Latency during DB Update

Post general support questions here that do not specifically fall into the Linux or Windows categories.

Moderators: Moderators, Developers

Post Reply
Author
Message
jhmiller3
Posts: 19
Joined: Mon Oct 24, 2011 9:50 am

Latency during DB Update

#1 Post by jhmiller3 » Fri Dec 20, 2019 3:56 pm

Cacti version 1.2.7 running on a VM (VMWare vsphere 6.7 U1) Red Hat 8.0 machine. Poller is cmd.php. After the poller runs (5 min. intervals) about 15 seconds after the application begins to write to the database we are seeing severe latency on the disk I/O. Has anyone else seen this and, if so, have you found a fix?

Thanks - John

jhmiller3
Posts: 19
Joined: Mon Oct 24, 2011 9:50 am

Re: Latency during DB Update

#2 Post by jhmiller3 » Thu Jan 02, 2020 4:35 pm

jovitit133 wrote:
Thu Jan 02, 2020 4:31 pm
jhmiller3 wrote:
Fri Dec 20, 2019 3:56 pm
Cacti version 1.2.7 running on a VM (VMWare vsphere 6.7 U1) Red Hat 8.0 machine. Poller is cmd.php. After the poller runs (5 min. intervals) about 15 seconds after the application begins to write to the database we are seeing severe latency on the disk I/O. Has anyone else seen this and, if so, have you found a fix?

Thanks - John
Hi buddy, any follow up on this? Facing kind of exact same situation.

Regard,
J. Smith
No, nothing yet. We are looking at the way the OS was installed, but with the holiday break we have nothing new to report. Will update if we find anything concrete. Thanks for the reply and let me know it you have any luck. Thanks - John

User avatar
TheWitness
Developer
Posts: 14855
Joined: Tue May 14, 2002 5:08 pm
Location: MI, USA
Contact:

Re: Latency during DB Update

#3 Post by TheWitness » Sun Jan 05, 2020 12:35 pm

Really not nearly enough information to go on here. You should post from statistics from your cacti log, also, you might want to turn on the slow query log to understand better what is going on.
True understanding begins only when we realize how little we truly understand...

Life is an adventure, let yours begin with Cacti!

Author of MacTrack, Boost, CLog, SpikeKill, Platform RTM, DSStats, maintainer of Spine, lot's of unpublished work and most of Cacti's bugs.
_________________
Official Cacti Documentation
GitHub Repository with Supported Plugins
Central Plugin Repository
Central Templates Repository


I'm still out there people. Getting excited for Cacti 1.2. I think it will be a great release.

jhmiller3
Posts: 19
Joined: Mon Oct 24, 2011 9:50 am

Re: Latency during DB Update

#4 Post by jhmiller3 » Mon Jan 06, 2020 12:31 pm

TheWitness wrote:
Sun Jan 05, 2020 12:35 pm
Really not nearly enough information to go on here. You should post from statistics from your cacti log, also, you might want to turn on the slow query log to understand better what is going on.
Really not sure what you are saying. There are no indications in the cacti.log file that show anything about this issue. It is an issue on the virtual servers and the latency issues are reported by the SAN, not Cacti. As to "slow query log" I don't know what means. If you could help me out I can try that to see if Cacti is reporting any currently unlogged errors.

Thanks - John

User avatar
TheWitness
Developer
Posts: 14855
Joined: Tue May 14, 2002 5:08 pm
Location: MI, USA
Contact:

Re: Latency during DB Update

#5 Post by TheWitness » Mon Jan 06, 2020 5:06 pm

You provided qualitative information, and what we need is quantitative.
True understanding begins only when we realize how little we truly understand...

Life is an adventure, let yours begin with Cacti!

Author of MacTrack, Boost, CLog, SpikeKill, Platform RTM, DSStats, maintainer of Spine, lot's of unpublished work and most of Cacti's bugs.
_________________
Official Cacti Documentation
GitHub Repository with Supported Plugins
Central Plugin Repository
Central Templates Repository


I'm still out there people. Getting excited for Cacti 1.2. I think it will be a great release.

jhmiller3
Posts: 19
Joined: Mon Oct 24, 2011 9:50 am

Re: Latency during DB Update

#6 Post by jhmiller3 » Mon Jan 06, 2020 5:14 pm

TheWitness wrote:
Mon Jan 06, 2020 5:06 pm
You provided qualitative information, and what we need is quantitative.
What more can I provide you? As I said, there is nothing in the cacti.log file to indicate any issues. All I am asking is has anyone running this version of Cacti in the virtual environment I specified seen any latency issues on their storage device and, if so, were you able to resolve it. I am not saying this is an issue with Cacti and, since there are no log entries to indicate a problem, I cannot provide you with anything from the logs. Please help me to help you to help me. I am not sure what you are asking for and, as I said, I do not know how to change the logging so if you can help me with that it would be greatly appreciated.

PLEASE NOTE - Cacti seems to be running fine. What we see is slowness (due to the disk latency in the SAN) on other VMs.

Thanks - John

cigamit
Developer
Posts: 2831
Joined: Thu Apr 07, 2005 3:29 pm
Location: B/CS Texas
Contact:

Re: Latency during DB Update

#7 Post by cigamit » Mon Jan 06, 2020 7:35 pm

The log generally isn't going to show much at its default setting, you would have to turn up the verbosity to start seeing any relevant.

Are the underlying SAN Disks platters or SSDs?
Are you utilizing "On-demand RRD Update Settings" (Settings >> Performance)?

That is normally how you combat Disk Latency. Be sure to use a Memory table for poller_output_boost. Basically queues up the data in memory, and will then write them all at once at a later time. It can also do on demand writing for individual graphs when viewed.

User avatar
Osiris
Cacti Pro User
Posts: 889
Joined: Mon Jan 05, 2015 10:10 am

Re: Latency during DB Update

#8 Post by Osiris » Mon Jan 06, 2020 8:25 pm

Per cigamit, disk latency is a killer.
Before history, there was a paradise, now dust.

jhmiller3
Posts: 19
Joined: Mon Oct 24, 2011 9:50 am

Re: Latency during DB Update

#9 Post by jhmiller3 » Tue Jan 07, 2020 9:42 am

cigamit wrote:
Mon Jan 06, 2020 7:35 pm
The log generally isn't going to show much at its default setting, you would have to turn up the verbosity to start seeing any relevant.

Are the underlying SAN Disks platters or SSDs?
Are you utilizing "On-demand RRD Update Settings" (Settings >> Performance)?

That is normally how you combat Disk Latency. Be sure to use a Memory table for poller_output_boost. Basically queues up the data in memory, and will then write them all at once at a later time. It can also do on demand writing for individual graphs when viewed.
The SAN is platter, NOT SSD. Currently not using "On-Demand Update Settings" but the question is this. If we use On-Demand does it just delay the issue to a later time?

As to the Memory table, where do you set that up? I looked at the settings after turning on the On-Demand and don't see anything like that. Basically, can you point me to something to guide me on this?

Thanks - John

User avatar
TheWitness
Developer
Posts: 14855
Joined: Tue May 14, 2002 5:08 pm
Location: MI, USA
Contact:

Re: Latency during DB Update

#10 Post by TheWitness » Tue Jan 07, 2020 9:49 am

First, the memory table is a bit dangerous. In the past, it was beneficial, but if you are using something like a recent MariaDB 10.2+, and you follow our recommendations, InnoDB is just fine. Additionally, there have been recent enhancements in version 1.2.24+ that improve performance.

Generally speaking, for large installs, you should use local PCIe or NVMe based flash, and then using daily rsync's to your SAN, and in general avoid network attached storage. However, boost is designed to basically minimize the impact of this slower disk on polling times. It's pretty fast too, but if you have more RRDfiles than you have disk cache, things start to slow down. In the past, I would say something like this; if you run the command below:

du -hs rra

If that number comes back and it's more than you have disk cache, you will need boost. I hope that makes sense.
True understanding begins only when we realize how little we truly understand...

Life is an adventure, let yours begin with Cacti!

Author of MacTrack, Boost, CLog, SpikeKill, Platform RTM, DSStats, maintainer of Spine, lot's of unpublished work and most of Cacti's bugs.
_________________
Official Cacti Documentation
GitHub Repository with Supported Plugins
Central Plugin Repository
Central Templates Repository


I'm still out there people. Getting excited for Cacti 1.2. I think it will be a great release.

Post Reply