How to increase performance of Spine polling time?

Post general support questions here that do not specifically fall into the Linux or Windows categories.

Moderators: Moderators, Developers

Post Reply
Author
Message
roxy
Posts: 17
Joined: Wed May 04, 2016 7:56 am

How to increase performance of Spine polling time?

#1 Post by roxy » Wed May 04, 2016 8:17 am

I am running Version 0.8.8a and Linux platform. I am running into an issue with high polling time and periodic gaps in my graphs. I have tried adjusting the number of threads and also tried to use multiple processes instead of one. But still there is no improvement on the total polling time. Do you have any suggestion for improving spine's performance - I have lots of devices to monitor in Cacti.

05/04/2016 03:09:26 PM - SYSTEM STATS: Time:264.7665 Method:spine Processes:24 Threads:20 Hosts:2262 HostsPerProcess:95 DataSources:254876 RRDsProcessed:92164
05/04/2016 03:05:27 PM - SYSTEM STATS: Time:324.9506 Method:spine Processes:24 Threads:20 Hosts:2262 HostsPerProcess:95 DataSources:254876 RRDsProcessed:92176
05/04/2016 02:58:41 PM - SYSTEM STATS: Time:218.8851 Method:spine Processes:24 Threads:20 Hosts:2262 HostsPerProcess:95 DataSources:254876 RRDsProcessed:92194
05/04/2016 02:53:43 PM - SYSTEM STATS: Time:221.7222 Method:spine Processes:24 Threads:20 Hosts:2262 HostsPerProcess:95 DataSources:254876 RRDsProcessed:92183
05/04/2016 02:48:38 PM - SYSTEM STATS: Time:216.6530 Method:spine Processes:24 Threads:20 Hosts:2262 HostsPerProcess:95 DataSources:254876 RRDsProcessed:92133
05/04/2016 02:43:41 PM - SYSTEM STATS: Time:219.5430 Method:spine Processes:24 Threads:20 Hosts:2262 HostsPerProcess:95 DataSources:254876 RRDsProcessed:92177
05/04/2016 02:39:56 PM - SYSTEM STATS: Time:294.3475 Method:spine Processes:24 Threads:20 Hosts:2262 HostsPerProcess:95 DataSources:254876 RRDsProcessed:92147
05/04/2016 02:33:47 PM - SYSTEM STATS: Time:225.3324 Method:spine Processes:24 Threads:20 Hosts:2262 HostsPerProcess:95 DataSources:254876 RRDsProcessed:92177


Cacti poller is scheduled every 5 mins. Please suggest how I can improve performance of spine and can reduce poller time so that I can add more devices in the system.

I am using a good configuration box here. 24 CPU, 6 Cores each, 64GB memory.

User avatar
phalek
Cacti Guru User
Posts: 2710
Joined: Thu Jan 31, 2008 6:39 am
Location: Kressbronn, Germany
Contact:

Re: How to increase performance of Spine polling time?

#2 Post by phalek » Wed May 04, 2016 8:22 am

You are probably hitting I/O limits.

Do you have SSD disks in place or "plain old harddisks" ?

You could look into boost to offload the rrd-updates.

You can also check if the script mentioned here helps you with the updating part: http://forums.cacti.net/viewtopic.php?f ... ol+wrapper

Check "top" for your io waits ...
Greetings,
Phalek
---
Need more help ? Read the Cacti documentation or my new Cacti 1.x Book
Need on-site support ? Look here Cacti Workshop
Need professional Cacti support ? Look here CereusService
---
Plugins : CereusTransporter | CereusReporting | nmidWebService | nmidSmokeping | nmidWeb2 |

Code: Select all

CereusServer Master:  SYSTEM STATS: Time:2.5621 Method:spine Processes:1 Threads:16 Hosts:446 HostsPerProcess:446 DataSources:14683 RRDsProcessed:7573
CereusServer Agent:   SYSTEM STATS: Time:27.4840 Method:spine Processes:1 Threads:8 Hosts:16 HostsPerProcess:16 DataSources:114576 RRDsProcessed:48061
[/size]

mikygee
Cacti User
Posts: 87
Joined: Thu Jul 30, 2009 3:13 am

Re: How to increase performance of Spine polling time?

#3 Post by mikygee » Wed May 04, 2016 4:58 pm

Hello,

I tend to have my rrd on a specific partition and mount that partition with the option noatime. This helps when files are read. In your case you might consider using a specific disk for your rrd files.
I don't use SSD drives because rrd tend to take space and ssd drives are more expensive.

Have you tried to increase the "The Maximum SNMP OID's Per SNMP Get Request" ?

You should also check if some devices are very slow to answer.
Maybe your cacti is not properly placed in your network ? If it's far from most devices then you'll have an extra overhead.

I monitor my I/O disk on my system.

Maybe it's not the disks but something else like the network card which receives a lot of little packets and interrupts the cpu all the time
Some info about my system
- Cacti: 0.8.8g / OpenBSD 5.8
- Third Party: Nginx 1.9.3p3 / PHP 5.6.11 / MariaDB 10.0.20 / NET-SNMP version 5.7.3 / RRDTool 1.4.9
- Plugins: Weathermap 0.97c / Settings 0.71 / Thold 0.5 / Flowview 1.1 / Realtime 0.5.2 / Predict 1.0

roxy
Posts: 17
Joined: Wed May 04, 2016 7:56 am

Re: How to increase performance of Spine polling time?

#4 Post by roxy » Thu May 05, 2016 5:09 am

I have already tried with higher "The Maximum SNMP OID's Per SNMP Get Request" but there is no improvement. RRD files are in local disk,

/dev/sda1 8063408 5442464 2211344 72% /
tmpfs 32982080 3205832 29776248 10% /dev/shm
/dev/sda6 2015824 1202856 710568 63% /home
/dev/sda5 4031680 2136088 1690792 56% /tmp
/dev/sda7 99150 5650 88380 7% /usr/app
/dev/sda2 8063408 5843136 1810672 77% /var


My Cacti is installed in /var/www/cacti and this is production system.

I have another system (dev/test) exactly similar setup with less number of devices (server with 2 CPU, 6 core each and 8 GB memory). It was taking around 190 - 210 Sec to complete poller and I installed boost to offload RRD update but I have not seen much improvement there as well.

05/05/2016 11:48:41 AM - SYSTEM STATS: Time:219.7454 Method:spine Processes:2 Threads:100 Hosts:1397 HostsPerProcess:699 DataSources:36567 RRDsProcessed:0
05/05/2016 11:43:26 AM - SYSTEM STATS: Time:204.4324 Method:spine Processes:2 Threads:100 Hosts:1397 HostsPerProcess:699 DataSources:36567 RRDsProcessed:0
05/05/2016 11:38:21 AM - SYSTEM STATS: Time:199.9224 Method:spine Processes:2 Threads:100 Hosts:1397 HostsPerProcess:699 DataSources:36567 RRDsProcessed:0
05/05/2016 11:33:28 AM - SYSTEM STATS: Time:206.3242 Method:spine Processes:2 Threads:100 Hosts:1397 HostsPerProcess:699 DataSources:36567 RRDsProcessed:0
05/05/2016 11:28:39 AM - SYSTEM STATS: Time:217.2716 Method:spine Processes:2 Threads:100 Hosts:1397 HostsPerProcess:699 DataSources:36567 RRDsProcessed:0
05/05/2016 11:23:17 AM - SYSTEM STATS: Time:194.6898 Method:spine Processes:2 Threads:100 Hosts:1397 HostsPerProcess:699 DataSources:36567 RRDsProcessed:0
05/05/2016 11:18:18 AM - SYSTEM STATS: Time:197.0485 Method:spine Processes:2 Threads:100 Hosts:1397 HostsPerProcess:699 DataSources:36567 RRDsProcessed:0


Boost plugin status:

Boost On Demand Updating: Idle
Total Data Sources: 36567
Pending Boost Records: 29302
Archived Boost Records: 0
Total Boost Records: 29302
Boost Storage Statistics
Database Engine: MEMORY
Current Boost Tables Size: 3 MBytes
Avg Bytes/Record: 91 Bytes
Max Record Length: 31 Bytes
Max Allowed Boost Table Size: 9 MBytes
Estimated Maximum Records: 99460 Records
Runtime Statistics
Last Start Time: 2016-05-05 11:48:42
Last Run Duration: 19 seconds (1% of update frequency)
RRD Updates: 32789
Peak Poller Memory: 14.04 MBytes
Detailed Runtime Timers: RRDUpdates:32789 TotalTime:20 get_records:0.92 results_cycle:18.63 rrd_filename_and_template:11.16 rrd_lastupdate:1.51 rrdupdate:1.56 delete:0.08 timer_overhead:~2
Max Poller Memory Allowed: 512 MBytes
Run Time Configuration
Update Frequency: 30 Minutes
Next Start Time: 2016-05-05 12:18:42
Maximum Records: 30000 Records
Maximum Allowed Runtime: 20 Minutes
Boost Server Details
Server Config Status: Disabled
Multiprocess Server: Multiple Process
Update Timeout: 2 Seconds
Server/Port: [email protected]
Authorized Update Web Servers: 127.0.0.1
RRDtool Binary Used: /usr/bin/rrdtool


Cacti is in local disk here as well.

/dev/sda1 20641788 10712324 8880824 55% /
tmpfs 4029856 0 4029856 0% /dev/shm
/dev/sda6 2063504 905368 1053316 47% /home
/dev/sda7 2063504 42304 1916380 3% /tmp
/dev/sda8 106031216 12809828 87835304 13% /tools
/dev/sda5 2063504 438944 1519740 23% /opt
/dev/sda2 6190692 2228884 3647336 38% /var


Before doing Boost installation on prod, I want to check improvement on this test environment first.

I have few big Weathermaps configured on Cacti. Is there a possibility that WM put extra load on poller?
(Cacti Version: 0.8.8a, Plugin Architecture Version: 3.1, Weathermap 0.97b)

User avatar
phalek
Cacti Guru User
Posts: 2710
Joined: Thu Jan 31, 2008 6:39 am
Location: Kressbronn, Germany
Contact:

Re: How to increase performance of Spine polling time?

#5 Post by phalek » Thu May 05, 2016 5:15 am

If boost doesn't give you much then it may really be network related. You can look into offloading the actual device polling to something like the multi pollen plugin
Greetings,
Phalek
---
Need more help ? Read the Cacti documentation or my new Cacti 1.x Book
Need on-site support ? Look here Cacti Workshop
Need professional Cacti support ? Look here CereusService
---
Plugins : CereusTransporter | CereusReporting | nmidWebService | nmidSmokeping | nmidWeb2 |

Code: Select all

CereusServer Master:  SYSTEM STATS: Time:2.5621 Method:spine Processes:1 Threads:16 Hosts:446 HostsPerProcess:446 DataSources:14683 RRDsProcessed:7573
CereusServer Agent:   SYSTEM STATS: Time:27.4840 Method:spine Processes:1 Threads:8 Hosts:16 HostsPerProcess:16 DataSources:114576 RRDsProcessed:48061
[/size]

roxy
Posts: 17
Joined: Wed May 04, 2016 7:56 am

Re: How to increase performance of Spine polling time?

#6 Post by roxy » Thu May 05, 2016 5:38 am

Thanks Phalek for your inputs.

I will try this plugin on test environment but Boost is already installed there so do I need to uninstall this first or Boost and Multi poller both can work together?

Any idea??

User avatar
phalek
Cacti Guru User
Posts: 2710
Joined: Thu Jan 31, 2008 6:39 am
Location: Kressbronn, Germany
Contact:

Re: How to increase performance of Spine polling time?

#7 Post by phalek » Thu May 05, 2016 5:41 am

Try without boost first. I didn't test this combination yet.
Greetings,
Phalek
---
Need more help ? Read the Cacti documentation or my new Cacti 1.x Book
Need on-site support ? Look here Cacti Workshop
Need professional Cacti support ? Look here CereusService
---
Plugins : CereusTransporter | CereusReporting | nmidWebService | nmidSmokeping | nmidWeb2 |

Code: Select all

CereusServer Master:  SYSTEM STATS: Time:2.5621 Method:spine Processes:1 Threads:16 Hosts:446 HostsPerProcess:446 DataSources:14683 RRDsProcessed:7573
CereusServer Agent:   SYSTEM STATS: Time:27.4840 Method:spine Processes:1 Threads:8 Hosts:16 HostsPerProcess:16 DataSources:114576 RRDsProcessed:48061
[/size]

roxy
Posts: 17
Joined: Wed May 04, 2016 7:56 am

Re: How to increase performance of Spine polling time?

#8 Post by roxy » Tue Jul 12, 2016 9:19 am

Hi Phalek,

I want to use Cacti Multipoller server 0.2.2b as it is compatible with my Cacti version as mentioned some forum. But I cannot see any download link to download this version.
Can you please guide me where I can download this version?

Can it be possible to configure multiple poller in a single server so that I can assign each poller to a class of network devices? This will help to complete polling time within 300 sec?

Is it possible to install Cacti multi-poller on Cacti 0.8.8a or I need to upgrade Cacti to 0.8.8d first to install this plugin?

User avatar
phalek
Cacti Guru User
Posts: 2710
Joined: Thu Jan 31, 2008 6:39 am
Location: Kressbronn, Germany
Contact:

Re: How to increase performance of Spine polling time?

#9 Post by phalek » Tue Jul 19, 2016 12:56 am

Go here for a list of all downloads: http://www.cacti-multipollerserver.de/i ... l?Itemid=0
Greetings,
Phalek
---
Need more help ? Read the Cacti documentation or my new Cacti 1.x Book
Need on-site support ? Look here Cacti Workshop
Need professional Cacti support ? Look here CereusService
---
Plugins : CereusTransporter | CereusReporting | nmidWebService | nmidSmokeping | nmidWeb2 |

Code: Select all

CereusServer Master:  SYSTEM STATS: Time:2.5621 Method:spine Processes:1 Threads:16 Hosts:446 HostsPerProcess:446 DataSources:14683 RRDsProcessed:7573
CereusServer Agent:   SYSTEM STATS: Time:27.4840 Method:spine Processes:1 Threads:8 Hosts:16 HostsPerProcess:16 DataSources:114576 RRDsProcessed:48061
[/size]

roxy
Posts: 17
Joined: Wed May 04, 2016 7:56 am

Re: How to increase performance of Spine polling time?

#10 Post by roxy » Mon Mar 06, 2017 10:02 am

I have recently built a test environment with 3 VM to implement Cacti Multi Poller installation so that I can see the performance. I faced lots of issue with poller time in my current system (Cacti single poller) as number of devices large and have heavy weathermaps.

I had below system with single poller (all in local system).
Cacti Version: 0.8.8a, Plugin Architecture Version: 3.1, MySQL 5.1.73

Now I have used clone of above in a new VM and then upgrade it to Cacti Version: 0.8.8d and installed Cacti Multi Poller v0.8.8d and configured all the VMs. Now I can see poller is working properly from all VMs and graphs are populated with data.

But I am facing an issue with graph tree. Graph tree view is broken after Cacti upgrade and installation of Multi-poller plugin. I am getting blank page when I am clicking on graph tree view.

Do you have any idea how can I fix this issue or how can I troubleshoot this issue?

Post Reply