Great Remote Poller Performance of Cacti 1.2

Beta testing discussion area

Moderators: Moderators, Developers

Post Reply
Author
Message
User avatar
phalek
Cacti Guru User
Posts: 2710
Joined: Thu Jan 31, 2008 6:39 am
Location: Kressbronn, Germany
Contact:

Great Remote Poller Performance of Cacti 1.2

#1 Post by phalek » Thu Oct 18, 2018 5:30 am

Just a remark on the Poller Performance of remote running pollers ... It's great !

The attached screenshot shows some of my pollers. All based on Raspberry Pis running the apt-get install of Cacti 1.1.38 upgraded to 1.2 from the Master.

As a side note: Make sure to have the master and remote pollers be as identical as possible. I'm running a CentOS 7 system as Master which has been isntalled from source and the Raspberry Pis are on Debian with the apt-get version.

The special setup of the apt-get install introduces a lot of issues for remote pollers. One is the symlink of css files which seem to force the poller to permanently trying to update these files causing up to 20 seconds delay on very far sites (ping > 250 ms).

All polling on the remote systems are from the mURLin plugin (php script server based) which has been changed to always use the local database for caching.
Attachments
Remote_Pollers.png
Remote_Pollers.png (261.09 KiB) Viewed 435 times
Greetings,
Phalek
---
Need more help ? Read the Cacti documentation or my new Cacti 1.x Book
Need on-site support ? Look here Cacti Workshop
Need professional Cacti support ? Look here CereusService
---
Plugins : CereusTransporter | CereusReporting | nmidWebService | nmidSmokeping | nmidWeb2 |

Code: Select all

CereusServer Master:  SYSTEM STATS: Time:2.5621 Method:spine Processes:1 Threads:16 Hosts:446 HostsPerProcess:446 DataSources:14683 RRDsProcessed:7573
CereusServer Agent:   SYSTEM STATS: Time:27.4840 Method:spine Processes:1 Threads:8 Hosts:16 HostsPerProcess:16 DataSources:114576 RRDsProcessed:48061
[/size]

netniV
Cacti Guru User
Posts: 2686
Joined: Sun Aug 27, 2017 12:05 am

Re: Great Remote Poller Performance of Cacti 1.2

#2 Post by netniV » Thu Oct 18, 2018 9:37 am

Very nice! Speaking of the mURLin plugin, have you committed that yet?

cigamit
Developer
Posts: 2771
Joined: Thu Apr 07, 2005 3:29 pm
Location: B/CS Texas
Contact:

Re: Great Remote Poller Performance of Cacti 1.2

#3 Post by cigamit » Tue Oct 23, 2018 11:14 pm

Nice. An interesting tidbit is some of my early testing on the remote poller was with Raspberry PIs, as it was easier to detect what changes actually had a performance impact.

User avatar
phalek
Cacti Guru User
Posts: 2710
Joined: Thu Jan 31, 2008 6:39 am
Location: Kressbronn, Germany
Contact:

Re: Great Remote Poller Performance of Cacti 1.2

#4 Post by phalek » Thu Oct 25, 2018 9:23 am

Hm ... one additional idea would be to actually start the polling only if the database sync and any file syncing has been completed.

Especially on very remote sites the file updates take a very long time and new pollers are being spawned all the way down the road making the system unresponsive.

Maybe a switch to manually enable/disable the actual polling (but leave the DB/File sync on) on the remote poller.
I think the current Enable/Disable functionality is a "Disable All" or "Enable All" thing, right ?

My total is 47 Raspberry Pi 3b+ system all over the world, so I can do a lot of testing here :-)
Greetings,
Phalek
---
Need more help ? Read the Cacti documentation or my new Cacti 1.x Book
Need on-site support ? Look here Cacti Workshop
Need professional Cacti support ? Look here CereusService
---
Plugins : CereusTransporter | CereusReporting | nmidWebService | nmidSmokeping | nmidWeb2 |

Code: Select all

CereusServer Master:  SYSTEM STATS: Time:2.5621 Method:spine Processes:1 Threads:16 Hosts:446 HostsPerProcess:446 DataSources:14683 RRDsProcessed:7573
CereusServer Agent:   SYSTEM STATS: Time:27.4840 Method:spine Processes:1 Threads:8 Hosts:16 HostsPerProcess:16 DataSources:114576 RRDsProcessed:48061
[/size]

netniV
Cacti Guru User
Posts: 2686
Joined: Sun Aug 27, 2017 12:05 am

Re: Great Remote Poller Performance of Cacti 1.2

#5 Post by netniV » Fri Oct 26, 2018 7:13 am

I think that would be wise, especially during upgrades. For example, a lot changes between 1.1.38 and 1.2, so syncing those changes means that not all files would be in alignment when a poller is kicked off if it's only got a 64k link (for want of the worst kind of example). I'm not too sure how the poller currently handles that situation, still learning all it's nuances.

User avatar
phalek
Cacti Guru User
Posts: 2710
Joined: Thu Jan 31, 2008 6:39 am
Location: Kressbronn, Germany
Contact:

Re: Great Remote Poller Performance of Cacti 1.2

#6 Post by phalek » Mon Nov 05, 2018 7:16 am

For slow links it's quite easy: The poller is being called multiple times resulting in a complete system instability and unresponsiveness.

For upgrades, it's complicated. When I upgraed to the 1.2 Beta, most pollers just stopped working as the cli_check php file was missing. Then afterwards it was complaining about missing features.

Although the database-based syncing is a convenient way of getting the pollers in sync with the master, it does have it's downsides.

A customer of mine managed to make the master fail by adding a 1GB test file for I/O measurement to the cacti main directory. Cacti tried to add that to the database and as a result the mysql connection constantly crashed (max_packet_size ... ).


Right now I have a LOCAL remote poller which is taking minutes to start as it tries to put the files from the DB to the local file system. Still trying to find the root cause of this, but if I just could disable this and enable the syncing one-by-one for each poller, that would make the whole setup a lot more stable. ( i.e. only sync if something has changed, and "something has changed" is defined by me and not a md5 hash .. )

INFO: I'm not filing a bug, as all these issues may just come from running a Centos (from source install) Master with Debian (apt-get installed) based remote pollers ...

UPDATE: I've now added a new global swithc "Enable Remote File Sync" to the poller tab and added some code to the resource_cache_out which check that settings and prevents the system to go through the resouce distribution. Based on that it looks like the "md5sum_path" function is actually causing the very long delay. need further checking on this ...
Greetings,
Phalek
---
Need more help ? Read the Cacti documentation or my new Cacti 1.x Book
Need on-site support ? Look here Cacti Workshop
Need professional Cacti support ? Look here CereusService
---
Plugins : CereusTransporter | CereusReporting | nmidWebService | nmidSmokeping | nmidWeb2 |

Code: Select all

CereusServer Master:  SYSTEM STATS: Time:2.5621 Method:spine Processes:1 Threads:16 Hosts:446 HostsPerProcess:446 DataSources:14683 RRDsProcessed:7573
CereusServer Agent:   SYSTEM STATS: Time:27.4840 Method:spine Processes:1 Threads:8 Hosts:16 HostsPerProcess:16 DataSources:114576 RRDsProcessed:48061
[/size]

netniV
Cacti Guru User
Posts: 2686
Joined: Sun Aug 27, 2017 12:05 am

Re: Great Remote Poller Performance of Cacti 1.2

#7 Post by netniV » Mon Nov 05, 2018 9:03 am

I think a size limit should certainly be in place, extremely large files hitting the database/syncing down to the pollers are not ideal. However, that might affect sending RRA's around.

User avatar
phalek
Cacti Guru User
Posts: 2710
Joined: Thu Jan 31, 2008 6:39 am
Location: Kressbronn, Germany
Contact:

Re: Great Remote Poller Performance of Cacti 1.2

#8 Post by phalek » Mon Nov 05, 2018 9:07 am

netniV wrote:I think a size limit should certainly be in place, extremely large files hitting the database/syncing down to the pollers are not ideal. However, that might affect sending RRA's around.
-> The size could be checked against the max_package setting of mysql, as anything bigger then that will cause the mysql connection to fail anyway ... ?
Greetings,
Phalek
---
Need more help ? Read the Cacti documentation or my new Cacti 1.x Book
Need on-site support ? Look here Cacti Workshop
Need professional Cacti support ? Look here CereusService
---
Plugins : CereusTransporter | CereusReporting | nmidWebService | nmidSmokeping | nmidWeb2 |

Code: Select all

CereusServer Master:  SYSTEM STATS: Time:2.5621 Method:spine Processes:1 Threads:16 Hosts:446 HostsPerProcess:446 DataSources:14683 RRDsProcessed:7573
CereusServer Agent:   SYSTEM STATS: Time:27.4840 Method:spine Processes:1 Threads:8 Hosts:16 HostsPerProcess:16 DataSources:114576 RRDsProcessed:48061
[/size]

netniV
Cacti Guru User
Posts: 2686
Joined: Sun Aug 27, 2017 12:05 am

Re: Great Remote Poller Performance of Cacti 1.2

#9 Post by netniV » Mon Nov 05, 2018 4:45 pm

I would agree with that, with a warning about the issue in cacti.log.

Post Reply