Cacti (home)ForumsDocumentation
Cacti: offical forums and support
It is currently Mon Nov 19, 2018 9:30 am

All times are UTC - 5 hours




Post new topic Reply to topic  [ 9 posts ] 
Author Message
 Post subject: Great Remote Poller Performance of Cacti 1.2
PostPosted: Thu Oct 18, 2018 5:30 am 
Offline
Cacti Guru User
User avatar

Joined: Thu Jan 31, 2008 6:39 am
Posts: 2698
Location: Kressbronn, Germany
Just a remark on the Poller Performance of remote running pollers ... It's great !

The attached screenshot shows some of my pollers. All based on Raspberry Pis running the apt-get install of Cacti 1.1.38 upgraded to 1.2 from the Master.

As a side note: Make sure to have the master and remote pollers be as identical as possible. I'm running a CentOS 7 system as Master which has been isntalled from source and the Raspberry Pis are on Debian with the apt-get version.

The special setup of the apt-get install introduces a lot of issues for remote pollers. One is the symlink of css files which seem to force the poller to permanently trying to update these files causing up to 20 seconds delay on very far sites (ping > 250 ms).

All polling on the remote systems are from the mURLin plugin (php script server based) which has been changed to always use the local database for caching.


Attachments:
Remote_Pollers.png
Remote_Pollers.png [ 261.09 KiB | Viewed 190 times ]

_________________
Greetings,
Phalek
---
Need more help ? Read the Cacti documentation or my new Cacti 1.x Book
Need on-site support ? Look here Cacti Workshop
Need professional Cacti support ? Look here CereusService
---
Plugins : CereusTransporter | CereusReporting | nmidWebService | nmidSmokeping | nmidWeb2 |

Code:
CereusServer Master:  SYSTEM STATS: Time:2.5621 Method:spine Processes:1 Threads:16 Hosts:446 HostsPerProcess:446 DataSources:14683 RRDsProcessed:7573
CereusServer Agent:   SYSTEM STATS: Time:27.4840 Method:spine Processes:1 Threads:8 Hosts:16 HostsPerProcess:16 DataSources:114576 RRDsProcessed:48061
Top
 Profile  
 
 Post subject: Re: Great Remote Poller Performance of Cacti 1.2
PostPosted: Thu Oct 18, 2018 9:37 am 
Offline
Cacti Guru User

Joined: Sun Aug 27, 2017 12:05 am
Posts: 1694
Very nice! Speaking of the mURLin plugin, have you committed that yet?

_________________
Official Cacti Developer

Cacti Resources:
Cacti Website (including releases)
Cacti Issues
Cacti Development Releases
Cacti Development Documentation

My resources:
How to submit Pull Requests
Development Wiki and How To's
Updated NetSNMP Memory template for Cacti 1.x
Cisco SFP template for Cacti 0.8.8


Top
 Profile  
 
 Post subject: Re: Great Remote Poller Performance of Cacti 1.2
PostPosted: Tue Oct 23, 2018 11:14 pm 
Offline
Developer

Joined: Thu Apr 07, 2005 3:29 pm
Posts: 2727
Location: B/CS Texas
Nice. An interesting tidbit is some of my early testing on the remote poller was with Raspberry PIs, as it was easier to detect what changes actually had a performance impact.

_________________
Report a bug
Download Releases
1.X Compatible Plugins


Top
 Profile  
 
 Post subject: Re: Great Remote Poller Performance of Cacti 1.2
PostPosted: Thu Oct 25, 2018 9:23 am 
Offline
Cacti Guru User
User avatar

Joined: Thu Jan 31, 2008 6:39 am
Posts: 2698
Location: Kressbronn, Germany
Hm ... one additional idea would be to actually start the polling only if the database sync and any file syncing has been completed.

Especially on very remote sites the file updates take a very long time and new pollers are being spawned all the way down the road making the system unresponsive.

Maybe a switch to manually enable/disable the actual polling (but leave the DB/File sync on) on the remote poller.
I think the current Enable/Disable functionality is a "Disable All" or "Enable All" thing, right ?

My total is 47 Raspberry Pi 3b+ system all over the world, so I can do a lot of testing here :-)

_________________
Greetings,
Phalek
---
Need more help ? Read the Cacti documentation or my new Cacti 1.x Book
Need on-site support ? Look here Cacti Workshop
Need professional Cacti support ? Look here CereusService
---
Plugins : CereusTransporter | CereusReporting | nmidWebService | nmidSmokeping | nmidWeb2 |

Code:
CereusServer Master:  SYSTEM STATS: Time:2.5621 Method:spine Processes:1 Threads:16 Hosts:446 HostsPerProcess:446 DataSources:14683 RRDsProcessed:7573
CereusServer Agent:   SYSTEM STATS: Time:27.4840 Method:spine Processes:1 Threads:8 Hosts:16 HostsPerProcess:16 DataSources:114576 RRDsProcessed:48061


Top
 Profile  
 
 Post subject: Re: Great Remote Poller Performance of Cacti 1.2
PostPosted: Fri Oct 26, 2018 7:13 am 
Offline
Cacti Guru User

Joined: Sun Aug 27, 2017 12:05 am
Posts: 1694
I think that would be wise, especially during upgrades. For example, a lot changes between 1.1.38 and 1.2, so syncing those changes means that not all files would be in alignment when a poller is kicked off if it's only got a 64k link (for want of the worst kind of example). I'm not too sure how the poller currently handles that situation, still learning all it's nuances.

_________________
Official Cacti Developer

Cacti Resources:
Cacti Website (including releases)
Cacti Issues
Cacti Development Releases
Cacti Development Documentation

My resources:
How to submit Pull Requests
Development Wiki and How To's
Updated NetSNMP Memory template for Cacti 1.x
Cisco SFP template for Cacti 0.8.8


Top
 Profile  
 
 Post subject: Re: Great Remote Poller Performance of Cacti 1.2
PostPosted: Mon Nov 05, 2018 7:16 am 
Offline
Cacti Guru User
User avatar

Joined: Thu Jan 31, 2008 6:39 am
Posts: 2698
Location: Kressbronn, Germany
For slow links it's quite easy: The poller is being called multiple times resulting in a complete system instability and unresponsiveness.

For upgrades, it's complicated. When I upgraed to the 1.2 Beta, most pollers just stopped working as the cli_check php file was missing. Then afterwards it was complaining about missing features.

Although the database-based syncing is a convenient way of getting the pollers in sync with the master, it does have it's downsides.

A customer of mine managed to make the master fail by adding a 1GB test file for I/O measurement to the cacti main directory. Cacti tried to add that to the database and as a result the mysql connection constantly crashed (max_packet_size ... ).


Right now I have a LOCAL remote poller which is taking minutes to start as it tries to put the files from the DB to the local file system. Still trying to find the root cause of this, but if I just could disable this and enable the syncing one-by-one for each poller, that would make the whole setup a lot more stable. ( i.e. only sync if something has changed, and "something has changed" is defined by me and not a md5 hash .. )

INFO: I'm not filing a bug, as all these issues may just come from running a Centos (from source install) Master with Debian (apt-get installed) based remote pollers ...

UPDATE: I've now added a new global swithc "Enable Remote File Sync" to the poller tab and added some code to the resource_cache_out which check that settings and prevents the system to go through the resouce distribution. Based on that it looks like the "md5sum_path" function is actually causing the very long delay. need further checking on this ...

_________________
Greetings,
Phalek
---
Need more help ? Read the Cacti documentation or my new Cacti 1.x Book
Need on-site support ? Look here Cacti Workshop
Need professional Cacti support ? Look here CereusService
---
Plugins : CereusTransporter | CereusReporting | nmidWebService | nmidSmokeping | nmidWeb2 |

Code:
CereusServer Master:  SYSTEM STATS: Time:2.5621 Method:spine Processes:1 Threads:16 Hosts:446 HostsPerProcess:446 DataSources:14683 RRDsProcessed:7573
CereusServer Agent:   SYSTEM STATS: Time:27.4840 Method:spine Processes:1 Threads:8 Hosts:16 HostsPerProcess:16 DataSources:114576 RRDsProcessed:48061


Top
 Profile  
 
 Post subject: Re: Great Remote Poller Performance of Cacti 1.2
PostPosted: Mon Nov 05, 2018 9:03 am 
Offline
Cacti Guru User

Joined: Sun Aug 27, 2017 12:05 am
Posts: 1694
I think a size limit should certainly be in place, extremely large files hitting the database/syncing down to the pollers are not ideal. However, that might affect sending RRA's around.

_________________
Official Cacti Developer

Cacti Resources:
Cacti Website (including releases)
Cacti Issues
Cacti Development Releases
Cacti Development Documentation

My resources:
How to submit Pull Requests
Development Wiki and How To's
Updated NetSNMP Memory template for Cacti 1.x
Cisco SFP template for Cacti 0.8.8


Top
 Profile  
 
 Post subject: Re: Great Remote Poller Performance of Cacti 1.2
PostPosted: Mon Nov 05, 2018 9:07 am 
Offline
Cacti Guru User
User avatar

Joined: Thu Jan 31, 2008 6:39 am
Posts: 2698
Location: Kressbronn, Germany
netniV wrote:
I think a size limit should certainly be in place, extremely large files hitting the database/syncing down to the pollers are not ideal. However, that might affect sending RRA's around.


-> The size could be checked against the max_package setting of mysql, as anything bigger then that will cause the mysql connection to fail anyway ... ?

_________________
Greetings,
Phalek
---
Need more help ? Read the Cacti documentation or my new Cacti 1.x Book
Need on-site support ? Look here Cacti Workshop
Need professional Cacti support ? Look here CereusService
---
Plugins : CereusTransporter | CereusReporting | nmidWebService | nmidSmokeping | nmidWeb2 |

Code:
CereusServer Master:  SYSTEM STATS: Time:2.5621 Method:spine Processes:1 Threads:16 Hosts:446 HostsPerProcess:446 DataSources:14683 RRDsProcessed:7573
CereusServer Agent:   SYSTEM STATS: Time:27.4840 Method:spine Processes:1 Threads:8 Hosts:16 HostsPerProcess:16 DataSources:114576 RRDsProcessed:48061


Top
 Profile  
 
 Post subject: Re: Great Remote Poller Performance of Cacti 1.2
PostPosted: Mon Nov 05, 2018 4:45 pm 
Offline
Cacti Guru User

Joined: Sun Aug 27, 2017 12:05 am
Posts: 1694
I would agree with that, with a warning about the issue in cacti.log.

_________________
Official Cacti Developer

Cacti Resources:
Cacti Website (including releases)
Cacti Issues
Cacti Development Releases
Cacti Development Documentation

My resources:
How to submit Pull Requests
Development Wiki and How To's
Updated NetSNMP Memory template for Cacti 1.x
Cisco SFP template for Cacti 0.8.8


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 9 posts ] 

All times are UTC - 5 hours


Who is online

Users browsing this forum: No registered users and 1 guest


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  

Protected by Anti-Spam ACP Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group