Cacti (home)ForumsDocumentation
Cacti: offical forums and support
It is currently Tue Jun 18, 2019 12:24 pm

All times are UTC - 5 hours




Post new topic Reply to topic  [ 12 posts ] 
Author Message
 Post subject: Cacti not spawning enough poller processes
PostPosted: Mon Mar 18, 2019 9:06 am 
Offline

Joined: Mon Mar 18, 2019 8:42 am
Posts: 6
We've got a medium-sized setup that was just upgraded from 1.x to 1.2.x via EPEL, and I am trying to determine if this is a bug or a failure of some kind related to the upgrade (the installation is fairly new so blowing it away and rebuilding from scratch will not end the world).

Since the upgrade, the Device Threads setting (under Configuration->Device Defaults) seems to have become the hard limit for every type of poller. The most copies of cmd.php that I can catch running are six. Switching over to Spine, again... six total. This despite the fact that I've configured Spine to max at 8, and cmd.php should be running to 64. The box itself has plenty of capacity for this (four virtual cores on a host with 16 available) and load doesn't even reach 1.

Because of this the machine struggles to check even the 100 hosts it's been told to track, because it takes longer than a minute to run all those tests when it'll only run a small number of them at a time. The majority of them don't even incur a host up check, because the bulk of the tests are simply collecting interface availability metrics, and that test takes a maximum of 8 seconds. After refactoring that down from 12 seconds, the run generally completes in about 50-55 seconds.

Any suggestions? I've seen quite a bit of verbiage in the transients over the configuration items about things being done differently with 1.2.x, and I suspect it might be an oversight that the interface will only allow you to set a maximum of six device threads.


Top
 Profile  
 
 Post subject: Re: Cacti not spawning enough poller processes
PostPosted: Mon Mar 18, 2019 11:33 am 
Offline
Cacti Guru User

Joined: Sun Aug 27, 2017 12:05 am
Posts: 2499
These poller settings are now on the individual settings of the data collectors with the defaults being just for when new one are created.

_________________
Official Cacti Developer

Cacti Resources:
Cacti Website (including releases)
Cacti Issues
Cacti Development Releases
Cacti Development Documentation

My resources:
How to submit Pull Requests
Development Wiki and How To's
Updated NetSNMP Memory template for Cacti 1.x
Cisco SFP template for Cacti 0.8.8


Top
 Profile  
 
 Post subject: Re: Cacti not spawning enough poller processes
PostPosted: Mon Mar 18, 2019 12:55 pm 
Offline

Joined: Mon Mar 18, 2019 8:42 am
Posts: 6
Yeah, I understand that, but despite what's under the other tabs (specifically, the Poller tab sporting a value of 64 for default data collector processes as well as 32 threads for Spine-specific settings) neither Spine nor cmd.php seem interested in spawning more than six instances at once. Either the database schema upgrade failed in a weird way, or both spine and cmd.php are looking at the wrong ceiling limit, because a few seconds after the top of the minute I see this with cmd.php:

Code:
apache    2007     1  4 12:37 ?        00:00:00 /bin/php -q /usr/share/cacti/cmd.php --poller=1 --first=15 --last=532 --mibs
apache    2019     1  3 12:37 ?        00:00:00 /bin/php -q /usr/share/cacti/cmd.php --poller=1 --first=533 --last=549 --mibs
apache    2031     1  3 12:37 ?        00:00:00 /bin/php -q /usr/share/cacti/cmd.php --poller=1 --first=550 --last=566 --mibs
apache    2041     1  3 12:37 ?        00:00:00 /bin/php -q /usr/share/cacti/cmd.php --poller=1 --first=567 --last=583 --mibs
apache    2053     1  3 12:37 ?        00:00:00 /bin/php -q /usr/share/cacti/cmd.php --poller=1 --first=584 --last=600 --mibs
apache    2063     1  3 12:37 ?        00:00:00 /bin/php -q /usr/share/cacti/cmd.php --poller=1 --first=601 --last=616 --mibs


...and flipping the switch to using Spine, I see this:

Code:
apache    3667     1  0 12:39 ?        00:00:00 /usr/bin/spine -C /etc/spine.conf --poller 1 --first 15 --last 532 --mibs
apache    3683     1  0 12:39 ?        00:00:00 /usr/bin/spine -C /etc/spine.conf --poller 1 --first 533 --last 549 --mibs
apache    3700     1  0 12:39 ?        00:00:00 /usr/bin/spine -C /etc/spine.conf --poller 1 --first 550 --last 566 --mibs
apache    3716     1  0 12:39 ?        00:00:00 /usr/bin/spine -C /etc/spine.conf --poller 1 --first 567 --last 583 --mibs
apache    3727     1  0 12:39 ?        00:00:00 /usr/bin/spine -C /etc/spine.conf --poller 1 --first 584 --last 600 --mibs
apache    3747     1  0 12:39 ?        00:00:00 /usr/bin/spine -C /etc/spine.conf --poller 1 --first 601 --last 616 --mibs


Load average (somewhat predictably) isn't even hitting single digits, and only after some refactoring do these checks (the majority of which are pretty simple ping -A -c 5 -w 8 hostname tests) all manage to complete in under a minute for per-minute polling. After the upgrade to 1.2.2-1.el7 only about 2/3 of the RRDs were actually seeing updates because the tests just didn't get run. Everything I'm looking at seems to indicate that the things are just doing a maximum of six tasks simultaneously. More baffling, the logs are showing different figures still...

2019-03-18 12:47:59 - SYSTEM STATS: Time:56.8681 Method:spine Processes:10 Threads:1 Hosts:100 HostsPerProcess:10 DataSources:177 RRDsProcessed:172

2019-03-18 12:28:57 - SYSTEM STATS: Time:55.4773 Method:cmd.php Processes:10 Threads:N/A Hosts:100 HostsPerProcess:10 DataSources:177 RRDsProcessed:172

At no point am I seeing ten simultaneous anythings, and as far as I know they should be reporting 32 threads, not 1, and a bit more than just ten processes. So, something ain't right... and it's not a problem to blow away the db, but I figured at the very least I should leave a trail of bread crumbs for anyone else who might run face-first into the same issue.


Top
 Profile  
 
 Post subject: Re: Cacti not spawning enough poller processes
PostPosted: Mon Mar 18, 2019 1:32 pm 
Offline

Joined: Fri Mar 08, 2019 11:39 am
Posts: 8
10 is the number of processes you have configured in 'Data collection -> Data Collector'. If you select spine you can select the number of threads there too. Try 3 processes (low as recommended) and 10 threads. top -H will do to show the 30 threads


Top
 Profile  
 
 Post subject: Re: Cacti not spawning enough poller processes
PostPosted: Mon Mar 18, 2019 1:43 pm 
Offline
Cacti Guru User

Joined: Sun Aug 27, 2017 12:05 am
Posts: 2499
You don't have installs in different folders which are different versions? some peopel do that and forget to update their cron task.

_________________
Official Cacti Developer

Cacti Resources:
Cacti Website (including releases)
Cacti Issues
Cacti Development Releases
Cacti Development Documentation

My resources:
How to submit Pull Requests
Development Wiki and How To's
Updated NetSNMP Memory template for Cacti 1.x
Cisco SFP template for Cacti 0.8.8


Top
 Profile  
 
 Post subject: Re: Cacti not spawning enough poller processes
PostPosted: Tue Mar 19, 2019 6:59 am 
Offline

Joined: Mon Mar 18, 2019 8:42 am
Posts: 6
The installation was via packages, and the upgrade was also via packages, and both packages appear to have been sticking everything in the same place (and otherwise some custom scripts would have fallen right off).

Also, cmd.php ran enough processes to be useful before the upgrade. If something was using stale configuration files that would be one thing, but it would at least be using the previous version's settings.


Top
 Profile  
 
 Post subject: Re: Cacti not spawning enough poller processes
PostPosted: Tue Mar 19, 2019 8:19 am 
Offline
Cacti Guru User

Joined: Sun Aug 27, 2017 12:05 am
Posts: 2499
Code:
select id, name, processes, threads from poller;
select * from settings where name like '%threa%' or name like '%proces%';

_________________
Official Cacti Developer

Cacti Resources:
Cacti Website (including releases)
Cacti Issues
Cacti Development Releases
Cacti Development Documentation

My resources:
How to submit Pull Requests
Development Wiki and How To's
Updated NetSNMP Memory template for Cacti 1.x
Cisco SFP template for Cacti 0.8.8


Top
 Profile  
 
 Post subject: Re: Cacti not spawning enough poller processes
PostPosted: Tue Mar 19, 2019 6:14 pm 
Offline

Joined: Mon Mar 18, 2019 8:42 am
Posts: 6
After wiping the thing and repopulating it (I also took a moment to spin up a brand new maria instance, so... zero chance of stale database information), there's probably something really knackered in the EPEL packages for 1.2.2, because I am pretty sure this output means exactly what it looks like it means.

Code:
MariaDB [cacti]> select id, name, processes, threads from poller;
+----+-------------+-----------+---------+
| id | name        | processes | threads |
+----+-------------+-----------+---------+
|  1 | Main Poller |         1 |       1 |
+----+-------------+-----------+---------+
1 row in set (0.00 sec)

MariaDB [cacti]> select * from settings where name like '%threa%' or name like '%proces%';
+---------------------------------------------+-------------------------------+
| name                                        | value                         |
+---------------------------------------------+-------------------------------+
| concurrent_processes                        | 32                            |
| device_threads                              | 5                             |
| install_table_automation_processes          | automation_processes          |
| install_table_poller_output_boost_processes | poller_output_boost_processes |
| max_threads                                 | 32                            |
| process_leveling                            | on                            |
+---------------------------------------------+-------------------------------+
6 rows in set (0.00 sec)


Top
 Profile  
 
 Post subject: Re: Cacti not spawning enough poller processes
PostPosted: Tue Mar 19, 2019 8:58 pm 
Offline

Joined: Mon Mar 18, 2019 8:42 am
Posts: 6
...and after poking higher numbers under threads/processes, almost like magic, things run more than just one at a time.

This doesn't appear to be a problem that's limited to the EPEL packages, either. Nothing in the web GUI appears to affect these settings on an entirely separate instance I run at home, on Slackware, where everything was built up from source tarballs.


Top
 Profile  
 
 Post subject: Re: Cacti not spawning enough poller processes
PostPosted: Wed Mar 20, 2019 3:51 am 
Offline
Cacti Guru User

Joined: Sun Aug 27, 2017 12:05 am
Posts: 2499
The default values (in the settings table) will not affect existing collectors. Only those in the pollers table affect that. I updated my test poller from 1 / 1 to 2 / 3, spine kicked in 3 threads so the functionality is working.

_________________
Official Cacti Developer

Cacti Resources:
Cacti Website (including releases)
Cacti Issues
Cacti Development Releases
Cacti Development Documentation

My resources:
How to submit Pull Requests
Development Wiki and How To's
Updated NetSNMP Memory template for Cacti 1.x
Cisco SFP template for Cacti 0.8.8


Top
 Profile  
 
 Post subject: Re: Cacti not spawning enough poller processes
PostPosted: Wed Mar 20, 2019 10:02 am 
Offline

Joined: Mon Mar 18, 2019 8:42 am
Posts: 6
It seems like there's somewhat of a glaring interface problem here, since most people doing a fresh install are going to expect the settings for the pollers to be under Configuration->Settings->Poller, or at least for those to at least have some impact on operation. Perhaps it's time to consider using a negative value or some other obvious canary to make a poller grab the defaults instead of staying set to the 1/1 they inherited when they were instantiated. Especially considering that the majority of Cacti installations are probably still using a single poller, having its configuration in an entirely separate section of the interface is likely to be a source of continued confusion.


Top
 Profile  
 
 Post subject: Re: Cacti not spawning enough poller processes
PostPosted: Wed Mar 20, 2019 6:31 pm 
Offline
Cacti Guru User

Joined: Sun Aug 27, 2017 12:05 am
Posts: 2499
if you look at the help for the settings configuration, it clearly states that they are for default values only. When upgrading existing systems, the defaults are transferred to the poller table for any existing pollers.

_________________
Official Cacti Developer

Cacti Resources:
Cacti Website (including releases)
Cacti Issues
Cacti Development Releases
Cacti Development Documentation

My resources:
How to submit Pull Requests
Development Wiki and How To's
Updated NetSNMP Memory template for Cacti 1.x
Cisco SFP template for Cacti 0.8.8


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 12 posts ] 

All times are UTC - 5 hours


Who is online

Users browsing this forum: No registered users and 9 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  

Protected by Anti-Spam ACP Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group