Cacti (home)ForumsDocumentation
Cacti: offical forums and support
It is currently Fri Apr 26, 2019 11:39 am

All times are UTC - 5 hours




Post new topic Reply to topic  [ 9 posts ] 
Author Message
 Post subject: [SOLVED] Poller Table Not Empty - how to fix?
PostPosted: Tue Feb 05, 2019 6:51 pm 
Offline
Cacti User

Joined: Mon Oct 01, 2018 10:09 am
Posts: 112
Hello All,

I have been looking all over the internet, trying to find some
reasonable cure for the poller not empty warning in cacti.
I am using cacti-1.1.38 in a test system with 12 cores.

I have two devices in the test setup, a router with 3 datasources
and a linux box - monitoring disk space and processes.

I was running a cron job every 5 minutes, with the poller using spine, set to
run every 30 seconds. I changed the cron job to run every 10 minutes, just for fun.
I am running 8 processes with 10 threads

I still get poller table not empty warnings for just one device with 4 graphs. It is monitoring
traffic thru the interfaces, in/out bits. The warning reports either 1 or two issues about every 4 minutes:
POLLER: Poller[Main Poller] WARNING: Poller Output Table not Empty. Issues: 2, Graphs[sbc-52-880-router - Traffic - GigabitEthernet0/1/10101, sbc-52-880-router - Traffic - GigabitEthernet0/1/10101] Graphs[sbc-52-880-router - Traffic - 10.48.11.7 (Vlan11), sbc-52-880-router - Traffic - 10.48.11.7 (Vlan11)] DS[sbc-52-880-router - Traffic - Gi0/1, sbc-52-880-router - Traffic - 10.48.11.7 - Vl11]

I have a production version of cacti that continues to get 10s of 10s of these warnings. I have yet to discover the fix or the solution. Any tips will help.

thanks,
eholz1


Last edited by eholz1 on Fri Feb 15, 2019 11:05 am, edited 1 time in total.

Top
 Profile  
 
 Post subject: Re: The Poller Table Not Empty - how to fix?
PostPosted: Wed Feb 06, 2019 7:23 am 
Offline
Cacti Guru User

Joined: Sun Aug 27, 2017 12:05 am
Posts: 2419
To be clear, there are these intervals you are dealing with here:

  • Cron interval: How often the PHP script poller.php is called by the operating system.
  • Poller interval: The maximum length of time the PHP script poller.php should run

If Cron interval and poller interval are not the same, you will see warnings that they are out of sync. This is normal and should be corrected to be in sync.

Now, it's also my understanding that if you get too many devices timing out extending the collection past it's interval, this could lead to you experiencing issues were not everything was successfully polled. The best way to cure that is to put constantly problematic devices onto a longer polling cycle.

It does seem that you are struggling with your cacti installation (based on recent questions and which isn't surprising as it can appear to be quite complex when your thrown in at the deep end) so it may be worth hiring someone with more experience help you out over the short term until your experience/understanding grows.

_________________
Official Cacti Developer

Cacti Resources:
Cacti Website (including releases)
Cacti Issues
Cacti Development Releases
Cacti Development Documentation

My resources:
How to submit Pull Requests
Development Wiki and How To's
Updated NetSNMP Memory template for Cacti 1.x
Cisco SFP template for Cacti 0.8.8


Top
 Profile  
 
 Post subject: Re: The Poller Table Not Empty - how to fix?
PostPosted: Wed Feb 06, 2019 11:52 am 
Offline
Cacti User

Joined: Mon Oct 01, 2018 10:09 am
Posts: 112
Thank you for the reply - always food for thought.

It is definately clear that the relationship between the number of devices,
the number of datasources, etc becomes critical with the combination of the
cron time (in my case every 5 minutes) and the poller interval (i.e. 30 secs), and the
number of processes and threads for the poller. This means gaps or no gaps,
snmp timeouts, and poller table not empty warnings.

I can understand how we may see many of the warnings on one system which has
386 devices and 3546 datasources - currently the poller for this setup takes about
19.5 seconds (12 processes and 10 threads for spine).

But I am puzzled as to why a system with 2 devices ( the linux host, and a basic router), and 6 datasources
gives poller table not empty warnings. The cron value is 5 minutes, the poller is set to every minute.
The poller is using spine, with 12 processes and 10 threads.

I will create a datasource profile with a longer collection rate and see what that does.
(perhaps I have too many processes and threads for a system with only two devices?)
So I will try reducing processes and threads, etc.

Thanks,
eholz1
(down but not out)


Top
 Profile  
 
 Post subject: Re: The Poller Table Not Empty - how to fix?
PostPosted: Thu Feb 07, 2019 5:48 pm 
Offline
Cacti User

Joined: Mon Oct 01, 2018 10:09 am
Posts: 112
Hello,

I have found something out. If I use spine, cron 5 min, poller 5min, sys default profile,
I get poller table not empty warnings.

If I switch from spine to cmd.php - I get no poller table not empty errors!
This is with NO change to cron or the poller config via the web GUI.

What is different about cmd.php that it works, and if I use spine, I get the warnings?

Any suggestions on this? I will continue to search for cmd.php/spine comparisons.
Oh - the datasource/query used is "Interface Traffic" which is based on the interface.xml
query.

Thanks,
Eholz1


Top
 Profile  
 
 Post subject: Re: The Poller Table Not Empty - how to fix?
PostPosted: Thu Feb 07, 2019 5:58 pm 
Offline
Cacti Pro User
User avatar

Joined: Mon Jan 05, 2015 10:10 am
Posts: 753
There were some issues with the 1.2.1 release. I suggest you download the latest lib/template.php from the Cacti github and then re-index the hosts that were experiencing this issue. That should clear things up. Please look for an issue like this logged on GitHub. You may also have to update lib/functions.php.

_________________
Before history, there was a paradise, now dust.


Top
 Profile  
 
 Post subject: Re: The Poller Table Not Empty - how to fix?
PostPosted: Thu Feb 07, 2019 6:10 pm 
Offline
Cacti Pro User
User avatar

Joined: Mon Jan 05, 2015 10:10 am
Posts: 753
Sorry lib/data_query.php, lib/functions.php, lib/template.php

_________________
Before history, there was a paradise, now dust.


Top
 Profile  
 
 Post subject: Re: The Poller Table Not Empty - how to fix?
PostPosted: Fri Feb 08, 2019 12:00 pm 
Offline
Cacti User

Joined: Mon Oct 01, 2018 10:09 am
Posts: 112
Will Do, and Will Try, and will get back to you,
and thanks for the support!

I seem to still be getting poller table warnings, and got this error:
2019/02/08 17:23:23 - CMDPHP PHP ERROR NOTICE Backtrace:
2019/02/08 17:23:23 - ERROR PHP NOTICE: Use of undefined constant MESSAGE_LEVEL_NONE - assumed 'MESSAGE_LEVEL_NONE' in file: /var/www/cacti-1.1.38/lib/functions.php on line: 666

I have 365 devices, with 3474 datasources, running cron at 5 min, poller at 1, with 12 processes and 12 threads.
The cpu (a virtual machine) has 12 cores.

I get poller table not empty warnings at every poller start, then the stats

Log [Total Lines: 16 - Admin View - No Other Filter in Affect]

2019/02/08 17:27:17 - SYSTEM THOLD STATS: Time:0.0280 Tholds:2 TotalDevices:395 DownDevices:44 NewDownDevices:0
2019/02/08 17:27:16 - SYSTEM STATS: Time:16.5268 Method:spine Processes:12 Threads:12 Hosts:395 HostsPerProcess:33 DataSources:3474 RRDsProcessed:1751

2019/02/08 17:26:59 - POLLER: Poller[STL Main Poller] WARNING: Poller Output Table not Empty. Issues: 154, Graphs[msa-50-531-rtr3 - Traffic - Gi0/1, msa-50-531-rtr3 - Traffic - Gi0/1] Graphs[msa-50-531-rtr3 - Traffic - Gi0/2, msa-50-531-rtr3 - Traffic - Gi0/2] Graphs[msa-50-531-rtr3 - Traffic - Gi0/2, msa-50-531-rtr3 - Traffic - Gi0/2] Graphs[msa-50-531-rtr3 - Traffic - Gi0/0, msa-50-531-rtr3 - Traffic - Gi0/0] Graphs[msa-50-531-rtr3 - Traffic - Gi0/1, msa-50-531-rtr3 - Traffic - Gi0/1] Graphs[sea-9-53-rsw - Traffic - Te1/27, sea-9-53-rsw - Traffic - Te1/27] Graphs[sea-9-53-rsw - Traffic - Te1/27, sea-9-53-rsw - Traffic - Te1/27] Graphs[sea-9-53-rsw - Traffic - Te1/24, sea-9-53-rsw - Traffic - Te1/24] Graphs[sea-9-53-rsw - Traffic - Te1/24, sea-9-53-rsw - Traffic - Te1/24] Graphs[sea-9-53-rsw - Traffic - Te1/23, sea-9-53-rsw - Traffic - Te1/23] Graphs[sea-9-53-rsw - Traffic - Te1/23, sea-9-53-rsw - Traffic - Te1/23] Graphs[sea-9-53-rsw - Traffic - Te1/22, sea-9-53-rsw - Traffic - Te1/22] Graphs[sea-9-53-rsw - Traffic - Te1/22, sea-9-53-rsw - Traffic - Te1/22] Graphs[sea-9-53-rsw - Traffic - Te1/21, sea-9-53-rsw - Traffic - Te1/21] Graphs[sea-9-53-rsw - Traffic - Te1/21, sea-9-53-rsw - Traffic - Te1/21] Graphs[sea-9-53-rsw - Traffic - Te1/13, sea-9-53-rsw - Traffic - Te1/13] Graphs[sea-9-53-rsw - Traffic - Te1/13, sea-9-53-rsw - Traffic - Te1/13] Graphs[sea-9-53-rsw - Traffic - Te1/32, sea-9-53-rsw - Traffic - Te1/32] Graphs[sea-9-53-rsw - Traffic - Te1/32, sea-9-53-rsw - Traffic - Te1/32] Graphs[sea-9-53-rsw - Traffic - Te2/1, sea-9-53-rsw - Traffic - Te2/1] DS[msa-50-531-rtr3 - Traffic - Gi0/1, msa-50-531-rtr3 - Traffic - |query_ifIP| - Gi0/2, msa-50-531-rtr3 - Traffic - |query_ifIP| - Gi0/2, msa-50-531-rtr3 - Traffic - |query_ifIP| - Gi0/0, msa-50-531-rtr3 - Traffic - Gi0/1, sea-9-53-rsw - Traffic - Te1/27 to sea-9-08-asw, sea-9-53-rsw - Traffic - Te1/27 to sea-9-08-asw, sea-9-53-rsw - Traffic - Te1/24 Trunk to sea-9-, sea-9-53-rsw - Traffic - Te1/24 Trunk to sea-9-, sea-9-53-rsw - Traffic - Te1/23 Trunk to dist s, sea-9-53-rsw - Traffic - Te1/23 Trunk to dist s, sea-9-53-rsw - Traffic - Te1/22 to sea-13-03-es, sea-9-53-rsw - Traffic - Te1/22 to sea-13-03-es, sea-9-53-rsw - Traffic - Te1/21 to sea-9-53-rtr, sea-9-53-rsw - Traffic - Te1/21 to sea-9-53-rtr, sea-9-53-rsw - Traffic - Te1/13 Palo Alto LBX (, sea-9-53-rsw - Traffic - Te1/13 Palo Alto LBX (, sea-9-53-rsw - Traffic - Te1/32 to sea-9-53-asw, sea-9-53-rsw - Traffic - Te1/32 to sea-9-53-asw, sea-9-53-rsw - Traffic - Te2/1], Additional Issues Remain. Only showing first 20
2019/02/08 17:26:18 - SYSTEM THOLD STATS: Time:0.0277 Tholds:2 TotalDevices:395 DownDevices:44 NewDownDevices:0
2019/02/08 17:26:17 - SYSTEM STATS: Time:16.5436 Method:spine Processes:12 Threads:12 Hosts:395 HostsPerProcess:33 DataSources:3474 RRDsProcessed:1751


sorry for all the garbage!

eholz1
"A cacti in time polls 3550"


Top
 Profile  
 
 Post subject: Re: The Poller Table Not Empty - how to fix?
PostPosted: Sun Feb 10, 2019 2:54 pm 
Offline
Cacti Pro User
User avatar

Joined: Mon Jan 05, 2015 10:10 am
Posts: 753
Re-index after updating files.

_________________
Before history, there was a paradise, now dust.


Top
 Profile  
 
 Post subject: Re: The Poller Table Not Empty - how to fix?
PostPosted: Thu Feb 14, 2019 6:02 pm 
Offline
Cacti User

Joined: Mon Oct 01, 2018 10:09 am
Posts: 112
Again,
Thanks for the support.
I will assume by "re-index" you mean run the
poller_reindex_hosts.php in the cacti/cli directory.

Will try.

Thanks

eholz1


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 9 posts ] 

All times are UTC - 5 hours


Who is online

Users browsing this forum: Google [Bot] and 6 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  

Protected by Anti-Spam ACP Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group