After upgrade cacti/cactid - poller not processing all RRDs

dbrummer · Joined: 19 Apr 2006 Posts: 28

Hello,
I have a large deployment of cacti with over 150 hosts and 25,000 data sources. I just recently updated to Cacti 0.8.6i and Cactid 0.8.6i. Nothing has been working as well as it used to after the upgrade. I am noticing the NaN issue in my graphs and I've followed all the steps in the NaN troubleshooting guide as well. I do not notice anything unusual except for the fact that not all of my RRD's are being processed. I can see in the log DEBUG Cactid polling my devices and putting the output into the MySQL database but not updating the RRD with the output. Any ideas?

As you can see the RRDProcessed number goes up and down. This number should be over 9500:

11/07/2006 08:38:15 AM - SYSTEM STATS: Time:193.5153 Method:cactid Processes:4 Threads:16 Hosts:166 HostsPerProcess:42 DataSources:27216 RRDsProcessed:5971
11/07/2006 08:34:10 AM - SYSTEM STATS: Time:248.9619 Method:cactid Processes:4 Threads:16 Hosts:166 HostsPerProcess:42 DataSources:27216 RRDsProcessed:5992
11/07/2006 08:28:04 AM - SYSTEM STATS: Time:182.5217 Method:cactid Processes:4 Threads:16 Hosts:166 HostsPerProcess:42 DataSources:27216 RRDsProcessed:9479
11/07/2006 08:21:47 AM - SYSTEM STATS: Time:105.3721 Method:cactid Processes:4 Threads:16 Hosts:166 HostsPerProcess:42 DataSources:27216 RRDsProcessed:5965
11/07/2006 08:18:16 AM - SYSTEM STATS: Time:194.9623 Method:cactid Processes:4 Threads:16 Hosts:166 HostsPerProcess:42 DataSources:27216 RRDsProcessed:5819
11/07/2006 08:14:21 AM - SYSTEM STATS: Time:259.4138 Method:cactid Processes:4 Threads:16 Hosts:166 HostsPerProcess:42 DataSources:27216 RRDsProcessed:9346
11/07/2006 08:07:56 AM - SYSTEM STATS: Time:175.4117 Method:cactid Processes:4 Threads:16 Hosts:166 HostsPerProcess:42 DataSources:27216 RRDsProcessed:5555

Your help in this is greatly apprecaited, thank you.

dbrummer · Joined: 19 Apr 2006 Posts: 28

Note: I have downgraded back to Cacti 0.8.6h and my problem is gone. I tried using Cactid 0.8.6i with my Cacti 0.8.6h installation and I do not experience the problem. Looks like it's just limited to Cacti 0.8.6i.

-Dan

knobdy · Cacti User Joined: 28 Sep 2005 Posts: 495

This appears to be the same problem I have:
http://forums.cacti.net/viewtopic.php?t=17433

However, cmd.php wasn't doing it for me and I could never get the older version of cactid to run. Hoping to get some support on fixing whatever this issue is before being forced to downgrade.

Rossco · Posted: Fri Nov 10, 2006 6:05 am Post subject: identical problem

I just upgraded to 0.8.6i from h and have the same problem on Win32 - I have over 16000 rrd's and am now only getting 6000-9000 processed according to the logs. There are no timeouts occuring that I can see.

Time to restore my database and copy back the files it seems

Anyone else come across this?

Rossco

TheWitness · Posted: Fri Nov 10, 2006 7:23 am Post subject:

We have identified this as a bug. I have corrected it in SVN (BRANCH_0_8_6). The two files of interest are:

<path_cacti>/poller.php
<path_cacti>/lib/poller.php

You can get to the Web based SVN by following the link below:

http://svn.cacti.net/cgi-bin/viewcvs.cgi/branches/BRANCH_0_8_6/cacti/

TheWitness

TheWitness · Posted: Fri Nov 10, 2006 7:31 am Post subject:

Please note, if you are using the Plugin Architecture, you should either wait for Cigamit to release his maintenance pack or perform a DIFF.

TheWitness

fmangeant · Posted: Fri Nov 10, 2006 8:28 am Post subject:

Hi Larry

I've made a diff against 0.8.6i + plugin architecture 1.0, but I don't see any difference :

before

Quote:

11/10/2006 02:22:09 PM - SYSTEM STATS: Time:127.6496 Method:cactid Processes:4 Threads:25 Hosts:228 HostsPerProcess:57 DataSources:6822 RRDsProcessed:3406
after

Quote:

11/10/2006 02:22:09 PM - SYSTEM STATS: Time:127.6496 Method:cactid Processes:4 Threads:25 Hosts:228 HostsPerProcess:57 DataSources:6822 RRDsProcessed:3406

Does this bug affect everyone using Cacti 0.8.6i ?

fmangeant · Posted: Fri Nov 10, 2006 8:32 am Post subject:

I've seen your other post

Rossco · Posted: Fri Nov 10, 2006 8:33 am Post subject:

Hi

I also can see no difference (other than plugin architecture hooks) between the last svn and the poller.php from plugin architecture. I can see there is the following additional lines/modified lines in the "lib\poller.php"

-----------------------
function process_poller_output($rrdtool_pipe, $remainder = FALSE) {

if ($remainder) {
$limit = "";
}else{
$limit = "LIMIT 10000";
}
-----------------------

Having just made the modifications I can still see no significant difference - half of the rrds are still not being processed. Am just rebuiolding poller cache to see if it makes a difference.

Thanks for your help so far.

Rossco

Rossco · Posted: Fri Nov 10, 2006 10:24 am Post subject:

Hi

I dread to think what I'm breaking by doing this, but by replacing the two poller.php files with two from my 0.8.6h install, I now seem to have everything working normally again.

Unless anything nasty happens I'll hang on like this until a patch comes out.

Cheers

Rossco

SYSTEM STATS: Time:61.0253 Method:cactid Processes:5 Threads:100 Hosts:316 HostsPerProcess:64 DataSources:47498 RRDsProcessed:16443

TheWitness · Posted: Fri Nov 10, 2006 3:24 pm Post subject:

Rossco,

If you take the two poller files from SVN, you should be Ok. Please note, there are two files. poller.php and lib/poller.php, you only noted lib/poller.php.

Larry

Jeb · Joined: 24 May 2005 Posts: 38

I have applied the patch but it seems that i have the same problem

11/18/2006 01:21:59 PM - SYSTEM STATS: Time:119.5609 Method:cactid Processes:2 Threads:5 Hosts:489 HostsPerProcess:245 DataSources:40374 RRDsProcessed:20030
11/18/2006 01:17:39 PM - SYSTEM STATS: Time:156.5171 Method:cactid Processes:2 Threads:5 Hosts:489 HostsPerProcess:245 DataSources:40374 RRDsProcessed:17289
11/18/2006 01:12:13 PM - SYSTEM STATS: Time:126.1268 Method:cactid Processes:2 Threads:5 Hosts:489 HostsPerProcess:245 DataSources:40374 RRDsProcessed:13170
11/18/2006 01:01:50 PM - SYSTEM STATS: Time:109.2950 Method:cactid Processes:2 Threads:5 Hosts:489 HostsPerProcess:245 DataSources:40374 RRDsProcessed:18813
11/18/2006 12:56:47 PM - SYSTEM STATS: Time:107.4840 Method:cactid Processes:2 Threads:5 Hosts:489 HostsPerProcess:245 DataSources:40374 RRDsProcessed:19608
11/18/2006 12:51:41 PM - SYSTEM STATS: Time:100.8533 Method:cactid Processes:2 Threads:5 Hosts:489 HostsPerProcess:245 DataSources:40374 RRDsProcessed:14910
11/18/2006 12:46:53 PM - SYSTEM STATS: Time:113.0835 Method:cactid Processes:2 Threads:5 Hosts:489 HostsPerProcess:245 DataSources:40374 RRDsProcessed:15473
11/18/2006 12:44:21 PM - SYSTEM STATS: Time:255.0518 Method:cactid Processes:2 Threads:5 Hosts:489 HostsPerProcess:245 DataSources:40374 RRDsProcessed:14577
11/18/2006 12:33:47 PM - SYSTEM STATS: Time:227.6766 Method:cactid Processes:2 Threads:5 Hosts:489 HostsPerProcess:245 DataSources:40374 RRDsProcessed:14100
11/18/2006 12:26:39 PM - SYSTEM STATS: Time:98.9665 Method:cactid Processes:2 Threads:5 Hosts:489 HostsPerProcess:245 DataSources:40374 RRDsProcessed:11509
11/18/2006 12:21:40 PM - SYSTEM STATS: Time:100.1794 Method:cactid Processes:2 Threads:5 Hosts:489 HostsPerProcess:245 DataSources:40374 RRDsProcessed:14311

Is there something to do after patching ?

Thanks,

TheWitness · Posted: Sat Nov 18, 2006 7:50 am Post subject:

increase your php memory and then edit you lib/poller.php. Look for a line "LIMIT 10000" and change to "LIMIT 50000". See if that helps.

TheWitness

Jeb · Joined: 24 May 2005 Posts: 38

I have increased php memory to 64MB and LIMIT to 50000 but i still have the same problem

TheWitness · Posted: Sat Nov 18, 2006 12:50 pm Post subject:

Do you have the 1 minute poller interval patch installed? You can tell by going to Settings->Poller and looking for a Poller Interval dropdown.

TheWitness