After upgrade cacti/cactid - poller not processing all RRDs

mgeni · Cacti User Joined: 25 Mar 2005 Posts: 65

I have similar problem, any Solutions?

Tomeq · Joined: 05 Dec 2002 Posts: 21

yardus9 · Cacti User Joined: 27 Jan 2004 Posts: 77

I am having a similar problem with spine.

I have 2 cacti servers, both CentOS. I recently upgraded one from 0.86i with cactid to 0.87 with spine0.87b without a problem.
However, after I upgraded my second server, when I switch from cactid to spine (by changing the path), I get this problem where some of my RRDs fail to update. This second server has the plugin architecture installed, the first one did not. I have changed the LIMIT in lib/poller.php to 50000, and my php.ini memory lilmit is 64MB.
I rebuilt the poller cache, did not seem to help.

Here is a look at my cacti stats, polling looks normal, however, the number of updated RRDs is short by about 1500 when I use the spine poller:

01/11/2008 04:05:46 PM - SYSTEM STATS: Time:43.3462 Method:spine Processes:4 Threads:10 Hosts:441 HostsPerProcess:111 DataSources:13378 RRDsProcessed:8328
01/11/2008 04:06:02 PM - RECACHE STATS: RecacheTime:15.0207 HostsRecached:2
01/11/2008 04:10:41 PM - SYSTEM STATS: Time:39.0059 Method:spine Processes:4 Threads:10 Hosts:441 HostsPerProcess:111 DataSources:13378 RRDsProcessed:8328
01/11/2008 04:10:55 PM - RECACHE STATS: RecacheTime:13.7780 HostsRecached:3
01/11/2008 04:19:59 PM - SYSTEM STATS: Time:295.2987 Method:spine Processes:4 Threads:10 Hosts:441 HostsPerProcess:111 DataSources:13378 RRDsProcessed:6721
01/11/2008 04:20:17 PM - RECACHE STATS: RecacheTime:18.1796 HostsRecached:2
01/11/2008 04:24:59 PM - SYSTEM STATS: Time:295.2255 Method:spine Processes:4 Threads:10 Hosts:441 HostsPerProcess:111 DataSources:13377 RRDsProcessed:6721
01/11/2008 04:25:37 PM - SYSTEM STATS: Time:33.4695 Method:spine Processes:4 Threads:10 Hosts:441 HostsPerProcess:111 DataSources:13378 RRDsProcessed:8317
01/11/2008 04:25:49 PM - RECACHE STATS: RecacheTime:11.8979 HostsRecached:2

The poll seems to fail on the more recently added hosts. When i run spine on one of these hosts from the command line, it runs fine.

Any ideas??
Would this patch be applicable on this version:
http://www.cacti.net/downloads/patches/0.8.6i/poller_output_remainder.patch

khufure · Posted: Sat Jan 26, 2008 5:38 pm Post subject:

Not sure if it applies to all but I was able to resolve a similar problem using http://forums.cacti.net/viewtopic.php?p=124130#124130.

Regards,
P

yardus9 · Cacti User Joined: 27 Jan 2004 Posts: 77

I tried both the suggestions from that Post, neither seems to have solved my problem.

Anybody else have any ideas?

I ran spine from the command line via strace. Can't really make heads or tails of the results. Seems to be cruising along, then tons of this:
nanosleep({0, 400000}, NULL) = 0
nanosleep({0, 400000}, NULL) = 0
gettimeofday({1201806333, 641847}, NULL) = 0
nanosleep({0, 400000}, NULL) = 0
nanosleep({0, 400000}, NULL) = 0
nanosleep({0, 400000}, NULL) = 0
nanosleep({0, 400000}, NULL) = 0
nanosleep({0, 400000}, NULL)

leonk · Joined: 20 Jun 2007 Posts: 2

yardus9 · Cacti User Joined: 27 Jan 2004 Posts: 77

I just built a new box from scratch.
CentOS5
cacti-0.8.7b
spine0.8.7b
mysql-5.0.22
php-5.1.6

Imported my database, and am still getting the same problem, over half my RRDs are not getting updated.

03/13/2008 06:49:56 PM - SYSTEM STATS: Time:295.0487 Method:spine Processes:4 Threads:10 Hosts:461 HostsPerProcess:116 DataSources:14676 RRDsProcessed:5967
03/13/2008 06:49:59 PM - RECACHE STATS: RecacheTime:3.1101 HostsRecached:1

I am getting a ton of these errors in the DEBUG log:
03/13/2008 06:42:57 PM - CMDPHP: Poller[0] DEBUG: SQL Assoc: "select poller_id,end_time from poller_time where poller_id=0"
03/13/2008 06:42:57 PM - CMDPHP: Poller[0] DEBUG: SQL Assoc: "select poller_output.output, poller_output.time, poller_output.local_data_id, poller_item.rrd_path, poller_item.rrd_name, poller_item.rrd_num from (poller_output,poller_item) where (poller_output.local_data_id=poller_item.local_data_id and poller_output.rrd_name=poller_item.rrd_name) LIMIT 10000"

I had the plugin architecture installed in a previous version while using this same database. Could that be causing a problem? Is there something in the database that I can strip out?

Any other ideas?

davromaniak · Joined: 31 Mar 2008 Posts: 2

matguitar · Joined: 07 May 2008 Posts: 1 Location: Münster

Hi,
I have the same problem: the gathering of the snmp data works perfect when collecting with cacti (0.8.7b) itself. But when switching to spine (0.8.7c) only a few results are being updated to the RRD DB so the graphs are looking quite ragged.
I'm running 64 bit Redhat 2.6.18-53.el5 #1 SMP x86_64 GNU/Linux
Please solve this problem!

gandalf

faman · Joined: 14 Jul 2008 Posts: 1 Location: Los Angeles, CA

Deleting all UCD 10 FS graphs actually helped me. Times went from constant timeouts after 50s from the poller, with the assosciated db insert errors, to sub 10s poller runs against 50 or so hosts in one of our networks. I did the same in another network with 87 hosts and poller times went from timout to about 20s per run.

These errors started happening when I went from 0.8.6j/cactid to cacti 0.8.7b and spine 0.8.7a. Under 0.8.6 I was using a 1 minute poll interval and didn't have issues.