After upgrade cacti/cactid - poller not processing all RRDs

Post support questions that directly relate to Linux/Unix operating systems.

Moderators: Moderators, Developers

Author
Message
aleto
Posts: 39
Joined: Wed May 25, 2005 3:57 am

#31 Post by aleto » Wed Nov 21, 2007 11:33 am

TheWitness wrote:What is your poller interval? What is your cron interval?

TheWitness
Hi,

Thanks for your reply.

My cron interval is 5 mins, my polling interval is 1 min

HTHH,

aleto

mgeni
Cacti User
Posts: 65
Joined: Fri Mar 25, 2005 8:12 am

#32 Post by mgeni » Fri Nov 30, 2007 4:48 pm

I have similar problem, any Solutions?

Tomeq
Posts: 24
Joined: Thu Dec 05, 2002 2:22 pm

#33 Post by Tomeq » Mon Dec 31, 2007 3:23 am

mgeni wrote:I have similar problem, any Solutions?
We had/and now have similar problem. It looks like that there is too much i/o's when updating the rrd files.

We had the problem, but switched to a more modern hardware and the problem was gone. For some time, until much more data sources and rrds arrived. Now, the server is writing rrds for 3-4 minutes (it is Xeon, with good and fast RAID set up) one minute more and the problem will back.


Greets,
Tomek

yardus9
Cacti User
Posts: 77
Joined: Tue Jan 27, 2004 5:32 pm

#34 Post by yardus9 » Wed Jan 16, 2008 12:00 pm

I am having a similar problem with spine.

I have 2 cacti servers, both CentOS. I recently upgraded one from 0.86i with cactid to 0.87 with spine0.87b without a problem.
However, after I upgraded my second server, when I switch from cactid to spine (by changing the path), I get this problem where some of my RRDs fail to update. This second server has the plugin architecture installed, the first one did not. I have changed the LIMIT in lib/poller.php to 50000, and my php.ini memory lilmit is 64MB.
I rebuilt the poller cache, did not seem to help.

Here is a look at my cacti stats, polling looks normal, however, the number of updated RRDs is short by about 1500 when I use the spine poller:

01/11/2008 04:05:46 PM - SYSTEM STATS: Time:43.3462 Method:spine Processes:4 Threads:10 Hosts:441 HostsPerProcess:111 DataSources:13378 RRDsProcessed:8328
01/11/2008 04:06:02 PM - RECACHE STATS: RecacheTime:15.0207 HostsRecached:2
01/11/2008 04:10:41 PM - SYSTEM STATS: Time:39.0059 Method:spine Processes:4 Threads:10 Hosts:441 HostsPerProcess:111 DataSources:13378 RRDsProcessed:8328
01/11/2008 04:10:55 PM - RECACHE STATS: RecacheTime:13.7780 HostsRecached:3
01/11/2008 04:19:59 PM - SYSTEM STATS: Time:295.2987 Method:spine Processes:4 Threads:10 Hosts:441 HostsPerProcess:111 DataSources:13378 RRDsProcessed:6721
01/11/2008 04:20:17 PM - RECACHE STATS: RecacheTime:18.1796 HostsRecached:2
01/11/2008 04:24:59 PM - SYSTEM STATS: Time:295.2255 Method:spine Processes:4 Threads:10 Hosts:441 HostsPerProcess:111 DataSources:13377 RRDsProcessed:6721
01/11/2008 04:25:37 PM - SYSTEM STATS: Time:33.4695 Method:spine Processes:4 Threads:10 Hosts:441 HostsPerProcess:111 DataSources:13378 RRDsProcessed:8317
01/11/2008 04:25:49 PM - RECACHE STATS: RecacheTime:11.8979 HostsRecached:2


The poll seems to fail on the more recently added hosts. When i run spine on one of these hosts from the command line, it runs fine.

Any ideas??
Would this patch be applicable on this version:
http://www.cacti.net/downloads/patches/ ... nder.patch

khufure
Cacti User
Posts: 203
Joined: Wed Oct 24, 2007 5:47 pm
Location: San Francisco, CA
Contact:

#35 Post by khufure » Sat Jan 26, 2008 5:38 pm

Not sure if it applies to all but I was able to resolve a similar problem using http://forums.cacti.net/viewtopic.php?p=124130#124130.

Regards,
P

yardus9
Cacti User
Posts: 77
Joined: Tue Jan 27, 2004 5:32 pm

#36 Post by yardus9 » Thu Jan 31, 2008 2:27 pm

I tried both the suggestions from that Post, neither seems to have solved my problem.

Anybody else have any ideas?

I ran spine from the command line via strace. Can't really make heads or tails of the results. Seems to be cruising along, then tons of this:
nanosleep({0, 400000}, NULL) = 0
nanosleep({0, 400000}, NULL) = 0
gettimeofday({1201806333, 641847}, NULL) = 0
nanosleep({0, 400000}, NULL) = 0
nanosleep({0, 400000}, NULL) = 0
nanosleep({0, 400000}, NULL) = 0
nanosleep({0, 400000}, NULL) = 0
nanosleep({0, 400000}, NULL)

leonk
Posts: 4
Joined: Wed Jun 20, 2007 2:16 am

#37 Post by leonk » Tue Feb 26, 2008 11:28 pm


I tried both the suggestions from that Post, neither seems to have solved my problem.

Anybody else have any ideas?

I ran spine from the command line via strace. Can't really make heads or tails of the results. Seems to be cruising along, then tons of this:
nanosleep({0, 400000}, NULL) = 0
nanosleep({0, 400000}, NULL) = 0
gettimeofday({1201806333, 641847}, NULL) = 0
nanosleep({0, 400000}, NULL) = 0
nanosleep({0, 400000}, NULL) = 0
nanosleep({0, 400000}, NULL) = 0
nanosleep({0, 400000}, NULL) = 0
nanosleep({0, 400000}, NULL)



ditto... Having the same problem.. Have tried all fixes mentioned. :(

yardus9
Cacti User
Posts: 77
Joined: Tue Jan 27, 2004 5:32 pm

#38 Post by yardus9 » Thu Mar 13, 2008 6:02 pm

I just built a new box from scratch.
CentOS5
cacti-0.8.7b
spine0.8.7b
mysql-5.0.22
php-5.1.6

Imported my database, and am still getting the same problem, over half my RRDs are not getting updated.

03/13/2008 06:49:56 PM - SYSTEM STATS: Time:295.0487 Method:spine Processes:4 Threads:10 Hosts:461 HostsPerProcess:116 DataSources:14676 RRDsProcessed:5967
03/13/2008 06:49:59 PM - RECACHE STATS: RecacheTime:3.1101 HostsRecached:1

I am getting a ton of these errors in the DEBUG log:
03/13/2008 06:42:57 PM - CMDPHP: Poller[0] DEBUG: SQL Assoc: "select poller_id,end_time from poller_time where poller_id=0"
03/13/2008 06:42:57 PM - CMDPHP: Poller[0] DEBUG: SQL Assoc: "select poller_output.output, poller_output.time, poller_output.local_data_id, poller_item.rrd_path, poller_item.rrd_name, poller_item.rrd_num from (poller_output,poller_item) where (poller_output.local_data_id=poller_item.local_data_id and poller_output.rrd_name=poller_item.rrd_name) LIMIT 10000"


I had the plugin architecture installed in a previous version while using this same database. Could that be causing a problem? Is there something in the database that I can strip out?

Any other ideas?

davromaniak
Posts: 2
Joined: Mon Mar 31, 2008 10:07 am

#39 Post by davromaniak » Thu Apr 03, 2008 5:07 am

yardus9 wrote:I just built a new box from scratch.
CentOS5
cacti-0.8.7b
spine0.8.7b
mysql-5.0.22
php-5.1.6

Imported my database, and am still getting the same problem, over half my RRDs are not getting updated.

03/13/2008 06:49:56 PM - SYSTEM STATS: Time:295.0487 Method:spine Processes:4 Threads:10 Hosts:461 HostsPerProcess:116 DataSources:14676 RRDsProcessed:5967
03/13/2008 06:49:59 PM - RECACHE STATS: RecacheTime:3.1101 HostsRecached:1

I am getting a ton of these errors in the DEBUG log:
03/13/2008 06:42:57 PM - CMDPHP: Poller[0] DEBUG: SQL Assoc: "select poller_id,end_time from poller_time where poller_id=0"
03/13/2008 06:42:57 PM - CMDPHP: Poller[0] DEBUG: SQL Assoc: "select poller_output.output, poller_output.time, poller_output.local_data_id, poller_item.rrd_path, poller_item.rrd_name, poller_item.rrd_num from (poller_output,poller_item) where (poller_output.local_data_id=poller_item.local_data_id and poller_output.rrd_name=poller_item.rrd_name) LIMIT 10000"


I had the plugin architecture installed in a previous version while using this same database. Could that be causing a problem? Is there something in the database that I can strip out?

Any other ideas?
I have the same problem.

I'm using cacti 0.8.7a and spine 0.8.7a on a Debian Etch, with PHP 5.2.0 and MySQL 5.0.32.

The SNMP requests are working very well, but only 452 out of 3006 RRD files are updated.

The RRD update worked when cacti was not updated, and worked under 0.8.6j version.

Regards.

matguitar
Posts: 1
Joined: Wed May 07, 2008 3:19 am
Location: Münster

Is there any news concerning this problem

#40 Post by matguitar » Wed May 07, 2008 3:31 am

Hi,
I have the same problem: the gathering of the snmp data works perfect when collecting with cacti (0.8.7b) itself. But when switching to spine (0.8.7c) only a few results are being updated to the RRD DB so the graphs are looking quite ragged.
I'm running 64 bit Redhat 2.6.18-53.el5 #1 SMP x86_64 GNU/Linux
Please solve this problem!

User avatar
gandalf
Developer
Posts: 22375
Joined: Thu Dec 02, 2004 2:46 am
Location: Muenster, Germany
Contact:

Re: Is there any news concerning this problem

#41 Post by gandalf » Fri May 09, 2008 7:44 am

matguitar wrote:Hi,
I have the same problem: the gathering of the snmp data works perfect when collecting with cacti (0.8.7b) itself. But when switching to spine (0.8.7c) only a few results are being updated to the RRD DB so the graphs are looking quite ragged.
I'm running 64 bit Redhat 2.6.18-53.el5 #1 SMP x86_64 GNU/Linux
Please solve this problem!
Hey matguitar! You're not really far away from me. We should have a beer instead of discussing boring IT stuff, then, hehe.
For me, spine 087c is broken. So I use 087b ATM. cmd.php is no option for you? Larry will address known problems, but this may still take some days
Reinhard

faman
Posts: 1
Joined: Mon Jul 14, 2008 5:10 pm
Location: Los Angeles, CA

#42 Post by faman » Mon Jul 14, 2008 5:14 pm

Deleting all UCD 10 FS graphs actually helped me. Times went from constant timeouts after 50s from the poller, with the assosciated db insert errors, to sub 10s poller runs against 50 or so hosts in one of our networks. I did the same in another network with 87 hosts and poller times went from timout to about 20s per run.

These errors started happening when I went from 0.8.6j/cactid to cacti 0.8.7b and spine 0.8.7a. Under 0.8.6 I was using a 1 minute poll interval and didn't have issues.

Aiyaya
Posts: 1
Joined: Tue Aug 30, 2011 3:47 am

Re: After upgrade cacti/cactid - poller not processing all R

#43 Post by Aiyaya » Tue Aug 30, 2011 4:30 am

I tried using Cactid 0.8.6i with my Cacti 0.8.6h installation and I do not experience the problem. Looks like it's just limited to Cacti 0.8.6i.

User avatar
gandalf
Developer
Posts: 22375
Joined: Thu Dec 02, 2004 2:46 am
Location: Muenster, Germany
Contact:

Re: After upgrade cacti/cactid - poller not processing all R

#44 Post by gandalf » Fri Sep 16, 2011 2:33 pm

086 should no longer be used, by no means
R.

Post Reply