Cacti (home)ForumsDocumentation
Cacti: offical forums and support
It is currently Sun May 28, 2017 3:48 am

All times are UTC - 5 hours




Post new topic Reply to topic  [ 44 posts ]  Go to page 1, 2, 3  Next
Author Message
 Post subject: After upgrade cacti/cactid - poller not processing all RRDs
PostPosted: Tue Nov 07, 2006 11:44 am 
Offline

Joined: Wed Apr 19, 2006 11:50 am
Posts: 29
Location: Las Vegas, NV
Hello,
I have a large deployment of cacti with over 150 hosts and 25,000 data sources. I just recently updated to Cacti 0.8.6i and Cactid 0.8.6i. Nothing has been working as well as it used to after the upgrade. I am noticing the NaN issue in my graphs and I've followed all the steps in the NaN troubleshooting guide as well. I do not notice anything unusual except for the fact that not all of my RRD's are being processed. I can see in the log DEBUG Cactid polling my devices and putting the output into the MySQL database but not updating the RRD with the output. Any ideas?

As you can see the RRDProcessed number goes up and down. This number should be over 9500:

11/07/2006 08:38:15 AM - SYSTEM STATS: Time:193.5153 Method:cactid Processes:4 Threads:16 Hosts:166 HostsPerProcess:42 DataSources:27216 RRDsProcessed:5971
11/07/2006 08:34:10 AM - SYSTEM STATS: Time:248.9619 Method:cactid Processes:4 Threads:16 Hosts:166 HostsPerProcess:42 DataSources:27216 RRDsProcessed:5992
11/07/2006 08:28:04 AM - SYSTEM STATS: Time:182.5217 Method:cactid Processes:4 Threads:16 Hosts:166 HostsPerProcess:42 DataSources:27216 RRDsProcessed:9479
11/07/2006 08:21:47 AM - SYSTEM STATS: Time:105.3721 Method:cactid Processes:4 Threads:16 Hosts:166 HostsPerProcess:42 DataSources:27216 RRDsProcessed:5965
11/07/2006 08:18:16 AM - SYSTEM STATS: Time:194.9623 Method:cactid Processes:4 Threads:16 Hosts:166 HostsPerProcess:42 DataSources:27216 RRDsProcessed:5819
11/07/2006 08:14:21 AM - SYSTEM STATS: Time:259.4138 Method:cactid Processes:4 Threads:16 Hosts:166 HostsPerProcess:42 DataSources:27216 RRDsProcessed:9346
11/07/2006 08:07:56 AM - SYSTEM STATS: Time:175.4117 Method:cactid Processes:4 Threads:16 Hosts:166 HostsPerProcess:42 DataSources:27216 RRDsProcessed:5555

Your help in this is greatly apprecaited, thank you.


Top
 Profile  
 
 Post subject:
PostPosted: Tue Nov 07, 2006 2:41 pm 
Offline

Joined: Wed Apr 19, 2006 11:50 am
Posts: 29
Location: Las Vegas, NV
Note: I have downgraded back to Cacti 0.8.6h and my problem is gone. I tried using Cactid 0.8.6i with my Cacti 0.8.6h installation and I do not experience the problem. Looks like it's just limited to Cacti 0.8.6i.

-Dan


Top
 Profile  
 
 Post subject:
PostPosted: Wed Nov 08, 2006 10:24 am 
Offline
Cacti User

Joined: Wed Sep 28, 2005 1:39 pm
Posts: 495
This appears to be the same problem I have:
http://forums.cacti.net/viewtopic.php?t=17433

However, cmd.php wasn't doing it for me and I could never get the older version of cactid to run. Hoping to get some support on fixing whatever this issue is before being forced to downgrade.


Top
 Profile  
 
 Post subject: identical problem
PostPosted: Fri Nov 10, 2006 6:05 am 
Offline
Cacti User

Joined: Sat Jan 15, 2005 4:52 pm
Posts: 76
Location: London, England
I just upgraded to 0.8.6i from h and have the same problem on Win32 - I have over 16000 rrd's and am now only getting 6000-9000 processed according to the logs. There are no timeouts occuring that I can see.

Time to restore my database and copy back the files it seems :-(

Anyone else come across this?

Rossco


Top
 Profile  
 
 Post subject:
PostPosted: Fri Nov 10, 2006 7:23 am 
Offline
Developer
User avatar

Joined: Tue May 14, 2002 5:08 pm
Posts: 14863
Location: MI, USA
We have identified this as a bug. I have corrected it in SVN (BRANCH_0_8_6). The two files of interest are:

<path_cacti>/poller.php
<path_cacti>/lib/poller.php

You can get to the Web based SVN by following the link below:

http://svn.cacti.net/cgi-bin/viewcvs.cg ... 8_6/cacti/

TheWitness

_________________
True understanding begins only when we realize how little we truly understand...

Life is an adventure, let yours begin with Cacti!

Author of MacTrack, Boost, CLog, SpikeKill, Platform RTM, DSStats, maintainer of Spine, lot's of unpublished work and most of Cacti's bugs.
_________________
Official Cacti Documentation
Gandalfs Official Debugging Help
Central Plugin Repository
Central Templates Repository


Top
 Profile  
 
 Post subject:
PostPosted: Fri Nov 10, 2006 7:31 am 
Offline
Developer
User avatar

Joined: Tue May 14, 2002 5:08 pm
Posts: 14863
Location: MI, USA
Please note, if you are using the Plugin Architecture, you should either wait for Cigamit to release his maintenance pack or perform a DIFF.

TheWitness

_________________
True understanding begins only when we realize how little we truly understand...

Life is an adventure, let yours begin with Cacti!

Author of MacTrack, Boost, CLog, SpikeKill, Platform RTM, DSStats, maintainer of Spine, lot's of unpublished work and most of Cacti's bugs.
_________________
Official Cacti Documentation
Gandalfs Official Debugging Help
Central Plugin Repository
Central Templates Repository


Top
 Profile  
 
 Post subject:
PostPosted: Fri Nov 10, 2006 8:28 am 
Offline
Cacti Guru User
User avatar

Joined: Fri Sep 19, 2003 8:36 am
Posts: 2326
Location: Sophia-Antipolis, France
Hi Larry

I've made a diff against 0.8.6i + plugin architecture 1.0, but I don't see any difference :
  • before
    Quote:
    11/10/2006 02:22:09 PM - SYSTEM STATS: Time:127.6496 Method:cactid Processes:4 Threads:25 Hosts:228 HostsPerProcess:57 DataSources:6822 RRDsProcessed:3406
  • after
    Quote:
    11/10/2006 02:22:09 PM - SYSTEM STATS: Time:127.6496 Method:cactid Processes:4 Threads:25 Hosts:228 HostsPerProcess:57 DataSources:6822 RRDsProcessed:3406
Does this bug affect everyone using Cacti 0.8.6i ?

_________________

HOWTOs :
Templates :


Top
 Profile  
 
 Post subject:
PostPosted: Fri Nov 10, 2006 8:32 am 
Offline
Cacti Guru User
User avatar

Joined: Fri Sep 19, 2003 8:36 am
Posts: 2326
Location: Sophia-Antipolis, France
I've seen your other post :oops:

TheWitness wrote:
This bug will only affect you if you have more than 10k Data Sources. So, it does not affect everyone. But it will affect larger sites.

My main Cacti server is still running 0.8.6h, so I'm fine...

_________________

HOWTOs :
Templates :


Top
 Profile  
 
 Post subject:
PostPosted: Fri Nov 10, 2006 8:33 am 
Offline
Cacti User

Joined: Sat Jan 15, 2005 4:52 pm
Posts: 76
Location: London, England
Hi

I also can see no difference (other than plugin architecture hooks) between the last svn and the poller.php from plugin architecture. I can see there is the following additional lines/modified lines in the "lib\poller.php"

-----------------------
function process_poller_output($rrdtool_pipe, $remainder = FALSE) {

if ($remainder) {
$limit = "";
}else{
$limit = "LIMIT 10000";
}
-----------------------

Having just made the modifications I can still see no significant difference - half of the rrds are still not being processed. Am just rebuiolding poller cache to see if it makes a difference.

Thanks for your help so far.

Rossco


Top
 Profile  
 
 Post subject:
PostPosted: Fri Nov 10, 2006 10:24 am 
Offline
Cacti User

Joined: Sat Jan 15, 2005 4:52 pm
Posts: 76
Location: London, England
Hi

I dread to think what I'm breaking by doing this, but by replacing the two poller.php files with two from my 0.8.6h install, I now seem to have everything working normally again.

Unless anything nasty happens I'll hang on like this until a patch comes out.

Cheers

Rossco

SYSTEM STATS: Time:61.0253 Method:cactid Processes:5 Threads:100 Hosts:316 HostsPerProcess:64 DataSources:47498 RRDsProcessed:16443


Top
 Profile  
 
 Post subject:
PostPosted: Fri Nov 10, 2006 3:24 pm 
Offline
Developer
User avatar

Joined: Tue May 14, 2002 5:08 pm
Posts: 14863
Location: MI, USA
Rossco,

If you take the two poller files from SVN, you should be Ok. Please note, there are two files. poller.php and lib/poller.php, you only noted lib/poller.php.

Larry

_________________
True understanding begins only when we realize how little we truly understand...

Life is an adventure, let yours begin with Cacti!

Author of MacTrack, Boost, CLog, SpikeKill, Platform RTM, DSStats, maintainer of Spine, lot's of unpublished work and most of Cacti's bugs.
_________________
Official Cacti Documentation
Gandalfs Official Debugging Help
Central Plugin Repository
Central Templates Repository


Top
 Profile  
 
 Post subject:
PostPosted: Sat Nov 18, 2006 7:30 am 
Offline

Joined: Tue May 24, 2005 2:50 am
Posts: 38
I have applied the patch but it seems that i have the same problem

11/18/2006 01:21:59 PM - SYSTEM STATS: Time:119.5609 Method:cactid Processes:2 Threads:5 Hosts:489 HostsPerProcess:245 DataSources:40374 RRDsProcessed:20030
11/18/2006 01:17:39 PM - SYSTEM STATS: Time:156.5171 Method:cactid Processes:2 Threads:5 Hosts:489 HostsPerProcess:245 DataSources:40374 RRDsProcessed:17289
11/18/2006 01:12:13 PM - SYSTEM STATS: Time:126.1268 Method:cactid Processes:2 Threads:5 Hosts:489 HostsPerProcess:245 DataSources:40374 RRDsProcessed:13170
11/18/2006 01:01:50 PM - SYSTEM STATS: Time:109.2950 Method:cactid Processes:2 Threads:5 Hosts:489 HostsPerProcess:245 DataSources:40374 RRDsProcessed:18813
11/18/2006 12:56:47 PM - SYSTEM STATS: Time:107.4840 Method:cactid Processes:2 Threads:5 Hosts:489 HostsPerProcess:245 DataSources:40374 RRDsProcessed:19608
11/18/2006 12:51:41 PM - SYSTEM STATS: Time:100.8533 Method:cactid Processes:2 Threads:5 Hosts:489 HostsPerProcess:245 DataSources:40374 RRDsProcessed:14910
11/18/2006 12:46:53 PM - SYSTEM STATS: Time:113.0835 Method:cactid Processes:2 Threads:5 Hosts:489 HostsPerProcess:245 DataSources:40374 RRDsProcessed:15473
11/18/2006 12:44:21 PM - SYSTEM STATS: Time:255.0518 Method:cactid Processes:2 Threads:5 Hosts:489 HostsPerProcess:245 DataSources:40374 RRDsProcessed:14577
11/18/2006 12:33:47 PM - SYSTEM STATS: Time:227.6766 Method:cactid Processes:2 Threads:5 Hosts:489 HostsPerProcess:245 DataSources:40374 RRDsProcessed:14100
11/18/2006 12:26:39 PM - SYSTEM STATS: Time:98.9665 Method:cactid Processes:2 Threads:5 Hosts:489 HostsPerProcess:245 DataSources:40374 RRDsProcessed:11509
11/18/2006 12:21:40 PM - SYSTEM STATS: Time:100.1794 Method:cactid Processes:2 Threads:5 Hosts:489 HostsPerProcess:245 DataSources:40374 RRDsProcessed:14311

Is there something to do after patching ?

Thanks,


Top
 Profile  
 
 Post subject:
PostPosted: Sat Nov 18, 2006 7:50 am 
Offline
Developer
User avatar

Joined: Tue May 14, 2002 5:08 pm
Posts: 14863
Location: MI, USA
increase your php memory and then edit you lib/poller.php. Look for a line "LIMIT 10000" and change to "LIMIT 50000". See if that helps.

TheWitness

_________________
True understanding begins only when we realize how little we truly understand...

Life is an adventure, let yours begin with Cacti!

Author of MacTrack, Boost, CLog, SpikeKill, Platform RTM, DSStats, maintainer of Spine, lot's of unpublished work and most of Cacti's bugs.
_________________
Official Cacti Documentation
Gandalfs Official Debugging Help
Central Plugin Repository
Central Templates Repository


Top
 Profile  
 
 Post subject:
PostPosted: Sat Nov 18, 2006 12:36 pm 
Offline

Joined: Tue May 24, 2005 2:50 am
Posts: 38
I have increased php memory to 64MB and LIMIT to 50000 but i still have the same problem


Top
 Profile  
 
 Post subject:
PostPosted: Sat Nov 18, 2006 12:50 pm 
Offline
Developer
User avatar

Joined: Tue May 14, 2002 5:08 pm
Posts: 14863
Location: MI, USA
Do you have the 1 minute poller interval patch installed? You can tell by going to Settings->Poller and looking for a Poller Interval dropdown.

TheWitness

_________________
True understanding begins only when we realize how little we truly understand...

Life is an adventure, let yours begin with Cacti!

Author of MacTrack, Boost, CLog, SpikeKill, Platform RTM, DSStats, maintainer of Spine, lot's of unpublished work and most of Cacti's bugs.
_________________
Official Cacti Documentation
Gandalfs Official Debugging Help
Central Plugin Repository
Central Templates Repository


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 44 posts ]  Go to page 1, 2, 3  Next

All times are UTC - 5 hours


Who is online

Users browsing this forum: No registered users and 2 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  

Protected by Anti-Spam ACP Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group