rrdtool update not running

Post support questions that directly relate to Linux/Unix operating systems.

Moderators: Moderators, Developers

Author
Message
tdkevin
Posts: 20
Joined: Wed Aug 11, 2010 12:27 pm

rrdtool update not running

#1 Post by tdkevin » Thu Feb 03, 2011 12:51 pm

Hello,

I have recently done a migration of my cacti server to a new machine. Everything has migrated fine, except now I notice my RRD graphs are no longer updating.

I have followed the troubleshooting documentation and everything (permissions, mysql, etc) checked out EXCEPT i cannot see any "rrdtool update" statements, as one of the steps on the troubleshooting documentation says to check the cacti.log for that :

Code: Select all

# cat cacti.log | grep -i rrdtool | grep -i update
#
So if I look in the poller cache for the paritcular RRD graph not updating (all of them are not updating), I can see that data is being pulled, but no data is being input in to the graphs :

Code: Select all

Data Source Debug

/usr/bin/rrdtool create \
/usr/local/cacti-0.8.7g/rra/web2-1_cpu_378.rrd \
--step 300  \
DS:cpu:GAUGE:600:0:100 \
RRA:AVERAGE:0.5:1:600 \
RRA:AVERAGE:0.5:6:700 \
RRA:AVERAGE:0.5:24:775 \
RRA:AVERAGE:0.5:288:797 \
RRA:MAX:0.5:1:600 \
RRA:MAX:0.5:6:700 \
RRA:MAX:0.5:24:775 \
RRA:MAX:0.5:288:797 \
Does anyone have suggestions? The troubleshooting documentation at docs.cacti.net don't mention what you should do next if rrdtool isnt updating , after checking the cacti.log file. Not sure where to go from here, so any help would be greatly appreciated!

Thanks

tdkevin
Posts: 20
Joined: Wed Aug 11, 2010 12:27 pm

Re: rrdtool update not running

#2 Post by tdkevin » Thu Feb 03, 2011 12:52 pm

I should note that all the graphs have NAN values.

tdkevin
Posts: 20
Joined: Wed Aug 11, 2010 12:27 pm

Re: rrdtool update not running

#3 Post by tdkevin » Thu Feb 03, 2011 1:03 pm

Here's an example of a blank graph :

Code: Select all

RRDTool Command:

/usr/bin/rrdtool graph - \
--imgformat=PNG \
--start=-86400 \
--end=-300 \
--title='TOPSMEDIA1 - CPU Usage' \
--rigid \
--base=1000 \
--height=120 \
--width=500 \
--alt-autoscale-max \
--lower-limit=0 \
--vertical-label='percent' \
--slope-mode \
--font TITLE:12: \
--font AXIS:8: \
--font LEGEND:10: \
--font UNIT:8: \
DEF:a="/usr/local/cacti-0.8.7g/rra/topsmedia1_cpu_system_705.rrd":cpu_system:AVERAGE \
DEF:b="/usr/local/cacti-0.8.7g/rra/topsmedia1_cpu_user_706.rrd":cpu_user:AVERAGE \
DEF:c="/usr/local/cacti-0.8.7g/rra/topsmedia1_cpu_nice_704.rrd":cpu_nice:AVERAGE \
CDEF:cdefbc=TIME,1296755696,GT,a,a,UN,0,a,IF,IF,TIME,1296755696,GT,b,b,UN,0,b,IF,IF,TIME,1296755696,GT,c,c,UN,0,c,IF,IF,+,+ \
AREA:a#FF0000FF:"System"  \
GPRINT:a:LAST:"Current\:%8.2lf %s"  \
GPRINT:a:AVERAGE:"Average\:%8.2lf %s"  \
GPRINT:a:MAX:"Maximum\:%8.2lf %s\n"  \
AREA:b#0000FFFF:"User":STACK \
GPRINT:b:LAST:"  Current\:%8.2lf %s"  \
GPRINT:b:AVERAGE:"Average\:%8.2lf %s"  \
GPRINT:b:MAX:"Maximum\:%8.2lf %s\n"  \
AREA:c#00FF00FF:"Nice":STACK \
GPRINT:c:LAST:"  Current\:%8.2lf %s"  \
GPRINT:c:AVERAGE:"Average\:%8.2lf %s"  \
GPRINT:c:MAX:"Maximum\:%8.2lf %s\n"  \
LINE1:cdefbc#000000FF:"Total"  \
GPRINT:cdefbc:LAST:" Current\:%8.2lf %s"  \
GPRINT:cdefbc:AVERAGE:"Average\:%8.2lf %s"  \
GPRINT:cdefbc:MAX:"Maximum\:%8.2lf %s" 

RRDTool Says:

OK

Image
Attachments
empty-graph.png
empty-graph.png (21.23 KiB) Viewed 13358 times

User avatar
gandalf
Developer
Posts: 22375
Joined: Thu Dec 02, 2004 2:46 am
Location: Muenster, Germany
Contact:

Re: rrdtool update not running

#4 Post by gandalf » Thu Feb 03, 2011 1:43 pm

When migrating, did paths change?
R.

tdkevin
Posts: 20
Joined: Wed Aug 11, 2010 12:27 pm

Re: rrdtool update not running

#5 Post by tdkevin » Thu Feb 03, 2011 1:49 pm

not at all -- i booted the new server into finnix and rsync'd the entire server over -- everything else works except the graphs within cacti.

i just tried moving the rrd files and seeing if it creates new ones, but it does not. I verified all the paths are valid in the cacti settings.

User avatar
gandalf
Developer
Posts: 22375
Joined: Thu Dec 02, 2004 2:46 am
Location: Muenster, Germany
Contact:

Re: rrdtool update not running

#6 Post by gandalf » Thu Feb 03, 2011 3:25 pm

Nevertheless, please run a rebuild poller cache. Verify, taht "System Utilities -> View Poller cache" shows correct paths and commands
R.

tdkevin
Posts: 20
Joined: Wed Aug 11, 2010 12:27 pm

Re: rrdtool update not running

#7 Post by tdkevin » Thu Feb 03, 2011 3:31 pm

Yes -- Following the documentation I did do that. I rebuilt the poller cache twice. For one of the records in the poller cache you can see its trying to create the graphs with proper data :

Code: Select all

/usr/bin/rrdtool create \
/usr/local/cacti-0.8.7g/rra/topsweb2-1_cpu_378.rrd \
--step 300  \
DS:cpu:GAUGE:600:0:100 \
RRA:AVERAGE:0.5:1:600 \
RRA:AVERAGE:0.5:6:700 \
RRA:AVERAGE:0.5:24:775 \
RRA:AVERAGE:0.5:288:797 \
RRA:MAX:0.5:1:600 \
RRA:MAX:0.5:6:700 \
RRA:MAX:0.5:24:775 \
RRA:MAX:0.5:288:797 \
But that graph still shows "nan" ... :(

tdkevin
Posts: 20
Joined: Wed Aug 11, 2010 12:27 pm

Re: rrdtool update not running

#8 Post by tdkevin » Thu Feb 03, 2011 4:46 pm

if theres something else i can check or verify or run manually, i'm open to anything at this point. i've done the debugging howto and couldn't find a root cause to this problem.

User avatar
gandalf
Developer
Posts: 22375
Joined: Thu Dec 02, 2004 2:46 am
Location: Muenster, Germany
Contact:

Re: rrdtool update not running

#9 Post by gandalf » Fri Feb 04, 2011 10:15 am

Let me sum up what I did understand:

- the poller itself is running
- it creates entries in cacti.log
- data is polled from the targets and seen in that log file
- you did redirect the poller's output by changing the crontab to ... > /var/www/html/cacti/log/poller.log 2>&1
- log/poller.log does show entries? But no "rrdtool" entries???
- are rrd files updated or not?

What happens when you run "php -q poller.php --force" as cactiuser?
R.

tdkevin
Posts: 20
Joined: Wed Aug 11, 2010 12:27 pm

Re: rrdtool update not running

#10 Post by tdkevin » Fri Feb 04, 2011 10:27 am

gandalf wrote: - the poller itself is running
Yes.
gandalf wrote: - it creates entries in cacti.log
Yes
gandalf wrote: - data is polled from the targets and seen in that log file
Yes
gandalf wrote: - you did redirect the poller's output by changing the crontab to ... > /var/www/html/cacti/log/poller.log 2>&1
- log/poller.log does show entries? But no "rrdtool" entries???
I did that , there are entries such as the following in the log file :

Code: Select all

OK u:0.00 s:0.03 r:4.16
OK u:0.00 s:0.03 r:4.16
OK u:0.00 s:0.03 r:4.16
OK u:0.00 s:0.03 r:4.16
OK u:0.00 s:0.03 r:4.16
OK u:0.00 s:0.03 r:4.16
OK u:0.00 s:0.03 r:4.16
OK u:0.00 s:0.03 r:4.16
OK u:0.00 s:0.03 r:4.16
OK u:0.00 s:0.03 r:4.16
OK u:0.00 s:0.03 r:4.16
ERROR: Not enough arguments
OK u:0.00 s:0.03 r:4.16
OK u:0.00 s:0.03 r:4.16
OK u:0.00 s:0.03 r:4.16
OK u:0.00 s:0.03 r:4.16
OK u:0.00 s:0.03 r:4.16
OK u:0.00 s:0.03 r:4.16
OK u:0.00 s:0.03 r:4.16
OK u:0.00 s:0.03 r:4.16
OK u:0.00 s:0.03 r:4.16
OK u:0.00 s:0.03 r:4.16
ERROR: Not enough arguments
OK u:0.00 s:0.03 r:4.16
OK u:0.00 s:0.03 r:4.16
OK u:0.00 s:0.03 r:4.16
OK u:0.00 s:0.03 r:4.16
OK u:0.00 s:0.03 r:4.16
ERROR: Not enough arguments
You can view the entire poller.log here : http://pastebin.com/cFCAw3gJ Please let me know if you see anything weird happening.
gandalf wrote: - are rrd files updated or not?
No they are not being updated. I have chmodded them 777 to rule out permissions.
gandalf wrote: What happens when you run "php -q poller.php --force" as cactiuser?
See the pastebin of the poller.log output.


If you want I can post the cacti.log file for you to review, but its rather large and I'm not sure if its necessary.

User avatar
gandalf
Developer
Posts: 22375
Joined: Thu Dec 02, 2004 2:46 am
Location: Muenster, Germany
Contact:

Re: rrdtool update not running

#11 Post by gandalf » Fri Feb 04, 2011 10:43 am

Thanks. Those "ok" messages are output by rrdtool commands. So there are such commands. They will be printed to the log only when using higher debugging levels as given by the debugging guide. So I really require access to such a log (a single run would be ok).
R.

tdkevin
Posts: 20
Joined: Wed Aug 11, 2010 12:27 pm

Re: rrdtool update not running

#12 Post by tdkevin » Fri Feb 04, 2011 11:05 am

I had to cut the cacti.log in half because pastebin said it was too large


cacti.log : http://pastebin.com/gx60nD97

poller.log : http://pastebin.com/hz7F7edQ



Both are now with debug / developer logging enabled. Please let me know if you need anything else.


Thanks

tdkevin
Posts: 20
Joined: Wed Aug 11, 2010 12:27 pm

Re: rrdtool update not running

#13 Post by tdkevin » Fri Feb 04, 2011 1:26 pm

I was able to resolve this issue. The cron poller entry was pointing to /cacti/lib/poller.php , but i changed it to /cacti/poller.php

The two files are different and I'm not sure why it exists in two locations. I am seeing graphs populate again!

This whole process was somewhat confusing, but Gandalf I thank you for offering your help. Going through this process has enlightened me towards more in-depth troubleshooting for cacti in general.

Thanks again!

User avatar
gandalf
Developer
Posts: 22375
Joined: Thu Dec 02, 2004 2:46 am
Location: Muenster, Germany
Contact:

Re: rrdtool update not running

#14 Post by gandalf » Fri Feb 04, 2011 6:50 pm

The lib/poller.php is a library file that defines some functions used by other modules
R.

samirs2k
Posts: 1
Joined: Wed Sep 03, 2014 8:51 am

Re: rrdtool update not running

#15 Post by samirs2k » Wed Sep 03, 2014 8:58 am

Dear Gandalf,

Even I am facing somewhat similar problem. My 1) My Device was not showing up 2)RRD files were not getting created 3) No graphs were generated. After going through forums I tried triggering the Poller.php manually. I could see 1) Device showing up 2) RRD files created 3) Empty graph getting generated. I concluded the poller is not running as per the CRON schedule. I tried finding out a solution but all in vain. Can you please provide me a solution where poller does its job properly as defined in CRON (5Minutes).

If I keep on manually ( Through commandline) triggering poller.php the graph does generate. I dont understand why its not getting automated.

Post Reply