Cacti (home)ForumsDocumentation
Cacti: offical forums and support
It is currently Sat Feb 23, 2019 6:47 pm

All times are UTC - 5 hours




Post new topic Reply to topic  [ 6 posts ] 
Author Message
 Post subject: NaN graphs and missing poller items from cache
PostPosted: Mon Jan 07, 2019 5:50 am 
Offline
Cacti User
User avatar

Joined: Wed Apr 27, 2005 9:37 am
Posts: 55
I am having trouble finding the source of a problem with some graphs not updating and showing NaN. I have probably 40 devices in Cacti and the three devices having issues are different from the rest in that that they use a different data profile. I am on 1.1.38 and spine.

I've gone through all the troubleshooting steps in the docs for identifying NaN issues, to no avail. One step indicates you should check the poller cache to see if the devices are missing, which they are, but it offers no suggestions on what to do if that's the case.

I've tried rebuilding my poller cache, both via the web and the CLI and I still cannot get the devices to show up in the poller cache. I'm not sure what happened here, as these three devices had been working for several months, when suddenly at 2AM one day, two of the devices stopped updating. When I started looking at the problem and running through the troubleshooting steps outlined in the docs, suddenly the third device stopped updating, without me having changed anything besides rebuilding the poller cache. I believe the issue has to do with the different data profile, but I'm not sure what to do to fix it. The data profile has the exact same Data Source profiles as the default, but there is one additional that defines a 10-Minute Average:

Attachment:
File comment: Data profile
data_profile.PNG
data_profile.PNG [ 23.05 KiB | Viewed 110 times ]


Can someone please point me in the right direction on how to get these three devices to start polling again? Don't ask me where I got the numbers for the steps/rows in the 'Semi-weekly' data source profile. I copied them from another Cacti installation with the same profile, which has been working for several years.

EDIT: I found the following when using the Data Source Info selection in the web GUI:

Code:
Number of rows for Cacti RRA id '2' should be '700'
Number of rows for Cacti RRA id '3' should be '775'
Number of rows for Cacti RRA id '4' should be '797'
Number of rows for Cacti RRA id '7' should be '700'
Number of rows for Cacti RRA id '8' should be '775'
Number of rows for Cacti RRA id '9' should be '797'
Number of rows for Cacti RRA id '12' should be '700'
Number of rows for Cacti RRA id '13' should be '775'
Number of rows for Cacti RRA id '14' should be '797'
Number of rows for Cacti RRA id '17' should be '700'
Number of rows for Cacti RRA id '18' should be '775'
Number of rows for Cacti RRA id '19' should be '797'
/bin/rrdtool resize /var/www/html/cacti/rra/r1_traffic_in_339.rrd 2 SHRINK 2800
rename /var/www/html/cacti/rra/resize.rrd to /var/www/html/cacti/rra/r1_traffic_in_339.rrd


I am pretty sure this is why this is happening. I'm not versed at all in rrdtool - can someone give me an idea what I'll need to do to get this fixed?


Top
 Profile  
 
 Post subject: Re: NaN graphs and missing poller items from cache
PostPosted: Mon Jan 07, 2019 6:47 pm 
Offline
Cacti Pro User
User avatar

Joined: Mon Jan 05, 2015 10:10 am
Posts: 744
Resize leaves Nan's in the new rows, which makes the graphs break for a while. There have been those that have written utilities to insert good data, but rrdtool does not.

_________________
Before history, there was a paradise, now dust.


Top
 Profile  
 
 Post subject: Re: NaN graphs and missing poller items from cache
PostPosted: Tue Jan 08, 2019 4:59 am 
Offline
Cacti User
User avatar

Joined: Wed Apr 27, 2005 9:37 am
Posts: 55
I'm not concerned about the missing data at this point. I just want Cacti to actually start polling the devices, as I believe I fixed the issues with the rrd files. Rebuilding the poller cache doesn't seem to work. I am able to create new graphs on the same hosts and they get added into the cache, but for some reason, these just will not. How can I get Cacti to start polling? Debug shows nothing because the devices are not being polled. If I try rebuilding the cache for just the singular devices via the CLI, I get no errors.


Top
 Profile  
 
 Post subject: Re: NaN graphs and missing poller items from cache
PostPosted: Tue Jan 08, 2019 7:06 pm 
Offline
Cacti Pro User
User avatar

Joined: Mon Jan 05, 2015 10:10 am
Posts: 744
You might want to 'resave' them as it will populate the poller cache if it's not already. This is kind of hard to solve over the forums, but basically there is a column in the poller_item tabled called rrd_next_step and when it goes negative, the data source will poll again. It decrements by the poller interval. So, if you are polling every 60 seconds, you will see the value slowly creep towards zero. Sometimes people set the heartbeat too low. Say for example, you poller interval is 300 seconds, but your heartbeat is set to 60 seconds. Those graphs will never show data.

Under the Data Sources page, there is an Info debug level that will show you what the RRDfiles look like, but some of this is hand viewing the poller cache and in rare cases (like a bug in Cacti) reviewing the rrd_next_step value for accuracy.

I hope this helps. There is a lot to review. One more thing, in Cacti 1.2, there is a troubleshooting pick and under that Data Debug that you can enable for the Data Sources in question.

_________________
Before history, there was a paradise, now dust.


Top
 Profile  
 
 Post subject: Re: NaN graphs and missing poller items from cache
PostPosted: Thu Jan 10, 2019 12:54 pm 
Offline
Cacti User
User avatar

Joined: Wed Apr 27, 2005 9:37 am
Posts: 55
I've tried everything I can think of to "re-save" them. I've disabled and re-enabled the hosts, changed settings in the individual graph pages and changed them back again. I've been trying to find how the poller cache is populated, but I'm not a coder and can't really figure it out from the PHP files. I am about to just delete the graphs, re-add them and use the old (fixed) RRD files over top of the new ones it creates.

Thanks for the info about the poller_item column. Unfortunately, these items are not even in the database at all, so there is no field to attempt to decrement. I can't create new graphs using the same template without deleting the existing ones first, so it does know that graphs were created. It just is not putting them into the poller_items table when rebuilding.

I did try to update to 1.2 the other day, but all my graphs stopped updating after doing so and nothing I was doing was working to get them back online. I got frustrated and just restored to 1.1.38.


Top
 Profile  
 
 Post subject: Re: NaN graphs and missing poller items from cache
PostPosted: Fri Jan 11, 2019 4:47 pm 
Offline
Cacti Pro User
User avatar

Joined: Mon Jan 05, 2015 10:10 am
Posts: 744
Okay, there was an issue found in 1.2.0. If you are running 1.2.0, do the following:

Quote:
UPDATE host SET deleted="" WHERE deleted=NULL;
ALTER TABLE host MODIFY COLUMN deleted CHAR(2) NOT NULL default "";

_________________
Before history, there was a paradise, now dust.


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 6 posts ] 

All times are UTC - 5 hours


Who is online

Users browsing this forum: No registered users and 5 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  

Protected by Anti-Spam ACP Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group