Sudden NAN graph issues under ubuntu 16.04 cacti 1.2.2

Post support questions that directly relate to Linux/Unix operating systems.

Moderators: Moderators, Developers

Post Reply
Author
Message
seanmancini
Posts: 47
Joined: Wed Mar 13, 2019 3:37 pm
Location: toronto
Contact:

Sudden NAN graph issues under ubuntu 16.04 cacti 1.2.2

#1 Post by seanmancini » Fri Apr 05, 2019 9:26 pm

Hello Everyone

I have a weird issue going on with graphs suddenly all of my graphs went into NAN
Realtime graphs work which is strange

RRD debug mode shows ok



RRDtool Command:

/usr/bin/rrdtool graph - \
--imgformat=PNG \
--start='-86400' \
--end='-60' \
--pango-markup \
--title=****- Traffic - *** xxx ***' \
--vertical-label='bits per second' \
--slope-mode \
--base=1000 \
--height=200 \
--width=700 \
--rigid \
--alt-autoscale-max \
--lower-limit='0' \
COMMENT:"From 2019/04/04 22\:15\:03 To 2019/04/05 22\:14\:03\c" \
COMMENT:" \n" \
--color BACK#F3F3F3 \
--color CANVAS#FDFDFD \
--color SHADEA#CBCBCB \
--color SHADEB#999999 \
--color FONT#000000 \
--color AXIS#2C4D43 \
--color ARROW#2C4D43 \
--color FRAME#2C4D43 \
--border 1 --font TITLE:11:'Arial' \
--font AXIS:8:'Arial' \
--font LEGEND:8:'Courier' \
--font UNIT:8:'Arial' \
--font WATERMARK:6:'Arial' \
--slope-mode \
--watermark 'Generated by Cacti®' \
DEF:a='/var/www/html/cacti/rra/xxx_in_302.rrd':'traffic_in':MAX \
DEF:b='/var/www/html/cacti/rra/xxx_***_com_traffic_in_302.rrd':'traffic_in':AVERAGE \
DEF:c='/var/www/html/cacti/rra/xxx_***_com_traffic_in_302.rrd':'traffic_out':MAX \
DEF:d='/var/www/html/cacti/rra/xxx_***_com_traffic_in_302.rrd':'traffic_out':AVERAGE \
CDEF:cdefa='a,8,*' \
CDEF:cdefb='b,8,*' \
CDEF:cdeff='c,8,*' \
CDEF:cdefg='d,8,*' \
LINE1:cdefa#00CF00FF: \
AREA:cdefb#00CF007F:'Inbound ' \
GPRINT:cdefb:LAST:'Current\:%8.2lf %s' \
GPRINT:cdefb:AVERAGE:'Average\:%8.2lf %s' \
GPRINT:cdefb:MAX:'Maximum\:%8.2lf %s\n' \
LINE1:cdeff#002A97FF: \
AREA:cdefg#002A977F:'Outbound' \
GPRINT:cdefg:LAST:'Current\:%8.2lf %s' \
GPRINT:cdefg:AVERAGE:'Average\:%8.2lf %s' \
GPRINT:cdefg:MAX:'Maximum\:%8.2lf %s\n'

RRDtool Says:

OK

Output from poller

OK u:0.02 s:0.02 r:1.12
OK u:0.02 s:0.02 r:1.13
OK u:0.02 s:0.02 r:1.13
OK u:0.02 s:0.02 r:1.15
OK u:0.02 s:0.02 r:1.15
OK u:0.02 s:0.02 r:1.17
OK u:0.02 s:0.02 r:1.17
OK u:0.02 s:0.02 r:1.18
OK u:0.02 s:0.02 r:1.18
2019/04/05 22:07:02 - SYSTEM STATS: Time:1.5721 Method:cmd.php Processes:1 Threads:N/A Hosts:9 HostsPerProcess:9 DataSources:16 RRDsProcessed:10

Here are my directory permissions

-rw-r--r-- 1 www-data www-data 4.3K Feb 24 07:37 about.php
-rw-r--r-- 1 www-data www-data 56K Feb 24 07:37 aggregate_graphs.php
-rw-r--r-- 1 www-data www-data 18K Feb 24 07:37 aggregate_items.php
-rw-r--r-- 1 www-data www-data 24K Feb 24 07:37 aggregate_templates.php
-rw-r--r-- 1 www-data www-data 14K Feb 24 07:37 auth_changepassword.php
-rw-r--r-- 1 www-data www-data 28K Feb 24 07:37 auth_login.php
-rw-r--r-- 1 www-data www-data 18K Feb 24 07:37 auth_profile.php
-rw-r--r-- 1 www-data www-data 23K Feb 24 07:37 automation_devices.php
-rw-r--r-- 1 www-data www-data 36K Feb 24 07:37 automation_graph_rules.php
-rw-r--r-- 1 www-data www-data 42K Feb 24 07:37 automation_networks.php
-rw-r--r-- 1 www-data www-data 30K Feb 24 07:37 automation_snmp.php
-rw-r--r-- 1 www-data www-data 19K Feb 24 07:37 automation_templates.php
-rw-r--r-- 1 www-data www-data 38K Feb 24 07:37 automation_tree_rules.php
-rw-r--r-- 1 www-data www-data 2.9K Feb 24 07:37 boost_rrdupdate.php
drwxr-xr-x 6 www-data www-data 4.0K Feb 24 07:37 cache
-rw-r--r-- 1 www-data www-data 119K Feb 24 07:37 cacti.sql
-rw-r--r-- 1 www-data www-data 29K Feb 24 07:37 cdef.php
-rw-r--r-- 1 www-data www-data 167K Feb 24 07:37 CHANGELOG
drwxr-xr-x 2 www-data www-data 4.0K Feb 24 07:37 cli
-rw-r--r-- 1 www-data www-data 1.9K Feb 24 07:37 clog.php
-rw-r--r-- 1 www-data www-data 1.9K Feb 24 07:37 clog_user.php
-rw-r--r-- 1 www-data www-data 30K Feb 24 07:37 cmd.php
-rw-r--r-- 1 www-data www-data 8.7K Feb 24 07:37 cmd_realtime.php
-rw-r--r-- 1 www-data www-data 24K Feb 24 07:37 color.php
-rw-r--r-- 1 www-data www-data 13K Feb 24 07:37 color_templates_items.php
-rw-r--r-- 1 www-data www-data 22K Feb 24 07:37 color_templates.php
-rw-r--r-- 1 www-data www-data 27K Feb 24 07:37 data_debug.php
-rw-r--r-- 1 www-data www-data 34K Feb 24 07:37 data_input.php
-rw-r--r-- 1 www-data www-data 49K Feb 24 07:37 data_queries.php
-rw-r--r-- 1 www-data www-data 34K Feb 24 07:37 data_source_profiles.php
-rw-r--r-- 1 www-data www-data 61K Feb 24 07:37 data_sources.php
-rw-r--r-- 1 www-data www-data 41K Feb 24 07:37 data_templates.php
drwxr-xr-x 2 www-data www-data 4.0K Feb 24 07:37 docs
drwxr-xr-x 2 www-data www-data 4.0K Feb 24 07:37 formats
drwxr-xr-x 3 www-data www-data 4.0K Feb 24 07:37 .github
-rw-r--r-- 1 www-data www-data 588 Feb 24 07:37 .gitignore
-rw-r--r-- 1 www-data www-data 14K Feb 24 07:37 gprint_presets.php
-rw-r--r-- 1 www-data www-data 5.4K Feb 24 07:37 graph_image.php
-rw-r--r-- 1 www-data www-data 8.7K Feb 24 07:37 graph_json.php
-rw-r--r-- 1 www-data www-data 21K Feb 24 07:37 graph.php
-rw-r--r-- 1 www-data www-data 14K Feb 24 07:37 graph_realtime.php
-rw-r--r-- 1 www-data www-data 25K Feb 24 07:37 graphs_items.php
-rw-r--r-- 1 www-data www-data 31K Feb 24 07:37 graphs_new.php
-rw-r--r-- 1 www-data www-data 80K Feb 24 07:37 graphs.php
-rw-r--r-- 1 www-data www-data 9.3K Feb 24 07:37 graph_templates_inputs.php
-rw-r--r-- 1 www-data www-data 26K Feb 24 07:37 graph_templates_items.php
-rw-r--r-- 1 www-data www-data 38K Feb 24 07:37 graph_templates.php
-rw-r--r-- 1 www-data www-data 29K Feb 24 07:37 graph_view.php
-rw-r--r-- 1 www-data www-data 12K Feb 24 07:37 graph_xport.php
-rw-r--r-- 1 www-data www-data 59K Feb 24 07:37 host.php
-rw-r--r-- 1 www-data www-data 30K Feb 24 07:37 host_templates.php
drwxr-xr-x 2 www-data www-data 4.0K Feb 24 07:37 images
drwxr-xr-x 8 www-data www-data 4.0K Apr 5 16:10 include
-rw-r--r-- 1 www-data www-data 5.6K Feb 24 07:37 index.php
drwxr-xr-x 4 www-data www-data 4.0K Feb 24 07:37 install
drwxr-xr-x 2 www-data www-data 4.0K Feb 24 07:37 lib
-rw-r--r-- 1 www-data www-data 15K Feb 24 07:37 LICENSE
-rw-r--r-- 1 www-data www-data 2.9K Feb 24 07:37 link.php
-rw-r--r-- 1 www-data www-data 22K Feb 24 07:37 links.php
drwxr-xr-x 4 www-data www-data 4.0K Feb 24 07:37 locales
drwxr-xr-x 2 www-data www-data 4.0K Apr 5 16:20 log
-rw-r--r-- 1 www-data www-data 4.4K Feb 24 07:37 logout.php
-rw-r--r-- 1 www-data www-data 38K Feb 24 07:37 managers.php
drwxr-xr-x 2 www-data www-data 4.0K Feb 24 07:37 mibs
-rw-r--r-- 1 www-data www-data 3.4K Feb 24 07:37 permission_denied.php
drwxr-xr-x 4 www-data www-data 4.0K Apr 5 18:24 plugins
-rw-r--r-- 1 www-data www-data 25K Feb 24 07:37 plugins.php
-rw-r--r-- 1 www-data www-data 36K Feb 24 07:37 poller_automation.php
-rw-r--r-- 1 www-data www-data 15K Feb 24 07:37 poller_boost.php
-rw-r--r-- 1 www-data www-data 6.7K Feb 24 07:37 poller_commands.php
-rw-r--r-- 1 www-data www-data 7.7K Feb 24 07:37 poller_dsstats.php
-rw-r--r-- 1 www-data www-data 19K Feb 24 07:37 poller_maintenance.php
-rw-r--r-- 1 www-data www-data 31K Feb 24 07:37 poller.php
-rw-r--r-- 1 www-data www-data 9.6K Feb 24 07:37 poller_realtime.php
-rw-r--r-- 1 www-data www-data 8.6K Feb 24 07:37 poller_recovery.php
-rw-r--r-- 1 www-data www-data 5.2K Feb 24 07:37 poller_reports.php
-rw-r--r-- 1 www-data www-data 34K Feb 24 07:37 pollers.php
-rw-r--r-- 1 www-data www-data 6.8K Feb 24 07:37 poller_spikekill.php
-rw-r--r-- 1 www-data www-data 8.0K Feb 24 07:37 README.md
-rw-r--r-- 1 www-data www-data 14K Feb 24 07:37 remote_agent.php
-rw-r--r-- 1 www-data www-data 3.3K Feb 24 07:37 reports_admin.php
-rw-r--r-- 1 www-data www-data 3.2K Feb 24 07:37 reports_user.php
drwxr-xr-x 5 www-data www-data 4.0K Feb 24 07:37 resource
drwxr-xr-x 2 www-data www-data 4.0K Apr 5 21:14 rra
-rw-r--r-- 1 www-data www-data 19K Feb 24 07:37 rrdcleaner.php
drwxr-xr-x 2 www-data www-data 4.0K Apr 5 19:20 scripts
-rw-r--r-- 1 www-data www-data 11K Feb 24 07:37 script_server.php
-rw-r--r-- 1 www-data www-data 37K Feb 24 07:37 settings.php
-rw-r--r-- 1 www-data www-data 19K Feb 24 07:37 sites.php
-rw-r--r-- 1 www-data www-data 3.7K Feb 24 07:37 snmpagent_mibcachechild.php
-rw-r--r-- 1 www-data www-data 2.4K Feb 24 07:37 snmpagent_mibcache.php
-rw-r--r-- 1 www-data www-data 5.4K Feb 24 07:37 snmpagent_persist.php
-rw-r--r-- 1 www-data www-data 3.9K Feb 24 07:37 spikekill.php
-rw-r--r-- 1 www-data www-data 6.4K Feb 24 07:37 templates_export.php
-rw-r--r-- 1 www-data www-data 5.7K Feb 24 07:37 templates_import.php
drwxr-xr-x 3 www-data www-data 4.0K Feb 24 07:37 tests
-rw-r--r-- 1 www-data www-data 1.5K Feb 24 07:37 .travis.yml
-rw-r--r-- 1 www-data www-data 62K Feb 24 07:37 tree.php
-rw-r--r-- 1 www-data www-data 100K Feb 24 07:37 user_admin.php
-rw-r--r-- 1 www-data www-data 30K Feb 24 07:37 user_domains.php
-rw-r--r-- 1 www-data www-data 86K Feb 24 07:37 user_group_admin.php
-rw-r--r-- 1 www-data www-data 91K Feb 24 07:37 utilities.php
-rw-r--r-- 1 www-data www-data 29K Feb 24 07:37 vdef.php

Here is my Crontab entry

*/1 * * * * www-data php /var/www/html/cacti/poller.php >> /var/log/cacti-cron-log.txt

Enviroment info


PHP 7.0.33-0ubuntu0.16.04.3 (cli) ( NTS )
Copyright (c) 1997-2017 The PHP Group
Zend Engine v3.0.0, Copyright (c) 1998-2017 Zend Technologies
with Zend OPcache v7.0.33-0ubuntu0.16.04.3, Copyright (c) 1999-2017, by Zend Technologies


No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 16.04.6 LTS
Release: 16.04
Codename: xenial


Any Ideas ?

netniV
Cacti Guru User
Posts: 2881
Joined: Sun Aug 27, 2017 12:05 am

Re: Sudden NAN graph issues under ubuntu 16.04 cacti 1.2.2

#2 Post by netniV » Sat Apr 06, 2019 2:41 am

Check the subfolders of the RRA and make sure that they are also of the right permissions.

seanmancini
Posts: 47
Joined: Wed Mar 13, 2019 3:37 pm
Location: toronto
Contact:

Re: Sudden NAN graph issues under ubuntu 16.04 cacti 1.2.2

#3 Post by seanmancini » Sat Apr 06, 2019 8:28 am

Thanks for the reply!

So the RRA folder permissions seem to be fine
The cron is being run under the www-data user
-rw-r--r-- 1 www-data www-data 185K Apr 6 01:34 881-r1_xxx-lab_local_traffic_in_104.rrd
-rw-r--r-- 1 www-data www-data 185K Apr 6 01:35 881-r1_xxx-lab_local_traffic_in_107.rrd
-rw-r--r-- 1 www-data www-data 185K Apr 6 01:35 881-r1_xxx-lab_local_traffic_in_110.rrd
-rw-r--r-- 1 www-data www-data 185K Apr 6 01:36 881-r1_xxx-lab_local_traffic_in_113.rrd


the strange thing is this morning graphs are working I added another device in and It seems to be taking a very long time for graphs to populate
I still get NAN for the new device even after several minutes

when I click real-time graphs I see results instantly I do see the file created in RRA as well so doesn't appear to be a permissions issue

The poller time is good
2019/04/06 09:19:03 - SYSTEM STATS: Time:1.3643 Method:cmd.php Processes:1 Threads:N/A Hosts:7 HostsPerProcess:7 DataSources:12 RRDsProcessed:7

I am at a loss

Post Reply