Cacti not creating data sources or rrd files when it should?

Post general support questions here that do not specifically fall into the Linux or Windows categories.

Moderators: Moderators, Developers

Author
Message
Azendale
Posts: 13
Joined: Sat Feb 21, 2015 6:15 pm

Cacti not creating data sources or rrd files when it should?

#1 Post by Azendale » Thu Jul 12, 2018 2:55 pm

I have some indexed SNMP queries. (I've attached the XML definitions.) I have these associated with a host templete. When I add a device/host with that template, and then go to the "Create graphs for this host" link, I can see in the tables that the indexed queries are running and are successful in returning data. Good.

But when I try to create graphs, the graphs are always replaced by an image that says no .rrd file was found. Checking in the filesystem of the server, there is no .rrd file, so it was never created.

I'm really not sure how to debug this. I tried turning up the debug to devel level, and then adding graphs to the host. I then looked at the logs and found and entry that was something like this:
(I'm paraphrasing because the page refreshed before I could copy it)
CACTI2RRD: (some RRDtool creation command) /some/path/to/new.rrd DS[1234] (Data source not found)

When I click on the link to go to the datasource it says it can't find, the Cacti WebUI says it doesn't exist.

So it seems that Cacti is not creating data sources both in the MySQL database and as .rrd files on the file system -- but only for some things. (Other graphs on different data queries are working just fine and the datasources and .rrd files create just fine.)

Why is this happening? What information do I need to collect to help debug this?
Attachments
huawei_cmts_cms.xml
Indexed query XML DOCSIS Modem stats
(2.5 KiB) Downloaded 62 times
huawei_cmts_downstream_utilization.xml
Indexed query XML DOCSIS DS utilization
(736 Bytes) Downloaded 58 times
huawei_cmts_upstream_utilization.xml
Indexed query XML DOCSIS US utilization
(732 Bytes) Downloaded 58 times

netniV
Cacti Guru User
Posts: 2666
Joined: Sun Aug 27, 2017 12:05 am

Re: Cacti not creating data sources or rrd files when it sho

#2 Post by netniV » Thu Jul 12, 2018 5:42 pm

Check that your folder has the correct permissions to allow both the poller user and website user to create / maintain the rrd files.

Azendale
Posts: 13
Joined: Sat Feb 21, 2015 6:15 pm

Re: Cacti not creating data sources or rrd files when it sho

#3 Post by Azendale » Thu Jul 12, 2018 7:27 pm

I don't think that is the problem, since other graphs and data sources can be created. But here's what I get trying to look at that:

Code: Select all

spruce:/usr/share/cacti/site# ls -lah |grep rra
lrwxrwxrwx  1 root root   18 Apr 26 04:25 rra -> /var/lib/cacti/rra
spruce:/usr/share/cacti/site# ls -lah /var/lib/cacti/ | grep rra
drwxr-xr-x  2 www-data www-data 692K Jul 12 10:50 rra
spruce:/usr/share/cacti/site# cat /etc/cron
cron.d/       cron.daily/   cron.hourly/  cron.monthly/ crontab       cron.weekly/  
spruce:/usr/share/cacti/site# cat /etc/cron.d
cron.d/     cron.daily/ 
spruce:/usr/share/cacti/site# cat /etc/cron.d/
cacti          certbot        php            php5           php5.dpkg-old  .placeholder   
spruce:/usr/share/cacti/site# cat /etc/cron.d/cacti 
MAILTO=root
*/5 * * * * www-data php /usr/share/cacti/site/poller.php 2>&1 >/dev/null | if [ -f /usr/bin/ts ] ; then ts ; else tee ; fi >> /var/log/cacti/poller-error.log
This is on a Debian server where the server normally would run PHP as www-data, at least to my understanding.

Also, I'm attaching some exports of what I have, in case there is something wrong with them (which I highly suspect; I just don't know what in particular is wrong..)

Well, at least the host template export. I can't seem to attach the graph templates as they are being rejected by the forum as "The upload was rejected because the uploaded file was identified as a possible attack vector."
Attachments
cacti_host_template_huawei_cmts.xml
Host template
(85.51 KiB) Downloaded 69 times

netniV
Cacti Guru User
Posts: 2666
Joined: Sun Aug 27, 2017 12:05 am

Re: Cacti not creating data sources or rrd files when it sho

#4 Post by netniV » Fri Jul 13, 2018 1:16 am

Rename them to .txt and it should be OK, are there any errors in the logs when you attempt to create from these sources?

Azendale
Posts: 13
Joined: Sat Feb 21, 2015 6:15 pm

Re: Cacti not creating data sources or rrd files when it sho

#5 Post by Azendale » Sun Jul 15, 2018 7:55 pm

I just tried renaming them, and the same "The upload was rejected because the uploaded file was identified as a possible attack vector." happened.

I just tried re-adding the graphs after deleting the old ones that didn't work and were showing the .rrd error. Nothing showed up in the logs related to that device or graphn with it set to "Low - statistics and errors".

Azendale
Posts: 13
Joined: Sat Feb 21, 2015 6:15 pm

Re: Cacti not creating data sources or rrd files when it sho

#6 Post by Azendale » Wed Jul 18, 2018 9:30 pm

Still having trouble with this, though maybe I found something.

But first, I'm working around the forum upload troubles. See http://users.eoni.com/~azendale/cacti/ for a list of files I was able to export on one of the problematic graphs.

Ok, back to what I noticed:

Here's the datasource for a graph on the device that works, with an indexed query:

Image

Here's one that doesn't:
Image

They are both indexed SNMP queries on the same device. Why does one have index not filled out and then won't let the other two fields be set? Am I on to something here?

(Since I've started this thread, I have tried removing the query and putting it back with a new data source and graph just to get around the issues, and today I noticed what I'm pointing out here after doing all that and not getting anywhere.)

This is a picture of what the problematic query shows on the add graphs page:

Image

This makes me think the query is fine, but maybe there is some problem with the query XML itself?

Working query:
http://users.eoni.com/~azendale/cacti/r ... annels.xml

Non working query:
http://users.eoni.com/~azendale/cacti/r ... ts_cms.xml

What am I missing between these two?

User avatar
Pucho
Cacti User
Posts: 180
Joined: Wed Jul 20, 2016 8:00 pm

Re: Cacti not creating data sources or rrd files when it sho

#7 Post by Pucho » Wed Jul 18, 2018 9:48 pm

How does your data source template look like? The one that's not working.
Cacti - 1.1.37
Poller Type - Spine
Weathermap 0.98a
Server Info - Linux 3.10.0 - Centos 7
Web Server - Apache/2.4.6 PHP 5.4.16
MySQL - 5.5 ;RRDTool - 1.4.8 ;SNMP - 5.7.2
Religion - Anti forum pets

Code: Select all

SYSTEM STATS: Time:17.7383 Method:spine Processes:1 Threads:10 Hosts:279 HostsPerProcess:279 DataSources:7309 RRDsProcessed:3718

Code: Select all

WEATHERMAP STATS: Weathermap 0.98 run complete - Thu, 16 Nov 17 08:42:56 +1300: 39 maps were run in 23.22 seconds with 0 warnings.

netniV
Cacti Guru User
Posts: 2666
Joined: Sun Aug 27, 2017 12:05 am

Re: Cacti not creating data sources or rrd files when it sho

#8 Post by netniV » Thu Jul 19, 2018 1:31 am

Your problem is quite simple. You've a difference in the index in the oid_index key so it never finds any indexes beneath it. Remove the 1979719681 or maybe add it depend on which one is working and which isn't.

Azendale
Posts: 13
Joined: Sat Feb 21, 2015 6:15 pm

Re: Cacti not creating data sources or rrd files when it sho

#9 Post by Azendale » Thu Jul 19, 2018 1:34 pm

Pucho wrote:How does your data source template look like? The one that's not working.
Here you go:
Image

Azendale
Posts: 13
Joined: Sat Feb 21, 2015 6:15 pm

Re: Cacti not creating data sources or rrd files when it sho

#10 Post by Azendale » Thu Jul 19, 2018 2:35 pm

netniV wrote:Your problem is quite simple. You've a difference in the index in the oid_index key so it never finds any indexes beneath it. Remove the 1979719681 or maybe add it depend on which one is working and which isn't.
Now I'm questioning what I know about indexed queries. I thought you wanted to find something you could SNMPwalk to get a list of "rows" in a table. Then you put that in the <oid_index> directive, and how to parse the index in the <out_index_parse> directive. It would then parse the 1979719681 that should be added on to get row one, and other suffixes that should be added to get the other rows. Or at least that's how I thought it worked. So I think the XML WITH the 1979719681 is incorrect. But it is the one that is working!

Here's what happens when I do SNMPwalks with the OIDs from the two XML files:

File that doesn't work, OID from <oid_index>:

Code: Select all

spruce:~# snmpwalk  -Of -c 0myHECK2 -v2c 10.5.6.2 .1.3.6.1.2.1.10.127.1.3.3.1.2
.iso.org.dod.internet.mgmt.mib-2.transmission.docsIfMib.docsIfMibObjects.docsIfCmtsObjects.docsIfCmtsCmStatusTable.docsIfCmtsCmStatusEntry.docsIfCmtsCmStatusMacAddress.1 = STRING: 3c:90:66:8f:a0:3a
.iso.org.dod.internet.mgmt.mib-2.transmission.docsIfMib.docsIfMibObjects.docsIfCmtsObjects.docsIfCmtsCmStatusTable.docsIfCmtsCmStatusEntry.docsIfCmtsCmStatusMacAddress.2 = STRING: 3c:90:66:8f:9f:38
.iso.org.dod.internet.mgmt.mib-2.transmission.docsIfMib.docsIfMibObjects.docsIfCmtsObjects.docsIfCmtsCmStatusTable.docsIfCmtsCmStatusEntry.docsIfCmtsCmStatusMacAddress.3 = STRING: 3c:90:66:8f:9f:44
.iso.org.dod.internet.mgmt.mib-2.transmission.docsIfMib.docsIfMibObjects.docsIfCmtsObjects.docsIfCmtsCmStatusTable.docsIfCmtsCmStatusEntry.docsIfCmtsCmStatusMacAddress.4 = STRING: 3c:90:66:8f:a0:ac
.iso.org.dod.internet.mgmt.mib-2.transmission.docsIfMib.docsIfMibObjects.docsIfCmtsObjects.docsIfCmtsCmStatusTable.docsIfCmtsCmStatusEntry.docsIfCmtsCmStatusMacAddress.5 = STRING: 3c:90:66:8f:9e:96
.iso.org.dod.internet.mgmt.mib-2.transmission.docsIfMib.docsIfMibObjects.docsIfCmtsObjects.docsIfCmtsCmStatusTable.docsIfCmtsCmStatusEntry.docsIfCmtsCmStatusMacAddress.6 = STRING: 3c:90:66:8f:9f:8c
.iso.org.dod.internet.mgmt.mib-2.transmission.docsIfMib.docsIfMibObjects.docsIfCmtsObjects.docsIfCmtsCmStatusTable.docsIfCmtsCmStatusEntry.docsIfCmtsCmStatusMacAddress.7 = STRING: 3c:90:66:8f:9e:a8
.iso.org.dod.internet.mgmt.mib-2.transmission.docsIfMib.docsIfMibObjects.docsIfCmtsObjects.docsIfCmtsCmStatusTable.docsIfCmtsCmStatusEntry.docsIfCmtsCmStatusMacAddress.8 = STRING: 3c:90:66:8f:9f:86
.iso.org.dod.internet.mgmt.mib-2.transmission.docsIfMib.docsIfMibObjects.docsIfCmtsObjects.docsIfCmtsCmStatusTable.docsIfCmtsCmStatusEntry.docsIfCmtsCmStatusMacAddress.9 = STRING: 3c:90:66:8f:a0:64
.iso.org.dod.internet.mgmt.mib-2.transmission.docsIfMib.docsIfMibObjects.docsIfCmtsObjects.docsIfCmtsCmStatusTable.docsIfCmtsCmStatusEntry.docsIfCmtsCmStatusMacAddress.10 = STRING: 3c:90:66:8f:9e:d8
.iso.org.dod.internet.mgmt.mib-2.transmission.docsIfMib.docsIfMibObjects.docsIfCmtsObjects.docsIfCmtsCmStatusTable.docsIfCmtsCmStatusEntry.docsIfCmtsCmStatusMacAddress.11 = STRING: 0:7:11:11:34:d2
File that did work, OID from <oid_index>:

Code: Select all

spruce:~# snmpwalk  -Of -c 0myHECK2 -v2c 10.5.6.2 .1.3.6.1.2.1.10.127.1.1.2.1.1.1979719681
.iso.org.dod.internet.mgmt.mib-2.transmission.docsIfMib.docsIfMibObjects.docsIfBaseObjects.docsIfUpstreamChannelTable.docsIfUpstreamChannelEntry.docsIfUpChannelId.1979719681 = INTEGER: 1
Hm. Only one row. Lets take off the suffix and try again:

Code: Select all

spruce:~# snmpwalk  -Of -c 0myHECK2 -v2c 10.5.6.2 .1.3.6.1.2.1.10.127.1.1.2.1.1
.iso.org.dod.internet.mgmt.mib-2.transmission.docsIfMib.docsIfMibObjects.docsIfBaseObjects.docsIfUpstreamChannelTable.docsIfUpstreamChannelEntry.docsIfUpChannelId.1979719681 = INTEGER: 1
.iso.org.dod.internet.mgmt.mib-2.transmission.docsIfMib.docsIfMibObjects.docsIfBaseObjects.docsIfUpstreamChannelTable.docsIfUpstreamChannelEntry.docsIfUpChannelId.1979719682 = INTEGER: 2
.iso.org.dod.internet.mgmt.mib-2.transmission.docsIfMib.docsIfMibObjects.docsIfBaseObjects.docsIfUpstreamChannelTable.docsIfUpstreamChannelEntry.docsIfUpChannelId.1979719683 = INTEGER: 3
.iso.org.dod.internet.mgmt.mib-2.transmission.docsIfMib.docsIfMibObjects.docsIfBaseObjects.docsIfUpstreamChannelTable.docsIfUpstreamChannelEntry.docsIfUpChannelId.1979719684 = INTEGER: 4
.iso.org.dod.internet.mgmt.mib-2.transmission.docsIfMib.docsIfMibObjects.docsIfBaseObjects.docsIfUpstreamChannelTable.docsIfUpstreamChannelEntry.docsIfUpChannelId.1979719685 = INTEGER: 5
.iso.org.dod.internet.mgmt.mib-2.transmission.docsIfMib.docsIfMibObjects.docsIfBaseObjects.docsIfUpstreamChannelTable.docsIfUpstreamChannelEntry.docsIfUpChannelId.1979719686 = INTEGER: 6
.iso.org.dod.internet.mgmt.mib-2.transmission.docsIfMib.docsIfMibObjects.docsIfBaseObjects.docsIfUpstreamChannelTable.docsIfUpstreamChannelEntry.docsIfUpChannelId.1979719687 = INTEGER: 7
.iso.org.dod.internet.mgmt.mib-2.transmission.docsIfMib.docsIfMibObjects.docsIfBaseObjects.docsIfUpstreamChannelTable.docsIfUpstreamChannelEntry.docsIfUpChannelId.1979719688 = INTEGER: 8
Ok, now that makes sense for what I would expect in the table.

Puzzled, I went to the device page and refreshed the working data query. It now only shows one row. "Fixed" the XML to use 10.5.6.2 .1.3.6.1.2.1.10.127.1.1.2.1.1 now I have the 8 rows I expect when I run the verbose query on the device page. So... is the index a side issue (probably introduced by me trying to find problems, but still working because I didn't re-create my graphs or the SNMP uptime didn't cause a re-index)?

User avatar
Pucho
Cacti User
Posts: 180
Joined: Wed Jul 20, 2016 8:00 pm

Re: Cacti not creating data sources or rrd files when it sho

#11 Post by Pucho » Thu Jul 19, 2018 3:44 pm

These two data source templates are using "Upgrade profile 1"?

If you can create other graphs without issues I wouldn't think it's a permission issue. I might be wrong but..

Can't spot anything weird though.
Cacti - 1.1.37
Poller Type - Spine
Weathermap 0.98a
Server Info - Linux 3.10.0 - Centos 7
Web Server - Apache/2.4.6 PHP 5.4.16
MySQL - 5.5 ;RRDTool - 1.4.8 ;SNMP - 5.7.2
Religion - Anti forum pets

Code: Select all

SYSTEM STATS: Time:17.7383 Method:spine Processes:1 Threads:10 Hosts:279 HostsPerProcess:279 DataSources:7309 RRDsProcessed:3718

Code: Select all

WEATHERMAP STATS: Weathermap 0.98 run complete - Thu, 16 Nov 17 08:42:56 +1300: 39 maps were run in 23.22 seconds with 0 warnings.

netniV
Cacti Guru User
Posts: 2666
Joined: Sun Aug 27, 2017 12:05 am

Re: Cacti not creating data sources or rrd files when it sho

#12 Post by netniV » Fri Jul 20, 2018 3:09 am

If you change the XML you should probably be running a poller output clear / reindex to ensure that everything is in sync.

Azendale
Posts: 13
Joined: Sat Feb 21, 2015 6:15 pm

Re: Cacti not creating data sources or rrd files when it sho

#13 Post by Azendale » Fri Jul 20, 2018 1:09 pm

Pucho wrote:These two data source templates are using "Upgrade profile 1"?

If you can create other graphs without issues I wouldn't think it's a permission issue. I might be wrong but..

Can't spot anything weird though.
Turns out I have two "Upgrade profile 1" profiles. Renamed them to Profile 1 and Profile 2 so I could tell them apart. But the working and the not working query use the same one. (Profile 2).

Azendale
Posts: 13
Joined: Sat Feb 21, 2015 6:15 pm

Re: Cacti not creating data sources or rrd files when it sho

#14 Post by Azendale » Fri Jul 20, 2018 6:48 pm

netniV wrote:If you change the XML you should probably be running a poller output clear / reindex to ensure that everything is in sync.
I did try that, and then re-creating graphs & data sources. It didn't seem to fix it.

At this point, I think I would be open to allowing a login to our server with SSH (send me a public SSH key / IP range to allow) and sharing a screen session to try to find the issue, if that helps.


Post Reply