POLLING PROBLEM

Post support questions that directly relate to Linux/Unix operating systems.

Moderators: Moderators, Developers

Post Reply
Author
Message
Lalbee99
Cacti User
Posts: 95
Joined: Mon Mar 03, 2008 2:07 pm

POLLING PROBLEM

#1 Post by Lalbee99 » Tue Sep 03, 2019 7:17 am

I have a cacti installation running 1.2.3 moitoring 109 clients. Everything has been running fine for a few months. I have recently added a client that has a swap graph that shows up on my cacti error check report. In this report I list RRD/Graphs that have not updated in over an hour. When I check this host I notice if I do a manual snmpwalk of the mib tree for the oid's associated with memTotalSwap, memAvailSwap it returns with no problems (see below). When I run spine in read only for the client I notice it also returns the values for memTotalSwap, memAvailSwap (see below). When I check the cacti.log I can also see values being returned by the poller for memTotalSwap, memAvailSwap (see below). When I perfomr a 4 hour dump of the round robin database associated with the graph I get alot NaN values with only 1 polled value. There are 94 other clients that are running the exact same swap template with no issues at all.
////////////
SNMPWALK
///////////
snmpwalk -v3 -l authNoPriv -u cactiuser -A XXXXXXX SERVERNAME .1.3.6.1.4.1.2021.4.4
UCD-SNMP-MIB::memAvailSwap.0 = INTEGER: 4194300 kB
snmpwalk -v3 -l authNoPriv -u cactiuser -A XXXXXXX SERVERNAME .1.3.6.1.4.1.2021.4.3
UCD-SNMP-MIB::memTotalSwap.0 = INTEGER: 4194300 kB

////////////
SPINE
////////////
/usr/local/spine/bin/spine -R -C /usr/local/spine/etc/spine.conf -H=395 -V 5
2019/09/03 07:53:37 - SPINE: Poller[1] Device[395] HT[1] DS[10938] SNMP: v3: SERVERNAME.DOMAIN.XXX, dsname: memAvailSwap, oid: .1.3.6.1.4.1.2021.4.4.0, value: 4194300
2019/09/03 07:53:37 - SPINE: Poller[1] Device[395] HT[1] DS[10938] SNMP: v3: SERVERNAME.DOMAIN.XXX, dsname: memTotalSwap, oid: .1.3.6.1.4.1.2021.4.3.0, value: 4194300

//////////
CACTI.LOG
//////////
2019/09/03 06:00:19 - SPINE: Poller[1] Device[395] HT[1] DS[10938] SNMP: v3: SERVERNAME.DOMAIN.XXX, dsname: memAvailSwap, oid: .1.3.6.1.4.1.2021.4.4.0, value: 4194300
2019/09/03 06:00:19 - SPINE: Poller[1] Device[395] HT[1] DS[10938] SNMP: v3: SERVERNAME.DOMAIN.XXX, dsname: memTotalSwap, oid: .1.3.6.1.4.1.2021.4.3.0, value: 4194300
2019/09/03 06:05:16 - SPINE: Poller[1] Device[395] HT[1] DS[10938] SNMP: v3: SERVERNAME.DOMAIN.XXX, dsname: memAvailSwap, oid: .1.3.6.1.4.1.2021.4.4.0, value: 4194300
2019/09/03 06:05:16 - SPINE: Poller[1] Device[395] HT[1] DS[10938] SNMP: v3: SERVERNAME.DOMAIN.XXX, dsname: memTotalSwap, oid: .1.3.6.1.4.1.2021.4.3.0, value: 4194300
2019/09/03 06:10:17 - SPINE: Poller[1] Device[395] HT[1] DS[10938] SNMP: v3: SERVERNAME.DOMAIN.XXX, dsname: memAvailSwap, oid: .1.3.6.1.4.1.2021.4.4.0, value: 4194300
2019/09/03 06:10:17 - SPINE: Poller[1] Device[395] HT[1] DS[10938] SNMP: v3: SERVERNAME.DOMAIN.XXX, dsname: memTotalSwap, oid: .1.3.6.1.4.1.2021.4.3.0, value: 4194300
2019/09/03 06:15:21 - SPINE: Poller[1] Device[395] HT[1] DS[10938] SNMP: v3: SERVERNAME.DOMAIN.XXX, dsname: memAvailSwap, oid: .1.3.6.1.4.1.2021.4.4.0, value: 4194300
2019/09/03 06:15:21 - SPINE: Poller[1] Device[395] HT[1] DS[10938] SNMP: v3: SERVERNAME.DOMAIN.XXX, dsname: memTotalSwap, oid: .1.3.6.1.4.1.2021.4.3.0, value: 4194300
2019/09/03 06:20:15 - SPINE: Poller[1] Device[395] HT[1] DS[10938] SNMP: v3: SERVERNAME.DOMAIN.XXX, dsname: memAvailSwap, oid: .1.3.6.1.4.1.2021.4.4.0, value: 4194300
2019/09/03 06:20:15 - SPINE: Poller[1] Device[395] HT[1] DS[10938] SNMP: v3: SERVERNAME.DOMAIN.XXX, dsname: memTotalSwap, oid: .1.3.6.1.4.1.2021.4.3.0, value: 4194300
2019/09/03 06:25:18 - SPINE: Poller[1] Device[395] HT[1] DS[10938] SNMP: v3: SERVERNAME.DOMAIN.XXX, dsname: memAvailSwap, oid: .1.3.6.1.4.1.2021.4.4.0, value: 4194300
2019/09/03 06:25:18 - SPINE: Poller[1] Device[395] HT[1] DS[10938] SNMP: v3: SERVERNAME.DOMAIN.XXX, dsname: memTotalSwap, oid: .1.3.6.1.4.1.2021.4.3.0, value: 4194300
2019/09/03 06:30:15 - SPINE: Poller[1] Device[395] HT[1] DS[10938] SNMP: v3: SERVERNAME.DOMAIN.XXX, dsname: memAvailSwap, oid: .1.3.6.1.4.1.2021.4.4.0, value: 4194300
2019/09/03 06:30:15 - SPINE: Poller[1] Device[395] HT[1] DS[10938] SNMP: v3: SERVERNAME.DOMAIN.XXX, dsname: memTotalSwap, oid: .1.3.6.1.4.1.2021.4.3.0, value: 4194300
2019/09/03 06:35:20 - SPINE: Poller[1] Device[395] HT[1] DS[10938] SNMP: v3: SERVERNAME.DOMAIN.XXX, dsname: memAvailSwap, oid: .1.3.6.1.4.1.2021.4.4.0, value: 4194300
2019/09/03 06:35:20 - SPINE: Poller[1] Device[395] HT[1] DS[10938] SNMP: v3: SERVERNAME.DOMAIN.XXX, dsname: memTotalSwap, oid: .1.3.6.1.4.1.2021.4.3.0, value: 4194300
2019/09/03 06:40:15 - SPINE: Poller[1] Device[395] HT[1] DS[10938] SNMP: v3: SERVERNAME.DOMAIN.XXX, dsname: memAvailSwap, oid: .1.3.6.1.4.1.2021.4.4.0, value: 4194300
2019/09/03 06:40:15 - SPINE: Poller[1] Device[395] HT[1] DS[10938] SNMP: v3: SERVERNAME.DOMAIN.XXX, dsname: memTotalSwap, oid: .1.3.6.1.4.1.2021.4.3.0, value: 4194300
2019/09/03 06:45:18 - SPINE: Poller[1] Device[395] HT[1] DS[10938] SNMP: v3: SERVERNAME.DOMAIN.XXX, dsname: memAvailSwap, oid: .1.3.6.1.4.1.2021.4.4.0, value: 4194300
2019/09/03 06:45:18 - SPINE: Poller[1] Device[395] HT[1] DS[10938] SNMP: v3: SERVERNAME.DOMAIN.XXX, dsname: memTotalSwap, oid: .1.3.6.1.4.1.2021.4.3.0, value: 4194300
2019/09/03 06:50:16 - SPINE: Poller[1] Device[395] HT[1] DS[10938] SNMP: v3: SERVERNAME.DOMAIN.XXX, dsname: memAvailSwap, oid: .1.3.6.1.4.1.2021.4.4.0, value: 4194300
2019/09/03 06:50:16 - SPINE: Poller[1] Device[395] HT[1] DS[10938] SNMP: v3: SERVERNAME.DOMAIN.XXX, dsname: memTotalSwap, oid: .1.3.6.1.4.1.2021.4.3.0, value: 4194300
2019/09/03 06:55:16 - SPINE: Poller[1] Device[395] HT[1] DS[10938] SNMP: v3: SERVERNAME.DOMAIN.XXX, dsname: memAvailSwap, oid: .1.3.6.1.4.1.2021.4.4.0, value: 4194300
2019/09/03 06:55:16 - SPINE: Poller[1] Device[395] HT[1] DS[10938] SNMP: v3: SERVERNAME.DOMAIN.XXX, dsname: memTotalSwap, oid: .1.3.6.1.4.1.2021.4.3.0, value: 4194300
2019/09/03 07:00:19 - SPINE: Poller[1] Device[395] HT[1] DS[10938] SNMP: v3: SERVERNAME.DOMAIN.XXX, dsname: memAvailSwap, oid: .1.3.6.1.4.1.2021.4.4.0, value: 4194300
2019/09/03 07:00:19 - SPINE: Poller[1] Device[395] HT[1] DS[10938] SNMP: v3: SERVERNAME.DOMAIN.XXX, dsname: memTotalSwap, oid: .1.3.6.1.4.1.2021.4.3.0, value: 4194300
2019/09/03 07:05:15 - SPINE: Poller[1] Device[395] HT[1] DS[10938] SNMP: v3: SERVERNAME.DOMAIN.XXX, dsname: memAvailSwap, oid: .1.3.6.1.4.1.2021.4.4.0, value: 4194300
2019/09/03 07:05:15 - SPINE: Poller[1] Device[395] HT[1] DS[10938] SNMP: v3: SERVERNAME.DOMAIN.XXX, dsname: memTotalSwap, oid: .1.3.6.1.4.1.2021.4.3.0, value: 4194300
2019/09/03 07:10:17 - SPINE: Poller[1] Device[395] HT[1] DS[10938] SNMP: v3: SERVERNAME.DOMAIN.XXX, dsname: memAvailSwap, oid: .1.3.6.1.4.1.2021.4.4.0, value: 4194300
2019/09/03 07:10:17 - SPINE: Poller[1] Device[395] HT[1] DS[10938] SNMP: v3: SERVERNAME.DOMAIN.XXX, dsname: memTotalSwap, oid: .1.3.6.1.4.1.2021.4.3.0, value: 4194300
2019/09/03 07:15:15 - SPINE: Poller[1] Device[395] HT[1] DS[10938] SNMP: v3: SERVERNAME.DOMAIN.XXX, dsname: memAvailSwap, oid: .1.3.6.1.4.1.2021.4.4.0, value: 4194300
2019/09/03 07:15:15 - SPINE: Poller[1] Device[395] HT[1] DS[10938] SNMP: v3: SERVERNAME.DOMAIN.XXX, dsname: memTotalSwap, oid: .1.3.6.1.4.1.2021.4.3.0, value: 4194300
2019/09/03 07:20:18 - SPINE: Poller[1] Device[395] HT[1] DS[10938] SNMP: v3: SERVERNAME.DOMAIN.XXX, dsname: memAvailSwap, oid: .1.3.6.1.4.1.2021.4.4.0, value: 4194300
2019/09/03 07:20:18 - SPINE: Poller[1] Device[395] HT[1] DS[10938] SNMP: v3: SERVERNAME.DOMAIN.XXX, dsname: memTotalSwap, oid: .1.3.6.1.4.1.2021.4.3.0, value: 4194300
2019/09/03 07:25:16 - SPINE: Poller[1] Device[395] HT[1] DS[10938] SNMP: v3: SERVERNAME.DOMAIN.XXX, dsname: memAvailSwap, oid: .1.3.6.1.4.1.2021.4.4.0, value: 4194300
2019/09/03 07:25:16 - SPINE: Poller[1] Device[395] HT[1] DS[10938] SNMP: v3: SERVERNAME.DOMAIN.XXX, dsname: memTotalSwap, oid: .1.3.6.1.4.1.2021.4.3.0, value: 4194300
2019/09/03 07:30:18 - SPINE: Poller[1] Device[395] HT[1] DS[10938] SNMP: v3: SERVERNAME.DOMAIN.XXX, dsname: memAvailSwap, oid: .1.3.6.1.4.1.2021.4.4.0, value: 4194300
2019/09/03 07:30:18 - SPINE: Poller[1] Device[395] HT[1] DS[10938] SNMP: v3: SERVERNAME.DOMAIN.XXX, dsname: memTotalSwap, oid: .1.3.6.1.4.1.2021.4.3.0, value: 4194300
2019/09/03 07:35:17 - SPINE: Poller[1] Device[395] HT[1] DS[10938] SNMP: v3: SERVERNAME.DOMAIN.XXX, dsname: memAvailSwap, oid: .1.3.6.1.4.1.2021.4.4.0, value: 4194300
2019/09/03 07:35:17 - SPINE: Poller[1] Device[395] HT[1] DS[10938] SNMP: v3: SERVERNAME.DOMAIN.XXX, dsname: memTotalSwap, oid: .1.3.6.1.4.1.2021.4.3.0, value: 4194300
2019/09/03 07:40:19 - SPINE: Poller[1] Device[395] HT[1] DS[10938] SNMP: v3: SERVERNAME.DOMAIN.XXX, dsname: memAvailSwap, oid: .1.3.6.1.4.1.2021.4.4.0, value: 4194300
2019/09/03 07:40:19 - SPINE: Poller[1] Device[395] HT[1] DS[10938] SNMP: v3: SERVERNAME.DOMAIN.XXX, dsname: memTotalSwap, oid: .1.3.6.1.4.1.2021.4.3.0, value: 4194300
2019/09/03 07:45:21 - SPINE: Poller[1] Device[395] HT[1] DS[10938] SNMP: v3: SERVERNAME.DOMAIN.XXX, dsname: memAvailSwap, oid: .1.3.6.1.4.1.2021.4.4.0, value: 4194300
2019/09/03 07:45:21 - SPINE: Poller[1] Device[395] HT[1] DS[10938] SNMP: v3: SERVERNAME.DOMAIN.XXX, dsname: memTotalSwap, oid: .1.3.6.1.4.1.2021.4.3.0, value: 4194300
2019/09/03 07:50:15 - SPINE: Poller[1] Device[395] HT[1] DS[10938] SNMP: v3: SERVERNAME.DOMAIN.XXX, dsname: memAvailSwap, oid: .1.3.6.1.4.1.2021.4.4.0, value: 4194300
2019/09/03 07:50:15 - SPINE: Poller[1] Device[395] HT[1] DS[10938] SNMP: v3: SERVERNAME.DOMAIN.XXX, dsname: memTotalSwap, oid: .1.3.6.1.4.1.2021.4.3.0, value: 4194300
2019/09/03 07:53:37 - SPINE: Poller[1] Device[395] HT[1] DS[10938] SNMP: v3: SERVERNAME.DOMAIN.XXX, dsname: memAvailSwap, oid: .1.3.6.1.4.1.2021.4.4.0, value: 4194300
2019/09/03 07:53:37 - SPINE: Poller[1] Device[395] HT[1] DS[10938] SNMP: v3: SERVERNAME.DOMAIN.XXX, dsname: memTotalSwap, oid: .1.3.6.1.4.1.2021.4.3.0, value: 4194300
2019/09/03 07:55:18 - SPINE: Poller[1] Device[395] HT[1] DS[10938] SNMP: v3: SERVERNAME.DOMAIN.XXX, dsname: memAvailSwap, oid: .1.3.6.1.4.1.2021.4.4.0, value: 4194300
2019/09/03 07:55:18 - SPINE: Poller[1] Device[395] HT[1] DS[10938] SNMP: v3: SERVERNAME.DOMAIN.XXX, dsname: memTotalSwap, oid: .1.3.6.1.4.1.2021.4.3.0, value: 4194300
2019/09/03 07:57:20 - SPINE: Poller[1] Device[395] HT[1] DS[10938] SNMP: v3: SERVERNAME.DOMAIN.XXX, dsname: memAvailSwap, oid: .1.3.6.1.4.1.2021.4.4.0, value: 4194300
2019/09/03 07:57:20 - SPINE: Poller[1] Device[395] HT[1] DS[10938] SNMP: v3: SERVERNAME.DOMAIN.XXX, dsname: memTotalSwap, oid: .1.3.6.1.4.1.2021.4.3.0, value: 4194300
2019/09/03 08:00:18 - SPINE: Poller[1] Device[395] HT[1] DS[10938] SNMP: v3: SERVERNAME.DOMAIN.XXX, dsname: memAvailSwap, oid: .1.3.6.1.4.1.2021.4.4.0, value: 4194300
2019/09/03 08:00:18 - SPINE: Poller[1] Device[395] HT[1] DS[10938] SNMP: v3: SERVERNAME.DOMAIN.XXX, dsname: memTotalSwap, oid: .1.3.6.1.4.1.2021.4.3.0, value: 4194300
2019/09/03 08:05:16 - SPINE: Poller[1] Device[395] HT[1] DS[10938] SNMP: v3: SERVERNAME.DOMAIN.XXX, dsname: memAvailSwap, oid: .1.3.6.1.4.1.2021.4.4.0, value: 4194300
2019/09/03 08:05:16 - SPINE: Poller[1] Device[395] HT[1] DS[10938] SNMP: v3: SERVERNAME.DOMAIN.XXX, dsname: memTotalSwap, oid: .1.3.6.1.4.1.2021.4.3.0, value: 4194300
2019/09/03 08:10:16 - SPINE: Poller[1] Device[395] HT[1] DS[10938] SNMP: v3: SERVERNAME.DOMAIN.XXX, dsname: memAvailSwap, oid: .1.3.6.1.4.1.2021.4.4.0, value: 4194300
2019/09/03 08:10:16 - SPINE: Poller[1] Device[395] HT[1] DS[10938] SNMP: v3: SERVERNAME.DOMAIN.XXX, dsname: memTotalSwap, oid: .1.3.6.1.4.1.2021.4.3.0, value: 4194300


////////////////////
RRDTOOL DUMP
///////////////////
rrdtool fetch cfextwebtst2_memtotalswap_10938.rrd AVERAGE -r 300 -s -4hr
memTotalSwap memAvailSwap

1567498500: -nan -nan
1567498800: -nan -nan
1567499100: -nan -nan
1567499400: -nan -nan
1567499700: -nan -nan
1567500000: -nan -nan
1567500300: -nan -nan
1567500600: -nan -nan
1567500900: -nan -nan
1567501200: -nan -nan
1567501500: -nan -nan
1567501800: -nan -nan
1567502100: -nan -nan
1567502400: -nan -nan
1567502700: -nan -nan
1567503000: -nan -nan
1567503300: -nan -nan
1567503600: -nan -nan
1567503900: -nan -nan
1567504200: -nan -nan
1567504500: -nan -nan
1567504800: -nan -nan
1567505100: -nan -nan
1567505400: -nan -nan
1567505700: -nan -nan
1567506000: -nan -nan
1567506300: -nan -nan
1567506600: -nan -nan
1567506900: -nan -nan
1567507200: -nan -nan
1567507500: -nan -nan
1567507800: -nan -nan
1567508100: -nan -nan
1567508400: -nan -nan
1567508700: -nan -nan
1567509000: -nan -nan
1567509300: -nan -nan
1567509600: -nan -nan
1567509900: -nan -nan
1567510200: -nan -nan
1567510500: -nan -nan
1567510800: -nan -nan
1567511100: -nan -nan
1567511400: 4.1943000000e+06 4.1943000000e+06
1567511700: -nan -nan
1567512000: -nan -nan
1567512300: -nan -nan
1567512600: -nan -nan
1567512900: -nan -nan

netniV
Cacti Guru User
Posts: 2881
Joined: Sun Aug 27, 2017 12:05 am

Re: POLLING PROBLEM

#2 Post by netniV » Tue Sep 03, 2019 9:05 am

Put that device into debug mode then check the logs to see what is occurring, it may be something outside of cacti, such as file permissions etc.

Lalbee99
Cacti User
Posts: 95
Joined: Mon Mar 03, 2008 2:07 pm

Re: POLLING PROBLEM

#3 Post by Lalbee99 » Tue Sep 03, 2019 9:19 am

That was the first thing that came to my mind. All the rrd's have the same owner/group & perms 664. The spine command has DEBUG on already and there was nothing jumping out. I will give you the full listing below:L

/usr/local/spine/bin/spine -R -C /usr/local/spine/etc/spine.conf -H=395 -V 5

SPINE: Using spine config file [/usr/local/spine/etc/spine.conf]
WARNING: DB_PreG is no longer supported
2019/09/03 10:16:03 - SPINE: Poller[1] DEBUG: The path_php_server variable is /cacti/cacti/script_server.php
2019/09/03 10:16:03 - SPINE: Poller[1] DEBUG: The path_cactilog variable is /cacti/cacti/log/cacti.log
2019/09/03 10:16:03 - SPINE: Poller[1] DEBUG: The log_destination variable is 1 (FILE)
2019/09/03 10:16:03 - SPINE: Poller[1] DEBUG: The path_php variable is /usr/bin/php
2019/09/03 10:16:03 - SPINE: Poller[1] DEBUG: The availability_method variable is 4
2019/09/03 10:16:03 - SPINE: Poller[1] DEBUG: The ping_recovery_count variable is 1
2019/09/03 10:16:03 - SPINE: Poller[1] DEBUG: The ping_failure_count variable is 2
2019/09/03 10:16:03 - SPINE: Poller[1] DEBUG: The ping_method variable is 1
2019/09/03 10:16:03 - SPINE: Poller[1] DEBUG: The ping_retries variable is 3
2019/09/03 10:16:03 - SPINE: Poller[1] DEBUG: The ping_timeout variable is 2000
2019/09/03 10:16:03 - SPINE: Poller[1] DEBUG: The snmp_retries variable is 3
2019/09/03 10:16:03 - SPINE: Poller[1] DEBUG: The log_perror variable is 1
2019/09/03 10:16:03 - SPINE: Poller[1] DEBUG: The log_pwarn variable is 0
2019/09/03 10:16:03 - SPINE: Poller[1] DEBUG: The boost_redirect variable is 1
2019/09/03 10:16:03 - SPINE: Poller[1] DEBUG: The boost_rrd_update_enable variable is 1
2019/09/03 10:16:03 - SPINE: Poller[1] DEBUG: The log_pstats variable is 0
2019/09/03 10:16:03 - SPINE: Poller[1] DEBUG: The threads variable is 60
2019/09/03 10:16:03 - SPINE: Poller[1] DEBUG: The polling interval is 300 seconds
2019/09/03 10:16:03 - SPINE: Poller[1] DEBUG: The number of concurrent processes is 2
2019/09/03 10:16:03 - SPINE: Poller[1] DEBUG: The script timeout is 20
2019/09/03 10:16:03 - SPINE: Poller[1] DEBUG: The selective_device_debug variable is 395
2019/09/03 10:16:03 - SPINE: Poller[1] DEBUG: The spine_log_level variable is 2
2019/09/03 10:16:03 - SPINE: Poller[1] DEBUG: The number of php script servers to run is 10
2019/09/03 10:16:03 - SPINE: Poller[1] DEBUG: Device List to be polled='395', TotalPHPScripts='1'
2019/09/03 10:16:03 - SPINE: Poller[1] DEBUG: The PHP Script Server is Required
2019/09/03 10:16:03 - SPINE: Poller[1] DEBUG: The Maximum SNMP OID Get Size is 80
2019/09/03 10:16:03 - SPINE: Poller[1] Selective Debug Devices 395
2019/09/03 10:16:03 - SPINE: Poller[1] Version 1.2.3 starting
2019/09/03 10:16:03 - SPINE: Poller[1] DEBUG: MySQL is Thread Safe!
2019/09/03 10:16:03 - SPINE: Poller[1] DEBUG: Spine running as 571 UID, 0 EUID
2019/09/03 10:16:03 - SPINE: Poller[1] DEBUG: Spine is running as root.
2019/09/03 10:16:03 - SPINE: Poller[1] DEBUG: Spine has got ICMP
2019/09/03 10:16:03 - SPINE: Poller[1] SPINE: Initializing Net-SNMP API
2019/09/03 10:16:03 - SPINE: Poller[1] DEBUG: Issues with SNMP Header Version information, assuming old version of Net-SNMP.
2019/09/03 10:16:03 - SPINE: Poller[1] SPINE: Initializing PHP Script Server(s)
2019/09/03 10:16:03 - SPINE: Poller[1] DEBUG: SS[0] PHP Script Server Routine Starting
2019/09/03 10:16:03 - SPINE: Poller[1] DEBUG: SS[0] PHP Script Server About to FORK Child Process
2019/09/03 10:16:03 - SPINE: Poller[1] DEBUG: SS[0] PHP Script Server Child FORK Success
2019/09/03 10:16:03 - SPINE: Poller[1] DEBUG: SS[0] Confirmed PHP Script Server running using readfd[7], writefd[6]
2019/09/03 10:16:03 - SPINE: Poller[1] DEBUG: SS[1] PHP Script Server Routine Starting
2019/09/03 10:16:03 - SPINE: Poller[1] DEBUG: SS[1] PHP Script Server About to FORK Child Process
2019/09/03 10:16:03 - SPINE: Poller[1] DEBUG: SS[1] PHP Script Server Child FORK Success
2019/09/03 10:16:03 - SPINE: Poller[1] DEBUG: SS[1] Confirmed PHP Script Server running using readfd[9], writefd[8]
2019/09/03 10:16:03 - SPINE: Poller[1] DEBUG: SS[2] PHP Script Server Routine Starting
2019/09/03 10:16:03 - SPINE: Poller[1] DEBUG: SS[2] PHP Script Server About to FORK Child Process
2019/09/03 10:16:03 - SPINE: Poller[1] DEBUG: SS[2] PHP Script Server Child FORK Success
2019/09/03 10:16:04 - SPINE: Poller[1] DEBUG: SS[2] Confirmed PHP Script Server running using readfd[11], writefd[10]
2019/09/03 10:16:04 - SPINE: Poller[1] DEBUG: SS[3] PHP Script Server Routine Starting
2019/09/03 10:16:04 - SPINE: Poller[1] DEBUG: SS[3] PHP Script Server About to FORK Child Process
2019/09/03 10:16:04 - SPINE: Poller[1] DEBUG: SS[3] PHP Script Server Child FORK Success
2019/09/03 10:16:04 - SPINE: Poller[1] DEBUG: SS[3] Confirmed PHP Script Server running using readfd[13], writefd[12]
2019/09/03 10:16:04 - SPINE: Poller[1] DEBUG: SS[4] PHP Script Server Routine Starting
2019/09/03 10:16:04 - SPINE: Poller[1] DEBUG: SS[4] PHP Script Server About to FORK Child Process
2019/09/03 10:16:04 - SPINE: Poller[1] DEBUG: SS[4] PHP Script Server Child FORK Success
2019/09/03 10:16:04 - SPINE: Poller[1] DEBUG: SS[4] Confirmed PHP Script Server running using readfd[15], writefd[14]
2019/09/03 10:16:04 - SPINE: Poller[1] DEBUG: SS[5] PHP Script Server Routine Starting
2019/09/03 10:16:04 - SPINE: Poller[1] DEBUG: SS[5] PHP Script Server About to FORK Child Process
2019/09/03 10:16:04 - SPINE: Poller[1] DEBUG: SS[5] PHP Script Server Child FORK Success
2019/09/03 10:16:05 - SPINE: Poller[1] DEBUG: SS[5] Confirmed PHP Script Server running using readfd[17], writefd[16]
2019/09/03 10:16:05 - SPINE: Poller[1] DEBUG: SS[6] PHP Script Server Routine Starting
2019/09/03 10:16:05 - SPINE: Poller[1] DEBUG: SS[6] PHP Script Server About to FORK Child Process
2019/09/03 10:16:05 - SPINE: Poller[1] DEBUG: SS[6] PHP Script Server Child FORK Success
2019/09/03 10:16:05 - SPINE: Poller[1] DEBUG: SS[6] Confirmed PHP Script Server running using readfd[19], writefd[18]
2019/09/03 10:16:05 - SPINE: Poller[1] DEBUG: SS[7] PHP Script Server Routine Starting
2019/09/03 10:16:05 - SPINE: Poller[1] DEBUG: SS[7] PHP Script Server About to FORK Child Process
2019/09/03 10:16:05 - SPINE: Poller[1] DEBUG: SS[7] PHP Script Server Child FORK Success
2019/09/03 10:16:05 - SPINE: Poller[1] DEBUG: SS[7] Confirmed PHP Script Server running using readfd[21], writefd[20]
2019/09/03 10:16:05 - SPINE: Poller[1] DEBUG: SS[8] PHP Script Server Routine Starting
2019/09/03 10:16:05 - SPINE: Poller[1] DEBUG: SS[8] PHP Script Server About to FORK Child Process
2019/09/03 10:16:05 - SPINE: Poller[1] DEBUG: SS[8] PHP Script Server Child FORK Success
2019/09/03 10:16:06 - SPINE: Poller[1] DEBUG: SS[8] Confirmed PHP Script Server running using readfd[23], writefd[22]
2019/09/03 10:16:06 - SPINE: Poller[1] DEBUG: SS[9] PHP Script Server Routine Starting
2019/09/03 10:16:06 - SPINE: Poller[1] DEBUG: SS[9] PHP Script Server About to FORK Child Process
2019/09/03 10:16:06 - SPINE: Poller[1] DEBUG: SS[9] PHP Script Server Child FORK Success
2019/09/03 10:16:06 - SPINE: Poller[1] DEBUG: SS[9] Confirmed PHP Script Server running using readfd[25], writefd[24]
2019/09/03 10:16:06 - SPINE: Poller[1] NOTE: Spine will support multithread device polling.
2019/09/03 10:16:06 - SPINE: Poller[1] DEBUG: Initial Value of Active Threads is 0
2019/09/03 10:16:06 - SPINE: Poller[1] DEBUG: Valid Thread to be Created
2019/09/03 10:16:06 - SPINE: Poller[1] SPINE: Active Threads is 1, Pending is 1
2019/09/03 10:16:06 - SPINE: Poller[1] DEBUG: In Poller, About to Start Polling of Device for Device ID 395
2019/09/03 10:16:06 - SPINE: Poller[1] Device[395] DEBUG: Entering ICMP Ping
2019/09/03 10:16:06 - SPINE: Poller[1] Device[395] DEBUG: ICMP Device Alive, Try Count:1, Time:2.0800 ms
2019/09/03 10:16:06 - SPINE: Poller[1] Device[395] PING Result: ICMP: Device is Alive
2019/09/03 10:16:06 - SPINE: Poller[1] Device[395] SNMP Result: SNMP not performed due to setting or ping result
2019/09/03 10:16:06 - SPINE: Poller[1] Updating Full System Information Table
2019/09/03 10:16:06 - SPINE: Poller[1] Device[395] HT[1] RECACHE: Processing 5 items in the auto reindex cache for 'SERVERNAME.DOMAIN.XXX'
2019/09/03 10:16:06 - SPINE: Poller[1] Device[395] HT[1] DQ[1] RECACHE OID: .1.3.6.1.2.1.1.3.0, (assert: 121227792 < output: 121233766)
2019/09/03 10:16:06 - SPINE: Poller[1] Device[395] HT[1] DQ[85] RECACHE OID: .1.3.6.1.2.1.1.3.0, (assert: 121227792 < output: 121233766)
2019/09/03 10:16:06 - SPINE: Poller[1] Device[395] HT[1] DQ[88] RECACHE OID: .1.3.6.1.2.1.1.3.0, (assert: 121227792 < output: 121233766)
2019/09/03 10:16:06 - SPINE: Poller[1] Device[395] HT[1] DQ[89] RECACHE OID: .1.3.6.1.2.1.1.3.0, (assert: 121227792 < output: 121233766)
2019/09/03 10:16:06 - SPINE: Poller[1] Device[395] HT[1] DQ[92] RECACHE OID: .1.3.6.1.2.1.1.3.0, (assert: 121227792 < output: 121233766)
2019/09/03 10:16:06 - SPINE: Poller[1] Device[395] HT[1] NOTE: There are '34' Polling Items for this Device
2019/09/03 10:16:06 - SPINE: Poller[1] Device[395] HT[1] DS[10698] SS[0] SERVER: /cacti/cacti/scripts/ss_fping.php ss_fping 'SERVERNAME.DOMAIN.XXX' '20' 'ICMP' , output: min:1.6980 avg:1.9672 max:2.3490 dev:0.1744 loss:0.0000
2019/09/03 10:16:06 - SPINE: Poller[1] Device[395] DEBUG: The NIFTY POPEN returned the following File Descriptor 28
2019/09/03 10:16:07 - SPINE: Poller[1] Device[395] HT[1] DS[10712] SCRIPT: perl /cacti/cacti/scripts/chb_rhel_memory.pl 'SERVERNAME.DOMAIN.XXX' 'get' 'ChbRhelFree' 'ChbRhelUsed', output: 3.4
2019/09/03 10:16:07 - SPINE: Poller[1] Device[395] DEBUG: The NIFTY POPEN returned the following File Descriptor 28
2019/09/03 10:16:08 - SPINE: Poller[1] Device[395] HT[1] DS[10712] SCRIPT: perl /cacti/cacti/scripts/chb_rhel_memory.pl 'SERVERNAME.DOMAIN.XXX' 'get' 'ChbRhelUsed' 'ChbRhelUsed', output: 0.3
2019/09/03 10:16:08 - SPINE: Poller[1] Device[395] DEBUG: The NIFTY POPEN returned the following File Descriptor 28
2019/09/03 10:16:09 - SPINE: Poller[1] Device[395] HT[1] DS[10712] SCRIPT: perl /cacti/cacti/scripts/chb_rhel_memory.pl 'SERVERNAME.DOMAIN.XXX' 'get' 'ChbRhelTotal' 'ChbRhelUsed', output: 3.7
2019/09/03 10:16:09 - SPINE: Poller[1] Device[395] DEBUG: The NIFTY POPEN returned the following File Descriptor 28
2019/09/03 10:16:11 - SPINE: Poller[1] Device[395] HT[1] DS[10712] SCRIPT: perl /cacti/cacti/scripts/chb_rhel_memory.pl 'SERVERNAME.DOMAIN.XXX' 'get' 'ChbRhelCached' 'ChbRhelUsed', output: 1.2
2019/09/03 10:16:11 - SPINE: Poller[1] Device[395] DEBUG: The NIFTY POPEN returned the following File Descriptor 28
2019/09/03 10:16:12 - SPINE: Poller[1] Device[395] HT[1] DS[10712] SCRIPT: perl /cacti/cacti/scripts/chb_rhel_memory.pl 'SERVERNAME.DOMAIN.XXX' 'get' 'ChbRhelBuffers' 'ChbRhelUsed', output: 0.4
2019/09/03 10:16:12 - SPINE: Poller[1] Device[395] HT[1] DS[10797] SNMP: v3: SERVERNAME.DOMAIN.XXX, dsname: ssCpuIdle, oid: .1.3.6.1.4.1.2021.11.11.0, value: 82
2019/09/03 10:16:12 - SPINE: Poller[1] Device[395] HT[1] DS[10797] SNMP: v3: SERVERNAME.DOMAIN.XXX, dsname: ssCpuUser, oid: .1.3.6.1.4.1.2021.11.9.0, value: 4
2019/09/03 10:16:12 - SPINE: Poller[1] Device[395] HT[1] DS[10797] SNMP: v3: SERVERNAME.DOMAIN.XXX, dsname: ssCpuSystem, oid: .1.3.6.1.4.1.2021.11.10.0, value: 13
2019/09/03 10:16:12 - SPINE: Poller[1] Device[395] HT[1] DS[10938] SNMP: v3: SERVERNAME.DOMAIN.XXX, dsname: memAvailSwap, oid: .1.3.6.1.4.1.2021.4.4.0, value: 4194300
2019/09/03 10:16:12 - SPINE: Poller[1] Device[395] HT[1] DS[10938] SNMP: v3: SERVERNAME.DOMAIN.XXX, dsname: memTotalSwap, oid: .1.3.6.1.4.1.2021.4.3.0, value: 4194300
2019/09/03 10:16:12 - SPINE: Poller[1] Device[395] HT[1] DS[10699] SNMP: v3: SERVERNAME.DOMAIN.XXX, dsname: ucd_load15min, oid: .1.3.6.1.4.1.2021.10.1.3.3, value: 0.05
2019/09/03 10:16:12 - SPINE: Poller[1] Device[395] HT[1] DS[10700] SNMP: v3: SERVERNAME.DOMAIN.XXX, dsname: ucd_load1min, oid: .1.3.6.1.4.1.2021.10.1.3.1, value: 0.10
2019/09/03 10:16:12 - SPINE: Poller[1] Device[395] HT[1] DS[10701] SNMP: v3: SERVERNAME.DOMAIN.XXX, dsname: ucd_load5min, oid: .1.3.6.1.4.1.2021.10.1.3.2, value: 0.05
2019/09/03 10:16:12 - SPINE: Poller[1] Device[395] HT[1] DS[10702] SNMP: v3: SERVERNAME.DOMAIN.XXX, dsname: ucd_hrSystemProcess, oid: .1.3.6.1.2.1.25.1.6.0, value: 182
2019/09/03 10:16:12 - SPINE: Poller[1] Device[395] HT[1] DS[10703] SNMP: v3: SERVERNAME.DOMAIN.XXX, dsname: users, oid: .1.3.6.1.2.1.25.1.5.0, value: 0
2019/09/03 10:16:12 - SPINE: Poller[1] Device[395] HT[1] DS[10704] SNMP: v3: SERVERNAME.DOMAIN.XXX, dsname: hrStorageSize, oid: 1.3.6.1.2.1.25.2.3.1.5.31, value: 2547525
2019/09/03 10:16:12 - SPINE: Poller[1] Device[395] HT[1] DS[10704] SNMP: v3: SERVERNAME.DOMAIN.XXX, dsname: hrStorageUsed, oid: 1.3.6.1.2.1.25.2.3.1.6.31, value: 1283643
2019/09/03 10:16:12 - SPINE: Poller[1] Device[395] HT[1] DS[10705] SNMP: v3: SERVERNAME.DOMAIN.XXX, dsname: hrStorageSize, oid: 1.3.6.1.2.1.25.2.3.1.5.58, value: 757700
2019/09/03 10:16:12 - SPINE: Poller[1] Device[395] HT[1] DS[10705] SNMP: v3: SERVERNAME.DOMAIN.XXX, dsname: hrStorageUsed, oid: 1.3.6.1.2.1.25.2.3.1.6.58, value: 53352
2019/09/03 10:16:12 - SPINE: Poller[1] Device[395] HT[1] DS[10706] SNMP: v3: SERVERNAME.DOMAIN.XXX, dsname: hrStorageSize, oid: 1.3.6.1.2.1.25.2.3.1.5.60, value: 1257376
2019/09/03 10:16:12 - SPINE: Poller[1] Device[395] HT[1] DS[10706] SNMP: v3: SERVERNAME.DOMAIN.XXX, dsname: hrStorageUsed, oid: 1.3.6.1.2.1.25.2.3.1.6.60, value: 7455
2019/09/03 10:16:12 - SPINE: Poller[1] Device[395] HT[1] DS[10707] SNMP: v3: SERVERNAME.DOMAIN.XXX, dsname: hrStorageSize, oid: 1.3.6.1.2.1.25.2.3.1.5.62, value: 3813466
2019/09/03 10:16:12 - SPINE: Poller[1] Device[395] HT[1] DS[10707] SNMP: v3: SERVERNAME.DOMAIN.XXX, dsname: hrStorageUsed, oid: 1.3.6.1.2.1.25.2.3.1.6.62, value: 10246
2019/09/03 10:16:12 - SPINE: Poller[1] Device[395] HT[1] DS[10708] SNMP: v3: SERVERNAME.DOMAIN.XXX, dsname: hrStorageSize, oid: 1.3.6.1.2.1.25.2.3.1.5.63, value: 1280537
2019/09/03 10:16:12 - SPINE: Poller[1] Device[395] HT[1] DS[10708] SNMP: v3: SERVERNAME.DOMAIN.XXX, dsname: hrStorageUsed, oid: 1.3.6.1.2.1.25.2.3.1.6.63, value: 38319
2019/09/03 10:16:12 - SPINE: Poller[1] Device[395] HT[1] DS[10709] SNMP: v3: SERVERNAME.DOMAIN.XXX, dsname: hrStorageSize, oid: 1.3.6.1.2.1.25.2.3.1.5.61, value: 2547525
2019/09/03 10:16:12 - SPINE: Poller[1] Device[395] HT[1] DS[10709] SNMP: v3: SERVERNAME.DOMAIN.XXX, dsname: hrStorageUsed, oid: 1.3.6.1.2.1.25.2.3.1.6.61, value: 408668
2019/09/03 10:16:12 - SPINE: Poller[1] Device[395] HT[1] DS[10710] SNMP: v3: SERVERNAME.DOMAIN.XXX, dsname: hrStorageSize, oid: 1.3.6.1.2.1.25.2.3.1.5.59, value: 249830
2019/09/03 10:16:12 - SPINE: Poller[1] Device[395] HT[1] DS[10710] SNMP: v3: SERVERNAME.DOMAIN.XXX, dsname: hrStorageUsed, oid: 1.3.6.1.2.1.25.2.3.1.6.59, value: 650
2019/09/03 10:16:12 - SPINE: Poller[1] Device[395] HT[1] DS[10713] SNMP: v3: SERVERNAME.DOMAIN.XXX, dsname: traffic_in, oid: .1.3.6.1.2.1.2.2.1.10.2, value: 1619311737
2019/09/03 10:16:12 - SPINE: Poller[1] Device[395] HT[1] DS[10713] SNMP: v3: SERVERNAME.DOMAIN.XXX, dsname: traffic_out, oid: .1.3.6.1.2.1.2.2.1.16.2, value: 1342127298
2019/09/03 10:16:12 - SPINE: Poller[1] Device[395] HT[1] DS[10714] SNMP: v3: SERVERNAME.DOMAIN.XXX, dsname: traffic_in, oid: .1.3.6.1.2.1.2.2.1.10.3, value: 1995835261
2019/09/03 10:16:12 - SPINE: Poller[1] Device[395] HT[1] DS[10714] SNMP: v3: SERVERNAME.DOMAIN.XXX, dsname: traffic_out, oid: .1.3.6.1.2.1.2.2.1.16.3, value: 12424
2019/09/03 10:16:12 - SPINE: Poller[1] Device[395] HT[1] Total Time: 5.9 Seconds
2019/09/03 10:16:12 - SPINE: Poller[1] Device[395] HT[1] DEBUG: HOST COMPLETE: About to Exit Device Polling Thread Function
2019/09/03 10:16:12 - SPINE: Poller[1] DEBUG: The Value of Active Threads is 0 for Device ID 395
2019/09/03 10:16:12 - SPINE: Poller[1] POLLER: Active Threads is 0, Pending is 0
2019/09/03 10:16:12 - SPINE: Poller[1] SPINE: The Final Value of Threads is 0
2019/09/03 10:16:12 - SPINE: Poller[1] DEBUG: Thread Cleanup Complete
2019/09/03 10:16:12 - SPINE: Poller[1] DEBUG: SS[0] Script Server Shutdown Started
2019/09/03 10:16:12 - SPINE: Poller[1] DEBUG: SS[1] Script Server Shutdown Started
2019/09/03 10:16:12 - SPINE: Poller[1] DEBUG: SS[2] Script Server Shutdown Started
2019/09/03 10:16:12 - SPINE: Poller[1] DEBUG: SS[3] Script Server Shutdown Started
2019/09/03 10:16:12 - SPINE: Poller[1] DEBUG: SS[4] Script Server Shutdown Started
2019/09/03 10:16:12 - SPINE: Poller[1] DEBUG: SS[5] Script Server Shutdown Started
2019/09/03 10:16:12 - SPINE: Poller[1] DEBUG: SS[6] Script Server Shutdown Started
2019/09/03 10:16:12 - SPINE: Poller[1] DEBUG: SS[7] Script Server Shutdown Started
2019/09/03 10:16:12 - SPINE: Poller[1] DEBUG: SS[8] Script Server Shutdown Started
2019/09/03 10:16:12 - SPINE: Poller[1] DEBUG: SS[9] Script Server Shutdown Started
2019/09/03 10:16:12 - SPINE: Poller[1] DEBUG: PHP Script Server Pipes Closed
2019/09/03 10:16:12 - SPINE: Poller[1] DEBUG: Allocated Variable Memory Freed
2019/09/03 10:16:12 - SPINE: Poller[1] DEBUG: MYSQL Free & Close Completed
2019/09/03 10:16:12 - SPINE: Poller[1] DEBUG: Net-SNMP Close Completed
2019/09/03 10:16:12 - SPINE: Poller[1] Time: 9.1468 s, Threads: 60, Devices: 1

netniV
Cacti Guru User
Posts: 2881
Joined: Sun Aug 27, 2017 12:05 am

Re: POLLING PROBLEM

#4 Post by netniV » Tue Sep 03, 2019 1:01 pm

How does it look when you set the device in debug mode from the poller? You don't have to set everything in debug mode, the standard logging levels at MEDIUM or LOW should do, as once that device is in debug mode we should see a lot more just for that device but in the context of the poller.

Lalbee99
Cacti User
Posts: 95
Joined: Mon Mar 03, 2008 2:07 pm

Re: POLLING PROBLEM

#5 Post by Lalbee99 » Wed Sep 04, 2019 11:39 am

If you are referring to choose "*Enable Device Debug" from the device page for that server. It is already enabled.

netniV
Cacti Guru User
Posts: 2881
Joined: Sun Aug 27, 2017 12:05 am

Re: POLLING PROBLEM

#6 Post by netniV » Wed Sep 04, 2019 3:39 pm

OK, I just would have expected more than simply spine logs. Do you want to email over your log file to [email protected]?

Lalbee99
Cacti User
Posts: 95
Joined: Mon Mar 03, 2008 2:07 pm

Re: POLLING PROBLEM

#7 Post by Lalbee99 » Thu Sep 05, 2019 6:42 am

I sent it to the email address you specified.

netniV
Cacti Guru User
Posts: 2881
Joined: Sun Aug 27, 2017 12:05 am

Re: POLLING PROBLEM

#8 Post by netniV » Thu Sep 05, 2019 8:12 am

So according to the log files, boost is trying to populate the rrd file with values that are five minutes apart. That you are not seeing these in the actual file would suggest a permission problem with the RRD folder I think. Both the polling user (normally defined in /etc/cron.d/cacti) and the webisite user/group should have read/write access to the RRA and log folders or it will definitely cause issues.

Lalbee99
Cacti User
Posts: 95
Joined: Mon Mar 03, 2008 2:07 pm

Re: POLLING PROBLEM

#9 Post by Lalbee99 » Thu Sep 05, 2019 8:43 am

I would agree with you but intermittently a value does get posted (in original output in this thread). If there was a perm problem it should be all or nothing.

Post Reply