totalling 22 Data Sources. Please Either Re-Index, Delete or[solved]

Post support questions that directly relate to Linux/Unix operating systems.

Moderators: Moderators, Developers

Author
Message
User avatar
camerabob
Cacti User
Posts: 290
Joined: Fri Feb 10, 2017 2:45 pm
Location: Long Island, New York, USA
Contact:

Re: totalling 22 Data Sources. Please Either Re-Index, Delete or

#16 Post by camerabob » Wed Aug 14, 2019 7:45 am

From your prior post, the poller is timing out before any data is received. Check your SNMP setting to see if reachability is an issue.

2019/08/14 15:40:00 - SYSTEM STATS: Time:298.6046 Method:spine Processes:1 Threads:1 Hosts:37 HostsPerProcess:37 DataSources:4456 RRDsProcessed:0
2019/08/14 15:40:00 - SYSTEM WARNING: Primary Admin account notifications disabled! Unable to send administrative Email.
2019/08/14 15:40:00 - POLLER: Poller[Main Poller] Maximum runtime of 298 seconds exceeded. Exiting.
See the Cacti 1.x templates I use at: http://www.camerabob.com/cacti

Live: Cacti 1.2.3 @ CentOS 7-6.1810.2.el7 & PHP 5.4.16-46.el7
Flowview @ 2.1
Mactrack @ 4.2
Maint @ 1.2
Monitor @ 2.3.6
Thold @ 1.2.4

Test: Cacti 1.2.6-1.el7 @ CentOS 7-7.1908.0.el7 & PHP 5.4.16-46.el7
Maint @ 1.2
Monitor @ 2.3.6
Thold @ 1.2.4


User avatar
camerabob
Cacti User
Posts: 290
Joined: Fri Feb 10, 2017 2:45 pm
Location: Long Island, New York, USA
Contact:

Re: totalling 22 Data Sources. Please Either Re-Index, Delete or

#18 Post by camerabob » Wed Aug 14, 2019 8:15 am

In 'Configuration' -> 'Settings' -> 'Poller', increase the number of threads per Spine process to perhaps get past the device which is stalling the processing. Also, try enabling 'Invalid Data Logging' to see more data.
See the Cacti 1.x templates I use at: http://www.camerabob.com/cacti

Live: Cacti 1.2.3 @ CentOS 7-6.1810.2.el7 & PHP 5.4.16-46.el7
Flowview @ 2.1
Mactrack @ 4.2
Maint @ 1.2
Monitor @ 2.3.6
Thold @ 1.2.4

Test: Cacti 1.2.6-1.el7 @ CentOS 7-7.1908.0.el7 & PHP 5.4.16-46.el7
Maint @ 1.2
Monitor @ 2.3.6
Thold @ 1.2.4

majed17
Cacti User
Posts: 58
Joined: Thu Jul 17, 2014 10:24 pm

Re: totalling 22 Data Sources. Please Either Re-Index, Delete or

#19 Post by majed17 » Wed Aug 14, 2019 9:04 am

i feel like a ginny pig, no more pills :)?
knock and it shall be opened, ask and you shall receive, believe in order for it to happen, cry and you shall be comforted!

majed17
Cacti User
Posts: 58
Joined: Thu Jul 17, 2014 10:24 pm

Re: totalling 22 Data Sources. Please Either Re-Index, Delete or

#20 Post by majed17 » Wed Aug 14, 2019 9:20 am

there is no snmp reachability problem...i increased Default Data Collector Processes to 2, spine log is set to detailed...
knock and it shall be opened, ask and you shall receive, believe in order for it to happen, cry and you shall be comforted!

majed17
Cacti User
Posts: 58
Joined: Thu Jul 17, 2014 10:24 pm

Re: totalling 22 Data Sources. Please Either Re-Index, Delete or

#21 Post by majed17 » Wed Aug 14, 2019 9:32 am

Code: Select all

2019/08/14 17:30:01 - POLLER: Poller[Main Poller] NOTE: Poller Int: '300', Cron Int: '300', Time Since Last: '300.69', Max Runtime '298', Poller Runs: '1'
2019/08/14 17:30:00 - REPORTS Cacti Reports reports found: 0
2019/08/14 17:30:00 - SYSTEM STATS: Time:298.5825 Method:spine Processes:1 Threads:1 Hosts:37 HostsPerProcess:37 DataSources:4456 RRDsProcessed:0
2019/08/14 17:30:00 - SYSTEM WARNING: Primary Admin account notifications disabled! Unable to send administrative Email.
2019/08/14 17:30:00 - POLLER: Poller[Main Poller] Maximum runtime of 298 seconds exceeded. Exiting. 
Default Threads per Process = 2
Default Data Collector Processes = 1
knock and it shall be opened, ask and you shall receive, believe in order for it to happen, cry and you shall be comforted!

majed17
Cacti User
Posts: 58
Joined: Thu Jul 17, 2014 10:24 pm

Re: totalling 22 Data Sources. Please Either Re-Index, Delete or

#22 Post by majed17 » Wed Aug 14, 2019 9:59 am

i removed memory routers from all related routers which was timing out but still no go..
knock and it shall be opened, ask and you shall receive, believe in order for it to happen, cry and you shall be comforted!

netniV
Cacti Guru User
Posts: 2872
Joined: Sun Aug 27, 2017 12:05 am

Re: totalling 22 Data Sources. Please Either Re-Index, Delete or

#23 Post by netniV » Wed Aug 14, 2019 12:06 pm

Try running spine manually from the command line in debug mode.
/usr/local/spine/bin/spine -R -S -V 3

majed17
Cacti User
Posts: 58
Joined: Thu Jul 17, 2014 10:24 pm

Re: totalling 22 Data Sources. Please Either Re-Index, Delete or

#24 Post by majed17 » Wed Aug 14, 2019 12:43 pm

ok, there was a nexus memory for a router.. and some invalid oid's... i got rid of them all, the db was never this perfect if not counting the start but still, no graphing
result of the debug

Code: Select all

POLLER: Active Threads is 0, Pending is 0
SPINE: The Final Value of Threads is 0
Time: 16.8693 s, Threads: 1, Devices: 37
log:

Code: Select all

2019/08/14 20:40:02 - WEBUI Found Valid 'Graph Template' priming cache
2019/08/14 20:40:02 - WEBUI Fetch InitRows:-1, Hash:1597d7f033a83a19f5b1a2356ce08ea0
2019/08/14 20:40:02 - WEBUI Obtaining 'Graph Template' cache
2019/08/14 20:40:02 - WEBUI Fetch InitRows:-2, Hash:80273e08c9e5f93672b7986f82875b97
2019/08/14 20:40:01 - POLLER: Poller[Main Poller] NOTE: Poller Int: '300', Cron Int: '300', Time Since Last: '300.61', Max Runtime '298', Poller Runs: '1'
2019/08/14 20:40:00 - REPORTS Cacti Reports reports found: 0
2019/08/14 20:40:00 - SYSTEM STATS: Time:298.6920 Method:spine Processes:1 Threads:1 Hosts:37 HostsPerProcess:37 DataSources:4338 RRDsProcessed:0
2019/08/14 20:40:00 - SYSTEM WARNING: Primary Admin account does not have an email address! Unable to send administrative Email.
2019/08/14 20:40:00 - POLLER: Poller[Main Poller] Maximum runtime of 298 seconds exceeded. Exiting. 
knock and it shall be opened, ask and you shall receive, believe in order for it to happen, cry and you shall be comforted!

majed17
Cacti User
Posts: 58
Joined: Thu Jul 17, 2014 10:24 pm

Re: totalling 22 Data Sources. Please Either Re-Index, Delete or

#25 Post by majed17 » Wed Aug 14, 2019 12:54 pm

i then restarted mysql, vixie cron and apache2 and got this:

Code: Select all

2019/08/14 20:52:37 - WEBUI Found Valid 'Graph Template' priming cache
2019/08/14 20:52:37 - WEBUI Fetch InitRows:-1, Hash:1597d7f033a83a19f5b1a2356ce08ea0
2019/08/14 20:52:37 - WEBUI Obtaining 'Graph Template' cache
2019/08/14 20:52:37 - WEBUI Fetch InitRows:-2, Hash:80273e08c9e5f93672b7986f82875b97
2019/08/14 20:50:01 - POLLER: Poller[Main Poller] NOTE: Poller Int: '300', Cron Int: '300', Time Since Last: '300.79', Max Runtime '298', Poller Runs: '1'
2019/08/14 20:50:00 - SNMPAGENT WARNING: SNMPAgent: MIB table does not exist
2019/08/14 20:50:00 - SNMPAGENT WARNING: SNMPAgent: MIB table does not exist
2019/08/14 20:50:00 - SNMPAGENT WARNING: SNMPAgent: MIB table does not exist
2019/08/14 20:50:00 - SNMPAGENT WARNING: SNMPAgent: MIB table does not exist
2019/08/14 20:50:00 - SNMPAGENT WARNING: SNMPAgent: MIB table does not exist
2019/08/14 20:50:00 - SNMPAGENT WARNING: SNMPAgent: MIB table does not exist
2019/08/14 20:50:00 - SNMPAGENT WARNING: SNMPAgent: MIB table does not exist
2019/08/14 20:50:00 - SNMPAGENT WARNING: SNMPAgent: MIB table does not exist
2019/08/14 20:50:00 - SNMPAGENT WARNING: SNMPAgent: MIB table does not exist
2019/08/14 20:50:00 - SNMPAGENT WARNING: SNMPAgent: MIB table does not exist
2019/08/14 20:50:00 - SNMPAGENT WARNING: SNMPAgent: MIB table does not exist
2019/08/14 20:50:00 - SYSTEM STATS: Time:298.4022 Method:spine Processes:1 Threads:1 Hosts:37 HostsPerProcess:37 DataSources:4336 RRDsProcessed:0
2019/08/14 20:50:00 - SNMPAGENT WARNING: SNMPAgent: MIB table does not exist
2019/08/14 20:50:00 - SYSTEM WARNING: Primary Admin account notifications disabled! Unable to send administrative Email. 
knock and it shall be opened, ask and you shall receive, believe in order for it to happen, cry and you shall be comforted!

majed17
Cacti User
Posts: 58
Joined: Thu Jul 17, 2014 10:24 pm

Re: totalling 22 Data Sources. Please Either Re-Index, Delete or

#26 Post by majed17 » Thu Aug 15, 2019 2:00 am

today i got this after changing log from debug to high:

Code: Select all

2019/08/15 09:55:01 - POLLER: Poller[Main Poller] NOTE: Poller Int: '300', Cron Int: '300', Time Since Last: '300.37', Max Runtime '298', Poller Runs: '1'
2019/08/15 09:55:00 - REPORTS Cacti Reports reports found: 0
2019/08/15 09:55:00 - POLLER: Poller[Main Poller] DEBUG: About to Spawn a Remote Process [CMD: /usr/bin/php, ARGS: -q '/var/www/localhost/htdocs/cacti/poller_maintenance.php']
2019/08/15 09:55:00 - POLLER: Poller[Main Poller] DEBUG: About to Spawn a Remote Process [CMD: /usr/bin/php, ARGS: -q '/var/www/localhost/htdocs/cacti/poller_automation.php' -M]
2019/08/15 09:55:00 - POLLER: Poller[Main Poller] DEBUG: About to Spawn a Remote Process [CMD: /usr/bin/php, ARGS: -q '/var/www/localhost/htdocs/cacti/poller_spikekill.php']
2019/08/15 09:55:00 - POLLER: Poller[Main Poller] DEBUG: About to Spawn a Remote Process [CMD: /usr/bin/php, ARGS: -q /var/www/localhost/htdocs/cacti/poller_reports.php]
2019/08/15 09:55:00 - SYSTEM STATS: Time:298.5956 Method:spine Processes:1 Threads:1 Hosts:37 HostsPerProcess:37 DataSources:4290 RRDsProcessed:0
2019/08/15 09:55:00 - SYSTEM WARNING: Primary Admin account does not have an email address! Unable to send administrative Email.
2019/08/15 09:55:00 - POLLER: Poller[Main Poller] Maximum runtime of 298 seconds exceeded. Exiting
what else could be the problem?
knock and it shall be opened, ask and you shall receive, believe in order for it to happen, cry and you shall be comforted!

majed17
Cacti User
Posts: 58
Joined: Thu Jul 17, 2014 10:24 pm

Re: totalling 22 Data Sources. Please Either Re-Index, Delete or

#27 Post by majed17 » Thu Aug 15, 2019 5:53 am

i got desperate, tried 777 on rra folder, also tried chown -R apache:cacti, but nothing is helping, must be something with spine...
knock and it shall be opened, ask and you shall receive, believe in order for it to happen, cry and you shall be comforted!

majed17
Cacti User
Posts: 58
Joined: Thu Jul 17, 2014 10:24 pm

Re: totalling 22 Data Sources. Please Either Re-Index, Delete or

#28 Post by majed17 » Thu Aug 15, 2019 9:33 am

i remember, once on nagios i was adding a host, and it just would not add! i just restarted the machine and voila it magically appeared! so i restarted the cacti machine but alas this is not nagios :D
knock and it shall be opened, ask and you shall receive, believe in order for it to happen, cry and you shall be comforted!

User avatar
camerabob
Cacti User
Posts: 290
Joined: Fri Feb 10, 2017 2:45 pm
Location: Long Island, New York, USA
Contact:

Re: totalling 22 Data Sources. Please Either Re-Index, Delete or

#29 Post by camerabob » Thu Aug 15, 2019 12:40 pm

Still only seeing one process and one thread....

Code: Select all

2019/08/15 09:55:00 - SYSTEM STATS: Time:298.5956 Method:spine Processes:1 Threads:1 Hosts:37 HostsPerProcess:37 DataSources:4290 RRDsProcessed:0
Go for 2 processes and 5 threads as a start.
See the Cacti 1.x templates I use at: http://www.camerabob.com/cacti

Live: Cacti 1.2.3 @ CentOS 7-6.1810.2.el7 & PHP 5.4.16-46.el7
Flowview @ 2.1
Mactrack @ 4.2
Maint @ 1.2
Monitor @ 2.3.6
Thold @ 1.2.4

Test: Cacti 1.2.6-1.el7 @ CentOS 7-7.1908.0.el7 & PHP 5.4.16-46.el7
Maint @ 1.2
Monitor @ 2.3.6
Thold @ 1.2.4

netniV
Cacti Guru User
Posts: 2872
Joined: Sun Aug 27, 2017 12:05 am

Re: totalling 22 Data Sources. Please Either Re-Index, Delete or

#30 Post by netniV » Thu Aug 15, 2019 3:05 pm

Make sure you are setting the values against the pollers not against the system defaults.

Post Reply