totalling 22 Data Sources. Please Either Re-Index, Delete or[solved]

Post support questions that directly relate to Linux/Unix operating systems.

Moderators: Moderators, Developers

Author
Message
majed17
Cacti User
Posts: 58
Joined: Thu Jul 17, 2014 10:24 pm

Re: totalling 22 Data Sources. Please Either Re-Index, Delete or

#31 Post by majed17 » Fri Aug 16, 2019 1:05 am

default data collector process are now 2
remote agent timeout = 5 sec
default threads per process =5
poller and cron interval are 5 mins
log:

Code: Select all

2019/08/16 09:05:02 - REPORTS Cacti Reports reports found: 0
2019/08/16 09:05:01 - SYSTEM STATS: Time:299.6893 Method:spine Processes:1 Threads:1 Hosts:37 HostsPerProcess:37 DataSources:4290 RRDsProcessed:0
2019/08/16 09:05:01 - SYSTEM WARNING: Primary Admin account does not have an email address! Unable to send administrative Email.
2019/08/16 09:05:01 - POLLER: Poller[Main Poller] Maximum runtime of 298 seconds exceeded. Exiting.
2019/08/16 09:05:01 - POLLER: Poller[Main Poller] NOTE: Poller Int: '300', Cron Int: '300', Time Since Last: '299.43', Max Runtime '298', Poller Runs: '1' 
knock and it shall be opened, ask and you shall receive, believe in order for it to happen, cry and you shall be comforted!

majed17
Cacti User
Posts: 58
Joined: Thu Jul 17, 2014 10:24 pm

Re: totalling 22 Data Sources. Please Either Re-Index, Delete or

#32 Post by majed17 » Fri Aug 16, 2019 2:21 am

just updated to the most recent 1.2.x and nothing improved for me.
knock and it shall be opened, ask and you shall receive, believe in order for it to happen, cry and you shall be comforted!

User avatar
camerabob
Cacti User
Posts: 290
Joined: Fri Feb 10, 2017 2:45 pm
Location: Long Island, New York, USA
Contact:

Re: totalling 22 Data Sources. Please Either Re-Index, Delete or

#33 Post by camerabob » Fri Aug 16, 2019 7:28 am

Something is wrong somewhere. Your second line posted above still shows 1/1. In comparison mine shows:

Code: Select all

2019/08/16 08:27:34 - SYSTEM STATS: Time:32.6822 Method:cmd.php Processes:18 Threads:N/A Hosts:61 HostsPerProcess:4 DataSources:189 RRDsProcessed:187 
See the Cacti 1.x templates I use at: http://www.camerabob.com/cacti

Live: Cacti 1.2.3 @ CentOS 7-6.1810.2.el7 & PHP 5.4.16-46.el7
Flowview @ 2.1
Mactrack @ 4.2
Maint @ 1.2
Monitor @ 2.3.6
Thold @ 1.2.4

Test: Cacti 1.2.6-1.el7 @ CentOS 7-7.1908.0.el7 & PHP 5.4.16-46.el7
Maint @ 1.2
Monitor @ 2.3.6
Thold @ 1.2.4

majed17
Cacti User
Posts: 58
Joined: Thu Jul 17, 2014 10:24 pm

Re: totalling 22 Data Sources. Please Either Re-Index, Delete or

#34 Post by majed17 » Fri Aug 16, 2019 7:52 am

well you do not use spine, i switched to your method and it didn't help

Code: Select all

2019/08/16 15:50:01 - POLLER: Poller[Main Poller] NOTE: Poller Int: '300', Cron Int: '300', Time Since Last: '300.51', Max Runtime '298', Poller Runs: '1'
2019/08/16 15:50:00 - REPORTS Cacti Reports reports found: 0
2019/08/16 15:50:00 - SYSTEM STATS: Time:298.6034 Method:cmd.php Processes:1 Threads:0 Hosts:37 HostsPerProcess:37 DataSources:4290 RRDsProcessed:0
2019/08/16 15:50:00 - SYSTEM WARNING: Primary Admin account notifications disabled! Unable to send administrative Email.
2019/08/16 15:50:00 - POLLER: Poller[Main Poller] Maximum runtime of 298 seconds exceeded. Exiting. 
knock and it shall be opened, ask and you shall receive, believe in order for it to happen, cry and you shall be comforted!

netniV
Cacti Guru User
Posts: 2872
Joined: Sun Aug 27, 2017 12:05 am

Re: totalling 22 Data Sources. Please Either Re-Index, Delete or

#35 Post by netniV » Fri Aug 16, 2019 10:21 am

As I said before, make sure you have set this against your poller not the system defaults. Console -> Data Colelctiojns -> Data Collectors -> Poller

User avatar
camerabob
Cacti User
Posts: 290
Joined: Fri Feb 10, 2017 2:45 pm
Location: Long Island, New York, USA
Contact:

Re: totalling 22 Data Sources. Please Either Re-Index, Delete or

#36 Post by camerabob » Fri Aug 16, 2019 11:20 am

As I said before, make sure you have set this against your poller not the system defaults. Console -> Data Collections -> Data Collectors -> Poller
Yes. My directions are wrong for the 1.2.x train. My apologies.
See the Cacti 1.x templates I use at: http://www.camerabob.com/cacti

Live: Cacti 1.2.3 @ CentOS 7-6.1810.2.el7 & PHP 5.4.16-46.el7
Flowview @ 2.1
Mactrack @ 4.2
Maint @ 1.2
Monitor @ 2.3.6
Thold @ 1.2.4

Test: Cacti 1.2.6-1.el7 @ CentOS 7-7.1908.0.el7 & PHP 5.4.16-46.el7
Maint @ 1.2
Monitor @ 2.3.6
Thold @ 1.2.4

majed17
Cacti User
Posts: 58
Joined: Thu Jul 17, 2014 10:24 pm

Re: totalling 22 Data Sources. Please Either Re-Index, Delete or

#37 Post by majed17 » Fri Aug 16, 2019 12:53 pm

Code: Select all

2019/08/16 20:45:01 - POLLER: Poller[Main Poller] NOTE: Poller Int: '300', Cron Int: '300', Time Since Last: '300.55', Max Runtime '298', Poller Runs: '1'
2019/08/16 20:45:00 - REPORTS Cacti Reports reports found: 0
2019/08/16 20:45:00 - SYSTEM STATS: Time:298.6666 Method:spine Processes:2 Threads:5 Hosts:37 HostsPerProcess:19 DataSources:4290 RRDsProcessed:0
2019/08/16 20:45:00 - SYSTEM WARNING: Primary Admin account notifications disabled! Unable to send administrative Email.
2019/08/16 20:45:00 - POLLER: Poller[Main Poller] Maximum runtime of 298 seconds exceeded. Exiting. 
Default Data Collector Processes = 2
Default Threads per Process = 5
in data collector
processes = 2
threads = 5
still not working
knock and it shall be opened, ask and you shall receive, believe in order for it to happen, cry and you shall be comforted!

netniV
Cacti Guru User
Posts: 2872
Joined: Sun Aug 27, 2017 12:05 am

Re: totalling 22 Data Sources. Please Either Re-Index, Delete or

#38 Post by netniV » Sun Aug 18, 2019 9:35 am

I published an update to spine this weekend that reduces timeout issues when a system isn't responding. It also highlighted a problem with one or two of my device configurations, the total timeout for each SNMP call will be:

Code: Select all

timeout + (retries * timeout)
This means that if you have a timeout of say 3 seconds (3000), and set the number of tries to 3, you actually end up with 4 tries per call totally 12 seconds of timeout. The issue with spine was that if it was attempting to get a full set of system information, that would be 5 calls. So for one device, 5 * 12 would be 60. Now, say you have several devices doing this, and they only start after 5 seconds due to their order in the list of devices, then that means it would take 70 seconds from the start of spine to error out on one thread. With 1 minute polling, the main spine thread would end at 60 seconds, so that last timeout would never actually be reached. Now, after the first call, it does honour the host_ignore flag which means that it wouldn't attempt the other 4 calls as one had already failed.

Now, you're using 5 minute polling so that shouldn't have been as big an issue but it's still worth trying with the updated version of spine to see if it does resolve things for you. Also, check what you are using for timeouts and retries against your hosts. Note, you can reset multiple ones at once by selecting devices and using the change device options.

majed17
Cacti User
Posts: 58
Joined: Thu Jul 17, 2014 10:24 pm

Re: totalling 22 Data Sources. Please Either Re-Index, Delete or

#39 Post by majed17 » Sun Aug 18, 2019 1:20 pm

i installed spine-develop, and now automake and tools, still no go:

Code: Select all

2019/08/18 21:15:04 - PCOMMAND Device[cisco skorini 2921] DQ[SNMP - Interface Statistics] RECACHE: Recached successfully.
2019/08/18 21:15:04 - AUTOM8 TRACE [PID: 15498] automation_execute_data_query() Device[cisco skorini 2921] - start - data query: 1
2019/08/18 21:15:04 - DSTRACE Field Name 'ifIP' found not suitable during Re-Index for Device[cisco skorini 2921], DQ[SNMP - Interface Statistics]
2019/08/18 21:15:04 - DSTRACE Field Name 'ifHwAddr' found not suitable. Non-unique and nonunique not specified during Re-Index for Device[cisco skorini 2921], DQ[SNMP - Interface Statistics]
2019/08/18 21:15:04 - DSTRACE Field Name 'ifHighSpeed' found not suitable during Re-Index for Device[cisco skorini 2921], DQ[SNMP - Interface Statistics]
2019/08/18 21:15:04 - DSTRACE Field Name 'ifSpeed' found not suitable during Re-Index for Device[cisco skorini 2921], DQ[SNMP - Interface Statistics]
2019/08/18 21:15:04 - DSTRACE Field Name 'ifType' found not suitable during Re-Index for Device[cisco skorini 2921], DQ[SNMP - Interface Statistics]
2019/08/18 21:15:04 - DSTRACE Field Name 'ifAlias' found not suitable during Re-Index for Device[cisco skorini 2921], DQ[SNMP - Interface Statistics]
2019/08/18 21:15:04 - DSTRACE Field Name 'ifName' found not suitable. Non-unique and nonunique not specified during Re-Index for Device[cisco skorini 2921], DQ[SNMP - Interface Statistics]
2019/08/18 21:15:04 - DSTRACE Field Name 'ifDescr' found suitable during Re-Index for Device[cisco skorini 2921], DQ[SNMP - Interface Statistics]
2019/08/18 21:15:04 - DSTRACE Field Name 'ifAdminStatus' found not suitable during Re-Index for Device[cisco skorini 2921], DQ[SNMP - Interface Statistics]
2019/08/18 21:15:04 - DSTRACE Field Name 'ifOperStatus' found not suitable during Re-Index for Device[cisco skorini 2921], DQ[SNMP - Interface Statistics]
2019/08/18 21:15:04 - DSTRACE Field Name 'ifIndex' is an SNMP index and suitable Re-Index for Device[cisco skorini 2921], DQ[SNMP - Interface Statistics]
2019/08/18 21:15:04 - DSTRACE Available Sort Fields for Re-Index for Device[cisco skorini 2921], DQ[SNMP - Interface Statistics] are [ifName:ifDescr:ifHwAddr:ifIndex]
2019/08/18 21:15:02 - WEBUI Found Valid 'Graph Template' priming cache
2019/08/18 21:15:02 - WEBUI Fetch InitRows:-1, Hash:1597d7f033a83a19f5b1a2356ce08ea0
2019/08/18 21:15:02 - WEBUI Obtaining 'Graph Template' cache
2019/08/18 21:15:01 - REPORTS Cacti Reports reports found: 0
2019/08/18 21:15:01 - POLLER: Poller[Main Poller] NOTE: Poller Int: '300', Cron Int: '300', Time Since Last: '299.8', Max Runtime '298', Poller Runs: '1'
2019/08/18 21:15:01 - DSTRACE Running Re-Index for Device[cisco skorini 2921], DQ[SNMP - Interface Statistics]
2019/08/18 21:15:01 - PCOMMAND Device[cisco skorini 2921] DQ[SNMP - Interface Statistics] RECACHE: Recache for Device started.
2019/08/18 21:15:01 - PCOMMAND Device[cisco skorini 2921] WARNING: Recache Event Detected for Device
2019/08/18 21:15:01 - WEBUI Fetch InitRows:-2, Hash:80273e08c9e5f93672b7986f82875b97
2019/08/18 21:15:01 - SYSTEM STATS: Time:298.6769 Method:spine Processes:2 Threads:5 Hosts:37 HostsPerProcess:19 DataSources:4290 RRDsProcessed:0
2019/08/18 21:15:01 - SYSTEM WARNING: Primary Admin account notifications disabled! Unable to send administrative Email.
2019/08/18 21:15:01 - POLLER: Poller[Main Poller] Maximum runtime of 298 seconds exceeded. Exiting. 
there are other hosts also in the output...
knock and it shall be opened, ask and you shall receive, believe in order for it to happen, cry and you shall be comforted!


majed17
Cacti User
Posts: 58
Joined: Thu Jul 17, 2014 10:24 pm

Re: totalling 22 Data Sources. Please Either Re-Index, Delete or

#41 Post by majed17 » Tue Aug 20, 2019 12:39 am

any results?
knock and it shall be opened, ask and you shall receive, believe in order for it to happen, cry and you shall be comforted!

majed17
Cacti User
Posts: 58
Joined: Thu Jul 17, 2014 10:24 pm

Re: totalling 22 Data Sources. Please Either Re-Index, Delete or

#42 Post by majed17 » Tue Aug 20, 2019 2:12 am

copied the yesterday commit but still,

Code: Select all

2019/08/20 10:05:50 - SNMP WARNING: SNMP Error:'Timeout (500 ms)', Device:'100.50.13.1', OID:'.1.3.6.1.2.1.31.1.1.1.6.22'
2019/08/20 10:05:30 - SNMP WARNING: SNMP Error:'Timeout (500 ms)', Device:'100.50.13.1', OID:'.1.3.6.1.2.1.31.1.1.1.10.22'
2019/08/20 10:05:09 - SNMP WARNING: SNMP Error:'Timeout (500 ms)', Device:'100.50.13.1', OID:'.1.3.6.1.2.1.31.1.1.1.10.22'
2019/08/20 10:05:07 - SNMP WARNING: SNMP Error:'Timeout (500 ms)', Device:'100.50.13.1', OID:'.1.3.6.1.2.1.31.1.1.1.6.22'
2019/08/20 10:05:02 - POLLER: Poller[Main Poller] NOTE: Poller Int: '300', Cron Int: '300', Time Since Last: '300.01', Max Runtime '298', Poller Runs: '1'
2019/08/20 10:05:01 - REPORTS Cacti Reports reports found: 0
2019/08/20 10:05:01 - SYSTEM STATS: Time:298.6180 Method:spine Processes:2 Threads:5 Hosts:37 HostsPerProcess:19 DataSources:4290 RRDsProcessed:0
2019/08/20 10:05:01 - SYSTEM WARNING: Primary Admin account notifications disabled! Unable to send administrative Email.
2019/08/20 10:05:01 - POLLER: Poller[Main Poller] Maximum runtime of 298 seconds exceeded. Exiting. 
knock and it shall be opened, ask and you shall receive, believe in order for it to happen, cry and you shall be comforted!

netniV
Cacti Guru User
Posts: 2872
Joined: Sun Aug 27, 2017 12:05 am

Re: totalling 22 Data Sources. Please Either Re-Index, Delete or

#43 Post by netniV » Tue Aug 20, 2019 7:22 am

Unfortunately, the output you sent was truncated. To get a proper understanding, it would need to be the full log.

majed17
Cacti User
Posts: 58
Joined: Thu Jul 17, 2014 10:24 pm

Re: totalling 22 Data Sources. Please Either Re-Index, Delete or

#44 Post by majed17 » Tue Aug 20, 2019 7:39 am

well i tried the output with > /tmp/cactispine but the file kept empty
knock and it shall be opened, ask and you shall receive, believe in order for it to happen, cry and you shall be comforted!

majed17
Cacti User
Posts: 58
Joined: Thu Jul 17, 2014 10:24 pm

Re: totalling 22 Data Sources. Please Either Re-Index, Delete or

#45 Post by majed17 » Tue Aug 20, 2019 9:35 am

wow a week of down time, this has never happened before
knock and it shall be opened, ask and you shall receive, believe in order for it to happen, cry and you shall be comforted!

Post Reply