Spine poller issues

[otisbrown]

I have been running Spine for quite some time and it has worked perfectly until recently. Over the past few weeks, I started seeing gaps in my graphs. Looking in the logs I found the following:

10/23/2008 10:45:01 AM - POLLER: Poller[0] WARNING: Poller Output Table not Empty. Potential Data Source Issues for Data Sources:

The list of data sources is very long, but does not seem to include every datasource. The list is not the same each time as well.

The above would eventually be followed by:

10/23/2008 10:58:00 AM - SYSTEM STATS: Time:1078.3264 Method:spine Processes:1 Threads:1 Hosts:74 HostsPerProcess:74 DataSources:12915 RRDsProcessed:2161

Doing the math on the time, it would be related to the poll run at 10:40, which explains the error above. However, I have the following entry in the log as well:
10/23/2008 10:40:47 AM - SPINE: Poller[0] Time: 45.4606 s, Threads: 1, Hosts: 74

The above to entries seem to contradict each other and I am not sure where to go from here to determine what is causing the problem. It almost seems the poller output is hanging.

Any help in isolating the issue would be greatly appreciated.

[gandalf]

poller.php does not only run spine, but does all of the rrdtool update. So no contradiction. You may want to increase the number of spine threads to 10-12.
Reinhard

[otisbrown]

Thanks Reinhold,

It was set to 10 when this issue first started. I lowered it thinking it was a hardware resource issue. I will up it to 11 and see if it stops.

[gandalf]

It would help to grep for those stats and post at least some 20-30 lines each
Reinhard

[otisbrown]

Reinhard,

I dont have any logs from when the issue first started. I only keep my logs for 7 days, and this has been going on a bit longer than that. I am certainly wishing, I kept them for longer than that.

Here is a snipet from the other day.

10/20/2008 03:05:47 PM - SYSTEM STATS: Time:46.3475 Method:spine Processes:1 Threads:1 Hosts:73 HostsPerProcess:73 DataSources:13026 RRDs
Processed:4372
10/20/2008 03:10:50 PM - SYSTEM STATS: Time:48.3887 Method:spine Processes:1 Threads:1 Hosts:73 HostsPerProcess:73 DataSources:13026 RRDs
Processed:4372
10/20/2008 03:15:50 PM - SYSTEM STATS: Time:48.4046 Method:spine Processes:1 Threads:1 Hosts:73 HostsPerProcess:73 DataSources:13026 RRDs
Processed:4372
10/20/2008 03:16:00 PM - SYSTEM STATS: Time:958.2887 Method:spine Processes:1 Threads:1 Hosts:73 HostsPerProcess:73 DataSources:13026 RRD
sProcessed:2194
10/20/2008 03:20:52 PM - SYSTEM STATS: Time:50.4259 Method:spine Processes:1 Threads:1 Hosts:73 HostsPerProcess:73 DataSources:13026 RRDs
Processed:4372
10/20/2008 03:43:05 PM - SYSTEM STATS: Time:1083.7639 Method:spine Processes:1 Threads:1 Hosts:73 HostsPerProcess:73 DataSources:13026 RR
DsProcessed:2194
10/20/2008 03:45:53 PM - SPINE: Poller[0] Time: 51.8964 s, Threads: 1, Hosts: 73
10/20/2008 03:45:53 PM - SYSTEM STATS: Time:52.0511 Method:spine Processes:1 Threads:1 Hosts:73 HostsPerProcess:73 DataSources:13026 RRDs
Processed:4372
10/20/2008 03:45:53 PM - CMDPHP: Poller[0] DEBUG: SQL Exec: "replace into settings (name,value) values ('stats_poller','Time:52.0511 Meth
od:spine Processes:1 Threads:1 Hosts:73 HostsPerProcess:73 DataSources:13026 RRDsProcessed:4372')"
10/20/2008 03:46:01 PM - SYSTEM STATS: Time:959.1739 Method:spine Processes:1 Threads:1 Hosts:73 HostsPerProcess:73 DataSources:13026 RRD
sProcessed:2180
10/20/2008 03:50:48 PM - SPINE: Poller[0] Time: 47.2342 s, Threads: 1, Hosts: 73
10/20/2008 03:50:49 PM - SYSTEM STATS: Time:47.8956 Method:spine Processes:1 Threads:1 Hosts:73 HostsPerProcess:73 DataSources:13026 RRDs
Processed:4372
10/20/2008 03:51:03 PM - SYSTEM STATS: Time:961.1964 Method:spine Processes:1 Threads:1 Hosts:73 HostsPerProcess:73 DataSources:13026 RRD
sProcessed:2112
10/20/2008 03:55:51 PM - SPINE: Poller[0] Time: 49.8378 s, Threads: 1, Hosts: 73
10/20/2008 03:55:52 PM - SYSTEM STATS: Time:50.2872 Method:spine Processes:1 Threads:1 Hosts:73 HostsPerProcess:73 DataSources:13026 RRDs
Processed:4372
10/20/2008 03:58:00 PM - SYSTEM STATS: Time:1078.2918 Method:spine Processes:1 Threads:1 Hosts:73 HostsPerProcess:73 DataSources:13026 RR
DsProcessed:2194
10/20/2008 04:00:52 PM - SPINE: Poller[0] Time: 50.5204 s, Threads: 1, Hosts: 73
10/20/2008 04:00:53 PM - SYSTEM STATS: Time:51.4739 Method:spine Processes:1 Threads:1 Hosts:73 HostsPerProcess:73 DataSources:13026 RRDs
Processed:4372
10/20/2008 04:05:50 PM - SPINE: Poller[0] Time: 48.1154 s, Threads: 1, Hosts: 73
10/20/2008 04:05:50 PM - SYSTEM STATS: Time:48.5993 Method:spine Processes:1 Threads:1 Hosts:73 HostsPerProcess:73 DataSources:13026 RRDs
Processed:4372



Here is a snipet from this morning. I think this morning it was caused from parsing the logs.

10/24/2008 09:00:14 AM - SYSTEM STATS: Time:12.4201 Method:spine Processes:1 Threads:11 Hosts:73 HostsPerProcess:73 DataSources:12897 RRDsProcessed:4384
10/24/2008 09:05:13 AM - SYSTEM STATS: Time:11.1829 Method:spine Processes:1 Threads:11 Hosts:73 HostsPerProcess:73 DataSources:12897 RRDsProcessed:4384
10/24/2008 09:10:12 AM - SYSTEM STATS: Time:10.6209 Method:spine Processes:1 Threads:11 Hosts:73 HostsPerProcess:73 DataSources:12897 RRDsProcessed:4384
10/24/2008 09:15:13 AM - SYSTEM STATS: Time:11.1228 Method:spine Processes:1 Threads:11 Hosts:73 HostsPerProcess:73 DataSources:12897 RRDsProcessed:4384
10/24/2008 09:20:13 AM - SYSTEM STATS: Time:11.3862 Method:spine Processes:1 Threads:11 Hosts:73 HostsPerProcess:73 DataSources:12897 RRDsProcessed:4384
10/24/2008 09:25:10 AM - POLLER: Poller[0] WARNING: Cron is out of sync with the Poller Interval! The Poller Interval is '300' seconds, with a maximum of a '300' second Cron, but 309 seconds have passed since the last poll!
10/24/2008 09:26:36 AM - SYSTEM STATS: Time:86.1219 Method:spine Processes:1 Threads:11 Hosts:73 HostsPerProcess:73 DataSources:12897 RRDsProcessed:4384
10/24/2008 09:32:11 AM - SYSTEM STATS: Time:128.6530 Method:spine Processes:1 Threads:11 Hosts:73 HostsPerProcess:73 DataSources:12897 RRDsProcessed:4384
10/24/2008 09:36:25 AM - SYSTEM STATS: Time:82.6991 Method:spine Processes:1 Threads:11 Hosts:73 HostsPerProcess:73 DataSources:12897 RRDsProcessed:4384
10/24/2008 09:40:12 AM - SYSTEM STATS: Time:11.0474 Method:spine Processes:1 Threads:11 Hosts:73 HostsPerProcess:73 DataSources:12897 RRDsProcessed:4384

[brian.nz]

had the same issues a while ago. Just check if you dont have any MySql issues.

check your mysql log files.

just my 2c worth

[otisbrown]

Thanks for the suggestion. I have looked at the log but it is clean. Things seemed to have settled down a bit. I do get snmp time outs from time to time, which don't make sense as the network itself is clean. I am leaning towards hardware at this point. We have ordered additional memory for this server, and it should arrive in a few days. I will post back with the results.

[TheWitness]

Please use the beta2 of spine-0.8.7c and see if things get better.

TheWitness

[otisbrown]

I have upgrade to spine -0.8.7c, but I am having the same result. I did attempt to compile spine with snmp support, but no matter how I specify the path it fails.

Any other ideas?

Here is a snippet of my log from today.

11/13/2008 08:50:11 AM - SYSTEM STATS: Time:9.2370 Method:spine Processes:1 Threads:12 Hosts:75 HostsPerProcess:75 DataSources:13084 RRDsProcessed:4451
11/13/2008 08:45:11 AM - SYSTEM STATS: Time:9.8297 Method:spine Processes:1 Threads:12 Hosts:75 HostsPerProcess:75 DataSources:13084 RRDsProcessed:4451
11/13/2008 08:40:17 AM - SYSTEM STATS: Time:15.8690 Method:spine Processes:1 Threads:12 Hosts:75 HostsPerProcess:75 DataSources:13084 RRDsProcessed:4451
11/13/2008 08:35:14 AM - SYSTEM STATS: Time:13.5403 Method:spine Processes:1 Threads:12 Hosts:75 HostsPerProcess:75 DataSources:13084 RRDsProcessed:4451
11/13/2008 08:30:12 AM - SYSTEM STATS: Time:10.0169 Method:spine Processes:1 Threads:12 Hosts:75 HostsPerProcess:75 DataSources:13084 RRDsProcessed:4451
11/13/2008 08:25:11 AM - SYSTEM STATS: Time:9.8273 Method:spine Processes:1 Threads:12 Hosts:75 HostsPerProcess:75 DataSources:13084 RRDsProcessed:4451
11/13/2008 08:20:12 AM - SYSTEM STATS: Time:10.8608 Method:spine Processes:1 Threads:12 Hosts:75 HostsPerProcess:75 DataSources:13084 RRDsProcessed:4451
11/13/2008 08:15:12 AM - SYSTEM STATS: Time:10.9079 Method:spine Processes:1 Threads:12 Hosts:75 HostsPerProcess:75 DataSources:13084 RRDsProcessed:4451
11/13/2008 08:10:13 AM - SYSTEM STATS: Time:11.7872 Method:spine Processes:1 Threads:12 Hosts:75 HostsPerProcess:75 DataSources:13084 RRDsProcessed:4451
11/13/2008 08:05:13 AM - SYSTEM STATS: Time:11.5705 Method:spine Processes:1 Threads:12 Hosts:75 HostsPerProcess:75 DataSources:13084 RRDsProcessed:4451
11/13/2008 08:00:15 AM - SYSTEM STATS: Time:13.4796 Method:spine Processes:1 Threads:12 Hosts:75 HostsPerProcess:75 DataSources:13084 RRDsProcessed:4451
11/13/2008 07:55:12 AM - SYSTEM STATS: Time:10.1428 Method:spine Processes:1 Threads:12 Hosts:75 HostsPerProcess:75 DataSources:13084 RRDsProcessed:4451
11/13/2008 07:50:09 AM - SYSTEM STATS: Time:8.2072 Method:spine Processes:1 Threads:12 Hosts:75 HostsPerProcess:75 DataSources:13084 RRDsProcessed:4451
11/13/2008 07:50:01 AM - POLLER: Poller[0] WARNING: Poller Output Table not Empty. Potential Data Source Issues for Data Sources: discards_out(DS[417]), errors_out(DS[417]), discards_in(DS[842]), discards_out(DS[842])
11/13/2008 07:45:16 AM - SYSTEM STATS: Time:13.9960 Method:spine Processes:1 Threads:12 Hosts:75 HostsPerProcess:75 DataSources:13084 RRDsProcessed:3817
11/13/2008 07:45:04 AM - SPINE: Poller[0] Host[17] DS[318] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.10'
11/13/2008 07:45:04 AM - SPINE: Poller[0] Host[18] DS[417] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.2'
11/13/2008 07:45:04 AM - SPINE: Poller[0] Host[26] DS[1233] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.14'
11/13/2008 07:45:04 AM - SPINE: Poller[0] Host[20] DS[612] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.4'
11/13/2008 07:45:04 AM - SPINE: Poller[0] Host[25] DS[1117] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.9'
11/13/2008 07:45:04 AM - SPINE: Poller[0] Host[27] DS[1302] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.18'
11/13/2008 07:45:04 AM - SPINE: Poller[0] Host[22] DS[842] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.6'
11/13/2008 07:45:04 AM - SPINE: Poller[0] Host[21] DS[708] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.5'
11/13/2008 07:45:04 AM - SPINE: Poller[0] Host[20] DS[612] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.4'
11/13/2008 07:45:04 AM - SPINE: Poller[0] Host[25] DS[1117] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.9'
11/13/2008 07:45:04 AM - SPINE: Poller[0] Host[26] DS[1233] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.14'
11/13/2008 07:45:04 AM - SPINE: Poller[0] Host[17] DS[318] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.10'
11/13/2008 07:45:04 AM - SPINE: Poller[0] Host[27] DS[1302] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.18'
11/13/2008 07:45:04 AM - SPINE: Poller[0] Host[17] DS[318] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.10'
11/13/2008 07:45:04 AM - SPINE: Poller[0] Host[20] DS[611] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.4'
11/13/2008 07:45:04 AM - SPINE: Poller[0] Host[25] DS[1117] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.9'
11/13/2008 07:45:04 AM - SPINE: Poller[0] Host[26] DS[1232] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.14'
11/13/2008 07:45:04 AM - SPINE: Poller[0] Host[18] DS[417] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.2'
11/13/2008 07:45:04 AM - SPINE: Poller[0] Host[27] DS[1302] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.18'
11/13/2008 07:45:04 AM - SPINE: Poller[0] Host[17] DS[318] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.10'
11/13/2008 07:45:04 AM - SPINE: Poller[0] Host[20] DS[611] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.4'
11/13/2008 07:45:04 AM - SPINE: Poller[0] Host[25] DS[1117] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.9'
11/13/2008 07:45:04 AM - SPINE: Poller[0] Host[26] DS[1232] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.14'
11/13/2008 07:45:04 AM - SPINE: Poller[0] Host[18] DS[416] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.2'
11/13/2008 07:45:04 AM - SPINE: Poller[0] Host[27] DS[1302] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.18'
11/13/2008 07:45:04 AM - SPINE: Poller[0] Host[17] DS[317] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.10'
11/13/2008 07:45:04 AM - SPINE: Poller[0] Host[20] DS[610] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.4'
11/13/2008 07:45:04 AM - SPINE: Poller[0] Host[25] DS[1116] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.9'
11/13/2008 07:45:04 AM - SPINE: Poller[0] Host[26] DS[1231] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.14'
11/13/2008 07:45:04 AM - SPINE: Poller[0] Host[18] DS[416] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.2'
11/13/2008 07:45:04 AM - SPINE: Poller[0] Host[27] DS[1301] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.18'
11/13/2008 07:45:04 AM - SPINE: Poller[0] Host[17] DS[317] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.10'
11/13/2008 07:45:04 AM - SPINE: Poller[0] Host[20] DS[610] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.4'
11/13/2008 07:45:04 AM - SPINE: Poller[0] Host[25] DS[1116] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.9'
11/13/2008 07:45:04 AM - SPINE: Poller[0] Host[26] DS[1231] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.14'
11/13/2008 07:45:04 AM - SPINE: Poller[0] Host[18] DS[416] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.2'
11/13/2008 07:45:04 AM - SPINE: Poller[0] Host[27] DS[1301] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.18'
11/13/2008 07:45:04 AM - SPINE: Poller[0] Host[17] DS[317] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.10'
11/13/2008 07:45:04 AM - SPINE: Poller[0] Host[20] DS[609] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.4'
11/13/2008 07:45:04 AM - SPINE: Poller[0] Host[25] DS[1116] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.9'
11/13/2008 07:45:04 AM - SPINE: Poller[0] Host[26] DS[1230] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.14'
11/13/2008 07:45:04 AM - SPINE: Poller[0] Host[18] DS[416] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.2'
11/13/2008 07:45:04 AM - SPINE: Poller[0] Host[27] DS[1301] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.18'
11/13/2008 07:45:04 AM - SPINE: Poller[0] Host[17] DS[317] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.10'
11/13/2008 07:45:04 AM - SPINE: Poller[0] Host[20] DS[609] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.4'
11/13/2008 07:45:04 AM - SPINE: Poller[0] Host[25] DS[1116] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.9'
11/13/2008 07:45:04 AM - SPINE: Poller[0] Host[26] DS[1230] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.14'
11/13/2008 07:45:04 AM - SPINE: Poller[0] Host[24] DS[1031] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.8'
11/13/2008 07:45:04 AM - SPINE: Poller[0] Host[18] DS[415] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.2'
11/13/2008 07:45:04 AM - SPINE: Poller[0] Host[27] DS[1301] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.18'
11/13/2008 07:45:04 AM - SPINE: Poller[0] Host[17] DS[316] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.10'
11/13/2008 07:45:04 AM - SPINE: Poller[0] Host[20] DS[608] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.4'
11/13/2008 07:45:04 AM - SPINE: Poller[0] Host[25] DS[1115] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.9'
11/13/2008 07:45:04 AM - SPINE: Poller[0] Host[26] DS[1229] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.14'
11/13/2008 07:45:04 AM - SPINE: Poller[0] Host[18] DS[415] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.2'
11/13/2008 07:45:04 AM - SPINE: Poller[0] Host[27] DS[1300] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.18'
11/13/2008 07:45:04 AM - SPINE: Poller[0] Host[17] DS[316] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.10'
11/13/2008 07:45:04 AM - SPINE: Poller[0] Host[20] DS[608] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.4'
11/13/2008 07:45:04 AM - SPINE: Poller[0] Host[25] DS[1115] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.9'
11/13/2008 07:45:04 AM - SPINE: Poller[0] Host[26] DS[1229] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.14'
11/13/2008 07:45:04 AM - SPINE: Poller[0] Host[18] DS[415] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.2'
11/13/2008 07:45:04 AM - SPINE: Poller[0] Host[27] DS[1300] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.18'
11/13/2008 07:45:04 AM - SPINE: Poller[0] Host[17] DS[316] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.10'
11/13/2008 07:45:04 AM - SPINE: Poller[0] Host[20] DS[607] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.4'
11/13/2008 07:45:04 AM - SPINE: Poller[0] Host[25] DS[1115] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.9'
11/13/2008 07:45:04 AM - SPINE: Poller[0] Host[26] DS[1228] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.14'
11/13/2008 07:45:04 AM - SPINE: Poller[0] Host[18] DS[415] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.2'
11/13/2008 07:45:04 AM - SPINE: Poller[0] Host[27] DS[1300] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.18'
11/13/2008 07:45:04 AM - SPINE: Poller[0] Host[18] DS[414] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.2'
11/13/2008 07:45:04 AM - SPINE: Poller[0] Host[21] DS[708] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.5'
11/13/2008 07:45:04 AM - SPINE: Poller[0] Host[21] DS[708] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.5'
11/13/2008 07:45:04 AM - SPINE: Poller[0] Host[21] DS[708] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.5'
11/13/2008 07:45:04 AM - SPINE: Poller[0] Host[21] DS[707] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.5'
11/13/2008 07:45:04 AM - SPINE: Poller[0] Host[21] DS[707] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.5'
11/13/2008 07:45:04 AM - SPINE: Poller[0] Host[21] DS[707] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.5'
11/13/2008 07:45:04 AM - SPINE: Poller[0] Host[21] DS[707] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.5'
11/13/2008 07:45:04 AM - SPINE: Poller[0] Host[21] DS[706] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.5'
11/13/2008 07:45:04 AM - SPINE: Poller[0] Host[21] DS[706] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.5'
11/13/2008 07:45:04 AM - SPINE: Poller[0] Host[21] DS[706] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.5'
11/13/2008 07:45:04 AM - SPINE: Poller[0] Host[22] DS[842] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.6'
11/13/2008 07:45:04 AM - SPINE: Poller[0] Host[22] DS[841] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.6'
11/13/2008 07:45:04 AM - SPINE: Poller[0] Host[22] DS[841] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.6'
11/13/2008 07:45:04 AM - SPINE: Poller[0] Host[22] DS[841] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.6'
11/13/2008 07:45:04 AM - SPINE: Poller[0] Host[22] DS[841] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.6'
11/13/2008 07:45:04 AM - SPINE: Poller[0] Host[22] DS[840] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.6'
11/13/2008 07:45:04 AM - SPINE: Poller[0] Host[22] DS[840] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.6'
11/13/2008 07:45:04 AM - SPINE: Poller[0] Host[22] DS[840] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.6'
11/13/2008 07:45:04 AM - SPINE: Poller[0] Host[22] DS[840] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.6'
11/13/2008 07:45:04 AM - SPINE: Poller[0] Host[22] DS[839] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.6'
11/13/2008 07:45:04 AM - SPINE: Poller[0] Host[23] DS[965] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.7'
11/13/2008 07:45:04 AM - SPINE: Poller[0] Host[24] DS[1031] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.8'
11/13/2008 07:45:04 AM - SPINE: Poller[0] Host[24] DS[1031] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.8'
11/13/2008 07:45:04 AM - SPINE: Poller[0] Host[24] DS[1031] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.8'
11/13/2008 07:45:04 AM - SPINE: Poller[0] Host[24] DS[1030] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.8'
11/13/2008 07:45:04 AM - SPINE: Poller[0] Host[24] DS[1030] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.8'
11/13/2008 07:45:04 AM - SPINE: Poller[0] Host[24] DS[1030] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.8'
11/13/2008 07:45:04 AM - SPINE: Poller[0] Host[24] DS[1030] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.8'
11/13/2008 07:45:04 AM - SPINE: Poller[0] Host[24] DS[1029] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.8'
11/13/2008 07:45:04 AM - SPINE: Poller[0] Host[24] DS[1029] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.8'
11/13/2008 07:45:04 AM - SPINE: Poller[0] Host[24] DS[1029] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.8'
11/13/2008 07:40:13 AM - SYSTEM STATS: Time:11.7608 Method:spine Processes:1 Threads:12 Hosts:75 HostsPerProcess:75 DataSources:13084 RRDsProcessed:4451
11/13/2008 07:35:11 AM - SYSTEM STATS: Time:9.4637 Method:spine Processes:1 Threads:12 Hosts:75 HostsPerProcess:75 DataSources:13084 RRDsProcessed:4451
11/13/2008 07:30:11 AM - SYSTEM STATS: Time:9.6980 Method:spine Processes:1 Threads:12 Hosts:75 HostsPerProcess:75 DataSources:13084 RRDsProcessed:4451
11/13/2008 07:25:11 AM - SYSTEM STATS: Time:9.8941 Method:spine Processes:1 Threads:12 Hosts:75 HostsPerProcess:75 DataSources:13084 RRDsProcessed:4451
11/13/2008 07:20:13 AM - SYSTEM STATS: Time:11.4608 Method:spine Processes:1 Threads:12 Hosts:75 HostsPerProcess:75 DataSources:13084 RRDsProcessed:4451
11/13/2008 07:15:15 AM - SYSTEM STATS: Time:13.1026 Method:spine Processes:1 Threads:12 Hosts:75 HostsPerProcess:75 DataSources:13084 RRDsProcessed:4451
11/13/2008 07:10:10 AM - SYSTEM STATS: Time:9.3914 Method:spine Processes:1 Threads:12 Hosts:75 HostsPerProcess:75 DataSources:13084 RRDsProcessed:4451
11/13/2008 07:05:12 AM - SYSTEM STATS: Time:10.6490 Method:spine Processes:1 Threads:12 Hosts:75 HostsPerProcess:75 DataSources:13084 RRDsProcessed:4451
11/13/2008 07:00:10 AM - SYSTEM STATS: Time:9.0171 Method:spine Processes:1 Threads:12 Hosts:75 HostsPerProcess:75 DataSources:13084 RRDsProcessed:4451
11/13/2008 06:55:12 AM - SYSTEM STATS: Time:10.3827 Method:spine Processes:1 Threads:12 Hosts:75 HostsPerProcess:75 DataSources:13084 RRDsProcessed:4451
11/13/2008 06:50:09 AM - SYSTEM STATS: Time:7.7291 Method:spine Processes:1 Threads:12 Hosts:75 HostsPerProcess:75 DataSources:13084 RRDsProcessed:4451
11/13/2008 06:45:12 AM - SYSTEM STATS: Time:10.4221 Method:spine Processes:1 Threads:12 Hosts:75 HostsPerProcess:75 DataSources:13084 RRDsProcessed:4451
11/13/2008 06:40:12 AM - SYSTEM STATS: Time:10.7400 Method:spine Processes:1 Threads:12 Hosts:75 HostsPerProcess:75 DataSources:13084 RRDsProcessed:4451
11/13/2008 06:35:12 AM - SYSTEM STATS: Time:10.4160 Method:spine Processes:1 Threads:12 Hosts:75 HostsPerProcess:75 DataSources:13084 RRDsProcessed:4451
11/13/2008 06:30:14 AM - SYSTEM STATS: Time:12.3115 Method:spine Processes:1 Threads:12 Hosts:75 HostsPerProcess:75 DataSources:13084 RRDsProcessed:4451
11/13/2008 06:25:11 AM - SYSTEM STATS: Time:9.4968 Method:spine Processes:1 Threads:12 Hosts:75 HostsPerProcess:75 DataSources:13084 RRDsProcessed:4451
11/13/2008 06:20:11 AM - SYSTEM STATS: Time:10.0448 Method:spine Processes:1 Threads:12 Hosts:75 HostsPerProcess:75 DataSources:13084 RRDsProcessed:4451
11/13/2008 06:15:12 AM - SYSTEM STATS: Time:10.2035 Method:spine Processes:1 Threads:12 Hosts:75 HostsPerProcess:75 DataSources:13084 RRDsProcessed:4451
11/13/2008 06:10:11 AM - SYSTEM STATS: Time:9.8119 Method:spine Processes:1 Threads:12 Hosts:75 HostsPerProcess:75 DataSources:13084 RRDsProcessed:4451
11/13/2008 06:05:12 AM - SYSTEM STATS: Time:10.2525 Method:spine Processes:1 Threads:12 Hosts:75 HostsPerProcess:75 DataSources:13084 RRDsProcessed:4451
11/13/2008 06:00:12 AM - SYSTEM STATS: Time:10.8081 Method:spine Processes:1 Threads:12 Hosts:75 HostsPerProcess:75 DataSources:13084 RRDsProcessed:4451
11/13/2008 05:55:12 AM - SYSTEM STATS: Time:9.9567 Method:spine Processes:1 Threads:12 Hosts:75 HostsPerProcess:75 DataSources:13084 RRDsProcessed:4451
11/13/2008 05:50:11 AM - SYSTEM STATS: Time:10.2994 Method:spine Processes:1 Threads:12 Hosts:75 HostsPerProcess:75 DataSources:13084 RRDsProcessed:4451
11/13/2008 05:45:12 AM - SYSTEM STATS: Time:10.4908 Method:spine Processes:1 Threads:12 Hosts:75 HostsPerProcess:75 DataSources:13084 RRDsProcessed:4451
11/13/2008 05:40:12 AM - SYSTEM STATS: Time:10.0842 Method:spine Processes:1 Threads:12 Hosts:75 HostsPerProcess:75 DataSources:13084 RRDsProcessed:4451
11/13/2008 05:35:11 AM - SYSTEM STATS: Time:9.7047 Method:spine Processes:1 Threads:12 Hosts:75 HostsPerProcess:75 DataSources:13084 RRDsProcessed:4451
11/13/2008 05:30:15 AM - SYSTEM STATS: Time:13.4550 Method:spine Processes:1 Threads:12 Hosts:75 HostsPerProcess:75 DataSources:13084 RRDsProcessed:4451
11/13/2008 05:25:11 AM - SYSTEM STATS: Time:9.6318 Method:spine Processes:1 Threads:12 Hosts:75 HostsPerProcess:75 DataSources:13084 RRDsProcessed:4451
11/13/2008 05:20:11 AM - SYSTEM STATS: Time:9.7922 Method:spine Processes:1 Threads:12 Hosts:75 HostsPerProcess:75 DataSources:13084 RRDsProcessed:4451
11/13/2008 05:20:01 AM - POLLER: Poller[0] WARNING: Poller Output Table not Empty. Potential Data Source Issues for Data Sources: discards_in(DS[660]), discards_out(DS[660]), discards_in(DS[837]), discards_out(DS[837]), discards_in(DS[953]), discards_out(DS[953])
11/13/2008 05:15:11 AM - SYSTEM STATS: Time:9.8175 Method:spine Processes:1 Threads:12 Hosts:75 HostsPerProcess:75 DataSources:13084 RRDsProcessed:4039
11/13/2008 05:15:04 AM - SPINE: Poller[0] Host[21] DS[758] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.5'
11/13/2008 05:15:04 AM - SPINE: Poller[0] Host[27] DS[1350] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.18'
11/13/2008 05:15:04 AM - SPINE: Poller[0] Host[24] DS[1041] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.8'
11/13/2008 05:15:04 AM - SPINE: Poller[0] Host[20] DS[660] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.4'
11/13/2008 05:15:04 AM - SPINE: Poller[0] Host[24] DS[1041] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.8'
11/13/2008 05:15:04 AM - SPINE: Poller[0] Host[24] DS[1041] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.8'
11/13/2008 05:15:04 AM - SPINE: Poller[0] Host[24] DS[1041] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.8'
11/13/2008 05:15:04 AM - SPINE: Poller[0] Host[21] DS[758] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.5'
11/13/2008 05:15:04 AM - SPINE: Poller[0] Host[21] DS[758] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.5'
11/13/2008 05:15:04 AM - SPINE: Poller[0] Host[21] DS[758] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.5'
11/13/2008 05:15:04 AM - SPINE: Poller[0] Host[27] DS[1350] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.18'
11/13/2008 05:15:04 AM - SPINE: Poller[0] Host[27] DS[1349] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.18'
11/13/2008 05:15:04 AM - SPINE: Poller[0] Host[20] DS[660] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.4'
11/13/2008 05:15:04 AM - SPINE: Poller[0] Host[20] DS[659] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.4'
11/13/2008 05:15:04 AM - SPINE: Poller[0] Host[20] DS[659] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.4'
11/13/2008 05:15:04 AM - SPINE: Poller[0] Host[20] DS[659] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.4'
11/13/2008 05:15:04 AM - SPINE: Poller[0] Host[21] DS[757] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.5'
11/13/2008 05:15:04 AM - SPINE: Poller[0] Host[24] DS[1040] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.8'
11/13/2008 05:15:04 AM - SPINE: Poller[0] Host[27] DS[1349] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.18'
11/13/2008 05:15:04 AM - SPINE: Poller[0] Host[20] DS[659] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.4'
11/13/2008 05:15:04 AM - SPINE: Poller[0] Host[24] DS[1040] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.8'
11/13/2008 05:15:04 AM - SPINE: Poller[0] Host[27] DS[1349] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.18'
11/13/2008 05:15:04 AM - SPINE: Poller[0] Host[21] DS[757] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.5'
11/13/2008 05:15:04 AM - SPINE: Poller[0] Host[20] DS[658] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.4'
11/13/2008 05:15:04 AM - SPINE: Poller[0] Host[21] DS[757] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.5'
11/13/2008 05:15:04 AM - SPINE: Poller[0] Host[24] DS[1040] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.8'
11/13/2008 05:15:04 AM - SPINE: Poller[0] Host[27] DS[1349] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.18'
11/13/2008 05:15:04 AM - SPINE: Poller[0] Host[20] DS[658] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.4'
11/13/2008 05:15:04 AM - SPINE: Poller[0] Host[25] DS[1157] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.9'
11/13/2008 05:15:04 AM - SPINE: Poller[0] Host[21] DS[757] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.5'
11/13/2008 05:15:04 AM - SPINE: Poller[0] Host[22] DS[837] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.6'
11/13/2008 05:15:04 AM - SPINE: Poller[0] Host[24] DS[1040] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.8'
11/13/2008 05:15:04 AM - SPINE: Poller[0] Host[27] DS[1348] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.18'
11/13/2008 05:15:04 AM - SPINE: Poller[0] Host[26] DS[1254] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.14'
11/13/2008 05:15:04 AM - SPINE: Poller[0] Host[20] DS[658] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.4'
11/13/2008 05:15:04 AM - SPINE: Poller[0] Host[24] DS[1039] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.8'
11/13/2008 05:15:04 AM - SPINE: Poller[0] Host[27] DS[1348] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.18'
11/13/2008 05:15:04 AM - SPINE: Poller[0] Host[21] DS[756] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.5'
11/13/2008 05:15:04 AM - SPINE: Poller[0] Host[21] DS[756] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.5'
11/13/2008 05:15:04 AM - SPINE: Poller[0] Host[20] DS[658] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.4'
11/13/2008 05:15:04 AM - SPINE: Poller[0] Host[24] DS[1039] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.8'
11/13/2008 05:15:04 AM - SPINE: Poller[0] Host[27] DS[1348] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.18'
11/13/2008 05:15:04 AM - SPINE: Poller[0] Host[23] DS[953] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.7'
11/13/2008 05:15:04 AM - SPINE: Poller[0] Host[27] DS[1348] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.18'
11/13/2008 05:15:04 AM - SPINE: Poller[0] Host[24] DS[1039] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.8'
11/13/2008 05:15:04 AM - SPINE: Poller[0] Host[21] DS[756] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.5'
11/13/2008 05:15:04 AM - SPINE: Poller[0] Host[20] DS[657] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.4'
11/13/2008 05:15:04 AM - SPINE: Poller[0] Host[27] DS[1347] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.18'
11/13/2008 05:15:04 AM - SPINE: Poller[0] Host[18] DS[454] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.2'
11/13/2008 05:15:04 AM - SPINE: Poller[0] Host[23] DS[953] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.7'
11/13/2008 05:15:04 AM - SPINE: Poller[0] Host[23] DS[952] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.7'
11/13/2008 05:15:04 AM - SPINE: Poller[0] Host[23] DS[952] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.7'
11/13/2008 05:15:04 AM - SPINE: Poller[0] Host[23] DS[952] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.7'
11/13/2008 05:15:04 AM - SPINE: Poller[0] Host[23] DS[952] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.7'
11/13/2008 05:15:04 AM - SPINE: Poller[0] Host[23] DS[951] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.7'
11/13/2008 05:15:04 AM - SPINE: Poller[0] Host[23] DS[951] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.7'
11/13/2008 05:15:04 AM - SPINE: Poller[0] Host[23] DS[951] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.7'
11/13/2008 05:15:04 AM - SPINE: Poller[0] Host[23] DS[951] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.7'
11/13/2008 05:15:04 AM - SPINE: Poller[0] Host[23] DS[950] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.7'
11/13/2008 05:15:04 AM - SPINE: Poller[0] Host[25] DS[1157] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.9'
11/13/2008 05:15:04 AM - SPINE: Poller[0] Host[25] DS[1157] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.9'
11/13/2008 05:15:04 AM - SPINE: Poller[0] Host[25] DS[1157] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.9'
11/13/2008 05:15:04 AM - SPINE: Poller[0] Host[25] DS[1156] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.9'
11/13/2008 05:15:04 AM - SPINE: Poller[0] Host[25] DS[1156] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.9'
11/13/2008 05:15:04 AM - SPINE: Poller[0] Host[25] DS[1156] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.9'
11/13/2008 05:15:04 AM - SPINE: Poller[0] Host[25] DS[1156] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.9'
11/13/2008 05:15:04 AM - SPINE: Poller[0] Host[25] DS[1155] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.9'
11/13/2008 05:15:04 AM - SPINE: Poller[0] Host[25] DS[1155] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.9'
11/13/2008 05:15:04 AM - SPINE: Poller[0] Host[25] DS[1155] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.9'
11/13/2008 05:15:04 AM - SPINE: Poller[0] Host[26] DS[1254] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.14'
11/13/2008 05:15:04 AM - SPINE: Poller[0] Host[26] DS[1254] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.14'
11/13/2008 05:15:04 AM - SPINE: Poller[0] Host[26] DS[1254] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.14'
11/13/2008 05:15:04 AM - SPINE: Poller[0] Host[26] DS[1253] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.14'
11/13/2008 05:15:04 AM - SPINE: Poller[0] Host[26] DS[1253] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.14'
11/13/2008 05:15:04 AM - SPINE: Poller[0] Host[26] DS[1253] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.14'
11/13/2008 05:15:04 AM - SPINE: Poller[0] Host[26] DS[1253] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.14'
11/13/2008 05:15:04 AM - SPINE: Poller[0] Host[26] DS[1252] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.14'
11/13/2008 05:15:04 AM - SPINE: Poller[0] Host[26] DS[1252] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.14'
11/13/2008 05:15:04 AM - SPINE: Poller[0] Host[26] DS[1252] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.14'
11/13/2008 05:15:04 AM - SPINE: Poller[0] Host[22] DS[837] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.6'
11/13/2008 05:15:04 AM - SPINE: Poller[0] Host[22] DS[836] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.6'
11/13/2008 05:15:04 AM - SPINE: Poller[0] Host[22] DS[836] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.6'
11/13/2008 05:15:04 AM - SPINE: Poller[0] Host[22] DS[836] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.6'
11/13/2008 05:15:04 AM - SPINE: Poller[0] Host[22] DS[836] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.6'
11/13/2008 05:15:04 AM - SPINE: Poller[0] Host[22] DS[835] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.6'
11/13/2008 05:15:04 AM - SPINE: Poller[0] Host[22] DS[835] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.6'
11/13/2008 05:15:04 AM - SPINE: Poller[0] Host[22] DS[835] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.6'
11/13/2008 05:15:04 AM - SPINE: Poller[0] Host[22] DS[835] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.6'
11/13/2008 05:15:04 AM - SPINE: Poller[0] Host[22] DS[834] WARNING: SNMP timeout detected [500 ms], ignoring host '172.16.200.6'
11/13/2008 05:10:11 AM - SYSTEM STATS: Time:10.4029 Method:spine Processes:1 Threads:12 Hosts:75 HostsPerProcess:75 DataSources:13084 RRDsProcessed:4451
11/13/2008 05:05:13 AM - SYSTEM STATS: Time:11.7485 Method:spine Processes:1 Threads:12 Hosts:75 HostsPerProcess:75 DataSources:13084 RRDsProcessed:4451
11/13/2008 05:00:11 AM - SYSTEM STATS: Time:9.4703 Method:spine Processes:1 Threads:12 Hosts:75 HostsPerProcess:75 DataSources:13084 RRDsProcessed:4451
11/13/2008 04:55:12 AM - SYSTEM STATS: Time:10.9890 Method:spine Processes:1 Threads:12 Hosts:75 HostsPerProcess:75 DataSources:13084 RRDsProcessed:4451
11/13/2008 04:50:13 AM - SYSTEM STATS: Time:11.2170 Method:spine Processes:1 Threads:12 Hosts:75 HostsPerProcess:75 DataSources:13084 RRDsProcessed:4451
11/13/2008 04:45:11 AM - SYSTEM STATS: Time:9.6915 Method:spine Processes:1 Threads:12 Hosts:75 HostsPerProcess:75 DataSources:13084 RRDsProcessed:4451
11/13/2008 04:40:12 AM - SYSTEM STATS: Time:10.5839 Method:spine Processes:1 Threads:12 Hosts:75 HostsPerProcess:75 DataSources:13084 RRDsProcessed:4451
11/13/2008 04:35:10 AM - SYSTEM STATS: Time:9.3675 Method:spine Processes:1 Threads:12 Hosts:75 HostsPerProcess:75 DataSources:13084 RRDsProcessed:4451
11/13/2008 04:30:13 AM - SYSTEM STATS: Time:11.5357 Method:spine Processes:1 Threads:12 Hosts:75 HostsPerProcess:75 DataSources:13084 RRDsProcessed:4451
11/13/2008 04:25:11 AM - SYSTEM STATS: Time:10.3779 Method:spine Processes:1 Threads:12 Hosts:75 HostsPerProcess:75 DataSources:13084 RRDsProcessed:4451
11/13/2008 04:20:11 AM - SYSTEM STATS: Time:9.9051 Method:spine Processes:1 Threads:12 Hosts:75 HostsPerProcess:75 DataSources:13084 RRDsProcessed:4451
11/13/2008 04:15:13 AM - SYSTEM STATS: Time:11.5155 Method:spine Processes:1 Threads:12 Hosts:75 HostsPerProcess:75 DataSources:13084 RRDsProcessed:4451
11/13/2008 04:10:11 AM - SYSTEM STATS: Time:9.6878 Method:spine Processes:1 Threads:12 Hosts:75 HostsPerProcess:75 DataSources:13084 RRDsProcessed:4451
11/13/2008 04:05:11 AM - SYSTEM STATS: Time:9.8219 Method:spine Processes:1 Threads:12 Hosts:75 HostsPerProcess:75 DataSources:13084 RRDsProcessed:4451

[otisbrown]

I may have found the cause of issue. During my investigation of the issue, I noticed both my NICs on the server were sending equal amounts of data. I would expect only one interface to be sending an receiving data as the bond is configured for active-backup. Ever since I disabled the one interface I have not had any issues. I will monitor this until next we before enabling the downed interface to see if the issue comes back.

[TheWitness]

Thanks for the tip. BTB, I have released beta3. You can find it in the Beta Section of the forums.