Very nice demonstration of positive impact.

How many hosts? How many data sources? How many RRDs? Show some before and after "STATS".

TheWitness

05/24/2007 02:19:22 PM - SYSTEM BOOST STATS: Time:221.6682 RRDUpdates:86368
05/24/2007 02:15:39 PM - SYSTEM STATS: Time:38.1070 Method:cactid Processes:8 Threads:16 Hosts:376 HostsPerProcess:47 DataSources:28971 RRDsProcessed:0
05/24/2007 02:10:40 PM - SYSTEM STATS: Time:39.4355 Method:cactid Processes:8 Threads:16 Hosts:376 HostsPerProcess:47 DataSources:28971 RRDsProcessed:0
05/24/2007 02:05:57 PM - SYSTEM STATS: Time:56.4675 Method:cactid Processes:8 Threads:16 Hosts:376 HostsPerProcess:47 DataSources:28970 RRDsProcessed:0
05/24/2007 02:05:42 PM - SYSTEM BOOST STATS: Time:299.0875 RRDUpdates:86055
05/24/2007 02:00:40 PM - SYSTEM STATS: Time:39.9136 Method:cactid Processes:8 Threads:16 Hosts:376 HostsPerProcess:47 DataSources:28970 RRDsProcessed:0
05/24/2007 01:55:38 PM - SYSTEM STATS: Time:37.3334 Method:cactid Processes:8 Threads:16 Hosts:376 HostsPerProcess:47 DataSources:28970 RRDsProcessed:0
05/24/2007 01:50:39 PM - SYSTEM STATS: Time:39.2261 Method:cactid Processes:8 Threads:16 Hosts:376 HostsPerProcess:47 DataSources:28970 RRDsProcessed:0
05/24/2007 01:49:16 PM - SYSTEM BOOST STATS: Time:213.1302 RRDUpdates:57637
05/24/2007 01:45:40 PM - SYSTEM STATS: Time:40.0143 Method:cactid Processes:8 Threads:16 Hosts:376 HostsPerProcess:47 DataSources:28970 RRDsProcessed:0
05/24/2007 01:40:39 PM - SYSTEM STATS: Time:38.7827 Method:cactid Processes:8 Threads:16 Hosts:376 HostsPerProcess:47 DataSources:28970 RRDsProcessed:0
05/24/2007 01:39:38 PM - SYSTEM BOOST STATS: Time:237.0712 RRDUpdates:86261
05/24/2007 01:35:39 PM - SYSTEM STATS: Time:38.9514 Method:cactid Processes:8 Threads:16 Hosts:376 HostsPerProcess:47 DataSources:28970 RRDsProcessed:0
05/24/2007 01:30:42 PM - SYSTEM STATS: Time:41.0048 Method:cactid Processes:8 Threads:16 Hosts:376 HostsPerProcess:47 DataSources:28970 RRDsProcessed:0
05/24/2007 01:25:40 PM - SYSTEM STATS: Time:39.4576 Method:cactid Processes:8 Threads:16 Hosts:376 HostsPerProcess:47 DataSources:28970 RRDsProcessed:0
05/24/2007 01:23:35 PM - SYSTEM BOOST STATS: Time:156.9922 RRDUpdates:57513
05/24/2007 01:20:54 PM - SYSTEM STATS: Time:54.0344 Method:cactid Processes:8 Threads:16 Hosts:376 HostsPerProcess:47 DataSources:28970 RRDsProcessed:0

Before... polling+update taken around 320seconds

This is really great ! I will be able to graph all things i want now
I was limited to around 30k datasource to be able to run the poller every 5 minutes...
I will post some graph later (i have just installed boost)
I will also try to do something to be able to update rrds used by weathermap after each poll (somethink like the on demand update for graphs)

Here is a pass of the new Cactid 0.8.6j/k and Boost 1.4

07/11/2007 05:05:40 AM - SYSTEM STATS: Time:38.6084 Method:cactid Processes:10 Threads:30 Hosts:3372 HostsPerProcess:338 DataSources:18583 RRDsProcessed:0

TheWitness

Is it "safe" to try version 1.4 on a 0.8.6j install, I'd like to try and lower my load average. I'm averaging about 4-5 right now, and would like to be closer to 3.

I would go to 1.5. However, I have found that if you have to turn it off, that it's not working too well. If you do have to turn it off for some reason, force a run and then comment it out from config.php.

Hmm, Did I release 1.5. Have to check...

TheWitness

After messing with my hardware (note, 15k RPM drives are a must with many RRDs to update) I am now able to poll my entire network in less than 3 minutes, and also update all the RRD's. I've been running Cricket for years...and to poll the entire network using Cricket required 8+ minutes with each poll. This was even after a complete re-write of Cricket to make it use threads.

This plugin and whole system is great!

Thanks!

A note working with another customer late last week.

1. Bump your PHP memory (php.ini -> memory_limit) up to match your memory size table (assuming you are using memory tables for poller_output_boost).
2. Move your boost update interval to about an hour or so.
3. Move your record limit up to be effectively an hour at a time (2+m records)
4. If all you are using for in input method is snmp and your maximum poller_output field length is less than 50, reduce the column width on the poller_output_boot table to 50. This will allow some 8-10x the number of records to be held in your poller_output_boost table.
5. If using memory tables, make sure you have enough memory to handle all 2+m records. Maybe "max_heap_table=1G" or so.

My observation is that you are doing updates too often. These steps will reduce your load average.

TheWitness

Larry, is it possible to read out the MySQL memory size table and dynamically change the php.ini's memory setting? Just to avoid such issues?
Reinhard

Hi Larry,
thanks for very interisting letter of recommendation. I'm busy oneself with similar considerations. We're currently moving to new productive hardware, as we come up to the polling cyle limit .

08/01/2007 10:28:25 PM - SYSTEM STATS: Time:204.9392 Method:cactid Processes:6 Threads:30 Hosts:1940 HostsPerProcess:324 DataSources:73158 RRDsProcessed:36870
08/01/2007 10:23:29 PM - SYSTEM STATS: Time:207.7885 Method:cactid Processes:6 Threads:30 Hosts:1940 HostsPerProcess:324 DataSources:73158 RRDsProcessed:36938

More HW details here http://forums.cacti.net/viewtopic.php?p ... ght=#51346

So we curently have a ratio of about 350 DS/sec. Compared with your stats you are at 500 DS/sec. I suppose it's mostly SNMP processing.

So my considerations are.

1. Where is the limit of DS/RRDs per box within the default polling cycle of 300 sec.
Is it theoretically limited at 150k DS/300sec ??
What are the key influence factors: I/O RAM, database RAM, PHP memory, CPUs, NFS mount performance, remote SQL performance ??

2. Where is the best balance between these 2 designs:
a.) Polling and database on a single box(8 core IBM445, 8GB RAM) -\- httpd and other plugins on the second box (equal HW)
b.) Only Polling on the single box (most memory for best I/O) -\- database and all other processes on the second box. With boost on the polling box and a high memory limit on the database box.

3. Will it be possible to build a determination matrix between all HW and SW influence factors and DS/RRDs per second?

Any further designs ideas and performance hints are desired.

Regards
Frizz

Yes. Good idea

Frizz,

Tobi has been performing some tests with RRDtool 1.3 and what he has found is that with the new design, he is able to get a 3 to 5 x performance increase during updates. So, I would definately add that to the list.

However, the largest factor is MEMORY and of course the speed of your disks. So, my "best" recommendation is that on the box that actually has the RRDfiles, you need the fastest disks possible. I have been recommending RAID10 with 6 or more disks in a striped arrangement and from anywhere to 8-32GBytes of RAM (Cheap relatively speaking now).

Then, for the largest installs, the database always goes on a separate box with memory requirements requisite on the size of the installation and the use of the Database server. Is it dual use? If not, then it only needs enough memory for the key cache (MyISAM) and your memory tabkes (if any). If not using Memory tables, you need enough disk cache (MyISAM) to keep the various tables in memory. Of course they should be close to one another, latency wise, to increase performance.

Regards,

Larry

Frizz,

One more note on DS/Sec. The next release of Cactid, is much more aggressive at collection from multiple hosts. I have designed it, latency aside, to launch 10k collection threads per minute.

Larry

Somebody would call this a DOS attack, won't you?
Reinhard

Assuming you have Cacti in the core and not out on the edge, it should only be considered efficient. In cases where you have concerns about crushing your network with snmp, I would not be too concerned as it is not a chatty protocol.

However, with 26 concurrent bulkwalk sessions on a Windows box, I was able to achieve 2.5mbps of network utilization. Not much these days.

TeWitness

Hey, this was a joke! I'm REALLY impressed about the constant improvements on cactid. Each time I suppose, the current version is as fast as lightning and the ultimate performance is reached, you're setting the limits higher.
Reinhard