Submit your BOOST Statistics Here

Important information about Cacti developments that all users should be interested in.

Moderators: Moderators, Developers

Author
Message
User avatar
TheWitness
Developer
Posts: 14817
Joined: Tue May 14, 2002 5:08 pm
Location: MI, USA
Contact:

Submit your BOOST Statistics Here

#1 Post by TheWitness » Sat Dec 09, 2006 3:55 am

All,

I have released the Boost Plugin, version 1.0. Once you have something good to say about boost, or bad for that matter, please post your before and after statistics as followups to this post.

Here are the main features:
On-Demand RRD Updating You don't have to update 40k RRD files every 5 minutes, you can wait hours before doing updates. If a user accesses a Graph before a major update, the "On-Demand" updater takes care of it.
Network Service to handle RRD Updating This service is handy when your Web Server does not have RW access to the RRA folder. This provides an added benefit of allowing for multiple Web Servers to be deployed within this framework to host Cacti.
PNG Caching Yes, that's right. This plugin will cache PNG's for as long as they are viable. Once they expire, the first person to view the graph uses the overhead of creating the next good image.

Regards,

TheWitness

http://forums.cacti.net/viewtopic.php?p=86053#86053
True understanding begins only when we realize how little we truly understand...

Life is an adventure, let yours begin with Cacti!

Author of MacTrack, Boost, CLog, SpikeKill, Platform RTM, DSStats, maintainer of Spine, lot's of unpublished work and most of Cacti's bugs.
_________________
Official Cacti Documentation
GitHub Repository with Supported Plugins
Central Plugin Repository
Central Templates Repository


I'm still out there people. Getting excited for Cacti 1.2. I think it will be a great release.

DAP
Posts: 43
Joined: Mon Jul 18, 2005 10:42 pm
Location: Melbourne, Australia

#2 Post by DAP » Fri Dec 15, 2006 1:01 am

Seems to be working fine.

12/15/2006 05:00:23 PM - SYSTEM BOOST STATS: Time:4.2656 RRDUpdates:1022

Love the cache feature :-)

Cheers

PS. View Boost Status not working.

User avatar
gandalf
Developer
Posts: 22375
Joined: Thu Dec 02, 2004 2:46 am
Location: Muenster, Germany
Contact:

#3 Post by gandalf » Sun Dec 17, 2006 2:35 pm

DAP wrote:PS. View Boost Status not working.
This requires a patch to plugin arch. In the meantime, you should be able to find it on the utilities page on the very left bottom side
Reinhard

wriley
Posts: 16
Joined: Fri May 19, 2006 1:17 pm
Location: Lake St Louis, MO USA
Contact:

#4 Post by wriley » Thu Jan 25, 2007 10:08 am

Boost is my new best friend! :) I had been struggling to tweak cacti to handle a huge amount of data sources. Thanks devs!

This system is monitoring nearly 70 Cisco 65xx routers, with a handful of other devices thrown in. This makes for a LOT of data sources.

SYSTEM: BOOST STATS: Time:985.7940 RRDUpdates:988752

SYSTEM: STATS: Time:59.0062 Method:cactid Processes:1 Threads:32 Hosts:74 HostsPerProcess:74 DataSources:75951 RRDsProcessed:0


Total Data Sources: 75951
Total Boost Records: 758095
Current Boost Table Size: 43 MBytes
Avg Bytes/Record: 45 Bytes
Max Allowed Boost Table Size: 4 GBytes
Estimated Maximum Records: 95443718 Records
Last edited by wriley on Mon Feb 12, 2007 1:38 pm, edited 1 time in total.
William Riley
http://workbench.freetcp.com/

User avatar
gandalf
Developer
Posts: 22375
Joined: Thu Dec 02, 2004 2:46 am
Location: Muenster, Germany
Contact:

#5 Post by gandalf » Thu Jan 25, 2007 1:55 pm

You may want to use my new Poller Stats, featuring Boost Statistics (runtime, amount of BOOST) http://forums.cacti.net/viewtopic.php?t=18057
Reinhard

wriley
Posts: 16
Joined: Fri May 19, 2006 1:17 pm
Location: Lake St Louis, MO USA
Contact:

#6 Post by wriley » Fri Jan 26, 2007 9:36 am

gandalf wrote:You may want to use my new Poller Stats, featuring Boost Statistics (runtime, amount of BOOST) http://forums.cacti.net/viewtopic.php?t=18057
Reinhard
I was already using your templates for poller statistics and subsequently turned on the boost stats as well. Thanks for the templates!


These graphs show the improvement when Boost was activated (at about two-thirds point). I also added the final 20 hosts, hence the jump in data sources at the same time.
Attachments
poller_stats.png
Poller Statistics
poller_stats.png (27.45 KiB) Viewed 42467 times
poller_runtime.png
Poller Runtime Statistics
poller_runtime.png (41.59 KiB) Viewed 42467 times
boost_statistics.png
Boost Statistics
boost_statistics.png (20.13 KiB) Viewed 42467 times
William Riley
http://workbench.freetcp.com/

User avatar
gandalf
Developer
Posts: 22375
Joined: Thu Dec 02, 2004 2:46 am
Location: Muenster, Germany
Contact:

#7 Post by gandalf » Fri Jan 26, 2007 2:04 pm

Do not worry, that the green line in the last image stops. This is now reported in the boost stats
Reinhard

User avatar
TheWitness
Developer
Posts: 14817
Joined: Tue May 14, 2002 5:08 pm
Location: MI, USA
Contact:

#8 Post by TheWitness » Fri Jan 26, 2007 8:49 pm

That is one BIG system. Glad to see it come back down to earth. I am sure the web site behaves better now too. I hope you enjoy this plugin I wrote for you. Not a lot of host's, but Lot's of Data Sources...

TheWitness
True understanding begins only when we realize how little we truly understand...

Life is an adventure, let yours begin with Cacti!

Author of MacTrack, Boost, CLog, SpikeKill, Platform RTM, DSStats, maintainer of Spine, lot's of unpublished work and most of Cacti's bugs.
_________________
Official Cacti Documentation
GitHub Repository with Supported Plugins
Central Plugin Repository
Central Templates Repository


I'm still out there people. Getting excited for Cacti 1.2. I think it will be a great release.

kanada
Cacti User
Posts: 137
Joined: Sun Aug 28, 2005 12:51 pm

#9 Post by kanada » Tue Mar 13, 2007 2:50 am

Depends my configuration i test cactid polling vs. booster.
On both settings, timers are same, about 366 sec. (~6 min.)

Polling interval 10 min.

Below are some from my system graphs (cpu, load average).

Current Booster settings:

Current Boost Status
Boost On Demand Updating: Idle
Total Data Sources: 38379
Total Boost Records: 22966

Table Size Estimates
Current Boost Table Size: 16 MBytes
Avg Bytes/Record: 127 Bytes
Max Allowed Boost Table Size: 4 GBytes
Estimated Maximum Records: 33818640 Records

Previous Runtime
Last Start Time: 2007-03-13 8:36:07
Last Run Duration: 319.6620 Seconds
RRD's Updated: 28646

Run Time Configuration
Update Frequency: 20 Minutes
Next Start Time: 2007-03-13 8:56:07
Maximum Records: 1000000 Records
Maximum Allowed Runtime: 20 Minutes

Boost Server Details
Server Config Status: Enabled
Multiprocess Server: Single Process
Update Timeout: 2 Seconds
Server/Port: [email protected]
Authorized Update Web Servers: 127.0.0.1
RRDtool Binary Used: /usr/bin/rrdtool

Seems, that Booster eat more system than cactid poller.
Its normal?

Best Regards,
Alex.
Attachments
cpu01.PNG
Daily, before a gap works on booster.
cpu01.PNG (26.43 KiB) Viewed 42022 times
cpu02.PNG
Weekly.
cpu02.PNG (41.03 KiB) Viewed 42022 times
load01.PNG
Daily, before 12h works on booster.
load01.PNG (34.35 KiB) Viewed 42022 times
load03.PNG
Monthly, from Week 10 system works on booster.
load03.PNG (46.5 KiB) Viewed 42022 times

User avatar
gandalf
Developer
Posts: 22375
Joined: Thu Dec 02, 2004 2:46 am
Location: Muenster, Germany
Contact:

#10 Post by gandalf » Tue Mar 13, 2007 11:18 am

From my point of view, boost is not a replacement for cactid. IMHO, the sequence would be

1) Use cmd.php as a start
If your systems grows, you may hit the wall unless ...
2) Use cactid as a fast (by order of a magnitude) replacement next
If your system still grows (we are speaking of at least 20,000 to 30,000 to 40,000 and more data sources) this will not be fast enough. rrdtool update consumes most of the time. But rrdtool features sth. like "multi-row insert". Boost will use this. If you hit the wall again ...
3) use cactid in combination with boost
This is now the ultimate step.
Reinhard

User avatar
TheWitness
Developer
Posts: 14817
Joined: Tue May 14, 2002 5:08 pm
Location: MI, USA
Contact:

#11 Post by TheWitness » Tue Mar 13, 2007 3:31 pm

Concurring with Gandolf.

TheWitness
True understanding begins only when we realize how little we truly understand...

Life is an adventure, let yours begin with Cacti!

Author of MacTrack, Boost, CLog, SpikeKill, Platform RTM, DSStats, maintainer of Spine, lot's of unpublished work and most of Cacti's bugs.
_________________
Official Cacti Documentation
GitHub Repository with Supported Plugins
Central Plugin Repository
Central Templates Repository


I'm still out there people. Getting excited for Cacti 1.2. I think it will be a great release.

kanada
Cacti User
Posts: 137
Joined: Sun Aug 28, 2005 12:51 pm

#12 Post by kanada » Wed Mar 14, 2007 3:10 am

Yes Gandalf you have right, as you see now i have about 40k Data Sources on 10k Hosts, and this number will grows..
Up i post my stats cacti+cactid working on normal polling vs. cacti+cactid+boost..

Graphs shows that in the same poll time (normal and booster check hosts in ~6 min) system utilization (CPU, Memory, HDD) is higher when work boost. I still search a 'gold center' to minimize a polling time (to 5 min.) && system use.

I will post there my test results, if you don't have nothing against that.
Thank you for advice..

Best Regards,
Alex.

User avatar
TheWitness
Developer
Posts: 14817
Joined: Tue May 14, 2002 5:08 pm
Location: MI, USA
Contact:

#13 Post by TheWitness » Thu Apr 12, 2007 6:43 pm

Alex,

I would move your update frequency to more like every 1-3 hours.

TheWitness
True understanding begins only when we realize how little we truly understand...

Life is an adventure, let yours begin with Cacti!

Author of MacTrack, Boost, CLog, SpikeKill, Platform RTM, DSStats, maintainer of Spine, lot's of unpublished work and most of Cacti's bugs.
_________________
Official Cacti Documentation
GitHub Repository with Supported Plugins
Central Plugin Repository
Central Templates Repository


I'm still out there people. Getting excited for Cacti 1.2. I think it will be a great release.

dawald
Posts: 22
Joined: Wed Feb 02, 2005 5:26 pm
Location: Zurich, Switzerland

#14 Post by dawald » Wed Apr 18, 2007 2:22 pm

This plugin is just amazing, finally I'm able to write all my data into the rra's without getting gaps in the graphs.

cacti: 0.8.6j
cactid: 0.8.6i

BOOST Stats:
***********

Code: Select all

04/18/2007 06:49:12 PM - SYSTEM BOOST STATS: Time:380.0422 RRDUpdates:548371
04/18/2007 07:49:09 PM - SYSTEM BOOST STATS: Time:377.6698 RRDUpdates:543419
POLLER Stats:
***********

Code: Select all

04/18/2007 08:32:53 PM - SYSTEM STATS: Time:51.5142 Method:cactid Processes:1 Threads:15 Hosts:251 HostsPerProcess:251 DataSources:50366 RRDsProcessed:0
04/18/2007 08:37:54 PM - SYSTEM STATS: Time:52.6713 Method:cactid Processes:1 Threads:15 Hosts:251 HostsPerProcess:251 DataSources:50366 RRDsProcessed:0
04/18/2007 08:42:51 PM - SYSTEM STATS: Time:49.4776 Method:cactid Processes:1 Threads:15 Hosts:251 HostsPerProcess:251 DataSources:50366 RRDsProcessed:0
Thanks a lot 'TheWitness', this definitly r0cks!!

cheers
:lol:

dawald
Posts: 22
Joined: Wed Feb 02, 2005 5:26 pm
Location: Zurich, Switzerland

#15 Post by dawald » Mon Apr 23, 2007 3:15 am

I would like to show you the impact of boost plugin on my disk i/o waittime. I also attachted a graph that show my loadavg. The first part of the graph shows statistics with the boost plugin. As soon as I switched back to the regular cactid poller method without the boost plugin, the red area (i/o wait) shows up...

cheers :roll:
Attachments
cacti_boost_plugin_iowait.png
iowait (red)
cacti_boost_plugin_iowait.png (38.61 KiB) Viewed 41227 times
cacti_boost_plugin_loadavg.png
another shot of the loadaverage
cacti_boost_plugin_loadavg.png (58.67 KiB) Viewed 41227 times

Post Reply