Cacti (home)ForumsDocumentation
Cacti: offical forums and support
It is currently Tue May 23, 2017 5:24 pm

All times are UTC - 5 hours




Post new topic Reply to topic  [ 33 posts ]  Go to page 1, 2, 3  Next
Author Message
 Post subject: DSStats - 1.1 Released (Data Source Statistics)
PostPosted: Thu Aug 20, 2009 8:12 pm 
Offline
Developer
User avatar

Joined: Tue May 14, 2002 5:08 pm
Posts: 14863
Location: MI, USA
All,

Version 1.1 has been released incorporating changes from Howie that allow for a more expansive use of the data that the plugin provides via either additional plugins, or new Graph Templates.

Regards,

TheWitness
_________________________________________________

All,

Please find the following user contributed plugin called DSStats. This plugin is an infrastructure plugin. What this means, is that you see nothing but slowed polling times by installing it, or at least on the surface.

However, if you are currently creating aggregate graphs and your savvy with SQL, you can now write script queries, that perform database calls to build those graphs further enhancing what we can already do with Aggregate and RRDtool.

Oh yeah right you say. Well how about a few examples. Let's say you are tracking the temperature of all equipment in a rack of servers, and want to know the average and peak temperatures within that rack, and you have some "MetaData" that tells you that so and so servers are in that rack.

Another example, would be to total all network bandwidth utilization within a state, city, country, floor, building, yah da yah da yah. Now, you can do it with a simple SQL query that sum's all data.

How about the top 10 Gigabit interfaces out of 10,000 in your company, either in the last hour, day, week, month, year. Now you can build a plugin that does that in an instant.

Well, I would like your comments, and donations to the Cacti Group. If you can spare it, please consider donating 10-100 dollars to the Cacti Group each year. It pays for lot's of beer and cider and keeps us going. Seriously!

And of course, before you turn this plugin loose, you must keep in mind the following:

1) The plugin uses two memory tables, do make sure you have enough memory to handle them.
2) If your poller is borderline in terms of making all polling in a poller cycle, then don't enable this without being prepared to immediately disable it.

Finally, good luck. Enjoy Cacti.

Regards,

TheWitness

Download Here

_________________
True understanding begins only when we realize how little we truly understand...

Life is an adventure, let yours begin with Cacti!

Author of MacTrack, Boost, CLog, SpikeKill, Platform RTM, DSStats, maintainer of Spine, lot's of unpublished work and most of Cacti's bugs.
_________________
Official Cacti Documentation
Gandalfs Official Debugging Help
Central Plugin Repository
Central Templates Repository


Last edited by TheWitness on Thu Apr 15, 2010 7:58 am, edited 6 times in total.

Top
 Profile  
 
 Post subject:
PostPosted: Fri Aug 21, 2009 4:35 am 
Offline
Cacti User
User avatar

Joined: Tue Feb 10, 2009 3:24 am
Posts: 209
Location: Frankfurt (a.M.), Germany
Hi there,

I guess this is kinda groundbreaking!
Cacti catches up with commercial solutions (I do often talk about Ipswitch Whatsup Gold) very fast and very professional, too.
I'll try this system ASAP.

May I donate beer? ;)

brgds

_________________
I'm a bit redundant. I repeat myself too ...


Top
 Profile  
 
 Post subject:
PostPosted: Fri Aug 21, 2009 6:21 am 
Offline
Developer
User avatar

Joined: Tue May 14, 2002 5:08 pm
Posts: 14863
Location: MI, USA
How about sponsorship so that we can visit where the beers made :)

TheWitness

_________________
True understanding begins only when we realize how little we truly understand...

Life is an adventure, let yours begin with Cacti!

Author of MacTrack, Boost, CLog, SpikeKill, Platform RTM, DSStats, maintainer of Spine, lot's of unpublished work and most of Cacti's bugs.
_________________
Official Cacti Documentation
Gandalfs Official Debugging Help
Central Plugin Repository
Central Templates Repository


Top
 Profile  
 
 Post subject: dsstats
PostPosted: Fri Aug 21, 2009 9:18 am 
Offline
Cacti User

Joined: Thu May 17, 2007 9:18 am
Posts: 234
how do we donate, can we buy something from Amazon?


Top
 Profile  
 
 Post subject:
PostPosted: Fri Aug 21, 2009 10:11 am 
Offline
Developer
User avatar

Joined: Tue May 14, 2002 5:08 pm
Posts: 14863
Location: MI, USA
We prefer PayPal.

http://www.cacti.net/donate.php

TheWitness

_________________
True understanding begins only when we realize how little we truly understand...

Life is an adventure, let yours begin with Cacti!

Author of MacTrack, Boost, CLog, SpikeKill, Platform RTM, DSStats, maintainer of Spine, lot's of unpublished work and most of Cacti's bugs.
_________________
Official Cacti Documentation
Gandalfs Official Debugging Help
Central Plugin Repository
Central Templates Repository


Top
 Profile  
 
 Post subject:
PostPosted: Mon Aug 24, 2009 7:56 am 
Offline
Cacti Guru User
User avatar

Joined: Thu Sep 16, 2004 5:53 am
Posts: 5097
Location: United Kingdom
Could the last calculated value also be put into data_source_stats_hourly_last?

To get a list of the most recent values for a particular subset of results, the query needs to have some time component that it wouldn't otherwise need.

Code:
select dss.*,dtd.name_cache,dtd.data_source_path from data_source_stats_hourly_cache dss, data_template_data dtd where value > 0 and (rrd_name='errors_in' or rrd_name='errors_out') and dtd.local_data_id = dss.local_data_id and dss.time > '2009-08-24 13:50' order by name_cache;


needs my script to figure out where I am relative to a poller cycle, and then what the '>' time should be. It could be a trivial (and static) query otherwise.

Or at least store the last start time somewhere?

_________________
Weathermap 0.98 is out! & QuickTree 0.2. Superlinks is over there now.
Some Other Cacti tweaks, including strip-graphs, icons and snmp/netflow stuff.
(Let me know if you have UK DevOps or Network Ops opportunities, too!)


Top
 Profile  
 
 Post subject:
PostPosted: Mon Aug 24, 2009 9:04 am 
Offline
Developer
User avatar

Joined: Tue May 14, 2002 5:08 pm
Posts: 14863
Location: MI, USA
Makes sense. Thought about that after I sent my PM. It makes for a time consuming query. Do you want to make the patch?

TheWitness

_________________
True understanding begins only when we realize how little we truly understand...

Life is an adventure, let yours begin with Cacti!

Author of MacTrack, Boost, CLog, SpikeKill, Platform RTM, DSStats, maintainer of Spine, lot's of unpublished work and most of Cacti's bugs.
_________________
Official Cacti Documentation
Gandalfs Official Debugging Help
Central Plugin Repository
Central Templates Repository


Top
 Profile  
 
 Post subject:
PostPosted: Mon Aug 24, 2009 9:05 am 
Offline
Developer
User avatar

Joined: Tue Jun 13, 2006 1:17 am
Posts: 786
Location: Cologne, Germany
removed

_________________
Hat das Blümchen einen Knick, war der Schmetterling zu dick! ;)
reportit v0.7.5a
SNMPAgent v0.2.3
Download ReportIt | Download SNMPAgent | ReportIt SVN | ReportIt Templates | Wish list


Top
 Profile  
 
 Post subject:
PostPosted: Mon Aug 24, 2009 10:18 am 
Offline
Cacti Guru User
User avatar

Joined: Thu Sep 16, 2004 5:53 am
Posts: 5097
Location: United Kingdom
To save someone else the research, I needed to alter my my.cnf:

Code:
# 64M heap table size
max_heap_table_size = 67108864
tmp_table_size = 67108864

even with a smallish system and one hour of history.

_________________
Weathermap 0.98 is out! & QuickTree 0.2. Superlinks is over there now.
Some Other Cacti tweaks, including strip-graphs, icons and snmp/netflow stuff.
(Let me know if you have UK DevOps or Network Ops opportunities, too!)


Top
 Profile  
 
 Post subject:
PostPosted: Mon Aug 24, 2009 10:18 am 
Offline
Cacti Guru User
User avatar

Joined: Thu Sep 16, 2004 5:53 am
Posts: 5097
Location: United Kingdom
TheWitness wrote:
Makes sense. Thought about that after I sent my PM. It makes for a time consuming query. Do you want to make the patch?

TheWitness


This seems to be working for me.

Code:
*** setup.php-orig      2009-08-24 15:17:21.000000000 +0100
--- setup.php   2009-08-24 16:00:16.000000000 +0100
***************
*** 187,190 ****
--- 187,191 ----
                        `rrd_name` varchar(19) NOT NULL,
                        `value` varchar(30) NOT NULL,
+                       `calculated` DOUBLE NOT NULL,
                        PRIMARY KEY  (`local_data_id`,`rrd_name`)
                        ) ENGINE=MEMORY DEFAULT CHARSET=UTF8;"
***************
*** 192,195 ****
--- 193,200 ----
        }

+       if (!sizeof(db_fetch_row("SHOW COLUMNS from data_source_stats_hourly_last where Field='calculated'"))) {
+               db_execute("ALTER TABLE data_source_stats_hourly_last ADD calculated double not null after value");
+       };
+
        if (!sizeof(db_fetch_row("SHOW TABLES LIKE 'data_source_stats_monthly'"))) {
                db_execute("CREATE TABLE `data_source_stats_monthly` (
***************
*** 315,319 ****
                        $outbuf           = "";
                        $sql_cache_prefix = "INSERT INTO data_source_stats_hourly_cache (local_data_id, rrd_name, time, `value`) VALUES";
!                       $sql_last_prefix  = "INSERT INTO data_source_stats_hourly_last (local_data_id, rrd_name, `value`) VALUES";
                        $sql_suffix       = " ON DUPLICATE KEY UPDATE `value`=VALUES(`value`)";
                        $overhead         = strlen($sql_prefix) + strlen($sql_suffix);
--- 320,324 ----
                        $outbuf           = "";
                        $sql_cache_prefix = "INSERT INTO data_source_stats_hourly_cache (local_data_id, rrd_name, time, `value`) VALUES";
!                       $sql_last_prefix  = "INSERT INTO data_source_stats_hourly_last (local_data_id, rrd_name, `value`, calculated) VALUES";
                        $sql_suffix       = " ON DUPLICATE KEY UPDATE `value`=VALUES(`value`)";
                        $overhead         = strlen($sql_prefix) + strlen($sql_suffix);
***************
*** 339,342 ****
--- 344,348 ----
                        $last_i       = 1;
                        $out_length   = 0;
+                       $last_length   = 0;
                        $lastbuf      = "";
                        $cachebuf     = "";
***************
*** 449,458 ****
                                                                $result["local_data_id"] . "','" .
                                                                $result["rrd_name"] . "','" .
!                                                               $lastval . "')";
                                                        $last_i++;
                                                }

                                                /* if we exceed out output buffer, it's time to write */
!                                               if (($out_length + $overhead) > $max_packet) {
                                                        db_execute($sql_cache_prefix . $cachebuf . $sql_suffix);

--- 455,466 ----
                                                                $result["local_data_id"] . "','" .
                                                                $result["rrd_name"] . "','" .
!                                                               $lastval . "','" .
!                                                               ($currentval != "U" ? $currentval:"-90909090909") . "')";
                                                        $last_i++;
+                                                       $last_length += strlen($lastbuf);
                                                }

                                                /* if we exceed out output buffer, it's time to write */
!                                               if ( ( ($out_length + $overhead) > $max_packet ) || ( ($last_length + $overhead) > $max_packet ) ) {
                                                        db_execute($sql_cache_prefix . $cachebuf . $sql_suffix);

***************
*** 464,467 ****
--- 472,476 ----
                                                        $lastbuf      = "";
                                                        $out_length   = 0;
+                                                       $last_length  = 0;
                                                        $cache_i      = 1;
                                                        $last_i       = 1;



Attachments:
ds-stats.patch [4.37 KiB]
Downloaded 297 times

_________________
Weathermap 0.98 is out! & QuickTree 0.2. Superlinks is over there now.
Some Other Cacti tweaks, including strip-graphs, icons and snmp/netflow stuff.
(Let me know if you have UK DevOps or Network Ops opportunities, too!)
Top
 Profile  
 
 Post subject:
PostPosted: Mon Aug 24, 2009 11:54 am 
Offline
Developer
User avatar

Joined: Tue May 14, 2002 5:08 pm
Posts: 14863
Location: MI, USA
Thanks Howie!

Larry

_________________
True understanding begins only when we realize how little we truly understand...

Life is an adventure, let yours begin with Cacti!

Author of MacTrack, Boost, CLog, SpikeKill, Platform RTM, DSStats, maintainer of Spine, lot's of unpublished work and most of Cacti's bugs.
_________________
Official Cacti Documentation
Gandalfs Official Debugging Help
Central Plugin Repository
Central Templates Repository


Top
 Profile  
 
 Post subject:
PostPosted: Tue Aug 25, 2009 2:58 am 
Offline
Cacti Guru User
User avatar

Joined: Thu Sep 16, 2004 5:53 am
Posts: 5097
Location: United Kingdom
TheWitness wrote:
Thanks Howie!

Larry


Not so fast... it seems to be working for most data, but not at all for what I was looking for (errors_in and errors_out) - they are all marked as NaNs, even though the _cache table has real data for them.

_________________
Weathermap 0.98 is out! & QuickTree 0.2. Superlinks is over there now.
Some Other Cacti tweaks, including strip-graphs, icons and snmp/netflow stuff.
(Let me know if you have UK DevOps or Network Ops opportunities, too!)


Top
 Profile  
 
 Post subject:
PostPosted: Tue Aug 25, 2009 3:49 am 
Offline
Cacti Guru User
User avatar

Joined: Thu Sep 16, 2004 5:53 am
Posts: 5097
Location: United Kingdom
:oops:

So it wasn't working for *any* data after the first inserts...

NOW I'm getting updates OK. Patch from 1.0, not from my previous.


Attachments:
dsstats patchv2.txt [3.92 KiB]
Downloaded 346 times

_________________
Weathermap 0.98 is out! & QuickTree 0.2. Superlinks is over there now.
Some Other Cacti tweaks, including strip-graphs, icons and snmp/netflow stuff.
(Let me know if you have UK DevOps or Network Ops opportunities, too!)
Top
 Profile  
 
 Post subject:
PostPosted: Tue Aug 25, 2009 4:19 am 
Offline
Cacti Guru User
User avatar

Joined: Thu Sep 16, 2004 5:53 am
Posts: 5097
Location: United Kingdom
and to go with that, here's a first cut of a report page for Superlinks that shows you all interfaces with non-zero error counters (assuming you added the error counter DS in the first place). Drop it into the content/ folder of Superlinks and add it in the usual way. Rename it to something.php first.

You need to have my patch applied to DSStats, because the query looks for the field that that patch adds.

I could do with some help on the query so that it picks up a more-correct graph for the link. I can't figure out how to get the error graph instead of just the first graph for each interface - on my install I sometimes get the errors graph and sometimes get the traffic graph, and sometimes a completely wrong graph.

Anyway, might be a useful start for other folks wanting to make top 10 type reports - no need to make a whole plugin unless you really need to :-)


Attachments:
all-errors.txt [2.04 KiB]
Downloaded 397 times

_________________
Weathermap 0.98 is out! & QuickTree 0.2. Superlinks is over there now.
Some Other Cacti tweaks, including strip-graphs, icons and snmp/netflow stuff.
(Let me know if you have UK DevOps or Network Ops opportunities, too!)
Top
 Profile  
 
 Post subject:
PostPosted: Tue Aug 25, 2009 3:38 pm 
Offline
Developer
User avatar

Joined: Tue May 14, 2002 5:08 pm
Posts: 14863
Location: MI, USA
Make it easy :)

Larry

_________________
True understanding begins only when we realize how little we truly understand...

Life is an adventure, let yours begin with Cacti!

Author of MacTrack, Boost, CLog, SpikeKill, Platform RTM, DSStats, maintainer of Spine, lot's of unpublished work and most of Cacti's bugs.
_________________
Official Cacti Documentation
Gandalfs Official Debugging Help
Central Plugin Repository
Central Templates Repository


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 33 posts ]  Go to page 1, 2, 3  Next

All times are UTC - 5 hours


Who is online

Users browsing this forum: No registered users and 2 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  

Protected by Anti-Spam ACP Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group