Virtual counter64 SNMP support

wheelear · Joined: 21 Oct 2005 Posts: 2

Most of the computers I monitor with Cacti do not seem to support 64bit counters. And, the switches I have which do support them seem to have a bug that makes the data unreliable.
So, I decided to fix this by writing a server that would run the needed queries once every 30 seconds, keeping track of the data using a 64bit counter, so I could effectively monitor gigabit interfaces.
Then when Cacti wanted the data, present it the locally maintained 64bit value.

This has been imlemented using perl for the server, and MySQL to hold the bits to query.

Is this something anyone else would be interested in?

The way it works now, when SNMP queries for either of the 2 counter64 interface OIDs are generated, they are converted to the 32bit queries, and the results stored in the database, which the server then updates every 30 seconds.

So, now I have counter64 support for all my systems.

If there is interest, I'll make the patch/server available here.

klavs · Joined: 01 Oct 2002 Posts: 18

I would very much like that patch/script. Have all Gigabit here interfaces here - and no graphs that works with 64-bit counters

wheelear · Joined: 21 Oct 2005 Posts: 2

Ok, here you go.
There are 4 files in the tarball:
SnmpProxy2.pl - The Perl script which runs a query every 30 seconds
snmp_counter64.sql - MySQL Code to put a new table in the database
cacti64php.tgz - Tarball containing replacement php files
README - Installation instructions and notes

Let me know if you have any questions.
As it says in the README, this should only affect queries that are for the 64bit counters. Your current 32bit counters should remain unaffected.

You must also be using the cmd.php poller. I think I'm going to modify cactid, but for now, it requires cmd.php.

prjctgeek · Joined: 13 Jan 2006 Posts: 1

very Creative workaround..

While looking at this idea I checked out the net-snmp site and discovered that the snmpd that comes with RH/Fedora(others?) doesn't have a flag that enables the IfXTable (64bit counters for interfaces). Reconfigure/recompile with the --enable-mfd-rewrites flag and eliminate the linux hosts from your problem list.

look at: http://www.net-snmp.org/about/ChangeLog.html.

snippet:
Ports:
Linux:
- new experimental tables/rewrites for Linux, including:
ifTable, ifXTable, inetCidrRouteTable, ipCidrRouteTable,
ipAddressTable, ipSystemStatsTable, ipNetToPhysicalTable.
- Enable these talbles by specifying --enable-mfd-rewrites to
configure.

fletch · Posted: Sun Jan 15, 2006 11:04 pm Post subject:

I'm about to try this, but am not clear on how to "tell cacti that you want to collect 64bit network statistics for some servers" - from your README.

How do I tell cacti to collect the 64 bit counters?

thanks,
fletch.

fletch · Posted: Sun Jan 15, 2006 11:55 pm Post subject:

answer: choose the DataSource and update bottom dropdown list to "In/Out Bits (64 Bit counters)

its working great - brilliant workaround - although I am afraid to hammer everything every 30 seconds...

thanks,
fletch

fletch · Posted: Tue Jan 17, 2006 2:32 pm Post subject:

Ok, there is a problem with some idle servers.

the graphs for these servers using your proxy script are showing a steady level of > 100Mbit traffic when there is really next to none.

I've guessing the is due to the logic in the script assuming the initial readings wrapped - when really they are not?

Any fix for this? (reset the DB counter?)

Below is an example of the issue - one reading taken from an unmodified 32 bit cacti installation - the other from the 64-bit SnmpProxy workaround modified version. The network is not carrying 200Mbit on this server...its not computing this correctly.

thanks,
fletch.

fletch · Posted: Tue Jan 17, 2006 6:58 pm Post subject:

The first graph is 32 bit counter (pre-patch) the second is the same web server network traffic graph post 64-bit patch - something is not right in the 64 bit version

fletch · Posted: Wed Jan 18, 2006 2:09 am Post subject:

Figured it out

the last32 field in the snmp_counter64 table is an int(32) which does not accept the updates from the pl script (mysql warns - data truncated) - therefore the subsequent differences between current and last grow and grow .
The fix - re-do the last32 as a varchar(20):

drop table snmp_counter64;
create table snmp_counter64(
host varchar(20) not null,
community varchar(20) not null,
oid varchar(255) not null,
version int default 1,
last32 varchar(20) not null default '0',
last64 varchar(20) not null default '0'
);

update the .pl update line to have single quotes around the last32 value:

$Query="UPDATE snmp_counter64 SET last32='$current32', last64='$last64' WHERE host='$host' AND oid='$oid'";

Cheers!
Fletch.

kbartoletta · Posted: Tue Apr 04, 2006 1:33 pm Post subject:

Just to chime in here.... along these same lines, I am trying to graph some items on an F5 BigIP load balencer - all of the values are reported in counter64 data type (64 bit counter) - Am I hearing from your discussion above that there is no way to tell cacti that these are 64 bit counters??? Currently, my graphs are all crazy and inaccurate...

robn · Joined: 25 Jan 2006 Posts: 9

There seems to be a bug somewhere - I ended up with 1000s of duplicate rows for one hosts - enough to ensue mysql took up 90% of the cpu time on a dual P3 1Ghz server

Anyway for now I've added a primary key to the database untill I can look through the code.

robn · Joined: 25 Jan 2006 Posts: 9

It def doesn't like this server - everytime the poller runs I'm getting this error.

SQL Exec Failed "INSERT INTO snmp_counter64(host, community, oid, version, last32, last64) VALUES('ilab.midyorks.nhs.uk:161', 'public', '.1.3.6.1.2.1.2.2.1.16.2', 1, 108951049, 108951049)"

The graph for this host still works fine - but it would be nice to get to the bottom of this and find out why this is happening

robn · Joined: 25 Jan 2006 Posts: 9

I've not got arround to fixing that bug yet, but I have made a few changes to the script

Basically I've changed it so it will run every ~60seconds no matter how long it takes to poll all the devices.

If it takes longer than 60seconds to poll the devices it will simply repeat as soon as its finished.

Why? Will we had an issue with it taking too long with the 30 seconds pause in it. I removed the pause and it was fine, untill this week when it started to take about 8seconds and was constantly updating the database.

These changes should mean that it will repeat striaght away when devices are slow to respond, but still delay things when all devices are responding normally.

robn · Joined: 25 Jan 2006 Posts: 9

I've made a change to the table to reduce disk i/o by using the heap storage method. All the data is lost on mysql restart, but this has little impact on the graphs (plus solves the issue of deleted devices not getting removed from the database).

Faruk · Joined: 04 Apr 2006 Posts: 10