Cacti (home)ForumsDocumentation
Cacti: offical forums and support
It is currently Wed May 22, 2019 2:20 am

All times are UTC - 5 hours




Post new topic Reply to topic  [ 58 posts ]  Go to page 1, 2, 3, 4  Next
Author Message
 Post subject: [HOWTO] Using Data Queries and Data Input Methods
PostPosted: Sat Jan 07, 2006 10:45 am 
Offline
Developer
User avatar

Joined: Thu Dec 02, 2004 2:46 am
Posts: 22376
Location: Muenster, Germany
This thread is not maintained any longer. Please refer to http://docs.cacti.net instead


Preface
This tutorial to help you figuring out, how to extend cacti's build-in capabilities by own scripts and queries. Some of them are of course part of the standard cacti distribution files. By those means, it is possible not only to work on SNMP related data but to retrieve virtually everything using custom-made code. This is not even restricted to certain programming languages; you'll find php, perl, shell/batch and more.
There a two ways extending cacti's build-in capabilities:
  • Data Input Methods for querying single readings or multiple, but non-indexed readings
  • Data Queries for indexed readings (like Network Interfaces or other SNMP Tables, you may even create Data Queries as scripts)
By using the Exporting and Importing Facilities, it is possible to share your results with others.
As always: Use this information at your own risk.

Common Tasks
In principle, it is possible to divide the following tasks into three different parts:
  • how to retrieve data
  • how to store data
  • how to present data
Image

Data Input Method
Find more about this topic in the cacti documentation: Chapter 9. Data Input Methods

Create a Data Input method to tell cacti how to gather data
Data Input Method returning a single value
Lets start with a simple script, that takes a hostname or IP address as input parameter, returning a single value. You may find this one as <path_cacti>/scripts/ping.pl:
Code:
#!/usr/bin/perl
$ping = `ping -c 1 $ARGV[0] | grep icmp_seq`;
$ping =~ m/(.*time=)(.*) (ms|usec)/;
print $2;
To define this script as a Data Input Method to cacti, please go to Data Input Methods and click Add. You should see:
Image

Please fill in Name, select Script/Command as Input Type and provide the command that should be used to retrieve the data. You may use <path_cacti> as a symbolical name for the path_to_your_cacti_installation. Those commands will be executed from crontab; so pay attention to providing full path to binaries if required (e.g. /usr/bin/perl instead of perl). Enter all Input Parameters in <> brackets. Click
create to see:
Image

Now lets define the Input Fields. Click Add as given above to see:
Image

The DropDown Field [Input] contains one single value only. This is taken from the Input String <host> above. Fill Friendly Name to serve your needs. The Special Type Code allows you to provide parameters from the current Device to be queried. In this case, the hostname will be taken from the current device.
Click create to see:
Image

At least, define the Output Fields. Again, click Add as described above:

Image

Provide a short Field [Output] name and a more meaningful Friendly Name. As you will want to save those data, select Update RRD File. Create to see:

Image

Click Save and you're done.

Create a Data Template to tell cacti how to store data
Now you want to tell cacti, how to store the data retrieved from this script. Please go to Data Templates and click Add. You should see:

Image

Fill in the Data Templates Name with a reasonable text. This name will be used to find this Template among others. Then, please fill in the Data Source Name. This
is the name given to the host-specific Data Source. The variable |host_description| is taken from the actual Device. This is to distinguish data sources for different devices. The Data Input Method is a DropDown containing all known scripts and the like. Select the Data Input Method you just created. The Associated RRA's is filled by default. At the moment there's no need to change this. The lower part of the screen looks like:

Image

The Internal Data Source Name may be defined at your wish. There's no need to use the same name as the Output Field of the Data Input Method, but it may look nicer.
Click create to see:

Image

Notice the new DropDown Output Field. As there is only one Output Field defined by our Data Input Method, you'll see only this. Here's how to connect the Data
Source Name (used in the rrd file) to the Output Field of the Script. Click Save and you're done.

Create a Graph Template to tell cacti how to present the data
Now you want to tell cacti, how to present the data retrieved from this script. Please go to Graph Templates and click Add. You should see:

Image

Fill in Name and Title. The variable |host_description| will again be filled from the Device's definition when generating the Graph. Keep the rest as is and Create. See:

Image

Now click Add to select the first item to be shown on the Graphs:

Image
Select the correct Data Source from the DropDown, fill in a color of your liking and select AREA as a Graph Item Type. You want to fill in a Text Format that will be shown underneath the Graph as a legend. Again, Create:

Image

Notice, that not only an entry was made under Graph Template Items, but under Graph Item Inputs as well. Don't bother with that now. Lets fill some more nice legends, see:

Image

Notice, that the Data Source is filled in automagically. Select LEGEND as Graph Item Type (it is not really a Graph Item Type in rrdtool-speak, but a nice time-saver), and click Create to see:

Image

Wow! Three items filled with one action! You may want to define a Vertical Label at the very bottom of the screen and Save.

Apply the Graph Template to your Device
Now go to the Devices and select the one of your choice. See the Associated Graph Templates in the middle of this page:

Image

Select your newly created Graph template from the Add Graph Template DropDown. Click Add to see:

Image

The Template is added and shown as Not Being Graphed. On the top of the page you'll find the Create Graphs for this Host link. Click this to see:

Image

Check the box that belongs to the new template and Create. See the results:

Image

This will automatically
  • create the needed Graph Description from the Graph Template. As you may notice from the success message, this Graph takes the hosts name in it: router - Test ping (router is the hosts name of this example).
  • create the needed Data Sources Description from the Data Template. Again, you will find the Hosts name replaced for |host_description|
  • create the needed rrd file with definitions from the Data Template. The name of this file is derived from the Host and the Data Template in conjunction with an auto-incrementing number.
  • create an entry to the poller_table to instruct cacti to gather data on each polling cycle
You'll have to wait at least for two polling cycles to find data in the Graph. Find your Graph by going to Graph Management, filtering for your host and selecting the appropriate Graph (there are other methods as well). This may look like:

Image

Data Query
What is a Data Query? Here's the text from cacti's website (Chapter 10. Data Queries):
Quote:
Data queries are not a replacement for data input methods in Cacti. Instead they provide an easy way to query, or list data based upon an index, making the data easier to graph. The most common use of a data query within Cacti is to retrieve a list of network interfaces via SNMP. .... While listing network interfaces is a common use for data queries, they also have other uses such as listing partitions, processors, or even cards in a router.

Quote:
One requirement for any data query in Cacti, is that it has some unique value that defines each row in the list. This concept follows that of a 'primary key' in SQL, and makes sure that each row in the list can be uniquely referenced. Examples of these index values are 'ifIndex' for SNMP network interfaces or the device name
for partitions.

Quote:
There are two types of data queries that you will see referred to throughout Cacti. They are script queries and SNMP queries. Script and SNMP queries are virtually identical in their functionality and only differ in how they obtain their information. A
script query will call an external command or script and an SNMP query will make an SNMP call to retrieve a list of data.

Quote:
All data queries have two parts, the XML file and the definition within Cacti. An XML file must be created for each query, that defines where each piece of information is and how to retrieve it. This could be thought of as the actual query. The second part is
a definition within Cacti, which tells Cacti where to find the XML file and associates the data query with one or more graph templates.


SNMP-Type Data Queries (SNMP Queries)
For SNMP Queries, you won't need to create a data retrieval script. Cacti will use SNMP to retrieve information. But cacti will need additional information on how the indexed data is structured. Think about a table (a MIB table in this case); you'll have to tell cacti about the table structure. This is done by defining an XML file (see: SNMP Query XML Syntax for all details). Basically, you have to define the index to tell cacti about the number of rows and about their unique index. This index is later used to access each rows data. Furthermore, you may define columns, that serve as descriptive fields to be shown in the selection table. The XML file knows them as
Code:
<direction>input</direction>
At last, you will have to define those fields, that will be queried for the readings, e.g. ifInOctets, ifOutOctets, ifInErrors, ... The XML file knows them as
Code:
<direction>output</direction>
Lets have an example: standard Interface MIB with the corresponding part of the <path_cacti>/resources/snmp_queries/interfaces.xml file are displayed using the following table:
Image
Image
Image
and see the corresponding table structure when defining New Graphs for that device (my laptop):

Image

Now you can map
  • Index: IF-MIB::ifIndex
  • Status: IF-MIB::ifOperStatus
  • Description: IF-MIB::ifDescr
  • Type: IF-MIB::ifType
  • Speed: IF-MIB::ifSpeed

All those are <direction>input</direction> Parameters. They serve as descriptive information to each row to help you identify the proper interface to use. Those parameters of <direction>output</direction> can be compared to output parameters of a script (see ping.pl script above). These are the readings from the device. By selecting the appropriate row (the one greyed out had been selected by me), you tell cacti to retrieve data from the interface defined by this index. But how does cacti know, what output parameters it shall retrieve? See the Select a Graph type DropDown. It specifies a Graph Template defined for this Data Query. The Graph Template in turn references a Data Template which incorporates the needed output parameters as Data Sources. This works quite the same way as defined for a Data Input Method.

To sum up: the SNMP XML file is somehow a replacement for the Data Input Method described above to be used on indexed values. It tells cacti, what data it should retrieve (direction: output). To help you identifying the relevant indexes, the XML defines descriptive parameters (direction: input) to be displayed in the selection table.
A walkthrough for this is given now. It is based on the already supplied interfaces.xml XML file.

Create a Data Query to tell cacti how to retrieve data
Go to Data Queries and click Add to see:

Image

Here, we are using the already existing interface.xml file. Select Get SNMP Data (Indexed) as Data Input Method. Create to see:

Image

See, that cacti found the XML file. Don't bother with the Associated Graph Templates at the moment. The success message does not include checking of the XML file's content. Not lets proceed to the next definitions.

Create a Data Template to tell cacti how to store data
This is the exact copy of the definitions made above. So I do not repeat everything here. Data Input Method must be selected as Get SNMP Data (Indexed). As this data source is a COUNTER type, select this as the Data Source Type. But after saving the new Data Source definition, you may want to define a second Data Source to the same Data Template. To do so, select New from the Data Source Item heading to see:

Image

The name of the Data Source (ifOutOctets) is not replaced in the Tab until you save your work. By default, Maximum Value is set to 100. This is way too low for an interface. All readings above this value will be stored as NaN by rrdtool. To avoid this, set to 0 (no clipping) or to a reasonable value (e.g. interface speed). Don't forget to specify COUNTER! You will have noticed, that the name of the data source does not match the Name in the interface.xml. Don't worry, the solution to this is given later on.
Before leaving, pay attention to the bottom of the page:
Image

This is specific to indexed SNMP Queries. You will have to check the last
three items to make indexing work. All other items should be left alone, there values will be taken from the appropriate device definitions. Now Save and you're done with this step.

Create a Graph Template to tell cacti how to present the data
Now you want to tell cacti, how to present the data retrieved from SNMP Query. Again, this is done by merely copying the procedure described above. When selecting the Data Source, pay attention to select from the just defined data sources.
The next step is new and applies only to Data Queries:

Add Graph Template to the Data Query
Now it's time to re-visit our Data Query. Remember the Associated Graph Template we've left alone in the very first step? Now it will get a meaning. Go to Data Queries and select our new one. Then Add a new Associated Graph Template:

Image

Give it a Name and select the generated Graph Template. Create.
Image

Select the correct Data Source, pay attention to checking the checkboxes of each row. Apply a name to the Data Template and a title to the Graph Template. Use cacti variables as defined in [url=http://www.cacti.net/downloads/docs/html/variables.html#GRAPH_VARIABLES">Chapter
15. Variables - Data Query Fields[/url]. You may use all XML fields defined as <direction>input</direction>; in this example the fields <ifName> and <ifDescr> of the
interface.xml were used. Add those Suggested Values. They will be used to distinguish Data Sources and Graphs for the same device; without this they all would carry the same name. At last: Save:

Image

Apply the Data Query to your Device
Now go to your Device to add the Associated Data Query:

Image

Click Add and then Create Graphs for this Host to see:
Image

Now select the wanted interface and Create to generate the Traffic Graph. As long as there's only one Associated Graph Template for that Data Query, here will be now Select a Graph Type DropDown.

Summing up
To be more precise, cacti's tasks sum up as following:
Image
You'll notice the association of Graph Templates to the Data Query as a last step. And a new theme has popped up, the Host Template. This one is for grouping Graph Templates and Data Queries with Associated Graph Templates together as a single Host Template. You may associate each Host to one of those Host Templates. This will ease the burden of associating endless lists of Graph Templates to dozens of hosts.

I appreciate any feedback to improve this document. You may find this as a PDF here
Reinhard

_________________
Official Cacti Documentation
Official Debugging Help
Central Plugin Repository
Central Templates Repository


Last edited by gandalf on Wed Jun 18, 2008 1:29 pm, edited 9 times in total.

Top
 Profile  
 
 Post subject:
PostPosted: Mon Jan 09, 2006 3:27 am 
Offline
Cacti Guru User
User avatar

Joined: Fri Sep 19, 2003 8:36 am
Posts: 2326
Location: Sophia-Antipolis, France
Hi lvm

many thanks for this excellent howto !

However, the PDF shows only a blank page; could you please upload it again ? Thanks in advance.

_________________

HOWTOs :
Templates :


Top
 Profile  
 
 Post subject:
PostPosted: Mon Jan 09, 2006 10:17 am 
Offline
Cacti User

Joined: Mon Nov 29, 2004 11:43 am
Posts: 53
WoW!!!!

I wanna thanks you so much for this works!!!!

I'm sure it will helps everybody here...


Top
 Profile  
 
 Post subject:
PostPosted: Mon Jan 09, 2006 12:07 pm 
Offline
Developer
User avatar

Joined: Thu Dec 02, 2004 2:46 am
Posts: 22376
Location: Muenster, Germany
fmangeant wrote:
Hi lvm

many thanks for this excellent howto !

However, the PDF shows only a blank page; could you please upload it again ? Thanks in advance.
Thank you for the hint. I'll correct that asap
Reinhard

_________________
Official Cacti Documentation
Official Debugging Help
Central Plugin Repository
Central Templates Repository


Top
 Profile  
 
 Post subject:
PostPosted: Mon Jan 09, 2006 9:49 pm 
Offline
Developer/Forum Admin
User avatar

Joined: Mon Nov 17, 2003 6:35 pm
Posts: 6016
Location: Michigan, USA
Mr. Lvm,

We need to talk about you and cacti. :)

_________________
Tony Roman
Experience is what causes a person to make new mistakes instead of old ones.
There are only 3 way to complete a project: Good, Fast or Cheap, pick two.
With age comes wisdom, what you choose to do with it determines whether or not you are wise.


Top
 Profile  
 
 Post subject:
PostPosted: Tue Jan 10, 2006 12:25 pm 
Offline
Developer
User avatar

Joined: Thu Dec 02, 2004 2:46 am
Posts: 22376
Location: Muenster, Germany
rony wrote:
Mr. Lvm,

We need to talk about you and cacti. :)
Always at your service :wink:
Reinhard

_________________
Official Cacti Documentation
Official Debugging Help
Central Plugin Repository
Central Templates Repository


Top
 Profile  
 
 Post subject:
PostPosted: Tue Jan 10, 2006 1:12 pm 
Offline
Developer
User avatar

Joined: Thu Dec 02, 2004 2:46 am
Posts: 22376
Location: Muenster, Germany
fmangeant wrote:
Hi lvm

many thanks for this excellent howto !

However, the PDF shows only a blank page; could you please upload it again ? Thanks in advance.
Issue should be corrected by now. Was a problem with OpenOffice 1.1 PDF export. Fixed in V2.0
Reinhard

_________________
Official Cacti Documentation
Official Debugging Help
Central Plugin Repository
Central Templates Repository


Top
 Profile  
 
 Post subject: this is OK theoretically, but how about a real example?
PostPosted: Sat Jan 21, 2006 9:54 pm 
Offline
Cacti User

Joined: Sat Jul 30, 2005 2:15 pm
Posts: 148
Using an existing XML files (interfaces.xml) is a bit of a cheat, don't you think? If the purpose of this is to show someone how to create a new data source for monitoring, it would be more informative if it didn't use any existing xml files.

I am trying to do some things that are not included in the stock cacti install and learning how best to do that would be a lot more useful.


Top
 Profile  
 
 Post subject:
PostPosted: Sun Jan 22, 2006 6:05 am 
Offline
Developer
User avatar

Joined: Thu Dec 02, 2004 2:46 am
Posts: 22376
Location: Muenster, Germany
I do agree partly. I was thinking about this very issue before writing this howto. As it shall only demonstrate the principles, it should not matter, what ressources are taken as an example. The advantage of the interface.xml is, that each installation has this bit. If I would create something from scratch (which I have done already), some users _may_ be able to implement and see it working, some others _might_ not.
And I did not target the creation of a new XML! Nor to explain all the options to be used in those XML definitions. I did merely want to show how all those resources are interconnected and how to _use_ them. The XML bit is somewhat more complex; perhaps this would make for another howto? Do you have any ideas for that? There are LOTS of possible solutions to think about: SNMP Queries versus Script Queries; all those REGEXP stuff and the like. But is there any _generic_ target example to explain this? Something everyone can do himself?
Again, if you have any ideas on this, I will think about implementing them as long as you do not want to do this yourself.
Reinhard
PS: I changed the title of this topic to show the intention more clearly

_________________
Official Cacti Documentation
Official Debugging Help
Central Plugin Repository
Central Templates Repository


Top
 Profile  
 
 Post subject: OK, now what? ;-)
PostPosted: Sun Jan 22, 2006 12:26 pm 
Offline
Cacti User

Joined: Sat Jul 30, 2005 2:15 pm
Posts: 148
I think cacti was more extensible when you didn't have to create a backing XML file (maybe some will say that was never the case, but I made some things work in earlier versions that I can't do now: I don't think it's early onset senility).

Even if I have an XML file from someone else's attempts, there are still some gaps in the process.

It would be a big help if you (or someone) could step through how to go from the data revealed by an snmpget and a graphic representation. Maybe it's easier that it looks, but as I mentioned, I have an XML file to start with and still seem to be missing something(s).

Interested?


Top
 Profile  
 
 Post subject: Re: OK, now what? ;-)
PostPosted: Sun Jan 22, 2006 2:30 pm 
Offline
Developer
User avatar

Joined: Thu Dec 02, 2004 2:46 am
Posts: 22376
Location: Muenster, Germany
paulbeard206 wrote:
It would be a big help if you (or someone) could step through how to go from the data revealed by an snmpget and a graphic representation. Maybe it's easier that it looks, but as I mentioned, I have an XML file to start with and still seem to be missing something(s).

Interested?
Of course! Please post your snmpwalk (perhaps walking an SNMP MIB table) as a NEW topic. Lets see, what we get. Your approach is interesting as a howto, if the starting point is nothing more than an snmpwalk ...
So lets walk through step by step
- snmpwalk output
- define SNMP XML file from this point
- define Data Query, Data Template and Graph Template is just as told above, but we may add this bit for convenience.
As to the second point, I must admit, that my knowledge of cacti's XML syntax is quite limited. There are that many options, I did not get through all of them yet. But this may be a good starting point for a complete howto...
Of course, there would be more ... Script Queries ...
Reinhard

_________________
Official Cacti Documentation
Official Debugging Help
Central Plugin Repository
Central Templates Repository


Top
 Profile  
 
 Post subject: OK, one new topic coming up
PostPosted: Sun Jan 22, 2006 3:30 pm 
Offline
Cacti User

Joined: Sat Jul 30, 2005 2:15 pm
Posts: 148
You're being a good sport about this: here's the topic with what I hope is a realistic definition of the problem.

http://forums.cacti.net/viewtopic.php?p=54649#54649


Top
 Profile  
 
 Post subject:
PostPosted: Tue Jan 24, 2006 8:45 am 
Offline
Cacti User

Joined: Mon Nov 24, 2003 10:22 am
Posts: 91
Location: Hungary
Hi,

Please tell me the purpose of the list in
Data Queries > SNMP - Interface Statistics > In/Out bits > Suggested Values like: "|host_description| - Traffic - |query_ifName|"
"|host_description| - Traffic - |query_ifIP| (|query_ifDescr|)"
I know that this values comes from snmp queries but which one will be the graph title if there is more than one title form in a list. The first one? I mean the top on the list?

One more thing.
How graph template names work? :)
If I set a |host_description| - |query_ifAlias| title on a graph template and there is no description on the interface the title gets the first alternative title form from the list I mentioned above? (Data Queries > SNMP - Interface Statistics > In/Out bits > Suggested Values Graph Template)

Thx

_________________
Sini


Top
 Profile  
 
 Post subject:
PostPosted: Tue Jan 24, 2006 1:37 pm 
Offline
Developer
User avatar

Joined: Thu Dec 02, 2004 2:46 am
Posts: 22376
Location: Muenster, Germany
sini wrote:
Hi,

Please tell me the purpose of the list in
Data Queries > SNMP - Interface Statistics > In/Out bits > Suggested Values like: "|host_description| - Traffic - |query_ifName|"
"|host_description| - Traffic - |query_ifIP| (|query_ifDescr|)"
I know that this values comes from snmp queries but which one will be the graph title if there is more than one title form in a list. The first one? I mean the top on the list?
Yep. You may change the sequence by means of the blue arrows. If cacti find the first match, it takes this.

Quote:
One more thing.
How graph template names work? :)
If I set a |host_description| - |query_ifAlias| title on a graph template and there is no description on the interface the title gets the first alternative title form from the list I mentioned above? (Data Queries > SNMP - Interface Statistics > In/Out bits > Suggested Values Graph Template)

Thx
See the Interface - Traffic Graph Templates for an example. You have to select Use Per-Graph Value (Ignore this Value) to make the title stuff work. If you uncheck, the title is as defined by the Graph Template itself. If you check, the title is as defined by the Data Query Suggested Name. You may Reapply Suggested Names from the Graph Management screen
Reinhard

_________________
Official Cacti Documentation
Official Debugging Help
Central Plugin Repository
Central Templates Repository


Top
 Profile  
 
 Post subject:
PostPosted: Wed Jan 25, 2006 9:44 am 
Offline
Cacti User

Joined: Mon Nov 24, 2003 10:22 am
Posts: 91
Location: Hungary
Thanks Reinhard!

As I thougth but now I sure about it.
And also many thanks for the nice howto. It is very handy!

_________________
Sini


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 58 posts ]  Go to page 1, 2, 3, 4  Next

All times are UTC - 5 hours


Who is online

Users browsing this forum: No registered users and 2 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  

Protected by Anti-Spam ACP Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group