[HOWTO] Define very BIG rra "without data loss"

[gandalf]

Please find the most current version here
Reinhard
Preface
This tutorial is based on cacti http://cacti.net, written by Ian Berry, Larry Adams and Tony Roman. Cacti itself is based on rrdtool http://www.rrdtool.org, written by Tobias Oetiker. It is written by Reinhard Scheck to help you figuring out, how to configure cacti for use with a single round robin archive (rra) without using consolidation (e.g. without averaging out some data points).

Be warned!
You won't really do that! Why? One of the inherent features of rrd's is: they never grow in space. In other words: When creating a new rrd, it is allocated with all space needed. See rrd-beginners tutorial at http://people.ee.ethz.ch/~oetiker/webtools/rrdtool/tut/rrd-beginners.en.html. As usually, you may use the information given at your own risk.

Here we go!
Cacti's logic to generate rrd files works as follows:

1. create a device (the host that shall be queried)
   
2. create a graph for this host (using a graph template or a data query that refers to graph templates)
   
3. each graph template refers to a data template
   
4. each data template defines one or more data sources
   
5. each data template uses one or more round robin archives (rra)
   
6. each of the data sources uses the same set of rra's
   
   

This tutorial works the way back.

Defining a new round robin archive (rra)
For the following, lets assume you are logged in with admin permissions and use the “console” tab.
1.Go to Management -> Data Sources -> RRAs
2.Click Add to add a new rra

Now fill in the data as follows and SAVE:

Name: you may choose your own
Consolidation function: AVERAGE needed
X-Files Factor: always 0.5
Steps: 1 (that is the number of data points to use for consolidation, 1 says: no consolidation at all)
Rows: 115200 = 400 days with 24 hours and 12 data points per hour (= 5 min interval)
Timespan: used for displaying 33,053,184 seconds = about 382 days (taken from other cacti rra)

Define a new data template
For ease of use (yes, I'm lazy), please copy an existing template. Goto Data Templates, and check the box on the right of Interface – Traffic:

Then scroll to the bottom of the page, select Duplicate and Go.

You will be prompted for a new name of this template:

Of course, you may choose your own name here. Now it is time to modify this template:

1.You may change the name of the template
2.Select the just created RRA (Don't worry about the other RRAs in this list; they are needed for the next tutorial ...)
Please leave the rest as is; SAVE. Of course, you may define a new data template from scratch. The only thing to keep in mind is to select the appropriate RRA. The data template is now done.

Define a new graph template
Well, you will imagine what comes next. Again, I decided to copy the appropriate graph template. So goto Graph Templates and repeat the steps above for the template Interface – Traffic (bits/sec). It will look like this:

Please pay attention to the next steps! You will have to delete both Graph Item Inputs, as they refer to the “wrong” data source. Please select the red X to the right of Inbound Data Source as well as Outbound Data Source.
Then you will have to add the newly generated data sources. In order to do that, please select each item of the list of Graph Items, one after the other. This will look like:

As Data Source you will choose the appropriate data source you generated in the previous step. Don't forget to do this for each and every item of the Graph Item list. When you're done, scroll to the bottom of the Graph Template definition and SAVE.

Modify Data Query to add Graph Template
This example uses Interface – Traffic Graph Template. This is referenced by the Data Query SNMP – Interface Statistics. Now we're going to add the newly defined Graph Template to this very Data Query.
If you have chosen some other Graph Template, e.g. ucd/net – Load Average, you will skip this step.
The Data Query goes like this. Goto Data Queries and select SNMP – Interface Statistics.

Now Add to see this:

Define a new name for this Associated Graph and CREATE.

Finally: Create Graphs for this Host
Goto Devices and select your favorite device to see the rra in action. If you have modified the SNMP – Interface Statistics Data Query, you may immediately select Create Graphs for this Host to see the following:

Select the interface as you would have done for any Traffic Graph. Then Select a graph type from the dropdown list (of course our newly defined Graph Template!) and CREATE. As usually, you will have to wait at least two polling cycles to get the graph generated and filled with the first value. Don't be impatient! Let it run for awhile.
Under the Graphs tab you will notice something like

Well, this looks like usual, doesn't it? You may wonder about the Outbound traffic displayed negative. Well, this is a little CDEF but is of no matter here. And of course, for the first two days you will not notice anything unusual. This is because the default cacti rra configuration keeps all data points without consolidation for 600 intervals (about 2 days).
Some advice:
Please do not click onto the graph too fast. I had to wait some time (don't remember exactly) before clicking gave a result like the next one:

This is already a zoomed image. You will notice, that my personal laptop isn't online for the whole day .

Now, where is the trick?
At first, you may wonder, whether only this one graph will be displayed. This is, because only one single rra exists. And cacti associated the time interval of the graph with each rra. Only one rra defined gives only one image displayed.
But you may zoom in at any place and will reach down to the 5 min intervals. This is, what had to be proved (q.e.d as the old romans said).

Something to keep in mind
Space allocation with rrdtool
The space needed is calculated from

• the number of data sources needed (e.g. traffic in and traffic out form two data sources)
  
• the number of rra's needed (e.g. one archive for storing original data points, a second one to hold averaged data points for some weeks, a third for holding averaged data points for some months ...)
  
• the number of data points to be stored in each rra
  
• some header space

If you omit consolidation (that is: averaging out some data points), you won't “loose data”. But you will loose space!
Example: Store data every 300 seconds for a whole year. This leads to 12 (data points each hour) * 24 (hours per day) * 365 (days per year) data point (= 105120). Each data point holds 8 bytes, so the whole rrd will occupy about 840,960 Bytes (plus some header space) for each single data source.

A closer look to rrd file properties
Please have a look at the file sizes on my computer:

[BSOD2600]

Nice guides, you slated to do some FAQ writing?

[disorganizer]

nice faq!
one point:
when using another function than average the graphs dont work, as they try to access the consolidation function average.

so it would be best to leave the function as average.

[gandalf]

[gandalf]

For ease of use I've added another Graph Template similar to the first one.
This time, it includes 3 RRAs (you'll see them already in the examples above).
Reinhard

[dtpiller]

Thanks for posting that section about allocating space with rrdtool. We are planning to make an rra of 10min average in 1 year, 1hour average in 2 years and 30min average in 2years but before we start with that, we want to know how big will the rrd files be with this rra.
Your example is simple and direct, but i just want to be sure how to calculate the total data points if i use consolidation. Example rrd:

/var/www/html/cacti/rra/test_host_traffic_in_183.rrd \
--step 300 \
DS:traffic_in:COUNTER:600:0:100000000 \
DS:traffic_out:COUNTER:600:0:100000000 \
RRA:AVERAGE:0.5:6:35040 \
RRA:AVERAGE:0.5:2:105120 \
RRA:AVERAGE:0.5:12:17520 \
RRA:MAX:0.5:6:35040 \
RRA:MAX:0.5:2:105120 \
RRA:MAX:0.5:12:17520 \ --> do i calculate the total data points in this RRA by 12 steps*17520? (1 hour average in 2 years)

Help will be much appreciated!

[gandalf]

[schef4711]

Hello,

is there also an easy way to create a template with 95th Percentile in 5min averrage for monthly and yearly ??

Because in the SNMP Data Queries there will not be a Interface for that.

bye alex

[gandalf]

[rony]

If the 5 minute data exists for the selected time range, month, year, it will use the 5 minute data to calculate the 95th percentile. Otherwise, it will use the consolidated data for that range.

[thisted]

[gandalf]

[james0]

I followed the directions listed on this thread for creating a one year RRA. It seems like it works since I can zoom all the way in to 5 min intervals, but there is one thing I don't understand:

When I first look at the RRA, it shows a whole year, and the Average and MAX are X and Y. When I zoom in on today

[james0]

**hmm, got an SQL error when posting the above... Posting again outside of IE7*


I followed the directions listed on this thread for creating a one year RRA. It seems like it works since I can zoom all the way in to 5 min intervals, but there is one thing I don't understand:

When I first look at the RRA, it shows a whole year, and the Average and MAX are X and Y. When I zoom in on today

[james0]

*ok, this time without apostrophes*

I followed the directions listed on this thread for creating a one year RRA. It seems like it works since I can zoom all the way in to 5 min intervals, but there is one thing I don't understand:

When I first look at the RRA, it shows a whole year, and the Average and MAX are X and Y. When I zoom in on todays data(and todays data is the only data in the RRA so far), the average and max increase. Why is this?

I was hoping that after doing these things I could look at a one year graph and see a realistic average and max that has not been averaged down to nothing. Have I missed a step? or a concept?


Another example of this is:
I took a clean cacti install, and changed all of the default RRA's to use a step of 1, and then multiplied the Rows by the original step value. I realize that this may not seem like an effective use of disk space, but I want to be able to quickly look at what is happening today on the graph without zooming in, and disk space is not an issue for me.
What I ended up with was something with the same problem I first described. Todays graph shows accurate data, and the average and max are X and Y. The next graph, being the weekly graph has a lower avg and MAX. Monthly and yearly say NAN, but I can zoom in on today and then values do show up. Where is this averaging happening? I do not understand this behavior.


-James