Cacti (home)ForumsDocumentation
Cacti: offical forums and support
It is currently Thu Jun 20, 2019 8:13 am

All times are UTC - 5 hours




Post new topic Reply to topic  [ 68 posts ]  Go to page 1, 2, 3, 4, 5  Next
Author Message
 Post subject: [HOWTO] RESIZE existing RRAs of existing RRDs
PostPosted: Sun Jan 15, 2006 3:22 pm 
Offline
Developer
User avatar

Joined: Thu Dec 02, 2004 2:46 am
Posts: 22376
Location: Muenster, Germany
Please find current version at http://docs.cacti.net


Preface
This tutorial is based on rrdtool, written by Tobias Oetiker. It is written by Reinhard Scheck to help you figuring out, how to resize an existing RRA of a given rrd file. This tutorial is based on Howto RESIZE existing RRAs of existing RRDs and extends this to some kind. Hope, this one will work even for windows users.

Be warned!
BACKUP ALL YOUR RRDs!
There's a good chance, that you will destroy all of your rrd files. I'm not joking! At the time of writing, rrdtool 1.2.12 is stable. Pay attention to older rrdtool-1.2.x version as they contain a bug when resizing rrd files created by rrdtool-1.0.x (see above reference for more).
SO BACKUP ALL YOUR RRDs! And check for sufficient file space!
As always: Use this information at your own risk.

Here we go!
As an attachment to the forum entry you will find a perl script resize.pl. It is necessary, to customize the /path/to/the/rrd/binary, e.g /usr/bin/rrdtool.

Help!
Put resize.pl wherever you want. There's no need to put it into the rrd working directory. But you will need some scratch space here for all rrds to be resized (due to the way rrdtool resize works). The user that runs this script must have
  • write permissions to the current directory used for scratch
  • read permissions on the original rrds to be resized
  • write permissions to the target directory to store the resized rrds in
The script does not care about space provided. To get help, simply type
Code:
perl resize.pl -h
you will receive
Code:
resize.pl Version 0.43 - resize an existing rrd
        Usage:    resize.pl
                                -f <filemask>
                                -r <rra> | -s <actual row size>
                                -o <output dir>
                                -g <growth>
                                -i
                           [-d <debug>]
        Requires: Getopt::Std, File::Basename, File::stat, File::Copy, File::KGlob, RRDp
        Author:   Reinhard Scheck
        Date:     2006-01-15
        Options:
                  -f, filemask of the source rrds
                  -r, rra to be changed (first rra denotes as -r 0)
                  -s, take only rra's with exactly that actual row size
                  -o, output directory for resized rrds
                  -g, growth (number of data points to be ADDED to those already defined)
                  -i, invoke rrdtool info instead of resizing
                  -d, debug level (0=standard, 1=function trace, 2=verbose)
                  -h, usage and options (this help)

        -s or -r must be given. -s will override -r option
        No parameter validation done. Hope you know what you're going to do!


Dry run
You may want to have a look at your rrds before resizing them. Specially for the required parameter -r (denoting the rra to be resized), you will want to have a look at those rras, that are defined in the rrd in question. Example (linefeeds only for ease of reading):
Code:
perl resize.pl -f "/var/www/html/cacti/rra/localhost_uptime_57.rrd" /
            -r 0 /
            -o /var/www/html/cacti/rra/resized/
            -g 8000 /
            -i
will result in:
Code:
-- RRDTOOL INFO localhost_uptime_57.rrd ...
ds[uptime].type = "GAUGE"
rra[0].cf = "AVERAGE"
rra[0].rows = 600
rra[1].cf = "AVERAGE"
rra[1].rows = 700
rra[2].cf = "AVERAGE"
rra[2].rows = 775
rra[3].cf = "AVERAGE"
rra[3].rows = 797
rra[4].cf = "MIN"
rra[4].rows = 600
rra[5].cf = "MIN"
rra[5].rows = 700
rra[6].cf = "MIN"
rra[6].rows = 775
rra[7].cf = "MIN"
rra[7].rows = 797
rra[8].cf = "MAX"
rra[8].rows = 600
rra[9].cf = "MAX"
rra[9].rows = 700
rra[10].cf = "MAX"
rra[10].rows = 775
rra[11].cf = "MAX"
rra[11].rows = 797
rra[12].cf = "LAST"
rra[12].rows = 600
rra[13].cf = "LAST"
rra[13].rows = 700
rra[14].cf = "LAST"
rra[14].rows = 775
rra[15].cf = "LAST"
rra[15].rows = 797

You may notice
  • a single data source (uptime)
  • four consolidation functions (AVERAGE, MIN, MAX, LAST)
  • four rra's for each of the consolidation functions
Of course, you may also enter a partly qualified dataset name. But it makes sense to take only those rrd's, that belong to the same datasource (e.g. with the same rrd file structure).

Resizing a single RRA of a single RRD
For ease of use, you may simply omit the trailing parameter -i. But pay attention to the parameter -r! In this example, only the first RRA of the consolidation function AVERAGE shall be resized. It depends on your needs, whether this will result in a correct RRD!
Code:
perl resize.pl -f "/var/www/html/cacti/rra/localhost_uptime_57.rrd" /
            -r 0 /
            -o /var/www/html/cacti/rra/resized/
            -g 8000
The output will look like:
Code:
-- RRDTOOL RESIZE localhost_uptime_57.rrd RRA (0) growing 8000.. (95328).. RRA#0.. (159328).. Done.
The first parenthesis contain the file size before resizing, the second one after resizing.

Resizing multiple RRA of a single RRD
Simply enter all RRAs to be resized in quotes:
Code:
perl resize.pl -f "/var/www/html/cacti/rra/localhost_uptime_57.rrd" /
            -r \u201c0 4 8 12\u201d /
            -o /var/www/html/cacti/rra/resized/
            -g 8000
to result in
Code:
-- RRDTOOL RESIZE localhost_uptime_57.rrd RRA (0 4 8 12) growing 8000.. (95328).. RRA#0#4#8#12.. (351328).. Done.


Resizing multiple RRAs of multiple RRDs
Please enter all RRAs to be resized in quotes and partly qualify alll RRDs:
Code:
perl resize.pl -f "/var/www/html/cacti/rra/*_uptime_*.rrd" /
            -r \u201c0 4 8 12\u201d /
            -o /var/www/html/cacti/rra/resized/
            -g 8000
to result in
Code:
-- RRDTOOL RESIZE router_uptime_59.rrd RRA (0 4 8 12) growing 8000.. (95328).. RRA#0#4#8#12.. (351328).. Done.
-- RRDTOOL RESIZE gandalf_uptime_58.rrd RRA (0 4 8 12) growing 8000.. (95328).. RRA#0#4#8#12.. (351328).. Done.
-- RRDTOOL RESIZE localhost_uptime_57.rrd RRA (0 4 8 12) growing 8000.. (95328).. RRA#0#4#8#12.. (351328).. Done.


Resizing all RRAs of a given row size
This is a new feature of this version. Use the parameter -s to specify the rowsize of the rra's you want to change. This parameter overrides the -r parameter, cause all relevant rra's will be calculated from the current rrd definition. This is useful if you're working on a list of files with different rrd structure (e.g. different Data Templates)
Code:
perl resize.pl -g 8000 -f "/var/www/html/workspace/branch/rra/gandalf*.rrd" /
            -s 600
            -o /var/www/html/cacti/rra/resized/
            -g 8000
to result in
Code:
... removed ...
-- RRDTOOL RESIZE gandalf_cpu_system_9.rrd RRA (0 4 ) growing 8000..47836.. RRA#0#4..175840.. Done.
-- RRDTOOL RESIZE gandalf_cpu_user_10.rrd RRA (0 4 ) growing 8000..47836.. RRA#0#4..175840.. Done.
-- RRDTOOL RESIZE gandalf_errors_in_18.rrd RRA (0 4 ) growing 8000..188308.. RRA#0#4..700312.. Done.
... removed ...
-- RRDTOOL RESIZE gandalf_unicast_in_20.rrd RRA (0 4 ) growing 8000..94660.. RRA#0#4..350664.. Done.
... removed ...
-- RRDTOOL RESIZE gandalf_uptime_58.rrd RRA (0 4 8 12 ) growing 8000..95328.. RRA#0#4#8#12..351328.. Done.
-- RRDTOOL RESIZE gandalf_users_89.rrd RRA (0 4 8 12 ) growing 8000..95328.. RRA#0#4#8#12..351328.. Done.
user time: 0.34 system time: 1.93 real time: 7.16
Please notice the last line of output, which reports the rrdtool runtime. If -s is given so that no rowsize of any rra will match, the corresponding rrd file is skipped:
Code:
perl resize.pl -g 8000 -f "/var/www/html/workspace/branch/rra/gandalf*.rrd" -s 601 -o new-resized
user time: 0.01 system time: 0.02 real time: 0.23


Something to keep in mind
Be warned!
You may even enter -o to resolve to the current RRD directory. This will result in overwriting your existing RRDs. YOU DON'T WANT TO DO THAT. Always look at the output after resizing. Try to generate graphs from them. Verify, that everything runs fine.
BACKUP YOUR ORIGINAL RRDs.
I appreciate any feedback to improve this document.
Reinhard Scheck aka lvm

Edit (Jan, 21th 2006): As I noticed recently, the module File::KGlob isn't really necessary. So if you get trouble with installing this really old module, simply delete this one. I checked this with some 1000 rrds without any problem


Attachments:
File comment: the perl script
resize.pl.gz [2.23 KiB]
Downloaded 3280 times

_________________
Official Cacti Documentation
Official Debugging Help
Central Plugin Repository
Central Templates Repository


Last edited by gandalf on Wed Jun 18, 2008 1:31 pm, edited 2 times in total.
Top
 Profile  
 
 Post subject:
PostPosted: Wed Feb 08, 2006 7:05 pm 
Offline
Cacti User

Joined: Sun Dec 04, 2005 10:59 pm
Posts: 55
Once I have resized my RRAs, how do I tell cacti not to consolidate?

Also, I can't seem to find RRDp.pm anywhere.


Top
 Profile  
 
 Post subject:
PostPosted: Thu Feb 09, 2006 3:02 pm 
Offline
Developer
User avatar

Joined: Thu Dec 02, 2004 2:46 am
Posts: 22376
Location: Muenster, Germany
evilzardoz wrote:
Once I have resized my RRAs, how do I tell cacti not to consolidate?
You can't. First, consolidation is a feature of rrdtool, not cacti. Second, as long as you keep rra's with steps>1, this will ask rrdtool to do consolidation. There's nothing bad in doing so. This is a very efficient way of keeping the important part of data. See my first few signatures to get an idea how to interpret MAXIMUM or AVERAGE values after consolidation.

Quote:
Also, I can't seem to find RRDp.pm anywhere.
This is part of the rrdtool rpms related to perl. When compiling on your own, there are some switches to set for perl modules to be generated.
Reinhard

_________________
Official Cacti Documentation
Official Debugging Help
Central Plugin Repository
Central Templates Repository


Top
 Profile  
 
 Post subject:
PostPosted: Sun Feb 12, 2006 8:19 pm 
Offline
Cacti User

Joined: Sun Dec 04, 2005 10:59 pm
Posts: 55
lvm - thanks for the help so far. We want to keep more detailed information for historical reasons which is why consolidation, while a good idea, isn't quite what we're looking for.

I've found some extra packages for RRD-perl stuff and they look like what I might need so I'll give those a shot.


Top
 Profile  
 
 Post subject:
PostPosted: Mon Feb 13, 2006 1:04 pm 
Offline
Developer
User avatar

Joined: Thu Dec 02, 2004 2:46 am
Posts: 22376
Location: Muenster, Germany
Of course you can "turn off consolidation" by only defining a single rra, consisting of LOTS of data points. That would be HUGE,of course. But for some very few data templates, I did this. Took me SOME discussions with those interpreting the results; this is a bit "funny". Try this, and you'll knwo what I mean. This effect will surely pertain to all other solutions (what do you think comes out, when you try to graph some 1000 data points when there are only about 500 pixels on the screen ...). Perhaps this would be a solution for intervals of 1h or so ...
Reinhard

_________________
Official Cacti Documentation
Official Debugging Help
Central Plugin Repository
Central Templates Repository


Top
 Profile  
 
 Post subject:
PostPosted: Mon Feb 13, 2006 10:21 pm 
Offline
Cacti User

Joined: Sun Dec 04, 2005 10:59 pm
Posts: 55
I still want the daily, monthly, yearly graphs.. but I don't want to loose granularity over time i.e. if I want to see what our network looked like three years ago.

Is this a bad idea?


Top
 Profile  
 
 Post subject:
PostPosted: Tue Feb 14, 2006 2:24 pm 
Offline
Developer
User avatar

Joined: Thu Dec 02, 2004 2:46 am
Posts: 22376
Location: Muenster, Germany
evilzardoz wrote:
I still want the daily, monthly, yearly graphs.. but I don't want to loose granularity over time i.e. if I want to see what our network looked like three years ago.

Is this a bad idea?
Not bad; not yet solved only :cry:
Some ideas but no solution :(
Reinhard

_________________
Official Cacti Documentation
Official Debugging Help
Central Plugin Repository
Central Templates Repository


Top
 Profile  
 
 Post subject:
PostPosted: Tue Feb 21, 2006 10:15 am 
Offline
Developer
User avatar

Joined: Thu Dec 02, 2004 2:46 am
Posts: 22376
Location: Muenster, Germany
Please define 4 new RRAs as shown (Xff is always 0.5) in picture 1. In your Data Templates, you will reference them as shown in picture 2 when creating the Data Source.
Afterwards, when the Data Sources are created (not earlier) you may change this to settings like picture 3. Now, when graphing, the commonly known 4 timespan graphs are shown.
If you add all 4 RRAs when creating the data sources, you'll get an rrd file with 4 rra's while you only want to have one. So follow the instructions carefully (perhaps a little code patch would supress generating weird rra definitions like those I mentioned).
In fact, I'd prefer to seperate the definition of graphing timespans from rra definitions. In "reality", the timespans to be defined for cacti's rra are not needed for rrdtool.
Reinhard


Attachments:
File comment: Define this new rra's
(Those ZLVM ... thingies)

No-Consolidation-RRAs-1.jpg
No-Consolidation-RRAs-1.jpg [ 105.96 KiB | Viewed 50433 times ]
File comment: Fisrt, check only the last of the 4 rra's
THIS IS IMPORTANT!

No-Consolidation-RRAs-2.jpg
No-Consolidation-RRAs-2.jpg [ 49.09 KiB | Viewed 50438 times ]
File comment: At last: check all 4 rra's
No-Consolidation-RRAs-3.jpg
No-Consolidation-RRAs-3.jpg [ 63.39 KiB | Viewed 50438 times ]

_________________
Official Cacti Documentation
Official Debugging Help
Central Plugin Repository
Central Templates Repository
Top
 Profile  
 
 Post subject: Some of my data sources do this automatically
PostPosted: Mon Feb 27, 2006 10:48 pm 
Offline

Joined: Mon Jun 27, 2005 6:30 pm
Posts: 32
I only just noticed today, but some of my data sources maintain 5 minute resolution without me having done anything. It is seemingly random in that I have used the same graph and data template multiple times and in some instances it keeps all the data while for other devices it hasn't. See a couple of graphs as an example, these are taken today (28/02/2006) by zooming in on my yearly graph - notice the dates and you can see it is still data per 5 minutes.

I'm not too sure where to poke around and what to look for to work out why this is happening either, but I've checked the RRD file sizes and the ones with 5 minute resolution are ~1.5M or 3M versus ~140K. This seems to have happened for about 1500 of my 3200 data sources. The time these data sources were created isn't of any significance either by the looks of it as some of the ones that are maintaining resolution are fairly old.

Tim


Attachments:
File comment: Same time frame, no 5 minute resolution
cpu not 5 mins.png
cpu not 5 mins.png [ 3.82 KiB | Viewed 50360 times ]
File comment: Notice the 5 minute resolution
cpu 5 mins.png
cpu 5 mins.png [ 3.87 KiB | Viewed 50360 times ]
File comment: Notice the 5 minute resolution
arp 5 mins.png
arp 5 mins.png [ 4.17 KiB | Viewed 50360 times ]
Top
 Profile  
 
 Post subject:
PostPosted: Wed Mar 01, 2006 1:35 pm 
Offline
Developer
User avatar

Joined: Thu Dec 02, 2004 2:46 am
Posts: 22376
Location: Muenster, Germany
The size is a good indication for those rrd's who got wrong. If you're using pure cacti, I'd swear there is ABSOLUTELY NO way to change rrd definitions if the file has once been created.
So it must have occured at generation time OR was changed manually afterwards.
You may check your RRA definitions to see, whether there is actually any RRA defining 5 min intervals for infinity. Another source may be importing tenplates. In each import you'll find rra definitions that will overwrite your settings. cacti always reports "updated" whether it really changed existing rra definitions or not. To be SURE, you would have to check rra definitions EACH time after importing. No joke, this is a very useful check; you should really do this (well, it already occured to me and I claim to be a somewhat experienced cacti user).
And if you have this unwanted settings for lots of rrds, this will surely slow cactis poller performance down. Changing this is near to magic ...
Perhaps dumping, changing XML and reloading would do ...
Reinhard

_________________
Official Cacti Documentation
Official Debugging Help
Central Plugin Repository
Central Templates Repository


Top
 Profile  
 
 Post subject:
PostPosted: Wed Mar 01, 2006 5:01 pm 
Offline

Joined: Mon Jun 27, 2005 6:30 pm
Posts: 32
I haven't changed anything manually, so perhaps things got changed with template imports as you suggested. What RRA definitions should I be checking exactly? I've looked at the rra definitions for the graphs above that have 5 minute data and they all have the Daily, Weekly, Monthly, Yearly RRAs associated with them.

It is not an unwanted setting, I actually prefer to keep all the data there as it allows me to look at old data with 5 minute accuracy. The reason I posted in this thread was because I thought maybe something happened with my installation that inadvertently caused some of my graphs to maintain 5 minute accuracy which is what a few people seem to want also.

It also doesn't seem to slow the poller down at all, my poller stats are:
Quote:
03/02/2006 08:55:32 AM - SYSTEM STATS: Time:30.3416 Method:cactid Processes:1 Threads:6 Hosts:172 HostsPerProcess:172 DataSources:5819 RRDsProcessed:3072


This is running on a P4 2.8G, 512 M ram and it's got 40 Gig hard disk (of which only 6 or 7 Gig is being used currently) so I have plenty of space to store the larger rrd files.


Top
 Profile  
 
 Post subject: question...
PostPosted: Thu Mar 16, 2006 2:47 pm 
Offline
Cacti User

Joined: Mon Feb 10, 2003 12:00 pm
Posts: 53
Location: Southern / CA
Reinhard,

I have a question about your insructions below:
Quote:
Please define 4 new RRAs as shown (Xff is always 0.5) in picture 1. In your Data Templates, you will reference them as shown in picture 2 when creating the Data Source.
Afterwards, when the Data Sources are created (not earlier) you may change this to settings like picture 3. Now, when graphing, the commonly known 4 timespan graphs are shown.
If you add all 4 RRAs when creating the data sources, you'll get an rrd file with 4 rra's while you only want to have one. So follow the instructions carefully (perhaps a little code patch would supress generating weird rra definitions like those I mentioned).
In fact, I'd prefer to seperate the definition of graphing timespans from rra definitions. In "reality", the timespans to be defined for cacti's rra are not needed for rrdtool.


If I want to switch all my customer bandwidth graphs to 1 year, 5 minute accuracy and I create the "ZLVM" RRAs, will I have to do something special each time I want to create a new graph?

Specifically do I have to go in and disable all the RRAs but the "yearly" one?

I want to make this easy for my network engineers to handle when they turn up a new customer.

Thx!


Top
 Profile  
 
 Post subject: getting an error
PostPosted: Mon Mar 27, 2006 1:00 am 
Offline
Cacti User

Joined: Mon Feb 10, 2003 12:00 pm
Posts: 53
Location: Southern / CA
Code:
./resize.pl -f "/var/www/html/cacti/rra.bak/ana_traffic_out_795.rrd" -s 600 -g 114600 -o /var/www/html/cacti/rra.bak/resized/ -i

-- RRDTOOL INFO ana_traffic_out_795.rrd ...
open2: exec of /usr/bin/rrdtool - failed at /usr/local/rrdtool-1.2.12/lib/perl/5.8.5/RRDp.pm line 120


Then it just hangs...


Top
 Profile  
 
 Post subject: Re: question...
PostPosted: Mon Mar 27, 2006 11:16 am 
Offline
Developer
User avatar

Joined: Thu Dec 02, 2004 2:46 am
Posts: 22376
Location: Muenster, Germany
warenet wrote:
If I want to switch all my customer bandwidth graphs to 1 year, 5 minute accuracy and I create the "ZLVM" RRAs, will I have to do something special each time I want to create a new graph?
Yes, these RRAs are faked. AFAIK, they will break rrd file generation (but I don't remember exactly) or at least produce some unreasonable rrd. My suggestion is a REALLY NASTY WORKAROUND, no more.
Reinhard

_________________
Official Cacti Documentation
Official Debugging Help
Central Plugin Repository
Central Templates Repository


Top
 Profile  
 
 Post subject: Re: getting an error
PostPosted: Mon Mar 27, 2006 11:20 am 
Offline
Developer
User avatar

Joined: Thu Dec 02, 2004 2:46 am
Posts: 22376
Location: Muenster, Germany
warenet wrote:
Code:
./resize.pl -f "/var/www/html/cacti/rra.bak/ana_traffic_out_795.rrd" -s 600 -g 114600 -o /var/www/html/cacti/rra.bak/resized/ -i

-- RRDTOOL INFO ana_traffic_out_795.rrd ...
open2: exec of /usr/bin/rrdtool - failed at /usr/local/rrdtool-1.2.12/lib/perl/5.8.5/RRDp.pm line 120


Then it just hangs...
Please change the script
Code:
# --- initialization ----------------------------------------------------------
my $rrd_binary  = "/usr/bin/rrdtool";   # path to rrd binary, to be customized! <<<<<<<<<<<<<<<<<<<<
as stated in my first post
Reinhard

_________________
Official Cacti Documentation
Official Debugging Help
Central Plugin Repository
Central Templates Repository


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 68 posts ]  Go to page 1, 2, 3, 4, 5  Next

All times are UTC - 5 hours


Who is online

Users browsing this forum: No registered users and 2 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  

Protected by Anti-Spam ACP Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group