Patch: "Structured RRD Path"

Addons for Cacti and discussion about those addons

Moderators: Moderators, Developers

Author
Message
User avatar
browniebraun
Developer
Posts: 788
Joined: Tue Jun 13, 2006 1:17 am
Location: Cologne, Germany

Patch: "Structured RRD Path"

#1 Post by browniebraun » Tue Jul 25, 2006 8:46 am

Hi CACTI community!

One disadvantage of the Cacti file structure is the flat file directory RRA, especially in great enviroments. For example we have 30.000 rrd's in one folder.
To overcome this, we decided to structure the directory in host_id/local_data_id.
A second issue was the naming of the rrd's with devicename_data_source_local_data_id when a hostname changed. The only needed identifier is the local_data_id. For human readability we plan a script to reverse lookup local_data_id to devicename.

Old structure of RRA folder:

/RRA/
......../labrt01_traffic_in_194.rrd

New structure of RRA folder:

/RRA/
..../$host_id/
................../$local_data_id.rrd

The patch is designed to give you the chance to switch between the old, flat file organization and a structured form with subfolders.
The screenshot displays the modified path setting checkbox.
The script (extended_RRA.php) make it possible to migrate a existing Cacti installation with all rrd files and database entries.
It's designed to run only from command line.

It's my first shot with PHP, so please be patient with me, and give me as much feedback as you can.

Please notice:
USE THIS PATCH AT YOUR OWN RISK and
PLEASE
READ THE README.TXT BEFORE


Best regards
BrownieBraun
Attachments
cacti.PNG
Modified setting entry
cacti.PNG (46.83 KiB) Viewed 25224 times
extended_RRA.tar
Patch, script and readme
(20 KiB) Downloaded 749 times

User avatar
rony
Developer/Forum Admin
Posts: 6016
Joined: Mon Nov 17, 2003 6:35 pm
Location: Michigan, USA
Contact:

#2 Post by rony » Tue Jul 25, 2006 9:15 am

Um... :)
[size=117][i][b]Tony Roman[/b][/i][/size]
[size=84][i]Experience is what causes a person to make new mistakes instead of old ones.[/i][/size]
[size=84][i]There are only 3 way to complete a project: Good, Fast or Cheap, pick two.[/i][/size]
[size=84][i]With age comes wisdom, what you choose to do with it determines whether or not you are wise.[/i][/size]

User avatar
gandalf
Developer
Posts: 22375
Joined: Thu Dec 02, 2004 2:46 am
Location: Muenster, Germany
Contact:

#3 Post by gandalf » Tue Jul 25, 2006 12:47 pm

Hey, not that bad ...
Lately, there were some requests for excatls this feature. Hope, they'll find this bit
Reinhard

User avatar
TheWitness
Developer
Posts: 14817
Joined: Tue May 14, 2002 5:08 pm
Location: MI, USA
Contact:

#4 Post by TheWitness » Tue Jul 25, 2006 9:12 pm

We had planned to forstall this feature till 0.9. But it's been on the request list for it seams like decades. Maybe we should do something about it.

TheWitness
True understanding begins only when we realize how little we truly understand...

Life is an adventure, let yours begin with Cacti!

Author of MacTrack, Boost, CLog, SpikeKill, Platform RTM, DSStats, maintainer of Spine, lot's of unpublished work and most of Cacti's bugs.
_________________
Official Cacti Documentation
GitHub Repository with Supported Plugins
Central Plugin Repository
Central Templates Repository


I'm still out there people. Getting excited for Cacti 1.2. I think it will be a great release.

User avatar
TheWitness
Developer
Posts: 14817
Joined: Tue May 14, 2002 5:08 pm
Location: MI, USA
Contact:

#5 Post by TheWitness » Tue Jul 25, 2006 9:14 pm

Oh, BTB, this is a core function, therefore, I would have difficulty supporting this unless it was a part of the "base" product. It is one clear feature that could make it difficult to support future upgrades for those who choose to implement it.

So, if you are reading this, implement at your own risk. I would hold off for a bit.

TheWitness
True understanding begins only when we realize how little we truly understand...

Life is an adventure, let yours begin with Cacti!

Author of MacTrack, Boost, CLog, SpikeKill, Platform RTM, DSStats, maintainer of Spine, lot's of unpublished work and most of Cacti's bugs.
_________________
Official Cacti Documentation
GitHub Repository with Supported Plugins
Central Plugin Repository
Central Templates Repository


I'm still out there people. Getting excited for Cacti 1.2. I think it will be a great release.

User avatar
fmangeant
Cacti Guru User
Posts: 2326
Joined: Fri Sep 19, 2003 8:36 am
Location: Sophia-Antipolis, France
Contact:

#6 Post by fmangeant » Wed Jul 26, 2006 9:06 am

TheWitness wrote:We had planned to forstall this feature till 0.9. But it's been on the request list for it seams like decades. Maybe we should do something about it.
Not decades :), but I've submitted a bug nearly 2 years ago : http://bugs.cacti.net/view.php?id=284

Believe me or not, I've changed <path_rra> for all of my ~ 10k data sources by hand...
And I still love Cacti !
[size=84]
[color=green]HOWTOs[/color] :
[list][*][url=http://forums.cacti.net/viewtopic.php?t=15353]Install and configure the Net-SNMP agent for Unix[/url]
[*][url=http://forums.cacti.net/viewtopic.php?t=26151]Install and configure the Net-SNMP agent for Windows[/url]
[*][url=http://forums.cacti.net/viewtopic.php?t=28175]Graph multiple servers using an SNMP proxy[/url][/list]
[color=green]Templates[/color] :
[list][*][url=http://forums.cacti.net/viewtopic.php?t=15412]Multiple CPU usage for Linux[/url]
[*][url=http://forums.cacti.net/viewtopic.php?p=125152]Memory & swap usage for Unix[/url][/list][/size]

User avatar
browniebraun
Developer
Posts: 788
Joined: Tue Jun 13, 2006 1:17 am
Location: Cologne, Germany

#7 Post by browniebraun » Wed Jul 26, 2006 10:44 am

Windows formatted readme file:
Attachments
reamde.tar
(10 KiB) Downloaded 523 times

zheka
Posts: 44
Joined: Tue May 23, 2006 11:52 am

#8 Post by zheka » Fri Jul 28, 2006 11:56 am

Hi there,

Recently bumped into this post. Great and awesome feature. I needed it too. It works with me, BUT I am always very picky. Is it possible to have the name of the host instead of the host ID in the path ../name/graph_xxx.rrd?
Structured RRA increments host id numbers for every added host. After adding more than dozen hosts you will forget which folder is which.

Eugene

User avatar
TheWitness
Developer
Posts: 14817
Joined: Tue May 14, 2002 5:08 pm
Location: MI, USA
Contact:

#9 Post by TheWitness » Fri Jul 28, 2006 1:22 pm

When we roll this into production, we will offer a drop down that let's you pick your method. Then, all hosts/graphs created after that point will be locked in. If you change in the middle though I do not propose modifying RRA file names from the user interface.

Ian has been against doing these types of things from the web interface due to security issues. Aka, read access is ok from the web site, but write is not.

This is one of the reasons we came up with the poller_commands table. It's intention is to perform various tasks after the poller has completed.

Regards,

TheWitness
True understanding begins only when we realize how little we truly understand...

Life is an adventure, let yours begin with Cacti!

Author of MacTrack, Boost, CLog, SpikeKill, Platform RTM, DSStats, maintainer of Spine, lot's of unpublished work and most of Cacti's bugs.
_________________
Official Cacti Documentation
GitHub Repository with Supported Plugins
Central Plugin Repository
Central Templates Repository


I'm still out there people. Getting excited for Cacti 1.2. I think it will be a great release.

zheka
Posts: 44
Joined: Tue May 23, 2006 11:52 am

#10 Post by zheka » Fri Jul 28, 2006 1:32 pm

Thanks, it is the answer from the real PRO indeed.
I will try to experiment with it anyway. I need it in production too!
Are there any hardware requirements for Cacti host that will graph and monitor about 50 hosts? I think my poor AMD 500 with 512 MB of RAM is not strong enough.

User avatar
browniebraun
Developer
Posts: 788
Joined: Tue Jun 13, 2006 1:17 am
Location: Cologne, Germany

#11 Post by browniebraun » Mon Jul 31, 2006 6:52 am

Hi CACTI community!

Last days I preferred to keep silent and to watch the trend of this discussion.

In my opinion it would be great to integrate a function in install.php,
so you can make a decision which RRD structure you want to use after you have installed cacti.
I agree with TheWitness's idea to lock it after you have picked your method. As you can
read it in the readme file I can not provide a mixed structure.

Using the poller_commands table would close the little security issue i have opened with this patch.

zheka: Please notice. In a great enviroment it's possible to have routers with the same name but different domains.
So if you change the name of the folder from $host_id to its name, you will lose the singleness which is needed.
The name of a router must not be forever. So what would you do if its name has been changed?
By using this structure ($host_id\local_data_id) in cacti you will only have to edit the description of the device under
"devices". Thats all!

A reverse lookup would be much more better in this case.
Here i'll see an synergy with "rrdcleaner" plugin.


Best regards
BrownieBraun

estair
Posts: 3
Joined: Mon Jan 09, 2006 4:32 pm

Inclusion of this patch in 0.86i?

#12 Post by estair » Tue Aug 22, 2006 1:17 pm

This is functional, and makes the device:file naming MUCH more elegant to work with behind the scenes (calling rrdtool from other apps). Is there any intent to include this work in Cacti? Is there any reason not to? IMO, this makes more sense than naming the RRD after sysName, and (potentially) having to change it. The ID can be pulled from the database easily, and will always* match the filename.

I for one definitely would like to see this in the next production release.

/eli

User avatar
rony
Developer/Forum Admin
Posts: 6016
Joined: Mon Nov 17, 2003 6:35 pm
Location: Michigan, USA
Contact:

Re: Inclusion of this patch in 0.86i?

#13 Post by rony » Tue Aug 22, 2006 1:24 pm

estair wrote:This is functional, and makes the device:file naming MUCH more elegant to work with behind the scenes (calling rrdtool from other apps). Is there any intent to include this work in Cacti? Is there any reason not to? IMO, this makes more sense than naming the RRD after sysName, and (potentially) having to change it. The ID can be pulled from the database easily, and will always* match the filename.

I for one definitely would like to see this in the next production release.

/eli
One issue, who says the device is SNMP enabled, or even has sysName set. :)
[size=117][i][b]Tony Roman[/b][/i][/size]
[size=84][i]Experience is what causes a person to make new mistakes instead of old ones.[/i][/size]
[size=84][i]There are only 3 way to complete a project: Good, Fast or Cheap, pick two.[/i][/size]
[size=84][i]With age comes wisdom, what you choose to do with it determines whether or not you are wise.[/i][/size]

estair
Posts: 3
Joined: Mon Jan 09, 2006 4:32 pm

Terminology correction :)

#14 Post by estair » Tue Aug 22, 2006 1:35 pm

I considered clarifying, but was too distracted... so thanks :)

I don't necessarily mean "SNMP sysName", moreso that the host->description in the database itself is less valid as a unique identifier than the 'id' row.

/eli

User avatar
rony
Developer/Forum Admin
Posts: 6016
Joined: Mon Nov 17, 2003 6:35 pm
Location: Michigan, USA
Contact:

#15 Post by rony » Tue Aug 22, 2006 1:37 pm

id is the only unique name that doesn't change with user input.

So, that being said, it will probably be used as the host directory name. :(
[size=117][i][b]Tony Roman[/b][/i][/size]
[size=84][i]Experience is what causes a person to make new mistakes instead of old ones.[/i][/size]
[size=84][i]There are only 3 way to complete a project: Good, Fast or Cheap, pick two.[/i][/size]
[size=84][i]With age comes wisdom, what you choose to do with it determines whether or not you are wise.[/i][/size]

Post Reply