Distributed Cacti - Ideas

[rcaston]

I'd like to start a post to discuss how we could handle using cacti to poll extremely large enterprise environments which would require more performance from a Cacti system than a single server can provide.

My thoughts are the problem could be addressed by proceeding with one of the following two approaches:

1) Distributing out the data collection: Setting it up so a number of specific devices are polled from remote pollers that do not run on the cacti server itself. Since some of the load of the poller is the initial mysql calls; I would imagine moving to a mysql read-tier (multiple mysql replicated servers) for the many distributed pollers to hit would be mandatory here.

or

2) Integrate the data presentation: Either thru a re-write of cacti or potentially via a plugin. My thoughts here would be a simple plugin which could load up the url to the other Cacti box when clicked. Currently this could be done using a modification of the "ntop" plugin example to call a url for exported cacti data; however - the data presentation of the exported data would include some of the table images (wrappers around the table data, images) etc, which would not look right. Also; for this to work, the current bugs in graph_export.php would need to be fixed.


I'd like to invite people to share your thoughts and or experiences with this issue. Hopefully this post goes sticky.

[gandalf]

[rcaston]

Yes, there would be an I/O bottleneck as the RRDs would end up existing on different systems; but given the idea that this is for the enterprise; I don't see that being an issue for those who need it. ie: if we have to we'll stick this on a high end NAS.

...

further thoughts on the "Distributed Collector" idea would be that a new field would be added to each device in the mysql database which would specify association between the device and a "poller template"

the 'poller template' would have the necessary information associated with it to determine which of several pollers will query and poll for that object.

example: 2 pollers -

Each would query his local replicated copy of the mysql cacti database for the list of objects which match his pollerid field. (ie: give me a list of stuff i am supposed to poll and collect rrds for)

this would keep the rrds and their pollers(various cactids) on different boxes.


---

all this being said; it would still likely be easier to go with idea #2 - which is to integrate the data presentation / web interface of several stand alone cacti boxes.

[rony]

FYI, distributed polling doesn't mean that the polling machine needs access to the rrdtool files.

In the model proposed for 0.9.0, the poller is distributable, but the rrdtool updates occur on a central machine.

[rcaston]

[rony]

With rrdtool files in a central store and boost plugin functionality being included in 0.9.0, I think it will resolve most of these problems.


There are many ways to skin this cat, but I believe that we can't rely on a distrubuted storage system. So, centralized rrdtool updates based upon access (boost), will vastly improve the polling system.

I should also note that the distributed polling system is being designed to be self healing.

For more information, we should really get Larry (TheWitness) involved in this post.

[rcaston]

[wjm]

I am very interested in a distributed model.
To change cacti from a tool my team uses for mostly routers and the dozen or so servers I care about, to a tool I can share with other teams would explode the amount of devices that could be polled.

[rcaston]

The problem I am facing with using cacti as a true large enterprise solution requires having multiple distributed pollers.

Any solution which results with only a single poller server can not scale well.

imagine trying to manage a pair of routers at about 30 physical sites with about 3,000 interfaces each.

one poller will not handle (2 x 30 x 3000 = 180,000 interfaces)

assuming you are only doing traffic in/out; that would be (2 x 180,000 = 360,000) RRA files. As nice as cactid is; has anyone ever had it to poll 360,000 elements in a 5 minute cycle?

or... Anyone ever tried to open a directory with 360,000 files in it?

[Howie]

[rcaston]

[gandalf]

[TheWitness]

Boost 1.x should scale to installation with approximately 200k data sources (directory access issue aside). My plans for boost 2.0 include a method to perform the following:

0) Associate a host with a poller.
1) Provide poller based directory structures
2) Create host subdirectories (aka each hosts RRD's are in a single direcory)
3) Ability to have RRD updates occur in a distributed fashion using boost servers (distributed as well).

I may pull 0 and 3 out till 0.9. It all depends on timing. In the mean time, that will resolve the directory access issues.

The real issue as I see it is that there will always be a site that is just to big for Cacti. Well, that applies for every other tool as well.

With Boost 1.2, you can however, do the following:

1) Primary Database Server with Either MYISAM or MEMORY based Boost
2) Primary Poller with Local Storage (Lot's of it) and Boost Server
3) Web Farm with one to many Web Servers using a smaller version of Cacti with sym links to the RRA and Boost Cache folders.
4) Load Ballancer in front of the Web Farm.

With this configuration, you will have an enterprise system. For most enterprises anyway.

TheWitness

[rcaston]

[gandalf]