InfluxDB/Grafana/Bosun + Cacti = CereusTransport - Updated!

Addons for Cacti and discussion about those addons

Moderators: Moderators, Developers

Post Reply
Author
Message
User avatar
phalek
Cacti Guru User
Posts: 2710
Joined: Thu Jan 31, 2008 6:39 am
Location: Kressbronn, Germany
Contact:

InfluxDB/Grafana/Bosun + Cacti = CereusTransport - Updated!

#1 Post by phalek » Mon Nov 30, 2015 8:01 am

Hi all,

I've created a small plugin which stores all of the polled data into an InfluxDB or Bosun. The data can then be used to be displayed using e.g. Grafana or processed further.

More information and a download link can be found here: https://www.urban-software.com/products ... ansporter/

Thanks to shd for adding bosun support to it :-)


Have a look at this screenshot of a working installation:
Grafana_InfluxDB_CereusTransporter.png
Grafana_InfluxDB_CereusTransporter.png (191.78 KiB) Viewed 12270 times
Grafana_InfluxDB_CereusTransporter_LoadAverages.png
Grafana_InfluxDB_CereusTransporter_LoadAverages.png (121.04 KiB) Viewed 12262 times
Last edited by phalek on Thu Jun 16, 2016 4:43 am, edited 5 times in total.
Greetings,
Phalek
---
Need more help ? Read the Cacti documentation or my new Cacti 1.x Book
Need on-site support ? Look here Cacti Workshop
Need professional Cacti support ? Look here CereusService
---
Plugins : CereusTransporter | CereusReporting | nmidWebService | nmidSmokeping | nmidWeb2 |

Code: Select all

CereusServer Master:  SYSTEM STATS: Time:2.5621 Method:spine Processes:1 Threads:16 Hosts:446 HostsPerProcess:446 DataSources:14683 RRDsProcessed:7573
CereusServer Agent:   SYSTEM STATS: Time:27.4840 Method:spine Processes:1 Threads:8 Hosts:16 HostsPerProcess:16 DataSources:114576 RRDsProcessed:48061
[/size]

shd
Cacti User
Posts: 96
Joined: Mon Apr 09, 2007 1:33 pm
Location: Nizhnevartovsk, Russia

Re: InfluxDB + Grafana + Cacti = CereusTransport

#2 Post by shd » Tue Dec 01, 2015 1:22 am

phalek wrote:I've created a small plugin which stores all of the polled data into an InfluxDB. The data can then be used to be displayed using e.g. Grafana or processed further.
Link? :)
CentOS 64-bit, 20Gb RAM, 8xCPU, some virtual datastore. 600+ hosts, 200k+ datasources, avg polltime 45 sec

User avatar
phalek
Cacti Guru User
Posts: 2710
Joined: Thu Jan 31, 2008 6:39 am
Location: Kressbronn, Germany
Contact:

Re: InfluxDB + Grafana + Cacti = CereusTransport

#3 Post by phalek » Tue Dec 01, 2015 4:12 am

Greetings,
Phalek
---
Need more help ? Read the Cacti documentation or my new Cacti 1.x Book
Need on-site support ? Look here Cacti Workshop
Need professional Cacti support ? Look here CereusService
---
Plugins : CereusTransporter | CereusReporting | nmidWebService | nmidSmokeping | nmidWeb2 |

Code: Select all

CereusServer Master:  SYSTEM STATS: Time:2.5621 Method:spine Processes:1 Threads:16 Hosts:446 HostsPerProcess:446 DataSources:14683 RRDsProcessed:7573
CereusServer Agent:   SYSTEM STATS: Time:27.4840 Method:spine Processes:1 Threads:8 Hosts:16 HostsPerProcess:16 DataSources:114576 RRDsProcessed:48061
[/size]

shd
Cacti User
Posts: 96
Joined: Mon Apr 09, 2007 1:33 pm
Location: Nizhnevartovsk, Russia

Re: InfluxDB + Grafana + Cacti = CereusTransport

#4 Post by shd » Tue Dec 01, 2015 11:58 pm

phalek wrote:Here you go:
http://blog.network-outsourcing.de/prod ... ansporter/
phalek wrote:This plugin heavily interferes with the actual poller process. 5 minute polling should be ok but 1 minute polling may not be able to finish in time. Keep a look at your polling times !
Have you tried batching multiple points at once, as per "Writing multiple points" part of documentation?

There is a possibility to hit max post size, so some size splitting may be needed.
CentOS 64-bit, 20Gb RAM, 8xCPU, some virtual datastore. 600+ hosts, 200k+ datasources, avg polltime 45 sec

User avatar
phalek
Cacti Guru User
Posts: 2710
Joined: Thu Jan 31, 2008 6:39 am
Location: Kressbronn, Germany
Contact:

Re: InfluxDB + Grafana + Cacti = CereusTransport

#5 Post by phalek » Wed Dec 02, 2015 12:07 am

Hence version 0.1. It's a proof-of-concept right now, working but not finished yet. Still fun to work with :-)

The next version will use a PHP module for interfacing with influxdb, allowing more possibilites like adding multiple data points or udp connections (Using https://github.com/influxdb/influxdb-php ).
Greetings,
Phalek
---
Need more help ? Read the Cacti documentation or my new Cacti 1.x Book
Need on-site support ? Look here Cacti Workshop
Need professional Cacti support ? Look here CereusService
---
Plugins : CereusTransporter | CereusReporting | nmidWebService | nmidSmokeping | nmidWeb2 |

Code: Select all

CereusServer Master:  SYSTEM STATS: Time:2.5621 Method:spine Processes:1 Threads:16 Hosts:446 HostsPerProcess:446 DataSources:14683 RRDsProcessed:7573
CereusServer Agent:   SYSTEM STATS: Time:27.4840 Method:spine Processes:1 Threads:8 Hosts:16 HostsPerProcess:16 DataSources:114576 RRDsProcessed:48061
[/size]

User avatar
phalek
Cacti Guru User
Posts: 2710
Joined: Thu Jan 31, 2008 6:39 am
Location: Kressbronn, Germany
Contact:

Re: InfluxDB + Grafana + Cacti = CereusTransport

#6 Post by phalek » Wed Dec 23, 2015 6:54 am

Update: I got the influxdb write down to a few seconds ( grab Version 0.2 from here: http://blog.network-outsourcing.de/prod ... ansporter/) , though I'm still unable to do a multi-data update.

Anyone knows a working way of doing mass-updated to a influxdb ? I've tried the following version:

Fails with a "bad timestamp" issue:

Code: Select all

curl -i -XPOST 'http://xliap114:8086/write?db=mydb' --data-binary 'load_5min,host=myTesthost,title="myTesthost\ -\ PRD\ -\ Load\ Average\ -\ 1\ Minute" value=1.68 1450870689673
load_1min,host=myTesthost,title="myTesthost\ -\ PRD\ -\ Load\ Average\ -\ 1\ Minute" value=1.68 1450870689673 
load_15min,host=myTesthost,title="myTesthost\ -\ PRD\ -\ Load\ Average\ -\ 1\ Minute" value=1.68 1450870689673'
Fails with an "invalid number" issue:

Code: Select all

curl -i -XPOST 'http://xliap114:8086/write?db=mydb' --data-binary 'load_5min,host=myTesthost,title="myTesthost\ -\ PRD\ -\ Load\ Average\ -\ 1\ Minute" value=1.68
load_1min,host=myTesthost,title="myTesthost\ -\ PRD\ -\ Load\ Average\ -\ 1\ Minute" value=1.68 
load_15min,host=myTesthost,title="myTesthost\ -\ PRD\ -\ Load\ Average\ -\ 1\ Minute" value=1.68' 
Succeeds, but doesn't update the database at all:

Code: Select all

curl -i -XPOST 'http://xliap114:8086/write?db=mydb' --data-binary '
load_5min,host=myTesthost,title="myTesthost\ -\ PRD\ -\ Load\ Average\ -\ 1\ Minute" value=1.68
load_1min,host=myTesthost,title="myTesthost\ -\ PRD\ -\ Load\ Average\ -\ 1\ Minute" value=1.68 
load_15min,host=myTesthost,title="myTesthost\ -\ PRD\ -\ Load\ Average\ -\ 1\ Minute" value=1.68
' 
Succeeds with timestamp, but doesn't update the database at all:

Code: Select all

curl -i -XPOST 'http://xliap114:8086/write?db=mydb' --data-binary '
load_5min,host=myTesthost,title="myTesthost\ -\ PRD\ -\ Load\ Average\ -\ 1\ Minute" value=1.68 1450870689673
load_1min,host=myTesthost,title="myTesthost\ -\ PRD\ -\ Load\ Average\ -\ 1\ Minute" value=1.68 1450870689673 
load_15min,host=myTesthost,title="myTesthost\ -\ PRD\ -\ Load\ Average\ -\ 1\ Minute" value=1.68 1450870689673
' 
The last one does complete within 2 seconds, so it's the fastest version, but as it doesn't update the influxdb it's of no use.
Anyone with some influxdb skills that may help out ?
Greetings,
Phalek
---
Need more help ? Read the Cacti documentation or my new Cacti 1.x Book
Need on-site support ? Look here Cacti Workshop
Need professional Cacti support ? Look here CereusService
---
Plugins : CereusTransporter | CereusReporting | nmidWebService | nmidSmokeping | nmidWeb2 |

Code: Select all

CereusServer Master:  SYSTEM STATS: Time:2.5621 Method:spine Processes:1 Threads:16 Hosts:446 HostsPerProcess:446 DataSources:14683 RRDsProcessed:7573
CereusServer Agent:   SYSTEM STATS: Time:27.4840 Method:spine Processes:1 Threads:8 Hosts:16 HostsPerProcess:16 DataSources:114576 RRDsProcessed:48061
[/size]

shd
Cacti User
Posts: 96
Joined: Mon Apr 09, 2007 1:33 pm
Location: Nizhnevartovsk, Russia

Re: InfluxDB + Grafana + Cacti = CereusTransport

#7 Post by shd » Fri Dec 25, 2015 3:04 am

phalek wrote:Update: I got the influxdb write down to a few seconds ( grab Version 0.2 from here: http://blog.network-outsourcing.de/prod ... ansporter/) , though I'm still unable to do a multi-data update.
btw, you have hardcoded host name for curl:

Code: Select all

                $cmd = "curl -i -XPOST 'http://xliap114:8086/write?db=mydb' --data-binary '"; 
CentOS 64-bit, 20Gb RAM, 8xCPU, some virtual datastore. 600+ hosts, 200k+ datasources, avg polltime 45 sec

User avatar
phalek
Cacti Guru User
Posts: 2710
Joined: Thu Jan 31, 2008 6:39 am
Location: Kressbronn, Germany
Contact:

Re: InfluxDB + Grafana + Cacti = CereusTransport

#8 Post by phalek » Fri Dec 25, 2015 4:25 am

fixed :-)

The settings are now in the "misc" tab, as was with v0.1
Greetings,
Phalek
---
Need more help ? Read the Cacti documentation or my new Cacti 1.x Book
Need on-site support ? Look here Cacti Workshop
Need professional Cacti support ? Look here CereusService
---
Plugins : CereusTransporter | CereusReporting | nmidWebService | nmidSmokeping | nmidWeb2 |

Code: Select all

CereusServer Master:  SYSTEM STATS: Time:2.5621 Method:spine Processes:1 Threads:16 Hosts:446 HostsPerProcess:446 DataSources:14683 RRDsProcessed:7573
CereusServer Agent:   SYSTEM STATS: Time:27.4840 Method:spine Processes:1 Threads:8 Hosts:16 HostsPerProcess:16 DataSources:114576 RRDsProcessed:48061
[/size]

User avatar
phalek
Cacti Guru User
Posts: 2710
Joined: Thu Jan 31, 2008 6:39 am
Location: Kressbronn, Germany
Contact:

Re: InfluxDB + Grafana + Cacti = CereusTransport

#9 Post by phalek » Mon Dec 28, 2015 6:03 am

Version 0.3 is now available. The multi update part has been fixed using a InfluxDB/PHP module available ( compatible with PHP >5.3 )

The update time is now taking about 1/4 of the original time ( Depending on the number of datasources per host ).

Please note: The configuration has been changed from URL based to host/port/db_name. So make sure to update your settings (misc tab) accordingly !

http://blog.network-outsourcing.de/prod ... ansporter/
Greetings,
Phalek
---
Need more help ? Read the Cacti documentation or my new Cacti 1.x Book
Need on-site support ? Look here Cacti Workshop
Need professional Cacti support ? Look here CereusService
---
Plugins : CereusTransporter | CereusReporting | nmidWebService | nmidSmokeping | nmidWeb2 |

Code: Select all

CereusServer Master:  SYSTEM STATS: Time:2.5621 Method:spine Processes:1 Threads:16 Hosts:446 HostsPerProcess:446 DataSources:14683 RRDsProcessed:7573
CereusServer Agent:   SYSTEM STATS: Time:27.4840 Method:spine Processes:1 Threads:8 Hosts:16 HostsPerProcess:16 DataSources:114576 RRDsProcessed:48061
[/size]

User avatar
phalek
Cacti Guru User
Posts: 2710
Joined: Thu Jan 31, 2008 6:39 am
Location: Kressbronn, Germany
Contact:

Re: InfluxDB + Grafana + Cacti = CereusTransport

#10 Post by phalek » Wed Dec 30, 2015 7:38 am

Version 0.4 is now available. This fixes an issue with data points getting lost ( same local_data_id in cacti, eg. traffic_in, traffic_out of network interface eth0 ).

http://blog.network-outsourcing.de/prod ... ansporter/
Greetings,
Phalek
---
Need more help ? Read the Cacti documentation or my new Cacti 1.x Book
Need on-site support ? Look here Cacti Workshop
Need professional Cacti support ? Look here CereusService
---
Plugins : CereusTransporter | CereusReporting | nmidWebService | nmidSmokeping | nmidWeb2 |

Code: Select all

CereusServer Master:  SYSTEM STATS: Time:2.5621 Method:spine Processes:1 Threads:16 Hosts:446 HostsPerProcess:446 DataSources:14683 RRDsProcessed:7573
CereusServer Agent:   SYSTEM STATS: Time:27.4840 Method:spine Processes:1 Threads:8 Hosts:16 HostsPerProcess:16 DataSources:114576 RRDsProcessed:48061
[/size]

theslim1
Posts: 4
Joined: Mon Jan 18, 2016 8:34 pm

Re: InfluxDB + Grafana + Cacti = CereusTransport

#11 Post by theslim1 » Mon Jan 18, 2016 8:43 pm

Great plugin! It's exactly what I've been looking for. However, I'm having a little issue. The plugin is not picking up all of my data sources and pushing them to InfluxDB, it's missing quite a few per host actually actually. See my debug log below:

Code: Select all

01/18/2016 05:18:05 PM - CereusTransporter: Poller[0] Finished adding [2] of data points for [192.168.1.3]
01/18/2016 05:18:05 PM - CereusTransporter: Poller[0] Adding [2] of data points for [192.168.1.3]
01/18/2016 05:18:05 PM - CereusTransporter: Poller[0] DEBUG: Appending the following data to request: [influxdb] [1453166282] [192.168.1.3] [cisco\ switch\ -\ Traffic\ -\ Nu0] [traffic_out] [0]
01/18/2016 05:18:05 PM - CereusTransporter: Poller[0] DEBUG: Appending the following data to request: [influxdb] [1453166282] [192.168.1.3] [cisco\ switch\ -\ Traffic\ -\ Nu0] [traffic_in] [0]
01/18/2016 05:18:05 PM - CereusTransporter: Poller[0] DEBUG: Adding the following data to influxdb: [1453166282] [192.168.1.3] 
The host (192.168.1.3) has many more data points then it is seeing. I'm experiencing this issue on about half of my hosts. I'm using the latest version of the plugin, with Cacti 0.8.8a. Here's some screenshots of the number of data sources and graphs that actually exist for the above host,

https://www.dropbox.com/s/bvmgu2xlfz5r6 ... 3.PNG?dl=0
https://www.dropbox.com/s/m6gp825968ugf ... h.PNG?dl=0

Any ideas?

User avatar
phalek
Cacti Guru User
Posts: 2710
Joined: Thu Jan 31, 2008 6:39 am
Location: Kressbronn, Germany
Contact:

Re: InfluxDB + Grafana + Cacti = CereusTransport

#12 Post by phalek » Tue Jan 19, 2016 8:50 am

Can you send me a complete log for one polling cycle containing only the debug messages of CereusTransport ?

It's processing the data ordered by "timestamp", so it may actually containg several entries for the same host but scattered around as the poller didn't collect them all at the very same second :-)
Greetings,
Phalek
---
Need more help ? Read the Cacti documentation or my new Cacti 1.x Book
Need on-site support ? Look here Cacti Workshop
Need professional Cacti support ? Look here CereusService
---
Plugins : CereusTransporter | CereusReporting | nmidWebService | nmidSmokeping | nmidWeb2 |

Code: Select all

CereusServer Master:  SYSTEM STATS: Time:2.5621 Method:spine Processes:1 Threads:16 Hosts:446 HostsPerProcess:446 DataSources:14683 RRDsProcessed:7573
CereusServer Agent:   SYSTEM STATS: Time:27.4840 Method:spine Processes:1 Threads:8 Hosts:16 HostsPerProcess:16 DataSources:114576 RRDsProcessed:48061
[/size]

theslim1
Posts: 4
Joined: Mon Jan 18, 2016 8:34 pm

Re: InfluxDB + Grafana + Cacti = CereusTransport

#13 Post by theslim1 » Tue Jan 19, 2016 12:51 pm

Here's the complete log,

https://www.dropbox.com/s/43lo2ek3iekia ... i.txt?dl=0

Let me know if you need anything else to help debug. I'm also using 1 minute polling, not sure if that causes issues or makes a difference.

shd
Cacti User
Posts: 96
Joined: Mon Apr 09, 2007 1:33 pm
Location: Nizhnevartovsk, Russia

Re: InfluxDB + Grafana + Cacti = CereusTransport

#14 Post by shd » Tue Jan 19, 2016 1:43 pm

I also saw that many datapoints were not being pushed into db, wasn't satisfied with the speed of script and with a lack of Bosun support. So I've started to change the script. Well, it became a rewrite.

Here are my changes:
0.45 (changes by [email protected])
* Added Bosun export support (http://bosun.org/). Also OpenTSDB. May have broken InfluxDB support, haven't tested it, but should be working.

* Changed settings back to DB URI instead of db, port and so on in separate settings. Made configuration simpler.

* Fixed population of data array by key - version 0.4 was loosing too much data (for example, if there was more than one traffic_out, only one was saved)

* Converted plugin_CereusTransporter_data to a Memory table. Nice improvement in speed. Make sure to make mysql memory table parameters bigger
(tmp_table_size and max_heap_table_size). Table will be recreated automatically by the upgrade script (hopefully).
Data is considered temporary and gets deleted on every sending pass.

* Added much more additional data to metrics: host and hostname, data source type (gauge, counter), index type and value if used, units and so on.
Unit name is taken from Vertical Label of Graph Template for datasource. If there are multiple Graph Templates, only one value will be taken.
Vertical Label text should be from this list, or bosun will skip it: https://godoc.org/bosun.org/metadata#Unit
Metrics info for Bosun is sent to different api outlet.

* Pre-populated all needed info from DB. No SQL queries are made for each metric. HUGE improvement in speed. Now it takes about 90 seconds on average
to send 168k data points into Bosun.

* Additional stats in poller logs. Example:
CereusTransporter: Poller[0] STATS: Time:98.82 LookupTableSize:32685 LookupUnitsTableSize:22363 MetricsCount:168862 AverageHostTime:0.12
SlowestHostnames:[host1: 3.544s, host2: 2.784s, host3: 2.216s, host4: 2.159s]
Changes are not much tested, but they are working in my environment. No guarantees.
Attachments
CereusTransporter-0.45.tar.gz
My take on transport plugin
(943.14 KiB) Downloaded 266 times
CentOS 64-bit, 20Gb RAM, 8xCPU, some virtual datastore. 600+ hosts, 200k+ datasources, avg polltime 45 sec

User avatar
phalek
Cacti Guru User
Posts: 2710
Joined: Thu Jan 31, 2008 6:39 am
Location: Kressbronn, Germany
Contact:

Re: InfluxDB + Grafana + Cacti = CereusTransport

#15 Post by phalek » Tue Jan 19, 2016 2:39 pm

shd wrote:I also saw that many datapoints were not being pushed into db, wasn't satisfied with the speed of script and with a lack of Bosun support. So I've started to change the script. Well, it became a rewrite.
thanks shd for this. I'll add it to my code and also upload it to my bitbucket.org account for better tracking.

It's still a kind of proof-of-concept plugin, but i'm glad some of you do find it useful and inspiring to enhance it.


For the data, I think there may be cases where the tables gets deleted (poller_top) while it's still working on the data ( overrunning poller processes ) , so maybe changing the deletion of old data should be done after the processing of it ( e.g delete from `plugin_CereusTransporter_data` where timestamp < oldest_processed_data_item_timestamp ).

It would be nice to know the difference in updating the rrd files and adding the data into inflixdb/bosun ....
Greetings,
Phalek
---
Need more help ? Read the Cacti documentation or my new Cacti 1.x Book
Need on-site support ? Look here Cacti Workshop
Need professional Cacti support ? Look here CereusService
---
Plugins : CereusTransporter | CereusReporting | nmidWebService | nmidSmokeping | nmidWeb2 |

Code: Select all

CereusServer Master:  SYSTEM STATS: Time:2.5621 Method:spine Processes:1 Threads:16 Hosts:446 HostsPerProcess:446 DataSources:14683 RRDsProcessed:7573
CereusServer Agent:   SYSTEM STATS: Time:27.4840 Method:spine Processes:1 Threads:8 Hosts:16 HostsPerProcess:16 DataSources:114576 RRDsProcessed:48061
[/size]

Post Reply