Cacti (home)ForumsDocumentation
Cacti: offical forums and support
It is currently Wed Jun 28, 2017 3:45 pm

All times are UTC - 5 hours




Post new topic Reply to topic  [ 109 posts ]  Go to page 1, 2, 3, 4, 5 ... 8  Next
Author Message
 Post subject: InfluxDB/Grafana/Bosun + Cacti = CereusTransport - Updated!
PostPosted: Mon Nov 30, 2015 8:01 am 
Offline
Cacti Guru User
User avatar

Joined: Thu Jan 31, 2008 6:39 am
Posts: 2589
Location: Kressbronn, Germany
Hi all,

I've created a small plugin which stores all of the polled data into an InfluxDB or Bosun. The data can then be used to be displayed using e.g. Grafana or processed further.

More information and a download link can be found here: https://www.urban-software.com/products ... ansporter/

Thanks to shd for adding bosun support to it :-)


Have a look at this screenshot of a working installation:
Attachment:
Grafana_InfluxDB_CereusTransporter.png
Grafana_InfluxDB_CereusTransporter.png [ 191.78 KiB | Viewed 7214 times ]

Attachment:
Grafana_InfluxDB_CereusTransporter_LoadAverages.png
Grafana_InfluxDB_CereusTransporter_LoadAverages.png [ 121.04 KiB | Viewed 7206 times ]

_________________
Greetings,
Phalek
---
Need more help ? Read the Cacti documentation or my new Cacti Book
Need on-site support ? Look here Cacti Workshop
Need professional Cacti support ? Look here CereusService
---
Plugins : CereusTransporter | CereusReporting | nmidWebService | nmidSmokeping | nmidWeb2 |

Code:
CereusServer Master:  SYSTEM STATS: Time:2.5621 Method:spine Processes:1 Threads:16 Hosts:446 HostsPerProcess:446 DataSources:14683 RRDsProcessed:7573
CereusServer Agent:   SYSTEM STATS: Time:27.4840 Method:spine Processes:1 Threads:8 Hosts:16 HostsPerProcess:16 DataSources:114576 RRDsProcessed:48061


Last edited by phalek on Thu Jun 16, 2016 4:43 am, edited 5 times in total.

Top
 Profile  
 
 Post subject: Re: InfluxDB + Grafana + Cacti = CereusTransport
PostPosted: Tue Dec 01, 2015 1:22 am 
Offline
Cacti User

Joined: Mon Apr 09, 2007 1:33 pm
Posts: 92
Location: Nizhnevartovsk, Russia
phalek wrote:
I've created a small plugin which stores all of the polled data into an InfluxDB. The data can then be used to be displayed using e.g. Grafana or processed further.

Link? :)

_________________
CentOS 6.7 64-bit, 12Gb RAM, 8xCPU, some virtual datastore. 497 hosts, 161598 datasources, avg polltime 33 sec


Top
 Profile  
 
 Post subject: Re: InfluxDB + Grafana + Cacti = CereusTransport
PostPosted: Tue Dec 01, 2015 4:12 am 
Offline
Cacti Guru User
User avatar

Joined: Thu Jan 31, 2008 6:39 am
Posts: 2589
Location: Kressbronn, Germany
Here you go:

http://blog.network-outsourcing.de/prod ... ansporter/

_________________
Greetings,
Phalek
---
Need more help ? Read the Cacti documentation or my new Cacti Book
Need on-site support ? Look here Cacti Workshop
Need professional Cacti support ? Look here CereusService
---
Plugins : CereusTransporter | CereusReporting | nmidWebService | nmidSmokeping | nmidWeb2 |

Code:
CereusServer Master:  SYSTEM STATS: Time:2.5621 Method:spine Processes:1 Threads:16 Hosts:446 HostsPerProcess:446 DataSources:14683 RRDsProcessed:7573
CereusServer Agent:   SYSTEM STATS: Time:27.4840 Method:spine Processes:1 Threads:8 Hosts:16 HostsPerProcess:16 DataSources:114576 RRDsProcessed:48061


Top
 Profile  
 
 Post subject: Re: InfluxDB + Grafana + Cacti = CereusTransport
PostPosted: Tue Dec 01, 2015 11:58 pm 
Offline
Cacti User

Joined: Mon Apr 09, 2007 1:33 pm
Posts: 92
Location: Nizhnevartovsk, Russia
phalek wrote:
Here you go:
http://blog.network-outsourcing.de/prod ... ansporter/
phalek wrote:
This plugin heavily interferes with the actual poller process. 5 minute polling should be ok but 1 minute polling may not be able to finish in time. Keep a look at your polling times !

Have you tried batching multiple points at once, as per "Writing multiple points" part of documentation?

There is a possibility to hit max post size, so some size splitting may be needed.

_________________
CentOS 6.7 64-bit, 12Gb RAM, 8xCPU, some virtual datastore. 497 hosts, 161598 datasources, avg polltime 33 sec


Top
 Profile  
 
 Post subject: Re: InfluxDB + Grafana + Cacti = CereusTransport
PostPosted: Wed Dec 02, 2015 12:07 am 
Offline
Cacti Guru User
User avatar

Joined: Thu Jan 31, 2008 6:39 am
Posts: 2589
Location: Kressbronn, Germany
Hence version 0.1. It's a proof-of-concept right now, working but not finished yet. Still fun to work with :-)

The next version will use a PHP module for interfacing with influxdb, allowing more possibilites like adding multiple data points or udp connections (Using https://github.com/influxdb/influxdb-php ).

_________________
Greetings,
Phalek
---
Need more help ? Read the Cacti documentation or my new Cacti Book
Need on-site support ? Look here Cacti Workshop
Need professional Cacti support ? Look here CereusService
---
Plugins : CereusTransporter | CereusReporting | nmidWebService | nmidSmokeping | nmidWeb2 |

Code:
CereusServer Master:  SYSTEM STATS: Time:2.5621 Method:spine Processes:1 Threads:16 Hosts:446 HostsPerProcess:446 DataSources:14683 RRDsProcessed:7573
CereusServer Agent:   SYSTEM STATS: Time:27.4840 Method:spine Processes:1 Threads:8 Hosts:16 HostsPerProcess:16 DataSources:114576 RRDsProcessed:48061


Top
 Profile  
 
 Post subject: Re: InfluxDB + Grafana + Cacti = CereusTransport
PostPosted: Wed Dec 23, 2015 6:54 am 
Offline
Cacti Guru User
User avatar

Joined: Thu Jan 31, 2008 6:39 am
Posts: 2589
Location: Kressbronn, Germany
Update: I got the influxdb write down to a few seconds ( grab Version 0.2 from here: http://blog.network-outsourcing.de/prod ... ansporter/ ) , though I'm still unable to do a multi-data update.

Anyone knows a working way of doing mass-updated to a influxdb ? I've tried the following version:

Fails with a "bad timestamp" issue:
Code:
curl -i -XPOST 'http://xliap114:8086/write?db=mydb' --data-binary 'load_5min,host=myTesthost,title="myTesthost\ -\ PRD\ -\ Load\ Average\ -\ 1\ Minute" value=1.68 1450870689673
load_1min,host=myTesthost,title="myTesthost\ -\ PRD\ -\ Load\ Average\ -\ 1\ Minute" value=1.68 1450870689673
load_15min,host=myTesthost,title="myTesthost\ -\ PRD\ -\ Load\ Average\ -\ 1\ Minute" value=1.68 1450870689673'


Fails with an "invalid number" issue:
Code:
curl -i -XPOST 'http://xliap114:8086/write?db=mydb' --data-binary 'load_5min,host=myTesthost,title="myTesthost\ -\ PRD\ -\ Load\ Average\ -\ 1\ Minute" value=1.68
load_1min,host=myTesthost,title="myTesthost\ -\ PRD\ -\ Load\ Average\ -\ 1\ Minute" value=1.68
load_15min,host=myTesthost,title="myTesthost\ -\ PRD\ -\ Load\ Average\ -\ 1\ Minute" value=1.68'


Succeeds, but doesn't update the database at all:
Code:
curl -i -XPOST 'http://xliap114:8086/write?db=mydb' --data-binary '
load_5min,host=myTesthost,title="myTesthost\ -\ PRD\ -\ Load\ Average\ -\ 1\ Minute" value=1.68
load_1min,host=myTesthost,title="myTesthost\ -\ PRD\ -\ Load\ Average\ -\ 1\ Minute" value=1.68
load_15min,host=myTesthost,title="myTesthost\ -\ PRD\ -\ Load\ Average\ -\ 1\ Minute" value=1.68
'


Succeeds with timestamp, but doesn't update the database at all:
Code:
curl -i -XPOST 'http://xliap114:8086/write?db=mydb' --data-binary '
load_5min,host=myTesthost,title="myTesthost\ -\ PRD\ -\ Load\ Average\ -\ 1\ Minute" value=1.68 1450870689673
load_1min,host=myTesthost,title="myTesthost\ -\ PRD\ -\ Load\ Average\ -\ 1\ Minute" value=1.68 1450870689673
load_15min,host=myTesthost,title="myTesthost\ -\ PRD\ -\ Load\ Average\ -\ 1\ Minute" value=1.68 1450870689673
'


The last one does complete within 2 seconds, so it's the fastest version, but as it doesn't update the influxdb it's of no use.
Anyone with some influxdb skills that may help out ?

_________________
Greetings,
Phalek
---
Need more help ? Read the Cacti documentation or my new Cacti Book
Need on-site support ? Look here Cacti Workshop
Need professional Cacti support ? Look here CereusService
---
Plugins : CereusTransporter | CereusReporting | nmidWebService | nmidSmokeping | nmidWeb2 |

Code:
CereusServer Master:  SYSTEM STATS: Time:2.5621 Method:spine Processes:1 Threads:16 Hosts:446 HostsPerProcess:446 DataSources:14683 RRDsProcessed:7573
CereusServer Agent:   SYSTEM STATS: Time:27.4840 Method:spine Processes:1 Threads:8 Hosts:16 HostsPerProcess:16 DataSources:114576 RRDsProcessed:48061


Top
 Profile  
 
 Post subject: Re: InfluxDB + Grafana + Cacti = CereusTransport
PostPosted: Fri Dec 25, 2015 3:04 am 
Offline
Cacti User

Joined: Mon Apr 09, 2007 1:33 pm
Posts: 92
Location: Nizhnevartovsk, Russia
phalek wrote:
Update: I got the influxdb write down to a few seconds ( grab Version 0.2 from here: http://blog.network-outsourcing.de/prod ... ansporter/ ) , though I'm still unable to do a multi-data update.

btw, you have hardcoded host name for curl:
Code:
                $cmd = "curl -i -XPOST 'http://xliap114:8086/write?db=mydb' --data-binary '"; 

_________________
CentOS 6.7 64-bit, 12Gb RAM, 8xCPU, some virtual datastore. 497 hosts, 161598 datasources, avg polltime 33 sec


Top
 Profile  
 
 Post subject: Re: InfluxDB + Grafana + Cacti = CereusTransport
PostPosted: Fri Dec 25, 2015 4:25 am 
Offline
Cacti Guru User
User avatar

Joined: Thu Jan 31, 2008 6:39 am
Posts: 2589
Location: Kressbronn, Germany
fixed :-)

The settings are now in the "misc" tab, as was with v0.1

_________________
Greetings,
Phalek
---
Need more help ? Read the Cacti documentation or my new Cacti Book
Need on-site support ? Look here Cacti Workshop
Need professional Cacti support ? Look here CereusService
---
Plugins : CereusTransporter | CereusReporting | nmidWebService | nmidSmokeping | nmidWeb2 |

Code:
CereusServer Master:  SYSTEM STATS: Time:2.5621 Method:spine Processes:1 Threads:16 Hosts:446 HostsPerProcess:446 DataSources:14683 RRDsProcessed:7573
CereusServer Agent:   SYSTEM STATS: Time:27.4840 Method:spine Processes:1 Threads:8 Hosts:16 HostsPerProcess:16 DataSources:114576 RRDsProcessed:48061


Top
 Profile  
 
 Post subject: Re: InfluxDB + Grafana + Cacti = CereusTransport
PostPosted: Mon Dec 28, 2015 6:03 am 
Offline
Cacti Guru User
User avatar

Joined: Thu Jan 31, 2008 6:39 am
Posts: 2589
Location: Kressbronn, Germany
Version 0.3 is now available. The multi update part has been fixed using a InfluxDB/PHP module available ( compatible with PHP >5.3 )

The update time is now taking about 1/4 of the original time ( Depending on the number of datasources per host ).

Please note: The configuration has been changed from URL based to host/port/db_name. So make sure to update your settings (misc tab) accordingly !

http://blog.network-outsourcing.de/prod ... ansporter/

_________________
Greetings,
Phalek
---
Need more help ? Read the Cacti documentation or my new Cacti Book
Need on-site support ? Look here Cacti Workshop
Need professional Cacti support ? Look here CereusService
---
Plugins : CereusTransporter | CereusReporting | nmidWebService | nmidSmokeping | nmidWeb2 |

Code:
CereusServer Master:  SYSTEM STATS: Time:2.5621 Method:spine Processes:1 Threads:16 Hosts:446 HostsPerProcess:446 DataSources:14683 RRDsProcessed:7573
CereusServer Agent:   SYSTEM STATS: Time:27.4840 Method:spine Processes:1 Threads:8 Hosts:16 HostsPerProcess:16 DataSources:114576 RRDsProcessed:48061


Top
 Profile  
 
 Post subject: Re: InfluxDB + Grafana + Cacti = CereusTransport
PostPosted: Wed Dec 30, 2015 7:38 am 
Offline
Cacti Guru User
User avatar

Joined: Thu Jan 31, 2008 6:39 am
Posts: 2589
Location: Kressbronn, Germany
Version 0.4 is now available. This fixes an issue with data points getting lost ( same local_data_id in cacti, eg. traffic_in, traffic_out of network interface eth0 ).

http://blog.network-outsourcing.de/prod ... ansporter/

_________________
Greetings,
Phalek
---
Need more help ? Read the Cacti documentation or my new Cacti Book
Need on-site support ? Look here Cacti Workshop
Need professional Cacti support ? Look here CereusService
---
Plugins : CereusTransporter | CereusReporting | nmidWebService | nmidSmokeping | nmidWeb2 |

Code:
CereusServer Master:  SYSTEM STATS: Time:2.5621 Method:spine Processes:1 Threads:16 Hosts:446 HostsPerProcess:446 DataSources:14683 RRDsProcessed:7573
CereusServer Agent:   SYSTEM STATS: Time:27.4840 Method:spine Processes:1 Threads:8 Hosts:16 HostsPerProcess:16 DataSources:114576 RRDsProcessed:48061


Top
 Profile  
 
 Post subject: Re: InfluxDB + Grafana + Cacti = CereusTransport
PostPosted: Mon Jan 18, 2016 8:43 pm 
Offline

Joined: Mon Jan 18, 2016 8:34 pm
Posts: 4
Great plugin! It's exactly what I've been looking for. However, I'm having a little issue. The plugin is not picking up all of my data sources and pushing them to InfluxDB, it's missing quite a few per host actually actually. See my debug log below:


Code:
01/18/2016 05:18:05 PM - CereusTransporter: Poller[0] Finished adding [2] of data points for [192.168.1.3]
01/18/2016 05:18:05 PM - CereusTransporter: Poller[0] Adding [2] of data points for [192.168.1.3]
01/18/2016 05:18:05 PM - CereusTransporter: Poller[0] DEBUG: Appending the following data to request: [influxdb] [1453166282] [192.168.1.3] [cisco\ switch\ -\ Traffic\ -\ Nu0] [traffic_out] [0]
01/18/2016 05:18:05 PM - CereusTransporter: Poller[0] DEBUG: Appending the following data to request: [influxdb] [1453166282] [192.168.1.3] [cisco\ switch\ -\ Traffic\ -\ Nu0] [traffic_in] [0]
01/18/2016 05:18:05 PM - CereusTransporter: Poller[0] DEBUG: Adding the following data to influxdb: [1453166282] [192.168.1.3]


The host (192.168.1.3) has many more data points then it is seeing. I'm experiencing this issue on about half of my hosts. I'm using the latest version of the plugin, with Cacti 0.8.8a. Here's some screenshots of the number of data sources and graphs that actually exist for the above host,

https://www.dropbox.com/s/bvmgu2xlfz5r6uc/cacti3.PNG?dl=0
https://www.dropbox.com/s/m6gp825968ugfyu/cacti_grpah.PNG?dl=0

Any ideas?


Top
 Profile  
 
 Post subject: Re: InfluxDB + Grafana + Cacti = CereusTransport
PostPosted: Tue Jan 19, 2016 8:50 am 
Offline
Cacti Guru User
User avatar

Joined: Thu Jan 31, 2008 6:39 am
Posts: 2589
Location: Kressbronn, Germany
Can you send me a complete log for one polling cycle containing only the debug messages of CereusTransport ?

It's processing the data ordered by "timestamp", so it may actually containg several entries for the same host but scattered around as the poller didn't collect them all at the very same second :-)

_________________
Greetings,
Phalek
---
Need more help ? Read the Cacti documentation or my new Cacti Book
Need on-site support ? Look here Cacti Workshop
Need professional Cacti support ? Look here CereusService
---
Plugins : CereusTransporter | CereusReporting | nmidWebService | nmidSmokeping | nmidWeb2 |

Code:
CereusServer Master:  SYSTEM STATS: Time:2.5621 Method:spine Processes:1 Threads:16 Hosts:446 HostsPerProcess:446 DataSources:14683 RRDsProcessed:7573
CereusServer Agent:   SYSTEM STATS: Time:27.4840 Method:spine Processes:1 Threads:8 Hosts:16 HostsPerProcess:16 DataSources:114576 RRDsProcessed:48061


Top
 Profile  
 
 Post subject: Re: InfluxDB + Grafana + Cacti = CereusTransport
PostPosted: Tue Jan 19, 2016 12:51 pm 
Offline

Joined: Mon Jan 18, 2016 8:34 pm
Posts: 4
Here's the complete log,

https://www.dropbox.com/s/43lo2ek3iekiac7/cereus_log_cacti.txt?dl=0

Let me know if you need anything else to help debug. I'm also using 1 minute polling, not sure if that causes issues or makes a difference.


Top
 Profile  
 
 Post subject: Re: InfluxDB + Grafana + Cacti = CereusTransport
PostPosted: Tue Jan 19, 2016 1:43 pm 
Offline
Cacti User

Joined: Mon Apr 09, 2007 1:33 pm
Posts: 92
Location: Nizhnevartovsk, Russia
I also saw that many datapoints were not being pushed into db, wasn't satisfied with the speed of script and with a lack of Bosun support. So I've started to change the script. Well, it became a rewrite.

Here are my changes:
Quote:
0.45 (changes by [email protected] )
* Added Bosun export support ( http://bosun.org/ ). Also OpenTSDB. May have broken InfluxDB support, haven't tested it, but should be working.

* Changed settings back to DB URI instead of db, port and so on in separate settings. Made configuration simpler.

* Fixed population of data array by key - version 0.4 was loosing too much data (for example, if there was more than one traffic_out, only one was saved)

* Converted plugin_CereusTransporter_data to a Memory table. Nice improvement in speed. Make sure to make mysql memory table parameters bigger
(tmp_table_size and max_heap_table_size). Table will be recreated automatically by the upgrade script (hopefully).
Data is considered temporary and gets deleted on every sending pass.

* Added much more additional data to metrics: host and hostname, data source type (gauge, counter), index type and value if used, units and so on.
Unit name is taken from Vertical Label of Graph Template for datasource. If there are multiple Graph Templates, only one value will be taken.
Vertical Label text should be from this list, or bosun will skip it: https://godoc.org/bosun.org/metadata#Unit
Metrics info for Bosun is sent to different api outlet.

* Pre-populated all needed info from DB. No SQL queries are made for each metric. HUGE improvement in speed. Now it takes about 90 seconds on average
to send 168k data points into Bosun.

* Additional stats in poller logs. Example:
CereusTransporter: Poller[0] STATS: Time:98.82 LookupTableSize:32685 LookupUnitsTableSize:22363 MetricsCount:168862 AverageHostTime:0.12
SlowestHostnames:[host1: 3.544s, host2: 2.784s, host3: 2.216s, host4: 2.159s]


Changes are not much tested, but they are working in my environment. No guarantees.


Attachments:
File comment: My take on transport plugin
CereusTransporter-0.45.tar.gz [943.14 KiB]
Downloaded 145 times

_________________
CentOS 6.7 64-bit, 12Gb RAM, 8xCPU, some virtual datastore. 497 hosts, 161598 datasources, avg polltime 33 sec
Top
 Profile  
 
 Post subject: Re: InfluxDB + Grafana + Cacti = CereusTransport
PostPosted: Tue Jan 19, 2016 2:39 pm 
Offline
Cacti Guru User
User avatar

Joined: Thu Jan 31, 2008 6:39 am
Posts: 2589
Location: Kressbronn, Germany
shd wrote:
I also saw that many datapoints were not being pushed into db, wasn't satisfied with the speed of script and with a lack of Bosun support. So I've started to change the script. Well, it became a rewrite.


thanks shd for this. I'll add it to my code and also upload it to my bitbucket.org account for better tracking.

It's still a kind of proof-of-concept plugin, but i'm glad some of you do find it useful and inspiring to enhance it.


For the data, I think there may be cases where the tables gets deleted (poller_top) while it's still working on the data ( overrunning poller processes ) , so maybe changing the deletion of old data should be done after the processing of it ( e.g delete from `plugin_CereusTransporter_data` where timestamp < oldest_processed_data_item_timestamp ).

It would be nice to know the difference in updating the rrd files and adding the data into inflixdb/bosun ....

_________________
Greetings,
Phalek
---
Need more help ? Read the Cacti documentation or my new Cacti Book
Need on-site support ? Look here Cacti Workshop
Need professional Cacti support ? Look here CereusService
---
Plugins : CereusTransporter | CereusReporting | nmidWebService | nmidSmokeping | nmidWeb2 |

Code:
CereusServer Master:  SYSTEM STATS: Time:2.5621 Method:spine Processes:1 Threads:16 Hosts:446 HostsPerProcess:446 DataSources:14683 RRDsProcessed:7573
CereusServer Agent:   SYSTEM STATS: Time:27.4840 Method:spine Processes:1 Threads:8 Hosts:16 HostsPerProcess:16 DataSources:114576 RRDsProcessed:48061


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 109 posts ]  Go to page 1, 2, 3, 4, 5 ... 8  Next

All times are UTC - 5 hours


Who is online

Users browsing this forum: No registered users and 1 guest


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  

Protected by Anti-Spam ACP Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group