Fix 64bit counters in Interface Traffic graphs

Announcements concerning Plugins for Cacti

Moderators: Moderators, Developers

Post Reply
Author
Message
coxxy
Posts: 12
Joined: Sun Mar 22, 2015 11:21 pm

Re: Fix 64bit counters in Interface Traffic graphs

#91 Post by coxxy » Tue Mar 24, 2015 9:55 pm

shd wrote:
coxxy wrote:I tried to make a 64bit graph with the same results, Im using a Dell Power edge server R220 running server 2012 with a 10 Gig Broadcomm NIC with a 3 Gbps Fiber connection

when it goes over 1Gbps the graph starts over so 1.2Gbps shows as 200Mbps

as far as i can tell its snmp v2
Fix64bit was intended to overcome problems with 32bit counters overflowing near 100Mbps. 1.2Gbps is something else. First of all, check RRD Maximum for you 64 bit graphs, set it to something more than 10^10. Then add some more zeroes at the end just to be sure. :)

As for timeouts, if the results are same, it could be something else. Try running spine in debug mode (-V=5 for verbosity, -R for read only, -H=yourhostid). see what will it say. Try also running tcpdump -vvv host youhostip for looking into other side.
Working on these steps now to see what happens, It weird how it only messes up at the 550 mb mark
i added an image ..

both these graphs are polled from identically configured servers - dell builds both running the same NIC drivers

the only difference the server having issues has a higher WAN fiber pipe

i have a bandwidth monitor ap running on the server so i can see the server is using over 1 Gbps but its not graphing
Attachments
graphs.png
graphs.png (41.94 KiB) Viewed 6045 times

coxxy
Posts: 12
Joined: Sun Mar 22, 2015 11:21 pm

Re: Fix 64bit counters in Interface Traffic graphs

#92 Post by coxxy » Wed Mar 25, 2015 12:11 am

coxxy wrote:
shd wrote:
coxxy wrote:I tried to make a 64bit graph with the same results, Im using a Dell Power edge server R220 running server 2012 with a 10 Gig Broadcomm NIC with a 3 Gbps Fiber connection

when it goes over 1Gbps the graph starts over so 1.2Gbps shows as 200Mbps

as far as i can tell its snmp v2
Fix64bit was intended to overcome problems with 32bit counters overflowing near 100Mbps. 1.2Gbps is something else. First of all, check RRD Maximum for you 64 bit graphs, set it to something more than 10^10. Then add some more zeroes at the end just to be sure. :)

As for timeouts, if the results are same, it could be something else. Try running spine in debug mode (-V=5 for verbosity, -R for read only, -H=yourhostid). see what will it say. Try also running tcpdump -vvv host youhostip for looking into other side.
Working on these steps now to see what happens, It weird how it only messes up at the 550 mb mark
i added an image ..

both these graphs are polled from identically configured servers - dell builds both running the same NIC drivers

the only difference the server having issues has a higher WAN fiber pipe

i have a bandwidth monitor ap running on the server so i can see the server is using over 1 Gbps but its not graphing


i tried to change the max values and now i get this in the log

03/25/2015 12:10:01 AM - POLLER: Poller[0] WARNING: Poller Output Table not Empty. Issues Found: 382, Data Sources: traffic_in(DS[47]), traffic_out(DS[47]), traffic_in(DS[48]), traffic_out(DS[48]), traffic_in(DS[49]), traffic_out(DS[49]),

i messed it up somewhere

coxxy
Posts: 12
Joined: Sun Mar 22, 2015 11:21 pm

Re: Fix 64bit counters in Interface Traffic graphs

#93 Post by coxxy » Wed Mar 25, 2015 11:48 pm

fixed the graphs but still only able to hit 600mb then the graph messes up ..so 800 down shows a 200 down ..its weird

shd
Cacti User
Posts: 96
Joined: Mon Apr 09, 2007 1:33 pm
Location: Nizhnevartovsk, Russia

Re: Fix 64bit counters in Interface Traffic graphs

#94 Post by shd » Thu Mar 26, 2015 8:31 am

coxxy wrote:fixed the graphs but still only able to hit 600mb then the graph messes up ..so 800 down shows a 200 down ..its weird
What does the command "rrdtool info you-rrd-file-with-that-graph.rrd" say? May be you should make rrd_maximum higher?
CentOS 64-bit, 20Gb RAM, 8xCPU, some virtual datastore. 600+ hosts, 200k+ datasources, avg polltime 45 sec

coxxy
Posts: 12
Joined: Sun Mar 22, 2015 11:21 pm

Re: Fix 64bit counters in Interface Traffic graphs

#95 Post by coxxy » Thu Mar 26, 2015 1:33 pm

shd wrote:
coxxy wrote:fixed the graphs but still only able to hit 600mb then the graph messes up ..so 800 down shows a 200 down ..its weird
What does the command "rrdtool info you-rrd-file-with-that-graph.rrd" say? May be you should make rrd_maximum higher?
im not 100% familliar with the CLI ...ill have to check what the command outputs and try the rrd max higher

coxxy
Posts: 12
Joined: Sun Mar 22, 2015 11:21 pm

Re: Fix 64bit counters in Interface Traffic graphs

#96 Post by coxxy » Fri Mar 27, 2015 12:56 pm

rrd_version = "0003"
step = 60
last_update = 1427478901
header_size = 1760
ds[traffic_in].index = 0
ds[traffic_in].type = "COUNTER"
ds[traffic_in].minimal_heartbeat = 120
ds[traffic_in].min = 0.0000000000e+00
ds[traffic_in].max = 1.0000000000e+09
ds[traffic_in].last_ds = "2426834031"
ds[traffic_in].value = 3.8675654746e+07
ds[traffic_in].unknown_sec = 0
ds[traffic_out].index = 1
ds[traffic_out].type = "COUNTER"
ds[traffic_out].minimal_heartbeat = 120
ds[traffic_out].min = 0.0000000000e+00
ds[traffic_out].max = 1.0000000000e+09
ds[traffic_out].last_ds = "2549141613"
ds[traffic_out].value = 3.6935149153e+06
ds[traffic_out].unknown_sec = 0
rra[0].cf = "AVERAGE"
rra[0].rows = 10080
rra[0].cur_row = 2924
rra[0].pdp_per_row = 1
rra[0].xff = 5.0000000000e-01
rra[0].cdp_prep[0].value = NaN
rra[0].cdp_prep[0].unknown_datapoints = 0
rra[0].cdp_prep[1].value = NaN
rra[0].cdp_prep[1].unknown_datapoints = 0
rra[1].cf = "AVERAGE"
rra[1].rows = 25920
rra[1].cur_row = 15964
rra[1].pdp_per_row = 5
rra[1].xff = 5.0000000000e-01
rra[1].cdp_prep[0].value = 0.0000000000e+00
rra[1].cdp_prep[0].unknown_datapoints = 0
rra[1].cdp_prep[1].value = 0.0000000000e+00
rra[1].cdp_prep[1].unknown_datapoints = 0
rra[2].cf = "MAX"
rra[2].rows = 10080
rra[2].cur_row = 4455
rra[2].pdp_per_row = 1
rra[2].xff = 5.0000000000e-01
rra[2].cdp_prep[0].value = NaN
rra[2].cdp_prep[0].unknown_datapoints = 0
rra[2].cdp_prep[1].value = NaN
rra[2].cdp_prep[1].unknown_datapoints = 0
rra[3].cf = "MAX"
rra[3].rows = 25920
rra[3].cur_row = 3591
rra[3].pdp_per_row = 5
rra[3].xff = 5.0000000000e-01
rra[3].cdp_prep[0].value = -inf
rra[3].cdp_prep[0].unknown_datapoints = 0
rra[3].cdp_prep[1].value = -inf
rra[3].cdp_prep[1].unknown_datapoints = 0
[email protected]:/usr/share/cacti/cli$

shd
Cacti User
Posts: 96
Joined: Mon Apr 09, 2007 1:33 pm
Location: Nizhnevartovsk, Russia

Re: Fix 64bit counters in Interface Traffic graphs

#97 Post by shd » Fri Mar 27, 2015 1:26 pm

coxxy wrote:rrd_version = "0003"
ds[traffic_in].index = 0
ds[traffic_in].max = 1.0000000000e+09
ds[traffic_in].last_ds = "2426834031"
ds[traffic_in].value = 3.8675654746e+07
ds[traffic_in].unknown_sec = 0
ds[traffic_out].index = 1
ds[traffic_out].max = 1.0000000000e+09
ds[traffic_out].last_ds = "2549141613"
ds[traffic_out].value = 3.6935149153e+06
[email protected]:/usr/share/cacti/cli$
Try changing rrdtool maximum:

Code: Select all

rrdtool tune your-rrd-file-with-that-graph.rrd -a traffic_in:1.0e+12
rrdtool tune your-rrd-file-with-that-graph.rrd -a traffic_out:1.0e+12
CentOS 64-bit, 20Gb RAM, 8xCPU, some virtual datastore. 600+ hosts, 200k+ datasources, avg polltime 45 sec

coxxy
Posts: 12
Joined: Sun Mar 22, 2015 11:21 pm

Re: Fix 64bit counters in Interface Traffic graphs

#98 Post by coxxy » Fri Mar 27, 2015 2:12 pm

Applied the change just gotta wait and see if it took it :) ...fingers crossed !!!

coxxy
Posts: 12
Joined: Sun Mar 22, 2015 11:21 pm

Re: Fix 64bit counters in Interface Traffic graphs

#99 Post by coxxy » Sat Mar 28, 2015 12:32 am

Seems to still do it ...i restarted the server so will keep watching it see if it gets better .
Attachments
Untitled.png
Untitled.png (66.09 KiB) Viewed 6009 times

coxxy
Posts: 12
Joined: Sun Mar 22, 2015 11:21 pm

Re: Fix 64bit counters in Interface Traffic graphs

#100 Post by coxxy » Sat Mar 28, 2015 2:42 am

coxxy wrote:Seems to still do it ...i restarted the server so will keep watching it see if it gets better .
DEF:a='/var/lib/cacti/rra/*rrd-file*_traffic_in_484.rrd':'traffic_out':AVERAGE \
DEF:b='/var/lib/cacti/rra/*rrd-file*_traffic_in_484.rrd':'traffic_in':AVERAGE \


When i do a graph debug it shows the 484.RRD but the graph number is 485 ...i ran the command on the 484.rrd ...does that sound right?

helpdeskdan
Posts: 1
Joined: Fri Jun 05, 2015 12:12 pm

Re: Fix 64bit counters in Interface Traffic graphs

#101 Post by helpdeskdan » Fri Jun 05, 2015 12:18 pm

I installed it, but graphs do nothing when I try to convert them, and the button does not show up. Logfile seems to indicate it tries to work. Odd.

dpotts
Posts: 1
Joined: Fri May 06, 2016 2:11 pm

Re: Fix 64bit counters in Interface Traffic graphs

#102 Post by dpotts » Fri May 06, 2016 2:13 pm

Awesome plugin.. Thanks!

Yas
Posts: 4
Joined: Fri Oct 20, 2017 9:45 am

Re: Fix 64bit counters in Interface Traffic graphs

#103 Post by Yas » Fri Oct 20, 2017 10:28 am

Hello guys,

thanks to author for all the work and everybody else for contributing!

I'm planning to use this fine plugin to convert several hundreds cacti graphs on our production network, but before doing that, I need to be sure nothing will go wrong.
I will greatly appreciate quick and precise answers.

Q1: How much time it takes for this plugin to convert, lets say, one thousand graps? In order not to lose any data, conversion must take less than 5-minutes poll time.
Q2: If something went not as planned, is there a way to revert to old 32-bit graphs without losing any data? Should I somehow backup something beforehand, and what exactly?
Q3: Is there a way to find out all 32-bit graps and convert those in 1-click?

Thanks in advance.

shd
Cacti User
Posts: 96
Joined: Mon Apr 09, 2007 1:33 pm
Location: Nizhnevartovsk, Russia

Re: Fix 64bit counters in Interface Traffic graphs

#104 Post by shd » Sat Oct 21, 2017 11:41 am

Yas wrote:Q1: How much time it takes for this plugin to convert, lets say, one thousand graps? In order not to lose any data, conversion must take less than 5-minutes poll time.
Depends on your setup, but it's quite fast. Couple of SQL updates and rrdtool tune command, which doesn't recreate graph, just updates one parameter in rrd file. A couple thousand of fixes should be performed quite fast. I think of seconds, not minutes.
Yas wrote: Q2: If something went not as planned, is there a way to revert to old 32-bit graphs without losing any data? Should I somehow backup something beforehand, and what exactly?
You should make database backup and rrd folder backup. The fixing process has some level of robustness, I haven't heard of any data corruption yet, but it's better to be on a safe side.
Yas wrote:Q3: Is there a way to find out all 32-bit graps and convert those in 1-click?
Check that you have the plugin configured in "Settings", select graphs that are matching your 32-bit template, select "Fix 64 bit counters" action, click Go and it will show you which graphs it will be converting. Plugin performs some checks before adding graph to the fixing queue.
CentOS 64-bit, 20Gb RAM, 8xCPU, some virtual datastore. 600+ hosts, 200k+ datasources, avg polltime 45 sec

Yas
Posts: 4
Joined: Fri Oct 20, 2017 9:45 am

Re: Fix 64bit counters in Interface Traffic graphs

#105 Post by Yas » Wed Oct 25, 2017 4:31 am

Thanks for clarification, conversion indeed went well without corrupting any data, we did not lose any statistics.
But for some reason we failed to batch-convert more than 1 interface graphs at once (cacti -> Graph Management -> select what we need and from pull-down menu choose "Fix 64bit counters" -> ok).
One-by-one conversion using a button to the right from a graph is perfectly fine, though.
Where should I start troubleshooting this inconvenience?
Cacti 0.8.7e, PA 2.6, fix64bit 0.4.

PS. No, we do not use thold plugin, and also cacti admin would rather not touch what is already working, so upgrading to newer version of cacti probably will not going to happen; is there any other way?

Post Reply