Cacti (home)ForumsDocumentation
Cacti: offical forums and support
It is currently Sat Jan 20, 2018 9:23 pm

All times are UTC - 5 hours




Post new topic Reply to topic  [ 110 posts ]  Go to page Previous  1 ... 4, 5, 6, 7, 8  Next
Author Message
 Post subject: Re: Fix 64bit counters in Interface Traffic graphs
PostPosted: Tue Mar 24, 2015 9:55 pm 
Offline

Joined: Sun Mar 22, 2015 11:21 pm
Posts: 12
shd wrote:
coxxy wrote:
I tried to make a 64bit graph with the same results, Im using a Dell Power edge server R220 running server 2012 with a 10 Gig Broadcomm NIC with a 3 Gbps Fiber connection

when it goes over 1Gbps the graph starts over so 1.2Gbps shows as 200Mbps

as far as i can tell its snmp v2

Fix64bit was intended to overcome problems with 32bit counters overflowing near 100Mbps. 1.2Gbps is something else. First of all, check RRD Maximum for you 64 bit graphs, set it to something more than 10^10. Then add some more zeroes at the end just to be sure. :)

As for timeouts, if the results are same, it could be something else. Try running spine in debug mode (-V=5 for verbosity, -R for read only, -H=yourhostid). see what will it say. Try also running tcpdump -vvv host youhostip for looking into other side.


Working on these steps now to see what happens, It weird how it only messes up at the 550 mb mark
i added an image ..

both these graphs are polled from identically configured servers - dell builds both running the same NIC drivers

the only difference the server having issues has a higher WAN fiber pipe

i have a bandwidth monitor ap running on the server so i can see the server is using over 1 Gbps but its not graphing


Attachments:
graphs.png
graphs.png [ 41.94 KiB | Viewed 5088 times ]
Top
 Profile  
 
 Post subject: Re: Fix 64bit counters in Interface Traffic graphs
PostPosted: Wed Mar 25, 2015 12:11 am 
Offline

Joined: Sun Mar 22, 2015 11:21 pm
Posts: 12
coxxy wrote:
shd wrote:
coxxy wrote:
I tried to make a 64bit graph with the same results, Im using a Dell Power edge server R220 running server 2012 with a 10 Gig Broadcomm NIC with a 3 Gbps Fiber connection

when it goes over 1Gbps the graph starts over so 1.2Gbps shows as 200Mbps

as far as i can tell its snmp v2

Fix64bit was intended to overcome problems with 32bit counters overflowing near 100Mbps. 1.2Gbps is something else. First of all, check RRD Maximum for you 64 bit graphs, set it to something more than 10^10. Then add some more zeroes at the end just to be sure. :)

As for timeouts, if the results are same, it could be something else. Try running spine in debug mode (-V=5 for verbosity, -R for read only, -H=yourhostid). see what will it say. Try also running tcpdump -vvv host youhostip for looking into other side.


Working on these steps now to see what happens, It weird how it only messes up at the 550 mb mark
i added an image ..

both these graphs are polled from identically configured servers - dell builds both running the same NIC drivers

the only difference the server having issues has a higher WAN fiber pipe

i have a bandwidth monitor ap running on the server so i can see the server is using over 1 Gbps but its not graphing




i tried to change the max values and now i get this in the log

03/25/2015 12:10:01 AM - POLLER: Poller[0] WARNING: Poller Output Table not Empty. Issues Found: 382, Data Sources: traffic_in(DS[47]), traffic_out(DS[47]), traffic_in(DS[48]), traffic_out(DS[48]), traffic_in(DS[49]), traffic_out(DS[49]),

i messed it up somewhere


Top
 Profile  
 
 Post subject: Re: Fix 64bit counters in Interface Traffic graphs
PostPosted: Wed Mar 25, 2015 11:48 pm 
Offline

Joined: Sun Mar 22, 2015 11:21 pm
Posts: 12
fixed the graphs but still only able to hit 600mb then the graph messes up ..so 800 down shows a 200 down ..its weird


Top
 Profile  
 
 Post subject: Re: Fix 64bit counters in Interface Traffic graphs
PostPosted: Thu Mar 26, 2015 8:31 am 
Offline
Cacti User

Joined: Mon Apr 09, 2007 1:33 pm
Posts: 95
Location: Nizhnevartovsk, Russia
coxxy wrote:
fixed the graphs but still only able to hit 600mb then the graph messes up ..so 800 down shows a 200 down ..its weird

What does the command "rrdtool info you-rrd-file-with-that-graph.rrd" say? May be you should make rrd_maximum higher?

_________________
CentOS 6.7 64-bit, 12Gb RAM, 8xCPU, some virtual datastore. 497 hosts, 161598 datasources, avg polltime 33 sec


Top
 Profile  
 
 Post subject: Re: Fix 64bit counters in Interface Traffic graphs
PostPosted: Thu Mar 26, 2015 1:33 pm 
Offline

Joined: Sun Mar 22, 2015 11:21 pm
Posts: 12
shd wrote:
coxxy wrote:
fixed the graphs but still only able to hit 600mb then the graph messes up ..so 800 down shows a 200 down ..its weird

What does the command "rrdtool info you-rrd-file-with-that-graph.rrd" say? May be you should make rrd_maximum higher?


im not 100% familliar with the CLI ...ill have to check what the command outputs and try the rrd max higher


Top
 Profile  
 
 Post subject: Re: Fix 64bit counters in Interface Traffic graphs
PostPosted: Fri Mar 27, 2015 12:56 pm 
Offline

Joined: Sun Mar 22, 2015 11:21 pm
Posts: 12
rrd_version = "0003"
step = 60
last_update = 1427478901
header_size = 1760
ds[traffic_in].index = 0
ds[traffic_in].type = "COUNTER"
ds[traffic_in].minimal_heartbeat = 120
ds[traffic_in].min = 0.0000000000e+00
ds[traffic_in].max = 1.0000000000e+09
ds[traffic_in].last_ds = "2426834031"
ds[traffic_in].value = 3.8675654746e+07
ds[traffic_in].unknown_sec = 0
ds[traffic_out].index = 1
ds[traffic_out].type = "COUNTER"
ds[traffic_out].minimal_heartbeat = 120
ds[traffic_out].min = 0.0000000000e+00
ds[traffic_out].max = 1.0000000000e+09
ds[traffic_out].last_ds = "2549141613"
ds[traffic_out].value = 3.6935149153e+06
ds[traffic_out].unknown_sec = 0
rra[0].cf = "AVERAGE"
rra[0].rows = 10080
rra[0].cur_row = 2924
rra[0].pdp_per_row = 1
rra[0].xff = 5.0000000000e-01
rra[0].cdp_prep[0].value = NaN
rra[0].cdp_prep[0].unknown_datapoints = 0
rra[0].cdp_prep[1].value = NaN
rra[0].cdp_prep[1].unknown_datapoints = 0
rra[1].cf = "AVERAGE"
rra[1].rows = 25920
rra[1].cur_row = 15964
rra[1].pdp_per_row = 5
rra[1].xff = 5.0000000000e-01
rra[1].cdp_prep[0].value = 0.0000000000e+00
rra[1].cdp_prep[0].unknown_datapoints = 0
rra[1].cdp_prep[1].value = 0.0000000000e+00
rra[1].cdp_prep[1].unknown_datapoints = 0
rra[2].cf = "MAX"
rra[2].rows = 10080
rra[2].cur_row = 4455
rra[2].pdp_per_row = 1
rra[2].xff = 5.0000000000e-01
rra[2].cdp_prep[0].value = NaN
rra[2].cdp_prep[0].unknown_datapoints = 0
rra[2].cdp_prep[1].value = NaN
rra[2].cdp_prep[1].unknown_datapoints = 0
rra[3].cf = "MAX"
rra[3].rows = 25920
rra[3].cur_row = 3591
rra[3].pdp_per_row = 5
rra[3].xff = 5.0000000000e-01
rra[3].cdp_prep[0].value = -inf
rra[3].cdp_prep[0].unknown_datapoints = 0
rra[3].cdp_prep[1].value = -inf
rra[3].cdp_prep[1].unknown_datapoints = 0
[email protected]:/usr/share/cacti/cli$


Top
 Profile  
 
 Post subject: Re: Fix 64bit counters in Interface Traffic graphs
PostPosted: Fri Mar 27, 2015 1:26 pm 
Offline
Cacti User

Joined: Mon Apr 09, 2007 1:33 pm
Posts: 95
Location: Nizhnevartovsk, Russia
coxxy wrote:
rrd_version = "0003"
ds[traffic_in].index = 0
ds[traffic_in].max = 1.0000000000e+09
ds[traffic_in].last_ds = "2426834031"
ds[traffic_in].value = 3.8675654746e+07
ds[traffic_in].unknown_sec = 0
ds[traffic_out].index = 1
ds[traffic_out].max = 1.0000000000e+09
ds[traffic_out].last_ds = "2549141613"
ds[traffic_out].value = 3.6935149153e+06
[email protected]:/usr/share/cacti/cli$

Try changing rrdtool maximum:
Code:
rrdtool tune your-rrd-file-with-that-graph.rrd -a traffic_in:1.0e+12
rrdtool tune your-rrd-file-with-that-graph.rrd -a traffic_out:1.0e+12

_________________
CentOS 6.7 64-bit, 12Gb RAM, 8xCPU, some virtual datastore. 497 hosts, 161598 datasources, avg polltime 33 sec


Top
 Profile  
 
 Post subject: Re: Fix 64bit counters in Interface Traffic graphs
PostPosted: Fri Mar 27, 2015 2:12 pm 
Offline

Joined: Sun Mar 22, 2015 11:21 pm
Posts: 12
Applied the change just gotta wait and see if it took it :) ...fingers crossed !!!


Top
 Profile  
 
 Post subject: Re: Fix 64bit counters in Interface Traffic graphs
PostPosted: Sat Mar 28, 2015 12:32 am 
Offline

Joined: Sun Mar 22, 2015 11:21 pm
Posts: 12
Seems to still do it ...i restarted the server so will keep watching it see if it gets better .


Attachments:
Untitled.png
Untitled.png [ 66.09 KiB | Viewed 5052 times ]
Top
 Profile  
 
 Post subject: Re: Fix 64bit counters in Interface Traffic graphs
PostPosted: Sat Mar 28, 2015 2:42 am 
Offline

Joined: Sun Mar 22, 2015 11:21 pm
Posts: 12
coxxy wrote:
Seems to still do it ...i restarted the server so will keep watching it see if it gets better .


DEF:a='/var/lib/cacti/rra/*rrd-file*_traffic_in_484.rrd':'traffic_out':AVERAGE \
DEF:b='/var/lib/cacti/rra/*rrd-file*_traffic_in_484.rrd':'traffic_in':AVERAGE \


When i do a graph debug it shows the 484.RRD but the graph number is 485 ...i ran the command on the 484.rrd ...does that sound right?


Top
 Profile  
 
 Post subject: Re: Fix 64bit counters in Interface Traffic graphs
PostPosted: Fri Jun 05, 2015 12:18 pm 
Offline

Joined: Fri Jun 05, 2015 12:12 pm
Posts: 1
I installed it, but graphs do nothing when I try to convert them, and the button does not show up. Logfile seems to indicate it tries to work. Odd.


Top
 Profile  
 
 Post subject: Re: Fix 64bit counters in Interface Traffic graphs
PostPosted: Fri May 06, 2016 2:13 pm 
Offline

Joined: Fri May 06, 2016 2:11 pm
Posts: 1
Awesome plugin.. Thanks!


Top
 Profile  
 
 Post subject: Re: Fix 64bit counters in Interface Traffic graphs
PostPosted: Fri Oct 20, 2017 10:28 am 
Offline

Joined: Fri Oct 20, 2017 9:45 am
Posts: 4
Hello guys,

thanks to author for all the work and everybody else for contributing!

I'm planning to use this fine plugin to convert several hundreds cacti graphs on our production network, but before doing that, I need to be sure nothing will go wrong.
I will greatly appreciate quick and precise answers.

Q1: How much time it takes for this plugin to convert, lets say, one thousand graps? In order not to lose any data, conversion must take less than 5-minutes poll time.
Q2: If something went not as planned, is there a way to revert to old 32-bit graphs without losing any data? Should I somehow backup something beforehand, and what exactly?
Q3: Is there a way to find out all 32-bit graps and convert those in 1-click?

Thanks in advance.


Top
 Profile  
 
 Post subject: Re: Fix 64bit counters in Interface Traffic graphs
PostPosted: Sat Oct 21, 2017 11:41 am 
Offline
Cacti User

Joined: Mon Apr 09, 2007 1:33 pm
Posts: 95
Location: Nizhnevartovsk, Russia
Yas wrote:
Q1: How much time it takes for this plugin to convert, lets say, one thousand graps? In order not to lose any data, conversion must take less than 5-minutes poll time.

Depends on your setup, but it's quite fast. Couple of SQL updates and rrdtool tune command, which doesn't recreate graph, just updates one parameter in rrd file. A couple thousand of fixes should be performed quite fast. I think of seconds, not minutes.

Yas wrote:
Q2: If something went not as planned, is there a way to revert to old 32-bit graphs without losing any data? Should I somehow backup something beforehand, and what exactly?

You should make database backup and rrd folder backup. The fixing process has some level of robustness, I haven't heard of any data corruption yet, but it's better to be on a safe side.

Yas wrote:
Q3: Is there a way to find out all 32-bit graps and convert those in 1-click?

Check that you have the plugin configured in "Settings", select graphs that are matching your 32-bit template, select "Fix 64 bit counters" action, click Go and it will show you which graphs it will be converting. Plugin performs some checks before adding graph to the fixing queue.

_________________
CentOS 6.7 64-bit, 12Gb RAM, 8xCPU, some virtual datastore. 497 hosts, 161598 datasources, avg polltime 33 sec


Top
 Profile  
 
 Post subject: Re: Fix 64bit counters in Interface Traffic graphs
PostPosted: Wed Oct 25, 2017 4:31 am 
Offline

Joined: Fri Oct 20, 2017 9:45 am
Posts: 4
Thanks for clarification, conversion indeed went well without corrupting any data, we did not lose any statistics.
But for some reason we failed to batch-convert more than 1 interface graphs at once (cacti -> Graph Management -> select what we need and from pull-down menu choose "Fix 64bit counters" -> ok).
One-by-one conversion using a button to the right from a graph is perfectly fine, though.
Where should I start troubleshooting this inconvenience?
Cacti 0.8.7e, PA 2.6, fix64bit 0.4.

PS. No, we do not use thold plugin, and also cacti admin would rather not touch what is already working, so upgrading to newer version of cacti probably will not going to happen; is there any other way?


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 110 posts ]  Go to page Previous  1 ... 4, 5, 6, 7, 8  Next

All times are UTC - 5 hours


Who is online

Users browsing this forum: No registered users and 1 guest


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  

Protected by Anti-Spam ACP Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group