Cacti (home)ForumsDocumentation
Cacti: offical forums and support
It is currently Fri Apr 26, 2019 8:55 am

All times are UTC - 5 hours




Post new topic Reply to topic  [ 18 posts ]  Go to page 1, 2  Next
Author Message
 Post subject: [HOWTO] Netflow
PostPosted: Thu Mar 09, 2006 10:33 am 
Offline

Joined: Thu Mar 09, 2006 9:38 am
Posts: 2
Ok, so here's my first post, but it seems like a needed one.

Things needed:

flow-tools
flowscan
CUFlow
cacti (I'm using 0.8.6h)


I'm using debian, so I was able to apt-get everything. Please note, though, that you need the testing version of flowscan to be able to use flow-tools.

So here's how it works:
Code:
Netflow from router -> flow-capture -> flowscan -> rrds -> cacti
                                 CUFlow -^


Netflow from Router:

On Cisco -
Code:
ip cef
! Do THIS ON EACH INTERFACE
interface e0/0
  ip route-cache flow
  no ip mroute-cache
!
ip flow-export version 5 peer-as bgp-nexthop
ip flow-export destination 1.2.3.4 9999
! where 1.2.3.4 is your flow colector's IP and 9999 is the port you'll choose


Easy enough...

flow-capture:

apt-get install flow-capture #I love good package management
vi /etc/flow-tools/flow-capture.conf

Add the following:

-z0 -V5 -n 288 -N0 -w /home/flows -E2G 0/0/9801

Feel free to man flow-capture for details. You should change these details to suite your needs. Of main interest is the last number, "9801", this is the port that needs to correspond with your previous entry into your cisco router. -z0 -V5 -N0 is important to have to work correctly with flowscan. -n 288 means to make a file capture every 5 minutes. -w is your save directory.

flowscan:

Pick your favorite method and install flowscan (note: needs to support flow-tools, not just cflowd, which ARE different)
apt-get install -t testing flowscan #good_package_management++

edit /etc/flowscan/flowscan.cf (or wherever your flowscan.cf is)

Change the defaults to:
Code:
FlowFileGlob /home/flows/ft-v05.*
ReportClasses CUFlow
WaitSeconds 30 # change  to 300 if you use a SLOW machine or you're processor paranoid
Verbose 1 # you can change this later when it works for sure

Flowscan didn't come with an init script, so I made one.
Code:
#!/bin/sh
#Description: Start/stop Flowscan

case "$1" in
'start')
/usr/bin/flowscan >> /var/log/flowscan 2>&1 </dev/null & >/dev/null
;;
'stop')
killall -9 flowscan
;;
*)
echo "Usage: $0 {start | stop }"
;;
esac
exit 0


Then, don't forget to add it to your startup scripts.


Optional:
Next, I wanted to keep 95th percentile marks ACCURATE for up to 2 months, so I had to change flowscan's way of making default rrds.

You don't have to do this if you don't care about accurate 95th percentile marks.

Edit your FlowScan.pm

Find the following and change the values to:

RRA:AVERAGE:.5:1:17280
RRA:AVERAGE:.5:6:1536
RRA:AVERAGE:.5:24:2268
RRA:AVERAGE:.5:288:1890
RRA:MAX:.5:24:2268
RRA:MAX:.5:288:1890

This SHOULD keep up to 60 days worth of non-compressed data before it starts to compress it down. Feel free to tell me if I'm wrong about this.


CUFlow:

Install cuflow perl module:
apt-get install -t testing flowscan-cuflow

edit /etc/flowscan/CUFlow.cf
Actually, go man CUFlow on google, as you'll need to customize this, however, I'll give an example:
Code:
Subnet 1.2.3.0/24 #my internal networks
Network 1.2.3.128/25 customera # Customer A
Network 1.2.3.0/25 customerb #Customer B
OutputDir /home/flows/rrds

Comment out the rest of the crap, unless you want a dozen graphs that do a dozen things. This config is where you make different graphs based on IP assignment. Very useful in an ISP environment.



Now, if all goes well, you should be making rrds in /home/flows/rrds

Also, if you want to save your raw flow files, create a directory called "saved" in your /home/flows directory, as flowscan deletes the processed flows if it doesn't find a ./saved directory.


rrds -> cacti:

Feel free to search for your favorite process to do this. There are lots of tutorials on HOW to do this already. I will, however, link my personal favorite.

http://forums.cacti.net/viewtopic.php?t=12202


Also, I just use duplicated the Traffic - (bits/sec, w/ 95th percentile) graph template for the netflows and it looks great!


Attachments:
File comment: A thing of beauty...
netflow-example.JPG
netflow-example.JPG [ 69.58 KiB | Viewed 133720 times ]
Top
 Profile  
 
 Post subject:
PostPosted: Thu Mar 09, 2006 1:43 pm 
Offline

Joined: Thu Mar 09, 2006 9:38 am
Posts: 2
Hmm, maybe this should be in the HOWTO section. oops! :o


Top
 Profile  
 
 Post subject:
PostPosted: Thu Mar 09, 2006 3:51 pm 
Offline
Cacti Guru User
User avatar

Joined: Thu Sep 16, 2004 5:53 am
Posts: 5301
Location: United Kingdom
It's worth mentioning that there are a couple of Unix tools to produce netflow data too (by sniffing a mirrored switch port, for example), if you don't have support for netflow in your routers.

On FreeBSD, check out ng_netflow, and also fprobe on Linux.

I've been using ng_netflow and nfsen for some time to monitor our transit usage better, but I hadn't thought about Cacti integration - that looks cool!

_________________
Weathermap 0.98 is out! & QuickTree 1.0. Superlinks is over there now (and built-in to Cacti 1.x).
Some Other Cacti tweaks, including strip-graphs, icons and snmp/netflow stuff.
(Let me know if you have UK DevOps or Network Ops opportunities, too!)


Top
 Profile  
 
 Post subject:
PostPosted: Mon Mar 13, 2006 7:51 pm 
Offline
Cacti User

Joined: Mon Feb 10, 2003 12:00 pm
Posts: 53
Location: Southern / CA
I'm a bit confused on the editing of the RRD metadata.

I've edited the default cacti RRA settings and basically set the monthly one to have "5 minutes" worth of data by changing the "Rows" to 8928.

I guess I'm confused about the relation between rows and timespan, but I'd think you'd have to change the timespan for the daily RRA?

My monthly RRA (that I thought had 5 minute averages) is:

X-files: 0.5
Steps: 1
Rows: 8928
Timespan: 2678400

Is this wrong?

TIA


Top
 Profile  
 
 Post subject: Netflow Template
PostPosted: Tue Apr 04, 2006 1:42 am 
Offline

Joined: Tue Apr 04, 2006 1:25 am
Posts: 2
Location: Phoenix
Good HOWTO. Thanks for the info. I'm using the
Code:
Netflow from router -> flow-capture -> flowscan -> rrds -> cacti
                                 CUFlow -^
setup and it works great. If anyone has any questions with this setup, feel free to e-mail me.

Do you have any more information about duplicating the Traffic - (bits/sec, w/ 95th percentile) graph template for the netflows? I'm having trouble with the process. I have duplicated the graph template, but I can't seem to figure out where to go from there.

Thanks again for the great HOWTO.

C

_________________
Quis custodiet ipsos custodes?


Top
 Profile  
 
 Post subject:
PostPosted: Sun Apr 09, 2006 4:19 pm 
Offline
User avatar

Joined: Sun Apr 09, 2006 4:16 pm
Posts: 41
Location: Spain
Nice howto...

However I think using ntop plugin for cacti offers much more information and is pretty easy to setup (enabling netflow add-on). Of course, it always depends on how much information you require :)

Regards,


Top
 Profile  
 
 Post subject:
PostPosted: Sun Jun 04, 2006 5:20 am 
Offline
Cacti User

Joined: Thu Feb 16, 2006 9:20 am
Posts: 98
Hello,

i tried the Howto in the follonwing link:
http://www.prolixium.com/sitenews.php?id=482
It looks like this tutorial but when i start flowscan, i have the following thing:

flowscan
2006/06/04 12:07:29 working on file /var/lib/netflow/ft/ft-v05.2006-06-04.120501+0200...
/var/lib/netflow/ft/ft-v05.2006-06-04.120501+0200: Invalid index in cflowd flow file: 0xCF100103! Version 5 flow-export is required with *all* fields being saved.2006/06/04 12:07:29 flowscan-1.020 CUFlow: Cflow::find took 0 wallclock secs ( 0.00 usr + 0.00 sys = 0.00 CPU) for 1980 flow file bytes, flow hit ratio: 0/0
2006/06/04 12:07:29 flowscan-1.020 CUFlow: report took 0 wallclock secs ( 0.00 usr + 0.00 sys = 0.00 CPU)
sleep 30...

When i try a rrdtool info on a rrd file just created, there is UNKN values inside.

rrdtool info total.rrd
filename = "total.rrd"
rrd_version = "0003"
step = 300
last_update = 1149416933
ds[in_bytes].type = "ABSOLUTE"
ds[in_bytes].minimal_heartbeat = 400
ds[in_bytes].min = NaN
ds[in_bytes].max = NaN
ds[in_bytes].last_ds = "UNKN"
ds[in_bytes].value = 0.0000000000e+00
ds[in_bytes].unknown_sec = 0
etc ...

Regards
QWertz


Top
 Profile  
 
 Post subject:
PostPosted: Sun Jun 04, 2006 3:05 pm 
Offline
Cacti User

Joined: Thu Feb 16, 2006 9:20 am
Posts: 98
hello:
here
http://www.onlamp.com/pub/a/bsd/2005/09 ... tml?page=3
I read
If FlowScan complains about an "Invalid index in cflowd flow file," you probably didn't install the newest Flowscan.pm module. This is perhaps the most common error people make with FlowScan. If you have this problem, go get the appropriate version of the module as described earlier.

So what?
what do i have to download or check?
Thanks
Qwertz


Top
 Profile  
 
 Post subject: no monthly graphs!!!
PostPosted: Thu Jul 27, 2006 11:29 pm 
Offline

Joined: Thu Jul 27, 2006 11:28 pm
Posts: 1
it worked just fine for 20 days, now i dont have monthly graphs!!!!

heeeeeelp!


Top
 Profile  
 
 Post subject:
PostPosted: Wed Aug 23, 2006 6:23 am 
Offline
Cacti User

Joined: Thu Feb 16, 2006 9:20 am
Posts: 98
Nice, i am very happy for you.

Which tutorial did you use?

On my side i had nothing in my rrdfile with ubuntu or Debian

Thanks

qwertz


Top
 Profile  
 
 Post subject:
PostPosted: Sun Oct 01, 2006 3:42 pm 
Offline

Joined: Sun Jul 03, 2005 1:30 pm
Posts: 12
Qwertz,

this error in flowscan logs means that CFlow is not compiled with flowtools support
Invalid index in cflowd flow file: 0xCF100103! Version 5 flow-export is required with *all* fields being saved.

to enable cflow support, ensure that you compile cflow from within flow-tools directory, right after you compile flow-tools. you will also see the -OSU flag during the compile process.


Top
 Profile  
 
 Post subject:
PostPosted: Sun Oct 01, 2006 11:55 pm 
Offline

Joined: Sun Jul 03, 2005 1:30 pm
Posts: 12
qwertz,

[[email protected] flow-tools-0.68]$ cd contrib/
[[email protected] contrib]$ tar xzf Cflow-1.051.tar.gz
[[email protected] contrib]$ cd Cflow-1.051
[[email protected] Cflow-1.051]$ perl Makefile.PL
Checking if your kit is complete...
Looks good
Found flow-tools... using "-DOSU -I../../lib -I../../lib/.. -L../../lib -lft -lz".

Anyway, i configured yet another system this morning


Top
 Profile  
 
 Post subject: No RRD file output...
PostPosted: Thu Oct 19, 2006 8:06 pm 
Offline

Joined: Thu Oct 19, 2006 7:59 pm
Posts: 9
Location: Weed, Siskiyou County, CA, USA
I've checked twice and I am using the latest (Debian unstable) version of all relevant packages. I've also run though the how-to steps several times and I am convinced that I did not miss anything.

Flowscan runs, and, it shows <i>sleeping 30...</i> in /var/log/flowscan.log on the scheduled intervals. However, nothing shows up in my netflows/rrds directory. I know that I am getting netflow traffic to my box, as well.

What is happening is files are created in netflows/ every 5 minutes ( -n 288). Following is a ls of my netflows/:

<i>
-rw-r--r-- 1 root root 10708 2006-10-19 17:51 ft-v05.2006-10-19.174952-0700
-rw-r--r-- 1 root root 15892 2006-10-19 17:56 ft-v05.2006-10-19.175118-0700
-rw-r--r-- 1 root root 19796 2006-10-19 18:01 ft-v05.2006-10-19.175617-0700
-rw-r--r-- 1 root root 15188 2006-10-19 18:06 ft-v05.2006-10-19.180116-0700
-rw-r--r-- 1 root root 22100 2006-10-19 18:11 ft-v05.2006-10-19.180615-0700
-rw-r--r-- 1 root root 15188 2006-10-19 18:16 ft-v05.2006-10-19.181113-0700
-rw-r--r-- 1 root root 19604 2006-10-19 18:21 ft-v05.2006-10-19.181613-0700
drwxrwxrwx 2 root root 4096 2006-10-19 09:53 rrds
drwxrwxrwx 2 root root 4096 2006-10-19 10:00 saved
-rw-r--r-- 1 root root 84 2006-10-19 18:21 tmp-v05.2006-10-19.182111-0700
</i>

In case it matters, I am running Ubuntu Edgy and am trying to receive flow data from several WRT-54's. DD-WRT's rflow-collector works fine under windows; but, I am transistioning to Linux.

Please direct me where to troubleshoot from. And, please be more verbose than you would with a longtime 'NIX sysadmin since I am not as skilled as most of you.

Thanks.

____________________

Fixed that problem. In CUFlow.cf, the networks must be subsets of the subnets. See http://www.columbia.edu/acis/networks/a ... UFlow.html for a good description of the cuflow options.


Top
 Profile  
 
 Post subject: How does flowscan know...
PostPosted: Tue Oct 31, 2006 9:10 pm 
Offline

Joined: Tue Oct 31, 2006 7:08 pm
Posts: 2
How does flowscan know where to load the config file from?

Is it compiled into the binary?

I think I'm starting my flowscan binary, but it's not reading my configuration file.

How do I make sure that my startup script tells flowscan where to look for my config?


Top
 Profile  
 
 Post subject: Also...
PostPosted: Wed Nov 01, 2006 9:00 am 
Offline

Joined: Tue Oct 31, 2006 7:08 pm
Posts: 2
This never worked for me

Code:
apt-get install -t testing flowscan-cuflow


I'm looking at the standard and unvirse apt lists and I can't find those anywhere.

Where did you get that?


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 18 posts ]  Go to page 1, 2  Next

All times are UTC - 5 hours


Who is online

Users browsing this forum: No registered users and 2 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  

Protected by Anti-Spam ACP Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group