Cacti (home)ForumsDocumentation
Cacti: offical forums and support
It is currently Thu Jul 20, 2017 5:56 pm

All times are UTC - 5 hours




Post new topic Reply to topic  [ 11 posts ] 

Are you interested in this project?
Sure! 100%  100%  [ 12 ]
No 0%  0%  [ 0 ]
Total votes : 12
Author Message
 Post subject: Netflow plugin
PostPosted: Tue Oct 08, 2013 4:30 am 
Offline
User avatar

Joined: Tue Oct 08, 2013 4:04 am
Posts: 20
Hello everyone. I have developed new plugin for cacti with name "NetFlow". I was inspired by other development - "FlowViewer". TBH: I even tried to contact with that developer, but he not answered on my requests... :-? Last update of FlowViewer says about end of life...

So I want to present you: NetFlow plugin. You can get it on the sourceforge site. It's still beta because of i had no testers for my system. So i would be very glad your tips about it.
I've started develop it in may 2013. It's includes collector's script and web interface. Web interface can be used as standalone, but also i've implemented feature of using as cacti plugin.

Current version is 0.0.7

Link to my project: https://sourceforge.net/projects/netflow/

FreeBSD port: http://master-dl.sourceforge.net/project/netflow/freebsd/nflows.tar.gz
FreeBSD installation instructions: http://master-dl.sourceforge.net/project/netflow/freebsd/README.txt

Centos RPM: http://master-dl.sourceforge.net/project/netflow/rpm/nflows-0.0.7-1.x86_64.rpm
CentOS installation instructions: http://master-dl.sourceforge.net/project/netflow/rpm/README.txt

Examples for Cisco Flexible Netflow configuration: http://forums.cacti.net/viewtopic.php?p=264037#p264037

Windows installation How-To: http://forums.cacti.net/viewtopic.php?p=264518#p264518

CHANGELOG:

0.0.7
27.07.2016 - Added database optimization. Less detalisation, but more performance.
27.07.2016 - Added access lists to avoid transfering from invalid sensors. (look /nflows/collector/threaded.pl -> @allowed variable that 0.0.0.0 by default)
27.07.2016 - Added scales for graphs.
27.07.2016 - Changed database structure.

0.0.6
05.05.2015 - Separated tables for each device should increase perfomance
05.05.2015 - Collector cleans old data every hour. No need to use cron - crutches
05.05.2015 - No need to create any table in database. Collector creates all required tables

0.0.5
14.10.2014 - Collector works as daemon now (freebsd)
14.10.2014 - Project can be installed as a port in freebsd (see the freebsd filefolder)
14.10.2014 - Fixed some bugs

0.0.4
07.10.2013 - Integration as plugin Cacti! ( http://www.cacti.net/ )
07.10.2013 - Finished developing time intervals on the web-interface
07.10.2013 - Fixed some bugs

0.0.3
09.09.2013 - Added interfaces discovering via SNMP
09.09.2013 - Improved Netflow-monitor interface
09.09.2013 - Changed database structure
09.09.2013 - Changed README file

0.0.2
20.08.2013 - Fixed chart generator (/nflows/collector/php)
20.08.2013 - Changed/Corrected README.TXT
20.08.2013 - Deprecated "mysql_" methods changed to "mysqli_"
20.08.2013 - Project moved in to "Netflow" project

0.0.1
19.08.2013 - Fixed a lot of bugs

My plans:

1) Implement Ipv6 protocol
2) Improve performance
3) Improve Access-lists
4) Improve UI.


Attachments:
output.png
output.png [ 45.89 KiB | Viewed 4212 times ]
input.png
input.png [ 46.39 KiB | Viewed 4212 times ]

_________________
My NetFlow monitor: https://sourceforge.net/projects/netflow/


Last edited by DreamHunter on Fri Feb 17, 2017 12:45 am, edited 9 times in total.
Top
 Profile  
 
 Post subject: Re: Netflow plugin
PostPosted: Wed Oct 09, 2013 10:49 pm 
Offline
Cacti Guru User

Joined: Mon Oct 16, 2006 5:57 am
Posts: 1876
Location: United Kingdom
This plugin looks like it has some real potential, although I can't see any real integration with Cacti as such. By this, I mean no hooks to existing devices etc - unless I am missing something ?

Would be happy to be a tester for this.

_________________
Quote:
Cacti Version 0.8.8b
Cacti OS Ubuntu LTS
RRDTool Version RRDTool 1.4.7
Poller Information
Type SPINE 0.8.8b


Top
 Profile  
 
 Post subject: Re: Netflow plugin
PostPosted: Wed Oct 09, 2013 11:53 pm 
Offline
User avatar

Joined: Tue Oct 08, 2013 4:04 am
Posts: 20
mcutting wrote:
This plugin looks like it has some real potential, although I can't see any real integration with Cacti as such. By this, I mean no hooks to existing devices etc - unless I am missing something ?

Would be happy to be a tester for this.

Basically yes, you right. This system uses separated database and different structure of data.
But there are reasons:
1) NetFlow data structure totally different compare to SNMP data. RRDTools are not applicable here. Also there are different principles of transfering/gathering the data. Impossible to implement Cacti's pooler.
2) My project started as standalone project. I just took a first step towards the development of Cacti plugin. So I want to check the demand for it.

_________________
My NetFlow monitor: https://sourceforge.net/projects/netflow/


Top
 Profile  
 
 Post subject: Re: Netflow plugin
PostPosted: Thu Oct 31, 2013 11:55 pm 
Offline
User avatar

Joined: Tue Oct 08, 2013 4:04 am
Posts: 20
added detailed guide how to install the tool. https://sourceforge.net/p/netflow/wiki/How-to-install/

_________________
My NetFlow monitor: https://sourceforge.net/projects/netflow/


Top
 Profile  
 
 Post subject: Re: Netflow plugin
PostPosted: Thu Apr 03, 2014 2:52 am 
Offline
Cacti User
User avatar

Joined: Mon Oct 02, 2006 1:22 pm
Posts: 193
Location: Belgium
Hi,

I'm also interested.
Can this be run on a seperate box with the web end as a cacti plugin ?
Just concious of the disk/cpu impact of netflows.

_________________
Joris.
http://www.routerjanitor.com


Top
 Profile  
 
 Post subject: Re: Netflow plugin
PostPosted: Mon Sep 05, 2016 9:44 pm 
Offline
User avatar

Joined: Tue Oct 08, 2013 4:04 am
Posts: 20
Wanna inform cacti community that my program got a user friendly ports (FreeBSD) and rpms (Centos). Also there are performance improvements.
Working on Ubuntu release.

_________________
My NetFlow monitor: https://sourceforge.net/projects/netflow/


Top
 Profile  
 
 Post subject: Re: Netflow plugin
PostPosted: Wed Nov 02, 2016 12:01 am 
Offline

Joined: Thu Apr 29, 2010 4:36 pm
Posts: 4
Can this be installed on Windows? I have a customer with a Cacti instance on Windows so curious if this can be added to that setup as well.

Thanks,


Top
 Profile  
 
 Post subject: Re: Netflow plugin
PostPosted: Mon Nov 28, 2016 3:23 am 
Offline
User avatar

Joined: Tue Oct 08, 2013 4:04 am
Posts: 20
weday0 wrote:
Can this be installed on Windows? I have a customer with a Cacti instance on Windows so curious if this can be added to that setup as well.

Thanks,

Yes you can install it in Windows. But you have to do it manually. Later i'll create some kind of "how-to".

in few words: you need install perl and additional modules to be able use the system. Of course Apache, PHP and mysql must be installed as well.

P.S. There are my e-mail address on sourceforge project page. Mail me.

_________________
My NetFlow monitor: https://sourceforge.net/projects/netflow/


Top
 Profile  
 
 Post subject: Re: Netflow plugin
PostPosted: Mon Nov 28, 2016 3:26 am 
Offline
User avatar

Joined: Tue Oct 08, 2013 4:04 am
Posts: 20
Flexible netflow config for my program:

Step1. Create a template.

Cisco ASR:
Code:
!
flow record ipv4flow
match ipv4 source address
match ipv4 destination address
match ipv4 protocol
collect timestamp sys-uptime first
collect timestamp sys-uptime last
collect routing next-hop address ipv4
collect interface input snmp
collect interface output snmp
collect counter bytes
collect counter packets
collect transport source-port
collect transport destination-port
collect transport tcp flags
collect ipv4 tos
collect routing source as
collect routing destination as
collect ipv4 source mask
collect ipv4 destination mask
!

Cisco 65xx:
Code:
!
flow record ipv4flow
match ipv4 tos
match ipv4 protocol
match ipv4 source address
match ipv4 destination address
match transport source-port
match transport destination-port
collect routing source as
collect routing destination as
collect routing next-hop address ipv4
collect ipv4 source mask
collect ipv4 destination mask
collect transport tcp flags
collect interface input
collect interface output
collect counter bytes
collect counter packets
collect timestamp sys-uptime first
collect timestamp sys-uptime last
!


Step 2. Create exporters:

Code:
!
flow exporter ipv4exp1
destination <COLLECTOR_IP_ADDR>
source Loopback0
transport udp 9999
!


Step 3. Create monitor with using exporters and template:

Code:
!
flow monitor ipv4mon
exporter ipv4exp1
cache timeout active 60
record ipv4flow
!


Step 4. Apply settings to interface:

Code:
!
interface XXXXXXXXEthernetx/x/x
ip flow monitor ipv4mon input
ip flow monitor ipv4mon output
!

_________________
My NetFlow monitor: https://sourceforge.net/projects/netflow/


Last edited by DreamHunter on Fri Feb 17, 2017 12:48 am, edited 2 times in total.

Top
 Profile  
 
 Post subject: Re: Netflow plugin
PostPosted: Sun Jan 15, 2017 3:12 am 
Offline
User avatar

Joined: Tue Oct 08, 2013 4:04 am
Posts: 20
Ok there... Now it's time to explain how to launch my program in Windows systems.

1) First we have to download netflow program:
Link to project page: https://sourceforge.net/projects/netflow/?source=directory
Attachment:
1.png
1.png [ 100.38 KiB | Viewed 2659 times ]

Link to downloads page: https://sourceforge.net/projects/netflow/files/?source=navbar
Attachment:
2.png
2.png [ 36.8 KiB | Viewed 2659 times ]


2) Unpack nflows.0.0.7.tar.gz and move unpacked files somewhere. For example into c:\

3) Second we need - perl for windows:
Link to download page: http://strawberryperl.com/
Attachment:
3.png
3.png [ 333.38 KiB | Viewed 2659 times ]


4) Cause of this is windows OS, we have to restart our computer.

5) If we will try to launch our program, we will get the following message:
Attachment:
4.png
4.png [ 10.97 KiB | Viewed 2654 times ]


6) Cause of this is a windows OS, we need to replace some lines in collector script.

Open C:\nflows.0.0.7\collector\threaded.pl in wordpad (not in notepad!!!) and replace the following lines:
Code:
use Proc::Daemon;
use Proc::PID::File;

# Daemonize
if ($ARGV[0] ne "nodaemon") {
    Proc::Daemon::Init();
}

# Exit if daemon already running
if (Proc::PID::File->running()) {
print "The program is already running\n";
exit 0;
}

by this code:
Code:
use Win32::Daemon;

# Tell the OS to start processing the service...
    Win32::Daemon::StartService();

# Wait until the service manager is ready for us to continue...
    while( SERVICE_START_PENDING != Win32::Daemon::State() )
    {
        sleep( 1 );
    }

    # Now let the service manager know that we are running...
    Win32::Daemon::State( SERVICE_RUNNING );


If you willtry install required modules instead, you will get the following error:


Attachments:
5.png
5.png [ 37.04 KiB | Viewed 2653 times ]

_________________
My NetFlow monitor: https://sourceforge.net/projects/netflow/


Last edited by DreamHunter on Fri Feb 17, 2017 12:54 am, edited 2 times in total.
Top
 Profile  
 
 Post subject: Re: Netflow plugin
PostPosted: Sun Jan 15, 2017 3:56 am 
Offline
User avatar

Joined: Tue Oct 08, 2013 4:04 am
Posts: 20
7) Third part is installing MySQL server:

Link for download page: https://dev.mysql.com/downloads/windows/installer/
Attachment:
6.png
6.png [ 57.89 KiB | Viewed 2651 times ]

I advice to choose installation type - Only server. And don't forget about root password!
tip: after install I've got a bug: MySQL server tries to create pid file in the programdata directory:
Code:
2017-01-15T10:50:51.723567Z 0 [ERROR] mysqld: Can't create/write to file 'C:\ProgramData\MySQL\MySQL Server 5.7\Data\noname-??.pid' (Errcode: 2 - No such file or directory)
2017-01-15T10:50:51.723567Z 0 [ERROR] Can't start server: can't create PID file: No such file or directory

So it seems you will need to add some settings to file
C:\ProgramData\MySQL\MySQL Server 5.7\my.cnf
Code:
[mysqld]
tmpdir=c:/temp
pid_file=c:/temp/mysql.pid

this is only because of stupid windows. don't blame MySQL and yourself. (AND DO NOT USE NOTEPAD!!!)

8 Now we need create database for our system:
C:\>cd Program Files\MySQL\MySQL Server 5.7\bin
C:\Program Files\MySQL\MySQL Server 5.7\bin>mysql.exe -u root -p
Enter password:
Code:
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 11
Server version: 5.7.17-log MySQL Community Server (GPL)

Copyright (c) 2000, 2016, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> CREATE DATABASE IF NOT EXISTS `flow` CHARACTER SET utf16;
Code:
Query OK, 1 row affected (0.00 sec)

mysql> USE `flow`;
Code:
Database changed

mysql> CREATE USER 'netflow'@'localhost' IDENTIFIED BY PASSWORD '*993AA45E0B64915AFBD1A5BE5713FD509A8E6C2C';
Code:
Query OK, 0 rows affected (0.00 sec)

mysql> GRANT ALL PRIVILEGES ON `flow` . * TO 'netflow'@'localhost' WITH GRANT OPTION;
Code:
Query OK, 0 rows affected (0.00 sec)

mysql> exit
Code:
Bye


9) Now we can check our collector:

Code:
C:\Program Files\MySQL\MySQL Server 5.7\bin>cd \nflows.0.0.7\collector
C:\nflows.0.0.7\collector>perl threaded.pl

Expected output:
Code:
C:\nflows.0.0.7\collector>perl threaded.pl
Smartmatch is experimental at threaded.pl line 64.
Smartmatch is experimental at threaded.pl line 64.
given is experimental at threaded.pl line 66.
when is experimental at threaded.pl line 67.
when is experimental at threaded.pl line 68.
Connect to DB via socket...
Check database structure:
Table "devices"
1. Field list
   device_id,device_header,device_description,device_data,device_snmpstr - OK
2. Structure check:
   device_id, int(10) unsigned, NO, PRI, , auto_increment - OK
   device_header, varchar(100), YES, , ,  - OK
   device_description, varchar(100), YES, , ,  - OK
   device_data, varchar(100), YES, , ,  - OK
   device_snmpstr, varchar(100), YES, , ,  - OK
Table "devices" - DONE
Table "interfaces"
1. Field list
   id,device_id,interface_id,interface_name,interface_description,interface_moni
toring - OK
2. Structure check:
   id, bigint(20) unsigned, NO, PRI, , auto_increment - OK
   device_id, int(10) unsigned, NO, , ,  - OK
   interface_id, int(10) unsigned, NO, , ,  - OK
   interface_name, varchar(256), YES, , ,  - OK
   interface_description, varchar(256), YES, , ,  - OK
   interface_monitoring, tinyint(1), NO, , 0,  - OK
Table "interfaces" - DONE
Table "ip4temp"
1. Field list
   id,device_id,dtime,srcaddr,dstaddr,nexthop,input,output,dpkts,doctets,srcport
,dstport,tcp_flags,prot,tos,src_as,dst_as,src_mask,dst_mask - OK
2. Structure check:
   id, bigint(20) unsigned, NO, PRI, , auto_increment - OK
   device_id, int(10) unsigned, NO, , ,  - OK
   dtime, int(10) unsigned, YES, , ,  - OK
   srcaddr, int(10) unsigned, YES, , ,  - OK
   dstaddr, int(10) unsigned, YES, , ,  - OK
   nexthop, int(10) unsigned, YES, , ,  - OK
   input, smallint(5) unsigned, YES, , ,  - OK
   output, smallint(5) unsigned, YES, , ,  - OK
   dpkts, int(10) unsigned, YES, , ,  - OK
   doctets, int(10) unsigned, YES, , ,  - OK
   srcport, smallint(5) unsigned, YES, , ,  - OK
   dstport, smallint(5) unsigned, YES, , ,  - OK
   tcp_flags, tinyint(3) unsigned, YES, , ,  - OK
   prot, tinyint(3) unsigned, YES, , ,  - OK
   tos, tinyint(3) unsigned, YES, , ,  - OK
   src_as, smallint(5) unsigned, YES, , ,  - OK
   dst_as, smallint(5) unsigned, YES, , ,  - OK
   src_mask, smallint(5) unsigned, YES, , ,  - OK
   dst_mask, smallint(5) unsigned, YES, , ,  - OK
Table "ip4temp" - DONE
Table "ip4temp1"
1. Field list
   id,device_id,dtime,srcaddr,dstaddr,nexthop,input,output,dpkts,doctets,srcport
,dstport,tcp_flags,prot,tos,src_as,dst_as,src_mask,dst_mask - OK
2. Structure check:
   id, bigint(20) unsigned, NO, PRI, , auto_increment - OK
   device_id, int(10) unsigned, NO, , ,  - OK
   dtime, int(10) unsigned, YES, , ,  - OK
   srcaddr, int(10) unsigned, YES, , ,  - OK
   dstaddr, int(10) unsigned, YES, , ,  - OK
   nexthop, int(10) unsigned, YES, , ,  - OK
   input, smallint(5) unsigned, YES, , ,  - OK
   output, smallint(5) unsigned, YES, , ,  - OK
   dpkts, int(10) unsigned, YES, , ,  - OK
   doctets, int(10) unsigned, YES, , ,  - OK
   srcport, smallint(5) unsigned, YES, , ,  - OK
   dstport, smallint(5) unsigned, YES, , ,  - OK
   tcp_flags, tinyint(3) unsigned, YES, , ,  - OK
   prot, tinyint(3) unsigned, YES, , ,  - OK
   tos, tinyint(3) unsigned, YES, , ,  - OK
   src_as, smallint(5) unsigned, YES, , ,  - OK
   dst_as, smallint(5) unsigned, YES, , ,  - OK
   src_mask, smallint(5) unsigned, YES, , ,  - OK
   dst_mask, smallint(5) unsigned, YES, , ,  - OK
Table "ip4temp1" - DONE


10) Start perl service as daemon:

create service by using sc command:
Code:
sc create netflow binPath= "C:\Strawberry\perl\bin\perl.exe c:\nflows.0.0.7\collector\threaded.pl"


now you have netflow service.

That's it folks! 8)

P.S.: I hate windows. Really... This system are not about server software. This is a system for user applications like a Internet browser, PC games and multimedia players...

_________________
My NetFlow monitor: https://sourceforge.net/projects/netflow/


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 11 posts ] 

All times are UTC - 5 hours


Who is online

Users browsing this forum: No registered users and 2 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  

Protected by Anti-Spam ACP Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group