A NetFlow probe for pfSense® software

Aim of the NetFlow-pfSense^® tutorial

While the Listening cases in the heart of the Internet network (Snowden case, Prism, French Big Brother, Echelon...) are brought to daylight, we thought it would be interesting to consider THE protocol which without a doubt came in to play for the collecting of network information... Indeed if the tools used to enable a semantic analysis of network frames were most likely customely developed by different intelligence agencies, the NetFlow protocol probably came in to play at least for the frame collecting.   

Then again the configuring of a specific 10G interface on a Cisco 12000 and dedicating it to the frame collecting is within reach of many IOS administrators on the planet... 

We are going to explain you :

1. how the protocol functions
2. how to use NetFlow with pfSense^® software
3. how to implement it on your network 
4. how to exploit the collected data

Our aim isn't to transforme you into a secret agent who works for an occult power or to spy every little movement of your users... but to have a level of control of your network which will enable you to go from an astonished analysis (such as "Apparently Monday morning we had a five minute network cut... What could have happened ?") to a more detailled presentation ("Monday morning at 11:32 we had a network cut of 4'23" because the computer of bobby22.bigbiz.local with the IP 192.168.23.234 initiated over three hundred connexions outwards mainly towards the website mongros.torrent.com where the person downloaded 4Go of data").

Having had the opportunity to go and see this famous «bobby22» with a printed chart showing the results of his transfers and the reason why the network collapsed, I guaranty you that it's : 

1. very funny (to be the Edward Snowden of your own LAN is very amusing !) 
2. very efficient (the user knows that you know which is very deterrent !) 
3. the proof that you control your network (maybe it's what you are paid to do ?)

How the protocol functions :

First of all NetFlow is an IP frame collector which works via routers or switches, themselves imbedded with this protocol.

The network equipement will collect the frames of all the flows which pass through it and will send them (generaly in UDP) to the "collector". The collector will organize and store the traces of frames in a way that follows the indications given by the system administrator. The data will then be available for an analysis of the envents that occurred on your network(s). 

The main premium brands of switches or routers come with their own implementation (compatible with NetFlow) : 

• Jlow or cflowd for Juniper
• NetStream for 3Com/HP
• NetStream for Huawei
• Cflowd for AlcatelLucent 
• Rflow for Ericsson 
• AppFlow for Citrix 

Not forgetting that pfSense is capable of generating NetFlow flows in different versions of the protocol (up to version 9). Linux and different variants of BSD can handle NetFlow, same applies to VMWare.

How to use NetFlow with pfSense^® software

pfSense has a NetFlow support thanks to a pfflowd package which enables the frame collecting and their export to a collector. You just need to set up the pfflowd sensor which is available in the pfSense packages.

Once installed, the packet needs a parameter setting of five variables :

• The collector's IP
• The port used by the collector 
• The IP source used by the collector
• The direction of the filtered frames
• The wanted protocol version of NetFlow (up to version 9) 

The deployment on pfSense^® software is the easiest task of the set up : you only need a few clicks to install the package and it's done !

How to implement NetFlow on your network

We have decided to use a Linux to deploy our NetFlow Collector. More precisely an Ubuntu Server 12.04.LTS which will enable us to benefit from a patched and secured collector for 5 years !

Once your Ubuntu 12.04.LTS is installed - do I need to specify that it's a headless server on which you have only activated SSH with a Public Key authentication. After all this collector will regroup all the frames which passed through your network. It is important to take the necessary measures in order to secure this black box ! 

You took good care of deploying this sensor in a DMZ administration to isolate it as much as possible. Finally its access will be limited by appropriate firewall rules.

Installing flowviewer

We have decided to install flowtools which will enable us to have a collector (flow-capture) and flowviewer as frame analysis graphs. Flowviewer also needs apache to be installed.

You will then need to configure in an appropriate way these different packages.
For the collector part we have used the following setting : 

Here flow-tools will create 287 files per day (approximately one capture file every five minutes). 
Then it will stock this from the directory defined by the variable -w"/var/flows/chabanais".
After the variable defines the storage structure which will be in the form YYYY/YYYY-MM/YYYY-MM-DD/flow-file.
Finally we define the addresses which the collector will listen to in the form of local IP's/distant IP's/port.

After all that you can start the collector and check that it runs correctly with the controls : 

How to exploit the collected data

Now it becomes a lot more difficult because their are many tools which can exploit your NetFlow data. From the basic online control to payable tools worth tens of thousands of euros.

Here we are interested in an OpenSource approach of NetFlow analysis tools implementation.

The approach of the storage and analysis of network frames by NetFlow is in many ways reserved to an elite of System and Network administration which already has a sufficient control on its environment and wants to go to the next level by personaly taking control of the main events !
Once again the OpenSource approach enables to choose good quality products by overcoming excessive licence costs !

Now that you have your pfSense router and a collector, you need to do a proper operating of the stored data. We have decided to choose flowviewer which enables us to have a good analysis tool and can be used to do a good reporting of your network activity.

I will skip the set up which is similar to the deployment of a HTTP server with the CGI setting.

After the set up of this program you will have at your disposal a complet frame analysis tool capable of generating custom graphs or to restore data according to your needs.

Here is the first page of your interface. Take good care of defining your pfSense router in the configuration files. You will then be able to choose it in the appropriate pop-up.

Here is the type of custom graph which the tool is capable of giving you.

As you have just seen : it's extremely precis and very effective. There are many ways to set up your probe and show the desired results...

If you do not wish to waist time with the setting up of your probe, we have NetFlow probes on offer ready to be used on your network !