[lk] Linux Network Connection Statistics

[Peter Teoh]

http://www.mm.kay-mueller.de/constat.html Deutsch Michael Mueller, IT Consulting Dipl. Math. Michael Mueller [Profile] [Training] [Sql Statistics] [Threads] [Solaris Device Drivers] [Linux Connection Statistics] [GDB for Sun CC] [Planets] [Contact] [Links] [Home] Linux Network Connection Statistics This software package allows you to easily log and evaluate network connections under Linux. It obtains its data from the Netfilter Connection Tracking layer of the kernel. This means that it is also able to log data traffic generated by connection-less protocols (except for icmp) such as UDP DNS queries and their respective responses. This software is published under GNU General Public License. It comes with absolutely no warranty. Download Description This package comprises two components: C program ipclog This programs runs as a daemon process and polls the pseudo-file /proc/net/nf_conntrack (or ip_conntrack in older Linux versions) for new and for terminated network connections. These connections are then written to a log file. The date and time of the start and end of the connection, the protocol, the source and desination addresses and the ports are logged for each connection or pseudo-connection (UDP, etc.). The addresses and change of addresses for local network interfaces (lo, eth0, ppp0, etc.) are also recorded. The interface names are used instead of the local addresses in the connection log. This is especially useful with regard to DHCP addresses. Alias names can be asigned for multipe addresses of a single interface using a configuration file. Shell Script constat This shell script evaluates the log files generated by ipclog and displays them in summarized form. The names of known and anticipated protocols, services and hosts can be entered in a lookup file, which is then used by constat. If no lookup file entry exists, the protocols, and services are shown as numbers and the hosts as IP addresses. If you enter all anticipated IP addresses in the lookup file, you will notice any unanticipated addresses immediately. Brief excerpt from ipclog P 2004-08-05-16:41:22 ipclog started H 2004-08-05-16:41:22 prot src dst sport dport fl time I 2004-08-05-16:41:22 UP lo 127.0.0.1 I 2004-08-05-16:41:22 UP eth0 192.168.0.1 I 2004-08-05-16:41:22 UP ppp0 217.235.218.69 O 2004-08-05-16:41:37 udp ppp0 217.237.151.97 32799 53 - - O 2004-08-05-16:41:37 tcp ppp0 212.227.15.145 32786 25 t - O 2004-08-05-16:42:22 tcp 192.168.0.2 212.227.15.149 1601 110 t - O 2004-08-05-16:43:22 tcp 192.168.0.3 eth0 1602 22 e - C 2004-08-05-16:41:37 tcp ppp0 212.227.15.145 32786 25 t 0 S 2004-08-05-16:43:47 tcp 192.168.0.3 eth0 1602 22 t 25 C 2004-08-05-16:42:22 tcp 192.168.0.2 212.227.15.149 1601 110 t 0 C 2004-08-05-16:41:37 udp ppp0 217.237.151.97 32799 53 - 0 O 2004-08-05-16:45:27 udp ppp0 217.237.151.97 32799 53 U - O 2004-08-05-16:45:32 tcp ppp0 212.227.119.68 32788 80 e - S 2004-08-05-16:45:37 tcp ppp0 212.227.119.68 32788 80 w 5 S 2004-08-05-16:45:42 tcp ppp0 212.227.119.68 32788 80 t 10 C 2004-08-05-16:43:47 tcp 192.168.0.3 eth0 1602 22 t 25 O 2004-08-05-16:45:47 tcp ppp0 212.227.119.68 32789 80 c - O 2004-08-05-16:45:47 tcp ppp0 212.227.119.68 32790 80 e - S 2004-08-05-16:45:52 tcp ppp0 212.227.119.68 32790 80 w 5 S 2004-08-05-16:45:57 tcp ppp0 212.227.119.68 32790 80 t 10 C 2004-08-05-16:45:47 tcp ppp0 212.227.119.68 32789 80 c 0 C 2004-08-05-16:45:42 tcp ppp0 212.227.119.68 32788 80 t 10 C 2004-08-05-16:45:57 tcp ppp0 212.227.119.68 32790 80 t 10 C 2004-08-05-16:45:47 udp ppp0 217.237.151.97 32799 53 - 20 P 2004-08-05-16:49:00 ipclog stopped, 0 errors The log shows three local interfaces as "UP": lo, eth0 and ppp0. A number of incoming, outgoing and forwarded connections were logged. The computer used in the example has been configured as a router (using NAT). Each connection has an "open" (O) and a "closed" (C) entry. The last column of the "closed" entry indicates the total connection time in seconds. There is also a "state change" (S) entry (time wait state = t) for tcp connections. The "H" entry indicates the column titles for only the "O", "C" and "S" entries. For more details, see the comments in the source code for ipclog. Other ipclog features If you link ipclog to the name "ipc" (identical to "ipclog -1") you create a user command (rather than a daemon) that indicates all currently open connections: $ ipc H 2004-08-31-12:40:06 prot src dst sport dport fl time O 2004-08-31-12:46:13 50 ppp0 195.135.220.3 - - - - If you are using NAT, ipclog always shows the original sources address of the incoming package and the final destination address of the outgoing package (which may have been modified by NAT). You can use the -N option to also display the outgoing source address (which may have been modified by NAT) and the original incoming destination address. Ipclog is not very well suited to displaying ICMP packages (such as, for example, ping) since the Netfilter layer of the kernal normally only makes these visible in /proc/net/nf_conntrack (or ip_conntrack) for a very brief period of time (i.e. milliseconds). Sample output from constat The call constat or constat -d 2004-08-05 provides an overview of the connections for the current date or a date you specify: scanning for date 2004-08-05 src prot dport dst n nopn avgtm maxtm --------------------------------------------------------------------------- 192.168.0.3 tcp ssh eth0 1 0 25.0 25 nat-pc tcp pop3 my-pop-server 1 0 0.0 0 ppp0 tcp http any 3 0 6.7 10 ppp0 tcp smtp my-smtp-server 1 0 0.0 0 ppp0 udp domain my-nameserver 2 0 10.0 20 0 ERRORS The output provides a summary of the ipclog entries shown earlier. The total number of connections, the number of connections currently open, and the average and maximum connection time are displayed in addition to the source, destination, protocol and port. All of the IP addresses, except 192.168.0.3, were entered in the lookup file, and their names are output. The same applies to all the services used (see the constat.lookup file in the download). This means that any addresses that have not been entered in the lookup file stand out and can be examined more closely. Because browsing Web usually creates a large number of connections with numerous Web servers within a relatively short period of time, all http and https connections are grouped together and referred to as "any" (and can be supressed using "constat -noany"). It is also possible to enter http and https connections in the lookup file and have them displayed individually. Usage usage: constat [options] options: -a: scan whole file -d 'jjjj-mm-dd': specific date -f logfile: scan logfile, default /var/logs/ipclog.log -h: help -noany: no 'any' dst hosts -nc: no name cache lookup Installation Download the file constat.tar.gz and unpack it using "tar xzvf constat.tar.gz". Compile ipclog.c by calling "make". Store ipclog and constat somewhere in the file system where you can call them easily, e.g. in /usr/local/bin. Create a link from ipclog to "ipc" if you want to use the ipc command. Select a "log-path" for the connection log file and start ipclog > log-path Ipclog executes as a daemon process in the background (with the exception of  "ipclog -d"). If you want to use ipclog regularly, you should incorporate it in the system boot process. Ipclog runs under any user as long as it has write permission for its own log file. Edit the contents of the variables LOGFILE and LOOKUPFILE in the shell script constat. LOGFILE should contain the absolute path name of the ipclog log file ("log-path"). LOOKUPFILE should contain the absolute path name of the lookup file. This variable can be empty if you do not want to use a lookup file. Create the lookup file if you want to use this feature. You can use the file in the download as a template. You will find more detailed information in the comments in the download file. You can now call "constat" whenever you want or inspect the log file directly.