Re: [pmacct-discussion] Virtual interfaces setup problem
Dear Paolo, Thanks a lot! That actually worked great. Traffic is being counted just the way I wanted it to be. As for your questions: a) I suppose an excerpt of my ifconfig will explain our setup best: eth1 Link encap:Ethernet Hardware Adresse fa:4d: c2:6f:92:01 inet Adresse:213.131.x.x Bcast:213.131.x.x Maske:255.255.255.x eth1:1Link encap:Ethernet Hardware Adresse fa:4d:c2:6c:92:09 inet Adresse:213.131.x.x Bcast:213.131.x.x Maske:255.255.255.x eth1:2Link encap:Ethernet Hardware Adresse fa:4d: c2:6c:92:02 inet Adresse:78.138.x.x Bcast:78.138.x.x Maske:255.255.255.x eth1:3Link encap:Ethernet Hardware Adresse fa:4d: c2:6c:88:03 inet Adresse:213.203.x.x Bcast:213.203.208.x Maske:255.255.255.x As you can see some interfaces are within the same subnet, others are in different ones. Does this answer the question? b) I've given it a try with tcpdump -i eth1 and as it seems all the addresses of the eth1 and eth1:x are being captured. Thank you very much again for your really helpful advice. I will of course check out the FAQ again but any further advice on how to proceed from here is still very much appreciated. Kind regards, Stefan -Ursprüngliche Nachricht- Von: pmacct-discussion [mailto:pmacct-discussion-boun...@pmacct.net] Im Auftrag von Paolo Lucente Gesendet: Donnerstag, 3. April 2014 07:50 An: pmacct-discussion@pmacct.net Betreff: Re: [pmacct-discussion] Virtual interfaces setup problem Hi Stefan, We crossed each other's email, very good. So if in your config you replace the aggregate: src_host,dst_host,proto line with: aggregate[in]: dst_host aggregate[out]: src_host ! imt_path[in]: /path/to/in.pipe imt_path[out]: /path/to/out.pipe Then query the memory tables: shell pmacct -s -p /path/to/in.pipe shell pmacct -s -p /path/to/out.pipe Do you achieve what you want? About how that will work with all the 50 virtual interfaces requires you to test and answer to my a), b) questions in the previous email. Cheers, Paolo On Thu, Apr 03, 2014 at 07:33:54AM +0200, Whisky wrote: Hi Mario. Thanks again for your reply. I'm really sorry to bother the list but I obviously don't really understand the configuration options although I've managed to narrow down the results I get. So I suggest we leave the MySQL problem aside for the time being. Here's my config on a local test system (hence the local ip addresses): daemonize: true pidfile: /var/run/pmacctd.pid syslog: daemon promisc: true interface: eth0 plugins: memory,memory[in], memory[out] plugin_pipe_size:1024000 plugin_buffer_size:8192 imt_buckets: 65537 imt_mem_pools_size: 1024000 aggregate: src_host,dst_host,proto aggregate_filter[in]: dst net 192.168.1.5 aggregate_filter[out]: src net 192.168.1.5 The ip address of eth0 is 192.168.1.5. As mentioned before all I want know is how much incoming and outgoing traffic is generated for that ip. Here's an excerpt of what I get as a result of pmacct -s: SRC_IP DST_IP PROTOCOLPACKETS BYTES 192.168.1.5195.20.242.89 tcp 142134 192.168.1.5192.168.1.75 tcp 490 77648 192.168.1.5192.168.1.1 udp 201430 192.168.1.5192.168.1.1 tcp 73021 8940812 192.168.1.5212.211.132.32 tcp 235 13626 192.168.1.5144.76.109.57 tcp 132 18032 192.168.1.5192.168.1.255 udp 4 964 192.168.1.5198.20.8.246 tcp 2711265 192.168.1.5198.20.8.241 tcp 323093 192.168.1.5141.76.2.4 tcp 8611184 As you can see 192.168.1.5 doesn't show up under DST_IP but in my opinion it should, because there obviously has to be incoming traffic. At least my idea was that incoming traffic has to show up under DST_IP - am I wrong here? So my current questions are: 1) How do I also get the incoming traffic? 2) What If that finally works and I wish to monitor about 50 virtual interfaces? Do I need a separate config for each of them? And if so, how do I get the results for each interface? Thank you very much for your patience, Stefan -Ursprüngliche Nachricht- Von: pmacct-discussion [mailto:pmacct-discussion-boun...@pmacct.net] Im Auftrag von Jentsch, Mario Gesendet: Mittwoch, 2. April 2014 11:43 An: pmacct-discussion@pmacct.net Betreff: Re: [pmacct-discussion] Virtual interfaces setup problem
Re: [pmacct-discussion] Virtual interfaces setup problem
Hi Stefan, Great. So the last two changes to make to your config are: interface: eth1 aggregate_filter[in]: dst net 213.131.x.x/y aggregate_filter[out]: src net 213.131.x.x/y You should essentially filter on the supernet that best represents all the IP addresses on the individual virtual interfaces. You can filter over multiple supernets if required. You can build and test filters with tcpdump (see their docs) then copy/paste them as aggregate_filter arguments. Cheers, Paolo On Thu, Apr 03, 2014 at 08:46:06AM +0200, Whisky wrote: Dear Paolo, Thanks a lot! That actually worked great. Traffic is being counted just the way I wanted it to be. As for your questions: a) I suppose an excerpt of my ifconfig will explain our setup best: eth1 Link encap:Ethernet Hardware Adresse fa:4d: c2:6f:92:01 inet Adresse:213.131.x.x Bcast:213.131.x.x Maske:255.255.255.x eth1:1Link encap:Ethernet Hardware Adresse fa:4d:c2:6c:92:09 inet Adresse:213.131.x.x Bcast:213.131.x.x Maske:255.255.255.x eth1:2Link encap:Ethernet Hardware Adresse fa:4d: c2:6c:92:02 inet Adresse:78.138.x.x Bcast:78.138.x.x Maske:255.255.255.x eth1:3Link encap:Ethernet Hardware Adresse fa:4d: c2:6c:88:03 inet Adresse:213.203.x.x Bcast:213.203.208.x Maske:255.255.255.x As you can see some interfaces are within the same subnet, others are in different ones. Does this answer the question? b) I've given it a try with tcpdump -i eth1 and as it seems all the addresses of the eth1 and eth1:x are being captured. Thank you very much again for your really helpful advice. I will of course check out the FAQ again but any further advice on how to proceed from here is still very much appreciated. Kind regards, Stefan -Ursprüngliche Nachricht- Von: pmacct-discussion [mailto:pmacct-discussion-boun...@pmacct.net] Im Auftrag von Paolo Lucente Gesendet: Donnerstag, 3. April 2014 07:50 An: pmacct-discussion@pmacct.net Betreff: Re: [pmacct-discussion] Virtual interfaces setup problem Hi Stefan, We crossed each other's email, very good. So if in your config you replace the aggregate: src_host,dst_host,proto line with: aggregate[in]: dst_host aggregate[out]: src_host ! imt_path[in]: /path/to/in.pipe imt_path[out]: /path/to/out.pipe Then query the memory tables: shell pmacct -s -p /path/to/in.pipe shell pmacct -s -p /path/to/out.pipe Do you achieve what you want? About how that will work with all the 50 virtual interfaces requires you to test and answer to my a), b) questions in the previous email. Cheers, Paolo On Thu, Apr 03, 2014 at 07:33:54AM +0200, Whisky wrote: Hi Mario. Thanks again for your reply. I'm really sorry to bother the list but I obviously don't really understand the configuration options although I've managed to narrow down the results I get. So I suggest we leave the MySQL problem aside for the time being. Here's my config on a local test system (hence the local ip addresses): daemonize: true pidfile: /var/run/pmacctd.pid syslog: daemon promisc: true interface: eth0 plugins: memory,memory[in], memory[out] plugin_pipe_size:1024000 plugin_buffer_size:8192 imt_buckets: 65537 imt_mem_pools_size: 1024000 aggregate: src_host,dst_host,proto aggregate_filter[in]: dst net 192.168.1.5 aggregate_filter[out]: src net 192.168.1.5 The ip address of eth0 is 192.168.1.5. As mentioned before all I want know is how much incoming and outgoing traffic is generated for that ip. Here's an excerpt of what I get as a result of pmacct -s: SRC_IP DST_IP PROTOCOLPACKETS BYTES 192.168.1.5195.20.242.89 tcp 142134 192.168.1.5192.168.1.75 tcp 490 77648 192.168.1.5192.168.1.1 udp 201430 192.168.1.5192.168.1.1 tcp 73021 8940812 192.168.1.5212.211.132.32 tcp 235 13626 192.168.1.5144.76.109.57 tcp 132 18032 192.168.1.5192.168.1.255 udp 4 964 192.168.1.5198.20.8.246 tcp 2711265 192.168.1.5198.20.8.241 tcp 323093 192.168.1.5141.76.2.4 tcp 8611184 As you can see 192.168.1.5 doesn't show up under DST_IP but in my opinion it should, because there obviously has to be incoming traffic. At least my idea was that incoming
Re: [pmacct-discussion] Virtual interfaces setup problem
Hey Stefan, I use nfacctd with a custom plugin, never used pmacctd nor one of the SQL plugins. My guess is that you don't use aggregate[]: none and sql_history[]: 1d. Whatever configuration proposal I give you is a shot in the dark. To have the data of all interfaces in one table I would create a pretag map file for each of them, setting the tag to an interface index I choose (e.g. tag=1 for eth0, tag=2 for eth1, ...) and use the tag primitive in the aggregate directive. MySQL table: CREATE TABLE `if_daily` ( `agent_id` INT(10) UNSIGNED NOT NULL, `stamp_inserted` DATETIME NOT NULL, `packets` INT(10) UNSIGNED NOT NULL, `bytes` BIGINT(20) UNSIGNED NOT NULL, `stamp_updated` DATETIME NULL DEFAULT NULL, PRIMARY KEY (`agent_id`, `stamp_inserted`) ) File pretag-eth0.map file: ! set_tag: 1 ! The configuration file: ! plugins: mysql ! sql_optimize_clauses: true sql_history: 1d sql_history_roundoff: d sql_table: if_daily sql_refresh_time: 300 ! aggregate: tag interface: eth0 pre_tag_map: pretag-eth0.map ! plugin_pipe_size: 1024 plugin_buffer_size: 102400 ! sql_host: mysqld-host sql_db: db-name sql_user: db-username sql_passwd: db-password ! agent_id in the MySQL table contains the interface id as of the pre-tagging. Data is updated every 300 seconds. Regards, Mario -Original Message- From: pmacct-discussion [mailto:pmacct-discussion-boun...@pmacct.net] On Behalf Of Whisky Sent: Dienstag, 1. April 2014 16:16 To: pmacct-discussion@pmacct.net Subject: Re: [pmacct-discussion] Virtual interfaces setup problem Hi Mario. Thanks for your message. I think what would help me most would be a correct configuration. As I said, I only need the total in and out values for one specific interface without ports or protocol. Shouldn't be too difficult but I just can't figure out how a fitting config would look like. Regards, Stefan ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] Virtual interfaces setup problem
Hi Stefan, Two questions for you: a) is each virtual interface configured with a different IP address (range) or is it placed in a different VLAN? b) if you bind pmacctd (or tcpdump) to the main/real interface, say, eth1 - are you able to see all in/out traffic of all eth1 virtual interfaces? Answer to these questions can help simplify your setup. Another way to proceed is you can share your current config since you got data - but as you say it's too detailed so it suggests you have simply put too much or are not filtering foreign IP addresses out. Finally, Q8 of the FAQS (*) document gives high-level orientation how you should be configuring pmacctd for scenarios like the one you are facing. Cheers, Paolo (*) http://wiki.pmacct.net/OfficialFAQs On Mon, Mar 31, 2014 at 12:01:17PM +0200, Whisky wrote: Dear pmacctd-list members. I've already tried to set up pmacct the way I need it but it seems I'm just too dumb to get it. Her's what I try to accomplish: We own several servers each of which has multiple IP addresses and accordingly multiple virtual network interfaces (up to 50 like eth1:1, eth1:2, etc). We would like to count the traffic per interface on a daily basis and we also wish to be able to generate weekly/monthly/yearly reports. A distinction between services / protocols is currently not necessary. We just want to know how much inbound and outbound traffic a single ip/interface has produced in total per day. We don't even need fancy diagrams although they would certainly be nice:) My first try with the MySQL plugin went ok but the results were way to detailed and the database grew to 450 MB within 17 days on a server that is not even actively used. As I am quite sure that I am not the only one with such a setup I kindly ask the list for help and/or example configurations. Any hint would be appreciated. Thank you very much in advance, Stefan ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
