Re: Traffic Accounting
Greetings! On 19 Jul 2003 23:35:08 +0300 kgb [EMAIL PROTECTED] wrote: Which is best way for traffic accounting i use ipac-ng but i don't like it anymore because it make my system under high load. If you don't want to mess around with IPtables just to do traffic accounting, you could try http://wyae.de/software/trafan/ which works even from a third machine - just plug in and be happy. I do not have any experiences with high load scenarios, though. Bye Volker Tanger -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Traffic Accounting
Volker Tanger wrote: Greetings! On 19 Jul 2003 23:35:08 +0300 kgb [EMAIL PROTECTED] wrote: Which is best way for traffic accounting i use ipac-ng but i don't like it anymore because it make my system under high load. If you don't want to mess around with IPtables just to do traffic accounting, you could try http://wyae.de/software/trafan/ which works even from a third machine - just plug in and be happy. I do not have any experiences with high load scenarios, though. Don't use it. I've been through many open source and self-made IP accounting tools, and using tcpdump is not what one would like. It gets really messy on high throughput. The greatest problem with ipac-ng is it's resource consumption under high loads. I've been through all of this, and built my own package. It uses iptables, because it's easy to set up and got relatively fast lookup times, a C program to parse iptables output and write database files, and some small shell/awk scripts to summarize the database. Data is stored inside a directory tree, nearly no data is looked up/parsed from that, and it's laid out that it's easy to summarize on a monthly basis. It works for me (on an E3) and at some customers' sites for at least 1.5 years, basically unchanged. System load maximizes at ~1.5 on a 1100 Athlon w/ 3xIntel eepro and 3 slow IDE HDDs. I'm planning to separate all those accounting chains by class-c though, this should speed up both kernel lookup latency and iptables output. I can make my scripts available, but (as it's not packaged in any way), only on personal request. Thomas -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Traffic Accounting
On 21 Jul 2003 at 8:50, Volker Tanger wrote: On 19 Jul 2003 23:35:08 +0300 kgb [EMAIL PROTECTED] wrote: Which is best way for traffic accounting i use ipac-ng but i don't like it anymore because it make my system under high load. If you don't want to mess around with IPtables just to do traffic accounting, you could try http://wyae.de/software/trafan/ which works even from a third machine - just plug in and be happy. I do not have any experiences with high load scenarios, though. Or have you maybe given netacctd a thought? Works fine here - even with a constant stream of about 30 MBit on the wire ... sometimes even higher. http://exorsus.net/projects/net-acct/ It can report traffic in regular intervals and write them to disk. Then you can write a separate tool to sum up the information you like before writing them to a database. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Traffic Accounting
Greetings! On Mon, 21 Jul 2003 10:20:05 +0200 Thomas Lamy [EMAIL PROTECTED] wrote: Don't use it. I've been through many open source and self-made IP accounting tools, and using tcpdump is not what one would like. It gets really messy on high throughput. Messy as in higher load than IPtables or as in packet drops - or how? Can you hint me at some ressources (URLs) on this? Thanks a lot for your input Volker Tanger PS: TrafAn was a quick-shot designed to give a rough estimate on intra-network protocol usage e.g. plugged into a SPAN-port of a switch. So using it for accounting is more a by-product... -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Traffic Accounting
Hello Russian Secret Service Agent... Am 23:35 2003-07-19 +0300 hat kgb geschrieben: Hello, Which is best way for traffic accounting i use ipac-ng but i don't like it anymore because it make my system under high load. Thanks in advanced. I think there is no other choice... I use ipac on a 100 MBit LAN where I count the traffic of five 11 MBit WaveLAN-Channels... where ipac has two NIC's and is In-Line between the Main-Router and the Switch where the Lucent ORINOCO COR-1100 and wireless Bridges are connected... Each channel has 120 Clients... I use a AMD Athlon XP 2400+ with 512 MByte of memory and the load is around 17... I have for each client (all fixed IP's) two rules (rx/tx) to the Internet and two rules (rx/tx) to the internal mail-Server. So I have completly 2400 rules plus som special-rules to count ftp, http, shttp and mail traffic. In summary around 2500 rules. What Do you have ??? Thanks Michelle -- Registered Linux-User #280138 with the Linux Counter, http://counter.li.org. +--+ | Michelle's Internet-ServiceInh. Michelle Konzack| | FunkLAN-Providerin | +--+ -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Traffic Accounting
On Sun, 2003-07-20 at 15:58, Michelle Konzack wrote: Hello Russian Secret Service Agent... Am 23:35 2003-07-19 +0300 hat kgb geschrieben: Hello, Which is best way for traffic accounting i use ipac-ng but i don't like it anymore because it make my system under high load. Thanks in advanced. I think there is no other choice... I use ipac on a 100 MBit LAN where I count the traffic of five 11 MBit WaveLAN-Channels... where ipac has two NIC's and is In-Line between the Main-Router and the Switch where the Lucent ORINOCO COR-1100 and wireless Bridges are connected... Each channel has 120 Clients... I use a AMD Athlon XP 2400+ with 512 MByte of memory and the load is around 17... I have for each client (all fixed IP's) two rules (rx/tx) to the Internet and two rules (rx/tx) to the internal mail-Server. So I have completly 2400 rules plus som special-rules to count ftp, http, shttp and mail traffic. In summary around 2500 rules. What Do you have ??? Thanks Michelle -- Registered Linux-User #280138 with the Linux Counter, http://counter.li.org. +--+ | Michelle's Internet-ServiceInh. Michelle Konzack| | FunkLAN-Providerin | +--+ I have over 2000 rules bgpeer tx/rx, internet tx/rx, local traffic tx/rx machine is AMD Athlon XP 1700+ with 1G ram i forgot how many rules are limit in iptables but when they are so many this is really sucks this is on 100Mbit LAN the problem is when fetchipac is running and ipacsum because file in /var/lib/ipac-ng/data.db is over 5G when file i smaller traffic is smaller or fetchipac and ipacsum is not running everything is fine i think thats can not be the only one way... -- Feci quod potui, faciant meliora potentes! signature.asc Description: This is a digitally signed message part
Re: Traffic Accounting
On Sun, Jul 20, 2003 at 07:01:24PM +0300, kgb wrote: On Sun, 2003-07-20 at 15:58, Michelle Konzack wrote: Hello Russian Secret Service Agent... Am 23:35 2003-07-19 +0300 hat kgb geschrieben: Hello, Which is best way for traffic accounting i use ipac-ng but i don't like it anymore because it make my system under high load. Thanks in advanced. I think there is no other choice... I use ipac on a 100 MBit LAN where I count the traffic of five 11 MBit WaveLAN-Channels... where ipac has two NIC's and is In-Line between the Main-Router and the Switch where the Lucent ORINOCO COR-1100 and wireless Bridges are connected... Each channel has 120 Clients... I use a AMD Athlon XP 2400+ with 512 MByte of memory and the load is around 17... I have for each client (all fixed IP's) two rules (rx/tx) to the Internet and two rules (rx/tx) to the internal mail-Server. So I have completly 2400 rules plus som special-rules to count ftp, http, shttp and mail traffic. In summary around 2500 rules. What Do you have ??? Thanks Michelle I have over 2000 rules bgpeer tx/rx, internet tx/rx, local traffic tx/rx machine is AMD Athlon XP 1700+ with 1G ram i forgot how many rules are limit in iptables but when they are so many this is really sucks this is on 100Mbit LAN the problem is when fetchipac is running and ipacsum because file in /var/lib/ipac-ng/data.db is over 5G when file i smaller traffic is smaller or fetchipac and ipacsum is not running everything is fine i think thats can not be the only one way... I don't run it, I'm just a by stander; but I bet you are not dealing with cpu issues but disk io. run top and compare system load to your cpu state % idle time. If you've got idle cpu, and load over one, you are most likely dealing with disk speed not cpu time for hardware scsi, striped raid, on 15k rpm disks :-P unfortunatly that's a lot more difficult and expensive than upgrading cpu and ram :-\ // George -- GEORGE GEORGALIS, System Admin/Architectcell: 646-331-2027IXOYE Security Services, Web, Mail,mailto:[EMAIL PROTECTED] Multimedia, DB, DNS and Metrics. http://www.galis.org/george -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Traffic Accounting
On Sun, 2003-07-20 at 19:27, George Georgalis wrote: On Sun, Jul 20, 2003 at 07:01:24PM +0300, kgb wrote: On Sun, 2003-07-20 at 15:58, Michelle Konzack wrote: Hello Russian Secret Service Agent... Am 23:35 2003-07-19 +0300 hat kgb geschrieben: Hello, Which is best way for traffic accounting i use ipac-ng but i don't like it anymore because it make my system under high load. Thanks in advanced. I think there is no other choice... I use ipac on a 100 MBit LAN where I count the traffic of five 11 MBit WaveLAN-Channels... where ipac has two NIC's and is In-Line between the Main-Router and the Switch where the Lucent ORINOCO COR-1100 and wireless Bridges are connected... Each channel has 120 Clients... I use a AMD Athlon XP 2400+ with 512 MByte of memory and the load is around 17... I have for each client (all fixed IP's) two rules (rx/tx) to the Internet and two rules (rx/tx) to the internal mail-Server. So I have completly 2400 rules plus som special-rules to count ftp, http, shttp and mail traffic. In summary around 2500 rules. What Do you have ??? Thanks Michelle I have over 2000 rules bgpeer tx/rx, internet tx/rx, local traffic tx/rx machine is AMD Athlon XP 1700+ with 1G ram i forgot how many rules are limit in iptables but when they are so many this is really sucks this is on 100Mbit LAN the problem is when fetchipac is running and ipacsum because file in /var/lib/ipac-ng/data.db is over 5G when file i smaller traffic is smaller or fetchipac and ipacsum is not running everything is fine i think thats can not be the only one way... I don't run it, I'm just a by stander; but I bet you are not dealing with cpu issues but disk io. run top and compare system load to your cpu state % idle time. If you've got idle cpu, and load over one, you are most likely dealing with disk speed not cpu time for hardware scsi, striped raid, on 15k rpm disks :-P unfortunatly that's a lot more difficult and expensive than upgrading cpu and ram :-\ // George -- GEORGE GEORGALIS, System Admin/Architectcell: 646-331-2027IXOYE Security Services, Web, Mail,mailto:[EMAIL PROTECTED] Multimedia, DB, DNS and Metrics. http://www.galis.org/george Yes, you're right but my question is, is there have other way to do accounting some bash, shell script to fetch traffic with tc command from cbq shaper ? -- Feci quod potui, faciant meliora potentes! signature.asc Description: This is a digitally signed message part
Re: Traffic Accounting
On Sun, Jul 20, 2003 at 08:02:07PM +0300, kgb wrote:
On Sun, 2003-07-20 at 19:27, George Georgalis wrote:
On Sun, Jul 20, 2003 at 07:01:24PM +0300, kgb wrote:
On Sun, 2003-07-20 at 15:58, Michelle Konzack wrote:
Hello Russian Secret Service Agent...
Am 23:35 2003-07-19 +0300 hat kgb geschrieben:
Hello,
Which is best way for traffic accounting i use ipac-ng but i don't like
it anymore because it make my system under high load.
Thanks in advanced.
I think there is no other choice...
I use ipac on a 100 MBit LAN where I count the traffic of five
11 MBit WaveLAN-Channels... where ipac has two NIC's and is
In-Line between the Main-Router and the Switch where the Lucent
ORINOCO COR-1100 and wireless Bridges are connected...
Each channel has 120 Clients...
I use a AMD Athlon XP 2400+ with 512 MByte of memory and the
load is around 17...
I have for each client (all fixed IP's) two rules (rx/tx) to the
Internet and two rules (rx/tx) to the internal mail-Server.
So I have completly 2400 rules plus som special-rules to count
ftp, http, shttp and mail traffic.
In summary around 2500 rules.
What Do you have ???
Thanks
Michelle
I have over 2000 rules bgpeer tx/rx, internet tx/rx, local traffic tx/rx
machine is AMD Athlon XP 1700+ with 1G ram i forgot how many rules are
limit in iptables but when they are so many this is really sucks this is
on 100Mbit LAN the problem is when fetchipac is running and ipacsum because
file in /var/lib/ipac-ng/data.db is over 5G when file i smaller traffic is smaller
or fetchipac and ipacsum is not running everything is fine i think thats can not
be
the only one way...
I don't run it, I'm just a by stander; but I bet you are not dealing
with cpu issues but disk io. run top and compare system load to your cpu
state % idle time.
If you've got idle cpu, and load over one, you are most likely dealing
with disk speed not cpu time for hardware scsi, striped raid, on 15k
rpm disks :-P unfortunatly that's a lot more difficult and expensive
than upgrading cpu and ram :-\
// George
Yes, you're right but my question is, is there have other way to do accounting
some bash, shell script to fetch traffic with tc command from cbq shaper ?
I don't really know that stuff... If you just want to log tcp/udp/icmp
ip use iptables:
iptables -N watchit
iptables -I watchit -s 10.1.0.0/24 -p tcp -m state --state NEW -j LOG --log-prefix
'##_NEW_## '
and periodically do something like
tablestats () {
iptables -vnL ${LOG}/iptablestats-${now}
iptables -t nat -vnL ${LOG}/iptablestats-${now}
}
or you may need qdisc routing and logging, I don't know much about
that. My favorite setup is an ebtables bridging router/fw (has no ip
address), patched to send packets through the netfilter tables. :)
That and iptable stats should probably cover your needs.
Just found these, should help with qdisc:
http://lartc.org/howto/index.html
http://lartc.org/howto/lartc.qdisc.html
http://lartc.org/howto/lartc.bridging.html
These are the links I saved from 6 or 8 months back.
http://plorf.net/linux-ip/html/
Guide to IP Layer Network Administration with Linux
http://users.pandora.be/bart.de.schuymer/ebtables/
http://users.pandora.be/bart.de.schuymer/ebtables/sourcecode.html
Ebtables homepage
http://www.tldp.org/HOWTO/Ethernet-Bridge-netfilter-HOWTO.html
Ethernet Bridge + netfilter Howto
http://www.sparkle-cc.co.uk/firewall/firewall.html
Implementing a Bridging Firewall By David Whitmarsh
http://www.compsci.lyon.edu/mcritch/dante/
Dante - Traffic control and QoS with Linux
http://lartc.org/
Linux Advanced Routing Traffic Control
http://lartc.org/howto/lartc.bridging.html
Building bridges, and pseudo-bridges with Proxy ARP
http://bridge.sourceforge.net/docs.html
http://bridge.sourceforge.net/docs/Firewalling for Free.pdf
Firewalling for Free, by Shawn Grimes.
http://www.pom.gr/ilisepe1/firewall_help.html#5
Transparent Firewall Bridging
http://plorf.net/linux-ip/html/ether-bridging.htm
Address Resolution Protocol and Bridging
http://www.zebra.org/
routing software
Have fun. Let us know what you come up with. :)
// George
--
GEORGE GEORGALIS, System Admin/Architectcell: 646-331-2027IXOYE
Security Services, Web, Mail,mailto:[EMAIL PROTECTED]
Multimedia, DB, DNS and Metrics. http://www.galis.org/george
--
To UNSUBSCRIBE, email to [EMAIL PROTECTED]
with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Traffic Accounting
On Sun, Jul 20, 2003 at 02:02:12PM -0400, George Georgalis wrote:
On Sun, Jul 20, 2003 at 08:02:07PM +0300, kgb wrote:
Yes, you're right but my question is, is there have other way to do accounting
some bash, shell script to fetch traffic with tc command from cbq shaper ?
I don't really know that stuff... If you just want to log tcp/udp/icmp
ip use iptables:
iptables -N watchit
iptables -I watchit -s 10.1.0.0/24 -p tcp -m state --state NEW -j LOG --log-prefix
'##_NEW_## '
and periodically do something like
tablestats () {
iptables -vnL ${LOG}/iptablestats-${now}
iptables -t nat -vnL ${LOG}/iptablestats-${now}
}
or you may need qdisc routing and logging, I don't know much about
that. My favorite setup is an ebtables bridging router/fw (has no ip
address), patched to send packets through the netfilter tables. :)
That and iptable stats should probably cover your needs.
Don't forget to use a good logging program like socklog!
also this is good doc:
On Fri, Jan 31, 2003 at 12:01:31AM -0600, Martin A. Brown wrote:
Yes, ip-cref.{ps,pdf}, and ip-tunnel.{ps,pdf} are immensely helpful.
This is Alexey Kuznetsov's documentation. He's one of the main
kernel developers for the IP network stack (as nearly as I can
tell).
// George
--
GEORGE GEORGALIS, System Admin/Architectcell: 646-331-2027IXOYE
Security Services, Web, Mail,mailto:[EMAIL PROTECTED]
Multimedia, DB, DNS and Metrics. http://www.galis.org/george
--
To UNSUBSCRIBE, email to [EMAIL PROTECTED]
with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Traffic Accounting
Am 12:27 2003-07-20 -0400 hat George Georgalis geschrieben: I don't run it, I'm just a by stander; but I bet you are not dealing with cpu issues but disk io. run top and compare system load to your cpu state % idle time. If you've got idle cpu, and load over one, you are most likely dealing with disk speed not cpu time for hardware scsi, striped raid, on 15k rpm disks :-P unfortunatly that's a lot more difficult and expensive than upgrading cpu and ram :-\ Hmm, I have a very low disk-usage... I save the results all 5 Minutes and this give a very short flash at the HD LED. Oh yes, I hav only a 5400 prm. All work of ipac is done in memory... Michelle -- Registered Linux-User #280138 with the Linux Counter, http://counter.li.org. +--+ | Michelle's Internet-ServiceInh. Michelle Konzack| | FunkLAN-Providerin | +--+ -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: traffic accounting
Hi On Thu, Jan 18, 2001 at 03:16:34PM +0100, Teun Vink wrote: I would like to setup up some sort of traffic accounting in our network. I know how to do this using ipchains rules, but the problem is that our network is completely redundant, so each machine in the network has two gateways (both Debian boxes). Does anybody know of a tool which can automatically combine the accounting of multiple routers into one set of statistics? Well, if you need graphical accounting you can try to stick with Hoth (incidentally written by me ;)). You can stack whatever data you want on the top of each other (the example graph on the page stacks tcp with icmp with irc, what is completely senseless...), so you can stack the traffic of two interfaces as well. It is based on RRDtool to store the data and the rest is a small perl script. See more at: http://joker.rhwd.de/software/hoth Biggest caveat: Not a seamless installation and almost no few docs. And if someone helps me to read the netlink sockets for accounting in Linux 2.4 I will port it as well. I wasn't successful yet in any way, neither in perl nor in python (help is really appreciated! :)).. MfG/Regards, Alexander -- Alexander Reelsen http://joker.rhwd.de [EMAIL PROTECTED] GnuPG: pub 1024D/F0D7313C sub 2048g/6AA2EDDB [EMAIL PROTECTED] 7D44 F4E3 1993 FDDF 552E 7C88 EE9C CBD1 F0D7 313C Securing Debian:http://joker.rhwd.de/doc/Securing-Debian-HOWTO -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: traffic accounting
On Thu, 18 Jan 2001, Alexander Reelsen wrote: Hi On Thu, Jan 18, 2001 at 03:16:34PM +0100, Teun Vink wrote: I would like to setup up some sort of traffic accounting in our network. I know how to do this using ipchains rules, but the problem is that our network is completely redundant, so each machine in the network has two gateways (both Debian boxes). Does anybody know of a tool which can automatically combine the accounting of multiple routers into one set of statistics? Well, if you need graphical accounting you can try to stick with Hoth (incidentally written by me ;)). You can stack whatever data you want on the top of each other (the example graph on the page stacks tcp with icmp with irc, what is completely senseless...), so you can stack the traffic of two interfaces as well. It is based on RRDtool to store the data and the rest is a small perl script. See more at: http://joker.rhwd.de/software/hoth Biggest caveat: Not a seamless installation and almost no few docs. And if someone helps me to read the netlink sockets for accounting in Linux 2.4 I will port it as well. I wasn't successful yet in any way, neither in perl nor in python (help is really appreciated! :)).. MfG/Regards, Alexander Well.. I especially need numbers, since we want to bill excessive traffic :-) But I be sure to take a look! Teun -- Teun Vink - [EMAIL PROTECTED] - icq: 15001247 - http://teun.moonblade.net -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: traffic accounting
On Thu, 18 Jan 2001, Teun Vink wrote: Hi, I would like to setup up some sort of traffic accounting in our network. I know how to do this using ipchains rules, but the problem is that our network is completely redundant, so each machine in the network has two gateways (both Debian boxes). Does anybody know of a tool which can automatically combine the accounting of multiple routers into one set of statistics? There is a tool called 'fipra' which I and a friend developed. it pulls what netblock it should log and to where from a mysql server. You can find it out on the net and it works with linux kernels up to 2.2.16.. I have a new patch done that works with later 2.2.x kernels and I will push that out before the weekend. it can easily handle accounting of 5000 ip's traffic att 30mbit or more, depending on the speed of the machine. Regards Roger A -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
RE: traffic accounting
I use fiprad ( Fast IP router accounting daemon) for logging traffic from multiple gateways to a central mSQL server. It uses stuff all CPU. I am very impressed with it. I have added a few small things of my own such as an fiprad.rc start/stop script and am working on some PHP scripts for interacting with the data on the mSQL server and a few other basic things. I intend to offer everything I have done to the maintainers of the package, so it can be included, if they dont produce something first that is. http://www.umplug.org/fipra/ Cheers, Richard -Original Message- From: Teun Vink [mailto:[EMAIL PROTECTED]] Sent: Friday, 19 January 2001 3:17 a.m. To: [EMAIL PROTECTED] Subject: traffic accounting Hi, I would like to setup up some sort of traffic accounting in our network. I know how to do this using ipchains rules, but the problem is that our network is completely redundant, so each machine in the network has two gateways (both Debian boxes). Does anybody know of a tool which can automatically combine the accounting of multiple routers into one set of statistics? Regards, Teun -- Teun Vink - [EMAIL PROTECTED] - icq: 15001247 - http://teun.moonblade.net -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED] -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
