Re: [ot] Re: Courier traffic accounting
On Sat, 2004-11-20 at 18:03, Henrique de Moraes Holschuh wrote: > Which I suppose is a good reason why we should work towards ASMTP, a 8-bit > clean, mandatory endpoint authenticated SMTP (as in no backscatter, > something using mandatory header signing). There is the possibility of using the current ASMTP (which is available in ESMTP) with SSL client certifcates, thus you would get signed mails. ;o) Regards, Philipp Kern signature.asc Description: This is a digitally signed message part
Re: [ot] Re: Courier traffic accounting
also sprach Henrique de Moraes Holschuh <[EMAIL PROTECTED]> [2004.11.20.1803 +0100]: > Actually... as far as a lot of users are capable of thinking, > that's exactly what SMTP should stand for: "I attach this file and > send it, could it be simpler?". And you know something? I can > see their point. Yes. I do too. > Which I suppose is a good reason why we should work towards ASMTP, > a 8-bit clean, mandatory endpoint authenticated SMTP (as in no > backscatter, something using mandatory header signing). > Beautiful! (it's just a dream, there is no such thing. Which is > fine right now, as chances are someone would have made it using > XML). I vote for WebDAV instead. -- Please do not send copies of list mail to me; I read the list! .''`. martin f. krafft <[EMAIL PROTECTED]> : :' :proud Debian developer, admin, user, and author `. `'` `- Debian - when you have better things to do than fixing a system Invalid/expired PGP subkeys? Use subkeys.pgp.net as keyserver! signature.asc Description: Digital signature
Re: [ot] Re: Courier traffic accounting
On Sat, 20 Nov 2004, martin f krafft wrote: > 478181 kilobytes in a POP3 session... teach those folks that SMTP is > not the simple mass transfer protocol. Actually... as far as a lot of users are capable of thinking, that's exactly what SMTP should stand for: "I attach this file and send it, could it be simpler?". And you know something? I can see their point. Which I suppose is a good reason why we should work towards ASMTP, a 8-bit clean, mandatory endpoint authenticated SMTP (as in no backscatter, something using mandatory header signing). Beautiful! (it's just a dream, there is no such thing. Which is fine right now, as chances are someone would have made it using XML). -- "One disk to rule them all, One disk to find them. One disk to bring them all and in the darkness grind them. In the Land of Redmond where the shadows lie." -- The Silicon Valley Tarot Henrique Holschuh -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: Courier traffic accounting
Hi there Martin! On Sat, 2004-11-20 at 17:21, martin f krafft wrote: > These are bytes. Be aware that this sort of accounting does not > include the respective protocol, or additional TCP, or IP traffic. Oh yes. I ignored them because in the small test session there was only protocol traffic. > I usually calculate 112% up to 100Mb and then 108% when more than > 100Mb has been transferred. With traffic >1Gb, it becomes > negligible. Exactly. Thank you for this information and also for the other replies. Regards, Philipp Kern signature.asc Description: This is a digitally signed message part
[ot] Re: Courier traffic accounting
also sprach Teófilo Ruiz Suárez <[EMAIL PROTECTED]> [2004.11.20.1733 +0100]: > > Nov 20 16:55:22 quebrantahuesos pop3d-ssl: LOGOUT, user=teo, > > ip=[:::217.125.62.238], top=0, retr=478181 > > > > The "retr" field is in KBytes. > > As madduck said in his mail, this are bytes :) Otherwise I'd have to shoot all the people in your addressbook, teo. 478181 kilobytes in a POP3 session... teach those folks that SMTP is not the simple mass transfer protocol. -- Please do not send copies of list mail to me; I read the list! .''`. martin f. krafft <[EMAIL PROTECTED]> : :' :proud Debian developer, admin, user, and author `. `'` `- Debian - when you have better things to do than fixing a system Invalid/expired PGP subkeys? Use subkeys.pgp.net as keyserver! signature.asc Description: Digital signature
Re: Courier traffic accounting
On Sat, Nov 20, 2004 at 05:20:07PM +0100, Teófilo Ruiz Suárez wrote: > On Sat, Nov 20, 2004 at 04:48:49PM +0100, Philipp Kern wrote: > > Dear debian-isp list members, > > > > are there any ways of traffic accounting related to Courier POP3d and > > IMAPd? We need this on a per-domain basis. The accounting for > > incoming/outgoing mail is easy, as our mailserver of choice -- Exim v4 > > -- logs the message size. When looking through Courier's logs I didn't > > notice something similar on the close of the connection. > > > > I would appreciate any hints; if it gets written down once by anything, > > I would write a tool to parse/summarise that stuff. > > I get that out of the box with Courier in Debian Sarge: > > 8< snip 8< > Nov 20 16:53:44 quebrantahuesos pop3d-ssl: LOGIN, user=teo, > ip=[:::217.125.62.238] > Nov 20 16:55:22 quebrantahuesos pop3d-ssl: LOGOUT, user=teo, > ip=[:::217.125.62.238], top=0, retr=478181 > > The "retr" field is in KBytes. As madduck said in his mail, this are bytes :) -- teo - http://blog.eltridente.org "Res publica non dominetur" signature.asc Description: Digital signature
Re: Courier traffic accounting
also sprach Philipp Kern <[EMAIL PROTECTED]> [2004.11.20.1648 +0100]:
> are there any ways of traffic accounting related to Courier POP3d and
> IMAPd? We need this on a per-domain basis. The accounting for
> incoming/outgoing mail is easy, as our mailserver of choice -- Exim v4
> -- logs the message size. When looking through Courier's logs I didn't
> notice something similar on the close of the connection.
gaia pop3d-ssl: LOGOUT, user=x, ip=[:::130.60.75.xxx],
top=0, retr=4253, time=0
imapd-ssl: LOGOUT, user=x, ip=[:::130.60.75.xxx],
headers=4241, body=290514, time=1216, starttls=1
^^
These are bytes. Be aware that this sort of accounting does not
include the respective protocol, or additional TCP, or IP traffic.
I usually calculate 112% up to 100Mb and then 108% when more than
100Mb has been transferred. With traffic >1Gb, it becomes
negligible.
--
Please do not send copies of list mail to me; I read the list!
.''`. martin f. krafft <[EMAIL PROTECTED]>
: :' :proud Debian developer, admin, user, and author
`. `'`
`- Debian - when you have better things to do than fixing a system
Invalid/expired PGP subkeys? Use subkeys.pgp.net as keyserver!
signature.asc
Description: Digital signature
Re: Courier traffic accounting
On Sat, Nov 20, 2004 at 04:48:49PM +0100, Philipp Kern wrote: > Dear debian-isp list members, > > are there any ways of traffic accounting related to Courier POP3d and > IMAPd? We need this on a per-domain basis. The accounting for > incoming/outgoing mail is easy, as our mailserver of choice -- Exim v4 > -- logs the message size. When looking through Courier's logs I didn't > notice something similar on the close of the connection. > > I would appreciate any hints; if it gets written down once by anything, > I would write a tool to parse/summarise that stuff. I get that out of the box with Courier in Debian Sarge: 8< snip 8< Nov 20 16:53:44 quebrantahuesos pop3d-ssl: LOGIN, user=teo, ip=[:::217.125.62.238] Nov 20 16:55:22 quebrantahuesos pop3d-ssl: LOGOUT, user=teo, ip=[:::217.125.62.238], top=0, retr=478181 The "retr" field is in KBytes. Regards, -- teo - http://blog.eltridente.org "Res publica non dominetur" -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Courier traffic accounting
Dear debian-isp list members, are there any ways of traffic accounting related to Courier POP3d and IMAPd? We need this on a per-domain basis. The accounting for incoming/outgoing mail is easy, as our mailserver of choice -- Exim v4 -- logs the message size. When looking through Courier's logs I didn't notice something similar on the close of the connection. I would appreciate any hints; if it gets written down once by anything, I would write a tool to parse/summarise that stuff. Regards, Philipp Kern signature.asc Description: This is a digitally signed message part
Re: Traffic Accounting
Greetings! On Mon, 21 Jul 2003 10:20:05 +0200 Thomas Lamy <[EMAIL PROTECTED]> wrote: > Don't use it. I've been through many open source and self-made IP > accounting tools, and using tcpdump is not what one would like. It > gets really messy on high throughput. "Messy" as in higher load than IPtables or as in packet drops - or how? Can you hint me at some ressources (URLs) on this? Thanks a lot for your input Volker Tanger PS: TrafAn was a quick-shot designed to give a rough estimate on intra-network protocol usage e.g. plugged into a SPAN-port of a switch. So using it for accounting is more a by-product... -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: Traffic Accounting
On 21 Jul 2003 at 8:50, Volker Tanger wrote: > On 19 Jul 2003 23:35:08 +0300 kgb <[EMAIL PROTECTED]> wrote: > > > Which is best way for traffic accounting i use ipac-ng but i don't > > like it anymore because it make my system under high load. > > If you don't want to mess around with IPtables just to do traffic > accounting, you could try > > http://wyae.de/software/trafan/ > > which works even from a third machine - just plug in and be happy. I > do not have any experiences with high load scenarios, though. Or have you maybe given netacctd a thought? Works fine here - even with a constant stream of about 30 MBit on the wire ... sometimes even higher. http://exorsus.net/projects/net-acct/ It can report traffic in regular intervals and write them to disk. Then you can write a separate tool to sum up the information you like before writing them to a database. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: Traffic Accounting
Volker Tanger wrote: > > Greetings! > > On 19 Jul 2003 23:35:08 +0300 kgb <[EMAIL PROTECTED]> wrote: > > > Which is best way for traffic accounting i use ipac-ng but i don't > > like it anymore because it make my system under high load. > > If you don't want to mess around with IPtables just to do traffic > accounting, you could try > > http://wyae.de/software/trafan/ > > which works even from a third machine - just plug in and be > happy. I do > not have any experiences with high load scenarios, though. > Don't use it. I've been through many open source and self-made IP accounting tools, and using tcpdump is not what one would like. It gets really messy on high throughput. The greatest problem with ipac-ng is it's resource consumption under high loads. I've been through all of this, and built my own package. It uses iptables, because it's easy to set up and got relatively fast lookup times, a C program to parse iptables output and write "database" files, and some small shell/awk scripts to summarize the database. Data is stored inside a directory tree, nearly no data is looked up/parsed from that, and it's laid out that it's easy to summarize on a monthly basis. It works for me (on an E3) and at some customers' sites for at least 1.5 years, basically unchanged. System load maximizes at ~1.5 on a 1100 Athlon w/ 3xIntel eepro and 3 slow IDE HDDs. I'm planning to separate all those accounting chains by class-c though, this should speed up both kernel lookup latency and iptables output. I can make my scripts available, but (as it's not packaged in any way), only on personal request. Thomas -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: Traffic Accounting
Greetings! On 19 Jul 2003 23:35:08 +0300 kgb <[EMAIL PROTECTED]> wrote: > Which is best way for traffic accounting i use ipac-ng but i don't > like it anymore because it make my system under high load. If you don't want to mess around with IPtables just to do traffic accounting, you could try http://wyae.de/software/trafan/ which works even from a third machine - just plug in and be happy. I do not have any experiences with high load scenarios, though. Bye Volker Tanger -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: Traffic Accounting
Am 12:27 2003-07-20 -0400 hat George Georgalis geschrieben: >I don't run it, I'm just a by stander; but I bet you are not dealing >with cpu issues but disk io. run top and compare system load to your cpu >state % idle time. > >If you've got idle cpu, and load over one, you are most likely dealing >with disk speed not cpu time for hardware scsi, striped raid, on 15k >rpm disks :-P unfortunatly that's a lot more difficult and expensive >than upgrading cpu and ram :-\ Hmm, I have a very low disk-usage... I save the results all 5 Minutes and this give a very short flash at the HD LED. Oh yes, I hav only a 5400 prm. All work of ipac is done in memory... Michelle -- Registered Linux-User #280138 with the Linux Counter, http://counter.li.org. +--+ | Michelle's Internet-ServiceInh. Michelle Konzack| | FunkLAN-Providerin | +--+ -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: Traffic Accounting
On Sun, Jul 20, 2003 at 02:02:12PM -0400, George Georgalis wrote:
>On Sun, Jul 20, 2003 at 08:02:07PM +0300, kgb wrote:
>>
>>Yes, you're right but my question is, is there have other way to do accounting
>>some bash, shell script to fetch traffic with "tc" command from cbq shaper ?
>
>I don't really know that stuff... If you just want to log tcp/udp/icmp
>ip use iptables:
>
>iptables -N watchit
>iptables -I watchit -s 10.1.0.0/24 -p tcp -m state --state NEW -j LOG --log-prefix
>'##_NEW_## '
>
>and periodically do something like
>
>tablestats () {
>iptables -vnL >>${LOG}/iptablestats-${now}
>iptables -t nat -vnL >>${LOG}/iptablestats-${now}
>}
>
>
>or you may need qdisc routing and logging, I don't know much about
>that. My favorite setup is an ebtables bridging router/fw (has no ip
>address), patched to send packets through the netfilter tables. :)
>That and iptable stats should probably cover your needs.
>
Don't forget to use a good logging program like socklog!
also this is good doc:
On Fri, Jan 31, 2003 at 12:01:31AM -0600, Martin A. Brown wrote:
>Yes, ip-cref.{ps,pdf}, and ip-tunnel.{ps,pdf} are immensely helpful.
>This is Alexey Kuznetsov's documentation. He's one of the main
>kernel developers for the IP network stack (as nearly as I can
>tell).
// George
--
GEORGE GEORGALIS, System Admin/Architectcell: 646-331-2027<
Security Services, Web, Mail,mailto:[EMAIL PROTECTED]
Multimedia, DB, DNS and Metrics. http://www.galis.org/george
--
To UNSUBSCRIBE, email to [EMAIL PROTECTED]
with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: Traffic Accounting
On Sun, Jul 20, 2003 at 08:02:07PM +0300, kgb wrote:
>On Sun, 2003-07-20 at 19:27, George Georgalis wrote:
>> On Sun, Jul 20, 2003 at 07:01:24PM +0300, kgb wrote:
>> >On Sun, 2003-07-20 at 15:58, Michelle Konzack wrote:
>> >> Hello Russian Secret Service Agent...
>> >>
>> >> Am 23:35 2003-07-19 +0300 hat kgb geschrieben:
>> >> >
>> >> >Hello,
>> >> >
>> >> >Which is best way for traffic accounting i use ipac-ng but i don't like
>> >> >it anymore because it make my system under high load.
>> >> >
>> >> >Thanks in advanced.
>> >>
>> >> I think there is no other choice...
>> >>
>> >> I use ipac on a 100 MBit LAN where I count the traffic of five
>> >> 11 MBit WaveLAN-Channels... where ipac has two NIC's and is
>> >> In-Line between the Main-Router and the Switch where the Lucent
>> >> ORINOCO COR-1100 and wireless Bridges are connected...
>> >>
>> >> Each channel has 120 Clients...
>> >>
>> >> I use a AMD Athlon XP 2400+ with 512 MByte of memory and the
>> >> load is around 17...
>> >>
>> >> I have for each client (all fixed IP's) two rules (rx/tx) to the
>> >> Internet and two rules (rx/tx) to the internal mail-Server.
>> >>
>> >> So I have completly 2400 rules plus som special-rules to count
>> >> ftp, http, shttp and mail traffic.
>> >>
>> >> In summary around 2500 rules.
>> >>
>> >> What Do you have ???
>> >>
>> >> Thanks
>> >> Michelle
>> >>
>> >I have over 2000 rules "bgpeer tx/rx", "internet tx/rx", "local traffic tx/rx"
>> >machine is AMD Athlon XP 1700+ with 1G ram i forgot how many rules are
>> >limit in iptables but when they are so many this is really sucks this is
>> >on 100Mbit LAN the problem is when fetchipac is running and ipacsum because
>> >file in /var/lib/ipac-ng/data.db is over 5G when file i smaller traffic is smaller
>> >or fetchipac and ipacsum is not running everything is fine i think thats can not
>> >be
>> >the only one way...
>> >
>>
>> I don't run it, I'm just a by stander; but I bet you are not dealing
>> with cpu issues but disk io. run top and compare system load to your cpu
>> state % idle time.
>>
>> If you've got idle cpu, and load over one, you are most likely dealing
>> with disk speed not cpu time for hardware scsi, striped raid, on 15k
>> rpm disks :-P unfortunatly that's a lot more difficult and expensive
>> than upgrading cpu and ram :-\
>>
>> // George
>>
>
>Yes, you're right but my question is, is there have other way to do accounting
>some bash, shell script to fetch traffic with "tc" command from cbq shaper ?
I don't really know that stuff... If you just want to log tcp/udp/icmp
ip use iptables:
iptables -N watchit
iptables -I watchit -s 10.1.0.0/24 -p tcp -m state --state NEW -j LOG --log-prefix
'##_NEW_## '
and periodically do something like
tablestats () {
iptables -vnL >>${LOG}/iptablestats-${now}
iptables -t nat -vnL >>${LOG}/iptablestats-${now}
}
or you may need qdisc routing and logging, I don't know much about
that. My favorite setup is an ebtables bridging router/fw (has no ip
address), patched to send packets through the netfilter tables. :)
That and iptable stats should probably cover your needs.
Just found these, should help with qdisc:
http://lartc.org/howto/index.html
http://lartc.org/howto/lartc.qdisc.html
http://lartc.org/howto/lartc.bridging.html
These are the links I saved from 6 or 8 months back.
http://plorf.net/linux-ip/html/
Guide to IP Layer Network Administration with Linux
http://users.pandora.be/bart.de.schuymer/ebtables/
http://users.pandora.be/bart.de.schuymer/ebtables/sourcecode.html
Ebtables homepage
http://www.tldp.org/HOWTO/Ethernet-Bridge-netfilter-HOWTO.html
Ethernet Bridge + netfilter Howto
http://www.sparkle-cc.co.uk/firewall/firewall.html
Implementing a Bridging Firewall By David Whitmarsh
http://www.compsci.lyon.edu/mcritch/dante/
Dante - Traffic control and QoS with Linux
http://lartc.org/
Linux Advanced Routing & Traffic Control
http://lartc.org/howto/lartc.bridging.html
Building bridges, and pseudo-bridges with Proxy ARP
http://bridge.sourceforge.net/docs.html
http://bridge.sourceforge.net/docs/Firewalling for Free.pdf
Firewalling for Free, by Shawn Grimes.
http://www.pom.gr/ilisepe1/firewall_help.html#5
Transparent Firewall Bridging
http://plorf.net/linux-ip/html/ether-bridging.htm
Address Resolution Protocol and Bridging
http://www.zebra.org/
routing software
Have fun. Let us know what you come up with. :)
// George
--
GEORGE GEORGALIS, System Admin/Architectcell: 646-331-2027<
Security Services, Web, Mail,mailto:[EMAIL PROTECTED]
Multimedia, DB, DNS and Metrics. http://www.galis.org/george
--
To UNSUBSCRIBE, email to [EMAIL PROTECTED]
with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: Traffic Accounting
On Sun, 2003-07-20 at 19:27, George Georgalis wrote: > On Sun, Jul 20, 2003 at 07:01:24PM +0300, kgb wrote: > >On Sun, 2003-07-20 at 15:58, Michelle Konzack wrote: > >> Hello Russian Secret Service Agent... > >> > >> Am 23:35 2003-07-19 +0300 hat kgb geschrieben: > >> > > >> >Hello, > >> > > >> >Which is best way for traffic accounting i use ipac-ng but i don't like > >> >it anymore because it make my system under high load. > >> > > >> >Thanks in advanced. > >> > >> I think there is no other choice... > >> > >> I use ipac on a 100 MBit LAN where I count the traffic of five > >> 11 MBit WaveLAN-Channels... where ipac has two NIC's and is > >> In-Line between the Main-Router and the Switch where the Lucent > >> ORINOCO COR-1100 and wireless Bridges are connected... > >> > >> Each channel has 120 Clients... > >> > >> I use a AMD Athlon XP 2400+ with 512 MByte of memory and the > >> load is around 17... > >> > >> I have for each client (all fixed IP's) two rules (rx/tx) to the > >> Internet and two rules (rx/tx) to the internal mail-Server. > >> > >> So I have completly 2400 rules plus som special-rules to count > >> ftp, http, shttp and mail traffic. > >> > >> In summary around 2500 rules. > >> > >> What Do you have ??? > >> > >> Thanks > >> Michelle > >> > >I have over 2000 rules "bgpeer tx/rx", "internet tx/rx", "local traffic tx/rx" > >machine is AMD Athlon XP 1700+ with 1G ram i forgot how many rules are > >limit in iptables but when they are so many this is really sucks this is > >on 100Mbit LAN the problem is when fetchipac is running and ipacsum because > >file in /var/lib/ipac-ng/data.db is over 5G when file i smaller traffic is smaller > >or fetchipac and ipacsum is not running everything is fine i think thats can not be > >the only one way... > > > > I don't run it, I'm just a by stander; but I bet you are not dealing > with cpu issues but disk io. run top and compare system load to your cpu > state % idle time. > > If you've got idle cpu, and load over one, you are most likely dealing > with disk speed not cpu time for hardware scsi, striped raid, on 15k > rpm disks :-P unfortunatly that's a lot more difficult and expensive > than upgrading cpu and ram :-\ > > // George > > > > -- > GEORGE GEORGALIS, System Admin/Architectcell: 646-331-2027< > Security Services, Web, Mail,mailto:[EMAIL PROTECTED] > Multimedia, DB, DNS and Metrics. http://www.galis.org/george Yes, you're right but my question is, is there have other way to do accounting some bash, shell script to fetch traffic with "tc" command from cbq shaper ? -- Feci quod potui, faciant meliora potentes! signature.asc Description: This is a digitally signed message part
Re: Traffic Accounting
On Sun, Jul 20, 2003 at 07:01:24PM +0300, kgb wrote: >On Sun, 2003-07-20 at 15:58, Michelle Konzack wrote: >> Hello Russian Secret Service Agent... >> >> Am 23:35 2003-07-19 +0300 hat kgb geschrieben: >> > >> >Hello, >> > >> >Which is best way for traffic accounting i use ipac-ng but i don't like >> >it anymore because it make my system under high load. >> > >> >Thanks in advanced. >> >> I think there is no other choice... >> >> I use ipac on a 100 MBit LAN where I count the traffic of five >> 11 MBit WaveLAN-Channels... where ipac has two NIC's and is >> In-Line between the Main-Router and the Switch where the Lucent >> ORINOCO COR-1100 and wireless Bridges are connected... >> >> Each channel has 120 Clients... >> >> I use a AMD Athlon XP 2400+ with 512 MByte of memory and the >> load is around 17... >> >> I have for each client (all fixed IP's) two rules (rx/tx) to the >> Internet and two rules (rx/tx) to the internal mail-Server. >> >> So I have completly 2400 rules plus som special-rules to count >> ftp, http, shttp and mail traffic. >> >> In summary around 2500 rules. >> >> What Do you have ??? >> >> Thanks >> Michelle >> >I have over 2000 rules "bgpeer tx/rx", "internet tx/rx", "local traffic tx/rx" >machine is AMD Athlon XP 1700+ with 1G ram i forgot how many rules are >limit in iptables but when they are so many this is really sucks this is >on 100Mbit LAN the problem is when fetchipac is running and ipacsum because >file in /var/lib/ipac-ng/data.db is over 5G when file i smaller traffic is smaller >or fetchipac and ipacsum is not running everything is fine i think thats can not be >the only one way... > I don't run it, I'm just a by stander; but I bet you are not dealing with cpu issues but disk io. run top and compare system load to your cpu state % idle time. If you've got idle cpu, and load over one, you are most likely dealing with disk speed not cpu time for hardware scsi, striped raid, on 15k rpm disks :-P unfortunatly that's a lot more difficult and expensive than upgrading cpu and ram :-\ // George -- GEORGE GEORGALIS, System Admin/Architectcell: 646-331-2027< Security Services, Web, Mail,mailto:[EMAIL PROTECTED] Multimedia, DB, DNS and Metrics. http://www.galis.org/george -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: Traffic Accounting
On Sun, 2003-07-20 at 15:58, Michelle Konzack wrote: > Hello Russian Secret Service Agent... > > Am 23:35 2003-07-19 +0300 hat kgb geschrieben: > > > >Hello, > > > >Which is best way for traffic accounting i use ipac-ng but i don't like > >it anymore because it make my system under high load. > > > >Thanks in advanced. > > I think there is no other choice... > > I use ipac on a 100 MBit LAN where I count the traffic of five > 11 MBit WaveLAN-Channels... where ipac has two NIC's and is > In-Line between the Main-Router and the Switch where the Lucent > ORINOCO COR-1100 and wireless Bridges are connected... > > Each channel has 120 Clients... > > I use a AMD Athlon XP 2400+ with 512 MByte of memory and the > load is around 17... > > I have for each client (all fixed IP's) two rules (rx/tx) to the > Internet and two rules (rx/tx) to the internal mail-Server. > > So I have completly 2400 rules plus som special-rules to count > ftp, http, shttp and mail traffic. > > In summary around 2500 rules. > > What Do you have ??? > > Thanks > Michelle > > -- > Registered Linux-User #280138 with the Linux Counter, http://counter.li.org. > +--+ > | Michelle's Internet-ServiceInh. Michelle Konzack| > | FunkLAN-Providerin | > +--+ I have over 2000 rules "bgpeer tx/rx", "internet tx/rx", "local traffic tx/rx" machine is AMD Athlon XP 1700+ with 1G ram i forgot how many rules are limit in iptables but when they are so many this is really sucks this is on 100Mbit LAN the problem is when fetchipac is running and ipacsum because file in /var/lib/ipac-ng/data.db is over 5G when file i smaller traffic is smaller or fetchipac and ipacsum is not running everything is fine i think thats can not be the only one way... -- Feci quod potui, faciant meliora potentes! signature.asc Description: This is a digitally signed message part
Re: Traffic Accounting
Hello Russian Secret Service Agent... Am 23:35 2003-07-19 +0300 hat kgb geschrieben: > >Hello, > >Which is best way for traffic accounting i use ipac-ng but i don't like >it anymore because it make my system under high load. > >Thanks in advanced. I think there is no other choice... I use ipac on a 100 MBit LAN where I count the traffic of five 11 MBit WaveLAN-Channels... where ipac has two NIC's and is In-Line between the Main-Router and the Switch where the Lucent ORINOCO COR-1100 and wireless Bridges are connected... Each channel has 120 Clients... I use a AMD Athlon XP 2400+ with 512 MByte of memory and the load is around 17... I have for each client (all fixed IP's) two rules (rx/tx) to the Internet and two rules (rx/tx) to the internal mail-Server. So I have completly 2400 rules plus som special-rules to count ftp, http, shttp and mail traffic. In summary around 2500 rules. What Do you have ??? Thanks Michelle -- Registered Linux-User #280138 with the Linux Counter, http://counter.li.org. +--+ | Michelle's Internet-ServiceInh. Michelle Konzack| | FunkLAN-Providerin | +--+ -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Traffic Accounting
Hello, Which is best way for traffic accounting i use ipac-ng but i don't like it anymore because it make my system under high load. Thanks in advanced. -- Feci quod potui, faciant meliora potentes! signature.asc Description: This is a digitally signed message part
Re: user traffic accounting
also sprach Marcel Hicking <[EMAIL PROTECTED]> [2002.01.09.1428 +0100]: > I'd go for real partitions. No worries with quotas, and > faster than NFS anyway. i guess, but then it couldn't use accounting on the IP level for that traffic. UML *does* support hostfs, which is wicked cool! so i'll use that for all partitions that i mount into the VM that aren't to be accounted. /usr for instance, and /usr/local. and /home. > Guess you could also use a hidden primary configuration. > Your publically announced NS is actually configured as > slave getting updates from the virtual binds. You might > even be ablel to run the official master bind on a > different machine for additional securtity. In case someone > manages to break out of the the virtual machine jail, he > won't be able to mess with your dns too much. > I run this sort of config here and there where somewhat > trusted customers want to have control over their zones. sure, but then their DNS traffic wouldn't be accounted. that's why their servers should actually answer. but in the end, DNS isn't a lot of traffic (if you're not a root server or otherwise big, and if you didn't screw up your SOA, so whatever... i might just ignore that.) > Basically this sounds fine to me. Not sure about the ssh > business, either. Not a nice and clean solution yet. yes. that and HTTPS. oh well... > I'd be really intersted in how the project goes. > Kept us up to date! will do. > > this all has to be implemented remotely ;) > > Apart from setting up a base system, i've never done > anything _not_ remotely ;-) me neither. word up! all it requires as root is a kernel install and a couple of configs like NFS (or coda). the rest is user-mode. i thought that the host kernel has to be majorly enabled, which would have been dangerous. but all i need it the TUN/TAP module support... -- martin; (greetings from the heart of the sun.) \ echo mailto: !#^."<*>"|tr "<*> mailto:"; [EMAIL PROTECTED] kermit: why are there so many songs about rainbows? fuzzy: that's part of what rainbows do. pgpD77KSl9NSw.pgp Description: PGP signature
Re: user traffic accounting
also sprach Mark Janssen <[EMAIL PROTECTED]> [2002.01.08.1847 +0100]: > I don't think this will work. I haven't used UML that much yet, but I > fear that you will not be able to run hundreds of UML's on a single > machine. You might be able to run 10 maybe 20 virtual linux-es on your > box, but it has a rather large overhead compared to a real box without > virtual linux boxes. quality assurance won't make me run more than 15 clients per machine anyway. so we'll see. it *does* have a large overhead, but on a test machine (P5-133, 96Mb), it runs quite quickly actually... i stripped the UM kernel to the bare minimum... > Yeah, it's really nice and secure to boot... but is the overhead and > administrative hassle worth it ?? it's not that much of a hassle actually. most of it was kernel compiles. now i simply get to play with postfix and bind, which i do anyway... > Especially if you are going to be running bind9, apache, postfix and > whatnot in every VM you will be having all those processes in memory all > the time (without them sharing the memory they would usually do when > they were running on the same machine (real vs virtual). valid point. still working on that one... -- martin; (greetings from the heart of the sun.) \ echo mailto: !#^."<*>"|tr "<*> mailto:"; [EMAIL PROTECTED] perl -e 'print $i=pack(c5,(41*2),sqrt(7056),(unpack(c,H)-2),oct(115),10);' pgpsdZmFo13Kc.pgp Description: PGP signature
Re: user traffic accounting
On 8 Jan 2002, at 18:25, martin f krafft wrote: > [cc'd to gr and peter because i think they might be > interested and because they might have valuable input. this > is about accounting on a user basis for each and every byte > a user or her domains cause. debian-isp is open to > posting... original post lives at [1]] > > also sprach Marcel Hicking <[EMAIL PROTECTED]> > [2002.01.08.1634 +0100]: > > User Mode Linux virtual > machines are networkable, > > to each other, to the host, > and to other physical > > machines. So, UML can be used to > set up a virtual > > network that allows setting up and > testing of > > experimental services. > > > http://user-mode-linux.sourceforge.net/ > > i.e. basically vmware for linux-on-linux only (for now), and > free... > > this is *very* cool, thanks so much. i mean, damn you, how > could you show me this, now i have something else to occupy > my time with ;) (i hope you aren't offended by my use of > "damn"). No, dammit, no prob ;-) > anyway, this is wicked, and i immediately want to give a > virtual machine to every single one of my users. since i > only have one IP (not true, but i don't have an IP per > user), i'd have to do MASQ along with proxies on the host, > but i think this could work. your comments on the following, > please... > > the best is, i think you could create *one* filesystem to > serve them all, mount it read-only, and then provide them > with /home/user - which is either NFS-mounted from the host, > or which is simply a partition mounted from a file in their > /home on the host. then again, i'd love to *not* have users > on the host then. that's the least trouble... I'd go for real partitions. No worries with quotas, and faster than NFS anyway. > let me start with constructing the hosting services before i > attack the tough nuts... so the system will have 1.2.3.4 as > the official IP, and a 172.16/16 network between the > official host and all the vm's. > > 1. postfix. there'll be a postfix running in each and every > vm, taking care of the hosted domains only. it is configured > to send via postfix on the master (smtp-relay), and the > master's postfix is configured to relay mail for all domains > in the VMs, using the transport table to then deliver it to > the vm's postfix on the 172.16/16 subnet. thus, even though > the mail traffic that my server farm sees isn't tthe same > that's flowing between the master and the vm, they are > (virtually) identical. because of received-headers adding > size, those users that only send will cause me some loss, > those that mostly receive will pay a little more. but it's > within the bytes to kilobytes range, thus no problem. > > 2. bind9. this is also moderately easy. the master runs a > bind9 server that's configured to go recursive for the > domains in the vmachines. the vm bind9 uses the master bind9 > as the only forwarder. Guess you could also use a hidden primary configuration. Your publically announced NS is actually configured as slave getting updates from the virtual binds. You might even be ablel to run the official master bind on a different machine for additional securtity. In case someone manages to break out of the the virtual machine jail, he won't be able to mess with your dns too much. I run this sort of config here and there where somewhat trusted customers want to have control over their zones. > 3. apache. things are getting more difficult. because of > virtual hosting, one would have to employ a transparent > squid proxy without caching abilities (maybe there's a > better, low-weight proxy for this) because what it should > really do is respond to a request for something like > vm1.madduck.net with the response it receives from a request > on the 172.16/16 subnet to the apache running in the > appropriate virtual machine. there are two problems i see: > logging - inside the vm, all requests for a domain's webpage > will appear to be coming from the proxy rather than the > original requester. i wonder if it's possible to have a > relay that reads ahead in the HTTP request to decide how to > forward/NAT the request before relaying it on the IP > level... the second problem is HTTPS, but then again, with a > single IP, you can't really run multiple HTTPS domains > anyway, so users simply won't get their own HTTPS server - > if they need HTTPS, then a special configuration could be > set up on the main HTTPS server, which NFS-mounts the > respective directory from the VM into the HTTPS ServerRoot, > which will at least account for the actual payload data even > if the request and HTTP response header are not going to be > included in the accounted traffic volume. oh well. > > 4. shell traffic. because 172.16/16 is illegal, masquerading > is done, which makes the master host be the upstream gateway > for the VMs. thus every byte will be registered by iptables > or ipac-ng as it passes through the master host's netfilter. > thus traffic caused on the shell will be counted without >
Re: user traffic accounting
also sprach Alexander Reelsen <[EMAIL PROTECTED]> [2002.01.09.0756 +0100]: > Anyone actually tried vserver? That might be what you are searching for > instead of UML... > > http://www.solucorp.qc.ca/miscprj/s_context.hc > > I think that's the right URL if I may believe my bookmarks. yeah, it works. i'll have a look. it's unbelievable how valuable the debian lists and you all are! thanks! -- martin; (greetings from the heart of the sun.) \ echo mailto: !#^."<*>"|tr "<*> mailto:"; [EMAIL PROTECTED] this site has moved. we'd tell you where, but then we'd have to delete you. pgpa7HcvGj61H.pgp Description: PGP signature
Re: user traffic accounting
also sprach Marcel Hicking <[EMAIL PROTECTED]> [2002.01.09.1428 +0100]: > I'd go for real partitions. No worries with quotas, and > faster than NFS anyway. i guess, but then it couldn't use accounting on the IP level for that traffic. UML *does* support hostfs, which is wicked cool! so i'll use that for all partitions that i mount into the VM that aren't to be accounted. /usr for instance, and /usr/local. and /home. > Guess you could also use a hidden primary configuration. > Your publically announced NS is actually configured as > slave getting updates from the virtual binds. You might > even be ablel to run the official master bind on a > different machine for additional securtity. In case someone > manages to break out of the the virtual machine jail, he > won't be able to mess with your dns too much. > I run this sort of config here and there where somewhat > trusted customers want to have control over their zones. sure, but then their DNS traffic wouldn't be accounted. that's why their servers should actually answer. but in the end, DNS isn't a lot of traffic (if you're not a root server or otherwise big, and if you didn't screw up your SOA, so whatever... i might just ignore that.) > Basically this sounds fine to me. Not sure about the ssh > business, either. Not a nice and clean solution yet. yes. that and HTTPS. oh well... > I'd be really intersted in how the project goes. > Kept us up to date! will do. > > this all has to be implemented remotely ;) > > Apart from setting up a base system, i've never done > anything _not_ remotely ;-) me neither. word up! all it requires as root is a kernel install and a couple of configs like NFS (or coda). the rest is user-mode. i thought that the host kernel has to be majorly enabled, which would have been dangerous. but all i need it the TUN/TAP module support... -- martin; (greetings from the heart of the sun.) \ echo mailto: !#^."<*>"|tr "<*> mailto:"; net@madduck kermit: why are there so many songs about rainbows? fuzzy: that's part of what rainbows do. msg04754/pgp0.pgp Description: PGP signature
Re: user traffic accounting
also sprach Mark Janssen <[EMAIL PROTECTED]> [2002.01.08.1847 +0100]: > I don't think this will work. I haven't used UML that much yet, but I > fear that you will not be able to run hundreds of UML's on a single > machine. You might be able to run 10 maybe 20 virtual linux-es on your > box, but it has a rather large overhead compared to a real box without > virtual linux boxes. quality assurance won't make me run more than 15 clients per machine anyway. so we'll see. it *does* have a large overhead, but on a test machine (P5-133, 96Mb), it runs quite quickly actually... i stripped the UM kernel to the bare minimum... > Yeah, it's really nice and secure to boot... but is the overhead and > administrative hassle worth it ?? it's not that much of a hassle actually. most of it was kernel compiles. now i simply get to play with postfix and bind, which i do anyway... > Especially if you are going to be running bind9, apache, postfix and > whatnot in every VM you will be having all those processes in memory all > the time (without them sharing the memory they would usually do when > they were running on the same machine (real vs virtual). valid point. still working on that one... -- martin; (greetings from the heart of the sun.) \ echo mailto: !#^."<*>"|tr "<*> mailto:"; net@madduck perl -e 'print $i=pack(c5,(41*2),sqrt(7056),(unpack(c,H)-2),oct(115),10);' msg04753/pgp0.pgp Description: PGP signature
Re: user traffic accounting
On 8 Jan 2002, at 18:25, martin f krafft wrote: > [cc'd to gr and peter because i think they might be > interested and because they might have valuable input. this > is about accounting on a user basis for each and every byte > a user or her domains cause. debian-isp is open to > posting... original post lives at [1]] > > also sprach Marcel Hicking <[EMAIL PROTECTED]> > [2002.01.08.1634 +0100]: > > User Mode Linux virtual > machines are networkable, > > to each other, to the host, > and to other physical > > machines. So, UML can be used to > set up a virtual > > network that allows setting up and > testing of > > experimental services. > > > http://user-mode-linux.sourceforge.net/ > > i.e. basically vmware for linux-on-linux only (for now), and > free... > > this is *very* cool, thanks so much. i mean, damn you, how > could you show me this, now i have something else to occupy > my time with ;) (i hope you aren't offended by my use of > "damn"). No, dammit, no prob ;-) > anyway, this is wicked, and i immediately want to give a > virtual machine to every single one of my users. since i > only have one IP (not true, but i don't have an IP per > user), i'd have to do MASQ along with proxies on the host, > but i think this could work. your comments on the following, > please... > > the best is, i think you could create *one* filesystem to > serve them all, mount it read-only, and then provide them > with /home/user - which is either NFS-mounted from the host, > or which is simply a partition mounted from a file in their > /home on the host. then again, i'd love to *not* have users > on the host then. that's the least trouble... I'd go for real partitions. No worries with quotas, and faster than NFS anyway. > let me start with constructing the hosting services before i > attack the tough nuts... so the system will have 1.2.3.4 as > the official IP, and a 172.16/16 network between the > official host and all the vm's. > > 1. postfix. there'll be a postfix running in each and every > vm, taking care of the hosted domains only. it is configured > to send via postfix on the master (smtp-relay), and the > master's postfix is configured to relay mail for all domains > in the VMs, using the transport table to then deliver it to > the vm's postfix on the 172.16/16 subnet. thus, even though > the mail traffic that my server farm sees isn't tthe same > that's flowing between the master and the vm, they are > (virtually) identical. because of received-headers adding > size, those users that only send will cause me some loss, > those that mostly receive will pay a little more. but it's > within the bytes to kilobytes range, thus no problem. > > 2. bind9. this is also moderately easy. the master runs a > bind9 server that's configured to go recursive for the > domains in the vmachines. the vm bind9 uses the master bind9 > as the only forwarder. Guess you could also use a hidden primary configuration. Your publically announced NS is actually configured as slave getting updates from the virtual binds. You might even be ablel to run the official master bind on a different machine for additional securtity. In case someone manages to break out of the the virtual machine jail, he won't be able to mess with your dns too much. I run this sort of config here and there where somewhat trusted customers want to have control over their zones. > 3. apache. things are getting more difficult. because of > virtual hosting, one would have to employ a transparent > squid proxy without caching abilities (maybe there's a > better, low-weight proxy for this) because what it should > really do is respond to a request for something like > vm1.madduck.net with the response it receives from a request > on the 172.16/16 subnet to the apache running in the > appropriate virtual machine. there are two problems i see: > logging - inside the vm, all requests for a domain's webpage > will appear to be coming from the proxy rather than the > original requester. i wonder if it's possible to have a > relay that reads ahead in the HTTP request to decide how to > forward/NAT the request before relaying it on the IP > level... the second problem is HTTPS, but then again, with a > single IP, you can't really run multiple HTTPS domains > anyway, so users simply won't get their own HTTPS server - > if they need HTTPS, then a special configuration could be > set up on the main HTTPS server, which NFS-mounts the > respective directory from the VM into the HTTPS ServerRoot, > which will at least account for the actual payload data even > if the request and HTTP response header are not going to be > included in the accounted traffic volume. oh well. > > 4. shell traffic. because 172.16/16 is illegal, masquerading > is done, which makes the master host be the upstream gateway > for the VMs. thus every byte will be registered by iptables > or ipac-ng as it passes through the master host's netfilter. > thus traffic caused on the shell will be counted without
Re: user traffic accounting
also sprach Alexander Reelsen <[EMAIL PROTECTED]> [2002.01.09.0756 +0100]: > Anyone actually tried vserver? That might be what you are searching for > instead of UML... > > http://www.solucorp.qc.ca/miscprj/s_context.hc > > I think that's the right URL if I may believe my bookmarks. yeah, it works. i'll have a look. it's unbelievable how valuable the debian lists and you all are! thanks! -- martin; (greetings from the heart of the sun.) \ echo mailto: !#^."<*>"|tr "<*> mailto:"; net@madduck this site has moved. we'd tell you where, but then we'd have to delete you. msg04749/pgp0.pgp Description: PGP signature
Re: user traffic accounting
Hi On Wed, Jan 09, 2002 at 03:12:09AM +0100, martin f krafft wrote: > also sprach Jeff Waugh <[EMAIL PROTECTED]> [2002.01.09.0257 +0100]: > > Nice idea, but it's not going to work. Perhaps with some real love and > > affection from someone who purely wanted to achieve this (and wasn't > > primarily interested in using it as a debugging tool), it may happen, but in > > its current state, UML is not appropriate for this. > i am doing so right now. since i don't have anything else to do (really, > for once). i'll report. Anyone actually tried vserver? That might be what you are searching for instead of UML... http://www.solucorp.qc.ca/miscprj/s_context.hc I think that's the right URL if I may believe my bookmarks. MfG/Regards, Alexander -- Alexander Reelsen http://joker.rhwd.de [EMAIL PROTECTED]GnuPG: pub 1024D/F0D7313C sub 2048g/6AA2EDDB 7D44 F4E3 1993 FDDF 552E 7C88 EE9C CBD1 F0D7 313C
Re: user traffic accounting
Hi On Wed, Jan 09, 2002 at 03:12:09AM +0100, martin f krafft wrote: > also sprach Jeff Waugh <[EMAIL PROTECTED]> [2002.01.09.0257 +0100]: > > Nice idea, but it's not going to work. Perhaps with some real love and > > affection from someone who purely wanted to achieve this (and wasn't > > primarily interested in using it as a debugging tool), it may happen, but in > > its current state, UML is not appropriate for this. > i am doing so right now. since i don't have anything else to do (really, > for once). i'll report. Anyone actually tried vserver? That might be what you are searching for instead of UML... http://www.solucorp.qc.ca/miscprj/s_context.hc I think that's the right URL if I may believe my bookmarks. MfG/Regards, Alexander -- Alexander Reelsen http://joker.rhwd.de [EMAIL PROTECTED]GnuPG: pub 1024D/F0D7313C sub 2048g/6AA2EDDB 7D44 F4E3 1993 FDDF 552E 7C88 EE9C CBD1 F0D7 313C -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: user traffic accounting
also sprach Jeff Waugh <[EMAIL PROTECTED]> [2002.01.09.0257 +0100]: > Nice idea, but it's not going to work. Perhaps with some real love and > affection from someone who purely wanted to achieve this (and wasn't > primarily interested in using it as a debugging tool), it may happen, but in > its current state, UML is not appropriate for this. i am doing so right now. since i don't have anything else to do (really, for once). i'll report. -- martin; (greetings from the heart of the sun.) \ echo mailto: !#^."<*>"|tr "<*> mailto:"; [EMAIL PROTECTED] man muss noch chaos in sich haben um einen tanzenden stern zu gebaehren. -- nietzsche pgpWPfrk93uKn.pgp Description: PGP signature
Re: user traffic accounting
> anyway, this is wicked, and i immediately want to give a virtual machine > to every single one of my users. Nice idea, but it's not going to work. Perhaps with some real love and affection from someone who purely wanted to achieve this (and wasn't primarily interested in using it as a debugging tool), it may happen, but in its current state, UML is not appropriate for this. - Jeff -- "I'm taking no part in your merry 5-way clusterfuck - sort that mess out between yourselves." - Alexander Viro
Re: user traffic accounting
also sprach Jeff Waugh <[EMAIL PROTECTED]> [2002.01.09.0257 +0100]: > Nice idea, but it's not going to work. Perhaps with some real love and > affection from someone who purely wanted to achieve this (and wasn't > primarily interested in using it as a debugging tool), it may happen, but in > its current state, UML is not appropriate for this. i am doing so right now. since i don't have anything else to do (really, for once). i'll report. -- martin; (greetings from the heart of the sun.) \ echo mailto: !#^."<*>"|tr "<*> mailto:"; net@madduck man muss noch chaos in sich haben um einen tanzenden stern zu gebaehren. -- nietzsche msg04742/pgp0.pgp Description: PGP signature
Re: user traffic accounting
> anyway, this is wicked, and i immediately want to give a virtual machine > to every single one of my users. Nice idea, but it's not going to work. Perhaps with some real love and affection from someone who purely wanted to achieve this (and wasn't primarily interested in using it as a debugging tool), it may happen, but in its current state, UML is not appropriate for this. - Jeff -- "I'm taking no part in your merry 5-way clusterfuck - sort that mess out between yourselves." - Alexander Viro -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: user traffic accounting
On Tue, Jan 08, 2002 at 06:25:12PM +0100, martin f krafft wrote: > > http://user-mode-linux.sourceforge.net/ > > i.e. basically vmware for linux-on-linux only (for now), and free... > > anyway, this is wicked, and i immediately want to give a virtual machine > to every single one of my users. since i only have one IP (not true, but I don't think this will work. I haven't used UML that much yet, but I fear that you will not be able to run hundreds of UML's on a single machine. You might be able to run 10 maybe 20 virtual linux-es on your box, but it has a rather large overhead compared to a real box without virtual linux boxes. Yeah, it's really nice and secure to boot... but is the overhead and administrative hassle worth it ?? Especially if you are going to be running bind9, apache, postfix and whatnot in every VM you will be having all those processes in memory all the time (without them sharing the memory they would usually do when they were running on the same machine (real vs virtual). Mark Janssen Unix / Linux, Open-Source and Internet Consultant @ SyConOS IT E-mail: mark(at)markjanssen.nl / maniac(at)maniac.nl GnuPG Key Id: 357D2178 Web: Maniac.nl Unix-God.[Net|Org] MarkJanssen.[com|net|org|nl] SyConOS.[com|nl]
Re: user traffic accounting
[cc'd to gr and peter because i think they might be interested and because they might have valuable input. this is about accounting on a user basis for each and every byte a user or her domains cause. debian-isp is open to posting... original post lives at [1]] also sprach Marcel Hicking <[EMAIL PROTECTED]> [2002.01.08.1634 +0100]: > > User Mode Linux virtual machines are networkable, > > to each other, to the host, and to other physical > > machines. So, UML can be used to set up a virtual > > network that allows setting up and testing of > > experimental services. > > http://user-mode-linux.sourceforge.net/ i.e. basically vmware for linux-on-linux only (for now), and free... this is *very* cool, thanks so much. i mean, damn you, how could you show me this, now i have something else to occupy my time with ;) (i hope you aren't offended by my use of "damn"). anyway, this is wicked, and i immediately want to give a virtual machine to every single one of my users. since i only have one IP (not true, but i don't have an IP per user), i'd have to do MASQ along with proxies on the host, but i think this could work. your comments on the following, please... the best is, i think you could create *one* filesystem to serve them all, mount it read-only, and then provide them with /home/user - which is either NFS-mounted from the host, or which is simply a partition mounted from a file in their /home on the host. then again, i'd love to *not* have users on the host then. that's the least trouble... let me start with constructing the hosting services before i attack the tough nuts... so the system will have 1.2.3.4 as the official IP, and a 172.16/16 network between the official host and all the vm's. 1. postfix. there'll be a postfix running in each and every vm, taking care of the hosted domains only. it is configured to send via postfix on the master (smtp-relay), and the master's postfix is configured to relay mail for all domains in the VMs, using the transport table to then deliver it to the vm's postfix on the 172.16/16 subnet. thus, even though the mail traffic that my server farm sees isn't tthe same that's flowing between the master and the vm, they are (virtually) identical. because of received-headers adding size, those users that only send will cause me some loss, those that mostly receive will pay a little more. but it's within the bytes to kilobytes range, thus no problem. 2. bind9. this is also moderately easy. the master runs a bind9 server that's configured to go recursive for the domains in the vmachines. the vm bind9 uses the master bind9 as the only forwarder. 3. apache. things are getting more difficult. because of virtual hosting, one would have to employ a transparent squid proxy without caching abilities (maybe there's a better, low-weight proxy for this) because what it should really do is respond to a request for something like vm1.madduck.net with the response it receives from a request on the 172.16/16 subnet to the apache running in the appropriate virtual machine. there are two problems i see: logging - inside the vm, all requests for a domain's webpage will appear to be coming from the proxy rather than the original requester. i wonder if it's possible to have a relay that reads ahead in the HTTP request to decide how to forward/NAT the request before relaying it on the IP level... the second problem is HTTPS, but then again, with a single IP, you can't really run multiple HTTPS domains anyway, so users simply won't get their own HTTPS server - if they need HTTPS, then a special configuration could be set up on the main HTTPS server, which NFS-mounts the respective directory from the VM into the HTTPS ServerRoot, which will at least account for the actual payload data even if the request and HTTP response header are not going to be included in the accounted traffic volume. oh well. 4. shell traffic. because 172.16/16 is illegal, masquerading is done, which makes the master host be the upstream gateway for the VMs. thus every byte will be registered by iptables or ipac-ng as it passes through the master host's netfilter. thus traffic caused on the shell will be counted without overlap, next to, and completely identical to the traffic caused by the daemons on the VM. 5. ssh. this is the real bitch! you can't proxy SSH, you can't really forward it. i could either give users accounts on the master host with their login shells configured to do host-based RSA authenticated login to their VM, or i could give out special SSH ports and forward those. for instance, user joe will be able to login to his VM at 172.16.101.123:22 via ssh to 1.2.3.4:22123. this is not a problem in terms of known_hosts because say joe owns joe.net, but he also helps to administer another domain, coop.net, which lives in another VM. while ssh'ing to joe.net via port 22123, his known_hosts will register the joe.net VM's RSA/DSA key with the IP 1.2.3.4 and hostname joe.net, when ssh'ing into coop.net via port
Re: user traffic accounting
On Tue, Jan 08, 2002 at 06:25:12PM +0100, martin f krafft wrote: > > http://user-mode-linux.sourceforge.net/ > > i.e. basically vmware for linux-on-linux only (for now), and free... > > anyway, this is wicked, and i immediately want to give a virtual machine > to every single one of my users. since i only have one IP (not true, but I don't think this will work. I haven't used UML that much yet, but I fear that you will not be able to run hundreds of UML's on a single machine. You might be able to run 10 maybe 20 virtual linux-es on your box, but it has a rather large overhead compared to a real box without virtual linux boxes. Yeah, it's really nice and secure to boot... but is the overhead and administrative hassle worth it ?? Especially if you are going to be running bind9, apache, postfix and whatnot in every VM you will be having all those processes in memory all the time (without them sharing the memory they would usually do when they were running on the same machine (real vs virtual). Mark Janssen Unix / Linux, Open-Source and Internet Consultant @ SyConOS IT E-mail: mark(at)markjanssen.nl / maniac(at)maniac.nl GnuPG Key Id: 357D2178 Web: Maniac.nl Unix-God.[Net|Org] MarkJanssen.[com|net|org|nl] SyConOS.[com|nl] -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: user traffic accounting
Just a quick thought, as it says on the website: > User Mode Linux virtual machines are networkable, > to each other, to the host, and to other physical > machines. So, UML can be used to set up a virtual > network that allows setting up and testing of > experimental services. http://user-mode-linux.sourceforge.net/ Maybe this allows accounting of the vm's, too. I remember a project here where BSD was used in a similar way and every virtual machine had it's own IP. Cheers, Marcel martin f krafft <[EMAIL PROTECTED]> 7 Jan 2002, at 14:26: > > --ReaqsoxgOBHFXBhH > Content-Type: text/plain; charset=iso-8859-15 > Content-Disposition: inline > Content-Transfer-Encoding: quoted-printable > > hi folks, > please direct me to some documentation on ways to account > for user traffic on a single machine, acting as BIND9, > apache, postfix, and sshd server for a number of users. i > need to get as close as possible to exact traffic volume > measurements to do proper billing, and (unfortunately), i > can't use an upstream router for that. > > i want to account for total traffic, which includes the > following challenges: > > - Shell: every user has ssh access. i need to be able to > keep track > of every byte coming in and out of sshd, but also any > data sent to or received from the internet while using > the shell account. > - HTTP: a user has zero or more domains hosted on the > system, all > request and response volume should be added to that > users accounting data. > - Mail: any mail that the user receives should be > byte-counted. the > same applies to mail sent from the user account via > sendmail, mail sent via port 25, and mail relayed (TLS > client authentication). > - BIND: c.f. with HTTP, basically the same applies. > =20 > if you ask me, this sounds like a horrible task. any tips > from the ISP experts? > > --=20 > martin; (greetings from the heart of the sun.) > \ echo mailto: !#^."<*>"|tr "<*> mailto:"; [EMAIL PROTECTED] > =20 > don't hate yourself in the morning -- sleep till noon. > > --ReaqsoxgOBHFXBhH > Content-Type: application/pgp-signature > Content-Disposition: inline > > -BEGIN PGP SIGNATURE- > Version: GnuPG v1.0.6 (GNU/Linux) > Comment: For info see http://www.gnupg.org > > iEYEARECAAYFAjw5ohYACgkQIgvIgzMMSnUSbQCfRrzmUHF9vYX3dVcJntpq > EwTl ik0AoJ7SNIpXyTKC2G2mjgPI5Y7Q0NlO =6Z3o -END PGP > SIGNATURE- > > --ReaqsoxgOBHFXBhH-- > > > -- > To UNSUBSCRIBE, email to [EMAIL PROTECTED] > with a subject of "unsubscribe". Trouble? Contact > [EMAIL PROTECTED] >
Re: user traffic accounting
[cc'd to gr and peter because i think they might be interested and because they might have valuable input. this is about accounting on a user basis for each and every byte a user or her domains cause. debian-isp is open to posting... original post lives at [1]] also sprach Marcel Hicking <[EMAIL PROTECTED]> [2002.01.08.1634 +0100]: > > User Mode Linux virtual machines are networkable, > > to each other, to the host, and to other physical > > machines. So, UML can be used to set up a virtual > > network that allows setting up and testing of > > experimental services. > > http://user-mode-linux.sourceforge.net/ i.e. basically vmware for linux-on-linux only (for now), and free... this is *very* cool, thanks so much. i mean, damn you, how could you show me this, now i have something else to occupy my time with ;) (i hope you aren't offended by my use of "damn"). anyway, this is wicked, and i immediately want to give a virtual machine to every single one of my users. since i only have one IP (not true, but i don't have an IP per user), i'd have to do MASQ along with proxies on the host, but i think this could work. your comments on the following, please... the best is, i think you could create *one* filesystem to serve them all, mount it read-only, and then provide them with /home/user - which is either NFS-mounted from the host, or which is simply a partition mounted from a file in their /home on the host. then again, i'd love to *not* have users on the host then. that's the least trouble... let me start with constructing the hosting services before i attack the tough nuts... so the system will have 1.2.3.4 as the official IP, and a 172.16/16 network between the official host and all the vm's. 1. postfix. there'll be a postfix running in each and every vm, taking care of the hosted domains only. it is configured to send via postfix on the master (smtp-relay), and the master's postfix is configured to relay mail for all domains in the VMs, using the transport table to then deliver it to the vm's postfix on the 172.16/16 subnet. thus, even though the mail traffic that my server farm sees isn't tthe same that's flowing between the master and the vm, they are (virtually) identical. because of received-headers adding size, those users that only send will cause me some loss, those that mostly receive will pay a little more. but it's within the bytes to kilobytes range, thus no problem. 2. bind9. this is also moderately easy. the master runs a bind9 server that's configured to go recursive for the domains in the vmachines. the vm bind9 uses the master bind9 as the only forwarder. 3. apache. things are getting more difficult. because of virtual hosting, one would have to employ a transparent squid proxy without caching abilities (maybe there's a better, low-weight proxy for this) because what it should really do is respond to a request for something like vm1.madduck.net with the response it receives from a request on the 172.16/16 subnet to the apache running in the appropriate virtual machine. there are two problems i see: logging - inside the vm, all requests for a domain's webpage will appear to be coming from the proxy rather than the original requester. i wonder if it's possible to have a relay that reads ahead in the HTTP request to decide how to forward/NAT the request before relaying it on the IP level... the second problem is HTTPS, but then again, with a single IP, you can't really run multiple HTTPS domains anyway, so users simply won't get their own HTTPS server - if they need HTTPS, then a special configuration could be set up on the main HTTPS server, which NFS-mounts the respective directory from the VM into the HTTPS ServerRoot, which will at least account for the actual payload data even if the request and HTTP response header are not going to be included in the accounted traffic volume. oh well. 4. shell traffic. because 172.16/16 is illegal, masquerading is done, which makes the master host be the upstream gateway for the VMs. thus every byte will be registered by iptables or ipac-ng as it passes through the master host's netfilter. thus traffic caused on the shell will be counted without overlap, next to, and completely identical to the traffic caused by the daemons on the VM. 5. ssh. this is the real bitch! you can't proxy SSH, you can't really forward it. i could either give users accounts on the master host with their login shells configured to do host-based RSA authenticated login to their VM, or i could give out special SSH ports and forward those. for instance, user joe will be able to login to his VM at 172.16.101.123:22 via ssh to 1.2.3.4:22123. this is not a problem in terms of known_hosts because say joe owns joe.net, but he also helps to administer another domain, coop.net, which lives in another VM. while ssh'ing to joe.net via port 22123, his known_hosts will register the joe.net VM's RSA/DSA key with the IP 1.2.3.4 and hostname joe.net, when ssh'ing into coop.net via por
Re: user traffic accounting
Just a quick thought, as it says on the website: > User Mode Linux virtual machines are networkable, > to each other, to the host, and to other physical > machines. So, UML can be used to set up a virtual > network that allows setting up and testing of > experimental services. http://user-mode-linux.sourceforge.net/ Maybe this allows accounting of the vm's, too. I remember a project here where BSD was used in a similar way and every virtual machine had it's own IP. Cheers, Marcel martin f krafft <[EMAIL PROTECTED]> 7 Jan 2002, at 14:26: > > --ReaqsoxgOBHFXBhH > Content-Type: text/plain; charset=iso-8859-15 > Content-Disposition: inline > Content-Transfer-Encoding: quoted-printable > > hi folks, > please direct me to some documentation on ways to account > for user traffic on a single machine, acting as BIND9, > apache, postfix, and sshd server for a number of users. i > need to get as close as possible to exact traffic volume > measurements to do proper billing, and (unfortunately), i > can't use an upstream router for that. > > i want to account for total traffic, which includes the > following challenges: > > - Shell: every user has ssh access. i need to be able to > keep track > of every byte coming in and out of sshd, but also any > data sent to or received from the internet while using > the shell account. > - HTTP: a user has zero or more domains hosted on the > system, all > request and response volume should be added to that > users accounting data. > - Mail: any mail that the user receives should be > byte-counted. the > same applies to mail sent from the user account via > sendmail, mail sent via port 25, and mail relayed (TLS > client authentication). > - BIND: c.f. with HTTP, basically the same applies. > =20 > if you ask me, this sounds like a horrible task. any tips > from the ISP experts? > > --=20 > martin; (greetings from the heart of the sun.) > \ echo mailto: !#^."<*>"|tr "<*> mailto:"; net@madduck > =20 > don't hate yourself in the morning -- sleep till noon. > > --ReaqsoxgOBHFXBhH > Content-Type: application/pgp-signature > Content-Disposition: inline > > -BEGIN PGP SIGNATURE- > Version: GnuPG v1.0.6 (GNU/Linux) > Comment: For info see http://www.gnupg.org > > iEYEARECAAYFAjw5ohYACgkQIgvIgzMMSnUSbQCfRrzmUHF9vYX3dVcJntpq > EwTl ik0AoJ7SNIpXyTKC2G2mjgPI5Y7Q0NlO =6Z3o -END PGP > SIGNATURE- > > --ReaqsoxgOBHFXBhH-- > > > -- > To UNSUBSCRIBE, email to [EMAIL PROTECTED] > with a subject of "unsubscribe". Trouble? Contact > [EMAIL PROTECTED] > -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: user traffic accounting
On 07/01/02, martin f krafft wrote: > please direct me to some documentation on ways to account for user > traffic on a single machine, acting as BIND9, apache, postfix, and sshd > server for a number of users. i need to get as close as possible to Sounds like those users need to have lots of money or be very careful how they use that machine as otherwise they go bankcruptcy. :-) > - Shell: every user has ssh access. i need to be able to keep track > of every byte coming in and out of sshd, but also any data sent to > or received from the internet while using the shell account. That could be the most difficult one as all traffic is encrypted and you have no chance to identify the user and figure out who is responsible for which traffic. Even when using a sniffer you'll only be able to figure out which traffic originates or was send to which ip. And using a sniffer could cause legal problems. > - HTTP: a user has zero or more domains hosted on the system, all > request and response volume should be added to that users accounting > data. Hm, that could be a bit easier, since at least for the incoming request it should be possible to get the http server to log not only the request and the origin of it, but also the size. The problem would be to identify exactly all outgoing traffic that is created as a response. > - Mail: any mail that the user receives should be byte-counted. the > same applies to mail sent from the user account via sendmail, mail > sent via port 25, and mail relayed (TLS client authentication). That again will be a bit difficult since most MTA don't log the size of the mail. I would suppose that accounting the outgoing traffic will be the biggest problem here, since mostly no logfile for a MTA will include information which user submitted a mail and how big it was. For incoming traffic, also called mails ;-), partly this could be solved by changing the setup to have the MTA first send the mail to some kind of content filter, which would then not only check for viruses, but also figure out to which user the mail was addressed by looking at some headers like Delievered-To and then calculate the exact size of the mail and write this information to some log before handing the mail to the MDA. > - BIND: c.f. with HTTP, basically the same applies. Again a big problem, since bind never logs the size of the request or answers. > if you ask me, this sounds like a horrible task. any tips from the ISP > experts? Yes, that's horrible and sounds like some sales people thought about ways to bill their customers more money without thinking about the technical problems or talking with an it staff about it. Christian -- Debian Developer (http://www.debian.org) 1024/26CC7853 31E6 A8CA 68FC 284F 7D16 63EC A9E6 67FF 26CC 7853 pgpacN8SSPvNf.pgp Description: PGP signature
Re: user traffic accounting
On 07/01/02, martin f krafft wrote: > please direct me to some documentation on ways to account for user > traffic on a single machine, acting as BIND9, apache, postfix, and sshd > server for a number of users. i need to get as close as possible to Sounds like those users need to have lots of money or be very careful how they use that machine as otherwise they go bankcruptcy. :-) > - Shell: every user has ssh access. i need to be able to keep track > of every byte coming in and out of sshd, but also any data sent to > or received from the internet while using the shell account. That could be the most difficult one as all traffic is encrypted and you have no chance to identify the user and figure out who is responsible for which traffic. Even when using a sniffer you'll only be able to figure out which traffic originates or was send to which ip. And using a sniffer could cause legal problems. > - HTTP: a user has zero or more domains hosted on the system, all > request and response volume should be added to that users accounting > data. Hm, that could be a bit easier, since at least for the incoming request it should be possible to get the http server to log not only the request and the origin of it, but also the size. The problem would be to identify exactly all outgoing traffic that is created as a response. > - Mail: any mail that the user receives should be byte-counted. the > same applies to mail sent from the user account via sendmail, mail > sent via port 25, and mail relayed (TLS client authentication). That again will be a bit difficult since most MTA don't log the size of the mail. I would suppose that accounting the outgoing traffic will be the biggest problem here, since mostly no logfile for a MTA will include information which user submitted a mail and how big it was. For incoming traffic, also called mails ;-), partly this could be solved by changing the setup to have the MTA first send the mail to some kind of content filter, which would then not only check for viruses, but also figure out to which user the mail was addressed by looking at some headers like Delievered-To and then calculate the exact size of the mail and write this information to some log before handing the mail to the MDA. > - BIND: c.f. with HTTP, basically the same applies. Again a big problem, since bind never logs the size of the request or answers. > if you ask me, this sounds like a horrible task. any tips from the ISP > experts? Yes, that's horrible and sounds like some sales people thought about ways to bill their customers more money without thinking about the technical problems or talking with an it staff about it. Christian -- Debian Developer (http://www.debian.org) 1024/26CC7853 31E6 A8CA 68FC 284F 7D16 63EC A9E6 67FF 26CC 7853 msg04731/pgp0.pgp Description: PGP signature
Re: user traffic accounting
also sprach Thedore Knab <[EMAIL PROTECTED]> [2002.01.07.1624 +0100]: > How would ipfm work for this? > > http://freshmeat.net/projects/ipfm/ this strikes me as a nice tool, but one that needs to run on a router/gateway/firewall, and one which can only differentiate according to IPs. if IPs were all i had to worry about, then i could just use iptables... thanks for replying though! have there been other replies to the list? i screwed my procmail temporarily and lost all mail since my original post... and the archives have nothing yet... -- martin; (greetings from the heart of the sun.) \ echo mailto: !#^."<*>"|tr "<*> mailto:"; [EMAIL PROTECTED] wind catches lily, scattering petals to the ground. segmentation fault. pgpEXglc1cGQ7.pgp Description: PGP signature
Re: user traffic accounting
How would ipfm work for this? http://freshmeat.net/projects/ipfm/ > please direct me to some documentation on ways to account for user > traffic on a single machine, acting as BIND9, apache, postfix, and sshd > server for a number of users. i need to get as close as possible to > exact traffic volume measurements to do proper billing, and > (unfortunately), i can't use an upstream router for that. -- -- GNU PGP public key http://www.annapolislinux.org/docs/public_key/GnuPG.txt - Ted Knab
Re: user traffic accounting
also sprach Thedore Knab <[EMAIL PROTECTED]> [2002.01.07.1624 +0100]: > How would ipfm work for this? > > http://freshmeat.net/projects/ipfm/ this strikes me as a nice tool, but one that needs to run on a router/gateway/firewall, and one which can only differentiate according to IPs. if IPs were all i had to worry about, then i could just use iptables... thanks for replying though! have there been other replies to the list? i screwed my procmail temporarily and lost all mail since my original post... and the archives have nothing yet... -- martin; (greetings from the heart of the sun.) \ echo mailto: !#^."<*>"|tr "<*> mailto:"; net@madduck wind catches lily, scattering petals to the ground. segmentation fault. msg04725/pgp0.pgp Description: PGP signature
user traffic accounting
hi folks, please direct me to some documentation on ways to account for user traffic on a single machine, acting as BIND9, apache, postfix, and sshd server for a number of users. i need to get as close as possible to exact traffic volume measurements to do proper billing, and (unfortunately), i can't use an upstream router for that. i want to account for total traffic, which includes the following challenges: - Shell: every user has ssh access. i need to be able to keep track of every byte coming in and out of sshd, but also any data sent to or received from the internet while using the shell account. - HTTP: a user has zero or more domains hosted on the system, all request and response volume should be added to that users accounting data. - Mail: any mail that the user receives should be byte-counted. the same applies to mail sent from the user account via sendmail, mail sent via port 25, and mail relayed (TLS client authentication). - BIND: c.f. with HTTP, basically the same applies. if you ask me, this sounds like a horrible task. any tips from the ISP experts? -- martin; (greetings from the heart of the sun.) \ echo mailto: !#^."<*>"|tr "<*> mailto:"; [EMAIL PROTECTED] don't hate yourself in the morning -- sleep till noon. pgp1NHNbcNofw.pgp Description: PGP signature
Re: user traffic accounting
How would ipfm work for this? http://freshmeat.net/projects/ipfm/ > please direct me to some documentation on ways to account for user > traffic on a single machine, acting as BIND9, apache, postfix, and sshd > server for a number of users. i need to get as close as possible to > exact traffic volume measurements to do proper billing, and > (unfortunately), i can't use an upstream router for that. -- -- GNU PGP public key http://www.annapolislinux.org/docs/public_key/GnuPG.txt - Ted Knab -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
user traffic accounting
hi folks, please direct me to some documentation on ways to account for user traffic on a single machine, acting as BIND9, apache, postfix, and sshd server for a number of users. i need to get as close as possible to exact traffic volume measurements to do proper billing, and (unfortunately), i can't use an upstream router for that. i want to account for total traffic, which includes the following challenges: - Shell: every user has ssh access. i need to be able to keep track of every byte coming in and out of sshd, but also any data sent to or received from the internet while using the shell account. - HTTP: a user has zero or more domains hosted on the system, all request and response volume should be added to that users accounting data. - Mail: any mail that the user receives should be byte-counted. the same applies to mail sent from the user account via sendmail, mail sent via port 25, and mail relayed (TLS client authentication). - BIND: c.f. with HTTP, basically the same applies. if you ask me, this sounds like a horrible task. any tips from the ISP experts? -- martin; (greetings from the heart of the sun.) \ echo mailto: !#^."<*>"|tr "<*> mailto:"; net@madduck don't hate yourself in the morning -- sleep till noon. msg04721/pgp0.pgp Description: PGP signature
Traffic shaping and traffic accounting on one box
Hello all, About a year ago I set up a traffic shaping router using debian and cbq.init to allocate measured bandwidth for a group of clients, and used ipac to measure the actual traffic. After a month or two, I found out that the reports generated by ipacsum were grossly inaccurate (up to 6 times as much traffic was reported as existed). As I was unable to find the error, I simply set up a different accounting package (trafstats) on another system, which works fine. I'm now being asked, however, to put trafstats and cbq on one box, and I've reached the tentative hypothesis that the original problem was not due to a bug in ipac, but because traffic *shaping* occurs at the point where packets exit the computer, while traffic *accounting* occurs at the point where packets arrive at the computer -- so trafstats will suffer the same problem. My gut instinct says I'm right, but can anyone here think of an obvious reason why this might be wrong? Cheers, Shad. -- Rens Houben |opinions are mine Resident linux guru and sysadmin | if my employers have one Systemec Internet Services. |they'll tell you themselves PGP public key at http://suzaku.systemec.nl/shadur.key.asc pgpSrPdxN53qi.pgp Description: PGP signature
Traffic shaping and traffic accounting on one box
Hello all, About a year ago I set up a traffic shaping router using debian and cbq.init to allocate measured bandwidth for a group of clients, and used ipac to measure the actual traffic. After a month or two, I found out that the reports generated by ipacsum were grossly inaccurate (up to 6 times as much traffic was reported as existed). As I was unable to find the error, I simply set up a different accounting package (trafstats) on another system, which works fine. I'm now being asked, however, to put trafstats and cbq on one box, and I've reached the tentative hypothesis that the original problem was not due to a bug in ipac, but because traffic *shaping* occurs at the point where packets exit the computer, while traffic *accounting* occurs at the point where packets arrive at the computer -- so trafstats will suffer the same problem. My gut instinct says I'm right, but can anyone here think of an obvious reason why this might be wrong? Cheers, Shad. -- Rens Houben |opinions are mine Resident linux guru and sysadmin | if my employers have one Systemec Internet Services. |they'll tell you themselves PGP public key at http://suzaku.systemec.nl/shadur.key.asc msg04359/pgp0.pgp Description: PGP signature
RE: traffic accounting
I use fiprad ( Fast IP router accounting daemon) for logging traffic from multiple gateways to a central mSQL server. It uses stuff all CPU. I am very impressed with it. I have added a few small things of my own such as an fiprad.rc start/stop script and am working on some PHP scripts for interacting with the data on the mSQL server and a few other basic things. I intend to offer everything I have done to the maintainers of the package, so it can be included, if they dont produce something first that is. http://www.umplug.org/fipra/ Cheers, Richard -Original Message- From: Teun Vink [mailto:[EMAIL PROTECTED]] Sent: Friday, 19 January 2001 3:17 a.m. To: [EMAIL PROTECTED] Subject: traffic accounting Hi, I would like to setup up some sort of traffic accounting in our network. I know how to do this using ipchains rules, but the problem is that our network is completely redundant, so each machine in the network has two gateways (both Debian boxes). Does anybody know of a tool which can automatically combine the accounting of multiple routers into one set of statistics? Regards, Teun -- Teun Vink - [EMAIL PROTECTED] - icq: 15001247 - http://teun.moonblade.net -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED] -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: traffic accounting
On Thu, 18 Jan 2001, Alexander Reelsen wrote: > Hi > > On Thu, Jan 18, 2001 at 03:34:52PM +0100, Teun Vink wrote: > > Well.. I especially need numbers, since we want to bill excessive traffic > Shouldn't it be sufficient then do sum up the netacct data of both > interfaces? > > > MfG/Regards, Alexander > > Yeah of course... but I wanted to know if there's a tool which can do that for me, instead of writing some scripts to combine data and add it up... Teun -- Teun Vink - [EMAIL PROTECTED] - icq: 15001247 - http://teun.moonblade.net -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: traffic accounting
On Thu, 18 Jan 2001, Teun Vink wrote: > > Hi, > > I would like to setup up some sort of traffic accounting in our > network. I know how to do this using ipchains rules, but the problem is > that our network is completely redundant, so each machine in the network > has two gateways (both Debian boxes). > > Does anybody know of a tool which can automatically combine the accounting > of multiple routers into one set of statistics? > There is a tool called 'fipra' which I and a friend developed. it pulls what netblock it should log and to where from a mysql server. You can find it out on the net and it works with linux kernels up to 2.2.16.. I have a new patch done that works with later 2.2.x kernels and I will push that out before the weekend. it can easily handle accounting of 5000 ip's traffic att 30mbit or more, depending on the speed of the machine. Regards Roger A -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: traffic accounting
Hi On Thu, Jan 18, 2001 at 03:34:52PM +0100, Teun Vink wrote: > Well.. I especially need numbers, since we want to bill excessive traffic Shouldn't it be sufficient then do sum up the netacct data of both interfaces? MfG/Regards, Alexander -- Alexander Reelsen http://joker.rhwd.de [EMAIL PROTECTED] GnuPG: pub 1024D/F0D7313C sub 2048g/6AA2EDDB [EMAIL PROTECTED] 7D44 F4E3 1993 FDDF 552E 7C88 EE9C CBD1 F0D7 313C Securing Debian:http://joker.rhwd.de/doc/Securing-Debian-HOWTO -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: traffic accounting
On Thu, 18 Jan 2001, Alexander Reelsen wrote: > Hi > > On Thu, Jan 18, 2001 at 03:16:34PM +0100, Teun Vink wrote: > > I would like to setup up some sort of traffic accounting in our > > network. I know how to do this using ipchains rules, but the problem is > > that our network is completely redundant, so each machine in the network > > has two gateways (both Debian boxes). > > > Does anybody know of a tool which can automatically combine the accounting > > of multiple routers into one set of statistics? > Well, if you need graphical accounting you can try to stick with Hoth > (incidentally written by me ;)). You can stack whatever data you want on > the top of each other (the example graph on the page stacks tcp with icmp > with irc, what is completely senseless...), so you can stack the traffic > of two interfaces as well. > > It is based on RRDtool to store the data and the rest is a small perl > script. See more at: > http://joker.rhwd.de/software/hoth > > Biggest caveat: Not a seamless installation and almost no few docs. > > And if someone helps me to read the netlink sockets for accounting in > Linux 2.4 I will port it as well. I wasn't successful yet in any way, > neither in perl nor in python (help is really appreciated! :)).. > > > MfG/Regards, Alexander > > Well.. I especially need numbers, since we want to bill excessive traffic :-) But I be sure to take a look! Teun -- Teun Vink - [EMAIL PROTECTED] - icq: 15001247 - http://teun.moonblade.net -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: traffic accounting
Hi On Thu, Jan 18, 2001 at 03:16:34PM +0100, Teun Vink wrote: > I would like to setup up some sort of traffic accounting in our > network. I know how to do this using ipchains rules, but the problem is > that our network is completely redundant, so each machine in the network > has two gateways (both Debian boxes). > Does anybody know of a tool which can automatically combine the accounting > of multiple routers into one set of statistics? Well, if you need graphical accounting you can try to stick with Hoth (incidentally written by me ;)). You can stack whatever data you want on the top of each other (the example graph on the page stacks tcp with icmp with irc, what is completely senseless...), so you can stack the traffic of two interfaces as well. It is based on RRDtool to store the data and the rest is a small perl script. See more at: http://joker.rhwd.de/software/hoth Biggest caveat: Not a seamless installation and almost no few docs. And if someone helps me to read the netlink sockets for accounting in Linux 2.4 I will port it as well. I wasn't successful yet in any way, neither in perl nor in python (help is really appreciated! :)).. MfG/Regards, Alexander -- Alexander Reelsen http://joker.rhwd.de [EMAIL PROTECTED] GnuPG: pub 1024D/F0D7313C sub 2048g/6AA2EDDB [EMAIL PROTECTED] 7D44 F4E3 1993 FDDF 552E 7C88 EE9C CBD1 F0D7 313C Securing Debian:http://joker.rhwd.de/doc/Securing-Debian-HOWTO -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
traffic accounting
Hi, I would like to setup up some sort of traffic accounting in our network. I know how to do this using ipchains rules, but the problem is that our network is completely redundant, so each machine in the network has two gateways (both Debian boxes). Does anybody know of a tool which can automatically combine the accounting of multiple routers into one set of statistics? Regards, Teun -- Teun Vink - [EMAIL PROTECTED] - icq: 15001247 - http://teun.moonblade.net -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
