Re: [LARTC] strange iptables mangle problem
foxy 202 wrote: > > Hi all, >I manage network with two connections with l00Mbit > In the past when network wasn't so load everything was OK, now > in pick hours load over border server from 1.0 to 1.5 / it isn't so > big / > and for me is very strange why I have increasing of ping timeout > from 0.5- 5ms in normal hour to 50-100 ms in pick hours.. > > server is with good hardware > AMD 64 Dualcore 3800+ > Intel Gigabit Ethernet > 1 GB RAM > Debian sarge 2.6.16 #2 SMP kernel > > I use about 240 mangle rules with iptables to mark download traffic > and to > limit it but when I try to load more rules server increase load and > begin to drop > packages :( > > my question is why when I try to load new 200 mangle rules / only > mangle rules / server increase load average and ping timeout increase > to 50-100 ms > and second is what is better solution for networks with more then > 100Mbit traffic .. > to use iptables mangle rules + u32 or to use more u32 filters and > less mangle rules ? > > Actually I don't have experience with so big traffic and I need any > advice is welcome. > > > Best Regards > Emil Emil, I don't have any real answers but I encountered the same problem you have, except your hardware is a lot better than mine. I'd load 255 rules and the keyboard would become unresponsive and the network was terribly slow. Not just pings, everything. I changed the NIC and that helped. I've forgotten what I replaced it with, but it uses the Tulip driver and it is 100Mbit. I changed iptables source code for connection tracking. TCP conntrack is set to track connections for 5 DAYS! If I recall correctly, I changed that to 20 minutes. That reduced the size of /proc/net/ip_conntrack and that at least made the keyboard OK, but it was still not enough. You should search the mailing list archives for hashing. (I gave up trying to maintain 255 marks.) -- gypsy ___ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
Re: [LARTC] created new q_disc, inserted module, tc tells me unknown qdisc
> George P Nychis wrote:
>>
>> Hi,
>>
>> I am trying to install a proprietary qdisc made for research, it is not
>> publically released yet, however its been used several times so i know
>> it works.
>>
>> The files included are: q_xcp.c: static int xcp_parse_opt() static int
>> xcp_print_opt() static int xcp_print_xstats() struct qdisc_util xcp_util
>> = { "NULL", "xcp" . };
>>
>> sch_xcp.c: static int xcp_enqueue() static int xcp_requeue() static struct
>> sk_buff * xcp_dequeue() struct Qdisc_ops xcp_qdisc_ops ={
>> NULL,NULL,"xcp", };
>>
>> printk(KERN_INFO "XCP qdisc module loaded.\n"); return
>> register_qdisc(&xcp_qdisc_ops);
>>
>> So, i make everything successfully, it creates q_xcp.so and copies it
>> to /usr/lib and sch_xcp.o which it copies to /lib/modules/... so then I
>> "insmod sch_xcp" and i see in dmesg: "XCP qdisc module loaded."
>>
>> I then try: "tc qdisc add dev eth0 root xcp capacity 10Mbit limit 500"
>> and get: "Unknown qdisc "xcp", hence option "capacity" is unparsable"
>>
>> So then I read the INSTALL further to find some sort of solution and it
>> mentions: This again assumes "tc" version is 2.4.7. If your "tc" is a
>> different version, download the iproute2 source code, and edit Makefile
>> to point "TC_INCLUDE" to "-I/iproute2/include -I/iproute2/tc"
>>
>> So, i did that, and i recompiled the q_xcp.so: lanthanum-ini src-1.0.1 #
>> make q_xcp.so cc -O2 -fPIC
>> -I/var/tmp/portage/iproute2-2.6.11.20050310-r1/work/iproute2-2.6.11/inc
>> lude/
>> -I/var/tmp/portage/iproute2-2.6.11.20050310-r1/work/iproute2-2.6.11/tc_
>> include -o q_xcp.o -c q_xcp.c ld -shared -o q_xcp.so q_xcp.o rm -f
>> q_xcp.o
>>
>> But i still get the same error so then my very final last effort
>> was to move q_xcp.c to my iproute2 source code tc/ directory and added
>> this to the makefile: TCMODULES += q_xcp.o
>>
>> Then I compiled tc, and i check tc to see if the xcp qdisc functions
>> were loaded: lanthanum-ini tc # nm tc | grep xcp 080531ec t xcp_parse_opt
>> 080533e0 t xcp_print_opt 08053426 t xcp_print_xstats 08070cc0 D xcp_util
>>
>>
>> And finally: lanthanum-ini tc # ./tc qdisc add dev ath0 root xcp
>> capacity 54Mbit limit 500 Unknown qdisc "xcp", hence option "capacity"
>> is unparsable
>>
>> I have no clue :( I figured that putting the .so into /usr/lib would
>> have been enough. Sorry for the long e-mail, I hope someone can help,
>> and thank you for your time even if you don't know the solution but
>> read this :)
>>
>> - George
>
> George,
>
> Please show us iproute/include/linux/pkt_sched.h
>
> There shouldn't be anything there the author wishes to keep private. --
> gypsy
>
>
They did not include the iproute source code that they used... they only
included the q_xcp.c to create the q_xcp.so ... therefore my pkt_sched.h i am
using is from this build: iproute2-2.6.11.20050310-r1
I can certainly post it if you need, just let me know
___
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
Re: [LARTC] created new q_disc, inserted module, tc tells me unknown qdisc
George P Nychis wrote:
>
> Hi,
>
> I am trying to install a proprietary qdisc made for research, it is not
> publically released yet, however its been used several times so i know it
> works.
>
> The files included are:
> q_xcp.c:
>static int xcp_parse_opt()
>static int xcp_print_opt()
>static int xcp_print_xstats()
>struct qdisc_util xcp_util = { "NULL", "xcp" . };
>
> sch_xcp.c:
>static int xcp_enqueue()
>static int xcp_requeue()
>static struct sk_buff * xcp_dequeue()
>
>
>struct Qdisc_ops xcp_qdisc_ops ={ NULL,NULL,"xcp", };
>
>printk(KERN_INFO "XCP qdisc module loaded.\n");
>return register_qdisc(&xcp_qdisc_ops);
>
> So, i make everything successfully, it creates q_xcp.so and copies it to
> /usr/lib and sch_xcp.o which it copies to /lib/modules/... so then I "insmod
> sch_xcp" and i see in dmesg:
> "XCP qdisc module loaded."
>
> I then try:
> "tc qdisc add dev eth0 root xcp capacity 10Mbit limit 500" and get:
> "Unknown qdisc "xcp", hence option "capacity" is unparsable"
>
> So then I read the INSTALL further to find some sort of solution and it
> mentions:
> This again assumes "tc" version is 2.4.7. If your "tc" is a different
> version, download the iproute2 source code, and edit Makefile to
> point "TC_INCLUDE" to "-I/iproute2/include -I/iproute2/tc"
>
> So, i did that, and i recompiled the q_xcp.so:
> lanthanum-ini src-1.0.1 # make q_xcp.so
> cc -O2 -fPIC
> -I/var/tmp/portage/iproute2-2.6.11.20050310-r1/work/iproute2-2.6.11/include/
> -I/var/tmp/portage/iproute2-2.6.11.20050310-r1/work/iproute2-2.6.11/tc_include
> -o q_xcp.o -c q_xcp.c
> ld -shared -o q_xcp.so q_xcp.o
> rm -f q_xcp.o
>
> But i still get the same error so then my very final last effort was to
> move q_xcp.c to my iproute2 source code tc/ directory and added this to the
> makefile:
> TCMODULES += q_xcp.o
>
> Then I compiled tc, and i check tc to see if the xcp qdisc functions were
> loaded:
> lanthanum-ini tc # nm tc | grep xcp
> 080531ec t xcp_parse_opt
> 080533e0 t xcp_print_opt
> 08053426 t xcp_print_xstats
> 08070cc0 D xcp_util
>
> And finally:
> lanthanum-ini tc # ./tc qdisc add dev ath0 root xcp capacity 54Mbit limit 500
> Unknown qdisc "xcp", hence option "capacity" is unparsable
>
> I have no clue :( I figured that putting the .so into /usr/lib would have
> been enough. Sorry for the long e-mail, I hope someone can help, and thank
> you for your time even if you don't know the solution but read this :)
>
> - George
George,
Please show us
iproute/include/linux/pkt_sched.h
There shouldn't be anything there the author wishes to keep private.
--
gypsy
___
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
RE: [LARTC] Shaping per IP in PPPoE
hi, i use the roaringpenguin pppoe-server and limit the bandwidth per
interface with this script:
(im using freeradius plugins too, thats the reason of the
/var/run/radattr.pppx file)
(/etc/ppp/ip-up.d/0pppx_up)
#!/bin/sh
DOWN=`cat /var/run/radattr.$1 | grep 'RP-Downstream-Speed-Limit' | cut -d '
' -f 2`
UP=`cat /var/run/radattr.$1 | grep 'RP-Upstream-Speed-Limit' | cut -d ' ' -f
2`
# limit Download Bandwidth with a simple htb qdisc and class (add QoS
here?...)
/sbin/tc qdisc add dev $1 root handle 1: htb default 1
/sbin/tc class add dev $1 parent 1: classid 1:1 htb rate ${DOWN}kbit ceil
${DOWN}kbit burst 1540
/sbin/tc qdisc add dev $1 handle : ingress
/sbin/tc filter add dev $1 parent : protocol ip prio 50 u32 \
match ip src 0.0.0.0/0 \
police rate ${UP}kbit burst 10k drop flowid :1
and have another script for deleting the rules
(/etc/ppp/ip-down.d/0pppx_down):
#!/bin/sh
/sbin/tc qdisc del dev $1 root
/sbin/tc qdisc del dev $1 ingress
ppp executes this scripts each time an interface gets up or down.
hope it helps.
Roberto Scattini
From: Rani Ahmed <[EMAIL PROTECTED]>
To: lartc@mailman.ds9a.nl
Subject: [LARTC] Shaping per IP in PPPoE
Date: Tue, 11 Apr 2006 18:49:29 +0300
hi all.
i am currently now serving PPPoE in my area.
i had a script generated from tcng that worked perfectly before i started
serving PPPoE.
the issue is not in the script it self BUT in that "tc" code is not shaping
on the ethernet anymore BUT INSTEAD
on the pppX devices. I tested it and talking jargon, what should i do?
The issue is that for each PPPoE login, PPPoE-server creates on the server a
pppX device. that is 10 logins means 10 ppp devices. from ppp0 till ppp9.
and one might die upon disconnection.
___
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
_
Sabe más sobre la próxima generación del MSN Messenger.
http://imagine-msn.com/minisites/messenger/default.aspx?locale=es-ar
___
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
Re: [LARTC] HTB statistics granularity
> b) Is there a way to get a precise rate and pps for the last 10 seconds? > (issue of rate taking 3-4 mins to get to zero as opposed to instantly) Run the command, record the number of bytes and packets sent. (tc -s class show dev ppp0) Wait exactly 10 seconds (sleep 10) Run the command again, record the new values (tc -s class show dev ppp0) Compare the difference between the bytes and packets sent, divide by 10. As to the rest of your questions, I have no idea. I've only ever used the rate from the commend to get a rough idea of the current rate. Otherwise I use the bytes sent measurement to generate some rrdgraph's in which case doing measurements of bytes sent every 60 seconds is perfectly accurate for those 60 seconds as long as roll-over hasn't occured. - Jody ___ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
Re: [LARTC] Re: created new q_disc, inserted module, tc tells me unknown qdisc
> Maybe i should add debugging in tc.c and see if it sees the .so and fails to > load it or something. Yes. I would do that next. Compile tc with -g and then run it under gdb and step through that part to see what happens. -Tim Shepard [EMAIL PROTECTED] ___ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
[LARTC] HTB statistics granularity
Hi,I am running HTB and using the following command to get rate and pps statistics:tc -s class show dev eth0However what I am seeing is that if I run the command over and over again, I see the sent bytes and sent packets increment however the rate and pps have the following issues:1. The rate and pps values only seem to update after every 15 seconds2. They do not show until about after five seconds that the traffic has been running?3. If I run traffic constantly at 1500 kbps and then stop it, it takes about 3 to 4 minutes for the rate to go back to 0 kbps.Does anyone know:a) How are the rate and pps being calculated?b) Is there a way to get a precise rate and pps for the last 10 seconds? (issue of rate taking 3-4 mins to get to zero as opposed to instantly)Thanks in advance! How low will we go? Check out Yahoo! Messengers low PC-to-Phone call rates.___ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
Re: [LARTC] Shaping per IP in PPPoE
Hello Rani,
: i am currently now serving PPPoE in my area. i had a script
: generated from tcng that worked perfectly before i started
: serving PPPoE. the issue is not in the script it self BUT in that
: "tc" code is not shaping on the ethernet anymore BUT INSTEAD on
: the pppX devices. I tested it and talking jargon, what should i
: do?
:
: The issue is that for each PPPoE login, PPPoE-server creates on
: the server a pppX device. that is 10 logins means 10 ppp devices.
: from ppp0 till ppp9. and one might die upon disconnection.
I'd suggest simply using the pppoe ip-up configuration scripts to
call the appropriate tc or tcng commands. Since ip-up should be
called something like this:
ip-up ppp0 $TTY $SPEED 192.168.0.4 10.0.0.4 $OTHER
Is ip-up called by YOUR pppoe-server binary? I am not able to test
this.
you should be able to create a script that would either execute tc
commands or a create tcng file on the fly. I created the basic
structure of such a script below, although you could probably
add/replace your own shell functions (tc_sfq, tc_my_complex_config)
with a much more complex traffic control configuration.
Good luck,
-Martin
#! /bin/bash
#
# -- add queuing to an interface brought up by pppd, 2006-04-11; -MAB
#GPL
#
# ip-up
dev="$1"&& shift
pty="$1"&& shift
spd="$1"&& shift
lip="$1"&& shift
rip="$1"&& shift
logger () { command logger -it "${0##*/}" -- "$@" ; }
abort () { logger "$@" ; exit 1 ; }
tc_tbf () {
local dev="$1"&& shift
local lip="$1"&& shift
local rip="$1"&& shift
test "$dev" = "" && abort "${FUNCNAME}() called with no device name"
test "$lip" = "" && abort "${FUNCNAME}() called with no local IP"
test "$rip" = "" && abort "${FUNCNAME}() called with no remote IP"
cat <<-EOTC
tc qdisc add dev $dev root handle 1:0 tbf rate 1544kbit limit 20kB
burst 3kB
EOTC
}
# -- run all commands in a single shell that we instruct to quit on any error
#
tc_tbf "$dev" "$lip" "$rip" | bash -e
# -- did the shell complete successfully?
#
test "$?" -gt 0 && abort "Could not install traffic control on $dev."
logger "Installed traffic control configuration on $dev."
# -- end of file
--
Martin A. Brown --- Wonderfrog Enterprises --- [EMAIL PROTECTED]
___
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
Re: [LARTC] Re: created new q_disc, inserted module, tc tells me unknown qdisc
I am getting closer...
I added debugging, and noticed that it looks for:
snprintf(buf, sizeof(buf), "%s_qdisc_util", str);
However in q_xcp.c it had:
struct qdisc_util xcp_util = {
so I changed that to xcp_qdisc_util, and now i run tc:
lanthanum-ini tc # tc qdisc add dev ath0 root xcp capacity 54Mbit limit 100
Segmentation fault
This happens on this line:
q = dlsym(dlh, buf);
Since this is very hard for people to help me without the source code, i did
ask the author if it has been release publically and am waiting for a response.
In the meantime, it seems as though maybe instead of trying to get this to
work with a newer version of tc, i should install an old version of tc that the
module was original made for.
Though if anyone else has ideas let me know.
Thanks for all the help
- George
>> On Mon, 10 Apr 2006 21:41:39 -0400 (EDT) "George P Nychis"
>> <[EMAIL PROTECTED]> wrote:
>>
>>> Hi,
>>>
>>> I am trying to install a proprietary qdisc made for research, it is
>>> not publically released yet, however its been used several times so i
>>> know it works.
>>>
>>> The files included are: q_xcp.c: static int xcp_parse_opt() static
>>> int xcp_print_opt() static int xcp_print_xstats() struct qdisc_util
>>> xcp_util = { "NULL", "xcp" . };
>>>
>>> sch_xcp.c: static int xcp_enqueue() static int xcp_requeue() static
>>> struct sk_buff * xcp_dequeue() struct Qdisc_ops
>>> xcp_qdisc_ops ={ NULL,NULL,"xcp", };
>>>
>>> printk(KERN_INFO "XCP qdisc module loaded.\n"); return
>>> register_qdisc(&xcp_qdisc_ops);
>>>
>>>
>>> So, i make everything successfully, it creates q_xcp.so and copies it
>>> to /usr/lib and sch_xcp.o which it copies to /lib/modules/... so
>>> then I "insmod sch_xcp" and i see in dmesg: "XCP qdisc module loaded."
>>>
>>>
>>> I then try: "tc qdisc add dev eth0 root xcp capacity 10Mbit limit
>>> 500" and get: "Unknown qdisc "xcp", hence option "capacity" is
>>> unparsable"
>>>
>>> So then I read the INSTALL further to find some sort of solution and
>>> it mentions: This again assumes "tc" version is 2.4.7. If your "tc"
>>> is a different version, download the iproute2 source code, and edit
>>> Makefile to point "TC_INCLUDE" to "-I/iproute2/include
>>> -I/iproute2/tc"
>>>
>>> So, i did that, and i recompiled the q_xcp.so: lanthanum-ini
>>> src-1.0.1 # make q_xcp.so cc -O2 -fPIC
>>> -I/var/tmp/portage/iproute2-2.6.11.20050310-r1/work/iproute2-2.6.11/i
>>> nc lude/
>>> -I/var/tmp/portage/iproute2-2.6.11.20050310-r1/work/iproute2-2.6.11/t
>>> c_ include -o q_xcp.o -c q_xcp.c ld -shared -o q_xcp.so q_xcp.o rm -f
>>> q_xcp.o
>>>
>>> But i still get the same error so then my very final last effort
>>> was to move q_xcp.c to my iproute2 source code tc/ directory and
>>> added this to the makefile: TCMODULES += q_xcp.o
>>>
>>> Then I compiled tc, and i check tc to see if the xcp qdisc functions
>>> were loaded: lanthanum-ini tc # nm tc | grep xcp 080531ec t
>>> xcp_parse_opt 080533e0 t xcp_print_opt 08053426 t xcp_print_xstats
>>> 08070cc0 D xcp_util
>>>
>>>
>>> And finally: lanthanum-ini tc # ./tc qdisc add dev ath0 root xcp
>>> capacity 54Mbit limit 500 Unknown qdisc "xcp", hence option
>>> "capacity" is unparsable
>>>
>>>
>>> I have no clue :( I figured that putting the .so into /usr/lib would
>>> have been enough. Sorry for the long e-mail, I hope someone can
>>> help, and thank you for your time even if you don't know the solution
>>> but read this :)
>>>
>>> - George
>>>
>>
>> The .so needs to go in /usr/lib/tc (assuming you are running relatively
>> recent version of iproute2 tools).
>>
>> Read source to tc.c where it calls dlopen.
>>
>>
>
> Still didn't seem to solve the problem :\
>
> In my tc.c i have: snprintf(buf, sizeof(buf), "/usr/lib/tc/q_%s.so", str);
>
>
> Also: lanthanum-ini tc # ls /usr/lib/tc experimental.dist normal.dist
> pareto.dist paretonormal.dist q_netem.so q_xcp.so
>
> And finally: lanthanum-ini tc # tc qdisc add dev ath0 root xcp capacity
> 54Mbit limit 500 Unknown qdisc "xcp", hence option "capacity" is
> unparsable
>
> Maybe i should add debugging in tc.c and see if it sees the .so and fails
> to load it or something.
>
> Any other suggestions?
>
> Thanks for all the responses, George
>
> ___ LARTC mailing list
> LARTC@mailman.ds9a.nl
> http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
>
>
--
___
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
[LARTC] Re: created new q_disc, inserted module, tc tells me unknown qdisc
> On Mon, 10 Apr 2006 21:41:39 -0400 (EDT) "George P Nychis"
> <[EMAIL PROTECTED]> wrote:
>
>> Hi,
>>
>> I am trying to install a proprietary qdisc made for research, it is not
>> publically released yet, however its been used several times so i know
>> it works.
>>
>> The files included are: q_xcp.c: static int xcp_parse_opt() static int
>> xcp_print_opt() static int xcp_print_xstats() struct qdisc_util xcp_util
>> = { "NULL", "xcp" . };
>>
>> sch_xcp.c: static int xcp_enqueue() static int xcp_requeue() static struct
>> sk_buff * xcp_dequeue() struct Qdisc_ops xcp_qdisc_ops ={
>> NULL,NULL,"xcp", };
>>
>> printk(KERN_INFO "XCP qdisc module loaded.\n"); return
>> register_qdisc(&xcp_qdisc_ops);
>>
>>
>> So, i make everything successfully, it creates q_xcp.so and copies it
>> to /usr/lib and sch_xcp.o which it copies to /lib/modules/... so then I
>> "insmod sch_xcp" and i see in dmesg: "XCP qdisc module loaded."
>>
>> I then try: "tc qdisc add dev eth0 root xcp capacity 10Mbit limit 500"
>> and get: "Unknown qdisc "xcp", hence option "capacity" is unparsable"
>>
>> So then I read the INSTALL further to find some sort of solution and it
>> mentions: This again assumes "tc" version is 2.4.7. If your "tc" is a
>> different version, download the iproute2 source code, and edit Makefile
>> to point "TC_INCLUDE" to "-I/iproute2/include -I/iproute2/tc"
>>
>> So, i did that, and i recompiled the q_xcp.so: lanthanum-ini src-1.0.1 #
>> make q_xcp.so cc -O2 -fPIC
>> -I/var/tmp/portage/iproute2-2.6.11.20050310-r1/work/iproute2-2.6.11/inc
>> lude/
>> -I/var/tmp/portage/iproute2-2.6.11.20050310-r1/work/iproute2-2.6.11/tc_
>> include -o q_xcp.o -c q_xcp.c ld -shared -o q_xcp.so q_xcp.o rm -f
>> q_xcp.o
>>
>> But i still get the same error so then my very final last effort
>> was to move q_xcp.c to my iproute2 source code tc/ directory and added
>> this to the makefile: TCMODULES += q_xcp.o
>>
>> Then I compiled tc, and i check tc to see if the xcp qdisc functions
>> were loaded: lanthanum-ini tc # nm tc | grep xcp 080531ec t xcp_parse_opt
>> 080533e0 t xcp_print_opt 08053426 t xcp_print_xstats 08070cc0 D xcp_util
>>
>>
>> And finally: lanthanum-ini tc # ./tc qdisc add dev ath0 root xcp
>> capacity 54Mbit limit 500 Unknown qdisc "xcp", hence option "capacity"
>> is unparsable
>>
>>
>> I have no clue :( I figured that putting the .so into /usr/lib would
>> have been enough. Sorry for the long e-mail, I hope someone can help,
>> and thank you for your time even if you don't know the solution but
>> read this :)
>>
>> - George
>>
>
> The .so needs to go in /usr/lib/tc (assuming you are running relatively
> recent version of iproute2 tools).
>
> Read source to tc.c where it calls dlopen.
>
>
Still didn't seem to solve the problem :\
In my tc.c i have:
snprintf(buf, sizeof(buf), "/usr/lib/tc/q_%s.so", str);
Also:
lanthanum-ini tc # ls /usr/lib/tc
experimental.dist normal.dist pareto.dist paretonormal.dist q_netem.so
q_xcp.so
And finally:
lanthanum-ini tc # tc qdisc add dev ath0 root xcp capacity 54Mbit limit 500
Unknown qdisc "xcp", hence option "capacity" is unparsable
Maybe i should add debugging in tc.c and see if it sees the .so and fails to
load it or something.
Any other suggestions?
Thanks for all the responses,
George
___
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
[LARTC] Re: created new q_disc, inserted module, tc tells me unknown qdisc
On Mon, 10 Apr 2006 21:41:39 -0400 (EDT)
"George P Nychis" <[EMAIL PROTECTED]> wrote:
> Hi,
>
> I am trying to install a proprietary qdisc made for research, it is not
> publically released yet, however its been used several times so i know it
> works.
>
> The files included are:
> q_xcp.c:
>static int xcp_parse_opt()
>static int xcp_print_opt()
>static int xcp_print_xstats()
>struct qdisc_util xcp_util = { "NULL", "xcp" . };
>
> sch_xcp.c:
>static int xcp_enqueue()
>static int xcp_requeue()
>static struct sk_buff * xcp_dequeue()
>
>
>struct Qdisc_ops xcp_qdisc_ops ={ NULL,NULL,"xcp", };
>
>printk(KERN_INFO "XCP qdisc module loaded.\n");
>return register_qdisc(&xcp_qdisc_ops);
>
>
> So, i make everything successfully, it creates q_xcp.so and copies it to
> /usr/lib and sch_xcp.o which it copies to /lib/modules/... so then I "insmod
> sch_xcp" and i see in dmesg:
> "XCP qdisc module loaded."
>
> I then try:
> "tc qdisc add dev eth0 root xcp capacity 10Mbit limit 500" and get:
> "Unknown qdisc "xcp", hence option "capacity" is unparsable"
>
> So then I read the INSTALL further to find some sort of solution and it
> mentions:
> This again assumes "tc" version is 2.4.7. If your "tc" is a different
> version, download the iproute2 source code, and edit Makefile to
> point "TC_INCLUDE" to "-I/iproute2/include -I/iproute2/tc"
>
> So, i did that, and i recompiled the q_xcp.so:
> lanthanum-ini src-1.0.1 # make q_xcp.so
> cc -O2 -fPIC
> -I/var/tmp/portage/iproute2-2.6.11.20050310-r1/work/iproute2-2.6.11/include/
> -I/var/tmp/portage/iproute2-2.6.11.20050310-r1/work/iproute2-2.6.11/tc_include
> -o q_xcp.o -c q_xcp.c
> ld -shared -o q_xcp.so q_xcp.o
> rm -f q_xcp.o
>
> But i still get the same error so then my very final last effort was to
> move q_xcp.c to my iproute2 source code tc/ directory and added this to the
> makefile:
> TCMODULES += q_xcp.o
>
> Then I compiled tc, and i check tc to see if the xcp qdisc functions were
> loaded:
> lanthanum-ini tc # nm tc | grep xcp
> 080531ec t xcp_parse_opt
> 080533e0 t xcp_print_opt
> 08053426 t xcp_print_xstats
> 08070cc0 D xcp_util
>
> And finally:
> lanthanum-ini tc # ./tc qdisc add dev ath0 root xcp capacity 54Mbit limit 500
> Unknown qdisc "xcp", hence option "capacity" is unparsable
>
>
> I have no clue :( I figured that putting the .so into /usr/lib would have
> been enough. Sorry for the long e-mail, I hope someone can help, and thank
> you for your time even if you don't know the solution but read this :)
>
> - George
>
The .so needs to go in /usr/lib/tc (assuming you are running relatively recent
version
of iproute2 tools).
Read source to tc.c where it calls dlopen.
___
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
RE: [LARTC] Problems matching by mac address
Sorry for bumping this one, but I wanted to know if anyone else had any ideas as well. I'm a bit baffled as to why this isn't working. Thanks. Adam -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Adam M. Towarnyckyj Sent: Friday, April 07, 2006 2:06 PM To: lartc@mailman.ds9a.nl Subject: RE: [LARTC] Problems matching by mac address Hey Alexey, Thanks for the input. I think that lack of two F's was a typo on my part but I tried it anyways and it still does not work. I also added the ceil to it with no luck. I'm a bit confused on what you meant by not having any rules to classify from root down to 12:. Can you elaborate or show me an example? As I stated before, this is pretty much the exact setup I used when I filtered by destination IP. The only thing I'm changing now is the actual filter command. Everything else has been in place for a while. Thanks. Adam -Original Message- From: Alexey Toptygin [mailto:[EMAIL PROTECTED] Sent: Tuesday, April 04, 2006 3:15 PM To: Adam M. Towarnyckyj Cc: lartc@mailman.ds9a.nl Subject: Re: [LARTC] Problems matching by mac address On Tue, 4 Apr 2006, Adam M. Towarnyckyj wrote: >I recently read on a prior post as well as the FAQ that > packets can be limited by mac address using the u32 filter. I attempted > this and, while all the commands went through with no errors, it is not > limiting at all. I'm attempting to limit all IP traffic to a specific > destination mac address (00:12:3f:05:43:7f). Here is a quick rundown of > the commands I've used: Not sure that this will help, but > tc qdisc add dev eth1 parent 1:2 handle 12: htb > tc class add dev eth1 parent 12: classid 12:10 htb rate 128kbit no ceil? > tc filter add dev eth1 protocol ip parent 12: prio 5 u32 match u16 > 0x0800 0x at -2 match u32 0x3f05437f 0xFF at -12 match u16 > 0x0012 0x at -14 flowid 12:10 Shouldn't that be "match u32 0x3f05437f 0x at -12" (2 more Fs) Also, what you sent didn't have any rules to classify from root down to 12: so the above filter won't be consulted... Alexey ___ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc ___ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
Re: [LARTC] I dont want to shape a host
Martin, Thanks for the answer. I will study your topology and try to make this happens. Att, Nataniel Klug Martin A. Brown escreveu: Nataniel, There are probably a handful of ways to solve this problem. Two pop to mind right away. : I am still reading about my QoS rules and I need that one of my : servers (that is into my LAN but has an routing ip address) did : not get into the qos rules I have. So I want that all traffic : coming or going to that specifc host did not get shapped by any : traffic control and do not get even into a QoS class. How can I : do this? Option A: specify "default 0" in your HTB qdisc declaration If you install the HTB qdisc with a "default 0" parameter, you are telling HTB to dequeue unclassified packets as fast as the hardware will accept the packets. Here's an example: tc qdisc add dev eth0 root handle 1:0 htb default 0 Now, any unclassified packets will simply be dequeued as fast as your hardware can do it. If you are trying to remain the bottleneck between you and the Internet, it is quite likely that this configuration will defeat your goal. Option B: make a deeper HTB tree Build the following: class 1:0, rate = ceil = hardware maximum bitrate class 2:0, rate = low, ceil = hardware maximum bitrate class 3:0, rate = low, ceil = maximum for everybody else root +--- HTB 2:0 --- your "routing ip" (public | / server?) goes here +-- HTB 1:0 --- \ +--- HTB 3:0 | +--- HTB 3:1 +--- HTB 3:2 +--- HTB 3:3 |... +--- HTB 3:N Now, you simply attach your filters to 1:0, like you did before, and put all traffic for your "routing ip" into the 2:0 class. If the rate on class 2:0 stays "low", but its ceiling is the same as the rate/ceil on 1:0, then you'll effectively get borrowing up to maximum available throughput for HTB 2:0. Good luck, -Martin ___ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
[LARTC] Shaping per IP in PPPoE
hi all. i am currently now serving PPPoE in my area. i had a script generated from tcng that worked perfectly before i started serving PPPoE. the issue is not in the script it self BUT in that "tc" code is not shaping on the ethernet anymore BUT INSTEAD on the pppX devices. I tested it and talking jargon, what should i do? The issue is that for each PPPoE login, PPPoE-server creates on the server a pppX device. that is 10 logins means 10 ppp devices. from ppp0 till ppp9. and one might die upon disconnection. ___ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
Re: [LARTC] Re: EF & AF filters with HTB
Hi Robert! It's good to see that I could help :). It took me a while to figure things out too. As for opalsoft.net, I suggest you to read all the linux diffserv stuff, it is very useful. I based many of my scripts on what can found there (read that site month ago and still I find new things), though I believe some examles, like af-htb are there for educational reasons, as he also claims it, not for practical usage. To sum up, I would suggest to only use EF and BE, or EF and AF11 first, test it with ping/iperf/etc. and tc qdisc/class/filter -s -d show, ethereal... Only then go for complex stuff. Script: Overall it looks good for me. Why not use 1:20 for AF if you used 1:10 for EF? Just clearer, I believe. I never used gred, so I can't help much on that. Anyway if you asked for that, I do not see why would you use 1:111 or such. Classids are only guidelines. parent ".." says where to attach the rule, flowid ".." says where to send the packet for classification. As your AF filters say, AF11,AF12,AF13 packets will all be directed to 1:11, and share the default 12kbit. To add: I believe you can concatenate matches if all will direct packets to same class, so can have a filter like: tc filter add dev $DEVICE parent 1:0 protocol ip prio 2 u32 match ip tos 0x28 0xff match ip tos 0x30 0xff match ip tos 0x38 0xff flowid 1:11 But check this, I'm not sure. Finally, I do not know what will happen to BE if you run this setup. Where will it go? I would create the main htb under 1:0 with default 30, so send anything unmatched (BE) to 1:30. Hope somewhere I wrote something you wanted,or I don't get your question. Ferenc __ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com ___ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
[LARTC] Htb queueing problem
A note to confirm that "-m physdev --physdev-is-bridged" in the iptables
command does enable iptables to work in a bridged environment. I was
fighting the same problem and this indeed solved it.
Below is my test script running on a two NIC Debian 3.1 266MHz bridge.
Before adding the physdev flag, only the "tc filter" commands worked but
now the iptables commands also correctly classify the packets both with
the MARK and CLASSIFY approaches. Note that the tc classes were setup
to give clear indication of which class was affecting the flow.
Edwin
-test tc script--
#!/bin/bash
RATE=8000
#if [ x$1 = 'xstop' ]; then
if [ tc ]; then
echo "Deleting qdisc for eth1"
tc qdisc del dev eth1 root
fi
tc qdisc add dev eth1 root handle 1:0 htb default 90
tc class add dev eth1 parent 1:0 classid 1:1 htb rate ${RATE}kbit ceil
${RATE}kbit
tc class add dev eth1 parent 1:1 classid 1:10 htb rate 3000kbit ceil
3000kbit
tc class add dev eth1 parent 1:1 classid 1:20 htb rate 1500kbit ceil
1500kbit
tc class add dev eth1 parent 1:1 classid 1:30 htb rate 1000kbit ceil
1000kbit
tc class add dev eth1 parent 1:1 classid 1:50 htb rate 500kbit ceil 500kbit
tc class add dev eth1 parent 1:1 classid 1:90 htb rate 256kbit ceil 256kbit
tc qdisc add dev eth1 parent 1:10 handle 10: sfq perturb 10
tc qdisc add dev eth1 parent 1:20 handle 20: sfq perturb 10
tc qdisc add dev eth1 parent 1:30 handle 30: sfq perturb 10
tc qdisc add dev eth1 parent 1:50 handle 50: sfq perturb 10
tc qdisc add dev eth1 parent 1:90 handle 90: sfq perturb 10
#tc filter add dev eth1 parent 1:0 protocol ip u32 match ip sport 80
0x classid 1:10
#tc filter add dev eth1 parent 1:0 protocol ip u32 match ip sport 22
0x classid 1:20
#tc filter add dev eth1 parent 1:0 protocol ip u32 match ip sport 25
0x classid 1:50
#tc filter add dev eth1 parent 1:0 protocol ip u32 match ip sport 110
0x classid 1:50
iptables -F -t mangle
#out
#iptables -t mangle -A POSTROUTING -m physdev --physdev-is-bridged -p
tcp --sport 80 -j MARK --set-mark 2
#iptables -t mangle -A FORWARD -o eth1 -p tcp --sport 22 -j MARK
--set-mark 1
#iptables -t mangle -A FORWARD -o eth1 -p tcp --sport 25 -j MARK
--set-mark 1
#iptables -t mangle -A FORWARD -o eth1 -p tcp --sport 110 -j MARK
--set-mark 1
#
iptables -t mangle -A POSTROUTING -m physdev --physdev-is-bridged -p tcp
--sport 80 -j CLASSIFY --set-class 1:50
iptables -t mangle -A POSTROUTING -m physdev --physdev-is-bridged -p tcp
--sport 139 -j CLASSIFY --set-class 1:10
iptables -t mangle -A POSTROUTING -m physdev --physdev-is-bridged -p tcp
--sport 22 -j CLASSIFY --set-class 1:10
iptables -t mangle -A POSTROUTING -m physdev --physdev-is-bridged -p tcp
--sport 25 -j CLASSIFY --set-class 1:10
iptables -t mangle -A POSTROUTING -m physdev --physdev-is-bridged -p tcp
--sport 110 -j CLASSIFY --set-class 1:10
#tc filter add dev eth1 parent 1:0 protocol ip handle 1 fw classid 1:10
#tc filter add dev eth1 parent 1:0 protocol ip handle 2 fw classid 1:20
#tc filter add dev eth1 parent 1:0 protocol ip handle 3 fw classid 1:30
#tc filter add dev eth1 parent 1:0 protocol ip handle 5 fw classid 1:50
#tc filter add dev eth1 parent 1:0 protocol ip handle 9 fw classid 1:90
--
<=+=+=+==+=+=+==+=+=+=+=+=+=+=+=>
Edwin Whitelaw, P.E.
New River Valley Unwired, LLC
2200 Lonesome Dove Dr
Christiansburg, VA 24073
540-239-0318
___
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
[LARTC] trying to find out how much is on a drive?
I have a Ferdora 2 machine with 2 drives in it one has the OS and the main drive for our Samba server on it and the other is the Mirror drive. What I am trying to find out is how much is on the primary drive. When I use the "du" command I get a number that seems to be off. I have the mirror drive mounted to the primary drive, and it apears as though when I do the "du" command from the root of the drive is adds in some of the files from the mirror drive. What I am trying to figure out is if there is a way to use the "du" command and exclude the mounted mirror drive in my total amount. I used the command du -ch, which gave the grand total in human form which is what I want but it seems to include the mirror drive also. How do I exclude it from adding in the mirror. Thanks, Yvon Dubinsky e-commerce admin ___ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
Re: [LARTC] Re: EF & AF filters with HTB
Hello again, Many thanks for your help. I have done as you suggested, I can't get the other way to work. I guess keep it simple, right? We might need to remark at a later stage, I want to test this. Have you looked at http://www.opalsoft.net/qos/DS-29.htm yet? Please could you confirm something? Is the below right in terms of flowids for the AF1 class? With another example they use flowid 1:111, flowid 1:112, flowid 1:113. # HTB echo -e "Installing HTB queue discipline...\n" tc qdisc add dev $DEVICE handle 1:0 root htb tc class add dev $DEVICE parent 1:0 classid 1:1 htb rate 128Kbit ceil 128Kbit # EF echo -e "Installing EF class...\n" tc class add dev $DEVICE parent 1:1 classid 1:10 htb rate 60Kbit ceil 128Kbit tc qdisc add dev $DEVICE parent 1:10 pfifo limit 5 tc filter add dev $DEVICE parent 1:0 protocol ip prio 1 u32 match ip tos 0xb8 0xff flowid 1:10 # AF 1 echo -e "Installing AF11, AF12, AF13 classes...\n" tc class add dev $DEVICE parent 1:1 classid 1:11 htb rate 12Kbit ceil 128Kbit tc qdisc add dev $DEVICE parent 1:11 gred setup DPs 3 default 2 grio tc qdisc change dev $DEVICE parent 1:11 gred limit 60KB min 15KB max 45KB \ burst 20 avpkt 1000 bandwidth 128Kbit DP 1 probability 0.02 prio 2 tc qdisc change dev $DEVICE parent 1:11 gred limit 60KB min 15KB max 45KB \ burst 20 avpkt 1000 bandwidth 128Kbit DP 2 probability 0.04 prio 3 tc qdisc change dev $DEVICE parent 1:11 gred limit 60KB min 15KB max 45KB \ burst 20 avpkt 1000 bandwidth 128Kbit DP 3 probability 0.06 prio 4 tc filter add dev $DEVICE parent 1:0 protocol ip prio 2 u32 match ip tos 0x28 0xff flowid 1:11 tc filter add dev $DEVICE parent 1:0 protocol ip prio 2 u32 match ip tos 0x30 0xff flowid 1:11 tc filter add dev $DEVICE parent 1:0 protocol ip prio 2 u32 match ip tos 0x38 0xff flowid 1:11 On 10/04/06, pfer <[EMAIL PROTECTED]> wrote: > Hi Robert! > > If I get you right, you wish to use HTB at a node > for doing some QoS for packets based the DSCP they > already have, and NOT marking them to have that AF, > etc. > > ( > have you checked > http://www.opalsoft.net/qos/DS-310.htm? > he has some nice scripts > ) > > Why not simply use an u32 match on the TOS field for > filtering under your main htb? Why do you need DSMARK? > (haven't tried, but should work fine) > > For ex. having htb main shaper at 1:0 gives > > tc filter add dev eth0 parent 1:0 protocol ip prio 2 > u32 match ip tos 0xb8 0xff flowid 1:10 > (the EF htb branch is at 1:10) > > tc filter add dev eth0 parent 1:0 protocol ip prio 3 > u32 match ip tos 0x28 0xff flowid 1:20 > (the AF11 htb branch is at 1:20) > > At for bulk, create the htb main with: "default 30", > and add an 1:30 class with needed rate + burst and > red/sfq/etc. > U won't need any filter for this, anything > unclassified > (anything other than EF or AF11) will get there. > > If I get you wrong, and you wish to re-mark those > packets, I have some scripts for that, too. > > Contact me at [EMAIL PROTECTED], and I will send > them to you. > > PS: > since I also have some scripts to test, and they > resemble yours, can you tell me whether they are > correct? > > First I wish to re-mark packets based on u32 ip src > match, with: > > tc qdisc add dev eth0 handle 1:0 root dsmark indices 4 > > tc class change dev eth0 classid 1:1 dsmark mask 0x3 > value 0xb8 > > ..some filters under 1:0 to send packets to 1:1 to > remark them... > > and then comes a htb at 2:0 having 1:0 (the dsmark > qdisc) as parent with few branches. > > Can I expect that htb qdisc at 2:0 will already have > the effects of dsmark at 1:0 (remarked packets)? > > Thanks, > > Ferenc > > __ > Do You Yahoo!? > Tired of spam? Yahoo! Mail has the best spam protection around > http://mail.yahoo.com > ___ > LARTC mailing list > LARTC@mailman.ds9a.nl > http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc > ___ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
[LARTC] strange iptables mangle problem
Hi all, I manage network with two connections with l00MbitIn the past when network wasn't so load everything was OK, nowin pick hours load over border server from 1.0 to 1.5 / it isn't so big /and for me is very strange why I have increasing of ping timeoutfrom 0.5- 5ms in normal hour to 50-100 ms in pick hours.. server is with good hardware AMD 64 Dualcore 3800+ Intel Gigabit Ethernet 1 GB RAM Debian sarge 2.6.16 #2 SMP kernel I use about 240 mangle rules with iptables to mark download traffic and tolimit it but when I try to load more rules server increase load and begin to droppackages :( my question is why when I try to load new 200 mangle rules / only mangle rules / server increase load average and ping timeout increase to 50-100 ms …and second is what is better solution for networks with more then 100Mbit traffic .. to use iptables mangle rules + u32 or to use more u32 filters and less mangle rules ? Actually I don't have experience with so big traffic and I need any advice is welcome. Best RegardsEmil ___ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
Re: [LARTC] Re: EF & AF filters with HTB
Hello Ferenc, The big picture is to put voice in EF class, have AF classes & BE class. Asterisk can mark packets, but I'm marking with 'iptables' & classifying (filtering) as per the requested help. We want HTB for EF class to guarantee the bandwidth & to be able to borrow/lend between classes should there be no voice or data on the link. We are looking to roll this out at many sites & to partake in DS domains, hence the need to fully understand DiffServ. . My biggest issue was understanding how to filter packets to their correct classes, then I found this: http://kabru.eecs.umich.edu/qos_network/diffserv/DiffServ_prototype/qdisc and explained very nicely: http://kabru.eecs.umich.edu/qos_network/diffserv/DiffServ_prototype/setting_qdisc.ps What you have shown me is good, it's simpler & will work just as well. Look at the above link for the script, I would like to run something like that but with HTB & PFIFO for the EF class. Their explanation helps, I'm busy trying to put something together now. I haven't tried to remark packets yet so I'm unsure as to what the exact procedure is. What frustrates me is that there is no debug or logging to test scenarios. Has anyone done voice HTB with EF, AF & BE classes? On 10/04/06, pfer <[EMAIL PROTECTED]> wrote: > Hi Robert! > > If I get you right, you wish to use HTB at a node > for doing some QoS for packets based the DSCP they > already have, and NOT marking them to have that AF, > etc. > > ( > have you checked > http://www.opalsoft.net/qos/DS-310.htm? > he has some nice scripts > ) > > Why not simply use an u32 match on the TOS field for > filtering under your main htb? Why do you need DSMARK? > (haven't tried, but should work fine) > > For ex. having htb main shaper at 1:0 gives > > tc filter add dev eth0 parent 1:0 protocol ip prio 2 > u32 match ip tos 0xb8 0xff flowid 1:10 > (the EF htb branch is at 1:10) > > tc filter add dev eth0 parent 1:0 protocol ip prio 3 > u32 match ip tos 0x28 0xff flowid 1:20 > (the AF11 htb branch is at 1:20) > > At for bulk, create the htb main with: "default 30", > and add an 1:30 class with needed rate + burst and > red/sfq/etc. > U won't need any filter for this, anything > unclassified > (anything other than EF or AF11) will get there. > > If I get you wrong, and you wish to re-mark those > packets, I have some scripts for that, too. > > Contact me at [EMAIL PROTECTED], and I will send > them to you. > > PS: > since I also have some scripts to test, and they > resemble yours, can you tell me whether they are > correct? > > First I wish to re-mark packets based on u32 ip src > match, with: > > tc qdisc add dev eth0 handle 1:0 root dsmark indices 4 > > tc class change dev eth0 classid 1:1 dsmark mask 0x3 > value 0xb8 > > ..some filters under 1:0 to send packets to 1:1 to > remark them... > > and then comes a htb at 2:0 having 1:0 (the dsmark > qdisc) as parent with few branches. > > Can I expect that htb qdisc at 2:0 will already have > the effects of dsmark at 1:0 (remarked packets)? > > Thanks, > > Ferenc > > __ > Do You Yahoo!? > Tired of spam? Yahoo! Mail has the best spam protection around > http://mail.yahoo.com > ___ > LARTC mailing list > LARTC@mailman.ds9a.nl > http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc > ___ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
