Re: [LARTC] Load balancing using connmark
Francis Brosnan Blazquez wrote: Hi, I've been implementing a load balancing solution using CONNMARK, based on solution described by Luciano Ruete at [1]. Gracias por el post y por apuntar en la dirección correcta Luciano! Once implemented, I've found that due to some reason packets aren't properly marked (or improperly remarked) and sent out using the wrong interface. snip iptables -t mangle -A POSTROUTING -m mark --mark ! 0 -j ACCEPT iptables -t mangle -A POSTROUTING -o eth1 -j MARK --set-mark 0x1 iptables -t mangle -A POSTROUTING -o eth2 -j MARK --set-mark 0x2 iptables -t mangle -A POSTROUTING -j CONNMARK --save-mark This is wrong. POSTROUTING is exactly what is is _POST_ routing. By the time you do your marks and stuff the kernel has _already_ assigned a packet to an interface, and you can not alter this anymore. After a bit of testing with the second solution, it seems to behave better, doing all marking job at the PREROUTING and OUTPUT. This is flawed too. OUTPUT suffers from the very same problem as POSTROUTING - by the time the packets hit the NF stack the process has already bound itself to an interface, which you can not change anymore. Peter Disagree with Peter. The marking in postrouting table is CONNMARK. This is for marking the connection, which has already had a route decided for it, so that all packets of the connection passes through this interface. This marking is done for packets with NEW state, see the check for mark==0 in the prev. line. The restore mark in PREROUTING will restore the connmark and route the subsequent packets. This approach will work, but you need some sort of stateful-ness in netfilter. The second point in Brosnan Blazquezs mail about shorewall: They seem to be doing Policy Routing, not real load balancing. ___ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
RE: [LARTC] Load balancing using connmark
On closer look, I am wrong about shorewall. It seems to be a different approach to load balancing. They connmark the incoming packets from WAN, rather than outgoing packets. I think it should work well, but I wonder why this approach is not popular. There must be some drawback to it. I cant think of one,though. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Salim S I Sent: Thursday, May 10, 2007 2:15 PM To: lartc@mailman.ds9a.nl Subject: Re: [LARTC] Load balancing using connmark Francis Brosnan Blazquez wrote: Hi, I've been implementing a load balancing solution using CONNMARK, based on solution described by Luciano Ruete at [1]. Gracias por el post y por apuntar en la dirección correcta Luciano! Once implemented, I've found that due to some reason packets aren't properly marked (or improperly remarked) and sent out using the wrong interface. snip iptables -t mangle -A POSTROUTING -m mark --mark ! 0 -j ACCEPT iptables -t mangle -A POSTROUTING -o eth1 -j MARK --set-mark 0x1 iptables -t mangle -A POSTROUTING -o eth2 -j MARK --set-mark 0x2 iptables -t mangle -A POSTROUTING -j CONNMARK --save-mark This is wrong. POSTROUTING is exactly what is is _POST_ routing. By the time you do your marks and stuff the kernel has _already_ assigned a packet to an interface, and you can not alter this anymore. After a bit of testing with the second solution, it seems to behave better, doing all marking job at the PREROUTING and OUTPUT. This is flawed too. OUTPUT suffers from the very same problem as POSTROUTING - by the time the packets hit the NF stack the process has already bound itself to an interface, which you can not change anymore. Peter Disagree with Peter. The marking in postrouting table is CONNMARK. This is for marking the connection, which has already had a route decided for it, so that all packets of the connection passes through this interface. This marking is done for packets with NEW state, see the check for mark==0 in the prev. line. The restore mark in PREROUTING will restore the connmark and route the subsequent packets. This approach will work, but you need some sort of stateful-ness in netfilter. The second point in Brosnan Blazquezs mail about shorewall: They seem to be doing Policy Routing, not real load balancing. ___ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
RE: [LARTC] Load balancing using connmark
El jue, 10-05-2007 a las 16:01 +0800, Salim S I escribió: Hi Salim, Thanks for your reply, On closer look, I am wrong about shorewall. It seems to be a different approach to load balancing. They connmark the incoming packets from WAN, rather than outgoing packets. I think it should work well, but I wonder why this approach is not popular. There must be some drawback to it. I can’t think of one,though. I think the main advantage of shorewall solution is that it applies connmark to incoming packets from the wan as you point, leaving load balancing to outgoing connections to the main table. In any case, with this second solution I don't see wrong routed packages on wan interfaces using tcpdump, whereas with the first solution I do. More testing is required. Regarding to your previous reply, can you elaborate more on ...This approach will work, but you need some sort of stateful-ness in netfilter... Cheers! -- Francis Brosnan Blazquez [EMAIL PROTECTED] Advanced Software Production Line, S.L. ___ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
FW: [LARTC] Load balancing using connmark
-Original Message- From: Salim S I [mailto:[EMAIL PROTECTED] Sent: Thursday, May 10, 2007 5:22 PM To: 'Francis Brosnan Blazquez' Subject: RE: [LARTC] Load balancing using connmark I think the main advantage of shorewall solution is that it applies connmark to incoming packets from the wan as you point, leaving load balancing to outgoing connections to the main table Actually, the main table/multipath route only routes the first packet of a connection. The subsequent routing for that connection is done based on connmark, for outgoing packets too. Otherwise replies to packets coming from WAN1 may go through WAN2. The difference in the two solutions is only in where packets are marked and which packets are marked. Routing is the same. For a detailed discussion on the first approach, you can refer to this thread. http://mailman.ds9a.nl/pipermail/lartc/2006q2/018964.html -Original Message- From: Francis Brosnan Blazquez [mailto:[EMAIL PROTECTED] Sent: Thursday, May 10, 2007 5:07 PM To: Salim S I Cc: lartc@mailman.ds9a.nl Subject: RE: [LARTC] Load balancing using connmark El jue, 10-05-2007 a las 16:01 +0800, Salim S I escribió: Hi Salim, Thanks for your reply, On closer look, I am wrong about shorewall. It seems to be a different approach to load balancing. They connmark the incoming packets from WAN, rather than outgoing packets. I think it should work well, but I wonder why this approach is not popular. There must be some drawback to it. I cant think of one,though. I think the main advantage of shorewall solution is that it applies connmark to incoming packets from the wan as you point, leaving load balancing to outgoing connections to the main table. In any case, with this second solution I don't see wrong routed packages on wan interfaces using tcpdump, whereas with the first solution I do. More testing is required. Regarding to your previous reply, can you elaborate more on ...This approach will work, but you need some sort of stateful-ness in netfilter... Cheers! -- Francis Brosnan Blazquez [EMAIL PROTECTED] Advanced Software Production Line, S.L. ___ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
Re: [LARTC] Load balancing using connmark
hi people Francis Brosnan Blazquez wrote: I've been implementing a load balancing solution using CONNMARK, based After giving a try during several days, I've found that another firewall solution, shorewall [2], implements built-in load balacing for free by using the following set of instructions: did somebody try the shorewall solution with centos 4? with centos 4 and the first solution i always had the problem, that it routes correctly only for passing through connections (forwarded). connections starting from the machine or hoing to the machine (input/output chain) had exactly the same behaviour as you stated before. i noticed with centos 4 that packets do not pass the prerouting magle chain if going to the local host (passing the input filter chain thereafter). therefore certainly the mark will not be restored and there will be no influence on the routing decision. someone noticed similar behaviour? peter -- :: e n d i a n :: open source - open minds :: peter warasin :: http://www.endian.com :: [EMAIL PROTECTED] ___ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
Re: [LARTC] Load balancing using connmark
Salim S I wrote: Francis Brosnan Blazquez wrote: Hi, I've been implementing a load balancing solution using CONNMARK, based on solution described by Luciano Ruete at [1]. Gracias por el post y por apuntar en la dirección correcta Luciano! Once implemented, I've found that due to some reason packets aren't properly marked (or improperly remarked) and sent out using the wrong interface. snip iptables -t mangle -A POSTROUTING -m mark --mark ! 0 -j ACCEPT iptables -t mangle -A POSTROUTING -o eth1 -j MARK --set-mark 0x1 iptables -t mangle -A POSTROUTING -o eth2 -j MARK --set-mark 0x2 iptables -t mangle -A POSTROUTING -j CONNMARK --save-mark This is wrong. POSTROUTING is exactly what is is _POST_ routing. By the time you do your marks and stuff the kernel has _already_ assigned a packet to an interface, and you can not alter this anymore. After a bit of testing with the second solution, it seems to behave better, doing all marking job at the PREROUTING and OUTPUT. This is flawed too. OUTPUT suffers from the very same problem as POSTROUTING - by the time the packets hit the NF stack the process has already bound itself to an interface, which you can not change anymore. Peter Disagree with Peter. The marking in postrouting table is CONNMARK. This is for marking the connection, which has already had a route decided for it, so that all packets of the connection passes through this interface. This marking is done for packets with NEW state, see the check for mark==0 in the prev. line. The restore mark in PREROUTING will restore the connmark and route the subsequent packets. This approach will work, but you need some sort of stateful-ness in netfilter. Connmark is exactly the statefullness you are talking about. The problem is that the marks by themselves do not mean anything. You mark packets and expect iproute to classify the packet in the correct routing table etc. CONNMARK is invisible to iproute - this is why you have only --save-mark and --restore-mark, and the rest of the rules deal with real MARKs. Further you (and the OP) seem to be confused by a mix of routing tasks. In the case of _forwarded_ traffic, you need to make sure that all packets within a connection leave to WAN over the same interface, and are SNATed to the same ip, so that they will come bak the same interface. The SNATting is trivial (as it can be done in POSTROUTING only), but you need to set all marks before the routing takes place (which is anywhere _but_ POSTROUTING). You might mark the connection with the proper CONNMARK. and subsequent packets might get routed correctly, but the _first_ packet (the one that you use to set the mark) is already assigned to an interface, and there is nothing you can do about it. In the case of _local_ traffic - it becomes even trickier. The problem is that when sockets are created they already have a source IP (the kernel determines that by looking at the default routing table, your marks do not exist yet). So since you can not alter the socket binding, the only way to make it leave on a different interface is by treating it as a forwarded connection and performing NAT on it. It is arguable if NATting locally originating connections is a good idea, but it can be done in OUTPUT just like it is done for forwarder connections in PREROUTING. I hope this clarifies things a bit, feel free to point out any inconsistencies you may find. Peter ___ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
Re: [LARTC] Load balancing using connmark
Peter Rabbitson wrote: ... In the case of _local_ traffic - it becomes even trickier. The problem is that when sockets are created they already have a source IP (the kernel determines that by looking at the default routing table, your marks do not exist yet). This is misleading - it will happen only when the application does not request a specific ip/interface to bind to. Only then the kernel default table is consulted, and the best interface is determined based on the destination that is supplied on socket creation. ___ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
RE: [LARTC] Load balancing using connmark
Let me explain why the marking is done in POSTROUTING. The first packet of any connection get routed by the multipath routing entry. This happens AFTER PREROUTING, as you know. And this is what we want, letting the kernel decide based on the weights. (some people do think that we shouldn't let multipath decide routing, but thatz a different story). So where can this packet be marked? Obviously in POSTROUTING (so that local pkts also can be caught). We mark it and save it.(connmark).The mark is decoded by the chosen interface. (eg:-o WAN1 --set mark 1,-o WAN2 --set-mark 2) In PREROUTING, there is a restore-mark. You see iptables -t mangle -A PREROUTING -j CONNMARK --restore-mark. If this packet belong to a connection that has already sent a packet,this will restore the mark set in POSTROUTING. Then it will be routed by the corresponding routing table.(wan1 table lookup mark1 and wan2 table lookup mark2) If it is a new pkt, it will be routed by multipath routing statement,since no mark exists. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Peter Rabbitson Sent: Thursday, May 10, 2007 6:51 PM Cc: lartc@mailman.ds9a.nl Subject: Re: [LARTC] Load balancing using connmark Salim S I wrote: Francis Brosnan Blazquez wrote: Hi, I've been implementing a load balancing solution using CONNMARK, based on solution described by Luciano Ruete at [1]. Gracias por el post y por apuntar en la dirección correcta Luciano! Once implemented, I've found that due to some reason packets aren't properly marked (or improperly remarked) and sent out using the wrong interface. snip iptables -t mangle -A POSTROUTING -m mark --mark ! 0 -j ACCEPT iptables -t mangle -A POSTROUTING -o eth1 -j MARK --set-mark 0x1 iptables -t mangle -A POSTROUTING -o eth2 -j MARK --set-mark 0x2 iptables -t mangle -A POSTROUTING -j CONNMARK --save-mark This is wrong. POSTROUTING is exactly what is is _POST_ routing. By the time you do your marks and stuff the kernel has _already_ assigned a packet to an interface, and you can not alter this anymore. After a bit of testing with the second solution, it seems to behave better, doing all marking job at the PREROUTING and OUTPUT. This is flawed too. OUTPUT suffers from the very same problem as POSTROUTING - by the time the packets hit the NF stack the process has already bound itself to an interface, which you can not change anymore. Peter Disagree with Peter. The marking in postrouting table is CONNMARK. This is for marking the connection, which has already had a route decided for it, so that all packets of the connection passes through this interface. This marking is done for packets with NEW state, see the check for mark==0 in the prev. line. The restore mark in PREROUTING will restore the connmark and route the subsequent packets. This approach will work, but you need some sort of stateful-ness in netfilter. Connmark is exactly the statefullness you are talking about. The problem is that the marks by themselves do not mean anything. You mark packets and expect iproute to classify the packet in the correct routing table etc. CONNMARK is invisible to iproute - this is why you have only --save-mark and --restore-mark, and the rest of the rules deal with real MARKs. Further you (and the OP) seem to be confused by a mix of routing tasks. In the case of _forwarded_ traffic, you need to make sure that all packets within a connection leave to WAN over the same interface, and are SNATed to the same ip, so that they will come bak the same interface. The SNATting is trivial (as it can be done in POSTROUTING only), but you need to set all marks before the routing takes place (which is anywhere _but_ POSTROUTING). You might mark the connection with the proper CONNMARK. and subsequent packets might get routed correctly, but the _first_ packet (the one that you use to set the mark) is already assigned to an interface, and there is nothing you can do about it. In the case of _local_ traffic - it becomes even trickier. The problem is that when sockets are created they already have a source IP (the kernel determines that by looking at the default routing table, your marks do not exist yet). So since you can not alter the socket binding, the only way to make it leave on a different interface is by treating it as a forwarded connection and performing NAT on it. It is arguable if NATting locally originating connections is a good idea, but it can be done in OUTPUT just like it is done for forwarder connections in PREROUTING. I hope this clarifies things a bit, feel free to point out any inconsistencies you may find. Peter ___ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc ___ LARTC mailing list LARTC@mailman.ds9a.nl
Re: [LARTC] Load balancing using connmark
Is there a good [single?] document explaining all of this and more? What the kernel does in POST vs PRE with respect to iproute2 and netfilter with CONNMARK and etc? Thank you, David Salim S I wrote: Let me explain why the marking is done in POSTROUTING. [...] ___ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
[LARTC] PRIO and TBF is much better than HTB??
Hello mailing list,
i stand bevor a mystery and cannot explain it J. I want to do shaping and
prioritization and I have done these following configurations and
simulations. I can´t explain, that the combination of PRIO and TBF is much
better than the HTB (with the prio parameter) alone or in combination with
the SFQ.
Here are my example configurations: 2 Traffic Classes http (80 = 0x50) and
ssh (22 = 0x16), and in my example, I want to prioritize the http-Traffic:
HTB: the results of the simulation ist here:
HTB cumulative: http://simo.mix4web.de/up/htb_cumul.jpg
HTB delay: http://simo.mix4web.de/up/htb_delay.jpg
HTB with prio parameter cumulative:
http://simo.mix4web.de/up/htb_cumul_prio_paramter.jpg
HTB with prio parameter delay:
http://simo.mix4web.de/up/htb_delay_prio_parameter.jpg
#define UPLOAD 1000kbps
dev eth0 1000 {
egress {
class ( $high ) if tcp_dport == 80;
class($low) if tcp_dport == 22;
htb () {
class ( rate UPLOAD, ceil UPLOAD) {
/* with the prio parameter : $high = class ( rate 700kbps, ceil UPLOAD,
prio 0); */
$high = class ( rate 700kbps, ceil UPLOAD);
/* with the prio parameter : $low = class ( rate 300kbps,
ceil UPLOAD, prio 0); */
$low = class ( rate 300kbps, ceil UPLOAD, prio 1);
}
}
}
}
/* 1Mbit 0.0008 = 100*8/10^6 */
every 0.0008s send TCP_PCK($tcp_dport=22) 0 x 60
/* 800kbit/s */
every 0.001s send TCP_PCK($tcp_dport=80) 0 x 60
time 2s
PRIO and TBF:
PRIO and TBF cumulative: http://simo.mix4web.de/up/prio_tbf_cumul.jpg
PRIO and TBF delay: http://simo.mix4web.de/up/prio_tbf_delay.jpg
#define UPLOAD 1000kbps
dev eth0 1000 {
egress {
class ( $high ) if tcp_dport == 80;
class($low) if tcp_dport == 22;
prio{
$high = class{ tbf (rate 700kbps, burst 1510B, mtu 1510B,
limit 3000B); }
$low = class{ tbf (rate 300kbps, burst 1510B, mtu 1510B, limit
3000B); }
}
}
}
/* 1Mbit 0.0008 = 100*8/10^6 */
every 0.0008s send TCP_PCK($tcp_dport=22) 0 x 60
/* 800kbit/s */
every 0.001s send TCP_PCK($tcp_dport=80) 0 x 60
time 2s
the delay by the combination of PRIO and TBF is much better than by the HTB.
(is it possible that pakets maybe dropped by the combination of PRIO and
TBF, that´s why the latency is so good???)
Have you an idea???
thanks
simo
-
In a world without walls who needs gates and windows?
___
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
Re: [LARTC] DGD patch not detecting dead gateway
I have a doubt. If you use such a script monitoring the link status with ping and then reconfiguring, why do you need the DGD patch? You need to do some reconfiguration (change multipath to a single default route) anyway if you use the script, right? Also, the DGD patch uses src to lookup the routing table entry, but if you have a dynamic IP for the WAN interface (PPPoE, DHCP etc), this approach is bound to fail, right? ___ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
