Hi Laura,
please find the log entries below (I changed farm names and addresses).
Kind Regards,
Stefan
Thu Jul 24 23:52:17 2014 - - - - running 'Stop write false' for
myfarm-1-443 farm l4xnat
Thu Jul 24 23:52:17 2014 - - - - deleteIptRules:: running
'/sbin/iptables -t mangle -D PREROUTING 17'
Thu Jul 24 23:52:17 2014 - - - - running 'Stop write false' for
myfarm-1-80 farm l4xnat
Thu Jul 24 23:52:17 2014 - - - - deleteIptRules:: delete netfilter
rule '17 0 0 MARK tcp -- * * 0.0.0.0/0
192.168.a.a recent: CHECK seconds: 43200 name:
_myfarm-1-443_0x208_sessions side: source multiport dports 443 /*
FARM_myfarm-1-443_0_ */ MARK set 0x208
'
Thu Jul 24 23:52:17 2014 - - - - deleteIptRules:: running
'/sbin/iptables -t mangle -D PREROUTING 16'
Thu Jul 24 23:52:17 2014 - - - - deleteIptRules:: delete netfilter
rule '16 0 0 MARK tcp -- * * 0.0.0.0/0
192.168.a.a statistic mode random probability
1.00000000000 multiport dports 443 /* FARM_myfarm-1-443_0_ */ MARK
set 0x208
'
Thu Jul 24 23:52:17 2014 - - - - deleteIptRules:: running
'/sbin/iptables -t mangle -D PREROUTING 17'
Thu Jul 24 23:52:17 2014 - - - - deleteIptRules:: running
'/sbin/iptables -t nat -D PREROUTING 8'
Thu Jul 24 23:52:17 2014 - - - - deleteIptRules:: delete netfilter
rule '17 0 0 MARK tcp -- * * 0.0.0.0/0
192.168.a.a recent: CHECK seconds: 43200 name:
_myfarm-1-80_0x20a_sessions side: source multiport dports 80 /*
FARM_myfarm-1-80_0_ */ MARK set 0x20a
'
Thu Jul 24 23:52:17 2014 - - - - deleteIptRules:: running
'/sbin/iptables -t mangle -D PREROUTING 16'
Thu Jul 24 23:52:17 2014 - - - - deleteIptRules:: delete netfilter
rule '8 0 0 DNAT tcp -- * * 0.0.0.0/0
0.0.0.0/0 mark match 0x208 recent: SET name:
_myfarm-1-443_0x208_sessions side: source /* FARM_myfarm-1-443_0_ */
to:192.168.b.a:443
'
Thu Jul 24 23:52:17 2014 - - - - deleteIptRules:: delete netfilter
rule '16 0 0 MARK tcp -- * * 0.0.0.0/0
192.168.a.a statistic mode random probability
1.00000000000 multiport dports 80 /* FARM_myfarm-1-80_0_ */ MARK set
0x20a
'
Thu Jul 24 23:52:17 2014 - - - - deleteIptRules:: running
'/sbin/iptables -t nat -D POSTROUTING 8'
Thu Jul 24 23:52:17 2014 - - - - deleteIptRules:: running
'/sbin/iptables -t nat -D PREROUTING 8'
Thu Jul 24 23:52:17 2014 - - - - deleteIptRules:: delete netfilter
rule '8 0 0 MASQUERADE tcp -- * * 0.0.0.0/0
0.0.0.0/0 mark match 0x208 /*
FARM_myfarm-1-443_0_ */
'
Thu Jul 24 23:52:17 2014 - - - - deleteIptRules:: delete netfilter
rule '8 0 0 DNAT tcp -- * * 0.0.0.0/0
0.0.0.0/0 mark match 0x20a recent: SET name:
_myfarm-1-80_0x20a_sessions side: source /* FARM_myfarm-1-80_0_ */
to:192.168.b.a:80
'
Thu Jul 24 23:52:17 2014 - - - - running 'Start write false' for
myfarm-1-443 farm l4xnat
Thu Jul 24 23:52:17 2014 - - - - deleteIptRules:: running
'/sbin/iptables -t nat -D POSTROUTING 8'
Thu Jul 24 23:52:17 2014 - - - - setting true to IP forwarding
Thu Jul 24 23:52:17 2014 - - - - deleteIptRules:: delete netfilter
rule '8 0 0 MASQUERADE tcp -- * * 0.0.0.0/0
0.0.0.0/0 mark match 0x20a /* FARM_myfarm-1-80_0_
*/
'
Thu Jul 24 23:52:17 2014 - - - - running 'Start write false' for
myfarm-1-80 farm l4xnat
Thu Jul 24 23:52:17 2014 - - - - setting true to IP forwarding
Thu Jul 24 23:52:32 2014 - - - - running 'Stop write false' for
myfarm-1-443 farm l4xnat
Thu Jul 24 23:52:32 2014 - - - - running 'Stop write false' for
myfarm-1-80 farm l4xnat
Thu Jul 24 23:52:32 2014 - - - - running 'Start write false' for
myfarm-1-443 farm l4xnat
Thu Jul 24 23:52:32 2014 - - - - running /sbin/iptables -t mangle
-A PREROUTING -m statistic --mode random --probability 1 -d
192.168.a.a -p tcp -m multiport --dports 443 -j MARK --set-mark 0x208
-m comment --comment ' FARM_myfarm-1-443_0_ '
Thu Jul 24 23:52:32 2014 - - - - running 'Start write false' for
myfarm-1-80 farm l4xnat
Thu Jul 24 23:52:32 2014 - - - - running /sbin/iptables -t mangle
-A PREROUTING -m recent --name "_myfarm-1-443_0x208_sessions" --rcheck
--seconds 43200 -d 192.168.a.a -p tcp -m multiport --dports 443 -j
MARK --set-mark 0x208 -m comment --comment ' FARM_myfarm-1-443_0_ '
Thu Jul 24 23:52:32 2014 - - - - running /sbin/iptables -t mangle
-A PREROUTING -m statistic --mode random --probability 1 -d
192.168.a.a -p tcp -m multiport --dports 80 -j MARK --set-mark 0x20a
-m comment --comment ' FARM_myfarm-1-80_0_ '
Thu Jul 24 23:52:32 2014 - - - - running /sbin/iptables -t nat -A
PREROUTING -m mark --mark 0x208 -j DNAT -p tcp --to-destination
192.168.b.a:443 -m recent --name "_myfarm-1-443_0x208_sessions" --set
-m comment --comment ' FARM_myfarm-1-443_0_ '
Thu Jul 24 23:52:32 2014 - - - - running /sbin/iptables -t mangle
-A PREROUTING -m recent --name "_myfarm-1-80_0x20a_sessions" --rcheck
--seconds 43200 -d 192.168.a.a -p tcp -m multiport --dports 80 -j MARK
--set-mark 0x20a -m comment --comment ' FARM_myfarm-1-80_0_ '
Thu Jul 24 23:52:32 2014 - - - - running /sbin/iptables -t nat -A
PREROUTING -m mark --mark 0x20a -j DNAT -p tcp --to-destination
192.168.b.a:80 -m recent --name "_myfarm-1-80_0x20a_sessions" --set -m
comment --comment ' FARM_myfarm-1-80_0_ '
Thu Jul 24 23:52:32 2014 - - - - running /sbin/iptables -t nat -A
POSTROUTING -m mark --mark 0x20a -j MASQUERADE -p tcp -m comment
--comment ' FARM_myfarm-1-80_0_ '
Thu Jul 24 23:52:32 2014 - - - - last command failed!
Thu Jul 24 23:52:32 2014 - - - - running /sbin/iptables -t nat -A
POSTROUTING -m mark --mark 0x208 -j MASQUERADE -p tcp -m comment
--comment ' FARM_myfarm-1-443_0_ '
Thu Jul 24 23:52:32 2014 - - - - setting true to IP forwarding
Thu Jul 24 23:52:32 2014 - - - - setting true to IP forwarding
On Thu, Aug 14, 2014 at 1:16 PM, Laura Garcia <[email protected]> wrote:
> Stefan, could you send the piece of zen logs where the iptables command is
> launched and failed? It seems that the command is malformed.
>
> Thanks
>
>
> On Thu, Aug 14, 2014 at 12:39 PM, Laura Garcia <[email protected]> wrote:
>>
>> Good one, let me check it out in our lab.
>>
>> Thanks.
>>
>>
>> On Thu, Aug 14, 2014 at 12:24 PM, Stefan <[email protected]> wrote:
>>>
>>> Hi all,
>>>
>>> I have recently been running into the problem that some of my l4xnat
>>> farms become unreachable every few days until I restart the problem
>>> farm(s). In the farmguardian logs I found messages like:
>>>
>>> iptables: Resource temporarily unavailable.
>>> iptables: Index of deletion too big.
>>>
>>> The above messages appeared at the time of backend status changes when
>>> farmguardian restarts the farm.
>>>
>>> Sometimes the problem farms were then actually shown as down in the
>>> web interface.
>>>
>>> In zenloadbalancer.log I found messages like:
>>> some date - - - - running 'Start write false'
>>> ...
>>> some date - - - - running /sbin/iptables ....
>>> some date - - - - last command failed!
>>> ...
>>>
>>> The more farms and thus farmguardian instances are running the more
>>> likely and frequent the problem occurs. And even more likely if
>>> multiple farms target the same backend server.
>>>
>>> So what I figured is that if the backends of multiple farms change
>>> status at the same time multiple farmguardian instances will restart
>>> multiple farms i.e. running many iptables commands at the same time
>>> which will cause some of those commands to fail.
>>>
>>> As a quick workaround I modified the farmguardian script to prevent
>>> multiple farmguardian instances restarting farms at the same time
>>> (using file locking). The problem seems to have disappeared now.
>>>
>>> ...
>>> use Fcntl qw(:flock);
>>> ...
>>> if ($type eq "l4xnat"){
>>> if (open(FGA,">/var/run/farmguardian_lock")){
>>> my $count = 1;
>>> while (!flock(FGA,LOCK_EX|LOCK_NB) && ($count < 11) ) {
>>> print("Restart farm $farmname blocked, waiting...
>>> $count\n");
>>> sleep(1);
>>> $count++;
>>> }
>>> }
>>> &_runFarmStop($farmname,"false");
>>> &setFarmBackendStatus($farmname,$j,"up");
>>> &_runFarmStart($farmname,"false");
>>> close(FGA);
>>> }
>>> ...
>>> if ($type eq "l4xnat"){
>>> if (open(FGA,">/var/run/farmguardian_lock")){
>>> my $count = 1;
>>> while (!flock(FGA,LOCK_EX|LOCK_NB) && ($count < 11) ) {
>>> print("Restart farm $farmname blocked, waiting...
>>> $count\n");
>>> sleep(1);
>>> $count++;
>>> }
>>> }
>>> &_runFarmStop($farmname,"false");
>>> &setFarmBackendStatus($farmname,$j,"fgDOWN");
>>> &_runFarmStart($farmname,"false");
>>> close(FGA);
>>> }
>>>
>>> I guess a better place to do something like this would be in
>>> farms_functions.cgi... Maybe the developers can have a look into this?
>>>
>>> Kind Regards,
>>> Stefan
>>>
>>>
>>> ------------------------------------------------------------------------------
>>> _______________________________________________
>>> Zenloadbalancer-support mailing list
>>> [email protected]
>>> https://lists.sourceforge.net/lists/listinfo/zenloadbalancer-support
>>
>>
>
>
> ------------------------------------------------------------------------------
>
> _______________________________________________
> Zenloadbalancer-support mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/zenloadbalancer-support
>
------------------------------------------------------------------------------
Slashdot TV.
Video for Nerds. Stuff that matters.
http://tv.slashdot.org/
_______________________________________________
Zenloadbalancer-support mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/zenloadbalancer-support
