Good one, let me check it out in our lab. Thanks.
On Thu, Aug 14, 2014 at 12:24 PM, Stefan <[email protected]> wrote: > Hi all, > > I have recently been running into the problem that some of my l4xnat > farms become unreachable every few days until I restart the problem > farm(s). In the farmguardian logs I found messages like: > > iptables: Resource temporarily unavailable. > iptables: Index of deletion too big. > > The above messages appeared at the time of backend status changes when > farmguardian restarts the farm. > > Sometimes the problem farms were then actually shown as down in the > web interface. > > In zenloadbalancer.log I found messages like: > some date - - - - running 'Start write false' > ... > some date - - - - running /sbin/iptables .... > some date - - - - last command failed! > ... > > The more farms and thus farmguardian instances are running the more > likely and frequent the problem occurs. And even more likely if > multiple farms target the same backend server. > > So what I figured is that if the backends of multiple farms change > status at the same time multiple farmguardian instances will restart > multiple farms i.e. running many iptables commands at the same time > which will cause some of those commands to fail. > > As a quick workaround I modified the farmguardian script to prevent > multiple farmguardian instances restarting farms at the same time > (using file locking). The problem seems to have disappeared now. > > ... > use Fcntl qw(:flock); > ... > if ($type eq "l4xnat"){ > if (open(FGA,">/var/run/farmguardian_lock")){ > my $count = 1; > while (!flock(FGA,LOCK_EX|LOCK_NB) && ($count < 11) ) { > print("Restart farm $farmname blocked, waiting... $count\n"); > sleep(1); > $count++; > } > } > &_runFarmStop($farmname,"false"); > &setFarmBackendStatus($farmname,$j,"up"); > &_runFarmStart($farmname,"false"); > close(FGA); > } > ... > if ($type eq "l4xnat"){ > if (open(FGA,">/var/run/farmguardian_lock")){ > my $count = 1; > while (!flock(FGA,LOCK_EX|LOCK_NB) && ($count < 11) ) { > print("Restart farm $farmname blocked, waiting... $count\n"); > sleep(1); > $count++; > } > } > &_runFarmStop($farmname,"false"); > &setFarmBackendStatus($farmname,$j,"fgDOWN"); > &_runFarmStart($farmname,"false"); > close(FGA); > } > > I guess a better place to do something like this would be in > farms_functions.cgi... Maybe the developers can have a look into this? > > Kind Regards, > Stefan > > > ------------------------------------------------------------------------------ > _______________________________________________ > Zenloadbalancer-support mailing list > [email protected] > https://lists.sourceforge.net/lists/listinfo/zenloadbalancer-support >
------------------------------------------------------------------------------
_______________________________________________ Zenloadbalancer-support mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/zenloadbalancer-support
