Hi Emilio, thanks for the update. I will gladly test your code but it will take a few days until I can do so.
Kind Regards, Stefan On Wed, Aug 20, 2014 at 6:07 PM, Emilio Campos <[email protected]> wrote: > Hi Stefan, I have been playing with your proposed code and I would like to > know if you could try the next one. > > > In FarmGuardian, delete your added source code for L4 farms: > > ... > if (open(FGA,">/var/run/ > farmguardian_lock")){ > my $count = 1; > while (!flock(FGA,LOCK_EX|LOCK_NB) && ($count < 11) ) { > print("Restart farm $farmname blocked, waiting... $count\n"); > sleep(1); > $count++; > } > } > .... > > and in farns_functions.cgi , function sub > setFarmBackendStatus($fname,$index,$stat), go to the line if ($type > eq "l4xnat"){ and change as follows : > > ...... > if ($type eq "l4xnat"){ > #ecm added lock in Tie. > use Fcntl ':flock'; > # > use Tie::File; > #tie @filelines, 'Tie::File', "$configdir\/$file"; > my $f = tie @filelines, 'Tie::File', "$configdir\/$file"; > $f->flock; > my $fileid = 0; > ..... > > I added use Fcntl and $f->flock for locking directly the file in place of an > auxiliary file, could you try it and let us know if it works? > > Thanks and regards > > > > > > > > 2014-08-14 13:16 GMT+02:00 Laura Garcia <[email protected]>: >> >> Stefan, could you send the piece of zen logs where the iptables command is >> launched and failed? It seems that the command is malformed. >> >> Thanks >> >> >> On Thu, Aug 14, 2014 at 12:39 PM, Laura Garcia <[email protected]> wrote: >>> >>> Good one, let me check it out in our lab. >>> >>> Thanks. >>> >>> >>> On Thu, Aug 14, 2014 at 12:24 PM, Stefan <[email protected]> wrote: >>>> >>>> Hi all, >>>> >>>> I have recently been running into the problem that some of my l4xnat >>>> farms become unreachable every few days until I restart the problem >>>> farm(s). In the farmguardian logs I found messages like: >>>> >>>> iptables: Resource temporarily unavailable. >>>> iptables: Index of deletion too big. >>>> >>>> The above messages appeared at the time of backend status changes when >>>> farmguardian restarts the farm. >>>> >>>> Sometimes the problem farms were then actually shown as down in the >>>> web interface. >>>> >>>> In zenloadbalancer.log I found messages like: >>>> some date - - - - running 'Start write false' >>>> ... >>>> some date - - - - running /sbin/iptables .... >>>> some date - - - - last command failed! >>>> ... >>>> >>>> The more farms and thus farmguardian instances are running the more >>>> likely and frequent the problem occurs. And even more likely if >>>> multiple farms target the same backend server. >>>> >>>> So what I figured is that if the backends of multiple farms change >>>> status at the same time multiple farmguardian instances will restart >>>> multiple farms i.e. running many iptables commands at the same time >>>> which will cause some of those commands to fail. >>>> >>>> As a quick workaround I modified the farmguardian script to prevent >>>> multiple farmguardian instances restarting farms at the same time >>>> (using file locking). The problem seems to have disappeared now. >>>> >>>> ... >>>> use Fcntl qw(:flock); >>>> ... >>>> if ($type eq "l4xnat"){ >>>> if (open(FGA,">/var/run/farmguardian_lock")){ >>>> my $count = 1; >>>> while (!flock(FGA,LOCK_EX|LOCK_NB) && ($count < 11) ) { >>>> print("Restart farm $farmname blocked, waiting... >>>> $count\n"); >>>> sleep(1); >>>> $count++; >>>> } >>>> } >>>> &_runFarmStop($farmname,"false"); >>>> &setFarmBackendStatus($farmname,$j,"up"); >>>> &_runFarmStart($farmname,"false"); >>>> close(FGA); >>>> } >>>> ... >>>> if ($type eq "l4xnat"){ >>>> if (open(FGA,">/var/run/farmguardian_lock")){ >>>> my $count = 1; >>>> while (!flock(FGA,LOCK_EX|LOCK_NB) && ($count < 11) ) { >>>> print("Restart farm $farmname blocked, waiting... >>>> $count\n"); >>>> sleep(1); >>>> $count++; >>>> } >>>> } >>>> &_runFarmStop($farmname,"false"); >>>> &setFarmBackendStatus($farmname,$j,"fgDOWN"); >>>> &_runFarmStart($farmname,"false"); >>>> close(FGA); >>>> } >>>> >>>> I guess a better place to do something like this would be in >>>> farms_functions.cgi... Maybe the developers can have a look into this? >>>> >>>> Kind Regards, >>>> Stefan >>>> >>>> >>>> ------------------------------------------------------------------------------ >>>> _______________________________________________ >>>> Zenloadbalancer-support mailing list >>>> [email protected] >>>> https://lists.sourceforge.net/lists/listinfo/zenloadbalancer-support >>> >>> >> >> >> >> ------------------------------------------------------------------------------ >> >> _______________________________________________ >> Zenloadbalancer-support mailing list >> [email protected] >> https://lists.sourceforge.net/lists/listinfo/zenloadbalancer-support >> > > > > -- > Load balancer distribution - Open Source Project > http://www.zenloadbalancer.com > Distribution list (subscribe): [email protected] > > ------------------------------------------------------------------------------ > Slashdot TV. > Video for Nerds. Stuff that matters. > http://tv.slashdot.org/ > _______________________________________________ > Zenloadbalancer-support mailing list > [email protected] > https://lists.sourceforge.net/lists/listinfo/zenloadbalancer-support > ------------------------------------------------------------------------------ Slashdot TV. Video for Nerds. Stuff that matters. http://tv.slashdot.org/ _______________________________________________ Zenloadbalancer-support mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/zenloadbalancer-support
