El vie, 05-04-2013 a las 13:17 -0700, Niphlod escribió: > more on the matter.... this is the result of 600 queued tasks all in > the main group, with two different schedulers started with -K appname, > with postgresql > > select assigned_worker_name, status, count(*) as howmany > from scheduler_task > group by assigned_worker_name, status > > "monster#4772";"COMPLETED";285 > "monster#4751";"COMPLETED";315 > > allright, they are not 300 and 300, but the more tasks they process > the more the "percentage gap" will reduce, eventually being a close > 50%-50% > maybe it was just that I went crazy when they do not pick up the first 10 tasks fairly. :-(
> On Friday, April 5, 2013 9:58:50 PM UTC+2, Niphlod wrote: > I get that if you "manually" force them to pick tasks from > different groups they are processed, but different schedulers > with the same group name should receive the same amount of > tasks anyway. > > On Friday, April 5, 2013 9:17:23 PM UTC+2, Yoel Benitez > Fonseca wrote: > El 5.4.2013 2:57 pm, Niphlod escribió: > > this is the second time the issue is posted and yet > I can't find a way > > to > > reproduce it.... It's true that I'm running two > scheduler processes on > > the > > same machine vs one scheduler on on machine and the > other on another > > separate one, but as far as the "assign" algorithm > is concerned, as > > long as > > there are two scheduler_workers rows, nothing should > change. > > Just as a proof, can you confirm that when you start > two schedulers on > > the > > same server you observe the same behaviour ? > > somehow I solved it: > > On each server the worker is started with a different > group name and > each task is assigned to a group name randomly chosen > from those > reported in the database: > > with n server start the workers like this: > > server1: python /home/www-data/web2py/web2py.py -K > myapp # group name > main > server2: python /home/www-data/web2py/web2py.py -K > myapp:srv2 > server3: python /home/www-data/web2py/web2py.py -K > myapp:srv3 > ... > serverN: python /home/www-data/web2py/web2py.py -K > myapp:srvN > > > for a group name at random from the available: > > def get_groups_names(): > rows = db(db.scheduler_worker.id > > 0).select(db.scheduler_worker.group_names).as_list() > mylist = [] > for row in rows: > mylist.extend(row['group_names']) > tmp = set(mylist) #make it a set > > return [item for item in tmp] > > and to assign a task to a group: > > ... > scheduler.queue_task(some_task, > ..., > group_name=random.choice(get_groups_names()) > ) > ... > > After doing that, the WORKERS started working without > problems and > tasks are not waiting if there are WORKERS idle. > > > On Friday, April 5, 2013 8:47:58 PM UTC+2, Yoel > Benitez Fonseca wrote: > >> > >> El 5.4.2013 6:47 am, Niphlod escribió: > >>> what db are you using ? > >> > >> postgres, is the same database shared between > servers > >> > >>> > >>> On Friday, April 5, 2013 6:24:13 AM UTC+2, Yoel > Benitez Fonseca > >>> wrote: > >>>> > >>>> i have this setup: > >>>> > >>>> 2 severs with the same web2py & app installed, > each of then run a > >>>> scheluder worker > >>>> > >>>> the problem is that only one of then get jobs, > and in > >>>> db.scheluder_worker the one with do all the work > is the only with > >>>> true > >>>> in is_ticker field, i just the 2 of em cooperate > and pickup the > >>>> available jobs. > >>>> > >>>> P.D: got them in diferent's groups and randomly > assing task to the > >>>> groups. > >>>> > >>>> -- > >>>> Yoel Benítez Fonseca > >>>> Tel: 573400 > >>>> > >> > >> -- > >> Yoel Benítez Fonseca > >> Tel: 573400 > >> > > -- > Yoel Benítez Fonseca > Tel: 573400 > > -- > > --- > You received this message because you are subscribed to the Google > Groups "web2py-users" group. > To unsubscribe from this group and stop receiving emails from it, send > an email to web2py+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/groups/opt_out. > > -- --- You received this message because you are subscribed to the Google Groups "web2py-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to web2py+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
