more on the matter.... this is the result of 600 queued tasks all in the main group, with two different schedulers started with -K appname, with postgresql
select assigned_worker_name, status, count(*) as howmany from scheduler_task group by assigned_worker_name, status "monster#4772";"COMPLETED";285 "monster#4751";"COMPLETED";315 allright, they are not 300 and 300, but the more tasks they process the more the "percentage gap" will reduce, eventually being a close 50%-50% On Friday, April 5, 2013 9:58:50 PM UTC+2, Niphlod wrote: > > I get that if you "manually" force them to pick tasks from different > groups they are processed, but different schedulers with the same group > name should receive the same amount of tasks anyway. > > On Friday, April 5, 2013 9:17:23 PM UTC+2, Yoel Benitez Fonseca wrote: >> >> El 5.4.2013 2:57 pm, Niphlod escribió: >> > this is the second time the issue is posted and yet I can't find a way >> > to >> > reproduce it.... It's true that I'm running two scheduler processes on >> > the >> > same machine vs one scheduler on on machine and the other on another >> > separate one, but as far as the "assign" algorithm is concerned, as >> > long as >> > there are two scheduler_workers rows, nothing should change. >> > Just as a proof, can you confirm that when you start two schedulers on >> > the >> > same server you observe the same behaviour ? >> >> somehow I solved it: >> >> On each server the worker is started with a different group name and >> each task is assigned to a group name randomly chosen from those >> reported in the database: >> >> with n server start the workers like this: >> >> server1: python /home/www-data/web2py/web2py.py -K myapp # group name >> main >> server2: python /home/www-data/web2py/web2py.py -K myapp:srv2 >> server3: python /home/www-data/web2py/web2py.py -K myapp:srv3 >> ... >> serverN: python /home/www-data/web2py/web2py.py -K myapp:srvN >> >> >> for a group name at random from the available: >> >> def get_groups_names(): >> rows = db(db.scheduler_worker.id > >> 0).select(db.scheduler_worker.group_names).as_list() >> mylist = [] >> for row in rows: mylist.extend(row['group_names']) >> tmp = set(mylist) #make it a set >> >> return [item for item in tmp] >> >> and to assign a task to a group: >> >> ... >> scheduler.queue_task(some_task, >> ..., >> group_name=random.choice(get_groups_names()) >> ) >> ... >> >> After doing that, the WORKERS started working without problems and >> tasks are not waiting if there are WORKERS idle. >> >> > On Friday, April 5, 2013 8:47:58 PM UTC+2, Yoel Benitez Fonseca wrote: >> >> >> >> El 5.4.2013 6:47 am, Niphlod escribió: >> >>> what db are you using ? >> >> >> >> postgres, is the same database shared between servers >> >> >> >>> >> >>> On Friday, April 5, 2013 6:24:13 AM UTC+2, Yoel Benitez Fonseca >> >>> wrote: >> >>>> >> >>>> i have this setup: >> >>>> >> >>>> 2 severs with the same web2py & app installed, each of then run a >> >>>> scheluder worker >> >>>> >> >>>> the problem is that only one of then get jobs, and in >> >>>> db.scheluder_worker the one with do all the work is the only with >> >>>> true >> >>>> in is_ticker field, i just the 2 of em cooperate and pickup the >> >>>> available jobs. >> >>>> >> >>>> P.D: got them in diferent's groups and randomly assing task to the >> >>>> groups. >> >>>> >> >>>> -- >> >>>> Yoel Benítez Fonseca >> >>>> Tel: 573400 >> >>>> >> >> >> >> -- >> >> Yoel Benítez Fonseca >> >> Tel: 573400 >> >> >> >> -- >> Yoel Benítez Fonseca >> Tel: 573400 >> > -- --- You received this message because you are subscribed to the Google Groups "web2py-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to web2py+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
