[web2py] Re: another scheduler Q
what db are you using ? On Friday, April 5, 2013 6:24:13 AM UTC+2, Yoel Benitez Fonseca wrote: i have this setup: 2 severs with the same web2py app installed, each of then run a scheluder worker the problem is that only one of then get jobs, and in db.scheluder_worker the one with do all the work is the only with true in is_ticker field, i just the 2 of em cooperate and pickup the available jobs. P.D: got them in diferent's groups and randomly assing task to the groups. -- Yoel Benítez Fonseca Tel: 573400 -- --- You received this message because you are subscribed to the Google Groups web2py-users group. To unsubscribe from this group and stop receiving emails from it, send an email to web2py+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: [web2py] Re: another scheduler Q
El 5.4.2013 6:47 am, Niphlod escribió: what db are you using ? postgres, is the same database shared between servers On Friday, April 5, 2013 6:24:13 AM UTC+2, Yoel Benitez Fonseca wrote: i have this setup: 2 severs with the same web2py app installed, each of then run a scheluder worker the problem is that only one of then get jobs, and in db.scheluder_worker the one with do all the work is the only with true in is_ticker field, i just the 2 of em cooperate and pickup the available jobs. P.D: got them in diferent's groups and randomly assing task to the groups. -- Yoel Benítez Fonseca Tel: 573400 -- Yoel Benítez Fonseca Tel: 573400 -- --- You received this message because you are subscribed to the Google Groups web2py-users group. To unsubscribe from this group and stop receiving emails from it, send an email to web2py+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: [web2py] Re: another scheduler Q
this is the second time the issue is posted and yet I can't find a way to reproduce it It's true that I'm running two scheduler processes on the same machine vs one scheduler on on machine and the other on another separate one, but as far as the assign algorithm is concerned, as long as there are two scheduler_workers rows, nothing should change. Just as a proof, can you confirm that when you start two schedulers on the same server you observe the same behaviour ? On Friday, April 5, 2013 8:47:58 PM UTC+2, Yoel Benitez Fonseca wrote: El 5.4.2013 6:47 am, Niphlod escribió: what db are you using ? postgres, is the same database shared between servers On Friday, April 5, 2013 6:24:13 AM UTC+2, Yoel Benitez Fonseca wrote: i have this setup: 2 severs with the same web2py app installed, each of then run a scheluder worker the problem is that only one of then get jobs, and in db.scheluder_worker the one with do all the work is the only with true in is_ticker field, i just the 2 of em cooperate and pickup the available jobs. P.D: got them in diferent's groups and randomly assing task to the groups. -- Yoel Benítez Fonseca Tel: 573400 -- Yoel Benítez Fonseca Tel: 573400 -- --- You received this message because you are subscribed to the Google Groups web2py-users group. To unsubscribe from this group and stop receiving emails from it, send an email to web2py+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: [web2py] Re: another scheduler Q
El 5.4.2013 2:57 pm, Niphlod escribió: this is the second time the issue is posted and yet I can't find a way to reproduce it It's true that I'm running two scheduler processes on the same machine vs one scheduler on on machine and the other on another separate one, but as far as the assign algorithm is concerned, as long as there are two scheduler_workers rows, nothing should change. Just as a proof, can you confirm that when you start two schedulers on the same server you observe the same behaviour ? somehow I solved it: On each server the worker is started with a different group name and each task is assigned to a group name randomly chosen from those reported in the database: with n server start the workers like this: server1: python /home/www-data/web2py/web2py.py -K myapp # group name main server2: python /home/www-data/web2py/web2py.py -K myapp:srv2 server3: python /home/www-data/web2py/web2py.py -K myapp:srv3 ... serverN: python /home/www-data/web2py/web2py.py -K myapp:srvN for a group name at random from the available: def get_groups_names(): rows = db(db.scheduler_worker.id 0).select(db.scheduler_worker.group_names).as_list() mylist = [] for row in rows: mylist.extend(row['group_names']) tmp = set(mylist) #make it a set return [item for item in tmp] and to assign a task to a group: ... scheduler.queue_task(some_task, ..., group_name=random.choice(get_groups_names()) ) ... After doing that, the WORKERS started working without problems and tasks are not waiting if there are WORKERS idle. On Friday, April 5, 2013 8:47:58 PM UTC+2, Yoel Benitez Fonseca wrote: El 5.4.2013 6:47 am, Niphlod escribió: what db are you using ? postgres, is the same database shared between servers On Friday, April 5, 2013 6:24:13 AM UTC+2, Yoel Benitez Fonseca wrote: i have this setup: 2 severs with the same web2py app installed, each of then run a scheluder worker the problem is that only one of then get jobs, and in db.scheluder_worker the one with do all the work is the only with true in is_ticker field, i just the 2 of em cooperate and pickup the available jobs. P.D: got them in diferent's groups and randomly assing task to the groups. -- Yoel Benítez Fonseca Tel: 573400 -- Yoel Benítez Fonseca Tel: 573400 -- Yoel Benítez Fonseca Tel: 573400 -- --- You received this message because you are subscribed to the Google Groups web2py-users group. To unsubscribe from this group and stop receiving emails from it, send an email to web2py+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: [web2py] Re: another scheduler Q
I get that if you manually force them to pick tasks from different groups they are processed, but different schedulers with the same group name should receive the same amount of tasks anyway. On Friday, April 5, 2013 9:17:23 PM UTC+2, Yoel Benitez Fonseca wrote: El 5.4.2013 2:57 pm, Niphlod escribió: this is the second time the issue is posted and yet I can't find a way to reproduce it It's true that I'm running two scheduler processes on the same machine vs one scheduler on on machine and the other on another separate one, but as far as the assign algorithm is concerned, as long as there are two scheduler_workers rows, nothing should change. Just as a proof, can you confirm that when you start two schedulers on the same server you observe the same behaviour ? somehow I solved it: On each server the worker is started with a different group name and each task is assigned to a group name randomly chosen from those reported in the database: with n server start the workers like this: server1: python /home/www-data/web2py/web2py.py -K myapp # group name main server2: python /home/www-data/web2py/web2py.py -K myapp:srv2 server3: python /home/www-data/web2py/web2py.py -K myapp:srv3 ... serverN: python /home/www-data/web2py/web2py.py -K myapp:srvN for a group name at random from the available: def get_groups_names(): rows = db(db.scheduler_worker.id 0).select(db.scheduler_worker.group_names).as_list() mylist = [] for row in rows: mylist.extend(row['group_names']) tmp = set(mylist) #make it a set return [item for item in tmp] and to assign a task to a group: ... scheduler.queue_task(some_task, ..., group_name=random.choice(get_groups_names()) ) ... After doing that, the WORKERS started working without problems and tasks are not waiting if there are WORKERS idle. On Friday, April 5, 2013 8:47:58 PM UTC+2, Yoel Benitez Fonseca wrote: El 5.4.2013 6:47 am, Niphlod escribió: what db are you using ? postgres, is the same database shared between servers On Friday, April 5, 2013 6:24:13 AM UTC+2, Yoel Benitez Fonseca wrote: i have this setup: 2 severs with the same web2py app installed, each of then run a scheluder worker the problem is that only one of then get jobs, and in db.scheluder_worker the one with do all the work is the only with true in is_ticker field, i just the 2 of em cooperate and pickup the available jobs. P.D: got them in diferent's groups and randomly assing task to the groups. -- Yoel Benítez Fonseca Tel: 573400 -- Yoel Benítez Fonseca Tel: 573400 -- Yoel Benítez Fonseca Tel: 573400 -- --- You received this message because you are subscribed to the Google Groups web2py-users group. To unsubscribe from this group and stop receiving emails from it, send an email to web2py+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: [web2py] Re: another scheduler Q
more on the matter this is the result of 600 queued tasks all in the main group, with two different schedulers started with -K appname, with postgresql select assigned_worker_name, status, count(*) as howmany from scheduler_task group by assigned_worker_name, status monster#4772;COMPLETED;285 monster#4751;COMPLETED;315 allright, they are not 300 and 300, but the more tasks they process the more the percentage gap will reduce, eventually being a close 50%-50% On Friday, April 5, 2013 9:58:50 PM UTC+2, Niphlod wrote: I get that if you manually force them to pick tasks from different groups they are processed, but different schedulers with the same group name should receive the same amount of tasks anyway. On Friday, April 5, 2013 9:17:23 PM UTC+2, Yoel Benitez Fonseca wrote: El 5.4.2013 2:57 pm, Niphlod escribió: this is the second time the issue is posted and yet I can't find a way to reproduce it It's true that I'm running two scheduler processes on the same machine vs one scheduler on on machine and the other on another separate one, but as far as the assign algorithm is concerned, as long as there are two scheduler_workers rows, nothing should change. Just as a proof, can you confirm that when you start two schedulers on the same server you observe the same behaviour ? somehow I solved it: On each server the worker is started with a different group name and each task is assigned to a group name randomly chosen from those reported in the database: with n server start the workers like this: server1: python /home/www-data/web2py/web2py.py -K myapp # group name main server2: python /home/www-data/web2py/web2py.py -K myapp:srv2 server3: python /home/www-data/web2py/web2py.py -K myapp:srv3 ... serverN: python /home/www-data/web2py/web2py.py -K myapp:srvN for a group name at random from the available: def get_groups_names(): rows = db(db.scheduler_worker.id 0).select(db.scheduler_worker.group_names).as_list() mylist = [] for row in rows: mylist.extend(row['group_names']) tmp = set(mylist) #make it a set return [item for item in tmp] and to assign a task to a group: ... scheduler.queue_task(some_task, ..., group_name=random.choice(get_groups_names()) ) ... After doing that, the WORKERS started working without problems and tasks are not waiting if there are WORKERS idle. On Friday, April 5, 2013 8:47:58 PM UTC+2, Yoel Benitez Fonseca wrote: El 5.4.2013 6:47 am, Niphlod escribió: what db are you using ? postgres, is the same database shared between servers On Friday, April 5, 2013 6:24:13 AM UTC+2, Yoel Benitez Fonseca wrote: i have this setup: 2 severs with the same web2py app installed, each of then run a scheluder worker the problem is that only one of then get jobs, and in db.scheluder_worker the one with do all the work is the only with true in is_ticker field, i just the 2 of em cooperate and pickup the available jobs. P.D: got them in diferent's groups and randomly assing task to the groups. -- Yoel Benítez Fonseca Tel: 573400 -- Yoel Benítez Fonseca Tel: 573400 -- Yoel Benítez Fonseca Tel: 573400 -- --- You received this message because you are subscribed to the Google Groups web2py-users group. To unsubscribe from this group and stop receiving emails from it, send an email to web2py+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: [web2py] Re: another scheduler Q
El vie, 05-04-2013 a las 13:17 -0700, Niphlod escribió: more on the matter this is the result of 600 queued tasks all in the main group, with two different schedulers started with -K appname, with postgresql select assigned_worker_name, status, count(*) as howmany from scheduler_task group by assigned_worker_name, status monster#4772;COMPLETED;285 monster#4751;COMPLETED;315 allright, they are not 300 and 300, but the more tasks they process the more the percentage gap will reduce, eventually being a close 50%-50% maybe it was just that I went crazy when they do not pick up the first 10 tasks fairly. :-( On Friday, April 5, 2013 9:58:50 PM UTC+2, Niphlod wrote: I get that if you manually force them to pick tasks from different groups they are processed, but different schedulers with the same group name should receive the same amount of tasks anyway. On Friday, April 5, 2013 9:17:23 PM UTC+2, Yoel Benitez Fonseca wrote: El 5.4.2013 2:57 pm, Niphlod escribió: this is the second time the issue is posted and yet I can't find a way to reproduce it It's true that I'm running two scheduler processes on the same machine vs one scheduler on on machine and the other on another separate one, but as far as the assign algorithm is concerned, as long as there are two scheduler_workers rows, nothing should change. Just as a proof, can you confirm that when you start two schedulers on the same server you observe the same behaviour ? somehow I solved it: On each server the worker is started with a different group name and each task is assigned to a group name randomly chosen from those reported in the database: with n server start the workers like this: server1: python /home/www-data/web2py/web2py.py -K myapp # group name main server2: python /home/www-data/web2py/web2py.py -K myapp:srv2 server3: python /home/www-data/web2py/web2py.py -K myapp:srv3 ... serverN: python /home/www-data/web2py/web2py.py -K myapp:srvN for a group name at random from the available: def get_groups_names(): rows = db(db.scheduler_worker.id 0).select(db.scheduler_worker.group_names).as_list() mylist = [] for row in rows: mylist.extend(row['group_names']) tmp = set(mylist) #make it a set return [item for item in tmp] and to assign a task to a group: ... scheduler.queue_task(some_task, ..., group_name=random.choice(get_groups_names()) ) ... After doing that, the WORKERS started working without problems and tasks are not waiting if there are WORKERS idle. On Friday, April 5, 2013 8:47:58 PM UTC+2, Yoel Benitez Fonseca wrote: El 5.4.2013 6:47 am, Niphlod escribió: what db are you using ? postgres, is the same database shared between servers On Friday, April 5, 2013 6:24:13 AM UTC+2, Yoel Benitez Fonseca wrote: i have this setup: 2 severs with the same web2py app installed, each of then run a scheluder worker the problem is that only one of then get jobs, and in db.scheluder_worker the one with do all the work is the only with true in is_ticker field, i just the 2 of em cooperate and pickup the available jobs. P.D: got them in diferent's groups and randomly assing task to the groups.
