[web2py] Re: tasks hierarchy
you should use "Jobs" for that. any task queued that needs another task to be executed before it should be linked with thejob.add_deps(dependant, parent) in your case: - cat3 is dependant from cat2 - cat2 is dependant from cat1 read more using the https://github.com/niphlod/w2p_scheduler_tests test app. On Sunday, October 23, 2016 at 11:44:12 AM UTC+2, Pierre wrote: > > Hi, > > Suppose a batch of tasks of 3 categories such that *cat1 > cat2 > cat3* > and a group of sheduler's processes to perform these as soon as they are > queued. > > How do i instruct workers to pick cat1 tasks first and then cat2 tasks > (only if no cat1 task is present in the queue) and finally cat3 tasks (if > no cat1 & cat2 tasks are queued) ? > -- Resources: - http://web2py.com - http://web2py.com/book (Documentation) - http://github.com/web2py/web2py (Source code) - https://code.google.com/p/web2py/issues/list (Report Issues) --- You received this message because you are subscribed to the Google Groups "web2py-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to web2py+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[web2py] Re: tasks hierarchy
yes i am aware of the scheduler new deps feature this would require to know every single task id to build the parent children relation I don't know what's in the queue, tasks are queued in arbitrary order and the queue evolves constantly so i can't scan single task ids alltogether the scheduler does a pretty good job I encounter this only once : *2016-10-25 14:55:28,137 - web2py.scheduler.hp533#4326 - ERROR - Error coordinating TICKER * is this some kind of fatal error ? -- Resources: - http://web2py.com - http://web2py.com/book (Documentation) - http://github.com/web2py/web2py (Source code) - https://code.google.com/p/web2py/issues/list (Report Issues) --- You received this message because you are subscribed to the Google Groups "web2py-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to web2py+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[web2py] Re: tasks hierarchy
BTW: "error coordinating ticker" happening seldomly is not a big issue: for that round no workers were eligible to be tickers (i.e. dispatchers) but the next round will elect one. -- Resources: - http://web2py.com - http://web2py.com/book (Documentation) - http://github.com/web2py/web2py (Source code) - https://code.google.com/p/web2py/issues/list (Report Issues) --- You received this message because you are subscribed to the Google Groups "web2py-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to web2py+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[web2py] Re: tasks hierarchy
hum...how can cat3 be queued if cat3 depends on cat2 being executed ? Or what you wanted to know is how to give a higher priority to cat1 tasks instead of cat2 tasks (i.e. they are not related, it's just that you prefer cat2 remaining in the queue a bit longer if there are cat1 tasks)? In that case, you should organize tasks into groups (i.e. cat1 with "high_prio", cat2 with "normal_prio", cat3 with "low_prio"), and then start the workers with group_names. Although the scenario for such things requires a bit of workers (i.e. 10-15) and to start 7 workers with high_prio, 2 with "normal_prio" and 1 with "low_prio" i.e. -K appname:high_prio,appname:high_prio,appname:high_prio,appname:high_prio,appname:high_prio,appname:high_prio,appname:high_prio,appname:normal_prio,appname:normal_prio,appname:low_prio and that you can make the scheduler know that it should assign more tasks of high_prio than normal_prio, but with a lower number of workers e.g. -K appname:high_prio:high_prio:high_prio:normal_prio:normal_prio:low_prio,appname:high_prio you'd get what I call a "proportional dispatch"meaning it would obviously "slim" the task queue of "high_prio" before "low_prio" just giving more chances to high_prio tasks to be executed vs low_prio. All this works best if you are in the situation of having a long queue of tasks ready to be processed ( start_time <= request.now). If you instead want to prioritize "scientificly" a little known fact (but it's shown in the scheduler test app and there's a regression test for it) is that tasks are picked up ordered by next_start_time. So, again, if you are queuing tasks that should be processed ASAP, just queue high_prio (your "cat1") with next_run_time = request.now - datetime.timedelta(days=90), normal_prio with next_run_time=request.now - datetime.timedelta(days=60) and low_prio with request.now. On Wednesday, October 26, 2016 at 4:01:07 PM UTC+2, Pierre wrote: > > yes i am aware of the scheduler new deps feature > this would require to know every single task id to build the parent > children relation > I don't know what's in the queue, tasks are queued in arbitrary order and > the queue evolves constantly so i can't scan single task ids > > alltogether the scheduler does a pretty good job > > I encounter this only once : > > *2016-10-25 14:55:28,137 - web2py.scheduler.hp533#4326 - ERROR - Error > coordinating TICKER * > > is this some kind of fatal error ? > > > -- Resources: - http://web2py.com - http://web2py.com/book (Documentation) - http://github.com/web2py/web2py (Source code) - https://code.google.com/p/web2py/issues/list (Report Issues) --- You received this message because you are subscribed to the Google Groups "web2py-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to web2py+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.