Re: [AIP-35] Add Signal Based Scheduling To Airflow

2020-06-23 Thread Becket Qin
ent a prototype and test it with our scenarios. Hopefully the implementation is mostly incremental and won't lead to patches hard to merge. But we would like to avoid moving into a direction different from where the community is heading to. So some advice would be appreciated. Thanks, Jiangjie (B

Re: [AIP-35] Add Signal Based Scheduling To Airflow

2020-06-19 Thread Becket Qin
vel, AirFlow has some predefined events -> action mappings for the operators. e.g. in case of task dependency based scheduling, on_event(task_finished) of an operator may be overridden to START, because the action is already defined when users construct the DAG with the API. - The RE

Re: [AIP-35] Add Signal Based Scheduling To Airflow

2020-06-17 Thread Becket Qin
run > three components: webserver, scheduler and signal service? Not sure if there is a problem, but I imagine for a local deployment, a signal service could be a part of the webserver, just with one more REST endpoint. Thanks again for all the great feedback and looking forward to your sugg