Re: Sensor slots utilization

2017-07-28 Thread Alex Guziel
I'm concerned that we would be making the logic more complex, unless the new sensor 'pokeonce' case is just a high number of retries. And the other overhead of course. Running the poke method inline wouldn't be great for perf either since it's a blocking I/O and would need to be handled async in or

Sensor slots utilization

2017-07-28 Thread Maxime Beauchemin
Thought his was interesting to bubble up to the mailing list. From: https://github.com/apache/incubator-airflow/pull/2423#issuecomment-318723842 This is about the issue around sensors utilizing a lot of worker slots. The context is a PR from @shaform introducing sensors that check once and give up