On 08/22/2018 06:56 PM, Tao Feng wrote: > FYI, there is an existing pr and proposal for improving sensor efficiency( > https://issues.apache.org/jira/browse/AIRFLOW-2747 and > https://github.com/apache/incubator-airflow/pull/3596/files) by the > community.
And I hope I'll find some time next week to continue working on it. More feedback and help is welcomed :) > For your idea, I am not sure if it is a good idea to add this dag dependent > functionality to the existing scheduler given its logic is already quite > complicated. > > On Wed, Aug 22, 2018 at 6:42 AM, Emmanuel Brard < > emmanuel.br...@getyourguide.com> wrote: > >> Hi everyone, >> >> I've recently looked at the implementation of the ExternalTaskSensor sensor >> and I was wondering if it would be a good idea to actually implement this >> check (these checks) at the scheduler level. Basically the >> ExternalTaskSensor runs a query against the backend database at regular >> intervals to know if the rule(s) have passed. This is also what the >> scheduler does to some extent checking against all task instances deps. So >> I though of something like adding the same (or similar) set of parameters >> of the ExternalTaskSensor directly to the operators directly and have the >> scheduler check for this. >> >> The idea around this is that the ExternalTaskSensors use resources to not >> do much actually. >> >> I have read somewhere that having the scheduler executing these kind of >> task could be a potential feature, it would already be a improvement but I >> suppose we would need the scheduler to "thread out" to do so, and this will >> just move the resources waste from the "executors" to the scheduler. >> >> What do you think ? >> >> Cheers, >> Emmanuel >> >> -- >> >> >> >> >> >> >> >> >> GetYourGuide AG >> >> Stampfenbachstrasse 48 >> >> 8006 Zürich >> >> Switzerland >> >> >> >> <https://www.facebook.com/GetYourGuide> >> <https://twitter.com/GetYourGuide> >> <https://www.instagram.com/getyourguide/> >> <https://www.linkedin.com/company/getyourguide-ag> >> <http://www.getyourguide.com> >> >> >> >> >> >> >> >> >
