FYI, there is an existing pr and proposal for improving sensor efficiency( https://issues.apache.org/jira/browse/AIRFLOW-2747 and https://github.com/apache/incubator-airflow/pull/3596/files) by the community.
For your idea, I am not sure if it is a good idea to add this dag dependent functionality to the existing scheduler given its logic is already quite complicated. On Wed, Aug 22, 2018 at 6:42 AM, Emmanuel Brard < emmanuel.br...@getyourguide.com> wrote: > Hi everyone, > > I've recently looked at the implementation of the ExternalTaskSensor sensor > and I was wondering if it would be a good idea to actually implement this > check (these checks) at the scheduler level. Basically the > ExternalTaskSensor runs a query against the backend database at regular > intervals to know if the rule(s) have passed. This is also what the > scheduler does to some extent checking against all task instances deps. So > I though of something like adding the same (or similar) set of parameters > of the ExternalTaskSensor directly to the operators directly and have the > scheduler check for this. > > The idea around this is that the ExternalTaskSensors use resources to not > do much actually. > > I have read somewhere that having the scheduler executing these kind of > task could be a potential feature, it would already be a improvement but I > suppose we would need the scheduler to "thread out" to do so, and this will > just move the resources waste from the "executors" to the scheduler. > > What do you think ? > > Cheers, > Emmanuel > > -- > > > > > > > > > GetYourGuide AG > > Stampfenbachstrasse 48 > > 8006 Zürich > > Switzerland > > > > <https://www.facebook.com/GetYourGuide> > <https://twitter.com/GetYourGuide> > <https://www.instagram.com/getyourguide/> > <https://www.linkedin.com/company/getyourguide-ag> > <http://www.getyourguide.com> > > > > > > > >
