Hi everyone, I've recently looked at the implementation of the ExternalTaskSensor sensor and I was wondering if it would be a good idea to actually implement this check (these checks) at the scheduler level. Basically the ExternalTaskSensor runs a query against the backend database at regular intervals to know if the rule(s) have passed. This is also what the scheduler does to some extent checking against all task instances deps. So I though of something like adding the same (or similar) set of parameters of the ExternalTaskSensor directly to the operators directly and have the scheduler check for this.
The idea around this is that the ExternalTaskSensors use resources to not do much actually. I have read somewhere that having the scheduler executing these kind of task could be a potential feature, it would already be a improvement but I suppose we would need the scheduler to "thread out" to do so, and this will just move the resources waste from the "executors" to the scheduler. What do you think ? Cheers, Emmanuel -- GetYourGuide AG Stampfenbachstrasse 48 8006 Zürich Switzerland <https://www.facebook.com/GetYourGuide> <https://twitter.com/GetYourGuide> <https://www.instagram.com/getyourguide/> <https://www.linkedin.com/company/getyourguide-ag> <http://www.getyourguide.com>
