Re: ExternalTaskSensor alternatives

[Stefan Seelmann]

On 08/22/2018 06:56 PM, Tao Feng wrote:
> FYI, there is an existing pr and proposal for improving sensor efficiency(
> https://issues.apache.org/jira/browse/AIRFLOW-2747 and
> https://github.com/apache/incubator-airflow/pull/3596/files) by the
> community.

And I hope I'll find some time next week to continue working on it. More
feedback and help is welcomed :)

> For your idea, I am not sure if it is a good idea to add this dag dependent
> functionality to the existing scheduler given its logic is already quite
> complicated.
> 
> On Wed, Aug 22, 2018 at 6:42 AM, Emmanuel Brard <
> emmanuel.br...@getyourguide.com> wrote:
> 
>> Hi everyone,
>>
>> I've recently looked at the implementation of the ExternalTaskSensor sensor
>> and I was wondering if it would be a good idea to actually implement this
>> check (these checks) at the scheduler level. Basically the
>> ExternalTaskSensor runs a query against the backend database at regular
>> intervals to know if the rule(s) have passed. This is also what the
>> scheduler does to some extent checking against all task instances deps. So
>> I though of something like adding the same (or similar) set of parameters
>> of the ExternalTaskSensor directly to the operators directly and have the
>> scheduler check for this.
>>
>> The idea around this is that the ExternalTaskSensors use resources to not
>> do much actually.
>>
>> I have read somewhere that having the scheduler executing these kind of
>> task could be a potential feature, it would already be a improvement but I
>> suppose we would need the scheduler to "thread out" to do so, and this will
>> just move the resources waste from the "executors" to the scheduler.
>>
>> What do you think ?
>>
>> Cheers,
>> Emmanuel
>>
>> --
>>
>>
>>
>>
>>
>>
>>
>>
>> GetYourGuide AG
>>
>> Stampfenbachstrasse 48
>>
>> 8006 Zürich
>>
>> Switzerland
>>
>>
>>
>>  
>> 
>> 
>> 
>> 
>>
>>
>>
>>
>>
>>
>>
>>
> 

Re: ExternalTaskSensor alternatives

[Tao Feng]

FYI, there is an existing pr and proposal for improving sensor efficiency(
https://issues.apache.org/jira/browse/AIRFLOW-2747 and
https://github.com/apache/incubator-airflow/pull/3596/files) by the
community.

For your idea, I am not sure if it is a good idea to add this dag dependent
functionality to the existing scheduler given its logic is already quite
complicated.

On Wed, Aug 22, 2018 at 6:42 AM, Emmanuel Brard <
emmanuel.br...@getyourguide.com> wrote:

> Hi everyone,
>
> I've recently looked at the implementation of the ExternalTaskSensor sensor
> and I was wondering if it would be a good idea to actually implement this
> check (these checks) at the scheduler level. Basically the
> ExternalTaskSensor runs a query against the backend database at regular
> intervals to know if the rule(s) have passed. This is also what the
> scheduler does to some extent checking against all task instances deps. So
> I though of something like adding the same (or similar) set of parameters
> of the ExternalTaskSensor directly to the operators directly and have the
> scheduler check for this.
>
> The idea around this is that the ExternalTaskSensors use resources to not
> do much actually.
>
> I have read somewhere that having the scheduler executing these kind of
> task could be a potential feature, it would already be a improvement but I
> suppose we would need the scheduler to "thread out" to do so, and this will
> just move the resources waste from the "executors" to the scheduler.
>
> What do you think ?
>
> Cheers,
> Emmanuel
>
> --
>
>
>
>
>
>
>
>
> GetYourGuide AG
>
> Stampfenbachstrasse 48
>
> 8006 Zürich
>
> Switzerland
>
>
>
>  
> 
> 
> 
> 
>
>
>
>
>
>
>
>