Re: ExternalTaskSensor alternatives

[Stefan Seelmann]

On 08/22/2018 06:56 PM, Tao Feng wrote:
> FYI, there is an existing pr and proposal for improving sensor efficiency(
> https://issues.apache.org/jira/browse/AIRFLOW-2747 and
> https://github.com/apache/incubator-airflow/pull/3596/files) by the
> community.

And I hope I'll find some time next week to continue working on it. More
feedback and help is welcomed :)

> For your idea, I am not sure if it is a good idea to add this dag dependent
> functionality to the existing scheduler given its logic is already quite
> complicated.
> 
> On Wed, Aug 22, 2018 at 6:42 AM, Emmanuel Brard <
> emmanuel.br...@getyourguide.com> wrote:
> 
>> Hi everyone,
>>
>> I've recently looked at the implementation of the ExternalTaskSensor sensor
>> and I was wondering if it would be a good idea to actually implement this
>> check (these checks) at the scheduler level. Basically the
>> ExternalTaskSensor runs a query against the backend database at regular
>> intervals to know if the rule(s) have passed. This is also what the
>> scheduler does to some extent checking against all task instances deps. So
>> I though of something like adding the same (or similar) set of parameters
>> of the ExternalTaskSensor directly to the operators directly and have the
>> scheduler check for this.
>>
>> The idea around this is that the ExternalTaskSensors use resources to not
>> do much actually.
>>
>> I have read somewhere that having the scheduler executing these kind of
>> task could be a potential feature, it would already be a improvement but I
>> suppose we would need the scheduler to "thread out" to do so, and this will
>> just move the resources waste from the "executors" to the scheduler.
>>
>> What do you think ?
>>
>> Cheers,
>> Emmanuel
>>
>> --
>>
>>
>>
>>
>>
>>
>>
>>
>> GetYourGuide AG
>>
>> Stampfenbachstrasse 48
>>
>> 8006 Zürich
>>
>> Switzerland
>>
>>
>>
>>  
>> 
>> 
>> 
>> 
>>
>>
>>
>>
>>
>>
>>
>>
> 

Re: ExternalTaskSensor alternatives

[Tao Feng]

FYI, there is an existing pr and proposal for improving sensor efficiency(
https://issues.apache.org/jira/browse/AIRFLOW-2747 and
https://github.com/apache/incubator-airflow/pull/3596/files) by the
community.

For your idea, I am not sure if it is a good idea to add this dag dependent
functionality to the existing scheduler given its logic is already quite
complicated.

On Wed, Aug 22, 2018 at 6:42 AM, Emmanuel Brard <
emmanuel.br...@getyourguide.com> wrote:

> Hi everyone,
>
> I've recently looked at the implementation of the ExternalTaskSensor sensor
> and I was wondering if it would be a good idea to actually implement this
> check (these checks) at the scheduler level. Basically the
> ExternalTaskSensor runs a query against the backend database at regular
> intervals to know if the rule(s) have passed. This is also what the
> scheduler does to some extent checking against all task instances deps. So
> I though of something like adding the same (or similar) set of parameters
> of the ExternalTaskSensor directly to the operators directly and have the
> scheduler check for this.
>
> The idea around this is that the ExternalTaskSensors use resources to not
> do much actually.
>
> I have read somewhere that having the scheduler executing these kind of
> task could be a potential feature, it would already be a improvement but I
> suppose we would need the scheduler to "thread out" to do so, and this will
> just move the resources waste from the "executors" to the scheduler.
>
> What do you think ?
>
> Cheers,
> Emmanuel
>
> --
>
>
>
>
>
>
>
>
> GetYourGuide AG
>
> Stampfenbachstrasse 48
>
> 8006 Zürich
>
> Switzerland
>
>
>
>  
> 
> 
> 
> 
>
>
>
>
>
>
>
>

ExternalTaskSensor alternatives

[Emmanuel Brard]

Hi everyone,

I've recently looked at the implementation of the ExternalTaskSensor sensor
and I was wondering if it would be a good idea to actually implement this
check (these checks) at the scheduler level. Basically the
ExternalTaskSensor runs a query against the backend database at regular
intervals to know if the rule(s) have passed. This is also what the
scheduler does to some extent checking against all task instances deps. So
I though of something like adding the same (or similar) set of parameters
of the ExternalTaskSensor directly to the operators directly and have the
scheduler check for this.

The idea around this is that the ExternalTaskSensors use resources to not
do much actually.

I have read somewhere that having the scheduler executing these kind of
task could be a potential feature, it would already be a improvement but I
suppose we would need the scheduler to "thread out" to do so, and this will
just move the resources waste from the "executors" to the scheduler.

What do you think ?

Cheers,
Emmanuel

-- 








GetYourGuide AG

Stampfenbachstrasse 48  

8006 Zürich

Switzerland