Sergei Iakhnin created AIRFLOW-193:
--------------------------------------
Summary: Allow a series of tasks to be executed on the same worker
Key: AIRFLOW-193
URL: https://issues.apache.org/jira/browse/AIRFLOW-193
Project: Apache Airflow
Issue Type: New Feature
Reporter: Sergei Iakhnin
Currently the only way to limit the execution of a series of tasks to a single
worker is via pools, however this is not a very convenient method when managing
hundreds of workers.
In the context of scientific workflows it is a common desire to be able to
retrieve a (possibly large) sample from a data repository (or object store),
then progressively elaborate it via a series of transformations, and finally
deposit the result back. From a modelling perspective it makes sense to have
the series of transformations each be encapsulated in a separate task. From
practical considerations (performance, network bandwidth) it would be desirable
to retrieve the sample to a single worker's local storage, where it would then
be worked upon until completion. This, of course, requires the ability to have
a slew of tasks to be bound to a particular worker.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
