Pavlo, Please follow the instructions in the Airflow Contributor Guide <https://cwiki.apache.org/confluence/display/AIRFLOW/Contributor+Guide> to create a mailing list account on dev and commits.
This will avoid us from having to accept/reject your individual emails. When you email dev-subscr...@airflow.incubator.apache.org, a bot will take care of your subscription. -s On Thu, May 19, 2016 at 3:07 PM, < dev-reject-1463670472.76373.oebhcfdbhheddaapm...@airflow.incubator.apache.org > wrote: > > To approve: > > dev-accept-1463670472.76373.oebhcfdbhheddaapm...@airflow.incubator.apache.org > To reject: > > dev-reject-1463670472.76373.oebhcfdbhheddaapm...@airflow.incubator.apache.org > To give a reason to reject: > %%% Start comment > %%% End comment > > > > ---------- Forwarded message ---------- > From: "Ryabchuk, Pavlo" <ext-pavlo.ryabc...@here.com> > To: "dev@airflow.incubator.apache.org" <dev@airflow.incubator.apache.org> > Cc: > Date: Thu, 19 May 2016 15:07:42 +0000 > Subject: Using CeleryExecutor to distribute SubDags, but running SubDAG > tasks locally one by one afterwards > > Hello, > > > > I am trying to fully use Airflow analytics and trying to make my tasks as > granular as possible still having the benefit of CeleryExecutor present. > > In general I want to make my DAG to consist of 100+ Subgags which are > actually distributed by Celery, but I want to have subdag tasks all > executed on same worker instance. > > SubDAG is in general this: copyData data from S3 to instance -> run > calculation -> copy result to S3. The reason I want to split it into 3 > tasks is to have ability to measure pure calculation time and aply SLA to > it and also get better statisticts on copy operations. > > So the main question is how to execute tasks of SubDAG one after another > on same instance? > > > > I’ve came across this issue/workaround here > https://issues.apache.org/jira/browse/AIRFLOW-74, but I believe it won’t > solve my issue. > > If it is not supported and I am not missing some magic configuration J > but still could be implemented with relatively small effort – I am in :) > > > > Best, > > Paul > >
