Hey Kevin, It looks like a good one for an AIF(https://cwiki.apache.org/ confluence/display/AIRFLOW/Airflow+Improvements+Proposals) proposal.
Thanks, -Tao On Fri, Aug 10, 2018 at 8:54 AM, Maxime Beauchemin < maximebeauche...@gmail.com> wrote: > Hey, > > I agree, I always thought this would be handled through a BaseOperator flag > `only_run_latest=True`. There are some potentially confusing / incompatible > scenarios like: > * can't have both depend_on_past and only_run_latest (obviously...) > * can't have any downstream tasks of only_run_latest have depend_on_past > * same 2 statements also apply to "wait_for_downstream" > * more? > > Max > > On Fri, Aug 10, 2018 at 12:43 AM Kevin Campbell <k...@diffractive.io> > wrote: > > > Hi everyone, > > > > We had some issues with the ShortCircuitOperator, so had to come up with > an > > alternative implementation. > > > > I've written up the details and the workaround in this blog posting: > > > > > > > > https://blog.diffractive.io/2018/08/07/replacement-shortcirc > uitoperator-for-airflow/ > > > > Seems that to fix this in airflow properly, it needs some design > changes. I > > think the sanest solution would be to have the scheduler handle skip > > operations, rather than implement this inside an operator. That would > mean > > either having the scheduler read xcom data, or elevating the status of > the > > python member to a first class attribute on a task. > > > > Anyway, hoping this posting might at least save some users some headaches > > if they want the ShortCircuitOperator to behave better. > > > > Comments and feedback appreciated. > > > > Regards, > > Kevin > > >
