We use LatestOnlyOperator in production. Generally our data is available on a regular schedule, and we update production services with it as soon as it is available; we might occasionally want to re-run historical days, in which case we want to run the same DAG but without interacting with live production services at all.
On Mon, Jul 23, 2018 at 2:18 PM, George Leslie-Waksman <waks...@gmail.com> wrote: > As the author of LatestOnlyOperator, the goal was as a stopgap until > catchup=False landed. > > There are some (very) fringe use cases where you might still want > LatestOnlyOperator but in almost all cases what you want is probably > catchup=False. > > The situations where LatestOnlyOperator is still useful are where you want > to run most of your DAG for every schedule interval but you want some of > the tasks to run only on the latest run (not catching up, not backfilling). > > It may be best to deprecate LatestOnlyOperator at this point to avoid > confusion. > > --George > > On Sat, Jul 21, 2018 at 7:34 PM Ben Tallman <btall...@gmail.com> wrote: > > > As the author of catch-up, the idea is that in many cases your data > > doesn't "window" nicely and you want instead to just run as if it were a > > brilliant Cron... > > > > Ben > > > > Sent from my iPhone > > > > > On Jul 20, 2018, at 11:39 PM, Shah Altaf <mend...@gmail.com> wrote: > > > > > > Hi my understanding is: if you use the LatestOnlyOperator then when you > > run > > > the DAG for the first time you'll see a whole bunch of DAG runs queued > > up, > > > and in each run the LatestOnlyOperator will cause the rest of the DAG > run > > > to be skipped. Only the latest DAG will run in 'full'. > > > > > > With catchup = False, you should just get just the latest DAG run. > > > > > > > > > On Fri, Jul 20, 2018 at 10:58 PM Shubham Gupta < > > shubham180695...@gmail.com> > > > wrote: > > > > > >> ---------- Forwarded message --------- > > >> From: Shubham Gupta <shubham180695...@gmail.com> > > >> Date: Fri, Jul 20, 2018 at 2:38 PM > > >> Subject: Catchup By default = False vs LatestOnlyOperator > > >> To: <dev-subscr...@airflow.incubator.apache.org> > > >> > > >> > > >> Hi! > > >> > > >> Can someone please explain the difference b/w catchup by default = > False > > >> and LatestOnlyOperator? > > >> > > >> Regarding > > >> Shubham Gupta > > >> > > >
