Good to know. -s On Wed, Aug 17, 2016 at 5:54 PM, Felix Cheung <felixcheun...@hotmail.com> wrote:
> Apache Oozie has LAST_ONLY execution policy in which only the current > instance will run when catching up or backfilling. > > https://oozie.apache.org/docs/4.2.0/CoordinatorFunctionalSpec. > html#a6.3._Synchronous_Coordinator_Application_Definition > > > _____________________________ > From: siddharth anand <san...@apache.org<mailto:san...@apache.org>> > Sent: Wednesday, August 17, 2016 5:39 PM > Subject: Re: ShortCircuitIfNotCurrentOperator > To: <dev@airflow.incubator.apache.org<mailto:dev@airflow. > incubator.apache.org>> > > > Hmn... none that I know of, but it seems pretty useful for us! > > On Wed, Aug 17, 2016 at 4:43 PM, Felix Cheung <felixcheun...@hotmail.com< > mailto:felixcheun...@hotmail.com>> > wrote: > > > Cool. Other scheduler has a concept called LAST_ONLY? > > > > > > > > > > > > > > On Wed, Aug 17, 2016 at 2:34 PM -0700, "siddharth anand" < > > san...@apache.org<mailto:san...@apache.org><mailto:san...@apache.org>> > wrote: > > > > For the longest time, we have been discussing supporting full cron > > functionality in Airflow. Currently, Airflow supports cron syntax with > one > > pretty visible issue. > > > > For example, if you wanted cron to schedule a job every day at midnight, > > you would specify 0 0 * * *. In Airflow, if you were to pause your dag > for > > a week and then unpause it, your dag would run the past 7 days' worth of > > dag runs. However, a common use-case for cron schedules is to use it for > > some routine job scheduling (e.g. full db backup, log file cleanup, > > minutely alert condition checking) that only needs to run at midnight, > but > > doesn't need to "backfill" the days missed. > > > > We had originally envisioned building this into the scheduler, but that > is > > unnecessarily complex. Instead, we have had the ShortCircuitOperator > around > > for a while. I plan to extend this to make a > > ShortCircuitIfNotCurrentOperator. This will skip downstream for dag runs > > that are not current. > > > > https://issues.apache.org/jira/browse/AIRFLOW-198 > > > > FYI.. alternate names for this operator are welcome. > > -s > > > > >
