Here is again a use case for dynamic scheduling: from the origin link: https://github.com/apache/airflow/issues/10449
Description There are several question (issues) on stack overflow, asking for the need of a dynamic schedule interval. This means, the ability to change the schedule interval after DAG creation programmatically via API or CLI. Further, user alo asking for a detailed discussion on some stackoverflow questions, which could not be find. With the ability to dynamically change DAG schedule intervals, airflow can increase user satisfaction and fully replace other custom cron like schedule systems. reference stack overflow links: * https://stackoverflow.com/questions/63494560/airflow-schedule-interval-change * https://stackoverflow.com/questions/63271671/can-we-parameterize-the-airflow-schedule-interval-dynamically-reading-from-the-v * https://stackoverflow.com/questions/37294560/airflow-changing-the-crontab-time-for-a-dag-in-airflow Within the Pitfalls it is also mentioned: * https://cwiki.apache.org/confluence/display/AIRFLOW/Common+Pitfalls Use case / motivation As a user I want to change the schedule intervall of an already defined DAG, so that the DAG can run dynamically at different time points depended on an external condition. The motivation is simple: there are a lot of use cases not fitting the only once cron like created definition of a schedule interval of a DAG. (e.g. event driven schedules) Use Case: In the morning there was a database update announced, due to ongoing work in the backend, the database should be updated always 2 hours after the announcement. Which could be always a different point in time, according to when the event happened. So the database update DAG needs to be scheduled on a new interval to run. Related Issues I could not find any directly related issues to that. On 2019/08/23 10:49:30, Ash Berlin-Taylor <a...@apache.org> wrote: > This has come up a few times before, someone has now opened a PR that makes this a global+per-dag setting: https://github.com/apache/airflow/pull/5787 and it also includes docs that I think does a good job of illustrating the two modes.> > > Does anyone object to this being merged? If no one says anything by midday on Tuesday I will take that as assent and will merge it.> > > The docs from the PR included below.> > > Thanks,> > Ash> > > Scheduled Time vs Execution Time> > ''''''''''''''''''''''''''''''''> > > A DAG with a ``schedule_interval`` will execute once per interval. By> > default, the execution of a DAG will occur at the **end** of the> > schedule interval.> > > A few examples:> > > - A DAG with ``schedule_interval='@hourly'``: The DAG run that processes> > 2019-08-16 17:00 will start running just after 2019-08-16 17:59:59,> > i.e. once that hour is over.> > - A DAG with ``schedule_interval='@daily'``: The DAG run that processes> > 2019-08-16 will start running shortly after 2019-08-17 00:00.> > > The reasoning behind this execution vs scheduling behaviour is that> > data for the interval to be processed won't be fully available until> > the interval has elapsed.> > > In cases where you wish the DAG to be executed at the **start** of the> > interval, specify ``schedule_at_interval_end=False``, either in> > ``airflow.cfg``, or on a per-DAG basis.>
