I start thinking more and more that we also should support this kind of case. We are not very far from it and people are using it this way anyway, so it would be a bit turning a blind eye if we don't try to accommodate - especially that it is not as difficult to do.
I think maybe we then should have the ETL (or rather Data interval) and non-Data Interval type of DAGs? And this actually goes hand-in-hand with the discussion about "schedule at the end or beginning of the interval". Maybe introducing the two kinds of Dags might kill two birds with the same stone? We could have "Data Interval DAGs" with cron specs pointing to the end of the interval, and "non-data-interval" ones triggered according to cron schedule. Those two seem related. Just a wild thought :) J. On Thu, Sep 3, 2020 at 1:44 PM Axovision Team <[email protected]> wrote: > > With the already very deep focus on ETL processes, there is a rising > interest in applying apache airflow also as a full stack scheduling system > for all types of processes. We are applying apache airflow as a scheduling > system at AXOVISION not only for ETL use cases, but also as a full > replacement of cron jobs. But still there is functionality missing for a > full stack scheduling system. But we think that adding these feature is not > to complex, because the fundamentals are already there. So it is like > adding simple new features to airflow to reach this stage of full stack > scheduling system, aside of ETL process management. The most important > features to add would be: *Dynamic schedule intervals *and *Unpause/Pause > without any catchup (no run of last recent) * > > Description > There are several question (issues) on stack overflow, asking for the need > of a dynamic schedule interval. This means, the ability to change the > schedule interval after DAG creation programmatically via API or CLI. > Further, user alo asking for a detailed discussion on some stackoverflow > questions, which could not be find. > With the ability to dynamically change DAG schedule intervals, airflow can > increase user satisfaction and fully replace other custom cron like > schedule systems. > > reference stack overflow links: > > - > > https://stackoverflow.com/questions/63494560/airflow-schedule-interval-change > - > > https://stackoverflow.com/questions/63271671/can-we-parameterize-the-airflow-schedule-interval-dynamically-reading-from-the-v > - > > https://stackoverflow.com/questions/37294560/airflow-changing-the-crontab-time-for-a-dag-in-airflow > > Within the Pitfalls it is also mentioned: > > - https://cwiki.apache.org/confluence/display/AIRFLOW/Common+Pitfalls > > Use case / motivation > As a user I want to change the schedule interval of an already defined DAG, > so that the DAG can run dynamically at different time points depended on an > external condition. > > The motivation is simple: there are a lot of use cases not fitting the only > once cron like created definition of a schedule interval of a DAG. (e.g. > event driven schedules) > > Use Case: > In the morning there was a database update announced, due to ongoing work > in the backend, the database should be updated always 2 hours after the > announcement. Which could be always a different point in time, according to > when the event happened. So the database update DAG needs to be scheduled > on a new interval to run. > > Related Issues > I could not find any directly related issues to that. -- Jarek Potiuk Polidea | Principal Software Engineer M: +48 660 796 129
