With the already very deep focus on ETL processes, there is a rising interest in applying apache airflow also as a full stack scheduling system for all types of processes. We are applying apache airflow as a scheduling system at AXOVISION not only for ETL use cases, but also as a full replacement of cron jobs. But still there is functionality missing for a full stack scheduling system. But we think that adding these feature is not to complex, because the fundamentals are already there. So it is like adding simple new features to airflow to reach this stage of full stack scheduling system, aside of ETL process management. The most important features to add would be: *Dynamic schedule intervals *and *Unpause/Pause without any catchup (no run of last recent) *
Description There are several question (issues) on stack overflow, asking for the need of a dynamic schedule interval. This means, the ability to change the schedule interval after DAG creation programmatically via API or CLI. Further, user alo asking for a detailed discussion on some stackoverflow questions, which could not be find. With the ability to dynamically change DAG schedule intervals, airflow can increase user satisfaction and fully replace other custom cron like schedule systems. reference stack overflow links: - https://stackoverflow.com/questions/63494560/airflow-schedule-interval-change - https://stackoverflow.com/questions/63271671/can-we-parameterize-the-airflow-schedule-interval-dynamically-reading-from-the-v - https://stackoverflow.com/questions/37294560/airflow-changing-the-crontab-time-for-a-dag-in-airflow Within the Pitfalls it is also mentioned: - https://cwiki.apache.org/confluence/display/AIRFLOW/Common+Pitfalls Use case / motivation As a user I want to change the schedule interval of an already defined DAG, so that the DAG can run dynamically at different time points depended on an external condition. The motivation is simple: there are a lot of use cases not fitting the only once cron like created definition of a schedule interval of a DAG. (e.g. event driven schedules) Use Case: In the morning there was a database update announced, due to ongoing work in the backend, the database should be updated always 2 hours after the announcement. Which could be always a different point in time, according to when the event happened. So the database update DAG needs to be scheduled on a new interval to run. Related Issues I could not find any directly related issues to that.
