BTW. Just to add to the discussion this is an attempt of "regular" users to use and modify the timetable as defined now: https://github.com/apache/airflow/issues/22242
To be honest, I don't even know what to answer the user. Seems that the user follows our advice - tries to work from our Workday examples but there is no way the current timetable approach can be easily tested or verified and if people will start using the current timetables to do anything, this will lead to similar problems and confusion. I honestly think we should provide the users with a few well tested (automatically) configurable timetable "implementations". J. On Tue, Mar 1, 2022 at 9:27 AM Jarek Potiuk <ja...@potiuk.com> wrote: >> >> >> This would be great. I agree that if there were ready-to-use >> timetables with a composable behavior then very few users will need to >> write custom timetables which is a good thing. >> > Cool! > >> >> > I do not think the current interface is "too complex". Not at all. But I >> > think that it is targeted to a different audience than Malthe and Bas talk >> > about. It is addressed for "power users" - not only because it requires >> > deep understanding of Airflow scheduling internals and optimizations but >> > also, because it requires "admin" rights to develop, test and install it. >> > Regular users. who are Dag authors cannot create new Timetables. This is >> > mostly because of security. The "regular users" need to convince the >> > admins to do so. And yes I am talking about the important segment of our >> > users where you have professional admins/devops configuring Airflow and >> > DAG authors who just write DAGs. I think this is the most interesting and >> > biggest segment of our users to be honest. We should always think about >> > this segment of our users first IMHO. >> >> Well I think they're too complex! >> >> And I actually elaborated quite a bit on that in this correspondence – >> including giving a concrete proposal which you did not consider or >> mention. But I'm happy to be proven wrong. Perhaps it is only me who >> thinks the interface is wrong. Let us see an example implementation of >> "-2 day of every month" or "every day after work", either using a >> declarative specification built on top of some composable timetable, >> or as a direct implementation. >> > > Precisely. I think we should implement those and then we can see if they can > be simplified. I think if we have a few customized timetables including > comprehensive unit tests, we will be able to see if we can simplify the whole > interface for Airflow 3. > But having those few predefined timetables and the unit tests for those - > will be of a great help when we will want to simplify it IMHO. Then for > Airflow 3 (which is still months away if not years) we will be in a much > better position to make even breaking changes. > >> >> >> - Composability. It should be possible to use and/or operators in a >> nested fashion to compose any timetable (e.g. every Friday OR every >> last weekday AND every day at 6pm). >> - Intervals. It should be possible to define a timetable which changes >> over time, as non-overlapping intervals (e.g. one of each year). >> - Exclude. It should be possible to exclude certain days (i.e. holidays). > > > By all means : - most of those were already mentioned as "future work" in > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-39+Richer+scheduler_interval > - composability, intervals. I think we should just follow up what has been > planned there. > >> >> I think this is a good starting point and it would allow users to meet >> most of their scheduling needs. >> >> Cheers
