I think this is a very valid use case, so +1 from me. The code logic is IMO pretty straight-forward: take the min(expr1, expr2, expr3, …) to compute the next interval. From the human point of view, it might become complex as Jarek says, but that’s up to the developer.
And also: a “helper” view in the UI visualizing the next schedules would be very helpful! E.g. a separate view to test scheduling expressions, or just a pop-up in the DAG to show the next X scheduled times. Bas > On 28 May 2020, at 14:12, Jarek Potiuk <jarek.pot...@polidea.com> wrote: > > I am for it as long as we can thoroughly test the edge cases and maybe have > some way to visualize resulting "data intervals" in this case. > > I think big part of Airflow is that it is not just a "scheduled job" > engine, but that it works on data intervals that are defined by the > schedules. > With regular schedules. it's fairly easy to reason about what data > intervals Airflow works on, but with complex multi-cron job expressions it > will become much less obvious > > In the example you mentioned above - - Mauricio - "every 10 min between > 16:30 and 18:10" we will have 10x 10 minutes data intervals ending at > 16:40, 16:50, 17:00, 17:10, 17:20, 17:30, 17:40, 17:50, 18:00, 18:10 and > one ~ 22.5h interval between 18:10 and 16:30 next day. > > This is a simple example of course and it is currently also possible for > fixed hours in cron, but I can imagine if we introduce the capability of > multiple cron job expressions that introduce arbitrary complex schedules, > the schedules might be super-difficult to reason about if you start mixing > them. > > I think it is fine if the users want to do it, but also for the convenience > of the users themselves. maybe there should be some way (Web UI? CLI?) > where you can take such a schedule and see the data intervals you can > expect to have? > > WDYT? > > J. > > On Thu, May 28, 2020 at 2:37 AM Shaw, Damian P. < > damian.sha...@credit-suisse.com> wrote: > >> Big +1 to anything that extends the limitations of Airflow's current >> scheduling capability. >> >> For me the only drawback of this is it doesn't go far enough and further >> additions would needed to be added later, it would still be difficult to >> express things that require updatable calendars like "Every Business Day" >> or things which are hard to express even with composible crontabs likes >> "The first week day of the month". >> >> But if this is an easy win I hope it's taken seriously. >> >> Damian. >> >> -----Original Message----- >> From: Mauricio De Diana <mdedi...@gmail.com> >> Sent: Wednesday, May 27, 2020 14:27 >> To: dev@airflow.apache.org >> Subject: Support for multiple cron expressions >> >> Hello all, >> >> At the moment some schedules are not possible in Airflow, for example, >> "every 10 min between 16:30 and 18:10". Such schedules would be possible if >> Airflow supported multiple cron expressions, as described in >> https://github.com/apache/airflow/issues/8649. In the issue, I was >> suggested to bring the discussion here because this may not be a desirable >> feature. >> >> In terms of implementation, I gave the idea a try and I have something >> working. For that, besides str, timedelta and relativedelta, a schedule >> interval can also be a list of strings representing cron expressions. There >> is a class that is a composite of croniter objects and providing the same >> methods. It works seamlessly for one or many cron expressions, so changes >> in the scheduler code are mostly replacing croniter with this class. >> >> I can create a PR if there is interest in discussing the implementation, >> but first I would like to learn opinions about this feature? Is it an idea >> worth following? >> >> Thanks, >> Mauricio >> >> >> >> =============================================================================== >> >> Please access the attached hyperlink for an important electronic >> communications disclaimer: >> http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html >> =============================================================================== >> >> > > > -- > > Jarek Potiuk > Polidea <https://www.polidea.com/> | Principal Software Engineer > > M: +48 660 796 129 <+48660796129> > [image: Polidea] <https://www.polidea.com/>
