It shouldn't need something that complex (or to my mind hacky) as in internal DAG.
The way the scheduler works now it just looks at two columns on the dag (model) table called I think "next_dagrun_after" (which is the earliest date that the dag run can be created, and "next execution date" (which is the value to put in the execution date of the dag run when it's created. Both these values are set by the dag parser process, which has full access to run code. What ever interface for defining new schedule expression should run in the existing process, much like how James C did in a subclass. Ash On 21 January 2021 18:21:58 GMT, Daniel Imberman <daniel.imber...@gmail.com> wrote: >I think James Idea sounds like a pretty good idea. What would you all think of >us doing something similar to how we handle smart sensors for how we implement >this? Have an internal DAG that reads all custom timetables and triggers a DAG >if the function returns True? Seems like a pretty simple/customizeable >solution. >On Wed, Jan 20, 2021 at 5:52 PM, James Timmins <ja...@astronomer.io> wrote: >Django provides a really good model for allowing users to customize the >behavior of Class Based Views. It's in line w/ what Daniel/Kaxil and co are >saying about a consistent backend class. It uses a standard base class as well >as a default concrete implementation. Customization then only requires setting >an explicit class if you're overriding the default. >Seems that the interface is more important than the backend mechanism to make >this work. There are multiple ways to make this work internally, but the >interface should be in line with future plans for hooks/extensible areas. >Just to make things concrete, here's my understanding of what that would look >like / what they're suggesting. >BaseTimetable abstract class - Defines a ` get_next_execution_time ` method. >This method accepts one argument, an arbitrary datetime value. Based on that >datetime, this method returns the next time the DAG should start. This makes >it easy to schedule past events, and also makes it easy to print out a "dry >run" of execution times for testing purposes. - Defines a >'_check_timetable_arguments ` method that looks for any existing timetable >args in the DAG and makes sure they're used by whatever Timetable class is >selected. Error checking. >CronTimetable - Default TimetableClass. Built on BaseTimetable. >If they want a different timetable, they can just extend BaseTimetable and >define a custom `get_next_execution_time` class. Then pass the class into the >DAG constructor under the `timetable_class` argument. So for `sunset` or >`sunrise`, they could easily create a `SolarTimetable` class and pass that in. >`get_next_execution_time` can then be called whenever DAGs are parsed or >whenever tasks run. >On Wed, Jan 20, 2021 at 3:53 PM James Coder < jcode...@gmail.com >[jcode...@gmail.com] > wrote: >Kaxil you beat me to it. I actually have a dag where I achieve an irregular >schedule by overriding DAG.next_dagrun_info(). If that method were swapped out >for an object it may be a semi-easy way to make the schedule “plugable”. > >James Coder >On Jan 20, 2021, at 6:37 PM, Kaxil Naik < kaxiln...@gmail.com >[kaxiln...@gmail.com] > wrote: > >"CronBackend" / "ScheduleIntervalBackend" :D similar to Xcom and Secrets >Backend >Would be definitely good to have Custom Schedule intervals using >functions/class that is Serializable too. > >On Wed, Jan 20, 2021 at 11:02 PM QP Hou <q...@scribd.com.invalid> wrote: >On Wed, Jan 20, 2021 at 10:22 AM Daniel Imberman >< daniel.imber...@gmail.com [daniel.imber...@gmail.com] > wrote: >> >> I love the idea of allowing users to create their own scheduling >> objects/scheduling python functions. They could either live in the scheduler >> or as a seperate process that trips some value in the DB when it is “true”. >> Would be great from a “marketplace” standpoint as well as users could post >> their custom scheduling objects for others to use. >> > >I like this idea as well, a quick escape patch for custom and complex >scheduling behaviors without having to wait for upstream support.
