Hi Ben,
Just copying my comment form ticket.
I think current airflow scheduler schedule DAGs only on time-basis (based
on cron schedule string). ***Is it correct understanding?*
How to approach a scenario where I want to trigger a DAG based on some
event which is not so predictable/regular on time basis?
- One is to use sensor and DAG flow will start by first running a sensor
task for checking an event and when an event happened actual processing get
a start.
- Second, is to have a DAG with None schedule and it gets triggered by
some other application or utility which checks for an event occurrence .
If your most of the flow(DAGs) are suppose to run on this logic, then
sensor doesn't make sense and hence left with the second approach only,
which is nothing but a custom scheduling of DAGs. ****Is it agreeable use
case for the custom scheduler?*
Thanks, {{Abhishek}}
On Thu, Jan 31, 2019 at 11:53 AM Ben Tallman <[email protected]> wrote:
> Can you explain a bit more what you are thinking for a custom scheduler?
> It's been awhile, but we added support for cron schedules without backfill
> awhile back, so I'm wondering what you are thinking of adding with this?
>
> Thanks,
> Ben
>
> --
> Ben Tallman - 503.680.5709
>
>
> On Thu, Jan 31, 2019 at 8:29 AM abhishek sharma <
> [email protected]>
> wrote:
>
> > Hi All,
> >
> > Created a ticket(https://issues.apache.org/jira/browse/AIRFLOW-3775) for
> > supporting custom scheduler in Airflow.
> >
> > The idea is to have a scheduler base class which can be extended for
> > writing a custom scheduler. The logic of custom scheduling is user
> > specific, and at the DAGs task level we can mention the scheduler type,
> and
> > that scheduler will be used for starting a task. [Naive Idea]
> >
> > Can we please discuss whether we need such functionality in Airflow or
> not
> > and If yes then we will proceed with the design and implementation.
> >
> > Thanks
> > Abhishek Sharma
> >
>
