+1 but in the future I think better would be
/dags/{dag_id}/dagRuns/{execution_date}/{run_number}. That would give an
automatic ordering between two runs, is a lot simpler than
"backfill_2020-03-16T00:00:00+00:00" and helps enable the multiple
dagruns per execution date that you mention.
On Thu, May 14, 2020 at 4:09 PM Jarek Potiuk <jarek.pot...@polidea.com>
wrote:
> +1
>
> On Thu, May 14, 2020 at 4:45 PM Kamil Breguła <kamil.breg...@polidea.com>
> wrote:
>
> > +1
> >
> >
> > On Wed, May 13, 2020 at 1:10 PM Kaxil Naik <kaxiln...@gmail.com> wrote:
> > >
> > > +1
> > >
> > > On Wed, May 13, 2020 at 11:54 AM Ash Berlin-Taylor <a...@apache.org>
> > wrote:
> > >
> > > > Hi all,
> > > >
> > > > The discussion about API spec made me think of something.
> > > >
> > > > Right now the primary key on TI is (dag_id, task_id, exeuction_date).
> > > > This is because when Airflow was original written, DagRun didn't
> exist
> > > > as a concept, so this was the natural PK.
> > > >
> > > > Then DagRun was added with a couple of unique constraints ( `(dag_id,
> > > > run_id)`, `(dag_id, execution_date)`) .But the PK on TI was never
> > changed.
> > > >
> > > >
> > > > Why does this matter?
> > > >
> > > > Well there's been often times people have asked to be able to have
> > > > multiple runs for the same execution_date. For example we might want
> to
> > > > have a machine learning pipeline run over the same set of data (so
> > > > having the same execution date) but use different hyperparameters
> > > > (passed in via dag_run.conf).
> > > >
> > > > This isn't supported _yet_ but with some adjusting of PKs,
> constraints
> > > > and relations in the TI this would be possible.
> > > >
> > > >
> > > > Why am I bringing this up now?
> > > >
> > > > Because as we are designing the API, it would be nice to change how
> we
> > > > refere to TIs changed before hand. (It would be possible to do it in
> a
> > > > compatible way, i.e. supporting both execution_date and run_id, but
> > it'd
> > > > be cleaner to only have to support one)
> > > >
> > > > So the proposal I am making right now is to change the API from:
> > > >
> > > > /dags/{dag_id}/dagRuns/{execution_date}/...
> > > >
> > > > to
> > > >
> > > > /dags/{dag_id}/dagRuns/{run_id}/...
> > > >
> > > > This alone doesn't give us the ability to have multiple dag runs for
> > the
> > > > same date, but it makes it easier to do so in the future without
> having
> > > > to change/redesign the API.
> > > >
> > > > What do people think?
> > > >
> > > > -ash
> > > >
> >
>
>
> --
>
> Jarek Potiuk
> Polidea <https://www.polidea.com/> | Principal Software Engineer
>
> M: +48 660 796 129 <+48660796129>
> [image: Polidea] <https://www.polidea.com/>
>
