+1 On Thu, May 14, 2020 at 4:45 PM Kamil Breguła <[email protected]> wrote:
> +1 > > > On Wed, May 13, 2020 at 1:10 PM Kaxil Naik <[email protected]> wrote: > > > > +1 > > > > On Wed, May 13, 2020 at 11:54 AM Ash Berlin-Taylor <[email protected]> > wrote: > > > > > Hi all, > > > > > > The discussion about API spec made me think of something. > > > > > > Right now the primary key on TI is (dag_id, task_id, exeuction_date). > > > This is because when Airflow was original written, DagRun didn't exist > > > as a concept, so this was the natural PK. > > > > > > Then DagRun was added with a couple of unique constraints ( `(dag_id, > > > run_id)`, `(dag_id, execution_date)`) .But the PK on TI was never > changed. > > > > > > > > > Why does this matter? > > > > > > Well there's been often times people have asked to be able to have > > > multiple runs for the same execution_date. For example we might want to > > > have a machine learning pipeline run over the same set of data (so > > > having the same execution date) but use different hyperparameters > > > (passed in via dag_run.conf). > > > > > > This isn't supported _yet_ but with some adjusting of PKs, constraints > > > and relations in the TI this would be possible. > > > > > > > > > Why am I bringing this up now? > > > > > > Because as we are designing the API, it would be nice to change how we > > > refere to TIs changed before hand. (It would be possible to do it in a > > > compatible way, i.e. supporting both execution_date and run_id, but > it'd > > > be cleaner to only have to support one) > > > > > > So the proposal I am making right now is to change the API from: > > > > > > /dags/{dag_id}/dagRuns/{execution_date}/... > > > > > > to > > > > > > /dags/{dag_id}/dagRuns/{run_id}/... > > > > > > This alone doesn't give us the ability to have multiple dag runs for > the > > > same date, but it makes it easier to do so in the future without having > > > to change/redesign the API. > > > > > > What do people think? > > > > > > -ash > > > > -- Jarek Potiuk Polidea <https://www.polidea.com/> | Principal Software Engineer M: +48 660 796 129 <+48660796129> [image: Polidea] <https://www.polidea.com/>
