Hi, If you are running on Postgres and 1.7.1.3 or Mysql/MariaDB and master triggering a dag run within the same second should work as long they are not triggered at the exact same time (6 fractions of a second).
Robin makes the correct case to make this work for exact same time triggers it would require rewiring task instances to use the primary key of dag runs. I’ve done this is in the past but received some push back on this from Jeremiah and thus removed it from the PR at that time. Bolke. > Op 21 dec. 2016, om 10:58 heeft Miller, Robin > <[email protected]> het volgende geschreven: > > Hi Olivier, > > > Someone correct me if I'm wrong, but my understanding is as follows: > > > At the moment the only way to identify a task_instance as for a specific > dag_run is by comparing the dag_id and execution_date. As such, having two > dag_runs with the same execution_date would make it impossible to correctly > identify the task states for each run. So that constraint is important. > > > In order to remove this constraint, I believe Airflow would need to assign a > unique key to a dag_run (possibly per dag_id, but that might be harder to > implement) and use that key to identify its task instances instead of the > execution date. > > > Regards, > > Robin Miller > OLIVER WYMAN > [email protected]<mailto:[email protected]> > www.oliverwyman.com<http://www.oliverwyman.com/> > > ________________________________ > From: Olivier Girardot <[email protected]> > Sent: 21 December 2016 07:02:05 > To: [email protected] > Subject: Re: AIRFLOW-699 - dags can't be triggered at the same second > > anyone ? > > > > > > > On Fri, Dec 16, 2016 2:46 PM, Olivier Girardot [email protected] > wrote: > Hi everyone,I wanted to talk about an issue I created AIRFLOW-699, there's an > integrity constraint regarding the dag_id and the execution_date that makes it > impossible to trigger two times with different parameters a specific dag at > the > same second. > As we are using airflow as the scheduler at the end of a complex notification > process, during high load scenario, these things happen. > Is there any hard reason for this uniqueness constraint ? > Regards, > Olivier Girardot > > > Olivier Girardot| Associé > [email protected] > +33 6 24 09 17 94 > > ________________________________ > This e-mail and any attachments may be confidential or legally privileged. If > you received this message in error or are not the intended recipient, you > should destroy the e-mail message and any attachments or copies, and you are > prohibited from retaining, distributing, disclosing or using any information > contained herein. Please inform us of the erroneous delivery by return > e-mail. Thank you for your cooperation.
