We are seeing another major issue with backfills where task instances are being deleted and marked as "removed", I am still investigating. Let's keep discussion about these in https://issues.apache.org/jira/browse/AIRFLOW-921 and the subtask comments to have it one place. I will look at the other points you cc'd me on too. Thanks for continuing to drive this forward!
On Wed, Mar 1, 2017 at 8:22 AM, Bolke de Bruin <bdbr...@gmail.com> wrote: > Hi, > > Just wanted to give an update about the progress getting to RC5. As > reported we have 6 blockers listed. > > 1. Double run job should not terminate the existing running job. -> Patch > Available > 2. Parallelize dag runs in backfills -> Patch Available, Tests need to be > updated, see below > 3. Setting a task to running manually breaks a DAGs UI -> Patch merged > 4. Can't mark non-existent tasks as successful from graph view -> > Workaround available (t.b.c.), Patch Available unit tests to be added > 5. (Named)HivePartitionSensor broken if hook attr not set -> Patch merged > 6. Skipped tasks potentially cause a dagrun to be marked as > failure/success prematurely -> see below > > On 2 I would like to have some more discussion of this would be acceptable > (https://github.com/apache/incubator-airflow/pull/2107). I have written > the patch for this, however we are not large backfill users. So I need > feedback specifically on ripping out the “executor” part: @dan, @max. > > On 6 Alex has reported this earlier and written a PR for this ( > https://github.com/apache/incubator-airflow/pull/1961). Maxime had some > thoughts about this, which are currently blocking the integration. However, > in testing it seems to solve the issue. Can we finalise the discussion > please @max @dan @alex? > > Cheers > Bolke
