FYI: When all root tasks (i.e. the last tasks to run) have succeeded the DagRun is considered successful and the scheduler will not consider any other tasks in the dag run. The code is here: https://github.com/apache/incubator-airflow/blob/master/airflow/models.py#L4095 for version 1.8, and hasn’t changed significantly since 1.7: https://github.com/apache/incubator-airflow/blob/airbnb_rb1.7.1_4/airflow/models.py#L2678 . As 1.7 is more task based and 1.8 more dag run based the behaviour between 1.7 and 1.8 might be different (Sid is investigating).
Thus to make sure the tasks run you will need to clear the Root task - thus downstream clearing will definitely work. I’m not sure this is a change in behaviour as explained above. As most of the use case will clear downstream (its the default), a workaround is available, I don’t consider it a blocker. - Bolke. > On 15 Mar 2017, at 09:17, Chris Riccomini <criccom...@apache.org> wrote: > > @Sid, does this happen if you clear downstream as well? > > On Wed, Mar 15, 2017 at 9:04 AM, Chris Riccomini <criccom...@apache.org> > wrote: > >> Has anyone been able to reproduce Sid's issue? >> >> On Tue, Mar 14, 2017 at 11:17 PM, Bolke de Bruin <bdbr...@gmail.com> >> wrote: >> >>> That is not an airflow error, but a Kerberos error. Try executing the >>> kinit command on the command line by yourself. >>> >>> Bolke >>> >>> Sent from my iPhone >>> >>>> On 14 Mar 2017, at 23:11, Ruslan Dautkhanov <dautkha...@gmail.com> >>> wrote: >>>> >>>> `airflow kerberos` is broken in 1.8-rc5 >>>> https://issues.apache.org/jira/browse/AIRFLOW-987 >>>> Hopefully fix can be part of the 1.8 release. >>>> >>>> >>>> >>>> -- >>>> Ruslan Dautkhanov >>>> >>>>> On Tue, Mar 14, 2017 at 6:19 PM, siddharth anand <san...@apache.org> >>> wrote: >>>>> >>>>> FYI, >>>>> I've just hit a major bug in the release candidate related to "clear >>> task" >>>>> behavior. >>>>> >>>>> I've been running airflow in both stage and prod since yesterday on >>> rc5 and >>>>> have reproduced this in both environments. I will file a JIRA for this >>>>> tonight, but wanted to send a note over email as well. >>>>> >>>>> In my example, I have a 2 task DAG. For a given DAG run that has >>> completed >>>>> successfully, if I >>>>> 1) clear task2 (leaf task in this case), the previously-successful DAG >>> Run >>>>> goes back to Running, requeues, and executes the task successfully. >>> The DAG >>>>> Run the returns from Running to Success. >>>>> 2) clear task1 (root task in this case), the previously-successful DAG >>> Run >>>>> goes back to Running, DOES NOT requeue or execute the task at all. The >>> DAG >>>>> Run the returns from Running to Success though it never ran the task. >>>>> >>>>> 1) is expected and previous behavior. 2) is a regression. >>>>> >>>>> The only workaround is to use the CLI to run the task cleared. Here are >>>>> some images : >>>>> *After Clearing the Tasks* >>>>> https://www.dropbox.com/s/wmuxt0krwx6wurr/Screenshot% >>>>> 202017-03-14%2014.09.34.png?dl=0 >>>>> >>>>> *After DAG Runs return to Success* >>>>> https://www.dropbox.com/s/qop933rzgdzchpd/Screenshot% >>>>> 202017-03-14%2014.09.49.png?dl=0 >>>>> >>>>> This is a major regression because it will force everyone to use the >>> CLI >>>>> for things that they would normally use the UI for. >>>>> >>>>> -s >>>>> >>>>> >>>>> -s >>>>> >>>>> >>>>>> On Tue, Mar 14, 2017 at 1:32 PM, Daniel Huang <dxhu...@gmail.com> >>> wrote: >>>>>> >>>>>> +1 (non-binding)! >>>>>> >>>>>> On Tue, Mar 14, 2017 at 11:35 AM, siddharth anand <san...@apache.org> >>>>>> wrote: >>>>>> >>>>>>> +1 (binding) >>>>>>> >>>>>>> >>>>>>> On Tue, Mar 14, 2017 at 8:42 AM, Maxime Beauchemin < >>>>>>> maximebeauche...@gmail.com> wrote: >>>>>>> >>>>>>>> +1 (binding) >>>>>>>> >>>>>>>> On Tue, Mar 14, 2017 at 3:59 AM, Alex Van Boxel <a...@vanboxel.be> >>>>>>> wrote: >>>>>>>> >>>>>>>>> +1 (binding) >>>>>>>>> >>>>>>>>> Note: we had to revert all our ONE_SUCCESS with ALL_SUCCESS trigger >>>>>>> rules >>>>>>>>> where the parent nodes where joining with a SKIP. But I can of >>>>> should >>>>>>>> have >>>>>>>>> known this was coming. Apart of that I had a successful run last >>>>>> night. >>>>>>>>> >>>>>>>>> >>>>>>>>> On Tue, Mar 14, 2017 at 1:37 AM siddharth anand <san...@apache.org >>>>>> >>>>>>>> wrote: >>>>>>>>> >>>>>>>>> I'm going to deploy this to staging now. Fab work Bolke! >>>>>>>>> -s >>>>>>>>> >>>>>>>>> On Mon, Mar 13, 2017 at 2:16 PM, Dan Davydov < >>>>> dan.davy...@airbnb.com >>>>>> . >>>>>>>>> invalid >>>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> I'll test this on staging as soon as I get a chance (the testing >>>>> is >>>>>>>>>> non-blocking on the rc5). Bolke very much in particular :). >>>>>>>>>> >>>>>>>>>> On Mon, Mar 13, 2017 at 10:46 AM, Jeremiah Lowin < >>>>>> jlo...@apache.org> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> +1 (binding) extremely impressed by the work and diligence all >>>>>>>>>> contributors >>>>>>>>>>> have put in to getting these blockers fixed, Bolke in >>>>> particular. >>>>>>>>>>> >>>>>>>>>>> On Mon, Mar 13, 2017 at 1:07 AM Arthur Wiedmer < >>>>>> art...@apache.org> >>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> +1 (binding) >>>>>>>>>>>> >>>>>>>>>>>> Thanks again for steering us through Bolke. >>>>>>>>>>>> >>>>>>>>>>>> Best, >>>>>>>>>>>> Arthur >>>>>>>>>>>> >>>>>>>>>>>> On Sun, Mar 12, 2017 at 9:59 PM, Bolke de Bruin < >>>>>>> bdbr...@gmail.com >>>>>>>>> >>>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Dear All, >>>>>>>>>>>>> >>>>>>>>>>>>> Finally, I have been able to make the FIFTH RELEASE >>>>> CANDIDATE >>>>>>> of >>>>>>>>>>> Airflow >>>>>>>>>>>>> 1.8.0 available at: https://dist.apache.org/repos/ >>>>>>>>>>>>> dist/dev/incubator/airflow/ <https://dist.apache.org/ >>>>>>>>>>>>> repos/dist/dev/incubator/airflow/> , public keys are >>>>>> available >>>>>>>> at >>>>>>>>>>>>> https://dist.apache.org/repos/dist/release/incubator/ >>>>>> airflow/ >>>>>>> < >>>>>>>>>>>>> https://dist.apache.org/repos/dist/release/incubator/ >>>>>> airflow/> >>>>>>> . >>>>>>>>> It >>>>>>>>>> is >>>>>>>>>>>>> tagged with a local version “apache.incubating” so it >>>>> allows >>>>>>>>>> upgrading >>>>>>>>>>>> from >>>>>>>>>>>>> earlier releases. >>>>>>>>>>>>> >>>>>>>>>>>>> Issues fixed since rc4: >>>>>>>>>>>>> >>>>>>>>>>>>> [AIRFLOW-900] Double trigger should not kill original task >>>>>>>> instance >>>>>>>>>>>>> [AIRFLOW-900] Fixes bugs in LocalTaskJob for double run >>>>>>>> protection >>>>>>>>>>>>> [AIRFLOW-932] Do not mark tasks removed when backfilling >>>>>>>>>>>>> [AIRFLOW-961] run onkill when SIGTERMed >>>>>>>>>>>>> [AIRFLOW-910] Use parallel task execution for backfills >>>>>>>>>>>>> [AIRFLOW-967] Wrap strings in native for py2 ldap >>>>>> compatibility >>>>>>>>>>>>> [AIRFLOW-941] Use defined parameters for psycopg2 >>>>>>>>>>>>> [AIRFLOW-719] Prevent DAGs from ending prematurely >>>>>>>>>>>>> [AIRFLOW-938] Use test for True in task_stats queries >>>>>>>>>>>>> [AIRFLOW-937] Improve performance of task_stats >>>>>>>>>>>>> [AIRFLOW-933] use ast.literal_eval rather eval because >>>>>>>>>> ast.literal_eval >>>>>>>>>>>>> does not execute input. >>>>>>>>>>>>> [AIRFLOW-919] Running tasks with no start date shouldn't >>>>>> break >>>>>>> a >>>>>>>>> DAGs >>>>>>>>>>> UI >>>>>>>>>>>>> [AIRFLOW-897] Prevent dagruns from failing with unfinished >>>>>>> tasks >>>>>>>>>>>>> [AIRFLOW-861] make pickle_info endpoint be login_required >>>>>>>>>>>>> [AIRFLOW-853] use utf8 encoding for stdout line decode >>>>>>>>>>>>> [AIRFLOW-856] Make sure execution date is set for local >>>>>> client >>>>>>>>>>>>> [AIRFLOW-830][AIRFLOW-829][AIRFLOW-88] Reduce Travis log >>>>>>>> verbosity >>>>>>>>>>>>> [AIRFLOW-794] Access DAGS_FOLDER and SQL_ALCHEMY_CONN >>>>>>> exclusively >>>>>>>>>> from >>>>>>>>>>>>> settings >>>>>>>>>>>>> [AIRFLOW-694] Fix config behaviour for empty envvar >>>>>>>>>>>>> [AIRFLOW-365] Set dag.fileloc explicitly and use for Code >>>>>> view >>>>>>>>>>>>> [AIRFLOW-931] Do not set QUEUED in TaskInstances >>>>>>>>>>>>> [AIRFLOW-899] Tasks in SCHEDULED state should be white in >>>>> the >>>>>>> UI >>>>>>>>>>> instead >>>>>>>>>>>>> of black >>>>>>>>>>>>> [AIRFLOW-895] Address Apache release incompliancies >>>>>>>>>>>>> [AIRFLOW-893][AIRFLOW-510] Fix crashing webservers when a >>>>>>> dagrun >>>>>>>>> has >>>>>>>>>> no >>>>>>>>>>>>> start date >>>>>>>>>>>>> [AIRFLOW-793] Enable compressed loading in S3ToHiveTransfer >>>>>>>>>>>>> [AIRFLOW-863] Example DAGs should have recent start dates >>>>>>>>>>>>> [AIRFLOW-869] Refactor mark success functionality >>>>>>>>>>>>> [AIRFLOW-856] Make sure execution date is set for local >>>>>> client >>>>>>>>>>>>> [AIRFLOW-814] Fix Presto*CheckOperator.__init__ >>>>>>>>>>>>> [AIRFLOW-844] Fix cgroups directory creation >>>>>>>>>>>>> >>>>>>>>>>>>> No known issues anymore. >>>>>>>>>>>>> >>>>>>>>>>>>> I would also like to raise a VOTE for releasing 1.8.0 based >>>>>> on >>>>>>>>>> release >>>>>>>>>>>>> candidate 5, i.e. just renaming release candidate 5 to >>>>> 1.8.0 >>>>>>>>> release. >>>>>>>>>>>>> >>>>>>>>>>>>> Please respond to this email by: >>>>>>>>>>>>> >>>>>>>>>>>>> +1,0,-1 with *binding* if you are a PMC member or >>>>>> *non-binding* >>>>>>>> if >>>>>>>>>> you >>>>>>>>>>>> are >>>>>>>>>>>>> not. >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks! >>>>>>>>>>>>> Bolke >>>>>>>>>>>>> >>>>>>>>>>>>> My VOTE: +1 (binding) >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> _/ >>>>>>>>> _/ Alex Van Boxel >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>> >> >>
