Confirmed that Bolke's PR above fixes the issue. Also, I agree this is not a blocker for the current airflow release, so my +1 (binding) stands. -s
On Wed, Mar 15, 2017 at 3:11 PM, Bolke de Bruin <bdbr...@gmail.com> wrote: > PR is available: https://github.com/apache/incubator-airflow/pull/2154 > > But marked for 1.8.1. > > - Bolke > > > On 15 Mar 2017, at 14:37, Bolke de Bruin <bdbr...@gmail.com> wrote: > > > > On second thought I do consider it a bug and can have a fix out pretty > quickly, but I don’t consider it a blocker. > > > > - B. > > > >> On 15 Mar 2017, at 14:21, Bolke de Bruin <bdbr...@gmail.com> wrote: > >> > >> Just to be clear: Also in 1.7.1 the DagRun was marked successful, but > its tasks continued to be scheduled. So one could also consider 1.7.1 > behaviour a bug. I am not sure here, but I think it kind of makes sense to > consider the behaviour of 1.7.1 a bug. It has been present throughout all > the 1.8 rc/beta/apha series. > >> > >> So yes it is a change in behaviour whether it is a regression or an > integrity improvement is up for discussion. Either way I don’t consider it > a blocker. > >> > >> Bolke. > >> > >>> On 15 Mar 2017, at 14:06, siddharth anand <san...@apache.org> wrote: > >>> > >>> Here's the JIRA : > >>> https://issues.apache.org/jira/browse/AIRFLOW-989 > >>> > >>> I confirmed it is a regression from 1.7.1.3, which I installed via pip > and > >>> tested against the same DAG in the JIRA. > >>> > >>> The issue occurs if a leaf / last / terminal downstream task is not > >>> cleared. You won't see this issue if you clear the entire DAG Run or > clear > >>> a task and all of its downstream tasks. If you truly want to only > clear and > >>> rerun a task, but not its downstream tasks, you can use the CLI to > execute > >>> a specific task (e.g. vial airflow run). > >>> > >>> This is a change in behavior -- if we do go ahead with the release, > then > >>> this JIRA should be in a list of JIRAs of known issues related to the > new > >>> version. > >>> -s > >>> > >>> On Wed, Mar 15, 2017 at 9:17 AM, Chris Riccomini < > criccom...@apache.org> > >>> wrote: > >>> > >>>> @Sid, does this happen if you clear downstream as well? > >>>> > >>>> On Wed, Mar 15, 2017 at 9:04 AM, Chris Riccomini < > criccom...@apache.org> > >>>> wrote: > >>>> > >>>>> Has anyone been able to reproduce Sid's issue? > >>>>> > >>>>> On Tue, Mar 14, 2017 at 11:17 PM, Bolke de Bruin <bdbr...@gmail.com> > >>>>> wrote: > >>>>> > >>>>>> That is not an airflow error, but a Kerberos error. Try executing > the > >>>>>> kinit command on the command line by yourself. > >>>>>> > >>>>>> Bolke > >>>>>> > >>>>>> Sent from my iPhone > >>>>>> > >>>>>>> On 14 Mar 2017, at 23:11, Ruslan Dautkhanov <dautkha...@gmail.com> > >>>>>> wrote: > >>>>>>> > >>>>>>> `airflow kerberos` is broken in 1.8-rc5 > >>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-987 > >>>>>>> Hopefully fix can be part of the 1.8 release. > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> -- > >>>>>>> Ruslan Dautkhanov > >>>>>>> > >>>>>>>> On Tue, Mar 14, 2017 at 6:19 PM, siddharth anand < > san...@apache.org> > >>>>>> wrote: > >>>>>>>> > >>>>>>>> FYI, > >>>>>>>> I've just hit a major bug in the release candidate related to > "clear > >>>>>> task" > >>>>>>>> behavior. > >>>>>>>> > >>>>>>>> I've been running airflow in both stage and prod since yesterday > on > >>>>>> rc5 and > >>>>>>>> have reproduced this in both environments. I will file a JIRA for > >>>> this > >>>>>>>> tonight, but wanted to send a note over email as well. > >>>>>>>> > >>>>>>>> In my example, I have a 2 task DAG. For a given DAG run that has > >>>>>> completed > >>>>>>>> successfully, if I > >>>>>>>> 1) clear task2 (leaf task in this case), the previously-successful > >>>> DAG > >>>>>> Run > >>>>>>>> goes back to Running, requeues, and executes the task > successfully. > >>>>>> The DAG > >>>>>>>> Run the returns from Running to Success. > >>>>>>>> 2) clear task1 (root task in this case), the previously-successful > >>>> DAG > >>>>>> Run > >>>>>>>> goes back to Running, DOES NOT requeue or execute the task at all. > >>>> The > >>>>>> DAG > >>>>>>>> Run the returns from Running to Success though it never ran the > task. > >>>>>>>> > >>>>>>>> 1) is expected and previous behavior. 2) is a regression. > >>>>>>>> > >>>>>>>> The only workaround is to use the CLI to run the task cleared. > Here > >>>> are > >>>>>>>> some images : > >>>>>>>> *After Clearing the Tasks* > >>>>>>>> https://www.dropbox.com/s/wmuxt0krwx6wurr/Screenshot% > >>>>>>>> 202017-03-14%2014.09.34.png?dl=0 > >>>>>>>> > >>>>>>>> *After DAG Runs return to Success* > >>>>>>>> https://www.dropbox.com/s/qop933rzgdzchpd/Screenshot% > >>>>>>>> 202017-03-14%2014.09.49.png?dl=0 > >>>>>>>> > >>>>>>>> This is a major regression because it will force everyone to use > the > >>>>>> CLI > >>>>>>>> for things that they would normally use the UI for. > >>>>>>>> > >>>>>>>> -s > >>>>>>>> > >>>>>>>> > >>>>>>>> -s > >>>>>>>> > >>>>>>>> > >>>>>>>>> On Tue, Mar 14, 2017 at 1:32 PM, Daniel Huang <dxhu...@gmail.com > > > >>>>>> wrote: > >>>>>>>>> > >>>>>>>>> +1 (non-binding)! > >>>>>>>>> > >>>>>>>>> On Tue, Mar 14, 2017 at 11:35 AM, siddharth anand < > >>>> san...@apache.org> > >>>>>>>>> wrote: > >>>>>>>>> > >>>>>>>>>> +1 (binding) > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> On Tue, Mar 14, 2017 at 8:42 AM, Maxime Beauchemin < > >>>>>>>>>> maximebeauche...@gmail.com> wrote: > >>>>>>>>>> > >>>>>>>>>>> +1 (binding) > >>>>>>>>>>> > >>>>>>>>>>> On Tue, Mar 14, 2017 at 3:59 AM, Alex Van Boxel < > a...@vanboxel.be > >>>>> > >>>>>>>>>> wrote: > >>>>>>>>>>> > >>>>>>>>>>>> +1 (binding) > >>>>>>>>>>>> > >>>>>>>>>>>> Note: we had to revert all our ONE_SUCCESS with ALL_SUCCESS > >>>> trigger > >>>>>>>>>> rules > >>>>>>>>>>>> where the parent nodes where joining with a SKIP. But I can of > >>>>>>>> should > >>>>>>>>>>> have > >>>>>>>>>>>> known this was coming. Apart of that I had a successful run > last > >>>>>>>>> night. > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> On Tue, Mar 14, 2017 at 1:37 AM siddharth anand < > >>>> san...@apache.org > >>>>>>>>> > >>>>>>>>>>> wrote: > >>>>>>>>>>>> > >>>>>>>>>>>> I'm going to deploy this to staging now. Fab work Bolke! > >>>>>>>>>>>> -s > >>>>>>>>>>>> > >>>>>>>>>>>> On Mon, Mar 13, 2017 at 2:16 PM, Dan Davydov < > >>>>>>>> dan.davy...@airbnb.com > >>>>>>>>> . > >>>>>>>>>>>> invalid > >>>>>>>>>>>>> wrote: > >>>>>>>>>>>> > >>>>>>>>>>>>> I'll test this on staging as soon as I get a chance (the > testing > >>>>>>>> is > >>>>>>>>>>>>> non-blocking on the rc5). Bolke very much in particular :). > >>>>>>>>>>>>> > >>>>>>>>>>>>> On Mon, Mar 13, 2017 at 10:46 AM, Jeremiah Lowin < > >>>>>>>>> jlo...@apache.org> > >>>>>>>>>>>>> wrote: > >>>>>>>>>>>>> > >>>>>>>>>>>>>> +1 (binding) extremely impressed by the work and diligence > all > >>>>>>>>>>>>> contributors > >>>>>>>>>>>>>> have put in to getting these blockers fixed, Bolke in > >>>>>>>> particular. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> On Mon, Mar 13, 2017 at 1:07 AM Arthur Wiedmer < > >>>>>>>>> art...@apache.org> > >>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> +1 (binding) > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Thanks again for steering us through Bolke. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Best, > >>>>>>>>>>>>>>> Arthur > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> On Sun, Mar 12, 2017 at 9:59 PM, Bolke de Bruin < > >>>>>>>>>> bdbr...@gmail.com > >>>>>>>>>>>> > >>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Dear All, > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Finally, I have been able to make the FIFTH RELEASE > >>>>>>>> CANDIDATE > >>>>>>>>>> of > >>>>>>>>>>>>>> Airflow > >>>>>>>>>>>>>>>> 1.8.0 available at: https://dist.apache.org/repos/ > >>>>>>>>>>>>>>>> dist/dev/incubator/airflow/ <https://dist.apache.org/ > >>>>>>>>>>>>>>>> repos/dist/dev/incubator/airflow/> , public keys are > >>>>>>>>> available > >>>>>>>>>>> at > >>>>>>>>>>>>>>>> https://dist.apache.org/repos/dist/release/incubator/ > >>>>>>>>> airflow/ > >>>>>>>>>> < > >>>>>>>>>>>>>>>> https://dist.apache.org/repos/dist/release/incubator/ > >>>>>>>>> airflow/> > >>>>>>>>>> . > >>>>>>>>>>>> It > >>>>>>>>>>>>> is > >>>>>>>>>>>>>>>> tagged with a local version “apache.incubating” so it > >>>>>>>> allows > >>>>>>>>>>>>> upgrading > >>>>>>>>>>>>>>> from > >>>>>>>>>>>>>>>> earlier releases. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Issues fixed since rc4: > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> [AIRFLOW-900] Double trigger should not kill original task > >>>>>>>>>>> instance > >>>>>>>>>>>>>>>> [AIRFLOW-900] Fixes bugs in LocalTaskJob for double run > >>>>>>>>>>> protection > >>>>>>>>>>>>>>>> [AIRFLOW-932] Do not mark tasks removed when backfilling > >>>>>>>>>>>>>>>> [AIRFLOW-961] run onkill when SIGTERMed > >>>>>>>>>>>>>>>> [AIRFLOW-910] Use parallel task execution for backfills > >>>>>>>>>>>>>>>> [AIRFLOW-967] Wrap strings in native for py2 ldap > >>>>>>>>> compatibility > >>>>>>>>>>>>>>>> [AIRFLOW-941] Use defined parameters for psycopg2 > >>>>>>>>>>>>>>>> [AIRFLOW-719] Prevent DAGs from ending prematurely > >>>>>>>>>>>>>>>> [AIRFLOW-938] Use test for True in task_stats queries > >>>>>>>>>>>>>>>> [AIRFLOW-937] Improve performance of task_stats > >>>>>>>>>>>>>>>> [AIRFLOW-933] use ast.literal_eval rather eval because > >>>>>>>>>>>>> ast.literal_eval > >>>>>>>>>>>>>>>> does not execute input. > >>>>>>>>>>>>>>>> [AIRFLOW-919] Running tasks with no start date shouldn't > >>>>>>>>> break > >>>>>>>>>> a > >>>>>>>>>>>> DAGs > >>>>>>>>>>>>>> UI > >>>>>>>>>>>>>>>> [AIRFLOW-897] Prevent dagruns from failing with unfinished > >>>>>>>>>> tasks > >>>>>>>>>>>>>>>> [AIRFLOW-861] make pickle_info endpoint be login_required > >>>>>>>>>>>>>>>> [AIRFLOW-853] use utf8 encoding for stdout line decode > >>>>>>>>>>>>>>>> [AIRFLOW-856] Make sure execution date is set for local > >>>>>>>>> client > >>>>>>>>>>>>>>>> [AIRFLOW-830][AIRFLOW-829][AIRFLOW-88] Reduce Travis log > >>>>>>>>>>> verbosity > >>>>>>>>>>>>>>>> [AIRFLOW-794] Access DAGS_FOLDER and SQL_ALCHEMY_CONN > >>>>>>>>>> exclusively > >>>>>>>>>>>>> from > >>>>>>>>>>>>>>>> settings > >>>>>>>>>>>>>>>> [AIRFLOW-694] Fix config behaviour for empty envvar > >>>>>>>>>>>>>>>> [AIRFLOW-365] Set dag.fileloc explicitly and use for Code > >>>>>>>>> view > >>>>>>>>>>>>>>>> [AIRFLOW-931] Do not set QUEUED in TaskInstances > >>>>>>>>>>>>>>>> [AIRFLOW-899] Tasks in SCHEDULED state should be white in > >>>>>>>> the > >>>>>>>>>> UI > >>>>>>>>>>>>>> instead > >>>>>>>>>>>>>>>> of black > >>>>>>>>>>>>>>>> [AIRFLOW-895] Address Apache release incompliancies > >>>>>>>>>>>>>>>> [AIRFLOW-893][AIRFLOW-510] Fix crashing webservers when a > >>>>>>>>>> dagrun > >>>>>>>>>>>> has > >>>>>>>>>>>>> no > >>>>>>>>>>>>>>>> start date > >>>>>>>>>>>>>>>> [AIRFLOW-793] Enable compressed loading in > S3ToHiveTransfer > >>>>>>>>>>>>>>>> [AIRFLOW-863] Example DAGs should have recent start dates > >>>>>>>>>>>>>>>> [AIRFLOW-869] Refactor mark success functionality > >>>>>>>>>>>>>>>> [AIRFLOW-856] Make sure execution date is set for local > >>>>>>>>> client > >>>>>>>>>>>>>>>> [AIRFLOW-814] Fix Presto*CheckOperator.__init__ > >>>>>>>>>>>>>>>> [AIRFLOW-844] Fix cgroups directory creation > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> No known issues anymore. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> I would also like to raise a VOTE for releasing 1.8.0 > based > >>>>>>>>> on > >>>>>>>>>>>>> release > >>>>>>>>>>>>>>>> candidate 5, i.e. just renaming release candidate 5 to > >>>>>>>> 1.8.0 > >>>>>>>>>>>> release. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Please respond to this email by: > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> +1,0,-1 with *binding* if you are a PMC member or > >>>>>>>>> *non-binding* > >>>>>>>>>>> if > >>>>>>>>>>>>> you > >>>>>>>>>>>>>>> are > >>>>>>>>>>>>>>>> not. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Thanks! > >>>>>>>>>>>>>>>> Bolke > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> My VOTE: +1 (binding) > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> -- > >>>>>>>>>>>> _/ > >>>>>>>>>>>> _/ Alex Van Boxel > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>> > >>>>> > >>>>> > >>>> > >> > > > >
