Re: [VOTE] Release Airflow 1.8.1 RC2

2017-05-03 Thread Jeremiah Lowin
+1 (binding) On Mon, May 1, 2017 at 1:58 PM Chris Riccomini wrote: > Dear All, > > _WARN: The package version for this RC is 1.8.1 (does not include RC2 in > version number). As such, any future 1.8.1 installatinos will have to be > force installed. PIP will not be able to distinguish between RC

Re: Discussion on Airflow 1.8.1 RC2

2017-05-03 Thread Dan Davydov
cc Alex and Rui who were working on fixes, I'm not sure if their commits got in before 1.8.1. On Wed, May 3, 2017 at 1:09 PM, Bolke de Bruin wrote: > Hi Dan, > > (Thread renamed to make sure it does not clash, dev@ now added) > > It surprises me that you found regression from 1.8.0 to 1.8.1 as 1

Discussion on Airflow 1.8.1 RC2

2017-05-03 Thread Bolke de Bruin
Hi Dan, (Thread renamed to make sure it does not clash, dev@ now added) It surprises me that you found regression from 1.8.0 to 1.8.1 as 1.8.1 is very much focused on bug fixes. Were the regressions shared yet? The whole 1.8.X release will be bug fix focused (per release management) and minor

Re: [VOTE] Release Airflow 1.8.1 RC2

2017-05-03 Thread Jakob Homan
+1 (binding), but with a concern: 11:58 $ cat setup.cfg | grep -i maxime author = Maxime Beauchemin author-email = maximebeauche...@gmail.com The setup.cfg still has Maxime listed as the author. I called this out on the last release as needing to be fixed. It has been fixed on trunk (AIRFLOW-99

Re: Custom DAG name?

2017-05-03 Thread EKC (Erik Cederstrand)
Hi Paul, Yes, I'm triggering the DAGs via "airflow trigger_dag". I somehow completely missed the "--run_id RUN_ID" command-line option. This will solve my problem very nicely. It's searchable from /admin/dagrun/. It would be nice, though, if the "Run Id" column in that list was clickable so I

Re: last task in the dag is not running

2017-05-03 Thread Maxime Beauchemin
One way to debug these "false starts" or tasks that don't even get to the point where the logging is initiated is to: 1. look at the scheduler log to get the exact command that is put in the queue for remote execution 2. copy the exact command 3. go on the worker and try to recreate the exact conte

Re: [VOTE] Release Airflow 1.8.1 RC2

2017-05-03 Thread Arthur Wiedmer
+1 binding Chris Thanks again for the hard work. Best, Arthur On Mon, May 1, 2017 at 10:58 AM, Chris Riccomini wrote: > Dear All, > > _WARN: The package version for this RC is 1.8.1 (does not include RC2 in > version number). As such, any future 1.8.1 installatinos will have to be > force inst

Re: Custom DAG name?

2017-05-03 Thread Maxime Beauchemin
You could add the run_id to the DagRun list view, would that help? Max On Wed, May 3, 2017 at 8:02 AM, Paul Zaczkiewicz wrote: > Sounds like an interesting application of airflow which may not be the best > use-case for it. > > How are you triggering these dag runs? airflow trigger_dag? If so,

Re: Custom DAG name?

2017-05-03 Thread Paul Zaczkiewicz
Sounds like an interesting application of airflow which may not be the best use-case for it. How are you triggering these dag runs? airflow trigger_dag? If so, you can find the logs for your specific RUN_ID in the ui without grepping through all logs first, but with hundreds of dag runs, it will s

Custom DAG name?

2017-05-03 Thread EKC (Erik Cederstrand)
Hi all, I have created a DAG, 'my_dag', that will always be triggered manually, and will always receive a unique ID from the command-line, specifying the entity the DAG must process. The DAG may be triggered hundreds of times a day. In the Airflow UI, the DAG will show up everywhere as 'my_da

Re: last task in the dag is not running

2017-05-03 Thread Bolke de Bruin
Hi Dmitry, Please provide more information, such as logs and the DAG definition itself. This is very little to go on unfortunately. Bolke > On 3 May 2017, at 10:22, Dmitry Smirnov wrote: > > Hi everyone, > > I'm using Airflow version 1.8.0, just upgraded from 1.7.1.3. The issue that > I'm go

last task in the dag is not running

2017-05-03 Thread Dmitry Smirnov
Hi everyone, I'm using Airflow version 1.8.0, just upgraded from 1.7.1.3. The issue that I'm going to describe started already in 1.7.1.3, I upgraded hoping it might help resolve it. I have several DAGs for which the *last* task is not moving from queued to running. These DAGs used to run fine so