dev@airflow.incubator.apache.org

thanks for sharing, was a good read! go Airflow go! On Tue, Feb 28, 2017 at 1:00 PM, Maxime Beauchemin < maximebeauche...@gmail.com> wrote: > Jointly publishing this short Q&A with Taylor D. Edmiston of Atronomer.io > > https://medium.com/the-astronomer-journey/airflow-and-the-future-of-data- > e

Re: Canceling a DAG run

Changing the state of a dag run in the UI will only stop the scheduler from evaluating that dag run, it does not sigterm or anything like that. "Clearing" running task instances (UI/CLI) will result in failing the task instances. Also as of 1.8.x if the state of the task in changed externally from

Re: [RESULT] [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc4

I wasn't able to reproduce the described behavior but replied in the JIRA (let's discuss there) https://issues.apache.org/jira/browse/AIRFLOW-920 On Tue, Feb 28, 2017 at 3:56 AM, Bolke de Bruin wrote: > Gotcha: > > It works, but then slightly different. > > If you added the subdag, do not zoom i

Task instance enqueued but never runs, holds up next run of the DAG

I have a large DAG (32 tasks) with concurrency=2 and max_active_runs=1. Most of the tasks also use a redshift_pool, and this is running the LocalExecutor on 1.8.0RC4. When the DAG kicks off things seem to generally function, but a few of the tasks get moved to queued status (appropriately) but the

dev@airflow.incubator.apache.org

Thanks for the share, Max! I really enjoyed working on and co-publishing the post. I know everyone is quite busy with the 1.8 release, so I especially appreciate taking the time to spread the word and think about the big picture. I'm happy to field any questions on the post or Astronomer. You c

dev@airflow.incubator.apache.org

Jointly publishing this short Q&A with Taylor D. Edmiston of Atronomer.io https://medium.com/the-astronomer-journey/airflow-and-the-future-of-data-engineering-a-q-a-266f68d956a9#.cpast6vjd

Canceling a DAG run

Hi folks, We have been using Airflow to run ETL jobs but sometimes require to cancel a DAG run. Since I could not find a way to cancel a DAG run (from CLI or UI), we are using a crude work-around that is to cancel the tasks that are running and thus failing the DAG run. What I expect (and I mi

Re: Failing jobs not properly terminated

Here's a follow-up with more information. I think my suspicion is supported by the Airflow database. It has the task marked as failed in the task_instance table, and the PID matches the dead job: airflow=> select job_id, task_id, dag_id, state, pid from task_instance where job_id=127022; job_i

Failing jobs not properly terminated

Hi folks, We're experiencing an issue where a failing DAG is not properly terminated. The log file contains a Python stack trace of the failure and then ends with: {base_task_runner.py:112} INFO - Running: ['bash', '-c', 'airflow run my_dag my_task 2017-02-22T00:00:00 --job_id 127022 --ra

Re: [RESULT] [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc4

Gotcha: It works, but then slightly different. If you added the subdag, do not zoom in, but click on the subdag in the main dag. Use mark success there. It will then allow you to mark all tasks successful that are part of the subdag. Do we still consider this a blocker? Imho, no as a workaro