Re: Upgrading to 1.7.1.2...

2016-08-01 Thread siddharth anand
It would be good to split release notes into separate sections of fixes and features. I recall that we generate the release notes from a git log command. We might want to make that script a little smarter - e.g. separate commits into the features and fixes buckets. We are committing a lot of fixes

DAG status still running when all its tasks are complete

2016-08-01 Thread Nadeem Ahmed Nazeer
Hello, I am facing a situation with Airflow where it doesn't flag the DAG's as success even though all of the tasks in that DAG are complete. I have a BranchPythonOperator which forks into running all downstream tasks or just a single task (dummy operator as an endpoint) depending if files exists

Re: MySQL operator that returns rows

2016-08-01 Thread Tamara Mendt
Hello Max, Thanks a lot for the thorough and nicely explained answer. I get your point of tasks being idempotent. Though I had read that tasks are not expected to move data between each other, I tried using XCom for this anyway because my use case was just to run a query against a DB replica to ge

Re: MySQL operator that returns rows

2016-08-01 Thread Maxime Beauchemin
Operators are meant to define an atomic, self-contained task. Meaning once the operator is done, the worker is done and no state persist (XCom are the exception to this, but let's ignore this for now). Ideally a task (the object returned when instantiating the operator) is idempotent and shouldn't

MySQL operator that returns rows

2016-08-01 Thread Tamara Mendt
Hello, I would like to use an operator which executes an SQL command and passes the result of this command to the next downstream operator. However, I noticed that the MySqlOperator only executes the command, without fetching the results. The solution I came up with was to create a new operator c