Re: scheduler running on multiple nodes

2017-02-23 Thread siddharth anand
I did run 2 or more schedulers with Local Executors up until mid last year. There have been enough changes to the code and feature additions that I don't think this is a recommended practice at this point. Also, there is not a lot of synchronization in the scheduler to ensure this will work. -s

Re: [RESULT] [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc4

2017-02-23 Thread siddharth anand
IMHO, a DAG run without a start date is non-sensical but is not enforced That said, our UI allows for the manual creation of DAG Runs without a start date as shown in the images below: - https://www.dropbox.com/s/3sxcqh04eztpl7p/Screenshot% 202017-02-22%2016.00.40.png?dl=0

Re: Meetup featuring an Airflow talk tomorrow

2017-02-23 Thread siddharth anand
Nice! https://twitter.com/ApacheAirflow/status/834945481440546816 Do please share slides and video so we can post both on via twitter & wiki. -s On Wed, Feb 22, 2017 at 3:00 PM, Maxime Beauchemin < maximebeauche...@gmail.com> wrote: > Hi, > > Just wanted to let you know that Arthur (one of our

Effort in doing a release

2017-02-23 Thread Bolke de Bruin
(Migrating this from the vote result thread). The git-fu part is quite limited. With the branching for release (test/stable) and then tagging it has become quite easy to get right thing is. In the beginning it was often syncing to master, now it is cherry-picking from master and making sure it

Re: [RESULT] [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc4

2017-02-23 Thread Maxime Beauchemin
Airbnb can commit to run the upcoming RC against our production as soon as it comes out. Max On Thu, Feb 23, 2017 at 2:01 PM, Chris Riccomini wrote: > Sounds good! :) > > On Thu, Feb 23, 2017 at 1:58 PM, Bolke de Bruin wrote: > > > I agree with the

Re: [RESULT] [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc4

2017-02-23 Thread Bolke de Bruin
I’m not particularly against another RC. On the IPMC there were some issues mentioned regarding licensing, which probably are blocking as well (eg. no LICENSE etc in the tar ball). I found some HighCharts left overs as well, while addressing the licensing issues. PR here:

Re: [RESULT] [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc4

2017-02-23 Thread Maxime Beauchemin
IMHO 1 is a blocker. The other issues could have been mitigated but 1 is a dealbreaker for Airbnb. We have lots of large, critical DAGs that would be in a standstill because of individual task failures, where in reality a lot of progress can be made. Airflow should really do as much work as

Re: [RESULT] [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc4

2017-02-23 Thread Bolke de Bruin
Gotcha. Will be patient. Good luck. Bolke > On 23 Feb 2017, at 21:12, Dan Davydov wrote: > > Here is an example for 1, you can see that there are some white tasks that > should have been run. I don't have time to create a skeleton DAG at the > moment

Re: [RESULT] [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc4

2017-02-23 Thread Dan Davydov
Some more issues found by our users in addition to the one Alex reported and the UI issue when a dagrun doesn't have a start date: 1. If a task fails it fails the whole dagrun immediately fails, this is a very large change to how control flow works as the rest of the tasks in the DAG are not run