Re: scheduler running on multiple nodes

I did run 2 or more schedulers with Local Executors up until mid last year. There have been enough changes to the code and feature additions that I don't think this is a recommended practice at this point. Also, there is not a lot of synchronization in the scheduler to ensure this will work. -s

Re: [RESULT] [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc4

IMHO, a DAG run without a start date is non-sensical but is not enforced That said, our UI allows for the manual creation of DAG Runs without a start date as shown in the images below: - https://www.dropbox.com/s/3sxcqh04eztpl7p/Screenshot% 202017-02-22%2016.00.40.png?dl=0

Re: Meetup featuring an Airflow talk tomorrow

Nice! https://twitter.com/ApacheAirflow/status/834945481440546816 Do please share slides and video so we can post both on via twitter & wiki. -s On Wed, Feb 22, 2017 at 3:00 PM, Maxime Beauchemin < maximebeauche...@gmail.com> wrote: > Hi, > > Just wanted to let you know that Arthur (one of our A

Effort in doing a release

(Migrating this from the vote result thread). The git-fu part is quite limited. With the branching for release (test/stable) and then tagging it has become quite easy to get right thing is. In the beginning it was often syncing to master, now it is cherry-picking from master and making sure it

Re: [RESULT] [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc4

Airbnb can commit to run the upcoming RC against our production as soon as it comes out. Max On Thu, Feb 23, 2017 at 2:01 PM, Chris Riccomini wrote: > Sounds good! :) > > On Thu, Feb 23, 2017 at 1:58 PM, Bolke de Bruin wrote: > > > I agree with the assessment. Let’s not rush and make sure both

Re: [RESULT] [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc4

Agreed! On Feb 23, 2017 2:01 PM, "Chris Riccomini" wrote: > Sounds good! :) > > On Thu, Feb 23, 2017 at 1:58 PM, Bolke de Bruin wrote: > > > I agree with the assessment. Let’s not rush and make sure both issues are > > properly fixed or understood (in case of Alex’) > > > > I will leave the vot

Re: [RESULT] [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc4

Sounds good! :) On Thu, Feb 23, 2017 at 1:58 PM, Bolke de Bruin wrote: > I agree with the assessment. Let’s not rush and make sure both issues are > properly fixed or understood (in case of Alex’) > > I will leave the vote at the IPMC open for another 10 hours or so, just to > get some more revi

Re: [RESULT] [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc4

I agree with the assessment. Let’s not rush and make sure both issues are properly fixed or understood (in case of Alex’) I will leave the vote at the IPMC open for another 10 hours or so, just to get some more reviews hopefully on the licensing part. - Bolke > On 23 Feb 2017, at 22:53, Dan Da

Re: [RESULT] [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc4

To expand on Max's point it doesn't concern me that this is a blocker for AirBnB, but it's not logical behavior and I'm sure many companies rely on the previous behavior (which I would say is the logically correct one). We are already running a fork of the release internally so we are unaffected, I

Re: [RESULT] [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc4

I’m not particularly against another RC. On the IPMC there were some issues mentioned regarding licensing, which probably are blocking as well (eg. no LICENSE etc in the tar ball). I found some HighCharts left overs as well, while addressing the licensing issues. PR here: https://github.com/apa

Re: [RESULT] [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc4

IMHO 1 is a blocker. The other issues could have been mitigated but 1 is a dealbreaker for Airbnb. We have lots of large, critical DAGs that would be in a standstill because of individual task failures, where in reality a lot of progress can be made. Airflow should really do as much work as possib

Re: [RESULT] [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc4

My 2c: I observed both #1 and #2 in Dan's list. I figured y'all had had a discussion about the change in behavior. :) In any case, I made my peace with it, and we've been running happily in production for weeks now, so I personally don't see it as a blocker. Obviously, if it's an issue for you guy

Re: [RESULT] [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc4

Gotcha. Will be patient. Good luck. Bolke > On 23 Feb 2017, at 21:12, Dan Davydov wrote: > > Here is an example for 1, you can see that there are some white tasks that > should have been run. I don't have time to create a skeleton DAG at the > moment unfortunately because of release-related f

Re: [RESULT] [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc4

Here is the DAG: http://imgur.com/a/zXXsS On Thu, Feb 23, 2017 at 12:18 PM, Arthur Wiedmer wrote: > Dan, > > Inline images get stripped by the mailing server. You will have to upload > to imgur or something. > > Best > Arthur > > On Feb 23, 2017 12:13 PM, "Dan Davydov" > wrote: > > > Here is an

Re: [RESULT] [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc4

Dan, Inline images get stripped by the mailing server. You will have to upload to imgur or something. Best Arthur On Feb 23, 2017 12:13 PM, "Dan Davydov" wrote: > Here is an example for 1, you can see that there are some white tasks that > should have been run. I don't have time to create a sk

Re: [RESULT] [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc4

Here is an example for 1, you can see that there are some white tasks that should have been run. I don't have time to create a skeleton DAG at the moment unfortunately because of release-related firefighting. Will hopefully post back here later once firefighting is done. [image: Inline image 1] On

Re: [RESULT] [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc4

Hey Dan, Alex, Indeed #1 seems serious, specifically the the second part - skipping the root task (root task of the whole DAG?). Do you have a skeleton DAG that exposes the issue? Is there a root cause analysis? When was the issue introduced? On the the issue Alex mentioned, we don’t see that a

Re: [RESULT] [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc4

Some more issues found by our users in addition to the one Alex reported and the UI issue when a dagrun doesn't have a start date: 1. If a task fails it fails the whole dagrun immediately fails, this is a very large change to how control flow works as the rest of the tasks in the DAG are not run (e