Re: [RESULT] [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc4

I wasn't able to reproduce the described behavior but replied in the JIRA (let's discuss there) https://issues.apache.org/jira/browse/AIRFLOW-920 On Tue, Feb 28, 2017 at 3:56 AM, Bolke de Bruin wrote: > Gotcha: > > It works, but then slightly different. > > If you added the subdag, do not zoom i

Re: [RESULT] [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc4

Gotcha: It works, but then slightly different. If you added the subdag, do not zoom in, but click on the subdag in the main dag. Use mark success there. It will then allow you to mark all tasks successful that are part of the subdag. Do we still consider this a blocker? Imho, no as a workaro

Re: [RESULT] [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc4

rc + your patch (and a couple of our own custom ones) On Mon, Feb 27, 2017 at 2:11 PM, Bolke de Bruin wrote: > Dan > > Btw are you running with my patch for this? Or still plain rc? > > Cheers > Bolke > > Sent from my iPhone > > > On 27 Feb 2017, at 22:46, Bolke de Bruin wrote: > > > > I'll hav

Re: [RESULT] [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc4

Dan Btw are you running with my patch for this? Or still plain rc? Cheers Bolke Sent from my iPhone > On 27 Feb 2017, at 22:46, Bolke de Bruin wrote: > > I'll have a look. I verified and the code is there to take of this. > > B. > > Sent from my iPhone > >> On 27 Feb 2017, at 22:34, Dan

Re: [RESULT] [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc4

I'll have a look. I verified and the code is there to take of this. B. Sent from my iPhone > On 27 Feb 2017, at 22:34, Dan Davydov wrote: > > Repro steps: > - Create a DAG with a dummy task > - Let this DAG run for one dagrun > - Add a new subdag operator that contains a dummy operator to th

Re: [RESULT] [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc4

Repro steps: - Create a DAG with a dummy task - Let this DAG run for one dagrun - Add a new subdag operator that contains a dummy operator to this DAG that has depends_on_past set to true - click on the white square for the new subdag operator in the DAGs first dagrun - Click "Zoom into subdag" (ta

Re: [RESULT] [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc4

Dan Can you elaborate on 2, cause I thought I specifically took care of that. Cheers Bolke Sent from my iPhone > On 27 Feb 2017, at 20:27, Dan Davydov wrote: > > I created https://issues.apache.org/jira/browse/AIRFLOW-921 to track the > pending issues. > > There are two more issues we found

Re: [RESULT] [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc4

I created https://issues.apache.org/jira/browse/AIRFLOW-921 to track the pending issues. There are two more issues we found which I included there: 1. Task instances that have their state manually set to running make the UI for their DAG unable to parse 2. Mark success doesn't work for non existen

Re: [RESULT] [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc4

Hey Max It is massive for sure. Sorry about that ;-). However it is not as massive as you might deduct from a first view. 0) run tasks concurrently across dag runs 1) ordering of the tasks was added to the loop. 2) calculating of deadlocks, running tasks, tasks to run was corrected, 3) relying

Re: [RESULT] [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc4

This PR is pretty massive and complex! It looks like solid work but let's be really careful around testing and rolling this out. This may be out of scope for this PR, but wanted to discuss the idea of using the scheduler's logic to perform backfills. It'd be nice to have that logic in one place, t

Re: [RESULT] [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc4

I have worked in the Backfill issue also in collaboration with Jeremiah. The refactor to use dag runs in backfills caused a regression in task execution performance as dag runs were executed sequentially. Next to that, the backfills were non deterministic due to the random execution of tasks, caus

Re: [RESULT] [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc4

I think the skipped task issue mentioned is the same issue described in AIRFLOW-872 . I had a DAG that was consistently hitting it. I believe the skipped task has to have at least one downstream task for this to occur (e.g. LatestOnlyOperator >> Du

Re: [RESULT] [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc4

I think what I observed was doesn't every time otherwise we would see it every time. I'll see if it happens this night again. On Sat, Feb 25, 2017 at 1:24 PM Jeremiah Lowin wrote: > Interesting if this is related to what I was seeing -- but to be clear the > error I observed is non-deterministic

Re: [RESULT] [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc4

Interesting if this is related to what I was seeing -- but to be clear the error I observed is non-deterministic and doesn't happen every time (obviously, because otherwise there would be no passing Travis runs). Is that the case for what you're describing, Dan/Alex? On Sat, Feb 25, 2017 at 4:13 A

Re: [RESULT] [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc4

About: Skipped tasks potentially cause a dagrun to be marked failure/success prematurely. Isn't that related to the discussion I had with Max about the ONE_SUCCESS trigger? When skipping tasks for now you need to put ONE_SUCCESS. I had kind of a fix but it was rejected because it changed behaviour

Re: [RESULT] [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc4

Not trying to muddy the waters, but the observation of Jeremiah (non deterministic outcomes) might have to do something with #3. I didn’t dive in deeper, yet. == ERROR: test_backfill_examples (tests.BackfillJobTest)

Re: [RESULT] [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc4

Hi Dan, - Backfill indeed runs only one dagrun at the time, see line 1755 of jobs.py. I’ll think about how to fix this over the weekend (I think it was my change that introduced this). Suggestions always welcome. Depending the impact it is a blocker or not. We don’t often use backfills and defi

Re: [RESULT] [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc4

Update on old pending issues: - Black Squares in UI: Fix merged - Double Trigger Issue That Alex G Mentioned: Alex has a PR in flight New Issues: - Backfill seems to be having issues (only running one dagrun at a time), we are still investigating - might be a blocker - High DB Load (~8x more than

Re: [RESULT] [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc4

Right now, we see some jobs getting run twice with the second run happening exactly one hour later. We can't isolate why. The current doublerun job prevention kills both jobs as it happens in the heartbeat and they run concurrently and is overeager in killing both. This PR changes it so the job wil

Re: [RESULT] [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc4

IMHO, a DAG run without a start date is non-sensical but is not enforced That said, our UI allows for the manual creation of DAG Runs without a start date as shown in the images below: - https://www.dropbox.com/s/3sxcqh04eztpl7p/Screenshot% 202017-02-22%2016.00.40.png?dl=0

Re: [RESULT] [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc4

Airbnb can commit to run the upcoming RC against our production as soon as it comes out. Max On Thu, Feb 23, 2017 at 2:01 PM, Chris Riccomini wrote: > Sounds good! :) > > On Thu, Feb 23, 2017 at 1:58 PM, Bolke de Bruin wrote: > > > I agree with the assessment. Let’s not rush and make sure both

Re: [RESULT] [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc4

Agreed! On Feb 23, 2017 2:01 PM, "Chris Riccomini" wrote: > Sounds good! :) > > On Thu, Feb 23, 2017 at 1:58 PM, Bolke de Bruin wrote: > > > I agree with the assessment. Let’s not rush and make sure both issues are > > properly fixed or understood (in case of Alex’) > > > > I will leave the vot

Re: [RESULT] [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc4

Sounds good! :) On Thu, Feb 23, 2017 at 1:58 PM, Bolke de Bruin wrote: > I agree with the assessment. Let’s not rush and make sure both issues are > properly fixed or understood (in case of Alex’) > > I will leave the vote at the IPMC open for another 10 hours or so, just to > get some more revi

Re: [RESULT] [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc4

I agree with the assessment. Let’s not rush and make sure both issues are properly fixed or understood (in case of Alex’) I will leave the vote at the IPMC open for another 10 hours or so, just to get some more reviews hopefully on the licensing part. - Bolke > On 23 Feb 2017, at 22:53, Dan Da

Re: [RESULT] [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc4

To expand on Max's point it doesn't concern me that this is a blocker for AirBnB, but it's not logical behavior and I'm sure many companies rely on the previous behavior (which I would say is the logically correct one). We are already running a fork of the release internally so we are unaffected, I

Re: [RESULT] [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc4

I’m not particularly against another RC. On the IPMC there were some issues mentioned regarding licensing, which probably are blocking as well (eg. no LICENSE etc in the tar ball). I found some HighCharts left overs as well, while addressing the licensing issues. PR here: https://github.com/apa

Re: [RESULT] [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc4

IMHO 1 is a blocker. The other issues could have been mitigated but 1 is a dealbreaker for Airbnb. We have lots of large, critical DAGs that would be in a standstill because of individual task failures, where in reality a lot of progress can be made. Airflow should really do as much work as possib

Re: [RESULT] [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc4

My 2c: I observed both #1 and #2 in Dan's list. I figured y'all had had a discussion about the change in behavior. :) In any case, I made my peace with it, and we've been running happily in production for weeks now, so I personally don't see it as a blocker. Obviously, if it's an issue for you guy

Re: [RESULT] [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc4

Gotcha. Will be patient. Good luck. Bolke > On 23 Feb 2017, at 21:12, Dan Davydov wrote: > > Here is an example for 1, you can see that there are some white tasks that > should have been run. I don't have time to create a skeleton DAG at the > moment unfortunately because of release-related f

Re: [RESULT] [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc4

Here is the DAG: http://imgur.com/a/zXXsS On Thu, Feb 23, 2017 at 12:18 PM, Arthur Wiedmer wrote: > Dan, > > Inline images get stripped by the mailing server. You will have to upload > to imgur or something. > > Best > Arthur > > On Feb 23, 2017 12:13 PM, "Dan Davydov" > wrote: > > > Here is an

Re: [RESULT] [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc4

Dan, Inline images get stripped by the mailing server. You will have to upload to imgur or something. Best Arthur On Feb 23, 2017 12:13 PM, "Dan Davydov" wrote: > Here is an example for 1, you can see that there are some white tasks that > should have been run. I don't have time to create a sk

Re: [RESULT] [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc4

Here is an example for 1, you can see that there are some white tasks that should have been run. I don't have time to create a skeleton DAG at the moment unfortunately because of release-related firefighting. Will hopefully post back here later once firefighting is done. [image: Inline image 1] On

Re: [RESULT] [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc4

Hey Dan, Alex, Indeed #1 seems serious, specifically the the second part - skipping the root task (root task of the whole DAG?). Do you have a skeleton DAG that exposes the issue? Is there a root cause analysis? When was the issue introduced? On the the issue Alex mentioned, we don’t see that a

Re: [RESULT] [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc4

Some more issues found by our users in addition to the one Alex reported and the UI issue when a dagrun doesn't have a start date: 1. If a task fails it fails the whole dagrun immediately fails, this is a very large change to how control flow works as the rest of the tasks in the DAG are not run (e

Re: [RESULT] [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc4

I have some concern that this change https://github.com/apache/incubator-airflow/pull/1939 [AIRFLOW-679] may be having issues because we are seeing lots of double triggers of tasks and tasks being killed as a result. On Wed, Feb 22, 2017 4:35 PM, Dan Davydov dan.davy...@airbnb.com.INVALID w

Re: [RESULT] [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc4

Bumping the thread so another user can comment. On Wed, Feb 22, 2017 at 3:12 PM, Maxime Beauchemin < maximebeauche...@gmail.com> wrote: > What I meant to ask is "how much engineering effort it takes to bake a > single RC?", I guess it depends on how much git-fu is necessary plus some > overhead c

Re: [RESULT] [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc4

What I meant to ask is "how much engineering effort it takes to bake a single RC?", I guess it depends on how much git-fu is necessary plus some overhead cost of doing the series of actions/commands/emails/jira. I can volunteer for 1.8.1 (hopefully I can get do it along another Airbnb engineer/vol

Re: [RESULT] [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc4

I thought you volunteered to baby sit 1.8.1 Chris ;-)? Sent from my iPhone > On 22 Feb 2017, at 23:31, Chris Riccomini wrote: > > I'm +1 for doing a 1.8.1 fast follow-on > > On Wed, Feb 22, 2017 at 2:26 PM, Maxime Beauchemin < > maximebeauche...@gmail.com> wrote: > >> Our database may have ed

Re: [RESULT] [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc4

Well now it took me almost 8 weeks :-). But if we are a bit faster and don't wait 9 months to accumulate commits I think it can be much faster. Taking betas and rcs into consideration and voting maybe 2-3 weeks. Voting needs to happen twice at the moment, but i would like to kick off a gradua

Re: [RESULT] [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc4

I'm +1 for doing a 1.8.1 fast follow-on On Wed, Feb 22, 2017 at 2:26 PM, Maxime Beauchemin < maximebeauche...@gmail.com> wrote: > Our database may have edge cases that could be associated with running any > previous version that may or may not have been part of an official release. > > Let's see

Re: [RESULT] [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc4

Our database may have edge cases that could be associated with running any previous version that may or may not have been part of an official release. Let's see if anyone else reports the issue. If no one does, one option is to release 1.8.0 as is with a comment in the release notes, and have a fu

Re: [RESULT] [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc4

So it is a database integrity issue? Afaik a start_date should always be set for a DagRun (create_dagrun) does so I didn't check the code though. Sent from my iPhone > On 22 Feb 2017, at 22:19, Dan Davydov wrote: > > Should clarify this occurs when a dagrun does not have a start date, not a

Re: [RESULT] [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc4

Should clarify this occurs when a dagrun does not have a start date, not a dag (which makes it even less likely to happen). I don't think this is a blocker for releasing. On Wed, Feb 22, 2017 at 1:15 PM, Dan Davydov wrote: > I rolled this out in our prod and the webservers failed to load due to

Re: [RESULT] [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc4

I rolled this out in our prod and the webservers failed to load due to this commit: [AIRFLOW-510] Filter Paused Dags, show Last Run & Trigger Dag 7c94d81c390881643f94d5e3d7d6fb351a445b72 This fixed it: - + This is caused by assuming that

Re: [RESULT] [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc4

Ack that the vote has already passed, but belated +1 (binding) On Tue, Feb 21, 2017 at 7:42 AM, Bolke de Bruin wrote: > IPMC Voting can be found here: > > http://mail-archives.apache.org/mod_mbox/incubator-general/201702.mbox/% > 3c676bdc9f-1b55-4469-92a7-9ff309ad0...@gmail.com%3e < > http://mai

Re: [RESULT] [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc4

IPMC Voting can be found here: http://mail-archives.apache.org/mod_mbox/incubator-general/201702.mbox/%3c676bdc9f-1b55-4469-92a7-9ff309ad0...@gmail.com%3e Kind regards,

Re: [RESULT] [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc4

Hello, Apache Airflow (incubating) 1.8.0 (based on RC4) has been accepted. 9 “+1” votes received: - Maxime Beauchemin (binding) - Arthur Wiedmer (binding) - Dan Davydov (binding) - Jeremiah Lowin (binding) - Siddharth Anand (binding) - Alex van Boxel (binding) - Bolke de Bruin (binding) - Jayes