Airflow 2017 Roadmap Survey

Hi folks, At the end of last year we had a list of possible roadmap items for 2017, this was created with the feedback from the contributors' meetings, meetups and thru the mailing list. The wiki was up for around a month for

Re: Airflow 1.8.0 alpha 4

Hi Laura, What version of MySQL are you using? 5.6.4 is the absolute minimum. - Bolke > On 11 Jan 2017, at 22:52, Laura Lorenz wrote: > > Hello! I just tried to install the release candidate alpha 4. > > I was previously on airflow 1.7.0, and am not able to upgrade my metadata > database on 1

Re: Airflow 1.8.0 alpha 4

Hello! I just tried to install the release candidate alpha 4. I was previously on airflow 1.7.0, and am not able to upgrade my metadata database on 1.8.0a4. I've detailed everything in a JIRA ticket at https://issues.apache.org/jira/browse/AIRFLOW-748. Let me know if I'm doing the install wrong, a

Re: Airflow 1.8.0 alpha 4

The task dependency engine code is well commented, but I can provide a high level overview specifically for developers if there is interest (note that this would be the first documentation of it's kind in that it would be developer-only documentation). The disadvantage is that it would create dupli

Re: Airflow 1.8.0 alpha 4

@bolke, this sounds like a good list. On Wed, Jan 11, 2017 at 12:01 PM, Bolke de Bruin wrote: > Ok. > > For now to call it “beta” 4 items seems to be left: > > Blocker: > * retry_delay not respected > * poison pill due to re-queue before process has finished (to be > investigated) > > Features:

Re: Airflow 1.8 not honoring retry_delay

First analysis: The dependency checker (at least the retry_delay checker) is not called in the scheduler for some reason. As it leaves the scheduler the state will be set to “queued” which won’t match the “is_premature” requirement of “UP_FOR_RETRY”. @Dan: are you able to help out here? Bolke

Re: Last minute open meetup speaking slot

Thank you Arthur for stepping up, and my sincere apologies. I just don't want to get anyone sick. I hope to present the talk at the next meetup. On Jan 10, 2017 11:22 PM, "George Leslie-Waksman" wrote: > Listing updated. Thanks for stepping in on short notice. > > On Tue, Jan 10, 2017 at 8:15 PM

Re: Airflow 1.8.0 alpha 4

Ok. For now to call it “beta” 4 items seems to be left: Blocker: * retry_delay not respected * poison pill due to re-queue before process has finished (to be investigated) Features: * cgroups + impersonation * dag.catchup (Ben Tallman -> Only documentation is missing). PRs that contain documen

Re: Airflow Release Planning and Supported Release Lifetime

Hey Bolke, This sounds great to me. +1 A quick blurb on the wiki about versioning (e.g. 1.8.0a3 vs 1.8.0, vs however the RC is going to be versioned) should be called out, too. Re: a vote, I'd say it's a good idea, regardless of whether its required or not, just to make sure everyone's on the sam

Re: Airflow 1.8.0 alpha 4

+1 We can always think about different ways of doing this later (fair share scheduling etc...) Best, Arthur On Wed, Jan 11, 2017 at 4:46 AM, Bolke de Bruin wrote: > Dear All, > > I would like to drop "Schedule all pending DAG runs in a single scheduler > loop” from the 1.8.0 release (updated:

Re: Airflow 1.8 not honoring retry_delay

Hi Bolke, Here is the JIRA issue, https://issues.apache.org/jira/browse/AIRFLOW-747. Harvey Xia | Software Engineer harvey...@spotify.com +1 (339) 225 1875 On Wed, Jan 11, 2017 at 11:32 AM, Bolke de Bruin wrote: > Hi Harvey, > > Thanks for reporting! Can you create a lira for this? I’ll have

Re: Airflow 1.8.0 alpha 4

Hey Bolke, I'm installing the latest alpha as we speak. As far as I'm concerned, we've hit no remaining blockers in our dev environment. And I'm ready for beta whenever you cut it (I'll install the beta in our prod cluster). Re: dropping scheduler change, I'm fine with that. I would rather favor

Provide failing unit tests along with bug reports if you can!

Hi all! I just had a thought about reporting bugs. Of course we don't expect everyone to jump in and send PRs fixing bugs, especially when the reported bug touches the core of Airflow and requires a lot of context to fix. If you can, one thing that can really help is to *provide a failing unit t

Re: Airflow 1.8 not honoring retry_delay

Hi Harvey, Thanks for reporting! Can you create a lira for this? I’ll have a look if I can reproduce it. - Bolke > On 11 Jan 2017, at 16:06, Harvey Xia wrote: > > Hi all, > > In Airflow 1.8 alpha 2, using LocalExecutor, DAGs do not seem to honor the > retry_delay parameter, i.e. the retries

Airflow 1.8 not honoring retry_delay

Hi all, In Airflow 1.8 alpha 2, using LocalExecutor, DAGs do not seem to honor the retry_delay parameter, i.e. the retries happen immediately one after the other without waiting the specific retry_delay time. However, the *number* of retries is honored. I am testing with the following code: from

Re: Refactoring Connection

Typo on the airflow link: it's https://issues.apache.org/jira/browse/AIRFLOW-746 Sorry Le mer. 11 janv. 2017 à 15:48, Gael Magnan a écrit : > Thanks for all the information, > > currently what I have is working iso with what was existing, just need to > add some doc and tests, but I have a few q

Re: Refactoring Connection

Thanks for all the information, currently what I have is working iso with what was existing, just need to add some doc and tests, but I have a few questions. Currently airflow/urtils/db.py initdb function creates a lot of connections by default. I understand that they are used for the tests, but

Re: Airflow 1.8.0 alpha 4

Hey Bolke, I'll be of for a few days but I started looking at the CHANGELIST to make sure we now have a list of changes for 1.8 (seems like a big list...). I'll will also drop the trigger issue for the 1.8 and see what we can do for the next version. I'll make it into a documentation issue. On We

Re: Airflow 1.8.0 alpha 4

Dear All, I would like to drop "Schedule all pending DAG runs in a single scheduler loop” from the 1.8.0 release (updated: https://github.com/apache/incubator-airflow/pull/1980 , original: https://github.com/apache/incubator-airflow/pull/1