Re: [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc5

2017-03-16 Thread Bolke de Bruin
I agree that it is not nice. I suggest that we revisit our fix and see if we can do better there (rather than adding new complexity). And get this into 1.8.1. Nevertheless, I consider the vote passed. Bolke > On 15 Mar 2017, at 19:12, Dan Davydov wrote: > > The only thing is that this is a c

[RESULT][VOTE]Release Airflow 1.8.0 based on Airflow 1.8.0rc5

2017-03-16 Thread Bolke de Bruin
Hello, Apache Airflow (incubating) 1.8.0 (RC5) has been accepted. 9 “+1” votes received: - Maxime Beauchemin (binding) - Chris Riccomini (binding) - Arthur Wiedmer (binding) - Jeremiah Lowin (binding) - Siddharth Anand (binding) - Alex van Boxel (binding) - Bolke de Bruin (binding) - Daniel Hua

[VOTE] Release Apache Airflow 1.8.0 (incubating)

2017-03-16 Thread Bolke de Bruin
Hello Incubator PMC’ers, The Apache Airflow community has voted and approved the proposal to release Apache Airflow 1.8.0 (incubating) based on 1.8.0 Release Candidate 5. We now kindly request the Incubator PMC members to review and vote on this incubator release. If the vote is successful we w

Re: [VOTE] Release Apache Airflow 1.8.0 (incubating)

2017-03-16 Thread Bolke de Bruin
Oops, wrong mailing list ;-). > On 16 Mar 2017, at 09:28, Bolke de Bruin wrote: > > Hello Incubator PMC’ers, > > The Apache Airflow community has voted and approved the proposal to release > Apache Airflow 1.8.0 (incubating) based on 1.8.0 Release Candidate 5. We now > kindly request the Inc

Airflow Committers: Landscape checks doing more harm than good?

2017-03-16 Thread Jeremiah Lowin
This may be an unpopular opinion, but most Airflow PRs have a little red "x" next to them not because they have failing unit tests, but because the Landscape check has decided they introduce bad code. Unfortunately Landscape is often wrong -- here it is telling me my latest PR introduced no less t

Re: Airflow Committers: Landscape checks doing more harm than good?

2017-03-16 Thread Bolke de Bruin
We can do it in Travis’ afaik. We should replace it. So +1. B. > On 16 Mar 2017, at 16:48, Jeremiah Lowin wrote: > > This may be an unpopular opinion, but most Airflow PRs have a little red > "x" next to them not because they have failing unit tests, but because the > Landscape check has decid

Re: Airflow Committers: Landscape checks doing more harm than good?

2017-03-16 Thread Dan Davydov
+1 as well though I have found it useful on larger PRs to help me catch some issues so it probably makes sense to make to add the travis linting at the same time we remove it in landscape. Not sure how much usability we lose by losing the landscape UI but I like that all of the errors would be in o

Re: Airflow Committers: Landscape checks doing more harm than good?

2017-03-16 Thread Maxime Beauchemin
+1 as well I'm disappointed because the service is inches away from getting everything right. As Bolke said, behind the cover it's little more than pylint, git hooks, and a somewhat-fancy ui. Operationally it's been getting in the way. There's a way to pipe the output of git diff into pylint and

Re: Airflow Committers: Landscape checks doing more harm than good?

2017-03-16 Thread Alex Guziel
+1 also We have code review already and the amount of false positives makes this useless. On Thu, Mar 16, 2017 at 5:02 PM, Maxime Beauchemin < maximebeauche...@gmail.com> wrote: > +1 as well > > I'm disappointed because the service is inches away from getting everything > right. As Bolke said, b

Re: Airflow Committers: Landscape checks doing more harm than good?

2017-03-16 Thread Jeremiah Lowin
Sounds like a probable consensus. I think this is a good time for me to admit I have no idea how to turn it off or move it to Travis. Could anyone please volunteer to take that on? On Thu, Mar 16, 2017 at 8:11 PM Alex Guziel wrote: > +1 also > > We have code review already and the amount of fal

Re: Airflow Committers: Landscape checks doing more harm than good?

2017-03-16 Thread Arthur Wiedmer
I just turned it off. I will look into enabling the right options for our Travis build. Best, Arthur On Thu, Mar 16, 2017 at 5:31 PM, Jeremiah Lowin wrote: > Sounds like a probable consensus. I think this is a good time for me to > admit I have no idea how to turn it off or move it to Travis.

Re: Airflow Committers: Landscape checks doing more harm than good?

2017-03-16 Thread Maxime Beauchemin
Let's wire a custom a linter command that can be called locally and respect an agreed upon set of parameters (pylint + config file, based off of our current .landscape.yml ). flake8 is far from being as good as pylint and can't be customized much AFAICT, but variations on the command bellow can he

Re: Adding Variables and Connections via script

2017-03-16 Thread siddharth anand
Yes.. I recall that PR and I also recall adding support to manage variables via the CLI a long time back.. both of those should be in the 1.8 release. -s On Wed, Mar 8, 2017 at 10:18 AM, Nicholas Hodgkinson < nik.hodgkin...@collectivehealth.com> wrote: > Thanks everyone! This is super helpful! >

Re: Make Scheduler More Centralized

2017-03-16 Thread Rui Wang
Thanks all your comments! Then looks like we should focus on scalability of scheduler now rather than adding more load on it. I will give up this centralized idea now. On Tue, Mar 14, 2017 at 3:08 PM, Rui Wang wrote: > Hi, > The design doc below I created is trying to make airflow scheduler mor

Re: Airflow Committers: Landscape checks doing more harm than good?

2017-03-16 Thread Jeremiah Lowin
FWIW I recently started using yapf (https://github.com/google/yapf) with a slightly custom config to format all of my projects. Rather than alert to discrete linting errors and concrete style rules (like PEP8) -- things I'm sure we all do anyway -- it reformats all code in compliance with your chos

Re: Airflow Committers: Landscape checks doing more harm than good?

2017-03-16 Thread siddharth anand
+1 for replacing it with travis linting. On Thu, Mar 16, 2017 at 7:59 PM, Jeremiah Lowin wrote: > FWIW I recently started using yapf (https://github.com/google/yapf) with a > slightly custom config to format all of my projects. Rather than alert to > discrete linting errors and concrete style ru

Re: Airflow Committers: Landscape checks doing more harm than good?

2017-03-16 Thread Maxime Beauchemin
Oh btw I think I tried to turn it off before and couldn't. I looked at all configuration options on both landscape and GH it just kept doing its thing no matter what. Ended up giving up. Curious to see how it goes this time. On Mar 16, 2017 8:57 PM, "siddharth anand" wrote: > +1 for replacing it