Re: Let's talk Airflow 2.0

2020-03-20 Thread Daniel Imberman
Great! Hope to get a few more folx to give +1's but I think we have a good path forward here :) On Fri, Mar 20, 2020 at 12:51 PM Jarek Potiuk wrote: > > > > > > I agree especially for larger-scale users migrations are a difficult > > process. Perhaps we can adopt something similar to a blockchai

Re: Let's talk Airflow 2.0

2020-03-20 Thread Jarek Potiuk
> > > I agree especially for larger-scale users migrations are a difficult > process. Perhaps we can adopt something similar to a blockchain fork (e.g. > determine X known airflow using companies, and start the countdown as soon > as Y% of them migrate). I really just want to make sure we don't end

Re: Let's talk Airflow 2.0

2020-03-20 Thread Daniel Imberman
Hi Jarek, thank you for your support on this effort! For sure. One comment here. One of the ways we discussed with Kamil > about the approach we want to take is to engage more people in > implementing it - so Kamil would create a basic framework and we might > have many people implementing and te

Re: Let's talk Airflow 2.0

2020-03-20 Thread Jarek Potiuk
> Hello, fellow Airflowers! > > I wanted to start this discussion in hopes of "kicking the tires" and > agreeing on a timeline for the Airflow 2.0 release. Cool! > > But I want to add _ feature which would be so awesome in a 2.0 press > release! > > We should also > consider that the 2.0 rele

Volunteers needed for 1.10 backport Airflow provider packages

2020-03-20 Thread Jarek Potiuk
TL;DR; We are looking for volunteers to help with testing backported providers package using airflow 1.10, so that we can release them. This way users will be able to use the latest and greatest Airflow providers in 1.10 without waiting for 2.0. We are just about to finish system-testing the backp

Let's talk Airflow 2.0

2020-03-20 Thread Daniel Imberman
Hello, fellow Airflowers! I wanted to start this discussion in hopes of "kicking the tires" and agreeing on a timeline for the Airflow 2.0 release. It seems to me that our 1.10 branch is straying farther and farther from master, and with that, there are two significant consequences. Releases beco

Re: [DISCUSS] Back to (some) dependency pinning

2020-03-20 Thread Jarek Potiuk
And rebased it right now and fixed automated requirements update. On Fri, Mar 20, 2020 at 2:28 PM Jarek Potiuk wrote: > Ah BTW. I just noticed that for some reason I pasted an old PR earlier in > the thread :(. > This is the one with requirements.txt I am talking about: > https://github.com/apac

Re: [DISCUSS] Back to (some) dependency pinning

2020-03-20 Thread Jarek Potiuk
Ah BTW. I just noticed that for some reason I pasted an old PR earlier in the thread :(. This is the one with requirements.txt I am talking about: https://github.com/apache/airflow/pull/7730 On Fri, Mar 20, 2020 at 2:26 PM Jarek Potiuk wrote: > Nope. Not blocking. I can work with my branch just

Re: [DISCUSS] Back to (some) dependency pinning

2020-03-20 Thread Jarek Potiuk
Nope. Not blocking. I can work with my branch just requirements.txt is enough for that :) I think the problem with semver is that it is loosely followed - we had a number of breakages in the past with minor version upgrades :(. J. On Fri, Mar 20, 2020 at 1:27 PM Kaxil Naik wrote: > Thanks fo

Re: [DISCUSS] Back to (some) dependency pinning

2020-03-20 Thread Kaxil Naik
Thanks for the detailed explanation Jarek. How about we have an upper limit for all our dependencies, example instead of "google-cloud-storage>=1.16", we have "google-cloud-storage>=1.16,<2.0" ? If a dependency breaks compatibility in minor versions, we can't do anything about it but if they foll

[VOTE] AIP-33 Secrets Backend

2020-03-20 Thread Kaxil Naik
This vote has passed with 12 +6 votes and 0 -1 votes and all PRs merged. Regards, Kaxil Naik On Mon, Mar 16, 2020 at 12:58 PM Tomasz Urbaszek wrote: > +1 binding > > On Mon, Mar 16, 2020 at 1:16 PM Driesprong, Fokko > wrote: > > > +1 > > > > Op ma 16 mrt. 2020 om 12:53 schreef Bas Harenslak >

Re: Can you give more information on GSoc Project "Add docs how to integrate with grafana and prometheus"

2020-03-20 Thread Jarek Potiuk
Hello Darshan, I think https://issues.apache.org/jira/browse/AIRFLOW-5154 says broadly what it is about. Some more details - ideally the document should describe (based on already implemented and running setup so that it is verified) how to configure and setup Gprahana/Prometheus in order to eff