Re: [PROPOSE] Ease future migration path to 2.0 by provider's operators/hook backporting to 1.10.*

Thanks Ash! It seems it works really well and is super simple! I have a POC working for Airflow: https://github.com/apache/airflow/pull/6507 I managed to build and pip-install two packages: 1) apache_airflow 2.0 -> which is the same as today - containing everything - including providers and gcp.

Re: [PROPOSE] Ease future migration path to 2.0 by provider's operators/hook backporting to 1.10.*

Yes let's just do (1) for now. On Tue, Nov 5, 2019, 08:48 Jarek Potiuk wrote: > Thanks Ash! It might indeed work. I will take it from there and try to make > a POC PR with airflow. > > It's a bit different approach than google-python libraries (they keep all > the libraries as separate sub-pac

Re: [PROPOSE] Ease future migration path to 2.0 by provider's operators/hook backporting to 1.10.*

Thanks Ash! It might indeed work. I will take it from there and try to make a POC PR with airflow. It's a bit different approach than google-python libraries (they keep all the libraries as separate sub-packages/mini projects inside the main project). The approach you propose is far less invasive

Re: [PROPOSE] Ease future migration path to 2.0 by provider's operators/hook backporting to 1.10.*

https://github.com/ashb/airflow-submodule-test That seems to work in any order things are installed, at least on python 3.7. I've had a stressful few days so I may have missed something. Please tell me if there's a case I've missed, or if this is

Re: [PROPOSE] Ease future migration path to 2.0 by provider's operators/hook backporting to 1.10.*

Pretty hard pass from me in airflow_ext. If it's released by airflow I want it to live under airflow.* (Anyone else is free to release packages under any namespace they choose) That said I think I've got something that works: /Users/ash/.virtualenvs/test-providers/lib/python3.7/site-packages/no

Re: [PROPOSE] Ease future migration path to 2.0 by provider's operators/hook backporting to 1.10.*

Hey Ash, Thanks for the offer. I must admin pkgutil and package namespaces are not the best documented part of python. I dug a deep deeper and I found a similar problem - https://github.com/pypa/setuptools/issues/895. Seems that even if it is not explicitly explained in pkgutil documentation, th

Re: [PROPOSE] Ease future migration path to 2.0 by provider's operators/hook backporting to 1.10.*

Let me run some tests too - I've used them a bit in the past. I thought since we only want to make airflow.providers a namespace package it might work for us. Will report back next week. -ash On 31 October 2019 15:58:22 GMT, Jarek Potiuk wrote: >The same repo (so mono-repo approach). All packa

Re: [PROPOSE] Ease future migration path to 2.0 by provider's operators/hook backporting to 1.10.*

The same repo (so mono-repo approach). All packages would be in "airflow_integrations" directory. It's mainly about moving the operators/hooks/sensor files to different directory structure. It might be done pretty much without changing the current installation/development model: 1) We can add set

Re: [PROPOSE] Ease future migration path to 2.0 by provider's operators/hook backporting to 1.10.*

So create another package in a different repo? or the same repo with a separate setup.py file that has airflow has dependency? On Thu, Oct 31, 2019 at 2:32 PM Jarek Potiuk wrote: > TL;DR; I did some more testing on how namespaces work. I still believe the > only way to use namespaces is to ha

Re: [PROPOSE] Ease future migration path to 2.0 by provider's operators/hook backporting to 1.10.*

TL;DR; I did some more testing on how namespaces work. I still believe the only way to use namespaces is to have separate (for example "airflow_integrations") package for all backportable packages. I am not sue if someone used namespaces before, but after reading and trying out , the main blocker

Re: [PROPOSE] Ease future migration path to 2.0 by provider's operators/hook backporting to 1.10.*

The namespace feature looks promising and from your tests, it looks like it would work well from Airflow 2.0 and onwards. I will look at it in-depth and see if I have more suggestions or opinion on it On Tue, Oct 29, 2019 at 3:32 PM Jarek Potiuk wrote: > TL;DR; We did some testing about namespa

Re: [PROPOSE] Ease future migration path to 2.0 by provider's operators/hook backporting to 1.10.*

TL;DR; We did some testing about namespaces and packaging (and potential backporting options for 1.10.* python3 Airflows) and we think it's best to use namespaces quickly and use different package name "airflow-integrations" for all non-fundamental integrations. Unless we missed some tricks, we ca

Re: [PROPOSE] Ease future migration path to 2.0 by provider's operators/hook backporting to 1.10.*

Tomasz and Ash got good points about the overhead of having separate repos. But while we grow bigger and more mature, I would prefer to have what was described in AIP-8. It shouldn't be extremely hard for us to come up with good strategies to handle the overhead. AIP-8 already talked about how it c

Re: [PROPOSE] Ease future migration path to 2.0 by provider's operators/hook backporting to 1.10.*

We are checking how we can use namespaces in back-portable way and we will have POC soon so that we all will be able to see how it will look like. J. On Mon, Oct 28, 2019 at 1:24 PM Ash Berlin-Taylor wrote: > I'll have to read your proposal in detail (sorry, no time right now!), but > I'm broad

Re: [PROPOSE] Ease future migration path to 2.0 by provider's operators/hook backporting to 1.10.*

I'll have to read your proposal in detail (sorry, no time right now!), but I'm broadly in favour of this approach, and I think keeping them _in_ the same repo is the best plan -- that makes writing and testing cross-cutting changes easier. -a > On 28 Oct 2019, at 12:14, Tomasz Urbaszek wrot

Re: [PROPOSE] Ease future migration path to 2.0 by provider's operators/hook backporting to 1.10.*

I think utilizing namespaces should reduce a lot of problems raised by using separate repos (who will manage it? how to release? where should be the repo?). Bests, Tomek On Sun, Oct 27, 2019 at 11:54 AM Jarek Potiuk wrote: > Thanks Bas for comments! Let me share my thoughts below. > > On Sun, O

Re: [PROPOSE] Ease future migration path to 2.0 by provider's operators/hook backporting to 1.10.*

Thanks Bas for comments! Let me share my thoughts below. On Sun, Oct 27, 2019 at 9:23 AM Bas Harenslak wrote: > Hi Jarek, I definitely see a future in creating separate installable > packages for various operators/hooks/etc (as in AIP-8). This would IMO > strip the “core” Airflow to only what’s

Re: [PROPOSE] Ease future migration path to 2.0 by provider's operators/hook backporting to 1.10.*

Hi Jarek, I definitely see a future in creating separate installable packages for various operators/hooks/etc (as in AIP-8). This would IMO strip the “core” Airflow to only what’s needed and result in a small package without a ton of dependencies (and make it more maintainable, shorter tests, et

Re: [PROPOSE] Ease future migration path to 2.0 by provider's operators/hook backporting to 1.10.*

Hello - any comments on that? I am happy to make it into an AIP :)? On Sun, Oct 13, 2019 at 5:53 PM Jarek Potiuk wrote: > *Motivation* > > I think we really should start thinking about making it easier to migrate > to 2.0 for our users. After implementing some recent changes related to > AIP-21

[PROPOSE] Ease future migration path to 2.0 by provider's operators/hook backporting to 1.10.*

*Motivation* I think we really should start thinking about making it easier to migrate to 2.0 for our users. After implementing some recent changes related to AIP-21- Changes in import paths I think I have an i