Yep. I am also curious what others think. We can do either or both or even change the approach over time. I made it flexible enough to handle all cases :).
J . pon., 10 lut 2020, 10:08 użytkownik Ash Berlin-Taylor <[email protected]> napisał: > I think the question to answer is: is this just a backport to make > adoption of Airflow 2 easier, or is this how we plan to release providers > in the future. > > If it is the way we want to release providers (i.e. separate from > "airflow-core") then separate packages per provider rather than one big one > would be the way to go -- that way each provider can operate at it's own > release schedule. (i.e. something similar to how Hashicorp's Terraform > handles its providers.) > What are people's thoughts on this? > On Feb 10 2020, at 8:48 am, Jarek Potiuk <[email protected]> wrote: > > Good point Kaxil - this is one of the reasons why we are not yet doing > > AIP-8 - because of potential dependency hell > > > > Currently we are pretty well synchronised w/regards dependencies. And the > > packages installs nicely on 1.10.9 without any dependency conflict. As > part > > of the > > implementation I am planning to test the packages for other versions as > > well and see if any fixes are needed. In fact the second PR will be > updated > > still so that > > Breeze becomes an easy test platform for all those older versions as > well. > > I have some ideas on how to make it really easy.. > > > > The only correction I had to make was docutils (limit the upper bound) - > > actually what makes it better is that we have "open" dependencies in > > released packages. > > > > As long as we do not increase the lower bounds of dependencies in airflow > > master, I think we are good as-is. > > > > For the future - if you look at the current setup.py - dependencies > > fot backported packages are reused from main airflow, but in case we need > > it, we can always > > modify the dependencies specifically for the backported packages. It's > now > > nicely organized per provider, so it should be manageable even if we > change > > a lot in the future. > > > > I also do not foresee that it will be very long. I think we should > release > > maybe a few backport packages and at some point in time when 2.0 is > > released we should stop doing it > > to encourage people to move to 2.0. > > > > J. > > > > On Mon, Feb 10, 2020 at 8:52 AM Kaxil Naik <[email protected]> wrote: > > > I think a single backported package makes sense for now. > > > However I have one question, how would we deal with a situation where > we > > > have dependecy conflict between the operator in Airflow 1.10.* and the > > > operator in the "backported" providers package. > > > > > > An example would be let's say the GCS Operator in Airflow 1.10.9 > requires > > > google-cloud-storage<0.10.0 and the one we use in Master / in > backported > > > package needs google-cloud-storage>=0.11.0 > > > > > > I am just making up this example, it is not the case with Gcs > operators. > > > And may be we might not even have any compatibility issues at all :) > but > > > just thinking it out loud. > > > > > > Regards, > > > Kaxil > > > > > > On Mon, Feb 10, 2020, 04:05 Jarek Potiuk <[email protected]> > wrote: > > > > Hello everyone, > > > > TL;DR; I've been quite busy recently as I was working on backported > > > > "providers" packages for Airflow 1.10.* and I have some pretty good > news > > > > > > on > > > > that front. I would love to have your comments and opinions on the > > > > > > current > > > > state of it. This is more 'information" on what is being implemented > > > > > > now - > > > > I will send a separate thread about some future decisions needed - > mostly > > > > from PMC side. > > > > > > > > I have two PRs that are relevant and I wanted to describe both here: > > > > 1) Preparing backportable packages for Airflow 1.10.* > > > > https://github.com/apache/airflow/pull/7391 > > > > > > > > This PR modifies setyp.py to enable preparation of backportable > packages > > > > for Airlfow 1.10.*. Using this version of setup.py we can prepare and > > > > release PIP packages of "providers" package that will be installable > for > > > > Airflow 1.10.* series. I managed to have it working without > converting > > > > packages to implicit namespaces (separate discussion on the devlist). > > > > > > > > I did it in a way that we can either prepare > "apache-airflow-providers" > > > > package (with all "providers" code in a single package) or we can > have > > > > "apache-airflow-providers-XXXXX" packages - separately for each > providers > > > > package we have. The latter approach produces many more smaller (and > > > > potentially inter-dependent) packages - something that in the future > > > > > > might > > > > be base for AIP-8 > > > > < > > > > > > > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=100827303&focusedCommentId=103093048 > > > > > > > > > - > > > > but we do not need it for now. It also nicely keeps dependencies > > > > > > separately > > > > - so each of the packages has only minimum set of dependencies > needed for > > > > each package. > > > > > > > > I would like to leave it for now, but for the purpose of backporting > I > > > > think releasing single "providers" package makes much more sense. > But if > > > > others think that we should release many more smaller "providers" > > > > > > packages > > > > separately - I am also quite OK with it. It's just the matter of > > > > testing/status of each package and some inter-dependencies (some > packages > > > > depend on each other) - especially for transfer operators. > > > > > > > > 2) System testing of backportable packages: > > > > https://github.com/apache/airflow/pull/7389 > > > > > > > > We need to have a way to test that the backported packages are > working > > > for > > > > Airflow 1.10. We cannot run all unit tests for Airflow 1.10, but we > can > > > > > > run > > > > some system end-2-end tests. While we do not have consistent system > > > > "end-2-end" tests for all operators we have quite extensive set of > system > > > > tests for GCP operators. Those tests run example dags from google > cloud > > > > platform operators - the example dags are used to both - provide > examples > > > > in the docs but also can be run (with appropriate environment) to > run the > > > > example dag automatically with a real external system (GCP in this > > > > > > case). I > > > > proposed this approach a long time ago in AIP-4 > > > > < > > > > > > > > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-4+Support+for+System+Tests+for+external+systems > > > > > > > > > and > > > > while it has not been "universally" accepted yet, we followed it > with GCP > > > > operator implementations (and we have all GCP operators automatically > > > > testable with system tests), With this PR I made the system test > approach > > > > nicely integrated with Pytest markers, Breeze and our test > environment - > > > > > > so > > > > it is now very easy to run system tests semi-automatically (and in > the > > > > future we can fully automate it when we switch to GitHub actions). > > > > > > > > We are planning to run all the system tests for all GCP operators, > but > > > when > > > > it's there it's also rather easy to add tests for other groups of > > > > > > operators > > > > so I am planning to have a community-driven effort to add more of > those > > > > system-tests (and make sure that backported packages can be safely > used > > > > > > in > > > > 1.10.* environment). > > > > > > > > > > > > J. > > > > > > > > > > > > > > > > > > > > -- > > > > Jarek Potiuk > > > > Polidea <https://www.polidea.com/> | Principal Software Engineer > > > > > > > > M: +48 660 796 129 <+48660796129> > > > > [image: Polidea] <https://www.polidea.com/> > > > > > > > > > > > > > -- > > Jarek Potiuk > > Polidea <https://www.polidea.com/> | Principal Software Engineer > > > > M: +48 660 796 129 <+48660796129> > > [image: Polidea] <https://www.polidea.com/> > > > >
