Any other comments? I'd love to hear your thoughts. It's the one thing that maybe not keeps me from prod image, But I would love to know if I can rely on the requirements.txt being part of the source code so that I can use it when building the prod image..
J. On Mon, Mar 16, 2020 at 12:16 PM Jarek Potiuk <[email protected]> wrote: > > On Mon, Mar 16, 2020 at 11:16 AM Driesprong, Fokko <[email protected]> > wrote: > >> Personally I don't like to have two versions in the PyPi repo. This also >> complicates the releases, since we need to test, release and verify two >> versions of Airflow >> > > Not necessarily. Release is automated (and I run both package builds in CI > now). > Regarding testing - at the moment of release 'apache-airflow` and > 'apache-airflow-pinned' are 1-1 equivalent. Their behaviour starts to be > different when something out of our control happens with dependencies after > the release. We can even have automated tests and install both > 'apache-airflow' and 'apache-airlfow-pinned' at release time and compare > the virtualenvs on binary level to verify that they are equivalent. This > way we can just test one of them. > > >> I'm afraid that this might confuse users. >> > > I think that the current situation is much more confusing for the users. > In order to install <1.10.8 for example you need to know that you should > run 'pip install apache-airflow==1.10.8 Werkzeug<1.0'. If you want to > install 1.10.2 you need to add other constraints. And tomorrow another > constraint might be needed to install 1.10.9. This is also a hard blocker > for the official, production image of Apache Airflow. One of the > constraints to make an official Docker image is repeatability > https://github.com/docker-library/official-images#repeatability . We > don't have repeatability now. > > We need to consider what our alternatives are. I think our problem is that: > > 1) we want to have it open for operator development > 2) we want the user (and official docker build) to have a reliable way to > install the released version. This already happened and will happen that > out of a sudden released version of airflow will stop installing without > magic 'Werkzeug <= 1.0' or similar. > > We already have good solution for 1) with open dependencies . I see two > solutions to that: > > *Solution 1) Ask the users to use requirements.txt as constraint file > manually:* > > a) manually download requirements.txt from the right release in github > b) pip install apache-airflow==1.10.9 --constraint requirements.txt > > Or > > *Solution 2) Use "-pinned" package (effectively does the same):* > > pip install apache-airflow-pinned==1.10.9 > > I find the second one much less confusing and straightforward.. > > > > I am all ears if we have any other solution? Can we discuss some potential > options here? > > > Besides that, it feels a bit like we're reinventing certain mechanisms that >> are already in tooling like Poetry. >> > > Unfortunately according to my checks - Poetry does not handle our case > where we are both library and application at the same time. Form what I > checked poetry uses .lock file and publishes the "pinned" version always in > such case. Hacking it to support both cases would be super-difficult and > defeat the purpose of using ready-to-use-tool, I looked at it and could not > find a way to achieve both 1) and 2). If you think we can do it, It would > be great to discuss the approach we can take. If we can agree that we only > publish pinned version, I would be super happy and switch to poetry > immediately - but we cannot do it I am afraid. There was a long discussion > about it and we cannot afford pinning by default, unfortunately. This has > already been discussed several times and unless this assumption has changed > - we cannot use poetry or similar. > > My 2ct. >> > -- > > Jarek Potiuk > Polidea <https://www.polidea.com/> | Principal Software Engineer > > M: +48 660 796 129 <+48660796129> > [image: Polidea] <https://www.polidea.com/> > > -- Jarek Potiuk Polidea <https://www.polidea.com/> | Principal Software Engineer M: +48 660 796 129 <+48660796129> [image: Polidea] <https://www.polidea.com/>
