I think irrespective of what we do about releasing a pinned version, using this approach so our prod image is repeatable sounds good!
On 17 March 2020 19:17:59 GMT, Jarek Potiuk <[email protected]> wrote: >Any other comments? > >I'd love to hear your thoughts. It's the one thing that maybe not keeps >me >from prod image, But I would love to know if I can rely on the >requirements.txt being part of the source code so that I can use it >when building the prod image.. > >J. > > >On Mon, Mar 16, 2020 at 12:16 PM Jarek Potiuk ><[email protected]> >wrote: > >> >> On Mon, Mar 16, 2020 at 11:16 AM Driesprong, Fokko ><[email protected]> >> wrote: >> >>> Personally I don't like to have two versions in the PyPi repo. This >also >>> complicates the releases, since we need to test, release and verify >two >>> versions of Airflow >>> >> >> Not necessarily. Release is automated (and I run both package builds >in CI >> now). >> Regarding testing - at the moment of release 'apache-airflow` and >> 'apache-airflow-pinned' are 1-1 equivalent. Their behaviour starts to >be >> different when something out of our control happens with dependencies >after >> the release. We can even have automated tests and install both >> 'apache-airflow' and 'apache-airlfow-pinned' at release time and >compare >> the virtualenvs on binary level to verify that they are equivalent. >This >> way we can just test one of them. >> >> >>> I'm afraid that this might confuse users. >>> >> >> I think that the current situation is much more confusing for the >users. >> In order to install <1.10.8 for example you need to know that you >should >> run 'pip install apache-airflow==1.10.8 Werkzeug<1.0'. If you want to >> install 1.10.2 you need to add other constraints. And tomorrow >another >> constraint might be needed to install 1.10.9. This is also a hard >blocker >> for the official, production image of Apache Airflow. One of the >> constraints to make an official Docker image is repeatability >> https://github.com/docker-library/official-images#repeatability . We >> don't have repeatability now. >> >> We need to consider what our alternatives are. I think our problem is >that: >> >> 1) we want to have it open for operator development >> 2) we want the user (and official docker build) to have a reliable >way to >> install the released version. This already happened and will happen >that >> out of a sudden released version of airflow will stop installing >without >> magic 'Werkzeug <= 1.0' or similar. >> >> We already have good solution for 1) with open dependencies . I see >two >> solutions to that: >> >> *Solution 1) Ask the users to use requirements.txt as constraint file >> manually:* >> >> a) manually download requirements.txt from the right release in >github >> b) pip install apache-airflow==1.10.9 --constraint requirements.txt >> >> Or >> >> *Solution 2) Use "-pinned" package (effectively does the same):* >> >> pip install apache-airflow-pinned==1.10.9 >> >> I find the second one much less confusing and straightforward.. >> >> >> >> I am all ears if we have any other solution? Can we discuss some >potential >> options here? >> >> >> Besides that, it feels a bit like we're reinventing certain >mechanisms that >>> are already in tooling like Poetry. >>> >> >> Unfortunately according to my checks - Poetry does not handle our >case >> where we are both library and application at the same time. Form what >I >> checked poetry uses .lock file and publishes the "pinned" version >always in >> such case. Hacking it to support both cases would be super-difficult >and >> defeat the purpose of using ready-to-use-tool, I looked at it and >could not >> find a way to achieve both 1) and 2). If you think we can do it, It >would >> be great to discuss the approach we can take. If we can agree that we >only >> publish pinned version, I would be super happy and switch to poetry >> immediately - but we cannot do it I am afraid. There was a long >discussion >> about it and we cannot afford pinning by default, unfortunately. This >has >> already been discussed several times and unless this assumption has >changed >> - we cannot use poetry or similar. >> >> My 2ct. >>> >> -- >> >> Jarek Potiuk >> Polidea <https://www.polidea.com/> | Principal Software Engineer >> >> M: +48 660 796 129 <+48660796129> >> [image: Polidea] <https://www.polidea.com/> >> >> > >-- > >Jarek Potiuk >Polidea <https://www.polidea.com/> | Principal Software Engineer > >M: +48 660 796 129 <+48660796129> >[image: Polidea] <https://www.polidea.com/>
