On Mon, Mar 16, 2020 at 11:16 AM Driesprong, Fokko <[email protected]> wrote:
> Personally I don't like to have two versions in the PyPi repo. This also > complicates the releases, since we need to test, release and verify two > versions of Airflow > Not necessarily. Release is automated (and I run both package builds in CI now). Regarding testing - at the moment of release 'apache-airflow` and 'apache-airflow-pinned' are 1-1 equivalent. Their behaviour starts to be different when something out of our control happens with dependencies after the release. We can even have automated tests and install both 'apache-airflow' and 'apache-airlfow-pinned' at release time and compare the virtualenvs on binary level to verify that they are equivalent. This way we can just test one of them. > I'm afraid that this might confuse users. > I think that the current situation is much more confusing for the users. In order to install <1.10.8 for example you need to know that you should run 'pip install apache-airflow==1.10.8 Werkzeug<1.0'. If you want to install 1.10.2 you need to add other constraints. And tomorrow another constraint might be needed to install 1.10.9. This is also a hard blocker for the official, production image of Apache Airflow. One of the constraints to make an official Docker image is repeatability https://github.com/docker-library/official-images#repeatability . We don't have repeatability now. We need to consider what our alternatives are. I think our problem is that: 1) we want to have it open for operator development 2) we want the user (and official docker build) to have a reliable way to install the released version. This already happened and will happen that out of a sudden released version of airflow will stop installing without magic 'Werkzeug <= 1.0' or similar. We already have good solution for 1) with open dependencies . I see two solutions to that: *Solution 1) Ask the users to use requirements.txt as constraint file manually:* a) manually download requirements.txt from the right release in github b) pip install apache-airflow==1.10.9 --constraint requirements.txt Or *Solution 2) Use "-pinned" package (effectively does the same):* pip install apache-airflow-pinned==1.10.9 I find the second one much less confusing and straightforward.. I am all ears if we have any other solution? Can we discuss some potential options here? Besides that, it feels a bit like we're reinventing certain mechanisms that > are already in tooling like Poetry. > Unfortunately according to my checks - Poetry does not handle our case where we are both library and application at the same time. Form what I checked poetry uses .lock file and publishes the "pinned" version always in such case. Hacking it to support both cases would be super-difficult and defeat the purpose of using ready-to-use-tool, I looked at it and could not find a way to achieve both 1) and 2). If you think we can do it, It would be great to discuss the approach we can take. If we can agree that we only publish pinned version, I would be super happy and switch to poetry immediately - but we cannot do it I am afraid. There was a long discussion about it and we cannot afford pinning by default, unfortunately. This has already been discussed several times and unless this assumption has changed - we cannot use poetry or similar. My 2ct. > -- Jarek Potiuk Polidea <https://www.polidea.com/> | Principal Software Engineer M: +48 660 796 129 <+48660796129> [image: Polidea] <https://www.polidea.com/>
