potiuk commented on a change in pull request #4938: [AIRFLOW-4117] Multi-staging Image - Travis CI tests [Step 3/3] URL: https://github.com/apache/airflow/pull/4938#discussion_r298472784
########## File path: Dockerfile ########## @@ -292,28 +320,32 @@ COPY --chown=airflow:airflow airflow/version.py ${AIRFLOW_SOURCES}/airflow/versi COPY --chown=airflow:airflow airflow/__init__.py ${AIRFLOW_SOURCES}/airflow/__init__.py COPY --chown=airflow:airflow airflow/bin/airflow ${AIRFLOW_SOURCES}/airflow/bin/airflow -# Airflow Extras installed -ARG AIRFLOW_EXTRAS="all" -ENV AIRFLOW_EXTRAS=${AIRFLOW_EXTRAS} -RUN echo "Installing with extras: ${AIRFLOW_EXTRAS}." - -# First install only dependencies but no Apache Airflow itself -# This way regular changes in sources of Airflow will not trigger reinstallation of all dependencies -# And this Docker layer will be reused between builds. -RUN pip install --no-use-pep517 -e ".[${AIRFLOW_EXTRAS}]" +# First upgrade everything to latest version and then run pip install -e .[extras] to downgrade the +# dependencies which should not be the latest ones according to setup.py +RUN pip list --outdated --format=freeze | grep -v '^\-e' | cut -d = -f 1 | xargs pip install -U && \ Review comment: Yeah. I will try to explain it a bit better in the comment. Basically what I am trying to do is to upgrade to the latest version of all dependencies that are still matching the setup.py limitations. There is no direct way to do it in pip (`pip install --upgrade -e .[all]` does not work as expected). So what I do instead is to upgrade all installed and outdated dependencies to latest version and then run `pip install -e .[all]` that will downgrade those dependencies that are not meeting setup.py limitations. This way we have at the end an equivalent of fresh install of latest possible dependencies as if installed today from the scratch but we build it on top of earlier requirements we already have (thus - having very fast 'pip install step') Example: 1) We have `moto==1.3.5` and `sshtunnel>=0.1.4,<0.2` in setup.py 2) In our virtualenv we have: `moto@1.3.5` and sshtunnel@0.1.4` 2) We run `pip install --upgrade moto sshtunnel` (both have newer versions in PIP 3) Moto is installed dragging upgrades of few other packages: ``` Installing collected packages: botocore, s3transfer, boto3, aws-sam-translator, jsonpointer, jsonpatch, cfn-lint, jsondiff, moto Found existing installation: botocore 1.10.84 Uninstalling botocore-1.10.84: Successfully uninstalled botocore-1.10.84 Found existing installation: s3transfer 0.1.13 Uninstalling s3transfer-0.1.13: Successfully uninstalled s3transfer-0.1.13 Found existing installation: boto3 1.7.84 Uninstalling boto3-1.7.84: Successfully uninstalled boto3-1.7.84 Found existing installation: jsondiff 1.1.1 Uninstalling jsondiff-1.1.1: Successfully uninstalled jsondiff-1.1.1 Found existing installation: moto 1.3.5 Uninstalling moto-1.3.5: Successfully uninstalled moto-1.3.5 Successfully installed aws-sam-translator-1.12.0 boto3-1.9.179 botocore-1.12.179 cfn-lint-0.22.0 jsondiff-1.1.2 jsonpatch-1.23 jsonpointer-2.0 moto-1.3.8 s3transfer-0.2.1 ``` 4) sshtunnel is upgraded to 0.1.5 5) We run `pip install -e .[all] 6) It downgrades moto to 1.3.5 and downgrades few packages that are not compatible: ``` Installing collected packages: botocore, jsondiff, s3transfer, boto3, moto, apache-airflow Found existing installation: botocore 1.12.179 Uninstalling botocore-1.12.179: Successfully uninstalled botocore-1.12.179 Found existing installation: jsondiff 1.1.2 Uninstalling jsondiff-1.1.2: Successfully uninstalled jsondiff-1.1.2 Found existing installation: s3transfer 0.2.1 Uninstalling s3transfer-0.2.1: Successfully uninstalled s3transfer-0.2.1 Found existing installation: boto3 1.9.179 Uninstalling boto3-1.9.179: Successfully uninstalled boto3-1.9.179 Found existing installation: moto 1.3.8 Uninstalling moto-1.3.8: Successfully uninstalled moto-1.3.8 Found existing installation: apache-airflow 2.0.0.dev0 Uninstalling apache-airflow-2.0.0.dev0: Successfully uninstalled apache-airflow-2.0.0.dev0 Running setup.py develop for apache-airflow Successfully installed apache-airflow boto3-1.7.84 botocore-1.10.84 jsondiff-1.1.1 moto-1.3.5 s3transfer-0.1.13 ``` 7) This way we updated sshtunnel to latest version compatible with setup.py and we kept moto at 1.3.5 - it should be equivalent of running `pip install -e .[all]` in a fresh virtualenv. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services