ashb commented on a change in pull request #4938: [AIRFLOW-4117] Multi-staging
Image - Travis CI tests [Step 3/3]
URL: https://github.com/apache/airflow/pull/4938#discussion_r299411577
##########
File path: Dockerfile
##########
@@ -278,42 +278,75 @@ RUN echo "Pip version: ${PIP_VERSION}"
RUN pip install --upgrade pip==${PIP_VERSION}
-# We are copying everything with airflow:airflow user:group even if we use
root to run the scripts
+ARG AIRFLOW_REPO=apache/airflow
+ENV AIRFLOW_REPO=${AIRFLOW_REPO}
+
+ARG AIRFLOW_BRANCH=master
+ENV AIRFLOW_BRANCH=${AIRFLOW_BRANCH}
+
+ENV
AIRFLOW_GITHUB_DOWNLOAD=https://raw.githubusercontent.com/${AIRFLOW_REPO}/${AIRFLOW_BRANCH}
+
+# We perform fresh dependency install at the beginning of each month from the
scratch
+# This way every month we re-test if fresh installation from the scratch
actually works
+# As opposed to incremental installations which does not upgrade already
installed packages unless it
+# is required by setup.py constraints.
+ARG BUILD_MONTH
+
+# We get Airflow dependencies (no Airflow sources) from the master version of
Airflow in order to avoid full
+# pip install layer cache invalidation when setup.py changes. This can be
reinstalled from the
+# latest master by increasing PIP_DEPENDENCIES_EPOCH_NUMBER.
+RUN mkdir -pv ${AIRFLOW_SOURCES}/airflow/bin \
+ && curl -L ${AIRFLOW_GITHUB_DOWNLOAD}/setup.py >${AIRFLOW_SOURCES}/setup.py \
+ && curl -L ${AIRFLOW_GITHUB_DOWNLOAD}/setup.cfg >${AIRFLOW_SOURCES}/setup.cfg
\
Review comment:
This feels like another image layer - rather than being in here. Would that
complicate things needlessly?
Is the goal here to speed up local dev, or does this help on PRs too? (If
the goal is to help local dev then setting a pip cache dir to a volume mount to
cache wheels/downloads go a long way towards speeding it up, without this odd
pattern. I'm not sure what I don't like about it.
I also wonder if we could do this differently:
```
RUN pip install --no-use-pep517 "pip install
"https://github.com/apache/airflow/archive/master.tar.gz#egg=apache-airflow[${AIRFLOW_EXTRAS}]";
\
&& pip uninstall --yes apache-airflow
```
Being able to do it in a single command/layer makes me happier.
(You can't use `-e` with an http download, but either way if we uninstall
the airflow package and leave the deps `-e` or not doesn't matter.)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
