[
https://issues.apache.org/jira/browse/AIRFLOW-5389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jarek Potiuk resolved AIRFLOW-5389.
-----------------------------------
Fix Version/s: 1.10.5
Resolution: Fixed
> Better organised/faster build for external dependencies in ci build
> -------------------------------------------------------------------
>
> Key: AIRFLOW-5389
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5389
> Project: Apache Airflow
> Issue Type: Improvement
> Components: ci
> Affects Versions: 2.0.0, 1.10.5
> Reporter: Jarek Potiuk
> Priority: Major
> Fix For: 1.10.5
>
>
> There are several entries with if's for CI image in the Dockerfile. They have
> weird syntax and they are slow (each if line is separately evaluated while
> Docker build).
> They can be moved to bash scripts and replaced with single RUN command (one
> per CI deps and one for main file). Then we will also use the power of
> shellcheck to verify those scripts.
> In the future, we also think about unifying the dockerfiles later one
> (production/CI/SLIM without multi-staging) and removing installation of
> hive/hadoop/minicluster to separate images. This might be first step to do so.
> Examples of the current code:
> CI deps:
> {code:java}
> RUN \
> if [[ "${APT_DEPS_IMAGE}" == "airflow-apt-deps-ci" ]]; then \
> mkdir -pv ${HADOOP_HOME} \
> && mkdir -pv ${HIVE_HOME} \
> && mkdir /tmp/minicluster \
> && mkdir -pv /user/hive/warehouse \
> && chmod -R 777 ${HIVE_HOME} \
> && chmod -R 777 /user/ \
> ;\
> fi
> # Install Hadoop
> # --absolute-names is a work around to avoid this issue
> https://github.com/docker/hub-feedback/issues/727
> RUN \
> if [[ "${APT_DEPS_IMAGE}" == "airflow-apt-deps-ci" ]]; then \
>
> HADOOP_URL=${HADOOP_URL}hadoop-${HADOOP_VERSION}-${HADOOP_DISTRO}${HADOOP_DISTRO_VERSION}.tar.gz
> \
> && HADOOP_TMP_FILE=/tmp/hadoop.tar.gz \
> && curl -sL ${HADOOP_URL} > ${HADOOP_TMP_FILE} \
> && tar xzf ${HADOOP_TMP_FILE} --absolute-names --strip-components 1 -C
> ${HADOOP_HOME} \
> && rm ${HADOOP_TMP_FILE} \
> ;\
> fi{code}
>
> main:
> {code:java}
> if [[ "${APT_DEPS_IMAGE}" == "airflow-apt-deps-ci" ]]; then \
> chmod +x "${HOME}/run-tests-complete" "${HOME}/run-tests" && \
> chown "${AIRFLOW_USER}.${AIRFLOW_USER}" "${HOME}/.bashrc" \
> "${HOME}/run-tests-complete" "${HOME}/run-tests"; \
> fi
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
