This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/airflow.git
The following commit(s) were added to refs/heads/master by this push:
new ec962b0 Adds dill exclusion to Dockerfiles to accomodate upcoming
beam fix (#15048)
ec962b0 is described below
commit ec962b01b72f76d6d2193342e52110996246a45a
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Sat Mar 27 23:41:53 2021 +0100
Adds dill exclusion to Dockerfiles to accomodate upcoming beam fix (#15048)
* Upgrades moto to newer version (~=2.0)
According to
https://github.com/spulec/moto/issues/3535#issuecomment-808706939
1.3.17 version of moto with a fix to be compatible with mock> 4.0.3 is
not going to be released because of breaking changes. Therefore we need
to migrate to newer version of moto.
At the same time we can get rid of the old botocore limitation, which
was added apparently to handle some test errors. We are relying fully
on what boto3 depends on.
Upgrading dependencies also discovered that mysql tests need to
be fixed because upgraded version of dependencies cause some test
failure (those turned out to be badly written tests).
* Adds dill exclusion to Dockerfiles to accomodate upcoming beam fix
With the upcoming apache-beam change where mock library will be
removed from install dependencies, we will be able to remove
`apache-beam` exclusion in our CI scripts. This will be a final
step of cleaning dependencies so that we have a truly
golden set of constraints that will allow to install airflow
and all community managed providers (we managed to fix all those
dependency issues for all packages but apache-beam).
The fix https://github.com/apache/beam/pull/14328 when merged
and Apache Beam is released will allow us to migrate to the new
version and get rid of the CI exclusion for beam.
Closes: #14994
---
Dockerfile | 3 ++-
Dockerfile.ci | 3 ++-
setup.py | 7 +++++--
3 files changed, 9 insertions(+), 4 deletions(-)
diff --git a/Dockerfile b/Dockerfile
index 5f8e666..202aa8f 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -276,7 +276,8 @@ ENV INSTALL_FROM_PYPI=${INSTALL_FROM_PYPI}
# * chardet<4 - required to keep snowflake happy
# * urllib3 - required to keep boto3 happy
# * pyjwt<2.0.0: flask-jwt-extended requires it
-ARG EAGER_UPGRADE_ADDITIONAL_REQUIREMENTS="chardet<4 urllib3<1.26 pyjwt<2.0.0"
+# * dill<0.3.3 required by apache-beam
+ARG EAGER_UPGRADE_ADDITIONAL_REQUIREMENTS="chardet<4 urllib3<1.26 pyjwt<2.0.0
dill<0.3.3"
WORKDIR /opt/airflow
diff --git a/Dockerfile.ci b/Dockerfile.ci
index 02a4325..46f0b36 100644
--- a/Dockerfile.ci
+++ b/Dockerfile.ci
@@ -348,7 +348,8 @@ COPY airflow/__init__.py
${AIRFLOW_SOURCES}/airflow/__init__.py
# * pyOpenSSL: required by snowflake provider
https://github.com/snowflakedb/snowflake-connector-python/blob/v2.3.6/setup.py#L201
# * urllib3<1.26: Required to keep boto3 happy
# * pyjwt<2.0.0: flask-jwt-extended requires it
-ARG EAGER_UPGRADE_ADDITIONAL_REQUIREMENTS="chardet<4 lazy-object-proxy<1.5.0
pyOpenSSL<20.0.0 urllib3<1.26 pyjwt<2.0.0"
+# * dill<0.3.3 required by apache-beam
+ARG EAGER_UPGRADE_ADDITIONAL_REQUIREMENTS="chardet<4 lazy-object-proxy<1.5.0
pyOpenSSL<20.0.0 urllib3<1.26 pyjwt<2.0.0 dill<0.3.3"
ENV
EAGER_UPGRADE_ADDITIONAL_REQUIREMENTS=${EAGER_UPGRADE_ADDITIONAL_REQUIREMENTS}
ARG CONTINUE_ON_PIP_CHECK_FAILURE="false"
diff --git a/setup.py b/setup.py
index 6a8aa00..a3662ab 100644
--- a/setup.py
+++ b/setup.py
@@ -743,8 +743,11 @@ PACKAGES_EXCLUDED_FOR_ALL.extend(
# This can be removed as soon as we get non-conflicting
# requirements for the apache-beam as well.
#
-# Currently Apache Beam has very narrow and old dependencies for 'dill' and
'mock' packages which
-# are required by our tests (but only for tests).
+# Currently Apache Beam has very narrow and old dependencies for 'mock'
package which
+# are required only for our tests.
+# once https://github.com/apache/beam/pull/14328 is solved and new version of
apache-beam is released
+# we will be able to remove this exclusion and get rid of
`install_remaining_dependencies`
+# function in `scripts/in_container`.
#
PACKAGES_EXCLUDED_FOR_CI = [
'apache-beam',
