This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git
The following commit(s) were added to refs/heads/main by this push:
new e95156c26d Add notes about Virtualenvs preventing the need for custom
images (#35306)
e95156c26d is described below
commit e95156c26de627eeb376bbe6179a80ab9b4147e2
Author: Jens Scheffler <95105677+jens-scheffler-bo...@users.noreply.github.com>
AuthorDate: Tue Oct 31 20:33:44 2023 +0100
Add notes about Virtualenvs preventing the need for custom images (#35306)
---
docs/apache-airflow/howto/docker-compose/index.rst | 7 +++++++
docs/docker-stack/build.rst | 7 +++++++
docs/helm-chart/quick-start.rst | 7 +++++++
3 files changed, 21 insertions(+)
diff --git a/docs/apache-airflow/howto/docker-compose/index.rst
b/docs/apache-airflow/howto/docker-compose/index.rst
index d493d4ba02..1533d0c813 100644
--- a/docs/apache-airflow/howto/docker-compose/index.rst
+++ b/docs/apache-airflow/howto/docker-compose/index.rst
@@ -294,6 +294,13 @@ to rebuild the images on-the-fly when you run other
``docker compose`` commands.
Examples of how you can extend the image with custom providers, python
packages,
apt packages and more can be found in :doc:`Building the image
<docker-stack:build>`.
+.. note::
+ Creating custom images means that you need to maintain also a level of
automation as you need to re-create the images
+ when either the packages you want to install or Airflow is upgraded. Please
do not forget about keeping these scripts.
+ Also keep in mind, that in cases when you run pure Python tasks, you can
use the
+ `Python Virtualenv functions <_howto/operator:PythonVirtualenvOperator>`_
which will
+ dynamically source and install python dependencies during runtime. With
Airflow 2.8.0 Virtualenvs can also be cached.
+
Special case - adding dependencies via requirements.txt file
============================================================
diff --git a/docs/docker-stack/build.rst b/docs/docker-stack/build.rst
index 5bad72c003..4397d59eca 100644
--- a/docs/docker-stack/build.rst
+++ b/docs/docker-stack/build.rst
@@ -66,6 +66,13 @@ as ``root`` will fail with an appropriate error message.
downgrade or upgrade of airflow. If you upgrade airflow base image, you
should also update the version
to match the new version of airflow.
+.. note::
+ Creating custom images means that you need to maintain also a level of
automation as you need to re-create the images
+ when either the packages you want to install or Airflow is upgraded. Please
do not forget about keeping these scripts.
+ Also keep in mind, that in cases when you run pure Python tasks, you can
use the
+ `Python Virtualenv functions
<https://airflow.apache.org/docs/apache-airflow/stable/howto/operator/python.html#pythonvirtualenvoperator>`_
+ which will dynamically source and install python dependencies during
runtime. With Airflow 2.8.0 Virtualenvs can also be cached.
+
.. exampleinclude:: docker-examples/extending/add-pypi-packages/Dockerfile
:language: Dockerfile
:start-after: [START Dockerfile]
diff --git a/docs/helm-chart/quick-start.rst b/docs/helm-chart/quick-start.rst
index 009c6e0d9d..df8239917e 100644
--- a/docs/helm-chart/quick-start.rst
+++ b/docs/helm-chart/quick-start.rst
@@ -91,6 +91,13 @@ The Apache Airflow community, releases Docker Images which
are ``reference image
However, when you try it out you want to add your own DAGs, custom
dependencies,
packages, or even custom providers.
+.. note::
+ Creating custom images means that you need to maintain also a level of
automation as you need to re-create the images
+ when either the packages you want to install or Airflow is upgraded. Please
do not forget about keeping these scripts.
+ Also keep in mind, that in cases when you run pure Python tasks, you can
use the
+ `Python Virtualenv functions
<https://airflow.apache.org/docs/apache-airflow/stable/howto/operator/python.html#pythonvirtualenvoperator>`_
+ which will dynamically source and install python dependencies during
runtime. With Airflow 2.8.0 Virtualenvs can also be cached.
+
The best way to achieve it, is to build your own, custom image.
Adding DAGs to your image
