This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git
The following commit(s) were added to refs/heads/main by this push: new e95156c26d Add notes about Virtualenvs preventing the need for custom images (#35306) e95156c26d is described below commit e95156c26de627eeb376bbe6179a80ab9b4147e2 Author: Jens Scheffler <95105677+jens-scheffler-bo...@users.noreply.github.com> AuthorDate: Tue Oct 31 20:33:44 2023 +0100 Add notes about Virtualenvs preventing the need for custom images (#35306) --- docs/apache-airflow/howto/docker-compose/index.rst | 7 +++++++ docs/docker-stack/build.rst | 7 +++++++ docs/helm-chart/quick-start.rst | 7 +++++++ 3 files changed, 21 insertions(+) diff --git a/docs/apache-airflow/howto/docker-compose/index.rst b/docs/apache-airflow/howto/docker-compose/index.rst index d493d4ba02..1533d0c813 100644 --- a/docs/apache-airflow/howto/docker-compose/index.rst +++ b/docs/apache-airflow/howto/docker-compose/index.rst @@ -294,6 +294,13 @@ to rebuild the images on-the-fly when you run other ``docker compose`` commands. Examples of how you can extend the image with custom providers, python packages, apt packages and more can be found in :doc:`Building the image <docker-stack:build>`. +.. note:: + Creating custom images means that you need to maintain also a level of automation as you need to re-create the images + when either the packages you want to install or Airflow is upgraded. Please do not forget about keeping these scripts. + Also keep in mind, that in cases when you run pure Python tasks, you can use the + `Python Virtualenv functions <_howto/operator:PythonVirtualenvOperator>`_ which will + dynamically source and install python dependencies during runtime. With Airflow 2.8.0 Virtualenvs can also be cached. + Special case - adding dependencies via requirements.txt file ============================================================ diff --git a/docs/docker-stack/build.rst b/docs/docker-stack/build.rst index 5bad72c003..4397d59eca 100644 --- a/docs/docker-stack/build.rst +++ b/docs/docker-stack/build.rst @@ -66,6 +66,13 @@ as ``root`` will fail with an appropriate error message. downgrade or upgrade of airflow. If you upgrade airflow base image, you should also update the version to match the new version of airflow. +.. note:: + Creating custom images means that you need to maintain also a level of automation as you need to re-create the images + when either the packages you want to install or Airflow is upgraded. Please do not forget about keeping these scripts. + Also keep in mind, that in cases when you run pure Python tasks, you can use the + `Python Virtualenv functions <https://airflow.apache.org/docs/apache-airflow/stable/howto/operator/python.html#pythonvirtualenvoperator>`_ + which will dynamically source and install python dependencies during runtime. With Airflow 2.8.0 Virtualenvs can also be cached. + .. exampleinclude:: docker-examples/extending/add-pypi-packages/Dockerfile :language: Dockerfile :start-after: [START Dockerfile] diff --git a/docs/helm-chart/quick-start.rst b/docs/helm-chart/quick-start.rst index 009c6e0d9d..df8239917e 100644 --- a/docs/helm-chart/quick-start.rst +++ b/docs/helm-chart/quick-start.rst @@ -91,6 +91,13 @@ The Apache Airflow community, releases Docker Images which are ``reference image However, when you try it out you want to add your own DAGs, custom dependencies, packages, or even custom providers. +.. note:: + Creating custom images means that you need to maintain also a level of automation as you need to re-create the images + when either the packages you want to install or Airflow is upgraded. Please do not forget about keeping these scripts. + Also keep in mind, that in cases when you run pure Python tasks, you can use the + `Python Virtualenv functions <https://airflow.apache.org/docs/apache-airflow/stable/howto/operator/python.html#pythonvirtualenvoperator>`_ + which will dynamically source and install python dependencies during runtime. With Airflow 2.8.0 Virtualenvs can also be cached. + The best way to achieve it, is to build your own, custom image. Adding DAGs to your image