This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git


The following commit(s) were added to refs/heads/main by this push:
     new f36ba1d282 Update instructions on how to update dependencies in Breeze 
(#33318)
f36ba1d282 is described below

commit f36ba1d282bb33827bd0ab28ed9daeed22e1d43b
Author: Jarek Potiuk <[email protected]>
AuthorDate: Fri Aug 11 18:53:45 2023 +0200

    Update instructions on how to update dependencies in Breeze (#33318)
    
    Noticed that the instructions were slightly out-dated. This change
    gives the precise step-by-step instructions on how to add Python,
    system and node dependencies to get them reflected in Breeze.
---
 BREEZE.rst | 107 ++++++++++++++++++++++++++++++++++++++++++++++++++-----------
 1 file changed, 89 insertions(+), 18 deletions(-)

diff --git a/BREEZE.rst b/BREEZE.rst
index 98f0b87f66..d7550b4506 100644
--- a/BREEZE.rst
+++ b/BREEZE.rst
@@ -2427,29 +2427,100 @@ If you set these variables, next time when you enter 
the environment the new por
 Managing Dependencies
 ---------------------
 
-If you need to change apt dependencies in the ``Dockerfile.ci``, add Python 
packages in ``setup.py``
-for airflow and in provider.yaml for packages. If you add any "node" 
dependencies in ``airflow/www``
-, you need to compile them in the host with ``breeze compile-www-assets`` 
command.
+There are couple of things you might want to do when adding/changing 
dependencies when developing with
+Breeze. You can add dependencies temporarily (which will last until you exit 
Breeze shell), or you might
+want to add them permanently (which require you to rebuild the image). Also 
there are different things
+you need to do when you are adding system level (debian) level, Python (pip) 
dependencies or Node (yarn)
+dependencies for the webserver.
 
-Adding Dependencies Permanently
-...............................
+Python dependencies
+~~~~~~~~~~~~~~~~~~~
+
+For temporary adding and modifying the dependencies, you just (in Breeze 
shell) run
+``pip install <dependency>`` or similar - in the same way as you would do it
+in your local environment. You can also use ``pip install -r 
/files/requirements.txt`` to install several
+dependencies - if you place your requirements file in ``files`` directory. 
Those dependencies will
+disappear when you exit Breeze shell.
+
+When you want to add dependencies permanently, then it depends what kind of 
dependency you add.
+
+If you want to add core dependency that should always be installed - you need 
to add it to ``setup.cfg``
+to ``install_requires`` section. If you want to add it to one of the optional 
core extras, you should
+add it in the extra definition in ``setup.py`` (you need to find out where it 
is defined). If you want
+to add it to one of the providers, you need to add it to the ``provider.yaml`` 
file in the provider
+directory - but remember that this should be followed by running pre-commit 
that will automatically update
+the ``generated/provider_dependencies.json`` directory with the new 
dependencies:
+
+```
+pre-commit run update-providers-dependencies  --all-files
+```
+
+You can also run the pre-commit by ``breeze static-checks --type 
update-providers-dependencies --all-files``
+command - which provides autocomplete.
+
+After you've updated the dependencies, you need to rebuild the image:
+
+Breeze will automatically detect when you updated dependencies and it will 
propose you to build image next
+time when you enter it. You can answer ``y`` during 10 seconds to get it done 
for you.
+
+```
+breeze ci-image build
+```
+
+Sometimes, when you have conflicting change in dependencies (i.e. dependencies 
in the old constraints
+are conflicting with the new specification, you might want to build the image 
with
+``--upgrade-to-newer-dependencies`` flag:
+
+```
+breeze ci-image build --upgrade-to-newer-dependencies
+```
+
+
+System (debian) dependencies
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+You can install ``apt-get`` dependencies temporarily by running ``apt-get 
install <dependency>`` in
+Breeze shell. Those dependencies will disappear when you exit Breeze shell.
+
+When you want to add dependencies permanently, you need to add them to 
``Dockerfile.ci``. But you need to
+do it indirectly via shell scripts that get automatically inlined in the 
``Dockerfile.ci``. Those
+scripts are present in ``scripts/docker`` directory and are aptly (!) named 
``install*.sh``. Most
+of the apt dependencies are installed in the ``install_os_dependencies.sh``, 
but some are installed in
+other scripts (for example ``install_postgres.sh`` or ``install_mysql.sh``).
+
+After you modify the dependencies in the scripts, you need to inline them by 
running pre-commit:
+
+```
+pre-commit run update-inlined-dockerfile-scripts --all-files
+```
+
+You can also run the pre-commit by ``breeze static-checks --type 
update-inlined-dockerfile-scripts --all-files``
+command - which provides autocomplete.
+
+
+After you've updated the dependencies, you need to rebuild the image:
+
+Breeze will automatically detect when you updated dependencies and it will 
propose you to build image next
+time when you enter it. You can answer ``y`` during 10 seconds to get it done 
for you.
+
+```
+breeze ci-image build
+```
+
+Sometimes, when you have conflicting change in dependencies (i.e. dependencies 
in the old constraints
+are conflicting with the new specification, you might want to build the image 
with
+``--upgrade-to-newer-dependencies`` flag:
 
-You can add dependencies to the ``Dockerfile.ci``, ``setup.py``.
-After you exit the container and re-run ``breeze``, Breeze detects changes in 
dependencies,
-asks you to confirm rebuilding the image and proceeds with rebuilding if you 
confirm (or skip it
-if you do not confirm). After rebuilding is done, Breeze drops you to shell. 
You may also use the
-``build`` command to only rebuild CI image and not to go into shell.
+```
+breeze ci-image build --upgrade-to-newer-dependencies
+```
 
-Incremental apt Dependencies in the Dockerfile.ci during development
-....................................................................
+Node (yarn) dependencies
+~~~~~~~~~~~~~~~~~~~~~~~~
 
-During development, changing dependencies in ``apt-get`` closer to the top of 
the ``Dockerfile.ci``
-invalidates cache for most of the image. It takes long time for Breeze to 
rebuild the image.
-So, it is a recommended practice to add new dependencies initially closer to 
the end
-of the ``Dockerfile.ci``. This way dependencies will be added incrementally.
+If you need to change "node" dependencies in ``airflow/www``, you need to 
compile them in the
+host with ``breeze compile-www-assets`` command. No need to rebuild the image.
 
-Before merge, these dependencies should be moved to the appropriate ``apt-get 
install`` command,
-which is already in the ``Dockerfile.ci``.
 
 Recording command output
 ========================

Reply via email to