uranusjr commented on code in PR #34714: URL: https://github.com/apache/airflow/pull/34714#discussion_r1357874912
########## INSTALL.md: ########## @@ -0,0 +1,236 @@ +# INSTALL / BUILD instructions for Apache Airflow + +This is a generic installation method that requires a number of dependencies to be installed. + +Depending on your system you might need different prerequisites, but the following +systems/prerequisites are known to work: + +Linux (Debian Bullseye and Linux Mint Debbie): + + sudo apt install build-essential python3-dev libsqlite3-dev openssl \ + sqlite default-libmysqlclient-dev libmysqlclient-dev postgresql + +On Ubuntu 20.04 you may get an error of ```mariadb_config not found``` +and mysql_config not found. + +Install MariaDB development headers: + + sudo apt-get install libmariadb-dev libmariadbclient-dev + +MacOS (Mojave/Catalina): + + brew install sqlite mysql postgresql + +- [Optional] Fetch the tarball and untar the source move into the directory that was untarred. +- [Optional] Run Apache RAT (release audit tool) to validate license headers +- RAT docs here: + ```https://creadur.apache.org/rat/.``` +- Requires Java and Apache Rat: + + java -jar apache-rat.jar -E ./.rat-excludes -d +- [Required] Instead of fetching and untarring the source tarball, you can use `pip` to directly install Apache Airflow with a specified version. + + pip install apache-airflow==<version> + +**[optional] Airflow pulls in quite a lot of dependencies in order +to connect to other services. You might want to test or run Airflow +from a virtual env to make sure those dependencies are separated +from your system wide versions** + + python3 -m venv PATH_TO_YOUR_VENV + source PATH_TO_YOUR_VENV/bin/activate + +**[Required] Building and installing by pip (preferred)** + + pip install . + + +**You can also install recommended version of the dependencies by using +constraint- ```python<PYTHON_MAJOR_MINOR_VERSION>.txt``` files as constraint file. This is needed in case +you have problems with installing the current requirements from PyPI. +There are different constraint files for different python versions. For example:** + + pip install . \ + --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-main/constraints-3.8.txt"; + + +By default `pip install` in Airflow 2.0 installs only the provider packages that are needed by the extras and +install them as packages from PyPI rather than from local sources: + + pip install .[google,amazon] \ + --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-main/constraints-3.8.txt"; + + +You can upgrade just airflow, without paying attention to provider's dependencies by using 'constraints-no-providers' +constraint files. This allows you to keep installed provider packages. + + pip install . --upgrade \ + --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-main/constraints-no-providers-3.8.txt"; + + +You can also install airflow in "editable mode" (with `-e`) flag and then provider packages are +available directly from the sources (and the provider packages installed from `PyPI` are UNINSTALLED in +order to avoid having providers in two places. And `provider.yaml` files are used to discover capabilities +of the providers which are part of the airflow source code. + +You can read more about `provider.yaml` and community-managed providers in +`https://airflow.apache.org/docs/apache-airflow-providers/index.html` for developing custom providers +and in ``CONTRIBUTING.rst`` for developing community maintained providers. + +This is useful if you want to develop providers: + + pip install -e . \ + --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-main/constraints-3.8.txt"; + +You can also skip installing provider packages from `PyPI` by setting `INSTALL_PROVIDERS_FROM_SOURCE` to "true". +In this case Airflow will be installed in non-editable mode with all providers installed from the sources. +Additionally `provider.yaml` files will also be copied to providers folders which will make the providers +discoverable by Airflow even if they are not installed from packages in this case. + + INSTALL_PROVIDERS_FROM_SOURCES="true" pip install . \ + --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-main/constraints-3.8.txt"; + +Airflow can be installed with extras to install some additional features (for example 'async' or 'doc' or +to install automatically providers and all dependencies needed by that provider: + + pip install .[async,google,amazon] \ + --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-main/constraints-3.8.txt"; + +The list of available extras: Review Comment: Are extras still recommended for deployment? IIRC they are mostly kept only for compatibility, and for an actual production installation it is recommended to install the actual providers you want instead, which would pull in the dependencies themselves. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
