This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/airflow.git
The following commit(s) were added to refs/heads/master by this push:
new 09a2413 Improves documentation regarding providers and custom
connections 2 (#13410)
09a2413 is described below
commit 09a2413fe633292a8314328608e4d04feac19607
Author: Bijan Soltani <[email protected]>
AuthorDate: Fri Jan 1 20:40:09 2021 +0100
Improves documentation regarding providers and custom connections 2 (#13410)
---
docs/apache-airflow-providers/index.rst | 33 +++++++++++++++++++++++++++++++--
1 file changed, 31 insertions(+), 2 deletions(-)
diff --git a/docs/apache-airflow-providers/index.rst
b/docs/apache-airflow-providers/index.rst
index 2f067ce..e60b771 100644
--- a/docs/apache-airflow-providers/index.rst
+++ b/docs/apache-airflow-providers/index.rst
@@ -179,6 +179,9 @@ Creating your own providers
You do not need to do anything special besides creating the
``apache_airflow_provider`` entry point
returning properly formatted meta-data (dictionary with ``extra-links`` and
``hook-class-names`` fields).
+Anyone who runs airflow in an environment that has your Python package
installed will be able to use the
+package as a provider package.
+
**What do I need to do to turn a package into a provider?**
You need to do the following to turn an existing Python package into a
provider (see below for examples):
@@ -189,9 +192,9 @@ You need to do the following to turn an existing Python
package into a provider
dictionary that contains all meta-data about your provider package; see also
``provider.yaml``
files in the community managed provider packages as examples
-Example ``setup.cfg``: cfg
+Example ``setup.cfg``:
-.. code-block::
+.. code-block:: cfg
[options.entry_points]
# the function get_provider_info is defined in myproviderpackage.somemodule
@@ -213,6 +216,32 @@ Example ``myproviderpackage/somemodule.py``:
'versions': ["1.0.0"],
}
+**How do provider packages work under the hood?**
+
+When running airflow with your provider package, there will be (at least)
three components to your airflow installation:
+
+* The installation itself (for example, a ``venv`` where you installed airflow
with ``pip install apache-airflow``)
+ together with the related files (e.g. ``dags`` folder)
+* The ``apache-airflow`` package
+* Your own ``myproviderpackage`` package that is independent of
``apache-airflow`` or your airflow installation, which
+ can be a local Python package (that you install via ``pip pip install -e
/path/to/my-package``), a normal pip package
+ (``pip install myproviderpackage``), or any other type of Python package
+
+In the ``myproviderpackage`` package you need to add the entry point and
provide the appropriate metadata as described above.
+If you have done that, airflow does the following at runtime:
+
+* Loop through ALL packages installed in your environment / ``venv``
+* For each package, if the package's ``setup.cfg`` has a section
``[options.entry_points]``, and if that section has a value
+ for ``apache_airflow_provider``, then get the value for ``provider_info``,
e.g. ``myproviderpackage.somemodule:get_provider_info``
+* That value works like an import statement:
``myproviderpackage.somemodule:get_provider_info`` translates to something like
+ ``from myproviderpackage.somemodule import get_provider_info``, and the
``get_provider_info`` that is being imported should be a
+ callable, i.e. a function
+* This function should return a dictionary with metadata
+* If you have custom connection types as part of your package, that metadata
will including a field called ``hook-class-names``
+ which should be a list of strings of your custom hooks - those strings
should also be in an import-like format, e.g.
+ ``myproviderpackage.hooks.source.SourceHook`` means that there is a class
``SourceHook`` in ``myproviderpackage/hooks/source.py``
+ - airflow then imports these hooks and looks for the functions
``get_ui_field_behaviour`` and ``get_connection_form_widgets``
+ (both optional) as well as the attributes ``conn_type`` and ``hook_name`` to
create the custom connection type in the airflow UI
**Should I name my provider specifically or should it be created in
``airflow.providers`` package?**
