This is an automated email from the ASF dual-hosted git repository.
dstandish pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git
The following commit(s) were added to refs/heads/main by this push:
new 66de4bdc9e Update public interface doc re operators (#36767)
66de4bdc9e is described below
commit 66de4bdc9e983cedfab29ccffeee3c503ba78b52
Author: Daniel Standish <15932138+dstand...@users.noreply.github.com>
AuthorDate: Fri Jan 19 09:51:49 2024 -0800
Update public interface doc re operators (#36767)
Base classes such as BaseOperator and BaseSensorOperator are public in the
traditional sense. We publish them explicitly for the purpose of being
extended.
But I think that derivatives of them are published only as end products, in
that their behavior and signature should be subject to semver, but not their
structure, which should be considered "internal" and not user-facing. Users
can extend these classes but they should do so at their own risk, with the
knowledge that we might refactor. Otherwise the "public interface" is just
unnecessarily big.
---
airflow/models/baseoperator.py | 6 ++-
airflow/operators/__init__.py | 7 ++-
airflow/sensors/__init__.py | 6 ++-
docs/apache-airflow/public-airflow-interface.rst | 66 +++++-------------------
docs/conf.py | 2 +
5 files changed, 30 insertions(+), 57 deletions(-)
diff --git a/airflow/models/baseoperator.py b/airflow/models/baseoperator.py
index 2d0244dbbf..1df771231f 100644
--- a/airflow/models/baseoperator.py
+++ b/airflow/models/baseoperator.py
@@ -15,7 +15,11 @@
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
-"""Base operator for all operators."""
+"""
+Base operator for all operators.
+
+:sphinx-autoapi-skip:
+"""
from __future__ import annotations
import abc
diff --git a/airflow/operators/__init__.py b/airflow/operators/__init__.py
index f79046e5ae..7fc63e1c8f 100644
--- a/airflow/operators/__init__.py
+++ b/airflow/operators/__init__.py
@@ -16,7 +16,12 @@
# specific language governing permissions and limitations
# under the License.
# fmt: off
-"""Operators."""
+"""
+Operators.
+
+:sphinx-autoapi-skip:
+"""
+
from __future__ import annotations
from airflow.utils.deprecation_tools import add_deprecated_classes
diff --git a/airflow/sensors/__init__.py b/airflow/sensors/__init__.py
index e987f6ef61..3d1627cc03 100644
--- a/airflow/sensors/__init__.py
+++ b/airflow/sensors/__init__.py
@@ -16,7 +16,11 @@
# specific language governing permissions and limitations
# under the License.
# fmt: off
-"""Sensors."""
+"""
+Sensors.
+
+:sphinx-autoapi-skip:
+"""
from __future__ import annotations
from airflow.utils.deprecation_tools import add_deprecated_classes
diff --git a/docs/apache-airflow/public-airflow-interface.rst
b/docs/apache-airflow/public-airflow-interface.rst
index bba6c2795a..c960c9c805 100644
--- a/docs/apache-airflow/public-airflow-interface.rst
+++ b/docs/apache-airflow/public-airflow-interface.rst
@@ -18,9 +18,9 @@
Public Interface of Airflow
...........................
-The Public Interface of Apache Airflow is a set of interfaces that allow
developers to interact
-with and access certain features of the Apache Airflow system. This includes
operations such as
-creating and managing DAGs (Directed Acyclic Graphs), managing tasks and their
dependencies,
+The Public Interface of Apache Airflow is the collection of interfaces and
behaviors in Apache Airflow
+whose changes are governed by semantic versioning. A user interacts with
Airflow's public interface
+by creating and managing DAGs, managing tasks and dependencies,
and extending Airflow capabilities by writing new executors, plugins,
operators and providers. The
Public Interface can be useful for building custom tools and integrations with
other systems,
and for automating certain aspects of the Airflow workflow.
@@ -28,17 +28,19 @@ and for automating certain aspects of the Airflow workflow.
Using Airflow Public Interfaces
===============================
-Using Airflow Public Interfaces is needed when you want to interact with
Airflow programmatically:
+The following are some examples of the public interface of Airflow:
-* When you are extending Airflow classes such as Operators and Hooks. This can
be done by DAG authors to add missing functionality in their DAGs or by those
who write reusable custom operators for other DAG authors.
+* When you are writing your own operators or hooks. This commonly done when no
hook or operator exists for your use case, or when perhaps when one exists but
you need to customize the behavior.
* When writing new :doc:`Plugins <authoring-and-scheduling/plugins>` that
extend Airflow's functionality beyond
DAG building blocks. Secrets, Timetables, Triggers, Listeners are all
examples of such functionality. This
is usually done by users who manage Airflow instances.
* Bundling custom Operators, Hooks, Plugins and releasing them together via
:doc:`provider packages <apache-airflow-providers:index>` - this is usually
done by those who intend to
provide a reusable set of functionality for external services or
applications Airflow integrates with.
+* Using the taskflow API to write tasks
+* Relying on the consistent behavior of Airflow objects
-All the ways above involve extending or using Airflow Python classes and
functions. The classes
+One aspect of "public interface" is extending or using Airflow Python classes
and functions. The classes
and functions mentioned below can be relied on to keep backwards-compatible
signatures and behaviours within
MAJOR version of Airflow. On the other hand, classes and methods starting with
``_`` (also known
as protected Python methods) and ``__`` (also known as private Python methods)
are not part of the Public
@@ -73,8 +75,6 @@ Airflow has a set of example DAGs that you can use to learn
how to write DAGs
You can read more about DAGs in :doc:`DAGs <core-concepts/dags>`.
-.. _pythonapi:operators:
-
References for the modules used in DAGs are here:
.. toctree::
@@ -95,57 +95,14 @@ Properties of a :class:`~airflow.models.dagrun.DagRun` can
also be referenced in
_api/airflow/models/dagrun/index
+.. _pythonapi:operators:
Operators
---------
-Operators allow for generation of certain types of tasks that become nodes in
-the DAG when instantiated.
-
-There are 3 main types of operators:
-
-- Operators that performs an **action**, or tell another system to
- perform an action
-- **Transfer** operators move data from one system to another
-- **Sensors** are a certain type of operator that will keep running until a
- certain criterion is met. Examples include a specific file landing in HDFS or
- S3, a partition appearing in Hive, or a specific time of the day. Sensors
- are derived from :class:`~airflow.sensors.base.BaseSensorOperator` and run a
poke
- method at a specified
:attr:`~airflow.sensors.base.BaseSensorOperator.poke_interval` until it
- returns ``True``.
-
-All operators are derived from
:class:`~airflow.models.baseoperator.BaseOperator` and acquire much
-functionality through inheritance. Since this is the core of the engine,
-it's worth taking the time to understand the parameters of
:class:`~airflow.models.baseoperator.BaseOperator`
-to understand the primitive features that can be leveraged in your DAGs.
-
-Airflow has a set of Operators that are considered public. You are also free
to extend their functionality
-by extending them:
-
-.. toctree::
- :includehidden:
- :glob:
- :maxdepth: 1
-
- _api/airflow/operators/index
-
- _api/airflow/sensors/index
-
-
-You can read more about the operators in :doc:`core-concepts/operators`,
:doc:`core-concepts/sensors`.
-Also you can learn how to write a custom operator in
:doc:`howto/custom-operator`.
-
-.. _pythonapi:hooks:
-
-References for the modules used in for operators are here:
-
-.. toctree::
- :includehidden:
- :glob:
- :maxdepth: 1
-
- _api/airflow/models/baseoperator/index
+The base classes :class:`~airflow.models.baseoperator.BaseOperator` and
:class:`~airflow.sensors.base.BaseSensorOperator` are public and may be
extended to make new operators.
+Subclasses of BaseOperator which are published in Apache Airflow are public in
*behavior* but not in *structure*. That is to say, the Operator's parameters
and behavior is governed by semver but the methods are subject to change at any
time.
Task Instances
--------------
@@ -175,6 +132,7 @@ Task instance keys are unique identifiers of task instances
in a DAG (in a DAG R
_api/airflow/models/taskinstancekey/index
+.. _pythonapi:hooks:
Hooks
-----
diff --git a/docs/conf.py b/docs/conf.py
index 73217330c9..069153e90a 100644
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -197,6 +197,8 @@ if PACKAGE_NAME == "apache-airflow":
exclude_patterns = [
# We only link to selected subpackages.
"_api/airflow/index.rst",
+ # "_api/airflow/operators/index.rst",
+ # "_api/airflow/sensors/index.rst",
# Included in the cluster-policies doc
"_api/airflow/policies/index.rst",
"README.rst",
