mik-laj commented on a change in pull request #10282:
URL: https://github.com/apache/airflow/pull/10282#discussion_r468911155
##########
File path: docs/concepts.rst
##########
@@ -1207,6 +1211,66 @@ queue during retries:
ti.queue = 'retry_queue'
+Cluster Policies for Custom Task Checks
+-----------------------------
+You may also use Cluster Policies to apply cluster-wide checks on Airflow
+tasks. You can raise :class:`~airflow.exceptions.AirflowClusterPolicyViolation`
+in a policy or task mutation hook (described below) to prevent a DAG from being
+imported or prevent a task from being executed if the task is not compliant
with
+your check.
+
+These checks are intended to help teams using Airflow to protect against common
+beginner errors that may get past a code reviewer, rather than as technical
+security controls.
+
+For example, don't run tasks without airflow owners:
+
+.. code-block:: python
+ def task_must_have_owners(task: BaseOperator):
+ if not task.owner or task.owner.lower() == "airflow":
+ raise AirflowClusterPolicyViolation(
+ f'''Task must have non-None non-'airflow' owner.
+ Current value: {task.owner}''')
+
+If you have multiple checks to apply, it is best practice to curate these rules
+in a separate python module and have a single policy / task mutation hook that
+performs multiple of these custom checks and aggregates the various error
+messages so that a single `AirflowClusterPolicyViolation` can be reported in
+the UI (and import errors table in the database).
+
+For Example in ``airflow_local_settings.py``:
+
+.. code-block:: python
+ from my_cluster_policies import rules
Review comment:
Ideally, the documentation should not contain any Python code, and if
needed, it would be loaded from the Python file and the Python files covered
with tests. It's going to be simple at this point, because you just need to
move this code snippet to variable ``SETTINGS_FILE_MUST_HAVE_OWNER_POLICY``.
Here is docs about this directive:
https://www.sphinx-doc.org/en/1.7/markup/code.html
You can even use `lines`, `start-after` and `end-before` if you only want to
include a part of the file.
Here is an example from project:
https://github.com/apache/airflow/blob/master/docs/howto/operator/google/cloud/dataproc.rst
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
