mik-laj commented on a change in pull request #10282:
URL: https://github.com/apache/airflow/pull/10282#discussion_r468185626
##########
File path: docs/concepts.rst
##########
@@ -1207,6 +1211,62 @@ queue during retries:
ti.queue = 'retry_queue'
+Cluster Policies for Custom Task Checks
+-----------------------------
+You may also use Cluster Policies to apply cluster-wide checks on Airflow
+tasks. You can raise `AirflowClusterPolicyViolation` in a policy or
+task mutation hook (described below) to prevent a DAG from being imported or
+prevent a task from being executed if the task is not compliant with your
check.
+
+These checks are intended to help teams using Airflow to protect against common
+newbie errors that may get past a code reviewer, rather than as technical
+security controls.
+
+For example, don't run tasks without airflow owners:
+
+.. code-block:: python
+ def task_must_have_owners(task: BaseOperator):
+ if not task.owner or task.owner.lower() == "airflow":
+ raise AirflowClusterPolicyViolation(
+ f'''Task must have non-None non-'airflow' owner.
+ Current value: {task.owner}''')
+
+If you have multiple checks to apply, it is best practice to curate these rules
+in a separate python module and have a single policy / task mutation hook that
+performs multiple of these custom checks and aggregates the various error
+messages so that a single `AirflowClusterPolicyViolation` can be reported in
+the UI (and import errors table in the database).
+
+For Example in ``airflow_local_settings.py``:
+
+.. code-block:: python
+ from my_cluster_policies import rules
+
+ TASK_RULES: List[Callable[[BaseOperator], None]] = [
+ rules.task_must_have_owners,
+ ]
+
+ def _check_task_rule(current_task: BaseOperator):
+ """Check task rules for given task."""
+ notices = []
+ for rule in TASK_RULES:
+ try:
+ rule(current_task)
+ except ClusterPolicyViolation as ex:
+ notices.append(str(ex))
+
+ if notices:
+ current_dag = current_task.dag
+ notices_list = " * " + "\n * ".join(notices)
+ # AirflowDagCycleException is only supported exception type by
Airflow.
Review comment:
I think we can use AirflowClusterPolicyViolation also
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
