1fanwang opened a new pull request, #71341:
URL: https://github.com/apache/airflow/pull/71341
Closes: #71309
## Why
Every list endpoint that hides unauthorized dags starts by materializing the
full set of dag
ids the caller may see, then passes that set to `dag_id IN (...)`.
`FabAuthManager` builds it
with a query that reads every row of `dag` when the user has a broad grant:
```python
return {dag.dag_id for dag in session.execute(select(DagModel.dag_id))}
```
So a request for 50 dags loads every dag id in the deployment, ships them
back to the API
server, and sends them out again as bind parameters. The work scales with
the size of the
deployment rather than the size of the page.
The manager already holds those grants in the same database the query runs
against, so it can
express the answer as SQL instead of as a set.
## What changed
`BaseAuthManager` gains `get_authorized_dag_ids_select`, which returns a
select of dag ids or
`None`. `None` is the default and keeps today's behaviour, so managers
backed by an external
policy service (Keycloak, Amazon Verified Permissions) are unaffected.
When a manager does return a select, the API applies it as `dag_id IN
(subquery)`. Filtering
and pagination then happen in one statement, and no dag ids cross the
process boundary.
`FabAuthManager` implements it by building the grant query it already knows
how to write.
## Testing Done
The filter subclasses each override `to_orm`, and every one of them wrote
`self.value or set()`. A select has no truth value, so that raises rather
than falling back.
Reverting only the accessor, keeping the tests, shows it:
```
$ pytest airflow-core/tests/unit/api_fastapi/core_api/test_security.py \
-k PermittedDagFilterSubquery -q
E TypeError: Boolean value of this clause is not defined
E TypeError: Boolean value of this clause is not defined
E TypeError: Boolean value of this clause is not defined
3 failed, 2 passed, 111 deselected
```
With the accessor in place:
```
$ pytest airflow-core/tests/unit/api_fastapi/core_api/test_security.py \
-k PermittedDagFilterSubquery -q
5 passed, 111 deselected
$ pytest
airflow-core/tests/unit/api_fastapi/auth/managers/test_base_auth_manager.py -q
61 passed
```
The five cover the default returning `None` and the set path staying as it
was, a returned
select reaching the SQL as a subquery rather than as bind parameters, an
empty select being
honoured as "nothing is permitted" instead of collapsing to no filter, and a
subclass filter
(`DagRun`) getting the same treatment as the base one.
The `create_app` fixtures in that file error out locally on `No module named
'airflow.www'`,
which is a stale plugin in my environment and unrelated to this change; CI
runs them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
