Andrushika commented on code in PR #69107:
URL: https://github.com/apache/airflow/pull/69107#discussion_r3490261503
##########
providers/keycloak/src/airflow/providers/keycloak/auth_manager/keycloak_auth_manager.py:
##########
@@ -457,10 +458,27 @@ def filter_authorized_dag_ids(
cache_key = (user.get_id(), method, team_name, frozenset(dag_ids))
def query_keycloak() -> set[str]:
- kwargs: dict = dict(dag_ids=dag_ids, user=user, method=method)
- if team_name is not None:
- kwargs["team_name"] = team_name
- return super(KeycloakAuthManager,
self).filter_authorized_dag_ids(**kwargs)
+ if not dag_ids:
+ return set()
+
+ max_workers = min(
+ len(dag_ids), conf.getint(CONF_SECTION_NAME,
CONF_REQUESTS_POOL_SIZE_KEY, fallback=10)
+ )
+
+ def check(dag_id: str) -> tuple[str, bool]:
+ details_kwargs: dict[str, Any] = {"id": dag_id}
+ if team_name is not None:
+ details_kwargs["team_name"] = team_name
+ return dag_id, self.is_authorized_dag(
Review Comment:
Good question! I didn't run an end-to-end benchmark against a real Keycloak
instance, and I'm still waiting for the reporter's response. The benchmark I
ran mocks the HTTP session, so it doesn't really trigger the request.
Since the HTTP-mocked benchmark still shows ~10s wall time on the old code,
the bottleneck is in our serial implementation rather than the HTTP layer. So I
think parallelizing is the right fix here, regardless of whether the HTTP
client is also a bottleneck in production.
On throughput beyond the pool size, it shouldn't help: every
`is_authorized_dag()` call shares the same `requests.Session`, and once all
connections in `HTTPAdapter(pool_maxsize=N)` are checked out, extra threads
block and wait for one to free up. That's why `max_workers` is capped at the
pool size.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
