Hi all,
I'd like to propose adding Variable.list() to the Task SDK to address
the gap left by the removal of direct ORM access in Airflow 3.
Background:
In Airflow 2.x, users could list all variables via:
from airflow.models import Variable
from airflow.utils.session import create_session
with create_session() as session:
variables = session.query(Variable).all()
In Airflow 3.x, this pattern raises:
RuntimeError: Direct database access via the ORM is not allowed in Airflow
3.0
There is currently no supported way to discover variable keys dynamically
when they are not known at DAG authoring time.
Proposal:
- Add Variable.list(prefix=None) to the Task SDK
- Scope is limited to the metadata database only (same as the old ORM pattern)
- Secrets backend support is intentionally out of scope, as it would
require a broader interface contract change and separate community
discussion
Related issue: https://github.com/apache/airflow/issues/61166
Draft PR: https://github.com/apache/airflow/pull/66022
I would appreciate any feedback or concerns from the community before
this moves forward.
Best Regards,
Jun Yeong Kim
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
