nailo2c opened a new issue, #64201:
URL: https://github.com/apache/airflow/issues/64201
### Apache Airflow version
3.1.8
### If "Other Airflow 3 version" selected, which one?
_No response_
### What happened?
`airflow.sdk.configuration.conf` does not resolve config options defined in
provider metadata (`provider.yaml`). For example, `conf.has_option("celery",
"pool")` returns `False` on the SDK conf, while the core conf correctly returns
`True`.
This was discovered during the #60000 conf import migration.
`celery_command.py` uses [`conf.has_option("celery",
"pool")`](https://github.com/apache/airflow/blob/main/providers/celery/src/airflow/providers/celery/cli/celery_command.py#L293)
to decide whether to pass `--pool` to the Celery worker. After switching to
the compat SDK conf, this returns `False` and the `--pool prefork` argument is
silently dropped.
From reading the code, this may be caused by a circular dependency in the
lookup sequence. When `has_option()` triggers
`_get_option_from_provider_metadata_config_fallbacks`
([parser.py:318](https://github.com/apache/airflow/blob/main/shared/configuration/src/airflow_shared/configuration/parser.py#L318)),
the cached property `_provider_metadata_config_fallback_default_values`
([parser.py:342](https://github.com/apache/airflow/blob/main/shared/configuration/src/airflow_shared/configuration/parser.py#L342))
calls `ProvidersManagerTaskRuntime().provider_configs`, which in turn calls
`conf.load_providers_configuration()`. `load_providers_configuration()` uses
`already_initialized_provider_configs`
([parser.py:1202](https://github.com/apache/airflow/blob/main/shared/configuration/src/airflow_shared/configuration/parser.py#L1202))
which may return an empty dict if initialization hasn't completed yet.
### What you think should happen instead?
`sdk_conf.has_option("celery", "pool")` should return `True`, consistent
with `core_conf.has_option("celery", "pool")`.
### How to reproduce
```python
from airflow.configuration import conf as core_conf
from airflow.sdk.configuration import conf as sdk_conf
print("core:", core_conf.has_option("celery", "pool")) # True
print("sdk:", sdk_conf.has_option("celery", "pool")) # False — expected
True
```
`celery.pool` is defined in
[`providers/celery/provider.yaml:287`](https://github.com/apache/airflow/blob/main/providers/celery/provider.yaml#L287)
with default `prefork`.
### Operating System
Breeze (Debian 12)
### Versions of Apache Airflow Providers
Apache Airflow `main` branch.
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
_No response_
### Anything else?
_No response_
### Are you willing to submit PR?
- [x] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
