Hello Anton,
Thanks for kicking off this discussion. I’d love to understand your
motivations a bit more on this front.
>From your PR, I am seeing that you are just not allowing addition of
multiple custom backends
but also changing the *default_backend* order. I am a bit torn on that
part.
The current design intentionally places the metadata DB backend at the
lowest precedence in the order,
since it’s meant to serve as the ultimate fallback source of truth. Any
additional configured
backends are prioritized higher than it by design.
With your changes, we now allow configurations like:
* @conf_vars({("secrets", "backends_order"):
"metastore,environment_variable,unsupported"}) def
test_backends_order_unsupported(self): with
pytest.raises(AirflowConfigException): ensure_secrets_loaded()*
I don’t fully understand the motivation behind supporting this level of
override, especially since it
could allow unsupported or unintended configurations. Additionally, with
Airflow 3.0+, we already support
a multi layered secret backend resolution capability with the introduction
of secrets backend for workers.
Order goes as:
*secrets backend on worker directly (optional) > env vars on worker > *
*reach out to api server [secrets backend defined here (optional) > env
vars on api server > metadata DB].*
You will have to consider this angle too.
In my opinion, a more practical and realistic use case would be to have the
ability to define multiple custom backends
both on worker or the API server.
Looking forward to hearing more from you.
Thanks & Regards,
Amogh Desai
On Wed, Jul 2, 2025 at 3:59 PM Anton Nitochkin <ant.nitoch...@gmail.com>
wrote:
> Hello,
>
> I'd like to discuss a new option that can be added via this PR:
> https://github.com/apache/airflow/pull/45931.
>
> Recently, I asked developers in Slack for their thoughts on the new
> variable [secrets]backend_order. Long story short: this option will
> introduce the ability to configure the backend order and control it using
> this variable. The default value will remain the same as in the current
> version, so for users who don't need it, things will stay as they are now.
>
> Jarek Potiuk advised starting a conversation and discussing the PR to reach
> a consensus with the community.
>
> Can you please share your thoughts on the option and its implementation?
>
> Anton Nitochkin
>
