Nataneljpwd commented on PR #61527:
URL: https://github.com/apache/airflow/pull/61527#issuecomment-3871981004
> thanks for the review and comments, however before I will address them, I
would like to also get your opinion on the secrets discovery method I chose -
the secret name mapping... which after more thinking is not a good ide. What do
you think about the options (I would go for option 2)
> ```yaml
> ---
> # labels (option 1)
> # backend_kwargs = {"connections_lable_type_name": "secret-type",
"connections_lable_name_name": "secret-name", ...}
> apiVersion: v1
> kind: Secret
> metadata:
> name: anything
> labels:
> airflow.apache.org/secret-type: connection
> airflow.apache.org/secret-name: <conn_name>
> data:
> value: <conn_value>
>
> ---
> # labels (options 2)
> # backend_kwargs = {"connections_secret_lable_name": "connection-name",
... }
> apiVersion: v1
> kind: Secret
> metadata:
> name: anything
> labels:
> airflow.apache.org/connection-name: <conn_name>
> data:
> value: <conn_value>
>
>
>
> ---
> # single secret to manage all connections (option 3)
> # backend_kwargs = {"connections_secret_name": "airflow-connections", ... }
> apiVersion: v1
> kind: Secret
> metadata:
> name: airflow-connections
> data:
> <conn_name1>: <conn_value>
> <conn_name2>: <conn_value>
>
> ```
>
> labels search could be equipped with same cacheing feature, if a secrets
has been discovered on previous runs, it does not perform search (i knows that
the secret/connection exists)
Option 1 seems reasonable, it allows to distinguish between connections and
variables easily, while option 2 does not, so here I would prefer option 1, yet
if you decide to make it with labels, you need to make sure that there are no
duplicated labels before the name was the unique constraint enforced by k8s,
now you will need to enforce uniqueness by yourself, in case you create, update
or delete a secret.
About the caching, we can just do a k8s query with a resource version of 0
(which is very quick) yet we will be updated on when the connection or variable
is deleted, and so caching on airflow side is probably not worth implementing,
as we can rely on k8s api server cache.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
