This is a very interesting question and I think it is very much relevant to our discussion we had yesterday at the Dev call where Dheeraj raised the point that lack of access to DB might turn people into defining connection to Airflow DB and using it anyway.
I think some of the access "needs" are very valid (like this one) and - yes if we will not provide a good replacement for some of those use cases, people will start doing this - connecting to Airflow DB with out-of-band credentials and modifying the DB. Which defeats not only the purpose of our task isolation, but also takes away the biggest benefit of task isolation - capability of releasing airflow independently on workers/ task-sdk. We might turn our blind eye on the fact that people might do this (i.e. access to database) - but since we already know that our first-movers are contemplating (and even doing it) opening a connection to DB because they cannot do something - and we ignore it, this is more like the proverbial ostrich hiding their hand in the sand and pretending nothing bad is happening (sorry that might be purely Polish proverb - but I hope you get the meaning of it). I see that case pretty useful - similarly as few other cases where indeed we should likely let the users interact (via Airflow REST API or task-sdk API) and do those things they want to do in a seamless way. Clear use cases that we see here - is the credential update described above. I've been thinking how else we could do it (maybe special auto-refreshing connection type?) but due to the way how most of those types of connections work, it really requires to refresh such tokens using long-lasting token, when they expire - and it has to be done pretty synchronously (i.e. when the credential expire, we should be able to transparently refresh - for example using long-living refresh token that is available via workload management or retrieved from secrets manager). This should happen on the worker, not on the scheduler (potentially custom code to run), so effectively we need a way to update such refresh the credential back to the connection - I don't see logically another way - we really need a worker -> DB update in this case. And that feature goes hand-in-hand with what we discussed yesterday: some way for the DAG author to interact with Airflow DB - in a limited (including limited permissions) way. The credentials allowing it should be automatically available in the task and they should be limited to (that's rough proposal - we likely need to get more detailed analysis and better "dynamic role" definition Read only: * that task * that dag * all other parts of our model connected with that particular dag and task Read/Write: * connections and variables belonging to the team where the DAG beliongs (when multi-team is enabled) - including those connections/variable that we mark as "DAG writeable" (I think it would be safest to limit write to those connections/variables that are explicitly marked as writeable) Again - those permissions above are rough - we likely need to review and make sure that we give access broad enough to cover most common use cases, and wide-enough to avoid people bypassing task isolation. Likely needs detailed AIP and deeper analysis. I think any other solution - where we delegate that task to our users (say configure special user in API and make credential available to DAG authors) is deeply flawed because it's next to impossible (or very difficult to keep it in sync) to limit the scope of such a request to DAG/TASK/TEAM - it seems much more reasonable that API-server creates detdcated credentials for the task - that contain enough information in the token (UUID) so that teh API call from task will be properly authenticated and has enough permissions for most of the things users would like to query for and update - without compromising our isolation. J. On Thu, Oct 23, 2025 at 11:23 PM Jens Scheffler <[email protected]> wrote: > Hi Mikhail, > > thanks for dropping the note. I personally never had such use case and > from security perspective I assume it depends very much on the > environment if it is acceptable to change connections. In most cases you > could impact operations of other workflows if the code is bad. So at > least this would require to be a configurable option and probably you 'd > like to limit this to certain Dags/Tasks only for certain Connections. > > As an alternative I'd suggest you use another connection to store > credentials to public API endpoint and then you could in a task use the > public API endpoint > > https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#operation/patch_connection > to update the connection from a "service Dag" i a regular interval. > > Jens > > On 23.10.25 17:22, Mikhail Ilchenko wrote: > > Hi everyone, > > > > I would like to start a discussion about whether Airflow should support > limited modification of Connection objects from within DAGs, similar to how > Variable updates are currently allowed. > > > > As I wrote in https://github.com/apache/airflow/discussions/53544, in > Airflow v2 I have specific DAG which > > > > • requests API to provide access token > > • updates connection > > > > This approach was convenient for automatically refreshing temporary > access tokens and persisting them for subsequent DAG runs. > > Trying to migrate to Airflow v3, however, I found out Connection objects > are effectively read-only for DAG authors — methods like set_extra and > updating via Session are no longer available. > > > > I fully understand the reasoning behind making Connection immutable in > most cases — it helps keep the configuration consistent and safe. > > However, there are real-world cases where programmatic updates of > connection metadata (like refreshed credentials, temporary URLs, or OAuth > tokens) are both safe and practical — especially when done in a controlled > way. > > > > I’d like to hear the community’s thoughts: > > > > • Would it make sense to have a documented, limited API for updating > connection attributes from DAGs? > > • Should we instead encourage another pattern and what it could be? > > > > > > Best regards, > > Mikhail > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > >
