javatarz commented on issue #31875:
URL: https://github.com/apache/airflow/issues/31875#issuecomment-1590298835
Thank you for the detailed response @potiuk! There's a bunch of perspectives
I hadn't thought of so I'll go through as much as I can answer now and for some
of it, I'll have to come back.
> I would welcome this one if enabling it solves the problem of what happens
when the user edits the connections of variables over UI or API only to find
out that they were deleted or modify at random deployment times. This is the
kind of surprises that our users do not want. If any kind of syncing is
proposed, it also must address this part and it has to be consistent solution
thinking about all the cases and scenarios.
Our team's perspective on this has been that any airflow variable changes
made through the UI will be temporary and will be over written in the next
deployment. If we don't want this to happen, the change should have come from
source control and not the UI. Ideally, we'd love to make the variables UI read
only if that was a configuration (this isn't an option so far I believe).
My proposal here would be to allow users to have an airflow config that
allows the edit options in the variables section to be disabled. If people keep
it enabled and use `airflow variables sync` in their deployments, we are OK
with these variables being reset. While that creates confusion for some users,
others might find it useful to make quick feature toggle/configuration changes
in Airflow variables and are OK with them being reset in the next deployment.
Changes proposed in this section:
1. CLI command for sync: `airflow variables sync file.json`
2. Configuration to disable UI edits for variables:
`AIRFLOW__UI__DISABLE_VARIABLE_EDITS`
What I dislike about this part of my proposal is that the default
implementation is to allow variables sync and an editable UI and this might be
confusing to some users. Changing the default behaviour will cause
significantly more challenges so this is the best suggestion I can think of so
far. What do others think?
> What happens if you sync connections and want to add a new one and keep it
persistent for example?
I'll try explaining how the sync works through an example
**Airflow state before sync:**
1. UnchangedVariable: UnchangedValue
2. SameKeyDifferentValue: OldValue
3. DeleteThisKey: DeleteThisValue
**Config file:**
```
{
"UnchangedVariable": "UnchangedValue",
"SameKeyDifferentValue": "NewValue",
"NewKey": "BrandNewValue"
}
```
**Airflow state after sync:**
1. UnchangedVariable: UnchangedValue
2. SameKeyDifferentValue: NewValue
3. NewKey: BrandNewValue
**Summary of changes:**
1. Keys and values that remain the same in Airflow vs the config will not be
touched
2. When values change, they will be updated
3. When keys are removed, they will be deleted from airflow
4. When keys are added, they will be added to airflow
> Another set of questions there regards the synchronisation scenarios in
the chart of our.
We do not use airflow's helm charts and will have to read up on these. I'd
like to get back to this section of questions a bit later.
> What happens if you sync connections and want to add a new one and keep it
persistent for example?
Aren't connections non-editable at this point? We've been deleting and
adding them for a while now :) I was hoping to start a separate card to talk
about what we should do with connections. I'd love a similar sync capability
but I'm not sure of the consequences of this. I'm not sure of the reasons why
connections aren't editable. Is there somewhere I can read up on that
discussion?
> Also we already have another solution for this kind of workflow when you
want to make sure that whatever you configure in a file is visible as
variable/connection namely `LocalFilesystemBackend`.
Hoping to stay away from local file system changes post the deployment.
Currently, I should be able to run this command if I have access to the
terminal on a machine (SSH or the likes) and I'd like to avoid a common file
system to keep all components in sync.
-----------
I'm still up for the challenge. I'll start a discussion on the dev mailing
list to talk about this change. Let's get consensus on what a consistent design
should be. In the meanwhile, let me read up on parts that I don't know as much
about yet (HelmCharts)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]