javatarz commented on issue #31875:
URL: https://github.com/apache/airflow/issues/31875#issuecomment-1590298835

   Thank you for the detailed response @potiuk! There's a bunch of perspectives 
I hadn't thought of so I'll go through as much as I can answer now and for some 
of it, I'll have to come back.
   
   > I would welcome this one if enabling it solves the problem of what happens 
when the user edits the connections of variables over UI or API only to find 
out that they were deleted or modify at random deployment times. This is the 
kind of surprises that our users do not want. If any kind of syncing is 
proposed, it also must address this part and it has to be consistent solution 
thinking about all the cases and scenarios.
   
   Our team's perspective on this has been that any airflow variable changes 
made through the UI will be temporary and will be over written in the next 
deployment. If we don't want this to happen, the change should have come from 
source control and not the UI. Ideally, we'd love to make the variables UI read 
only if that was a configuration (this isn't an option so far I believe).
   
   My proposal here would be to allow users to have an airflow config that 
allows the edit options in the variables section to be disabled. If people keep 
it enabled and use `airflow variables sync` in their deployments, we are OK 
with these variables being reset. While that creates confusion for some users, 
others might find it useful to make quick feature toggle/configuration changes 
in Airflow variables and are OK with them being reset in the next deployment.
   
   Changes proposed in this section:
   1. CLI command for sync: `airflow variables sync file.json`
   2. Configuration to disable UI edits for variables: 
`AIRFLOW__UI__DISABLE_VARIABLE_EDITS`
   
   What I dislike about this part of my proposal is that the default 
implementation is to allow variables sync and an editable UI and this might be 
confusing to some users. Changing the default behaviour will cause 
significantly more challenges so this is the best suggestion I can think of so 
far. What do others think?
   
   > What happens if you sync connections and want to add a new one and keep it 
persistent for example?
   I'll try explaining how the sync works through an example
   
   **Airflow state before sync:**
   1. UnchangedVariable: UnchangedValue
   2. SameKeyDifferentValue: OldValue
   3. DeleteThisKey: DeleteThisValue
   
   **Config file:**
   ```
   {
     "UnchangedVariable": "UnchangedValue",
     "SameKeyDifferentValue": "NewValue",
     "NewKey": "BrandNewValue"
   }
   ```
   
   **Airflow state after sync:**
   1. UnchangedVariable: UnchangedValue
   2. SameKeyDifferentValue: NewValue
   3. NewKey: BrandNewValue
   
   **Summary of changes:**
   1. Keys and values that remain the same in Airflow vs the config will not be 
touched
   2. When values change, they will be updated
   3. When keys are removed, they will be deleted from airflow
   4. When keys are added, they will be added to airflow
   
   > Another set of questions there regards the synchronisation scenarios in 
the chart of our.
   
   We do not use airflow's helm charts and will have to read up on these. I'd 
like to get back to this section of questions a bit later.
   
   > What happens if you sync connections and want to add a new one and keep it 
persistent for example?
   
   Aren't connections non-editable at this point? We've been deleting and 
adding them for a while now :) I was hoping to start a separate card to talk 
about what we should do with connections. I'd love a similar sync capability 
but I'm not sure of the consequences of this. I'm not sure of the reasons why 
connections aren't editable. Is there somewhere I can read up on that 
discussion?
   
   > Also we already have another solution for this kind of workflow when you 
want to make sure that whatever you configure in a file is visible as 
variable/connection namely `LocalFilesystemBackend`.
   
   Hoping to stay away from local file system changes post the deployment. 
Currently, I should be able to run this command if I have access to the 
terminal on a machine (SSH or the likes) and I'd like to avoid a common file 
system to keep all components in sync.
   
   -----------
   
   I'm still up for the challenge. I'll start a discussion on the dev mailing 
list to talk about this change. Let's get consensus on what a consistent design 
should be. In the meanwhile, let me read up on parts that I don't know as much 
about yet (HelmCharts)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to