potiuk commented on issue #31875:
URL: https://github.com/apache/airflow/issues/31875#issuecomment-1589784684

   We've been discussing those ideas few times and the result is pretty much 
always going the same route.
   
   This is quite a bigger task than you think it is and a number of things have 
to be discussed/proposed in order to be "ready" to accept it.
   
   I am not against the concept, I just want to be sure that whoever is going 
to propose a solution, is also going to see through all the consequences and 
proposes a solution that addresses the few issues issues I mention below.
   
   I propose that the author leads and gather feedback from others on the way 
how to solve that problem - and I think those who look for having this PR 
implemented should work out a complete and consistent proposal together. I have 
no strong opinions on that, I just know that this part should be addressed, so 
will keep listening and when the solution is agreed. Moreover we need to have 
documentation and be able to explain to our users what is the workflow they 
shoudl do and expect and have syncing implemented as feature in our Helm Chart. 
I think without it will be toothless - and it will raise a lot of support 
questions by people wanting to use it in our chart, so in order to accep that 
we need to have answers. Also it's not entirely clear why 
LocalFilesystemBackend of ours is not fitting the use case needed here.
   
   ## More context:
   
   ### Editability of the connection
   
   I would welcome this one if enabling it solves the problem of what happens 
when the user edits the connections of variables over UI or API only to find 
out that they were deleted or modify at random deployment times. This is the 
kind of surprises that our users do not want. If any  kind of syncing is 
proposed, it also must address this part and it has to be  consistent solution 
thinking about all the cases and scenarios. 
   
   There are questions regarding which/whether and in which circumstances, the 
variables should be allowed to be changed in the database. What happens if you 
sync connections and want to add a new one and keep it persistent for example? 
How do define which of the connections are managed by the sync and which are 
managed outside? Or maybe the connections should not be editable at all? How to 
protect against such changes. How the users should see the connections 
/variables when they access the UI? should they see them at all? How to 
communiicate what is changeable and what is not. This means also probably 
proposing some UI changes and API changes that accomodate that. Also some 
thinking how the "airflow connection" and "aiflow variable" CLI commands should 
behave. Connected to that - how to control whether the connections can by 
synced/edited? Should it be controlled by configuration? 
   
   ### Chart integration
   
   Another set of questions there regards the synchronisation scenarios in the 
chart of our. When and how such synchronisation should happen ? At the 
deployment time? When the file gets updated? Should it be via secret 
modification? Or  via some other ways? When you re-synchronise (and potentially 
update the new values? This feature will only be really useful if we have a 
workflow defined in our chart that and instructions explaining how to do the 
synchronisation by the users. What the users should do to edit their 
connections? Should they redeploy chart? Or run some kind of migration? Or 
upgrade Airflow? 
   
   ### LocalFileSystemBackend as an alterntaitve
   
   Also we already have another solution for this kind of make sure that 
whatever you configure in a file is visible as variable/connection namely 
`LocalFilesystemBackend`. It will take variables connections/variables from a 
file without syncing and make it available for workers and dag file processor, 
without introducing . Might be not perfect (requires file syncing between all 
the components).  and maybe there are other reasons why it's not good.
   
   
https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/security/secrets/secrets-backend/local-filesystem-secrets-backend.html
   
   I think - somewhat surprisingly - the scope of this change is at the border 
at requiring an Airflow Improvement Proposal  
https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals
 . At the very least, this proposal should end up as a consistent proposal 
explaining all the details above in a devlist thread and at least calling for a 
`[LAZY CONSENSUS]` (https://www.apache.org/foundation/voting.html) because it 
has som unobvious consequences.
   
   Again - I have no strong opinions on those, I hope the discussion - here or 
on the devlist can propose  a consistent solution for those problems (ideall 
worked out by those who are eager to have it) and then I am happy to  accept it 
as is (as long as it is consistent and addresses those issues above).
   
   @javatarz - are you up to the challenge? and others - would you be happy in 
helping @javatarz to make a consistent proposal?
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to