potiuk commented on issue #31875: URL: https://github.com/apache/airflow/issues/31875#issuecomment-1589784684
We've been discussing those ideas few times and the result is pretty much always going the same route. This is quite a bigger task than you think it is and a number of things have to be discussed/proposed in order to be "ready" to accept it. I am not against the concept, I just want to be sure that whoever is going to propose a solution, is also going to see through all the consequences and proposes a solution that addresses the few issues issues I mention below. I propose that the author leads and gather feedback from others on the way how to solve that problem - and I think those who look for having this PR implemented should work out a complete and consistent proposal together. I have no strong opinions on that, I just know that this part should be addressed, so will keep listening and when the solution is agreed. Moreover we need to have documentation and be able to explain to our users what is the workflow they shoudl do and expect and have syncing implemented as feature in our Helm Chart. I think without it will be toothless - and it will raise a lot of support questions by people wanting to use it in our chart, so in order to accep that we need to have answers. Also it's not entirely clear why LocalFilesystemBackend of ours is not fitting the use case needed here. ## More context: ### Editability of the connection I would welcome this one if enabling it solves the problem of what happens when the user edits the connections of variables over UI or API only to find out that they were deleted or modify at random deployment times. This is the kind of surprises that our users do not want. If any kind of syncing is proposed, it also must address this part and it has to be consistent solution thinking about all the cases and scenarios. There are questions regarding which/whether and in which circumstances, the variables should be allowed to be changed in the database. What happens if you sync connections and want to add a new one and keep it persistent for example? How do define which of the connections are managed by the sync and which are managed outside? Or maybe the connections should not be editable at all? How to protect against such changes. How the users should see the connections /variables when they access the UI? should they see them at all? How to communiicate what is changeable and what is not. This means also probably proposing some UI changes and API changes that accomodate that. Also some thinking how the "airflow connection" and "aiflow variable" CLI commands should behave. Connected to that - how to control whether the connections can by synced/edited? Should it be controlled by configuration? ### Chart integration Another set of questions there regards the synchronisation scenarios in the chart of our. When and how such synchronisation should happen ? At the deployment time? When the file gets updated? Should it be via secret modification? Or via some other ways? When you re-synchronise (and potentially update the new values? This feature will only be really useful if we have a workflow defined in our chart that and instructions explaining how to do the synchronisation by the users. What the users should do to edit their connections? Should they redeploy chart? Or run some kind of migration? Or upgrade Airflow? ### LocalFileSystemBackend as an alterntaitve Also we already have another solution for this kind of make sure that whatever you configure in a file is visible as variable/connection namely `LocalFilesystemBackend`. It will take variables connections/variables from a file without syncing and make it available for workers and dag file processor, without introducing . Might be not perfect (requires file syncing between all the components). and maybe there are other reasons why it's not good. https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/security/secrets/secrets-backend/local-filesystem-secrets-backend.html I think - somewhat surprisingly - the scope of this change is at the border at requiring an Airflow Improvement Proposal https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals . At the very least, this proposal should end up as a consistent proposal explaining all the details above in a devlist thread and at least calling for a `[LAZY CONSENSUS]` (https://www.apache.org/foundation/voting.html) because it has som unobvious consequences. Again - I have no strong opinions on those, I hope the discussion - here or on the devlist can propose a consistent solution for those problems (ideall worked out by those who are eager to have it) and then I am happy to accept it as is (as long as it is consistent and addresses those issues above). @javatarz - are you up to the challenge? and others - would you be happy in helping @javatarz to make a consistent proposal? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
