[GitHub] [airflow] potiuk commented on issue #11708: Git sync for plugins in the Helm Chart
potiuk commented on issue #11708: URL: https://github.com/apache/airflow/issues/11708#issuecomment-1495766319 > There are various scenarios and cases when the plugins can be reloaded: https://airflow.apache.org/docs/apache-airflow/stable/plugins.html#when-are-plugins-re-loaded From what I see @quocanh-upmesh (Also see the docs above) nothing changed. As explained above - things get implemented here when somoene implements it (and the most certain way to get somethig implemented is to either implement it or find someone who will do it). If you look above this would not only require change in the Helm Chart to sync the dags but also likely restarting the right components when they change (depending on the type of plugin and component). And answering your question - apparently there is no progress, otherwise semeone who would work on it would likely mention it here. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #11708: Git sync for plugins in the Helm Chart
potiuk commented on issue #11708: URL: https://github.com/apache/airflow/issues/11708#issuecomment-1371554995 > hey guys, just curious: is this approach even possible? I mean: sync plugins with git-sync. I had impression that 'plugins' must be in place before airflow starts. Otherwise if you start airflow with empty folder and then try to put new modules - it returns exception 'plugin is not registered'. Correct me if i'm wrong. There are various scenarios and cases when the plugins can be reloaded: https://airflow.apache.org/docs/apache-airflow/stable/plugins.html#when-are-plugins-re-loaded -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #11708: Git sync for plugins in the Helm Chart
potiuk commented on issue #11708: URL: https://github.com/apache/airflow/issues/11708#issuecomment-1312920790 Would be great. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #11708: Git sync for plugins in the Helm Chart
potiuk commented on issue #11708: URL: https://github.com/apache/airflow/issues/11708#issuecomment-1293269651 And just to add a bit - If you look at our repo - Airflow is created by >2200 contrbutors, mostly people like you who wanted something they missed and added it :). Adding such a feature is a nice way not only good for you (because you will be able to use it) but also for others (they will be able to use it too) but also a great way to contribute back for the software you got absolutely for free :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #11708: Git sync for plugins in the Helm Chart
potiuk commented on issue #11708: URL: https://github.com/apache/airflow/issues/11708#issuecomment-1293266402 Just to set expectations and explain how it works here - @tomrutter - as everything in open-source, things get done when somene does it :). There is no "magical" management and roadmap planning that you can have single authoritative response on. There is no roadmap unless someone decides to take on responsibility and manage, prepare and follow up the roadmap. That someone can be you if you want to get things implemented (which means often implementing things yourself and providing a PR or finding someone who would volunteer and do it. You are absolutely free to take a lead on it :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #11708: Git sync for plugins in the Helm Chart
potiuk commented on issue #11708: URL: https://github.com/apache/airflow/issues/11708#issuecomment-925725607 This is a good idea to have GitSync for plugins, but I am afraid the solution where you keep your plugins together with DAGs is a very bad idea for Airflow 2, one that should be heavily discouraged. There are two reasons (actually the root cause is isolation and security): 1) Plugins should not be "modifiable" by DAG writers. Plugins give you more "powers" over Airlfow that DAGs. The Plugins can modify UI of airflow (similarly as providers - with connections) and they are executed in the context of scheduler (not in the context of FileProcessor as all DAGs are) and you should not have the same people being able to write DAGs and modify plugins. This is something that wil be only strenghtened in the future when we increase isolation of Ailrflow components and it's a prerequisite to future Airflow multi-tenancy. But more importantly: 2) In Airflow 2 DAGs are not available as mounted volume for Webserver in our Helm Chart (and they should NOT be). See the Official Helm Chart. This is precisely because of the increased isolation and the fact that webserver does not need them any more (it uses Serialized form of the DAGs from the DB). So if your Plugins modify the UI of Airflow, this solution will not work (unless you also additionally mount the DAG folder to webserver - but this violates the isolation that was improved in Airflow 2 - see point 1). Generally speaking - you SHOULD separate DAGs and Plugins. Access to modigy those two in many cases should be controlled differently for security reasons. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
