potiuk commented on issue #29192: URL: https://github.com/apache/airflow/issues/29192#issuecomment-1406352484
That goes exactly against the way how Airflow DAG philosophy about distributing DAG files is. Airflow REST API is NOT about adding or modifying DAGs. It's about managing DAGs that ar distributed using other mechanism. Your interface to deal with DAG files is not Airflow REST API. Your API forr that is the API of the filesytem that you are using for distributing DAGS: * S3 -> use S3 API * Shared EFS volume -> use Shared EFS * GitSync -> use Git commands * etc. etc. This is NOT the question of doing post/pre-processing of what happens with the remote call over the API. This is also (and most importantly) the question of security and separation of concerns. Airlfow's REST API and Webserver have very, very low permissions when it comes to producing or manipulating an executable Python Code. Simply speaking, by interfacing with webserver or REST API, it's just impossible to modify any code that is submitted by a DAG Author and executed as part of tasks, It's not "disallowed" - it's far more than that - it's not POSSIBLE, because neither API nor webserver have (in serious deployments) have access to the DAG files on the shared volume, git sync. More often than not the webserver (due to the fact that it is exposed to external world via http/https and security requirements) will be running with it's own sandbox with only necessary ports and IP addresses it can communicate with for outwards communication - which will make it imposible to directly talk to s3 for example. The webserver / API server does not only miss credentials to do so, but they don't even have shared folders with DAGs mounted. AT ALL. Being able to delete a file, means that you have to have credentials to do so exposed to the webserver/api as well as you have to have those files mounted/accessible. So what you are asking for is breaking absolutely basic and foundational assumptions of Airlfow about security. And it's very straightforward "no-go". And it makes no sense whatsoever even from your perspective as someone who wants to automate this. If you were to write the python code to communicate with S3 for example and delete the files, you not only have to implement the code to do it and use the right API (S3 for example), but you also need to deploy that code with Airflow, and make sure it is integrated with this new futnctionality, distribute and (somehow) protect the credentials that are needed to do so (possibly to multiple Webserver/API components). If you do it directly in the client of yours that originate the "delete" request and does such removal directly, you have MUCH simpler job, and also you are not limited to using Python. You can for example write a bash script that runs: ``` aws s3 rm "generated URL to your dag file" ``` as part of your client deletion process. And all the credentials might be precisely where you need them and when you can secure them (via oauth or any other mechanism) - at the client which runs the deletion request. It's just **simpler** for you than wrapping the same code with Airlfow REST API call and integrating it with Airflow installation. If you want to delete DAG files, please continue using S3 API to delete your if you use S3. Or EFS API if you use EFS. Or "Git commit" if you are using Git Sync. Those are excellent APIs for managing files. Far better than REST API can ever be. They are very good for automation and you can make sure in your system that you have the right, properly secured credentials on the client side that originates the delete action. Do not push responsibility for that to Airflow webserver/ API server and do not add extra REST layer on top of those. It makes no sense. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
