Dear All,
After my talk at the Airflow Summit ("Operators need to die" :-)). I
decided to put my money where my mouth is so to speak. I have therefore
created a proposal to include Airflow FS;
With Airflow FS (AFS) we want to address the challenges of fragmented code,
improve XCom, and DAG processing complexity by providing a unified
abstraction for accessing object storage systems. By doing so, we aim to
enhance the overall user experience, promote code reusability, and simplify
the development of Airflow workflows that interact with different storage
backends.
You can find the AIP at
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=263430565
And its implementation
https://github.com/apache/airflow/pull/34729
Note that some discussion already happened on the PR. That might be
worthwhile reading if you have questions, suggestions, improvements.
Bolke
--
--
Bolke de Bruin
[email protected]
