Hi all, We, at Astronomer, are going to spend time working on DAG Serialisation. There are 2 AIPs that are somewhat related to what we plan to work on:
- AIP-18 Persist all information from DAG file in DB <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-18+Persist+all+information+from+DAG+file+in+DB> - AIP-19 Making the webserver stateless <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-19+Making+the+webserver+stateless> We plan to use JSON as the Serialisation format and store it as a blob in metadata DB. *Goals:* - Make Webserver Stateless - Use the same version of the DAG across Webserver & Scheduler - Keep backward compatibility and have a flag (globally & at DAG level) to turn this feature on/off - Enable DAG Versioning (extended Goal) We will be preparing a proposal (AIP) after some research and some initial work and open it for the suggestions of the community. We already had some good brain-storming sessions with Twitter folks (DanD & Sumit), folks from GoDataDriven (Fokko & Bas) & Alex (from Uber) which will be a good starting point for us. If anyone in the community is interested in it or has some experience about the same and want to collaborate please let me know and join #dag-serialisation channel on Airflow Slack. Regards, Kaxil
