This is awesome, and will bring tons of value to all Airflow users, thank you for driving this!
On Fri, Jul 26, 2019 at 5:37 PM Kaxil Naik <kaxiln...@gmail.com> wrote: > Hi all, > > We, at Astronomer, are going to spend time working on DAG Serialisation. > There are 2 AIPs that are somewhat related to what we plan to work on: > > - AIP-18 Persist all information from DAG file in DB > < > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-18+Persist+all+information+from+DAG+file+in+DB > > > - AIP-19 Making the webserver stateless > < > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-19+Making+the+webserver+stateless > > > > We plan to use JSON as the Serialisation format and store it as a blob in > metadata DB. > > *Goals:* > > - Make Webserver Stateless > - Use the same version of the DAG across Webserver & Scheduler > - Keep backward compatibility and have a flag (globally & at DAG level) > to turn this feature on/off > - Enable DAG Versioning (extended Goal) > > > We will be preparing a proposal (AIP) after some research and some initial > work and open it for the suggestions of the community. > > We already had some good brain-storming sessions with Twitter folks (DanD & > Sumit), folks from GoDataDriven (Fokko & Bas) & Alex (from Uber) which will > be a good starting point for us. > > If anyone in the community is interested in it or has some experience about > the same and want to collaborate please let me know and join > #dag-serialisation channel on Airflow Slack. > > Regards, > Kaxil >
