Yes, it’s something really worth exploring. Recently I just finished a POC on DAG serialisation as well, while the main objective was to allow “rewriting” some components, like worker (for specific operators/sensors), in languages other than Python. We’re also choosing JSON, so that most languages can easily process the information from DAGs.
Happy to know how this topic will be processed. XD > On 26 Jul 2019, at 10:36 PM, Kaxil Naik <kaxiln...@gmail.com> wrote: > > Hi all, > > We, at Astronomer, are going to spend time working on DAG Serialisation. > There are 2 AIPs that are somewhat related to what we plan to work on: > > - AIP-18 Persist all information from DAG file in DB > > <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-18+Persist+all+information+from+DAG+file+in+DB> > - AIP-19 Making the webserver stateless > > <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-19+Making+the+webserver+stateless> > > We plan to use JSON as the Serialisation format and store it as a blob in > metadata DB. > > *Goals:* > > - Make Webserver Stateless > - Use the same version of the DAG across Webserver & Scheduler > - Keep backward compatibility and have a flag (globally & at DAG level) > to turn this feature on/off > - Enable DAG Versioning (extended Goal) > > > We will be preparing a proposal (AIP) after some research and some initial > work and open it for the suggestions of the community. > > We already had some good brain-storming sessions with Twitter folks (DanD & > Sumit), folks from GoDataDriven (Fokko & Bas) & Alex (from Uber) which will > be a good starting point for us. > > If anyone in the community is interested in it or has some experience about > the same and want to collaborate please let me know and join > #dag-serialisation channel on Airflow Slack. > > Regards, > Kaxil
