Great! That's definitely one of the most painful aspects of Airflow. Happy to help/comment/take part in the discussions and later in the implementation.
On Fri, Jul 26, 2019 at 4:48 PM Deng Xiaodong <xd.den...@gmail.com> wrote: > Yes, it’s something really worth exploring. > > Recently I just finished a POC on DAG serialisation as well, while the > main objective was to allow “rewriting” some components, like worker (for > specific operators/sensors), in languages other than Python. We’re also > choosing JSON, so that most languages can easily process the information > from DAGs. > > Happy to know how this topic will be processed. > > > XD > > > On 26 Jul 2019, at 10:36 PM, Kaxil Naik <kaxiln...@gmail.com> wrote: > > > > Hi all, > > > > We, at Astronomer, are going to spend time working on DAG Serialisation. > > There are 2 AIPs that are somewhat related to what we plan to work on: > > > > - AIP-18 Persist all information from DAG file in DB > > < > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-18+Persist+all+information+from+DAG+file+in+DB > > > > - AIP-19 Making the webserver stateless > > < > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-19+Making+the+webserver+stateless > > > > > > We plan to use JSON as the Serialisation format and store it as a blob in > > metadata DB. > > > > *Goals:* > > > > - Make Webserver Stateless > > - Use the same version of the DAG across Webserver & Scheduler > > - Keep backward compatibility and have a flag (globally & at DAG level) > > to turn this feature on/off > > - Enable DAG Versioning (extended Goal) > > > > > > We will be preparing a proposal (AIP) after some research and some > initial > > work and open it for the suggestions of the community. > > > > We already had some good brain-storming sessions with Twitter folks > (DanD & > > Sumit), folks from GoDataDriven (Fokko & Bas) & Alex (from Uber) which > will > > be a good starting point for us. > > > > If anyone in the community is interested in it or has some experience > about > > the same and want to collaborate please let me know and join > > #dag-serialisation channel on Airflow Slack. > > > > Regards, > > Kaxil > > -- Jarek Potiuk Polidea <https://www.polidea.com/> | Principal Software Engineer M: +48 660 796 129 <+48660796129> [image: Polidea] <https://www.polidea.com/>
