Great to see this happening! On Fri, Jul 26, 2019 at 8:54 AM Jarek Potiuk <jarek.pot...@polidea.com> wrote:
> Great! That's definitely one of the most painful aspects of Airflow. Happy > to help/comment/take part in the discussions and later in the > implementation. > > On Fri, Jul 26, 2019 at 4:48 PM Deng Xiaodong <xd.den...@gmail.com> wrote: > > > Yes, it’s something really worth exploring. > > > > Recently I just finished a POC on DAG serialisation as well, while the > > main objective was to allow “rewriting” some components, like worker (for > > specific operators/sensors), in languages other than Python. We’re also > > choosing JSON, so that most languages can easily process the information > > from DAGs. > > > > Happy to know how this topic will be processed. > > > > > > XD > > > > > On 26 Jul 2019, at 10:36 PM, Kaxil Naik <kaxiln...@gmail.com> wrote: > > > > > > Hi all, > > > > > > We, at Astronomer, are going to spend time working on DAG > Serialisation. > > > There are 2 AIPs that are somewhat related to what we plan to work on: > > > > > > - AIP-18 Persist all information from DAG file in DB > > > < > > > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-18+Persist+all+information+from+DAG+file+in+DB > > > > > > - AIP-19 Making the webserver stateless > > > < > > > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-19+Making+the+webserver+stateless > > > > > > > > > We plan to use JSON as the Serialisation format and store it as a blob > in > > > metadata DB. > > > > > > *Goals:* > > > > > > - Make Webserver Stateless > > > - Use the same version of the DAG across Webserver & Scheduler > > > - Keep backward compatibility and have a flag (globally & at DAG > level) > > > to turn this feature on/off > > > - Enable DAG Versioning (extended Goal) > > > > > > > > > We will be preparing a proposal (AIP) after some research and some > > initial > > > work and open it for the suggestions of the community. > > > > > > We already had some good brain-storming sessions with Twitter folks > > (DanD & > > > Sumit), folks from GoDataDriven (Fokko & Bas) & Alex (from Uber) which > > will > > > be a good starting point for us. > > > > > > If anyone in the community is interested in it or has some experience > > about > > > the same and want to collaborate please let me know and join > > > #dag-serialisation channel on Airflow Slack. > > > > > > Regards, > > > Kaxil > > > > > > -- > > Jarek Potiuk > Polidea <https://www.polidea.com/> | Principal Software Engineer > > M: +48 660 796 129 <+48660796129> > [image: Polidea] <https://www.polidea.com/> >
