I looked over your shoulders while you were implementing it. Big +1 from me.
On Tue, Oct 15, 2019 at 2:57 AM Kaxil Naik <kaxiln...@gmail.com> wrote: > Hello, Airflow community, > > This email calls for a vote to add the DAG Serialization feature at > https://github.com/apache/airflow/pull/5743. > > *AIP*: > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-24+DAG+Persistence+in+DB+using+JSON+for+Airflow+Webserver+and+%28optional%29+Scheduler > > *Previous Mailing List discussion*: > https://lists.apache.org/thread.html/65d282368e0a7c19815badb8b1c6c8d72b0975ce94f601e13af44f74@%3Cdev.airflow.apache.org%3E > . > > *Authors*: Kaxil Naik, Zhou Fang, Ash-Berlin Taylor > > *Summary*: > > - DAGs are serialized using JSON format and stored in a SerializedDag > table > - The Webserver now instead of having to parse the DAG file again, > reads the serialized DAGs in JSON, de-serializes them and creates the > DagBag and uses it to show in the UI. > - Instead of loading an entire DagBag when the WebServer starts we > only load each DAG on demand from the Serialized Dag table. This helps > reduce Webserver startup time and memory. The reduction is notable when you > have a large number of DAGs. > - A JSON Schema has been defined and we validate the serialized dag > before writing it to the database > > [image: image.png] > > A PR (https://github.com/apache/airflow/pull/5743) is ready for review > from the committers and community. > > We also have a WIP PR (https://github.com/apache/airflow/pull/5992) to > backport this feature to 1.10.* branch. > > A big thank you to Zhou and Ash for their continuous help in improving > this feature/PR. > > This email is formally calling for a vote to accept the AIP and PR. Please > note that we will update the PR / feature to fix bugs if we find any. > > Cheers, > Kaxil > > > > > > > -- Jarek Potiuk Polidea <https://www.polidea.com/> | Principal Software Engineer M: +48 660 796 129 <+48660796129> [image: Polidea] <https://www.polidea.com/>
