It is really great news and would be a big plus. Thanks Kaxil, Ash, Anita, and Zhou Fang!
XD > On 14 Mar 2020, at 2:31 AM, Kevin Yang <[email protected]> wrote: > > This is thrilling news! Completely upleveling the experience using and > maintaining the webserver. Thank you so much everyone who contributed to > this initiative! > > > Cheers, > Kevin Y > > On Fri, Mar 13, 2020 at 6:24 PM Kaxil Naik <[email protected]> wrote: > >> Hi all, >> >> Happy to tell you all that we have completed the first phase of DAG >> Serialisation i.e. the Webserver is stateless and can now run without >> access to DAG Files. >> >> The 2 limitations we had in 1.10.7-1.10.9 ( >> https://airflow.apache.org/docs/1.10.7/dag-serialization.html#limitations) >> have been resolved. >> >> Special thanks to @ash for his continuous guidance and contributions. >> >> Also a special mention to Anita Fronczak and Zhou Fang for their >> contributions along the way. >> >> The next step is to remove SimpleDag representation in the Scheduler and >> replace it with Serialized DAG (WIP PR: >> https://github.com/apache/airflow/pull/7694) >> >> *Advantages*: >> >> - *Reduction in Webserver startup time* for large number of DAGs. >> Without DAG Serialization all the DAGs are loaded in the DagBag during >> the >> Webserver startup. With DAG Serialization, an empty DagBag is created >> and >> Dags are loaded from DB only when needed (i.e. when a particular DAG is >> clicked on in the home page) >> - *No DAG Parsing / Consistency*: Webserver would load DAGs from DB and >> won't even need the DAG Files when DAG Serialization is turned on. DAGs >> are >> parsed, serialized and stored in DB by the Scheduler. >> - Rendered Templates for TasksInstances that have already run will now >> correctly display their value which was true at the time of the run >> instead >> of the current value. >> - Paves way for* DAG Versioning* (more details on it when I create a >> separate AIP / update an existing AIP for it) and *Scheduler HA *(AIP-15 >> < >> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=103092651 >>> >> ). >> >> >> I will create new JIRA issues for further steps with DAG Serialization and >> DAG Versioning and would discuss them in our next sig-dag-serialization >> call (later this month). >> >> Regards, >> Kaxil >>
