Agreed on complexities (I think deprecating Jinja templates for webserver rendering is one thing), but I'm not sure I understand on the falling down on code changes part, mind providing an example?
On Thu, Jan 31, 2019 at 12:22 PM Ash Berlin-Taylor <a...@apache.org> wrote: > That sounds like a good idea at first, but falls down with possible code > changes in operators between one task and the next. > > (I would like this, but there are definite complexities) > > -ash > > > On 31 January 2019 16:56:54 GMT, Dan Davydov <ddavy...@twitter.com.INVALID> > wrote: > >I feel the right higher-level solution to this problem (which is > >"Adding > >Consistency to Airflow") is DAG serialization, that is all DAGs should > >be > >represented as e.g. JSON (similar to the current SimpleDAGBag object > >used > >by the Scheduler). This solves the webserver issue, and also adds > >consistency between Scheduler/Workers (all DAGruns can be ensured to > >run at > >the same version of a DAG instead of whatever happens to live on the > >worker > >at the time). > > > >On Thu, Jan 31, 2019 at 9:44 AM Peter van ‘t Hof < > >petervant...@godatadriven.com> wrote: > > > >> Hi All, > >> > >> As most of you guys know, airflow got an issue when loading new dags > >where > >> the webserver sometimes sees it and sometimes not. > >> Because of this we did wrote this AIP to solve this issue: > >> > >> > > > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-12+Persist+DAG+into+DB > >> > >> Any feedback is welcome. > >> > >> Gr, > >> Peter van 't Hof > >> Big Data Engineer > >> > >> GoDataDriven > >> Wibautstraat 202 > >> 1091 GS Amsterdam > >> https://godatadriven.com > >> > >> >
