Hi Julian, Thanks for adding that exhaustive list of changes that are needed for each view. Assuming we went with option 2b for obtaining information about the edges:
> Adding the current state of the DAG in the database, so that edges reflect the most recent version of DAG as it was parsed. Do you have a proposal for how the database schema will change? i.e., what tables are being added, their schema, any changes to columns in existing tables, etc. This has come up before in AIP-12 <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-12+Persist+DAG+into+DB> and its accompanying PR <https://github.com/apache/airflow/pull/4396/files#diff-ad4989e508949997ebe0f59574dc287f> - will we use the same schema as was proposed there? Thanks, Andrew Stahlman On Fri, Apr 12, 2019 at 7:43 AM Julian De Ruiter < julianderui...@godatadriven.com> wrote: > > Dear all, > > As an update on AIP-19, I have added the list of expected changes to the discussion of the AIP: https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-19+Making+the+webserver+stateless . > > Does anybody have more feedback on the proposal? > > Best, > Julian de Ruiter > > > On 27 Mar 2019, at 20:03, Julian De Ruiter < julianderui...@godatadriven.com<mailto:julianderui...@godatadriven.com>> wrote: > > Dear all, > > Last week we added AIP-19 ( https://cwiki.apache.org/confluence/display/AIRFLOW/AIP+19+-+Making+the+webserver+stateless), which aims to address various stability issues in the Airflow webserver stemming from differences in DagBag state between the different processes of the webserver. These stability issues are illustrated in this video: https://youtu.be/sNrBruPS3r4. > > Our AIP aims to solve these issues by moving DAG-related information into the database, rather than querying DAG metadata from the DagBag instance of the given webserver process. By obtaining this information from the database, we can ensure that there is a single-source-of-truth for DAG-related metadata, thus avoiding differences in state between webserver processes. To keep this AIP tractable, we propose to leverage the existing ORM models for storing and querying DAG metadata from the database. > > More information information on this AIP is available in cwiki. Feedback on the AIP is more than welcome! However, to keep the discussion centralized, I propose to discuss this AIP proposal in the comment section of cwiki. > > Best regards / met vriendelijke groet, > > Julian de Ruiter > Machine learning engineer > > ▉▉▉▉▉▉▉ GoDataDriven > Proudly part of the Xebia group > > M: +31 6 30 61 26 24 > W: http://www.godatadriven.com >
