Fokko commented on a change in pull request #4328: AIRFLOW-3530 - No re-parsing of dag files URL: https://github.com/apache/incubator-airflow/pull/4328#discussion_r242421319
########## File path: airflow/utils/dag_processing.py ########## @@ -44,101 +45,13 @@ # To avoid circular imports import airflow.models from airflow import configuration as conf -from airflow.dag.base_dag import BaseDag, BaseDagBag -from airflow.exceptions import AirflowException from airflow.settings import logging_class_path from airflow.utils import timezone from airflow.utils.db import provide_session from airflow.utils.log.logging_mixin import LoggingMixin from airflow.utils.state import State -class SimpleDag(BaseDag): Review comment: The SimpleDag is almost an exact copy of the dag, with the addition that it is serializable. I had a discussion with Peter, and I fully agree with him that the additional complexity of having this class hierarchy (`SimpleDag`, `BaseDag`, `Dag`) isn't worth the upsides that it brings. Firstly, my question would be, can we make the `Dag` in such a way that we can serialize this one as well. Secondly, why do we serialize in general? Afaik it was because of performance reasons (the webserver shows the dag based on the pickled instance), but I'm in doubt if we don't parse all the dags all the time if we still need this mechanism. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
