Re: [PR] Skip DB writes for unchanged Dags after serialization [airflow]
ephraimbuddy closed pull request #60670: Skip DB writes for unchanged Dags after serialization URL: https://github.com/apache/airflow/pull/60670 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Skip DB writes for unchanged Dags after serialization [airflow]
github-actions[bot] commented on PR #60670: URL: https://github.com/apache/airflow/pull/60670#issuecomment-4426291465 This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Skip DB writes for unchanged Dags after serialization [airflow]
uranusjr commented on code in PR #60670: URL: https://github.com/apache/airflow/pull/60670#discussion_r2851175075 ## airflow-core/src/airflow/dag_processing/collection.py: ## @@ -404,6 +598,16 @@ def update_dag_parsing_results_in_db( # Retry 'DAG.bulk_write_to_db' & 'SerializedDagModel.bulk_sync_to_db' in case # of any Operational Errors # In case of failures, provide_session handles rollback +dags_list = list(dags) Review Comment: I checked the code, `update_dag_parsing_results_in_db` is actually always called with a list, so let’s just change the function annotation to `Sequence[LazyDeserializedDAG]` instead and remove `dags_list`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Skip DB writes for unchanged Dags after serialization [airflow]
ephraimbuddy commented on code in PR #60670:
URL: https://github.com/apache/airflow/pull/60670#discussion_r2741286432
##
airflow-core/src/airflow/dag_processing/collection.py:
##
@@ -245,6 +248,190 @@ def _serialize_dag_capturing_errors(
]
+def _update_unchanged_dag_metadata(
+dag: LazyDeserializedDAG, *, bundle_name: str, session: Session
+) -> list[tuple[tuple[str, str], str]]:
+"""
+Update DagCode and Dag permissions for a Dag whose serialized payload is
unchanged.
+
+This keeps behavior consistent with _serialize_dag_capturing_errors
without re-serializing.
+"""
+from airflow.models.dagcode import DagCode
+
+try:
+DagCode.update_source_code(dag_id=dag.dag_id, fileloc=dag.fileloc,
session=session)
+if "FabAuthManager" in conf.get("core", "auth_manager"):
+_sync_dag_perms(dag, session=session)
+return []
+except OperationalError:
+raise
+except Exception:
+log.exception("Failed to update Dag metadata dag_id=%s fileloc=%s",
dag.dag_id, dag.fileloc)
+dagbag_import_error_traceback_depth = conf.getint("core",
"dagbag_import_error_traceback_depth")
+return [
+(
+(bundle_name, dag.relative_fileloc),
+
traceback.format_exc(limit=-dagbag_import_error_traceback_depth),
+)
+]
+
+
+def _update_dag_models_for_unchanged_dags(
+dags: Collection[LazyDeserializedDAG],
+*,
+bundle_name: str,
+bundle_version: str | None,
+parse_duration: float | None,
+session: Session,
+) -> None:
+if not dags:
+return
+log.debug("Updating DagModel for unchanged Dags", count=len(dags))
+dag_op = DagModelOperation(
+bundle_name=bundle_name,
+bundle_version=bundle_version,
+dags={dag.dag_id: dag for dag in dags},
+)
+orm_dags = dag_op.add_dags(session=session)
+dag_op.update_dags(orm_dags, parse_duration, session=session)
Review Comment:
I have implemented updates for specific fields.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
