kaxil commented on code in PR #74789: URL: https://github.com/apache/airflow/pull/74789#discussion_r4483631645
########## airflow-core/docs/administration-and-deployment/dag-bundles.rst: ########## @@ -261,6 +261,20 @@ Some caveats: * A failing refresh looks like one that is not due, since a refresh that raises is logged and leaves ``last_refreshed`` at its last success. Check the Dag processor logs, and ``/api/v2/jobs?job_type=DagProcessorJob`` for a live processor. + If the refresh may have changed files, parsing pauses until a refresh succeeds. The processor + logs the affected bundle and reports the total number of deferred files through + ``dag_processing.files_awaiting_source``. On metrics backends that support tags, + ``dag_processing.files_awaiting_source_by_bundle`` provides counts tagged with ``bundle_name``, and + with ``team_name`` in multi-team mode. + A provider that reports an unchanged source after failure lets ordinary parsing continue against + the previous source while refresh retries continue. Currently only ``GitDagBundle`` reports this, + for credential and fetch failures, in git provider releases that support it; the S3, GCS and + Azure Blob Storage bundles pause on any refresh failure. Explicit reparse requests wait for a + successful refresh. A Dag processor started with ``--num-runs`` does not exit while files wait for + a paused bundle, so it finishes only after that bundle refreshes successfully. Review Comment: Before this, a failed S3, GCS or Azure refresh was logged and parsing carried on. Now that bundle's Dags stop updating until a refresh succeeds, and a `--num-runs` processor won't exit meanwhile. Should this get a `significant` newsfragment so upgraders see it? ########## airflow-core/docs/administration-and-deployment/dag-bundles.rst: ########## @@ -261,6 +261,20 @@ Some caveats: * A failing refresh looks like one that is not due, since a refresh that raises is logged and leaves ``last_refreshed`` at its last success. Check the Dag processor logs, and ``/api/v2/jobs?job_type=DagProcessorJob`` for a live processor. + If the refresh may have changed files, parsing pauses until a refresh succeeds. The processor + logs the affected bundle and reports the total number of deferred files through + ``dag_processing.files_awaiting_source``. On metrics backends that support tags, + ``dag_processing.files_awaiting_source_by_bundle`` provides counts tagged with ``bundle_name``, and + with ``team_name`` in multi-team mode. + A provider that reports an unchanged source after failure lets ordinary parsing continue against + the previous source while refresh retries continue. Currently only ``GitDagBundle`` reports this, + for credential and fetch failures, in git provider releases that support it; the S3, GCS and + Azure Blob Storage bundles pause on any refresh failure. Explicit reparse requests wait for a + successful refresh. A Dag processor started with ``--num-runs`` does not exit while files wait for + a paused bundle, so it finishes only after that bundle refreshes successfully. + If listing the bundle's Dag files or recording the refresh fails after a successful refresh, the + processor keeps parsing the files it listed before, from the refreshed source, and retries the + refresh no later than the bundle's :ref:`config:dag_processor__refresh_interval`. Review Comment: `_delay_draining_refresh_retry` holds the retry for at least `refresh_interval` on Git, S3 and GCS bundles, so I think this wants "after" rather than "no later than". The "files it listed before" bit is only true when the listing fails; line 1078 of `manager.py` takes the new listing when just `update_bundle_state` fails. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
