hkc-8010 commented on code in PR #66854: URL: https://github.com/apache/airflow/pull/66854#discussion_r3619808054
########## airflow-core/newsfragments/66854.significant.rst: ########## @@ -0,0 +1,33 @@ +Asset events from successful tasks are registered asynchronously + +When a task succeeds, its asset events (and the resulting scheduling of downstream +asset-triggered Dags) are no longer always written synchronously inside the task +state-update request. The execution API now attempts the write once the task state is +committed and its row lock released, and if it cannot complete inline (for example under +database lock contention) it hands the events to a new ``asset_event_queue`` table that the +scheduler drains and retries until they are registered. This removes the long-running asset +work from the task-completion request path, which previously held the ``task_instance`` row +lock and could pile up requests and exhaust the API server under high fan-out. + +As a result there can be a short delay between a task succeeding and its asset events being +visible (typically one scheduler loop, longer under sustained load or contention). Asset +events are never dropped: the scheduler keeps retrying a deferred registration until it Review Comment: Good catch, reworded. It now says a deferred registration is retried up to `asset_event_queue_max_attempts` times, then parked and surfaced via the `asset.event_queue.failures` metric, and it calls out that `dag.test` has no scheduler to drain the queue. Same fix applied to the asset-scheduling doc note. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
