Andrushika commented on code in PR #66854: URL: https://github.com/apache/airflow/pull/66854#discussion_r3610816513
########## airflow-core/newsfragments/66854.significant.rst: ########## @@ -0,0 +1,33 @@ +Asset events from successful tasks are registered asynchronously + +When a task succeeds, its asset events (and the resulting scheduling of downstream +asset-triggered Dags) are no longer always written synchronously inside the task +state-update request. The execution API now attempts the write once the task state is +committed and its row lock released, and if it cannot complete inline (for example under +database lock contention) it hands the events to a new ``asset_event_queue`` table that the +scheduler drains and retries until they are registered. This removes the long-running asset +work from the task-completion request path, which previously held the ``task_instance`` row +lock and could pile up requests and exhaust the API server under high fan-out. + +As a result there can be a short delay between a task succeeding and its asset events being +visible (typically one scheduler loop, longer under sustained load or contention). Asset +events are never dropped: the scheduler keeps retrying a deferred registration until it Review Comment: Small wording thing: "never dropped" and "keeps retrying until it succeeds" promise a bit more than the code does. The drain gives up after `asset_event_queue_max_attempts` and parks the row, and under `dag.test` there is no scheduler to drain at all. Maybe something like "retried up to `asset_event_queue_max_attempts` times, then parked and reported via the asset.event_queue.failures metric"? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
