uranusjr commented on code in PR #67523: URL: https://github.com/apache/airflow/pull/67523#discussion_r3393548657
########## airflow-core/docs/authoring-and-scheduling/event-scheduling.rst: ########## @@ -150,24 +150,180 @@ In other words, the savings is at the poll-loop and upstream-I/O layer, not at t Suitable upstreams ^^^^^^^^^^^^^^^^^^ -The shared-stream channel is **one-way** today: events flow from -``open_shared_stream`` out to each subscriber's ``filter_shared_stream``, -and there is no way for a subscriber to tell the producer "I accepted / -dropped / committed this event". That restricts the pattern to upstreams -whose consumption does **not** depend on a side effect on a handle that -only the producer holds. Good fits: +Good fits for the shared-stream pattern: * Idempotent / read-only reads — directory scans, polling REST APIs. * Subscriber-side-effect cleanup, where the trigger's per-event action (``unlink``, local marking, …) goes through APIs the subscriber owns independently of the shared producer handle. +* Message-broker upstreams (Kafka, SQS, Pub/Sub, Azure Service Bus) where + the producer must commit/delete/ack after all subscribers have processed + the message — use the ack channel described below. + +Producer-side ack channel +^^^^^^^^^^^^^^^^^^^^^^^^^ + +For upstreams where the producer must advance (commit, delete, or ack) only +after all subscribers have processed an event, override +:py:meth:`~airflow.triggers.base.BaseEventTrigger.create_shared_stream_producer` +to return a :py:class:`~airflow.triggers.shared_stream.SharedStreamProducer`. Review Comment: I think `:py` is not needed, it is inferred by Sphinx when you write simply `:meth:` and `:class:`. Not entirely sure. Not too big a problem if it’s spelt out explitly either. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
