> Is this because all messages created by the producer are put into a > message queue for that topic? The work flow being the entire queue is > flushed to the broker and the client will continue to re-send until > broker ack's. Is that correct?
Correct. For the implementation, we rely on TCP to ensure the messages are ordered while in transit from client to broker (and from broker to bookkeeper). Then, the client, broker, and bookkeeper are carefully implemented to ensure order is maintained. > This is the scenario I was thinking about. Messages 1, 2, 3 are sent and > ack'ed by the broker. Messages 4, 5, 6 are sent but somehow 4 and 5 did > not get ack'ed by the broker but 6 did. This scenario should not happen and would be a bug if it did. Pulsar is designed so that message 6 can only be persisted if message 4 and message 5 are persisted. > With Pulsar and the async message queue, I'm assuming 4, 5 and 6 would > be sent as one "unit" to the broker so all succeed or do not? There is no requirement for all to succeed. The only guarantee is that they will persist in order. As such, 4 could succeed while 5 and 6 fail. - Michael On Tue, Mar 14, 2023 at 12:01 PM Adam Brenner via users <users@pulsar.apache.org> wrote: > > Matteo: thanks for the fast reply! Few follow up questions inline: > > On 3/13/23 14:52, Matteo Merli wrote: > > Hi Adam, > > > > On Mon, Mar 13, 2023 at 2:24 PM Adam Brenner via users > > <users@pulsar.apache.org <mailto:users@pulsar.apache.org>> wrote: > > > > Hello, > > > > I have a few questions regarding Pulsar Async mode and what, if any, > > guarantees are made with respect to packet ordering: > > > > 1) Does Pulsar offer any guarantee packet ordering in Async send mode? > > Let us assume best case with single producer and exclusive access mode. > > > > > > Yes, messages are guaranteed to be sent in order in both sync & async mode. > > Is this because all messages created by the producer are put into a > message queue for that topic? The work flow being the entire queue is > flushed to the broker and the client will continue to re-send until > broker ack's. Is that correct? > > Since everything is being operated as a message queue, order is thus > guaranteed? > > > > > 2) Does the client library do any coordination with Pulsar Broker to > > guarantee packet ordering in async send mode? > > > > > > In case of failures, the client library ensures that all messages > > pending in the producer queue are resent in the same order. > > > > This could lead to a sequence of dups, (unless the exactly once mode is > > enabled), though you wouldn't see a message out of order. > > > > NOTE: to have a 100% guarantee you need to disable publish timeout > > (sendTimeout=0). This is mostly to ease you application code. > > In the presence of a timeout, Pulsar client guarantees anyway that any > > message present in the queue will also fail, even if not all of them > > have already timed out. > > > The application though could be already trying to publish the next > > message. Unless there's proper synchronization. For that, the easier way > > is to disable timeouts and let the Pulsar client retrying until a > > successful publish. > > This is the scenario I was thinking about. Messages 1, 2, 3 are sent and > ack'ed by the broker. Messages 4, 5, 6 are sent but somehow 4 and 5 did > not get ack'ed by the broker but 6 did. > > With Pulsar and the async message queue, I'm assuming 4, 5 and 6 would > be sent as one "unit" to the broker so all succeed or do not? > > > > > > > 3) Would using batch mode with async send mode help in any way to > > guaranteed packet ordering? > > > > > > Batching will improve the throughput and the overall efficiency, less > > CPU usage, and better compression ratio, though it doesn't affect ordering. > > > > > > > > -- > > Matteo Merli > > <mme...@apache.org <mailto:mme...@apache.org>> > > Thanks, > -Adam >
