Re: Kafka response ordering guarantees
Thanks Joel, it is clear now. /Ivan - Original message - From: Joel Koshy <jjkosh...@gmail.com> To: "users@kafka.apache.org" <users@kafka.apache.org> Subject: Re: Kafka response ordering guarantees Date: Thu, 18 Feb 2016 11:24:48 -0800 > > Does this mean that when a client is sending more than one in-flight >> request per connection, the server does not guarantee that responses will >> be sent in the same order as requests? > > > No - the server does provide this guarantee - i.e., responses will always > be sent in the same order as requests received on the same TCP connection. > *"The > server guarantees that on a single TCP connection, requests will be > processed in the order they are sent and responses will return in that > order as well."* The subsequent statements just describe how this is > achieved (i.e., we process only one request at a time in the API handlers). > To further clarify: the above is true even if the client sends more than on in-flight request per connection. > > On Thu, Feb 18, 2016 at 5:59 AM, Ivan Dyachkov <i...@dyachkov.org> wrote: > >> Thanks Ben. >> >> As I mentioned, I'm developing a kafka library and not using standard >> java producer. >> >> My question is really about protocol guarantees. >> >> /Ivan >> >> - Original message - >> From: Ben Stopford <b...@confluent.io> >> To: users@kafka.apache.org >> Subject: Re: Kafka response ordering guarantees >> Date: Wed, 17 Feb 2016 14:48:59 -0800 >> >> So long as you set max.inflight.requests.per.connection = 1 Kafka should >> provide strong ordering within a partition (so use the same key for >> messages that should retain their order). There is a bug currently raised >> agaisnt this feature though where there is an edge case that can cause >> ordering issues. >> >> https://issues.apache.org/jira/browse/KAFKA-3197 >> > On 17 Feb 2016, at 07:17, Ivan Dyachkov <d...@dyachkov.org> wrote: >> > >> > Hello all. >> > >> > I'm developing a kafka client and have a question about kafka server >> guarantees. >> > >> > A statement from >> https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-Network >> makes me a bit confused: >> > >> > "The server guarantees that on a single TCP connection, requests will >> be processed in the order they are sent and responses will return in that >> order as well. The broker's request processing allows only a single >> in-flight request per connection in order to guarantee this ordering. Note >> that clients can (and ideally should) use non-blocking IO to implement >> request pipelining and achieve higher throughput. i.e., clients can send >> requests even while awaiting responses for preceding requests since the >> outstanding requests will be buffered in the underlying OS socket buffer. >> All requests are initiated by the client, and result in a corresponding >> response message from the server except where noted." >> > >> > Does this mean that when a client is sending more than one in-flight >> request per connection, the server does not guarantee that responses will >> be sent in the same order as requests? >> > >> > In other words, if I have a strictly monotonically increasing integer >> as a correlation id for all requests, can I rely on Kafka that correlation >> id in responses will also have this property? >> > >> > Thanks. >> > >> > /Ivan >> >> >
Re: Kafka response ordering guarantees
> > Does this mean that when a client is sending more than one in-flight >> request per connection, the server does not guarantee that responses will >> be sent in the same order as requests? > > > No - the server does provide this guarantee - i.e., responses will always > be sent in the same order as requests received on the same TCP connection. > *"The > server guarantees that on a single TCP connection, requests will be > processed in the order they are sent and responses will return in that > order as well."* The subsequent statements just describe how this is > achieved (i.e., we process only one request at a time in the API handlers). > To further clarify: the above is true even if the client sends more than on in-flight request per connection. > > On Thu, Feb 18, 2016 at 5:59 AM, Ivan Dyachkov <i...@dyachkov.org> wrote: > >> Thanks Ben. >> >> As I mentioned, I'm developing a kafka library and not using standard >> java producer. >> >> My question is really about protocol guarantees. >> >> /Ivan >> >> - Original message - >> From: Ben Stopford <b...@confluent.io> >> To: users@kafka.apache.org >> Subject: Re: Kafka response ordering guarantees >> Date: Wed, 17 Feb 2016 14:48:59 -0800 >> >> So long as you set max.inflight.requests.per.connection = 1 Kafka should >> provide strong ordering within a partition (so use the same key for >> messages that should retain their order). There is a bug currently raised >> agaisnt this feature though where there is an edge case that can cause >> ordering issues. >> >> https://issues.apache.org/jira/browse/KAFKA-3197 >> > On 17 Feb 2016, at 07:17, Ivan Dyachkov <d...@dyachkov.org> wrote: >> > >> > Hello all. >> > >> > I'm developing a kafka client and have a question about kafka server >> guarantees. >> > >> > A statement from >> https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-Network >> makes me a bit confused: >> > >> > "The server guarantees that on a single TCP connection, requests will >> be processed in the order they are sent and responses will return in that >> order as well. The broker's request processing allows only a single >> in-flight request per connection in order to guarantee this ordering. Note >> that clients can (and ideally should) use non-blocking IO to implement >> request pipelining and achieve higher throughput. i.e., clients can send >> requests even while awaiting responses for preceding requests since the >> outstanding requests will be buffered in the underlying OS socket buffer. >> All requests are initiated by the client, and result in a corresponding >> response message from the server except where noted." >> > >> > Does this mean that when a client is sending more than one in-flight >> request per connection, the server does not guarantee that responses will >> be sent in the same order as requests? >> > >> > In other words, if I have a strictly monotonically increasing integer >> as a correlation id for all requests, can I rely on Kafka that correlation >> id in responses will also have this property? >> > >> > Thanks. >> > >> > /Ivan >> >> >
Re: Kafka response ordering guarantees
> Does this mean that when a client is sending more than one in-flight > request per connection, the server does not guarantee that responses will > be sent in the same order as requests? No - the server does provide this guarantee - i.e., responses will always be sent in the same order as requests received on the same TCP connection. *"The server guarantees that on a single TCP connection, requests will be processed in the order they are sent and responses will return in that order as well."* The subsequent statements just describe how this is achieved (i.e., we process only one request at a time in the API handlers). On Thu, Feb 18, 2016 at 5:59 AM, Ivan Dyachkov <i...@dyachkov.org> wrote: > Thanks Ben. > > As I mentioned, I'm developing a kafka library and not using standard java > producer. > > My question is really about protocol guarantees. > > /Ivan > > - Original message - > From: Ben Stopford <b...@confluent.io> > To: users@kafka.apache.org > Subject: Re: Kafka response ordering guarantees > Date: Wed, 17 Feb 2016 14:48:59 -0800 > > So long as you set max.inflight.requests.per.connection = 1 Kafka should > provide strong ordering within a partition (so use the same key for > messages that should retain their order). There is a bug currently raised > agaisnt this feature though where there is an edge case that can cause > ordering issues. > > https://issues.apache.org/jira/browse/KAFKA-3197 > > On 17 Feb 2016, at 07:17, Ivan Dyachkov <d...@dyachkov.org> wrote: > > > > Hello all. > > > > I'm developing a kafka client and have a question about kafka server > guarantees. > > > > A statement from > https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-Network > makes me a bit confused: > > > > "The server guarantees that on a single TCP connection, requests will be > processed in the order they are sent and responses will return in that > order as well. The broker's request processing allows only a single > in-flight request per connection in order to guarantee this ordering. Note > that clients can (and ideally should) use non-blocking IO to implement > request pipelining and achieve higher throughput. i.e., clients can send > requests even while awaiting responses for preceding requests since the > outstanding requests will be buffered in the underlying OS socket buffer. > All requests are initiated by the client, and result in a corresponding > response message from the server except where noted." > > > > Does this mean that when a client is sending more than one in-flight > request per connection, the server does not guarantee that responses will > be sent in the same order as requests? > > > > In other words, if I have a strictly monotonically increasing integer as > a correlation id for all requests, can I rely on Kafka that correlation id > in responses will also have this property? > > > > Thanks. > > > > /Ivan > >
Re: Kafka response ordering guarantees
Thanks Ben. As I mentioned, I'm developing a kafka library and not using standard java producer. My question is really about protocol guarantees. /Ivan - Original message - From: Ben Stopford <b...@confluent.io> To: users@kafka.apache.org Subject: Re: Kafka response ordering guarantees Date: Wed, 17 Feb 2016 14:48:59 -0800 So long as you set max.inflight.requests.per.connection = 1 Kafka should provide strong ordering within a partition (so use the same key for messages that should retain their order). There is a bug currently raised agaisnt this feature though where there is an edge case that can cause ordering issues. https://issues.apache.org/jira/browse/KAFKA-3197 > On 17 Feb 2016, at 07:17, Ivan Dyachkov <d...@dyachkov.org> wrote: > > Hello all. > > I'm developing a kafka client and have a question about kafka server > guarantees. > > A statement from > https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-Network > makes me a bit confused: > > "The server guarantees that on a single TCP connection, requests will be > processed in the order they are sent and responses will return in that order > as well. The broker's request processing allows only a single in-flight > request per connection in order to guarantee this ordering. Note that clients > can (and ideally should) use non-blocking IO to implement request pipelining > and achieve higher throughput. i.e., clients can send requests even while > awaiting responses for preceding requests since the outstanding requests will > be buffered in the underlying OS socket buffer. All requests are initiated by > the client, and result in a corresponding response message from the server > except where noted." > > Does this mean that when a client is sending more than one in-flight request > per connection, the server does not guarantee that responses will be sent in > the same order as requests? > > In other words, if I have a strictly monotonically increasing integer as a > correlation id for all requests, can I rely on Kafka that correlation id in > responses will also have this property? > > Thanks. > > /Ivan
Re: Kafka response ordering guarantees
So long as you set max.inflight.requests.per.connection = 1 Kafka should provide strong ordering within a partition (so use the same key for messages that should retain their order). There is a bug currently raised agaisnt this feature though where there is an edge case that can cause ordering issues. https://issues.apache.org/jira/browse/KAFKA-3197 > On 17 Feb 2016, at 07:17, Ivan Dyachkovwrote: > > Hello all. > > I'm developing a kafka client and have a question about kafka server > guarantees. > > A statement from > https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-Network > makes me a bit confused: > > "The server guarantees that on a single TCP connection, requests will be > processed in the order they are sent and responses will return in that order > as well. The broker's request processing allows only a single in-flight > request per connection in order to guarantee this ordering. Note that clients > can (and ideally should) use non-blocking IO to implement request pipelining > and achieve higher throughput. i.e., clients can send requests even while > awaiting responses for preceding requests since the outstanding requests will > be buffered in the underlying OS socket buffer. All requests are initiated by > the client, and result in a corresponding response message from the server > except where noted." > > Does this mean that when a client is sending more than one in-flight request > per connection, the server does not guarantee that responses will be sent in > the same order as requests? > > In other words, if I have a strictly monotonically increasing integer as a > correlation id for all requests, can I rely on Kafka that correlation id in > responses will also have this property? > > Thanks. > > /Ivan
Kafka response ordering guarantees
Hello all. I'm developing a kafka client and have a question about kafka server guarantees. A statement from https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-Network makes me a bit confused: "The server guarantees that on a single TCP connection, requests will be processed in the order they are sent and responses will return in that order as well. The broker's request processing allows only a single in-flight request per connection in order to guarantee this ordering. Note that clients can (and ideally should) use non-blocking IO to implement request pipelining and achieve higher throughput. i.e., clients can send requests even while awaiting responses for preceding requests since the outstanding requests will be buffered in the underlying OS socket buffer. All requests are initiated by the client, and result in a corresponding response message from the server except where noted." Does this mean that when a client is sending more than one in-flight request per connection, the server does not guarantee that responses will be sent in the same order as requests? In other words, if I have a strictly monotonically increasing integer as a correlation id for all requests, can I rely on Kafka that correlation id in responses will also have this property? Thanks. /Ivan