Adam Dratwinski created KAFKA-6873:
--------------------------------------
Summary: Broker is not returning data including requested offset
Key: KAFKA-6873
URL: https://issues.apache.org/jira/browse/KAFKA-6873
Project: Kafka
Issue Type: Bug
Components: consumer
Affects Versions: 1.1.0
Environment: ubuntu
Reporter: Adam Dratwinski
After upgrading Kafka to 1.1.0 from 0.9.x I experience issues related with
broker returning incomplete responses. This happens for my all log compacted
topics. I am using Golang client (Sarama).
I debugged the issue and found that for some requests brokers return
FetchResponse with all messages having offsets lower then requested. For
example, I request for offset 1078831, I get FetchResponse with only one
message having offset 1078830, which produces missing blocks error. If I
request the next offset (1078832), then I get a block with many messages,
starting with much higher offset (e.g 1083813). There is a big gap in offsets
between these records, probably because I am using log compacted topics, but
all expected messages are there.
Sarama client treats this as consumer error:
{quote}kafka: response did not contain all the expected topic/partition blocks
{quote}
For build-in java client this issue is not happening. Looks like it is less
restrict regarding the data order, and when the offset is missing in the
returned block, it just simply request the next offset.
I reported this issue at Shopify/sarama Github project (see
[https://github.com/Shopify/sarama/issues/1087)], where I got response, that
this seems to be Kafka bug, as according to the documentation, in this
situation broker should never return only messages having lower offsets.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
