Re: Compression - MessageSet size

2016-02-17 Thread Alexis Midon
- it would be interesting to see the actual ProduceRequests/Responses and FetchReq/Resp. - at this point I would dive into the broker source code and follow the fetch request handling. https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/server/KafkaApis.scala#L418 On Tue, Feb

Re: Compression - MessageSet size

2016-02-16 Thread Oleksiy Krivoshey
I'm not using Java client, I'm developing my own: https://github.com/oleksiyk/kafka And I'm talking about MessageSet on the wire, not the one passed to library user. Example: Consumer is not started. Producer sends a batch #1 of 5 messages, compressed in a single message. delay 50 ms Producer

Re: Compression - MessageSet size

2016-02-16 Thread Alexis Midon
What makes you think there are 2? would you have data or code to share? When compression is enabled, multiple messages will be packed and compressed in a MessageSet. That MessageSet will then have a single message. The interface however will let you iterate over the unpacked messages. See

Compression - MessageSet size

2016-02-16 Thread Oleksiy Krivoshey
Hi! The Kafka 0.9 protocol guide https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-Compression explicitly states that there should be only single compressed message in a MessageSet, but I'm definitely receiving two compressed messages in a