[
https://issues.apache.org/jira/browse/KAFKA-2189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Olson,Andrew updated KAFKA-2189:
--------------------------------
Component/s: compression
build
> Snappy compression of message batches less efficient in 0.8.2.1
> ---------------------------------------------------------------
>
> Key: KAFKA-2189
> URL: https://issues.apache.org/jira/browse/KAFKA-2189
> Project: Kafka
> Issue Type: Bug
> Components: build, compression, log
> Affects Versions: 0.8.2.1
> Reporter: Olson,Andrew
> Assignee: Jay Kreps
>
> We are using snappy compression and noticed a fairly substantial increase
> (about 2.25x) in log filesystem space consumption after upgrading a Kafka
> cluster from 0.8.1.1 to 0.8.2.1. We found that this is caused by messages
> being seemingly recompressed individually (or possibly with a much smaller
> buffer or dictionary?) instead of as a batch as sent by producers. We
> eventually tracked down the change in compression ratio/scope to this [1]
> commit that updated the snappy version from 1.0.5 to 1.1.1.3. The Kafka
> client version does not appear to be relevant as we can reproduce this with
> both the 0.8.1.1 and 0.8.2.1 Producer.
> Here are the log files from our troubleshooting that contain the same set of
> 1000 messages, for batch sizes of 1, 10, 100, and 1000. f9d9b was the last
> commit with 0.8.1.1-like behavior prior to f5ab8 introducing the issue.
> {noformat}
> -rw-rw-r-- 1 kafka kafka 404967 May 12 11:45
> /var/kafka2/f9d9b-batch-1-0/00000000000000000000.log
> -rw-rw-r-- 1 kafka kafka 119951 May 12 11:45
> /var/kafka2/f9d9b-batch-10-0/00000000000000000000.log
> -rw-rw-r-- 1 kafka kafka 89645 May 12 11:45
> /var/kafka2/f9d9b-batch-100-0/00000000000000000000.log
> -rw-rw-r-- 1 kafka kafka 88279 May 12 11:45
> /var/kafka2/f9d9b-batch-1000-0/00000000000000000000.log
> -rw-rw-r-- 1 kafka kafka 402837 May 12 11:41
> /var/kafka2/f5ab8-batch-1-0/00000000000000000000.log
> -rw-rw-r-- 1 kafka kafka 382437 May 12 11:41
> /var/kafka2/f5ab8-batch-10-0/00000000000000000000.log
> -rw-rw-r-- 1 kafka kafka 364791 May 12 11:41
> /var/kafka2/f5ab8-batch-100-0/00000000000000000000.log
> -rw-rw-r-- 1 kafka kafka 380693 May 12 11:41
> /var/kafka2/f5ab8-batch-1000-0/00000000000000000000.log
> {noformat}
> [1]
> https://github.com/apache/kafka/commit/f5ab8e1780cf80f267906e3259ad4f9278c32d28
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
