[
https://issues.apache.org/jira/browse/KAFKA-527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13741294#comment-13741294
]
Guozhang Wang commented on KAFKA-527:
-------------------------------------
One alternative approach would be like this:
Currently in the compression code (ByteBufferMessageSet.create), for each
message we write 1) the incrementing logical offset in LONG, 2) the message
byte size in INT, and 3) the message payload.
The idea is that since the logical offset is just incrementing, hence with a
compressed message, as long as we know the offset of the first message, we
would know the offset of the rest messages without even reading the offset
field.
So we can ignore reading the offset of each message inside of the compressed
message but only the offset of the wrapper message which is the offset of the
last message + 1, and then in assignOffsets just modify the offset of the
wrapper message. Another change would be at the consumer side, the iterator
would need to be smart of interpreting the offsets of messages while
deep-iterating the compressed message.
As Jay pointed out, this method would not work with log compaction since it
would break the assumption that offsets increments continuously. Two
workarounds of this issue:
1) In log compaction, instead of deleting the to-be-deleted-message just
setting its payload to null but keep its header and hence keeping its slot in
the incrementing offset.
2) During the compression process, instead of writing the absolute value of the
logical offset of messages, write the deltas of their offset compared with the
offset of the wrapper message. So -1 would mean continuously decrementing from
the wrapper message offset, and -2/3/... would be skipping holes in side the
compressed message.
> Compression support does numerous byte copies
> ---------------------------------------------
>
> Key: KAFKA-527
> URL: https://issues.apache.org/jira/browse/KAFKA-527
> Project: Kafka
> Issue Type: Bug
> Reporter: Jay Kreps
> Attachments: java.hprof.no-compression.txt, java.hprof.snappy.text
>
>
> The data path for compressing or decompressing messages is extremely
> inefficient. We do something like 7 (?) complete copies of the data, often
> for simple things like adding a 4 byte size to the front. I am not sure how
> this went by unnoticed.
> This is likely the root cause of the performance issues we saw in doing bulk
> recompression of data in mirror maker.
> The mismatch between the InputStream and OutputStream interfaces and the
> Message/MessageSet interfaces which are based on byte buffers is the cause of
> many of these.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
