[
https://issues.apache.org/jira/browse/KAFKA-374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jay Kreps updated KAFKA-374:
----------------------------
Attachment: KAFKA-374-draft.patch
The following is a draft patch that transliterates the Hadoop CRC32
implementation into scala and swaps it in for our own crc32 implementation. I
ran a sanity check on a few hundred million values to check I got the same crcs.
The CRC code is essentially unreadable. So it is not something you can really
edit.
I wrote a simple test that just sequentially creates messages (which requires
computing the checksum). On this test I see about a 30-40% improvement, roughly
across the board on Linux. I will include full results when it completes.
I suspect this really only helps very trivial cases: benchmarks, mirroring, or
other applications which do no real processing. It should speed up both the
broker, the producer, and the consumer.
> Move to java CRC32 implementation
> ---------------------------------
>
> Key: KAFKA-374
> URL: https://issues.apache.org/jira/browse/KAFKA-374
> Project: Kafka
> Issue Type: New Feature
> Components: core
> Affects Versions: 0.8
> Reporter: Jay Kreps
> Priority: Minor
> Labels: newbie
> Attachments: KAFKA-374-draft.patch
>
>
> We keep a per-record crc32. This is fairly cheap algorithm, but the java
> implementation uses JNI and it seems to be a bit expensive for small records.
> I have seen this before in Kafka profiles, and I noticed it on another
> application I was working on. Basically with small records the native
> implementation can only checksum < 100MB/sec. Hadoop has done some analysis
> of this and replaced it with a Java implementation that is 2x faster for
> large values and 5-10x faster for small values. Details are here HADOOP-6148.
> We should do a quick read/write benchmark on log and message set iteration
> and see if this improves things.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
