[ https://issues.apache.org/jira/browse/KAFKA-3174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15126959#comment-15126959 ]
Jiangjie Qin commented on KAFKA-3174: ------------------------------------- [~ijuma] My test code was as below. Initially I only tested 1MB size. I added testing on different size after saw your comments above. {code} public static void main(String[] args) { int[] sizes = {8, 16, 32, 128, 1024, 65536, 1048576}; for (int size : sizes) { byte[] bytes = new byte[size]; Random random = new Random(); random.nextBytes(bytes); int loop = 5000 * (1048576 / size); long start = System.currentTimeMillis(); for (int i = 0; i < loop; i++) { Crc32 crc32 = new Crc32(); crc32.update(bytes, 0, bytes.length); } System.out.println(String.format("KCrc32: Size = %d\t, time = %d", size, (System.currentTimeMillis() - start))); start = System.currentTimeMillis(); for (int i = 0; i < loop; i++) { CRC32 crc32 = new CRC32(); crc32.update(bytes, 0, bytes.length); } System.out.println(String.format("JCrc32: Size = %d\t, time = %d\n", size, (System.currentTimeMillis() - start))); } } {code} And here is the output: {code} KCrc32: Size = 8 , time = 10400 JCrc32: Size = 8 , time = 9907 KCrc32: Size = 16 , time = 6959 JCrc32: Size = 16 , time = 8419 KCrc32: Size = 32 , time = 5596 JCrc32: Size = 32 , time = 5587 KCrc32: Size = 128 , time = 4397 JCrc32: Size = 128 , time = 3305 KCrc32: Size = 1024 , time = 4115 JCrc32: Size = 1024 , time = 2392 KCrc32: Size = 65536 , time = 4087 JCrc32: Size = 65536 , time = 2296 KCrc32: Size = 1048576 , time = 4078 JCrc32: Size = 1048576 , time = 2298 {code} >From the output above, it looks for size < 32 bytes KCrc32 and JCrc32 is >comparable (except 16 bytes). After size >= 64, JCrc32 is faster. My ~2x >result came from 1MB size. In this Intel paper, they mentioned that the CRC32 instruction is actually in SSE4.2, which was introduced in Nov 2008. http://www.intel.com/content/dam/www/public/us/en/documents/white-papers/crc-iscsi-polynomial-crc32-instruction-paper.pdf Wikipedia says the same thing. https://en.wikipedia.org/wiki/SSE4#SSE4.2 AMD started to support SSE4.2 in Oct 2011. I ran the above test on both my desktop (Intel(R) Xeon(R) CPU E5-2620 v2 @ 2.10GHz) and macbook (Intel(R) Core(TM) i7-4558U CPU @ 2.80GHz), both of them have SSE4.2 support. > Re-evaluate the CRC32 class performance. > ---------------------------------------- > > Key: KAFKA-3174 > URL: https://issues.apache.org/jira/browse/KAFKA-3174 > Project: Kafka > Issue Type: Improvement > Affects Versions: 0.9.0.0 > Reporter: Jiangjie Qin > Assignee: Jiangjie Qin > Fix For: 0.9.0.1 > > > We used org.apache.kafka.common.utils.CRC32 in clients because it has better > performance than java.util.zip.CRC32 in Java 1.6. > In a recent test I ran it looks in Java 1.8 the CRC32 class is 2x as fast as > the Crc32 class we are using now. We may want to re-evaluate the performance > of Crc32 class and see it makes sense to simply use java CRC32 instead. -- This message was sent by Atlassian JIRA (v6.3.4#6332)