divijvaidya opened a new pull request, #13312: URL: https://github.com/apache/kafka/pull/13312
# Motivation Reading/writing the protocol buffer varInt32 and varInt64 (also called varLong in our code base) is in the hot path of data plane code in Apache Kafka. We read multiple varInt in a record and in long. Hence, even a minor change in performance could extrapolate to larger performance benefit. # Changes This change uses loop unrolling and reduces the amount of repetition of calculations. # Results Performance has been evaluated using JMH benchmarks. The benefits of loop unrolling diminish as the size of loop increases (from Int32 to Int64) ``` ByteUtilsBenchmark.testUnsignedReadVarintNew thrpt 5 51187.160 ± 778.033 ops/s ByteUtilsBenchmark.testUnsignedReadVarintOld thrpt 5 43441.115 ± 1281.592 ops/s ByteUtilsBenchmark.testUnsignedReadVarlongNew thrpt 5 28293.582 ± 952.091 ops/s ByteUtilsBenchmark.testUnsignedReadVarlongOld thrpt 5 22734.384 ± 1132.111 ops/s ByteUtilsBenchmark.testUnsignedWriteVarintNew thrpt 5 134848.804 ± 2464.908 ops/s ByteUtilsBenchmark.testUnsignedWriteVarintOld thrpt 5 73333.919 ± 3306.315 ops/s ByteUtilsBenchmark.testUnsignedWriteVarlongNew thrpt 5 40252.868 ± 2551.431 ops/s ByteUtilsBenchmark.testUnsignedWriteVarlongOld thrpt 5 34948.229 ± 1877.987 ops/s ``` # Testing New tests have been added which validate that the value produced by prior algorithm is same as new algorithm. This validates correctness. UnitTest and IntegrationTest are successful locally. # References - https://steinborn.me/posts/performance/how-fast-can-you-write-a-varint/ is a nice blog which explains the optimizations for writing varInt32. - https://github.com/netty/netty/blob/59aa6e635b9996cf21cd946e64353270679adc73/codec/src/main/java/io/netty/handler/codec/protobuf/ProtobufVarint32FrameDecoder.java#L73 is Netty's unrolled implementation of readVarInt32 - https://github.com/protocolbuffers/protobuf/blob/22.x/java/core/src/main/java/com/google/protobuf/CodedOutputStream.java#L1345 is ProtoBuf's loop based implementation -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
