mingyen066 commented on PR #23477: URL: https://github.com/apache/kafka/pull/23477#issuecomment-5693217585
Benchmarks (JMH with `-prof gc`; `gc.alloc.rate.norm` is bytes allocated per operation; single fork, 3 warmup + 5 measurement iterations of 1s; Apple Silicon, JDK 17): 1. `RecordBatchIterationBenchmark`, already in trunk: `measureStreamingIteratorForVariableBatchSize` vs `measureSkipIteratorForVariableBatchSize` (v2, maxBatchSize=200, messageSize=1000, RANDOM, NO_CACHING; 100 batches per op). | compression | streaming B/op | skip B/op | saved | |---|---:|---:|---:| | NONE | 90,120 | 90,120 | 0% | | GZIP | 243,530 | 32,811 | 87% | | LZ4 | 285,592 | 74,873 | 74% | | ZSTD | 359,726 | 154,769 | 57% | 2. `FileRecords.searchForTimestamp`, trunk vs this branch, on a segment of 10 batches x 100 records with the target in the last record of the first batch, so each op decodes one full batch through `FileChannelRecordBatch`. This is a one-off benchmark that is not part of the PR; source available on request. | compression | value bytes | trunk B/op | branch B/op | alloc saved | trunk us/op | branch us/op | |---|---:|---:|---:|---:|---:|---:| | NONE | 100 | 29,008 | 28,984 | 0% | 4.5 | 4.4 | | NONE | 1,000 | 119,008 | 119,008 | 0% | 11.2 | 10.5 | | NONE | 10,000 | 1,019,208 | 1,019,208 | 0% | 70.8 | 63.7 | | LZ4 | 100 | 110,168 | 86,248 | 22% | 7.2 | 6.6 | | LZ4 | 1,000 | 290,272 | 176,584 | 39% | 18.0 | 12.2 | | LZ4 | 10,000 | 2,091,249 | 1,077,624 | 48% | 118.5 | 71.8 | | ZSTD | 100 | 189,496 | 165,576 | 13% | 28.4 | 26.6 | | ZSTD | 1,000 | 369,984 | 256,384 | 31% | 48.4 | 47.5 | | ZSTD | 10,000 | 2,170,202 | 1,156,601 | 47% | 280.9 | 238.7 | Uncompressed batches are unchanged, as expected: both iterators return the same zero-copy iterator for them. What the branch still allocates for compressed batches is the batch buffer loaded from the file, the decompression buffers (NO_CACHING) and the per-record `PartialDefaultRecord` objects. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
