[
https://issues.apache.org/jira/browse/KAFKA-21082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chia-Ping Tsai updated KAFKA-21082:
-----------------------------------
Summary: Build offset map without materializing record bodies (was: Reuse
the buffer in decompression path if the records are dropped quickly)
> Build offset map without materializing record bodies
> ----------------------------------------------------
>
> Key: KAFKA-21082
> URL: https://issues.apache.org/jira/browse/KAFKA-21082
> Project: Kafka
> Issue Type: Improvement
> Reporter: Chia-Ping Tsai
> Assignee: majialong
> Priority: Minor
>
> -There are a few paths with the following pattern:-
> # -the records could be compressed-
> # -the records are dropped quickly (they never outlive the iterator)-
> # -the key and value are needed, so {{skipKeyValueIterator}} is not an
> option-
> -That means numerous buffers will be created for those records and then get
> dropped along with the records. That is wasteful, so we could pass a
> {{BufferSupplier}} to {{streamingIterator}} for the record bodies and release
> the buffers when the iterator is closed.-
> -{{Cleaner.buildOffsetMapForSegment}} has the first priority, since most
> normal records get compressed. {{RecordsIterator}} and
> {{CoordinatorLoaderImpl}} are candidates, but I'm not sure whether they get
> compressed usually 🙂-
> another idea:Â The record is valid but exceeds the limit. All the cleaner
> needs is the "key's hash" and "whether value is null". Hence, we could add a
> variant of readPartiallyFrom which feeds the key into the digest in fixed
> chunks (OffsetMap only needs the hash anyway) and skips the value, since null
> can be checked by its size.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
