[
https://issues.apache.org/jira/browse/KAFKA-21082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chia-Ping Tsai updated KAFKA-21082:
-----------------------------------
Description:
-There are a few paths with the following pattern:-
# -the records could be compressed-
# -the records are dropped quickly (they never outlive the iterator)-
# -the key and value are needed, so {{skipKeyValueIterator}} is not an option-
-That means numerous buffers will be created for those records and then get
dropped along with the records. That is wasteful, so we could pass a
{{BufferSupplier}} to {{streamingIterator}} for the record bodies and release
the buffers when the iterator is closed.-
-{{Cleaner.buildOffsetMapForSegment}} has the first priority, since most normal
records get compressed. {{RecordsIterator}} and {{CoordinatorLoaderImpl}} are
candidates, but I'm not sure whether they get compressed usually 🙂-
another idea:Â The record is valid but exceeds the limit. All the cleaner needs
is the "key's hash" and "whether value is null". Hence, we could add a variant
of readPartiallyFrom which feeds the key into the digest in fixed chunks
(OffsetMap only needs the hash anyway) and skips the value, since null can be
checked by its size.
was:
-There are a few paths with the following pattern:-
# -the records could be compressed-
# -the records are dropped quickly (they never outlive the iterator)-
# -the key and value are needed, so {{skipKeyValueIterator}} is not an option-
-That means numerous buffers will be created for those records and then get
dropped along with the records. That is wasteful, so we could pass a
{{BufferSupplier}} to {{streamingIterator}} for the record bodies and release
the buffers when the iterator is closed.-
-{{Cleaner.buildOffsetMapForSegment}} has the first priority, since most normal
records get compressed. {{RecordsIterator}} and {{CoordinatorLoaderImpl}} are
candidates, but I'm not sure whether they get compressed usually 🙂-
> Reuse the buffer in decompression path if the records are dropped quickly
> -------------------------------------------------------------------------
>
> Key: KAFKA-21082
> URL: https://issues.apache.org/jira/browse/KAFKA-21082
> Project: Kafka
> Issue Type: Improvement
> Reporter: Chia-Ping Tsai
> Assignee: majialong
> Priority: Minor
>
> -There are a few paths with the following pattern:-
> # -the records could be compressed-
> # -the records are dropped quickly (they never outlive the iterator)-
> # -the key and value are needed, so {{skipKeyValueIterator}} is not an
> option-
> -That means numerous buffers will be created for those records and then get
> dropped along with the records. That is wasteful, so we could pass a
> {{BufferSupplier}} to {{streamingIterator}} for the record bodies and release
> the buffers when the iterator is closed.-
> -{{Cleaner.buildOffsetMapForSegment}} has the first priority, since most
> normal records get compressed. {{RecordsIterator}} and
> {{CoordinatorLoaderImpl}} are candidates, but I'm not sure whether they get
> compressed usually 🙂-
> another idea:Â The record is valid but exceeds the limit. All the cleaner
> needs is the "key's hash" and "whether value is null". Hence, we could add a
> variant of readPartiallyFrom which feeds the key into the digest in fixed
> chunks (OffsetMap only needs the hash anyway) and skips the value, since null
> can be checked by its size.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
