[
https://issues.apache.org/jira/browse/KAFKA-9373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17009314#comment-17009314
]
ASF GitHub Bot commented on KAFKA-9373:
---------------------------------------
efeg commented on pull request #7900: KAFKA-9373: Improve shutdown performance
via lazy accessing the offset and time indices
URL: https://github.com/apache/kafka/pull/7900
KAFKA-7283 enabled lazy mmap on index files by initializing indices
on-demand rather than performing costly disk/memory operations when creating
all indices on broker startup. This helped reducing the startup time of
brokers. However, segment indices are still created on closing segments,
regardless of whether they need to be closed or not.
This patch:
* Improves shutdown performance via lazy accessing the offset and time
indices.
* Eliminates redundant disk accesses and memory mapped operations while
deleting or renaming files that back segment indices.
* Prevents illegal accesses to underlying indices of a closed segment, which
would lead to memory leaks due to recreation of the underlying memory mapped
objects.
In our evaluations in a cluster with 31 brokers, where each broker has 13K
to 20K segments, we observed up to 2 orders of magnitude faster LogManager
shutdown times with this patch -- i.e. dropping the LogManager shutdown time of
each broker from 10s of seconds to 100s of milliseconds.
### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Improve shutdown performance via lazy accessing the offset and time indices.
> ----------------------------------------------------------------------------
>
> Key: KAFKA-9373
> URL: https://issues.apache.org/jira/browse/KAFKA-9373
> Project: Kafka
> Issue Type: Bug
> Components: log
> Affects Versions: 2.3.0, 2.4.0, 2.3.1
> Reporter: Adem Efe Gencer
> Assignee: Adem Efe Gencer
> Priority: Major
> Fix For: 2.3.0, 2.4.0, 2.3.1
>
>
> KAFKA-7283 enabled lazy mmap on index files by initializing indices on-demand
> rather than performing costly disk/memory operations when creating all
> indices on broker startup. This helped reducing the startup time of brokers.
> However, segment indices are still created on closing segments, regardless of
> whether they need to be closed or not.
>
> Ideally we should:
> * Improve shutdown performance via lazy accessing the offset and time
> indices.
> * Eliminate redundant disk accesses and memory mapped operations while
> deleting or renaming files that back segment indices.
> * Prevent illegal accesses to underlying indices of a closed segment, which
> would lead to memory leaks due to recreation of the underlying memory mapped
> objects.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
