[jira] [Commented] (KAFKA-7557) optimize LogManager.truncateFullyAndStartAt()

2018-10-26 Thread Jun Rao (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-7557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16665737#comment-16665737
 ] 

Jun Rao commented on KAFKA-7557:


To improve this, instead of calling 
Log.deleteSnapshotsAfterRecoveryPointCheckpoint() on all logs, we could 
probably call only the one that's being truncated.

> optimize LogManager.truncateFullyAndStartAt()
> -
>
> Key: KAFKA-7557
> URL: https://issues.apache.org/jira/browse/KAFKA-7557
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 2.0.0, 2.1.0
>Reporter: Jun Rao
>Priority: Major
>
> When a ReplicaFetcherThread calls LogManager.truncateFullyAndStartAt() for a 
> partition, we call LogManager.checkpointLogRecoveryOffsetsInDir() and then 
> Log.deleteSnapshotsAfterRecoveryPointCheckpoint() on all the logs in that 
> directory. This requires listing all the files in each log dir to figure out 
> the snapshot files. If some logs have many log segment files. This could take 
> some time. The can potentially block a replica fetcher thread, which 
> indirectly causes the request handler threads to be blocked.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (KAFKA-7557) optimize LogManager.truncateFullyAndStartAt()

2018-10-27 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-7557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1250#comment-1250
 ] 

ASF GitHub Bot commented on KAFKA-7557:
---

huxihx opened a new pull request #5848: KAFKA-7557: optimize 
LogManager.truncateFullyAndStartAt()
URL: https://github.com/apache/kafka/pull/5848
 
 
   Instead of calling `deleteSnapshotsAfterRecoveryPointCheckpoint` for 
`allLogs`, a possible optimization could be invoking it only for the logs being 
truncated.
   
   *More detailed description of your change,
   if necessary. The PR title and PR message become
   the squashed commit message, so use a separate
   comment to ping reviewers.*
   
   *Summary of testing strategy (including rationale)
   for the feature or bug fix. Unit and/or integration
   tests are expected for any behaviour change and
   system tests should be considered for larger changes.*
   
   ### Committer Checklist (excluded from commit message)
   - [ ] Verify design and implementation 
   - [ ] Verify test coverage and CI build status
   - [ ] Verify documentation (including upgrade notes)
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> optimize LogManager.truncateFullyAndStartAt()
> -
>
> Key: KAFKA-7557
> URL: https://issues.apache.org/jira/browse/KAFKA-7557
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 2.0.0, 2.1.0
>Reporter: Jun Rao
>Priority: Major
>
> When a ReplicaFetcherThread calls LogManager.truncateFullyAndStartAt() for a 
> partition, we call LogManager.checkpointLogRecoveryOffsetsInDir() and then 
> Log.deleteSnapshotsAfterRecoveryPointCheckpoint() on all the logs in that 
> directory. This requires listing all the files in each log dir to figure out 
> the snapshot files. If some logs have many log segment files. This could take 
> some time. The can potentially block a replica fetcher thread, which 
> indirectly causes the request handler threads to be blocked.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)