[
https://issues.apache.org/jira/browse/HDDS-16281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HDDS-16281:
----------------------------------
Labels: pull-request-available (was: )
> Reduce per-key overhead in Recon reprocess tasks
> ------------------------------------------------
>
> Key: HDDS-16281
> URL: https://issues.apache.org/jira/browse/HDDS-16281
> Project: Apache Ozone
> Issue Type: Improvement
> Reporter: Huang Kuan Hao
> Assignee: Huang Kuan Hao
> Priority: Major
> Labels: pull-request-available
>
> On the Recon reprocess (full rebuild) loop over the whole keyspace, the OBS
> task does a raw RocksDB point-read per key to fetch the bucket, even though
> consecutive keys share a bucket (the incremental process() path caches via
> lookupBucketCached):
> String bucketDBKey = omMetadataManager.getBucketKey(volumeName, bucketName);
> OmBucketInfo omBucketInfo =
> omMetadataManager.getBucketTable().getSkipCache(bucketDBKey);
> and each worker re-fetches its own accumulation map per key by boxing the
> thread id:
> Map<..> localMap = allLocalMaps.computeIfAbsent(
> Thread.currentThread().getId(), k -> new ConcurrentHashMap<>());
> Fix: (a) add a per-worker bucket cache (reprocess is multi-threaded, so
> per-worker / thread-safe); (b) resolve the worker map once via a ThreadLocal,
> keeping a concurrent registry of worker maps for the final merge
> (allLocalMaps.values() is iterated to flush). Behavior unchanged.
> Files: NSSummaryTaskWithOBS.java:122,315; ContainerKeyMapperHelper.java:149;
> FileSizeCountTaskHelper.java:167; NSSummaryTaskWithFSO.java:282
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
