spacemonkd commented on code in PR #10795:
URL: https://github.com/apache/ozone/pull/10795#discussion_r3955990700
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/service/QuotaRepairTask.java:
##########
@@ -554,6 +570,50 @@ private void recalculateDeletedDirNamespace(
}
}
+ /**
+ * Counts committed parts of incomplete multipart uploads from the active DB
checkpoint.
+ * Parts of a snapshot's incomplete uploads stay charged to that snapshot's
own bucket,
+ * so only the active multipartInfoTable is scanned here.
+ */
+ private void recalculateMultipartUsages(
+ OMMetadataManager metadataManager, Map<String, CountPair> mpuCountMap)
+ throws UncheckedIOException, UncheckedExecutionException {
+ try (Table.KeyValueIterator<String, OmMultipartKeyInfo> keyIter
+ = metadataManager.getMultipartInfoTable().iterator()) {
Review Comment:
Since `multipartInfoTable` is prefix-ordered by `/vol/bucket/` and the parts
table already supports prefix iteration, for a targeted bucket we could iterate
`getMultipartInfoTable().iterator(bucketPrefix)` per requested bucket and never
touch other buckets.
Refer to the below comment for more context
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/service/QuotaRepairTask.java:
##########
@@ -554,6 +570,50 @@ private void recalculateDeletedDirNamespace(
}
}
+ /**
+ * Counts committed parts of incomplete multipart uploads from the active DB
checkpoint.
+ * Parts of a snapshot's incomplete uploads stay charged to that snapshot's
own bucket,
+ * so only the active multipartInfoTable is scanned here.
+ */
+ private void recalculateMultipartUsages(
+ OMMetadataManager metadataManager, Map<String, CountPair> mpuCountMap)
+ throws UncheckedIOException, UncheckedExecutionException {
+ try (Table.KeyValueIterator<String, OmMultipartKeyInfo> keyIter
+ = metadataManager.getMultipartInfoTable().iterator()) {
+ scanTableInBatches(executor, keyIter, "Multipart upload usages",
MPU_BATCH_SIZE,
+ kv -> extractMultipartCount(kv, mpuCountMap, metadataManager));
+ } catch (IOException ex) {
+ throw new UncheckedIOException(ex);
+ }
+ }
+
+ private static void extractMultipartCount(
+ Table.KeyValue<String, OmMultipartKeyInfo> kv, Map<String, CountPair>
mpuCountMap,
+ OMMetadataManager metadataManager) throws UncheckedIOException {
+ try {
+ CountPair usage = mpuCountMap.get(getVolumeBucketPrefix(kv.getKey()));
+ if (usage == null) {
+ return;
Review Comment:
It seems this early return works after the iterator had already decoded the
value. This means for a single bucket or prefix specific repair we are
decoding the legacy rows in unrelated buckets as well as part of
recalculateMultipartUsages().
Check [this](https://github.com/apache/ozone/pull/10795/changes#r3955990700)
comment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
