chihsuan commented on code in PR #10795:
URL: https://github.com/apache/ozone/pull/10795#discussion_r3968653443
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/service/QuotaRepairTask.java:
##########
@@ -86,12 +91,19 @@ public class QuotaRepairTask {
QuotaRepairTask.class);
@VisibleForTesting
static final int BATCH_SIZE = 5000;
+ /**
+ * A legacy schema multipart upload holds every committed part inline, so
its rows are far
+ * larger than a key or file row: measured at roughly 320 bytes per part, a
1000 part upload
+ * decodes to about 320KB. Batch fewer of them to keep the in-flight batches
bounded.
+ */
+ @VisibleForTesting
+ static final int MPU_BATCH_SIZE = 100;
Review Comment:
@spacemonkd After rethinking this, I went a slightly different way. I
realized all three points come back to one cause. The producer thread **decoded
the row before anything decided it was needed**.
Therefore, I updated the PR in
https://github.com/apache/ozone/pull/10795/changes/e8adc666319b3a72be6ccf57a302a4e4cc0a9602
so the scan now iterates `KEY_ONLY` and reads each row after the bucket check.
If you would rather keep two routes, we could open the iterator on the bucket
prefix when specific buckets are requested. I think this version is simpler,
but happy to change it.
For reference, I ran a local benchmark with 2000 part legacy rows:
| | before | after |
|----------------|--------------------------|--------------------------|
| full repair | 264-271 ms / +293-315 MB | 170-187 ms / +120-199 MB |
| 1 of 8 buckets | 249-255 ms / +259-328 MB | 53-55 ms / +43-49 MB |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
