spacemonkd commented on code in PR #10795:
URL: https://github.com/apache/ozone/pull/10795#discussion_r3955388350
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/service/QuotaRepairTask.java:
##########
@@ -86,12 +91,19 @@ public class QuotaRepairTask {
QuotaRepairTask.class);
@VisibleForTesting
static final int BATCH_SIZE = 5000;
+ /**
+ * A legacy schema multipart upload holds every committed part inline, so
its rows are far
+ * larger than a key or file row: measured at roughly 320 bytes per part, a
1000 part upload
+ * decodes to about 320KB. Batch fewer of them to keep the in-flight batches
bounded.
+ */
+ @VisibleForTesting
+ static final int MPU_BATCH_SIZE = 100;
Review Comment:
On the legacy MPU row for a 1000 part upload it goes to 320KB. But S3 specs
allow upto 10,000 parts, which means in the worst case it will be upto 3.2MB.
Taking a sample worst case scenario (which is quite possible in production)
of 100 such rows in a batch it goes to ~320MB.
Since TASK_THREAD_CNT is 3 + upto 3 worker threads + 1 build we might be
consuming a lot of heap memory just for MPU repair.
I feel like it might be better to just process the legacy rows individually
instead of batching them, because adding the parts inline is easier on the CPU.
We can use batching only when schemaVersion = 1 i.e. new MPU format.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
