lpavanvenkat opened a new pull request, #11685:
URL: https://github.com/apache/ozone/pull/11685
## What changes were proposed in this pull request?
Recon reports incomplete S3 multipart uploads (MPUs) inconsistently across
its open key endpoints. When an MPU is
initiated, OM writes a multipart open key with size 0 to the open key/file
table; committed parts are recorded in
`multipartInfoTable` (legacy layout) or `multipartPartsTable` (split
layout), and the multipart open key never grows.
As a result, for one MPU with 10 MiB of committed parts:
| Endpoint | Before | After |
|---|---|---|
| `/keys/open` row | 0 | 10 MiB |
| `/keys/open/summary` | 0 | 10 MiB |
| `/keys/open/mpu/summary` | 10 MiB | 10 MiB (unchanged) |
Changes:
- **`/keys/open` rows**: for a multipart open key, `OMDBInsightEndpoint`
reads the upload ID from the DB key, looks up
the upload in `multipartInfoTable`, and sets `size`/`replicatedSize` to
the total of its committed parts. The new
`MultipartInfoInsightHandler.getCommittedPartsSize` handles both layouts
and reuses the existing
`OMMultipartUploadUtils` helpers, so it matches how the MPU summary is
computed. Other open keys, including part
open keys still being written, are unchanged. The page totals now add up
the row sizes.
- **`/keys/open/summary`**: the new
`ReconGlobalMetricsService.getOpenKeySummaryWithMPU` adds the MPU summary sizes
to
the open key sizes. The count is not changed, since each MPU is already
counted once through its multipart open
key. `/storageDistribution` already adds the two summaries itself, so it
keeps using the open-table-only
`getOpenKeySummary()` and does not count MPU bytes twice.
- **MPU size stats after a Recon restart**: `MultipartInfoInsightHandler`
caches each split-layout MPU's replication
config to compute the replicated size of part events (HDDS-16363). The
cache was filled only on
`multipartInfoTable` PUT/UPDATE, so after a Recon restart, aborting or
completing an MPU that was already open
skipped every part delete (`No cached replication config ... skipping part
size update`), leaving its bytes in the
MPU stats. Since `/keys/open/summary` now includes those stats, the
Overview card showed "0 open keys" with
non-zero sizes. OM deletes the `multipartInfoTable` row before its
`multipartPartsTable` rows in the same batch, so
the handler now also caches the config on the `multipartInfoTable` DELETE
event.
Known limitations, not changed here:
- MPUs initiated before HDDS-9017 do not have the multipart flag on their
open key, so their `/keys/open` row still
shows 0 (their bytes are included in `/keys/open/summary`).
- The path of a nested FSO multipart open key shows its parent directories
twice
(e.g. `dir1/dir2/dir1/dir2/key`). This is an existing
`ReconUtils.constructFullPath` issue and will be fixed
separately.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-16364
## How was this patch tested?
- New unit tests:
- `TestOmDBInsightEndPoint#testOpenKeySummaryIncludesMultipartUploads`:
summary sizes include MPU sizes; count is
unchanged.
- `TestOmDBInsightEndPoint#testGetOpenKeyInfoForMultipartOpenKeys`: row
sizes and page totals for a split-layout OBS
MPU, a legacy-layout FSO MPU with a nested key, and a multipart open key
without a `multipartInfoTable` entry.
-
`TestOmTableInsightTask#testProcessForSplitSchemaMPUDeletedAfterRestart`: a new
task instance (empty cache)
processes an MPU delete followed by its part deletes and the sizes
return to 0. It fails without the fix.
- `mvn -pl :ozone-recon test`: 876 tests, 0 failures. `checkstyle.sh`: 0
violations.
- Manual test on the `ozone` compose cluster (3 datanodes), with FSO and OBS
buckets: MPUs with RATIS/ONE and
RATIS/THREE parts, a nested key, an MPU without parts, a part re-upload,
FSO/non-FSO filters, prefix search,
complete and abort. All endpoints agreed at every step and
`/storageDistribution` was never doubled. The Recon
restart was repeated with MPUs open: a part upload, a complete and aborts
after the restart all updated the sizes,
ending at 0 with no "No cached replication config" warnings.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
