bernardodemarco opened a new pull request, #13776:
URL: https://github.com/apache/cloudstack/pull/13776
### Description
Each Management Server node runs a background task every hour to fetch
bucket usage information for each Object Storage provider registered in the ACS
environment. To retrieve the statistics for Ceph RGW, the workflow triggers the
`listBucketInfo()` S3 client method, which is equivalent to the `radosgw-admin
bucket stats --bucket=<bucket-name>` command.
The size of each bucket is obtained from the `size_kb` field contained in
the `usage` JSON object. However, the workflow does not handle the case where
the `usage` object is empty, which primarily occurs for empty S3 buckets. The
response schema for an empty bucket is shown below:
<details>
<summary><code>radosgw-admin bucket stats</code> response schema for an
empty bucket</summary>
```json
{
"bucket": "empty-bucket-outside-acs",
"tenant": "",
"versioning": "off",
"zonegroup": "13e2e428-c665-4fba-9a40-79c966da029c",
"placement_rule": "default-placement",
"explicit_placement": {
"data_pool": "",
"data_extra_pool": "",
"index_pool": ""
},
"id": "d25c72a0-a1f0-4033-a954-53bdff513512.54133.5",
"marker": "d25c72a0-a1f0-4033-a954-53bdff513512.54133.5",
"index_type": "Normal",
"index_generation": 0,
"num_shards": 11,
"object_lock_enabled": false,
"mfa_enabled": false,
"owner": "admin-s3-user",
"ver": "0#1,1#1,2#1,3#1,4#1,5#1,6#1,7#1,8#1,9#1,10#1",
"master_ver": "0#0,1#0,2#0,3#0,4#0,5#0,6#0,7#0,8#0,9#0,10#0",
"mtime": "2026-08-03T13:43:38.160905Z",
"creation_time": "2026-08-03T13:43:38.152185Z",
"max_marker": "0#,1#,2#,3#,4#,5#,6#,7#,8#,9#,10#",
"usage": {},
"bucket_quota": {
"enabled": false,
"check_on_raw": false,
"max_size": -1,
"max_size_kb": 0,
"max_objects": -1
}
}
```
</details>
And, the response schema for a non-empty bucket is shown below:
<details>
<summary><code>radosgw-admin bucket stats</code> response schema for a
non-empty bucket</summary>
```json
{
"bucket": "screenshots",
"tenant": "",
"versioning": "off",
"zonegroup": "13e2e428-c665-4fba-9a40-79c966da029c",
"placement_rule": "default-placement",
"explicit_placement": {
"data_pool": "",
"data_extra_pool": "",
"index_pool": ""
},
"id": "d25c72a0-a1f0-4033-a954-53bdff513512.54133.4",
"marker": "d25c72a0-a1f0-4033-a954-53bdff513512.54133.4",
"index_type": "Normal",
"index_generation": 0,
"num_shards": 11,
"object_lock_enabled": false,
"mfa_enabled": false,
"owner": "3af0d458-ecbd-11f0-8ad4-0e2ebc784cf3",
"ver": "0#3,1#3,2#1,3#4,4#1,5#2,6#4,7#3,8#9,9#3,10#2",
"master_ver": "0#0,1#0,2#0,3#0,4#0,5#0,6#0,7#0,8#0,9#0,10#0",
"mtime": "2026-08-03T13:41:32.888630Z",
"creation_time": "2026-08-03T13:41:21.864423Z",
"max_marker": "0#,1#,2#,3#,4#,5#,6#,7#,8#,9#,10#",
"usage": {
"rgw.main": {
"size": 35420402,
"size_actual": 35459072,
"size_utilized": 35420402,
"size_kb": 34591,
"size_kb_actual": 34628,
"size_kb_utilized": 34591,
"num_objects": 24
}
},
"bucket_quota": {
"enabled": true,
"check_on_raw": false,
"max_size": 10737418240,
"max_size_kb": 10485760,
"max_objects": -1
}
}
```
</details>
As a consequence, if there are any empty buckets in the Ceph RGW provider,
regardless of whether they are managed by ACS or not, the following NPE is
thrown and the entire bucket usage retrieval workflow is aborted. As a result,
the size of all buckets is reported as zero.
<details>
<summary>NPE that causes the whole buckets usage retrieval workflow to be
aborted</summary>
```
2026-08-03 13:09:57,577 ERROR [o.a.c.s.o.B.BucketUsageTask]
(Bucket-Usage-1:[ctx-72301d37]) (logid:0e547d8a) Failed to get bucket usage for
Object Store "rgw". Skipping this store.
com.cloud.utils.exception.CloudRuntimeException: Cannot invoke
"org.twonote.rgwadmin4j.model.BucketInfo$Usage$RgwMain.getSize_kb()" because
the return value of
"org.twonote.rgwadmin4j.model.BucketInfo$Usage.getRgwMain()" is null
at
org.apache.cloudstack.storage.datastore.driver.CephObjectStoreDriverImpl.getAllBucketsUsage(CephObjectStoreDriverImpl.java:313)
at
org.apache.cloudstack.storage.object.store.ObjectStoreImpl.getAllBucketsUsage(ObjectStoreImpl.java:155)
at
org.apache.cloudstack.storage.object.BucketApiServiceImpl$BucketUsageTask.runInContext(BucketApiServiceImpl.java:325)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at
java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
```
</details>
Therefore, this PR fixes this issue by skipping the retrieval of bucket size
information when the `usage` JSON object is empty.
---
Fixes #13224
### Types of changes
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] New feature (non-breaking change which adds functionality)
- [X] Bug fix (non-breaking change which fixes an issue)
- [ ] Enhancement (improves an existing feature and functionality)
- [ ] Cleanup (Code refactoring and cleanup, that may add test cases)
- [ ] Build/CI
- [ ] Test (unit or integration test code)
### Feature/Enhancement Scale or Bug Severity
#### Bug Severity
- [ ] BLOCKER
- [ ] Critical
- [ ] Major
- [X] Minor
- [ ] Trivial
### Screenshots (if appropriate):
### How Has This Been Tested?
- Registered a Ceph RGW provider;
- Created two buckets through ACS;
- Uploaded some objects to these buckets;
- Created empty buckets not managed by ACS;
- Created an empty bucket through ACS;
- Verified the logs of the bucket usage retrieval workflow;
<details>
<summary>Logs</summary>
```
root@acs:~# grep 'logid:231b7399'
/var/log/cloudstack/management/management-server.log
2026-08-03 14:39:13,690 DEBUG [o.a.c.s.o.B.BucketUsageTask]
(Bucket-Usage-1:[ctx-e0eeb37f]) (logid:231b7399) Getting bucket usage for
Object Store "rgw"
2026-08-03 14:39:13,775 DEBUG [o.a.c.s.d.d.CephObjectStoreDriverImpl]
(Bucket-Usage-1:[ctx-e0eeb37f]) (logid:231b7399) Fetching statistics for all
buckets accessible by the RGW administrator of the provider with ID [1].
2026-08-03 14:39:13,828 DEBUG [o.a.c.s.d.d.CephObjectStoreDriverImpl]
(Bucket-Usage-1:[ctx-e0eeb37f]) (logid:231b7399) Skipping bucket [test] because
RGW usage information is unavailable.
2026-08-03 14:39:13,828 DEBUG [o.a.c.s.d.d.CephObjectStoreDriverImpl]
(Bucket-Usage-1:[ctx-e0eeb37f]) (logid:231b7399) Skipping bucket
[empty-bucket-managed-by-acs] because RGW usage information is unavailable.
2026-08-03 14:39:13,828 DEBUG [o.a.c.s.d.d.CephObjectStoreDriverImpl]
(Bucket-Usage-1:[ctx-e0eeb37f]) (logid:231b7399) Skipping bucket
[empty-bucket-outside-acs] because RGW usage information is unavailable.
2026-08-03 14:39:13,837 DEBUG [o.a.c.s.o.B.BucketUsageTask]
(Bucket-Usage-1:[ctx-e0eeb37f]) (logid:231b7399) Completed updating bucket
usage for all object stores
```
</details>
- Verified the return of the `listBuckets` API:
<details>
<summary><code>listBuckets</code> API execution</summary>
```
(admin) 🐱 > list buckets filter=size,name,
{
"bucket": [
{
"name": "rgw-admin-bucket",
"size": 24318
},
{
"name": "screenshots",
"size": 34591
},
{
"name": "empty-bucket-managed-by-acs",
"size": 0
}
],
"count": 3
}
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
