jojochuang commented on PR #11068:
URL: https://github.com/apache/ozone/pull/11068#issuecomment-5431120657
Two performance notes on the CodecBuffer `multiGetFromTable` path:
### 1. Upfront allocation: `n × initial` direct memory
The first pass allocates a direct value buffer for **every** key in the
batch before calling RocksDB:
```java
final int initialCapacity = bufferCapacity.get(); // default 4 KB
for (KEY key : keys) {
...
final CodecBuffer valueBuffer =
CodecBuffer.allocateDirect(initialCapacity);
...
}
```
For a batch of size `n`, that is roughly **`n × initialCapacity`** of direct
memory reserved up front (plus `n` direct key buffers), even when most values
are small or missing. With large batches this can create noticeable
direct-memory pressure and GC churn when buffers are closed in the finally
blocks.
The byte[] path avoids this — RocksDB allocates only the actual value sizes.
Worth calling out in the PR description if callers may pass large key lists
(e.g. OM/SCM metadata scans). A possible follow-up would be lazy/per-key
allocation or capping batch size at the call site.
### 2. `bufferCapacity` hint not updated after oversize retry
Single-key reads adapt the shared hint when a value exceeds the current
capacity:
```java
// getFromTable() — after resize/retry loop
bufferCapacity.increase(required);
```
`multiGetFromTable` reads `bufferCapacity.get()` for the initial pass but
**never calls `bufferCapacity.increase()`** when keys are retried with
`requiredSize`. On a table whose values are consistently larger than the
default (4 KB), every `multiGetSkipCache` call will pay the two-pass penalty
(first pass discovers oversize, second pass re-allocates per oversize key),
while single `get()` calls on the same table would have warmed the hint.
Consider updating the hint after the retry pass, e.g.
`bufferCapacity.increase(Collections.max(retrySizes))`, so subsequent multiGets
on that table start with a better initial capacity. Low priority for
correctness, but it would align multi-get behavior with the existing single-get
adaptive sizing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
