danny0405 opened a new pull request, #19980:
URL: https://github.com/apache/hudi/pull/19980
### Describe the issue this Pull Request addresses
The Flink RocksDB lookup cache formats each serialized key byte with
`String.format()` on insertion and lookup, and writes each loaded row
separately. Lookups also retain intermediate key/value and byte-array
collections before deserializing the final rows. The shared prefix scan logs
every lookup at INFO and measures deserialization time for every entry.
These operations add CPU, allocation, and write overhead when loading or
querying a dimension table.
### Summary and Changelog
- Replace per-byte formatting with `StringUtils.toHexString()`.
- Encapsulate pending serialized rows and byte accounting in `WriteBuffer`,
using `RocksDBDAO.writeBatch()` when the buffer reaches 1,024 rows or 1 MiB of
serialized key/value data.
- Add a default `LookupCache.flush()` hook and flush the final batch before
marking a cache load complete. Reads flush pending writes; clear and close
discard them.
- Add a synchronous callback overload of `RocksDBDAO.prefixSearch()` that
closes its iterator even when the callback fails. Deserialize lookup rows
directly into the result list while preserving the existing stream-returning
overload's materialized results.
- Move prefix-scan diagnostics to DEBUG and retain only total scan timing.
- Cover batch boundaries, reused input rows, large rows, pending-write
clearing, callback failures, and cache loading with both heap and RocksDB
implementations.
No code was copied from another project.
### Impact
Reduces per-row writes and temporary lookup allocations for RocksDB-backed
Flink lookup joins. The shared DAO gains an additive callback API, and the
internal lookup-cache interface gains a default flush method; existing callers
remain compatible. Existing prefix-search callers also avoid INFO logging and
per-entry timing overhead.
No changes to Hudi storage formats, configuration defaults, or lookup
results. Performance improvements have not been benchmarked.
### Risk Level
medium
The change touches a shared RocksDB scan implementation and defers cache
writes until batch flushes. Existing stream semantics and iterator cleanup are
preserved, and pending writes are flushed before reads and successful load
completion.
Validation: all 18 focused tests passed (`TestRocksDBDAO`: 8,
`TestRocksDBLookupCache`: 5, `TestHoodieLookupFunction`: 5), with Checkstyle
and license checks passing in the Maven reactor. `git diff --check` also passed.
```shell
mvn -o -pl hudi-common,hudi-flink-datasource/hudi-flink -am \
-Dtest=TestRocksDBDAO,TestRocksDBLookupCache,TestHoodieLookupFunction \
-Dsurefire.failIfNoSpecifiedTests=false -DfailIfNoTests=false test
```
The full repository test suite was not run.
### Documentation Update
none — this is an internal performance improvement with no new user-facing
feature or configuration. The new callback and flush contracts are documented
in Javadoc.
### Contributor's checklist
- [x] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
