hudi-agent commented on code in PR #19373:
URL: https://github.com/apache/hudi/pull/19373#discussion_r3656656565
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadata.java:
##########
@@ -571,9 +573,10 @@ private ClosableIterator<IndexedRecord>
readSliceWithFilter(Predicate predicate,
.explicitInstants(validInstantTimestamps).build());
// If reuse is enabled and full scan is allowed for the partition, we can
reuse the file readers for base files and the reader context for the log files.
+ boolean shouldReuse = reuse &&
isFullScanAllowedForPartition(fileSlice.getPartitionPath());
+ boolean useLsmReader = !shouldReuse &&
shouldUseLsmReader(metadataMetaClient, fileSlice);
Review Comment:
🤖 When `useLsmReader` is chosen (reuse off), the key filter carried by
`readerContext` doesn't get applied to native log records. Native parquet log
readers return `false` from
`supportKeyPredicate()`/`supportKeyPrefixPredicate()`, so `LsmFileIterators`
reads every record in the log file, whereas the classic path derives a
`KeySpec` from the same filter (`HoodieMergedLogRecordReader.createKeySpec`)
and filters log records by the requested keys/prefixes. Since
`getRecordsByKeyPrefixes`/`readIndexRecordsWithKeys` map every returned record
by its own key with no post-reader key re-filter, the LSM reader can emit
records for keys/prefixes that weren't requested (and only partially merged,
since their base entries weren't seeked). FILES lookups happen to be safe
because consumers look up by the requested key, but
RECORD_INDEX/SECONDARY_INDEX/COLUMN_STATS point/prefix lookups (reuse off)
would be affected. @nsivabalan could you confirm whether the
requested-key/prefix filter needs to
be applied to the LSM log records here?
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag
quality.</i></sub>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
