JingsongLi commented on code in PR #10226:
URL: https://github.com/apache/paimon/pull/10226#discussion_r4295311487
##########
paimon-common/src/main/java/org/apache/paimon/data/variant/PaimonShreddingUtils.java:
##########
@@ -720,7 +748,7 @@ private static Object extractField(
return null;
}
GenericVariant v =
- new GenericVariant(binaryBuffer(row, variantIdx),
topLevelMetadata);
+ new GenericVariant(binaryBuffer(row, variantIdx),
metadata.buffer());
Review Comment:
[P2] Avoid decoding the whole dictionary for raw fallback lookups
Projecting a field outside the shredding schema still reaches this
`metadata.buffer()` call. `GenericVariant.getFieldByKey` then uses the existing
raw-byte binary search and does not use the decoded-key cache, but `buffer()`
now forces `ensureDecoded()` to compare the entire metadata buffer on every
row. I reproduced this through actual Parquet writes/reads: physical shredding
schema contains only `$.typed`, the query projects `$.leftover_42` as INT, and
each row has 10,000 leftover keys (158,905 metadata bytes). On the same
5,000-row file, independent JDK 8/G1 JVMs in BASE/HEAD/HEAD/BASE order measured
medians 7.14/30.23/29.02/7.18 ms, checking every result equals 42 and the row
count/checksum. The fully typed projection control no longer regresses. This
adds about 795 MB of unnecessary comparisons to one scalar scan. Please
separate synchronous raw metadata access from dictionary decoding/snapshot
adoption for fallback lookups, while preserving backing-buffer reuse isolation
for
cached dictionary/rebuild consumers, and add this unshredded-field projection
to the performance regression coverage.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
