JingsongLi commented on code in PR #1092:
URL: https://github.com/apache/paimon-rust/pull/1092#discussion_r4244759779
##########
crates/paimon/src/table/table_read.rs:
##########
@@ -1302,6 +1302,21 @@ impl<'a> PaimonTableRead<'a> {
if resolve_predicate_views {
output_views.retain(|field| !predicate_views.contains(field));
}
+ // Merge can rebuild larger batches. Yield one authorized row per
lookup
+ // so the downstream LIMIT can stop before the next reference is
resolved.
+ let stream: ArrowRecordBatchStream = if resolve_predicate_views &&
self.limit.is_some() {
+ Box::pin(async_stream::try_stream! {
+ let mut stream = stream;
+ while let Some(batch) = stream.next().await {
+ let batch = batch?;
+ for row in 0..batch.num_rows() {
+ yield batch.slice(row, 1);
Review Comment:
[P2] Keep dependency lookups batched up to the remaining output quota
Every LIMIT now slices authorized candidates into one-row batches. For
distinct references, BlobViewLookup::load_missing calls load once per row; load
re-fetches the upstream table and re-plans/scans it, so the existing descriptor
cache does not preserve grouped reads. Reproduced with real REST HTTP and
actual Parquet/BLOB files: 64 authorized rows, 64 distinct references to one
upstream table, picture IS NOT NULL, projection id, and LIMIT 64 produces 64
GetTable requests and 64 output batches (about 122ms locally). Removing only
this new slicing block produces the identical rows with 1 request and 1 batch
(about 8ms); the unlimited path also uses 1 request. A larger result quota
therefore converts batched remote metadata/data reads into a serial per-row
path. Please cap candidate chunks by the remaining output quota and decrement
that quota after predicate evaluation, allowing grouped lookups while still
never resolving references past the first LIMIT qualifying rows; keep the ne
w rejected-candidate/missing-reference tests and add a request-count
regression.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
