plusplusjiajia commented on code in PR #1092:
URL: https://github.com/apache/paimon-rust/pull/1092#discussion_r4245522199
##########
crates/paimon/src/table/table_read.rs:
##########
@@ -1302,6 +1302,21 @@ impl<'a> PaimonTableRead<'a> {
if resolve_predicate_views {
output_views.retain(|field| !predicate_views.contains(field));
}
+ // Merge can rebuild larger batches. Yield one authorized row per
lookup
+ // so the downstream LIMIT can stop before the next reference is
resolved.
+ let stream: ArrowRecordBatchStream = if resolve_predicate_views &&
self.limit.is_some() {
+ Box::pin(async_stream::try_stream! {
+ let mut stream = stream;
+ while let Some(batch) = stream.next().await {
+ let batch = batch?;
+ for row in 0..batch.num_rows() {
+ yield batch.slice(row, 1);
Review Comment:
Thanks, updated: chunks are now capped by the remaining LIMIT quota, which
only rows that pass the predicates consume, so lookups stay grouped without
resolving past LIMIT. Kept the existing tests and added a request-count
regression.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
