singhpk234 commented on code in PR #10943:
URL: https://github.com/apache/iceberg/pull/10943#discussion_r1720266361
##########
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkScanBuilder.java:
##########
@@ -407,14 +422,35 @@ public Scan build() {
private Scan buildBatchScan() {
Schema expectedSchema = schemaWithMetadataColumns();
+ org.apache.iceberg.Scan scan =
+ buildIcebergBatchScan(false /* not include Column Stats */,
expectedSchema);
+ if (pushedLimit > 0 && hasDeletes(scan)) {
+ LOG.info("Skipping limit pushdown: detected row level deletes");
+ pushedLimit = -1;
+ }
Review Comment:
Was wondering if the filter is partition filter or no filter at all and we
know the amount of files we are gonna scan and have enough confidence that we
will get those number of records can we set this to false then, hinting spark
that global limit is not required, though it will be a very niche use case and
maybe not worth it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
