cloud-fan commented on issue #19424: [SPARK-22197][SQL] push down operators to
data source before planning
URL: https://github.com/apache/spark/pull/19424#issuecomment-540890347
Usually a data source scans its data incrementally. So when the query has a
limit, Spark stops consuming the iterator from data source, and data source
won't scan all the data.
But limit pushdown does have use cases. For now the only way is to write a
catalyst rule, to catch the limit + scan query plan, and convert it to a custom
query plan with limit pushed down.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org