Hyukjin Kwon created SPARK-60734:
------------------------------------
Summary: Prune the columns that the scans of Data Source V2
streaming sources read
Key: SPARK-60734
URL: https://issues.apache.org/jira/browse/SPARK-60734
Project: Spark
Issue Type: Improvement
Components: Structured Streaming
Affects Versions: 5.0.0
Reporter: Hyukjin Kwon
The micro-batch and continuous executions push all the columns of the output of
a Data Source V2 streaming source into its scan builder, so a streaming scan
reads every column even when the query needs only some of them, unlike a batch
scan of the same source. This makes them push only the top-level columns that
the streaming query needs into the scan builders that support column pruning
(SupportsPushDownRequiredColumns), behind the internal configuration
spark.sql.streaming.v2ColumnPruning.enabled (true by default).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
