HyukjinKwon commented on PR #59931: URL: https://github.com/apache/spark/pull/59931#issuecomment-6627079281
870e3fa makes the read schema of `ColumnarScan` independent of the call order: it was the schema of the batch, which created the batch planner, until `toMicroBatchStream` set a flag, which @dongjoon-hyun pointed out in #59943 (https://github.com/apache/spark/pull/59943#discussion_r4463353297), where the same code is. The read schema is now the required columns, and only `toBatch` creates a planner, whose batch drops the other columns from the batches of the reader if the planner does not prune them. `ColumnarDataSourceSuite` checks the read schema of a streaming scan before its stream is created, and a batch read whose planner does not prune; `NativeDataSourceSuite` passes locally. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
