HyukjinKwon commented on PR #59943: URL: https://github.com/apache/spark/pull/59943#issuecomment-6679427488
Thank you for the eleventh round, @dongjoon-hyun. c92a4d8 addresses it: - **Main concern:** `spark.sql.streaming.v2ColumnPruning.enabled` is now `false` by default, so for the first release the pruning is opt-in: a table must declare `STREAMING_COLUMN_PRUNING` *and* the query must enable the conf. Falling back to a full scan when a column is missing is not possible once the stream exists, as I explained in the thread, which also goes through the sources of divergence you listed. - **Heuristics:** every value the optimizer computes from the clock or the session has `CURRENT_LIKE`, including `current_timezone()` in your example. A new parameterized test puts each of them in a `CASE WHEN`. In the thread I explained why I kept the optimizer rather than a pass without folding. - **Startup cost:** measured. It is about one optimizer run on the same plan, and less than a micro-batch's query planning. It runs once per query start, not per continuous reconfiguration. The conf doc now mentions it. - **`ColumnarTable`:** the native readers check the schema of each partition's data against the pruned read schema. Details are in the thread, and there is a note next to the capability. - **Error names:** both errors name the source with the same `sourceName`. - **`logChanges`:** `setSessionConf` is back to master's version. The startup uses `setConfWithoutLogging`, and a test checks that it sets the same values as `setSessionConf`. `StreamingV2ScanPushDownSuite`, `StreamingV2ScanPushDownRealTimeSuite`, `StreamingDataSourceV2Suite`, `OffsetSeqLogSuite` and `ColumnarDataSourceSuite` pass locally (171 tests), and the new tests fail without the changes. I updated the PR description. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
