HyukjinKwon commented on PR #59943: URL: https://github.com/apache/spark/pull/59943#issuecomment-6676959726
Thank you for the tenth round, @dongjoon-hyun. 39c2140 addresses it: - **Static CTE:** the startup keeps the `CTERelationDef`, with a placeholder of its plan, so `InlineCTE` no longer fails and silently turns the pruning off. New test. - **Subqueries and normalization:** every subtree of a subquery plan without an outer reference is now a placeholder. The plan is normalized with `planNormalizationRules` before it is optimized, as a micro-batch's is. New test covering `(SELECT true)` and a correlated `EXISTS`. - **`StartupValue.deterministic`** now also requires the wrapped expression to be deterministic. - **Shared extractor:** `visit` and `DataSourceV2Strategy` now use the same `ScanOfStream`. - **Error and conf:** `STREAMING_SCAN_INVALID_READ_SCHEMA` quotes the source as one name part and names the conf. With the conf off, the read schema of a full scan no longer fails the query, so the conf is a complete escape hatch. Tested. - **`newInstance`** nit done. - On the design question (analyzed-plan references) and continuous `start()`: I replied in the threads with why I kept the current approach. Locally, `StreamingV2ScanPushDownSuite`, `StreamingV2ScanPushDownRealTimeSuite` and `StreamingDataSourceV2Suite` pass (94 tests), and the new tests fail without the changes. I updated the PR description. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
