longvu-db opened a new pull request, #55571: URL: https://github.com/apache/spark/pull/55571
### What changes were proposed in this pull request? Add 11 new tests in `DataSourceV2TempViewConnectSuite` that verify temporary view behavior with stored plans when the underlying DSv2 table changes, using Spark Connect. The tests cover all 7 scenarios from the DSv2 table refresh design doc: 1. **Session and external writes** (scenarios 1.1, 1.2): Temp view with filter reflects new data after session or external writes. 2. **Adding columns with data** (scenarios 2.1, 2.2): Temp view preserves original schema after ADD COLUMN, both session and external. 3. **Removing columns** (scenario 3.2): Temp view detects external column removal via catalog API. 4. **Drop and recreate table** (scenario 4.2): Temp view resolves to externally recreated table. 5. **Drop and re-add column with same type** (scenarios 5.1, 5.2): Schema validation passes, view continues working. 6. **Drop and re-add column with different type** (scenarios 6.1, 6.2): Temp view detects type change. 7. **Type widening** (scenario 7): Temp view detects INT to BIGINT type change. Each test creates a DSv2 table via a Connect session, inserts initial data, builds a temp view with a filter (`salary < 999`) to demonstrate stored plan behavior, and verifies the view after table modifications. External writes use the direct catalog API (`loadTable` + `withData`) to simulate writes from outside the session. This is the Spark Connect counterpart of #55540. ### Why are the changes needed? The existing SPARK-53924 tests in `DataSourceV2DataFrameSuite` cover basic schema change detection for temp views on DSv2 tables but are missing: - Filter-based stored plan tests (the design doc uses a filter to demonstrate non-trivial stored plans) - External write variants (using catalog API to simulate writes from outside the session) - Scenarios 5, 6, 7 (drop+re-add column same/different type, type widening) These tests provide comprehensive coverage for the DSv2 table refresh design doc's Section 1 (Temp views with stored plans), exercised through Spark Connect. ### Does this PR introduce _any_ user-facing change? No. This PR only adds tests. ### How was this patch tested? All 11 new tests pass: ``` build/sbt 'connect/testOnly *DataSourceV2TempViewConnectSuite' ``` ### Was this patch authored or co-authored using generative AI tooling? Yes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
