szehon-ho commented on PR #58370: URL: https://github.com/apache/spark/pull/58370#issuecomment-5458555223
Thanks for the detailed review. Addressed in `467323b126c`: 1. Fully-pushed nested references are now validated eagerly through `checkRuntimeFilteringInterfaces()`, including queries without scalar-subquery filters. 2. Restored the resolution requirements in the interface docs and documented root-attribute eligibility and possible sibling-field predicates. 3. Runtime references now resolve as `NamedExpression`. Row-level group filtering materializes nested build keys before aggregation and uses the underlying nested expression for pruning. Added an end-to-end nested DELETE regression. 4. Updated the in-memory fixtures to use the configured resolver and added coverage for malformed nested references over non-struct columns. 5. Agreed that this fix should be backported to `branch-4.x`. Validation: - `DataSourceV2CatalystRuntimeFilterSuite`: 20 passed - `GroupBasedRowLevelOperationCatalystRuntimeFilterSuite`: 4 passed - Focused V1/V2 nested DPP regressions: 2 passed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
