szehon-ho commented on PR #58503: URL: https://github.com/apache/spark/pull/58503#issuecomment-5530257338
This is the upstream follow-up requested during the review of #58412. The changes map to that feedback as follows: - **Transform safety:** all affected in-memory runtime-filtering fixtures now advertise and bind only identity-transform source attributes, while retaining their real partition-key ordinals. This includes `InMemoryRowLevelOperationTable`, which was called out in the follow-up review. - **Fully-pushed validation:** `fullyPushedFilterAttributes()` must be an exact subset of `filterAttributes()`. This rejects a top-level struct root when only a nested path is filterable and exercises the previously unused invalid-declaration test hook. - **Consistent diagnostics:** DPP, row-level group filtering, and iterative pushdown now use the same structured runtime-filter attribute resolution instead of surfacing raw resolution errors. This is why `RowLevelOperationRuntimeGroupFiltering.scala` is part of the patch. - **Tests and cleanup:** transformed-source regressions cover Catalyst, V1/V2 DPP, and both row-level implementations; expected nested paths are represented once without lossy dot splitting; the V1/V2 evaluator guards remain minimal; and the transformed-partition test name describes fixture behavior. - **API wording:** the runtime-filter interfaces now say Spark *currently* tracks eligibility by root attribute. The focused suites pass with the documented local-network setting (`SPARK_LOCAL_IP=localhost`), along with compilation, error-condition validation, scalastyle, and checkstyle. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
