zhengruifeng opened a new pull request, #58892: URL: https://github.com/apache/spark/pull/58892
### What changes were proposed in this pull request? This PR separates source ownership from test ownership in `dev/sparktestsupport/modules.py`. Each runnable test module is paired with an internal source module. Files under JVM `src/test` directories belong to the runnable test module, and PySpark test files are matched to the module that lists them in `python_test_goals`. Source changes continue to propagate through the module dependency graph. Test-only changes are leaf changes and run only their owning test module. ### Why are the changes needed? The previous module graph used the same node for source code and test code. As a result, changing a test was treated like changing the module's source and triggered every dependent module. For example, #58883 changes only a Spark Connect Python test, but it triggered all pandas API on Spark test shards because those modules depend on PySpark SQL and Spark Connect source code. Separating source and test nodes avoids this unnecessary CI fan-out while preserving coverage for source changes. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Added doctests covering Spark Connect Python test ownership and Scala test ownership. Also ran: ``` PYTHONPATH=dev conda run -n spark-dev-313 python -m doctest dev/sparktestsupport/modules.py PYTHONPATH=dev conda run -n spark-dev-313 python -m doctest dev/sparktestsupport/utils.py conda run -n spark-dev-313 dev/lint-python --ruff ``` Representative Scala source, Scala test, Scala test-resource, and PySpark test paths were also checked directly against `determine_modules_for_files` and `determine_modules_to_test`. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: OpenAI Codex (GPT-5) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
