zhengruifeng commented on PR #57461: URL: https://github.com/apache/spark/pull/57461#issuecomment-5099328339
Thanks for the thoughtful feedback. This is an initial, non-final version; we can iterate on the prompt, validation rules, and evaluation data to improve accuracy. The workflow is intended as a non-blocking, best-effort post-merge signal: it selects a small set of likely-relevant tests for faster feedback. Failures remain visible and should be investigated, but periodic build_main provides the broader deterministic backstop. build_main does not provide complete coverage today - for example, Pandas API on Spark (PS) tests are always skipped. With limited CI capacity, the question is how to make coverage tradeoffs more informed. Removing old or slow tests is valuable but difficult and higher risk, since they may cover historical regressions or non-obvious interactions. Selecting a small relevant set is more incremental: it does not delete tests or claim they have no value, while broader periodic CI can still run them. We should measure the selector cost and recall before relying on it more broadly. A rule-based dependency/import-graph selector is also a useful complementary direction, especially for PySpark. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
