kerwin-zk commented on PR #13283: URL: https://github.com/apache/gluten/pull/13283#issuecomment-6361933464
[Blocking] The new manager-level regression test now exercises the correct ownership and shutdown paths, but it is not runnable on the normal `gluten-celeborn` Maven test classpath for two independent reasons. 1. Running `build/mvn -ntp -pl :gluten-celeborn -Pspark-3.5 -Pceleborn test` fails at [`CelebornShuffleManagerTest.java:74`](https://github.com/apache/gluten/blob/05dfbe61d2cc77ab3ed8de9af25667dc62942a18/gluten-celeborn/src/test/java/org/apache/spark/shuffle/gluten/celeborn/CelebornShuffleManagerTest.java#L72-L78) with `NoClassDefFoundError: org/apache/hadoop/fs/FSDataInputStream`. The module test classpath has no Hadoop artifact, so the test fails while initializing `SparkConf`, before reaching any ownership assertion. 2. Supplying the missing Hadoop classes reveals another setup failure when the test constructs `CelebornShuffleManager`: its [`ServiceLoader`-based static initializer](https://github.com/apache/gluten/blob/05dfbe61d2cc77ab3ed8de9af25667dc62942a18/gluten-celeborn/src/main/java/org/apache/spark/shuffle/gluten/celeborn/CelebornShuffleManager.java#L65-L84) throws `IllegalStateException: No Celeborn shuffle writer factory found`. The two new test factory implementations are present, but there are no corresponding `src/test/resources/META-INF/services/...` descriptors, so `ServiceLoader` cannot discover them. Could we add the required Hadoop test dependency and register both test providers, then verify that the normal module test command passes? The production ownership fix and the manager-level test flow themselves look correct. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
