[
https://issues.apache.org/jira/browse/SPARK-32954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yang Jie updated SPARK-32954:
-----------------------------
Description:
When I tried to verify that the resource-managers yarn module passed all UTs in
Scala 2.13 , I found that there are some problems blocking the test:
# resource-managers/yarn module is not explicit dependency on sql/hive module
but some case need HiveConf like `run Python application in yarn-client mode`
in YarnClusterSuite, so we can't test resource-managers/yarn module
independently as follow:
## mvn clean install -DskipTests -pl resource-managers/yarn -Pyarn
-Pscala-2.13 -am
## mvn clean test -pl resource-managers/yarn -Pyarn -Pscala-2.13
# There are more than one `servlet-api` dependency in spark now, one is
`javax.servlet:javax.servlet-api:3.10:compile` we config in core/pom.xml, the
other is `jakarta.servlet:jakarta.servlet-api:4.0.3:test` cascaded by
`org.glassfish.jersey.test-framework.providers`, we can use `mvn
dependency:tree` to check it .So when we execute resource-managers yarn module
test independently, if the position of
`javax.servlet:javax.servlet-api:3.10:compile` in the in classpath is before
`jakarta.servlet:jakarta.servlet-api:4.0.3:test`, there are some cases failed
in YarnClusterSuite, YarnShuffleIntegrationSuite and YarnShuffleAuthSuite
was:
When I tried to verify that the resource-managers yarn module passed all UTs in
Scala 2.13 , I found that there are some problems blocking the test:
# resource-managers/yarn module is not explicit dependency on sql/hive module
but some case need HiveConf like `run Python application in yarn-client mode`
in YarnClusterSuite, so we can't test resource-managers/yarn module
independently as follow:
## mvn clean install -DskipTests -pl resource-managers/yarn -Pyarn
-Pscala-2.13 -am
## mvn clean test -pl resource-managers/yarn -Pyarn -Pscala-2.13
# There are more than one `servlet-api` dependency in spark now, one is
`javax.servlet:javax.servlet-api:3.10:compile` we config in core/pom.xml, the
other is `jakarta.servlet:jakarta.servlet-api:4.0.3:test` cascaded by
`org.glassfish.jersey.test-framework.providers`, we can use `mvn
dependency:tree` to check it .So when we execute resource-managers yarn module
test independently, if the position of
`javax.servlet:javax.servlet-api:3.10:compile` in the in classpath is before
`jakarta.servlet:jakarta.servlet-api:4.0.3:test`, there are some cases failed
in YarnClusterSuite, YarnShuffleIntegrationSuite and YarnShuffleAuthSuite
> Let resource-managers yarn module can be tested independently
> -------------------------------------------------------------
>
> Key: SPARK-32954
> URL: https://issues.apache.org/jira/browse/SPARK-32954
> Project: Spark
> Issue Type: Improvement
> Components: YARN
> Affects Versions: 3.1.0
> Reporter: Yang Jie
> Priority: Minor
>
> When I tried to verify that the resource-managers yarn module passed all UTs
> in Scala 2.13 , I found that there are some problems blocking the test:
> # resource-managers/yarn module is not explicit dependency on sql/hive
> module but some case need HiveConf like `run Python application in
> yarn-client mode` in YarnClusterSuite, so we can't test
> resource-managers/yarn module independently as follow:
> ## mvn clean install -DskipTests -pl resource-managers/yarn -Pyarn
> -Pscala-2.13 -am
> ## mvn clean test -pl resource-managers/yarn -Pyarn -Pscala-2.13
> # There are more than one `servlet-api` dependency in spark now, one is
> `javax.servlet:javax.servlet-api:3.10:compile` we config in core/pom.xml, the
> other is `jakarta.servlet:jakarta.servlet-api:4.0.3:test` cascaded by
> `org.glassfish.jersey.test-framework.providers`, we can use `mvn
> dependency:tree` to check it .So when we execute resource-managers yarn
> module test independently, if the position of
> `javax.servlet:javax.servlet-api:3.10:compile` in the in classpath is before
> `jakarta.servlet:jakarta.servlet-api:4.0.3:test`, there are some cases failed
> in YarnClusterSuite, YarnShuffleIntegrationSuite and YarnShuffleAuthSuite
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
