[
https://issues.apache.org/jira/browse/SPARK-26299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Liu, Linhong updated SPARK-26299:
---------------------------------
Description:
In spark root pom, there is a test-jar execution for all sub-projects including
spark-streaming. This will attach an artifact:
org.apache.spark:spark-streaming_2.11:*{color:#ff0000}test-jar{color}*:tests:2.3.2
Also, in streaming pom, there is a shade test configuration, it will attach an
artifact:
org.apache.spark:spark-streaming_2.11:{color:#ff0000}*jar*{color}:tests:2.3.2
But two artifacts actually point to the same file:
spark-streaming_2.11-2.3.2-tests.jar
So, when deploy spark to nexus using mvn deploy, maven will upload the test.jar
twice since it belongs to 2 artifacts. Then deploy fails due to nexus don't
allow overrides existing file for non-SNAPSHOT release.
What's more, after checking the spark-streaming, shaded test jar is exactly
same to original test jar. It seems we can just delete the related shade config.
```
<build>
<outputDirectory>target/scala-${scala.binary.version}/classes</outputDirectory>
<testOutputDirectory>target/scala-${scala.binary.version}/test-classes</testOutputDirectory>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<configuration>
<shadeTestJar>true</shadeTestJar>
</configuration>
</plugin>
</plugins>
</build>
```
was:
In spark root pom, there is a test-jar execution for all sub-projects including
spark-streaming. This will attach an artifact:
org.apache.spark:spark-streaming_2.11:*{color:#FF0000}test-jar{color}*:tests:2.3.2
Also, in streaming pom, there is a shade test configuration, it will attach an
artifact:
org.apache.spark:spark-streaming_2.11:{color:#FF0000}*jar*{color}:tests:2.3.2
But two artifacts actually point to the same file:
spark-streaming_2.11-2.3.2-tests.jar
So, when deploy spark to nexus using mvn deploy,
> [MAVEN] Shaded test jar in spark-streaming cause deploy test jar twice
> ----------------------------------------------------------------------
>
> Key: SPARK-26299
> URL: https://issues.apache.org/jira/browse/SPARK-26299
> Project: Spark
> Issue Type: Bug
> Components: Deploy
> Affects Versions: 2.3.2, 2.4.0
> Reporter: Liu, Linhong
> Priority: Major
>
> In spark root pom, there is a test-jar execution for all sub-projects
> including spark-streaming. This will attach an artifact:
> org.apache.spark:spark-streaming_2.11:*{color:#ff0000}test-jar{color}*:tests:2.3.2
> Also, in streaming pom, there is a shade test configuration, it will attach
> an artifact:
> org.apache.spark:spark-streaming_2.11:{color:#ff0000}*jar*{color}:tests:2.3.2
> But two artifacts actually point to the same file:
> spark-streaming_2.11-2.3.2-tests.jar
> So, when deploy spark to nexus using mvn deploy, maven will upload the
> test.jar twice since it belongs to 2 artifacts. Then deploy fails due to
> nexus don't allow overrides existing file for non-SNAPSHOT release.
> What's more, after checking the spark-streaming, shaded test jar is exactly
> same to original test jar. It seems we can just delete the related shade
> config.
> ```
> <build>
>
> <outputDirectory>target/scala-${scala.binary.version}/classes</outputDirectory>
>
> <testOutputDirectory>target/scala-${scala.binary.version}/test-classes</testOutputDirectory>
> <plugins>
> <plugin>
> <groupId>org.apache.maven.plugins</groupId>
> <artifactId>maven-shade-plugin</artifactId>
> <configuration>
> <shadeTestJar>true</shadeTestJar>
> </configuration>
> </plugin>
> </plugins>
> </build>
> ```
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
