[
https://issues.apache.org/jira/browse/KUDU-1817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Todd Lipcon updated KUDU-1817:
------------------------------
Fix Version/s: (was: 1.3.0)
1.2.0
> Failed to execute spark-shell with kudu-spark2 package
> ------------------------------------------------------
>
> Key: KUDU-1817
> URL: https://issues.apache.org/jira/browse/KUDU-1817
> Project: Kudu
> Issue Type: Bug
> Components: client, spark
> Affects Versions: 1.1.0
> Reporter: Jun He
> Assignee: Jun He
> Priority: Blocker
> Fix For: 1.2.0
>
>
> Tried to run
> {code}spark-shell --packages org.apache.kudu:kudu-spark2_2.11:1.1.0{code} and
> failed with the below error message:
> {quote}
> :: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS
> Exception in thread "main" java.lang.RuntimeException: [unresolved
> dependency: org.apache.kudu#kudu-spark2_2.11;1.1.0: java.text.ParseException:
> inconsistent module descriptor file found in
> 'https://repo1.maven.org/maven2/org/apache/kudu/kudu-spark2_2.11/1.1.0/kudu-spark2_2.11-1.1.0.pom':
> bad module name: expected='kudu-spark2_2.11' found='kudu-spark_2.10'; ]
> at
> org.apache.spark.deploy.SparkSubmitUtils$.resolveMavenCoordinates(SparkSubmit.scala:1076)
> at
> org.apache.spark.deploy.SparkSubmit$.prepareSubmitEnvironment(SparkSubmit.scala:294)
> at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:158)
> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:124)
> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
> {quote}
> I checked
> http://repo1.maven.org/maven2/org/apache/kudu/kudu-spark2_2.11/1.1.0/kudu-spark2_2.11-1.1.0.pom.
> The artifactId in pom.xml is
> {code}kudu-${spark.version.label}_${scala.binary.version}{code}
> As the properties are
> {code}
> <properties>
> <scala.version>2.10.4</scala.version>
> <compat.src>src/main/spark1</compat.src>
> <scala.binary.version>2.10</scala.binary.version>
> <spark.version.label>spark</spark.version.label>
> <top.dir>${project.basedir}/..</top.dir>
> <spark.version>1.6.1</spark.version>
> </properties>
> {code}
> So it will be translated to `kafka-spark_2.10`.
> By checking the code, I think this is caused by how maven shade plugin
> generates pom file.
> During running {code}mvn clean package -P spark2_2.11{code}
> maven shade plugin will generate a dependency-reduced-pom.xml, which will be
> used later in the release pom file.
> In dependency-reduced-pom.xml, the shade plugin will only explicitly parse
> properties for all dependencies. However, it won't parse artifact or plugin
> configurations. So we see
> {code}kudu-${spark.version.label}_${scala.binary.version}{code} in the
> release.
> It will cause the problem when other applications try to load package because
> the properties values are for spark1 and profile values for spark2 won't be
> used when loading the package.
> As pom files are supposed to be static in maven, a quick fix will be creating
> two new modules (spark1 and spark2) to build them separately.
> Please let me know your comments. Thanks.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
