[GitHub] spark pull request #21588: [WIP][SPARK-24590][BUILD] Make Jenkins tests pass...
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/21588#discussion_r196689287
--- Diff: dev/run-tests.py ---
@@ -305,8 +305,8 @@ def get_hadoop_profiles(hadoop_version):
"""
sbt_maven_hadoop_profiles = {
-"hadoop2.6": ["-Phadoop-2.6"],
-"hadoop2.7": ["-Phadoop-2.7"],
+"hadoop2.6": ["-Phadoop-3.1"],
+"hadoop2.7": ["-Phadoop-3.1"],
--- End diff --
I see.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21588: [WIP][SPARK-24590][BUILD] Make Jenkins tests pass...
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/21588#discussion_r196686546
--- Diff: dev/run-tests.py ---
@@ -305,8 +305,8 @@ def get_hadoop_profiles(hadoop_version):
"""
sbt_maven_hadoop_profiles = {
-"hadoop2.6": ["-Phadoop-2.6"],
-"hadoop2.7": ["-Phadoop-2.7"],
+"hadoop2.6": ["-Phadoop-3.1"],
+"hadoop2.7": ["-Phadoop-3.1"],
--- End diff --
@jerryshao, Here, replaced the profile to point out `-Phadoop-3.1` by force
:-)
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21588: [WIP][SPARK-24590][BUILD] Make Jenkins tests pass...
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/21588#discussion_r196646202 --- Diff: pom.xml --- @@ -123,7 +123,7 @@ 1.6.0 3.4.6 2.6.0 -org.spark-project.hive +com.github.hyukjinkwon --- End diff -- I am going to revert this change and the changes in ` dev/run-tests.py` back to `org.spark-project.hive` once the tests pass. I redistributed `org.spark-project.hive` having one liner fix for Hadoop 3 support in order to use it in this PR. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21588: [WIP][SPARK-24590][BUILD] Make Jenkins tests pass...
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/21588#discussion_r196646070
--- Diff: project/SparkBuild.scala ---
@@ -464,7 +464,20 @@ object DockerIntegrationTests {
*/
object DependencyOverrides {
lazy val settings = Seq(
-dependencyOverrides += "com.google.guava" % "guava" % "14.0.1")
+dependencyOverrides += "com.google.guava" % "guava" % "14.0.1",
+dependencyOverrides += "com.fasterxml.jackson.core" %
"jackson-annotations" % "2.6.7",
+dependencyOverrides += "com.fasterxml.jackson.core" % "jackson-core" %
"2.6.7",
+dependencyOverrides += "com.fasterxml.jackson.core" %
"jackson-module-jaxb-annotations" % "2.6.7",
+dependencyOverrides += "com.fasterxml.jackson.core" %
"jackson-databind" % "2.6.7")
+}
+
+/**
+ * Exclusions to work around sbt's dependency resolution being different
from Maven's.
+ */
+object ExcludeDependencies {
+ lazy val settings = Seq(
+excludeDependencies += "com.fasterxml.jackson.jaxrs" %
"jackson-jaxrs-json-provider",
+excludeDependencies += "javax.ws.rs" % "jsr311-api")
--- End diff --
I think this also should have been excluded by Jersey. Seems it's
difference between Maven and SBT if I am not mistaken.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21588: [WIP][SPARK-24590][BUILD] Make Jenkins tests pass...
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/21588#discussion_r196645986
--- Diff: project/SparkBuild.scala ---
@@ -464,7 +464,20 @@ object DockerIntegrationTests {
*/
object DependencyOverrides {
lazy val settings = Seq(
-dependencyOverrides += "com.google.guava" % "guava" % "14.0.1")
+dependencyOverrides += "com.google.guava" % "guava" % "14.0.1",
+dependencyOverrides += "com.fasterxml.jackson.core" %
"jackson-annotations" % "2.6.7",
+dependencyOverrides += "com.fasterxml.jackson.core" % "jackson-core" %
"2.6.7",
+dependencyOverrides += "com.fasterxml.jackson.core" %
"jackson-module-jaxb-annotations" % "2.6.7",
+dependencyOverrides += "com.fasterxml.jackson.core" %
"jackson-databind" % "2.6.7")
--- End diff --
These look coming from `jackson-jaxrs-json-provider` where it looks the
resolution is different between Maven and SBT. I had to manually override and
exclude.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21588: [WIP][SPARK-24590][BUILD] Make Jenkins tests pass...
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/21588#discussion_r196645845
--- Diff:
sql/hive/src/test/scala/org/apache/spark/sql/hive/client/HiveVersionSuite.scala
---
@@ -26,7 +27,6 @@ import org.apache.spark.sql.hive.HiveUtils
private[client] abstract class HiveVersionSuite(version: String) extends
SparkFunSuite {
override protected val enableAutoThreadAudit = false
- protected var client: HiveClient = null
--- End diff --
This was only used in `HiveClientSuite.scala`.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
