[SPARK-43225][BUILD][SQL] Remove jackson-core-asl and jackson-mapper-asl from pre-built distribution <https://github.com/apache/spark/pull/40893>
tor. 2. nov. 2023 kl. 09:15 skrev Bjørn Jørgensen <[email protected] >: > In spark 3.5.0 removed jackson-core-asl and jackson-mapper-asl those > are with groupid org.codehaus.jackson. > > Those others jackson-* are with groupid com.fasterxml.jackson.core > > > tor. 2. nov. 2023 kl. 01:43 skrev [email protected] <[email protected]>: > >> Hi， >> Please check the versions of jar files starting with "jackson-". Make >> sure all versions are consistent. >> jackson jar list in spark-3.3.0: >> ================ >> 2022/06/10 04:37 75,714 jackson-annotations-*2.13.3*.jar >> 2022/06/10 04:37 374,895 jackson-core-*2.13.3*.jar >> 2022/06/10 04:37 232,248 jackson-core-asl-1.9.13.jar >> 2022/06/10 04:37 1,536,542 jackson-databind-*2.13.3*.jar >> 2022/06/10 04:37 52,020 jackson-dataformat-yaml-*2.13.3*.jar >> 2022/06/10 04:37 121,201 jackson-datatype-jsr310-*2.13.3*.jar >> 2022/06/10 04:37 780,664 jackson-mapper-asl-1.9.13.jar >> 2022/06/10 04:37 458,981 jackson-module-scala_2.12-*2.13.3* >> .jar >> ================ >> >> Spark 3.3.0 uses Jackson version 2.13.3, while Spark 3.5.0 uses Jackson >> version 2.15.2. >> I think you can remove the lower version of Jackson package to keep the >> versions consistent. >> eabour >> >> >> *From:* [email protected] >> *Date:* 2023-11-01 15:03 >> *To:* [email protected] >> *CC:* 'Saar Barhoom' <[email protected]> >> *Subject:* jackson-databind version mismatch >> >> Hi Spark team, >> >> >> >> On upgrading spark version from 3.2.1 to 3.4.1 got the following issue: >> >> *java.lang.NoSuchMethodError: 'com.fasterxml.jackson.core.JsonGenerator >> com.fasterxml.jackson.databind.ObjectMapper.createGenerator(java.io.OutputStream, >> com.fasterxml.jackson.core.JsonEncoding)'* >> >> * at >> org.apache.spark.util.JsonProtocol$.toJsonString(JsonProtocol.scala:75)* >> >> * at >> org.apache.spark.SparkThrowableHelper$.getMessage(SparkThrowableHelper.scala:74)* >> >> * at >> org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$7(SQLExecution.scala:127)* >> >> * at scala.Option.map(Option.scala:230)* >> >> * at >> org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$6(SQLExecution.scala:125)* >> >> * at >> org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:195)* >> >> * at >> org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:103)* >> >> * at >> org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:827)* >> >> * at >> org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:65)* >> >> * at >> org.apache.spark.sql.Dataset.withAction(Dataset.scala:4165)* >> >> * at org.apache.spark.sql.Dataset.head(Dataset.scala:3161)* >> >> * at org.apache.spark.sql.Dataset.take(Dataset.scala:3382)* >> >> * at >> org.apache.spark.sql.Dataset.takeAsList(Dataset.scala:3405)* >> >> * at >> com.crossix.safemine.cloud.utils.DebugRDDLogger.showDataset(DebugRDDLogger.java:84)* >> >> * at >> com.crossix.safemine.cloud.components.statistics.spark.StatisticsTransformer.getFillRateCountsWithSparkQuery(StatisticsTransformer.java:122)* >> >> * at >> com.crossix.safemine.cloud.components.statistics.spark.StatisticsTransformer.calculateStatistics(StatisticsTransformer.java:61)* >> >> * at >> com.crossix.safemine.cloud.components.statistics.spark.SparkFileStatistics.execute(SparkFileStatistics.java:102)* >> >> * at >> com.crossix.safemine.cloud.StatisticsFlow.calculateAllStatistics(StatisticsFlow.java:146)* >> >> * at >> com.crossix.safemine.cloud.StatisticsFlow.runStatistics(StatisticsFlow.java:119)* >> >> * at >> com.crossix.safemine.cloud.StatisticsFlow.initialFileStatistics(StatisticsFlow.java:77)* >> >> * at >> com.crossix.safemine.cloud.SMCFlow.process(SMCFlow.java:221)* >> >> * at >> com.crossix.safemine.cloud.SMCFlow.execute(SMCFlow.java:132)* >> >> * at >> com.crossix.safemine.cloud.SMCFlow.run(SMCFlow.java:91)* >> >> >> >> I see that that spark package contains the dependency: >> >> com.fasterxml.jackson.core:jackson-databind:jar:2.10.5:compile >> >> >> >> But jackson-databind 2.10.5 does not contain >> *ObjectMapper.createGenerator(java.io.OutputStream, >> com.fasterxml.jackson.core.JsonEncoding)* >> >> It was added on 2.11.0 >> >> >> >> Trying to upgrade jackson-databind fails with: >> >> *com.fasterxml.jackson.databind.JsonMappingException: Scala module 2.10.5 >> requires Jackson Databind version >= 2.10.0 and < 2.11.0* >> >> >> >> According to spark 3.3.0 release notes: "Upgrade Jackson to 2.13.3" but >> spark package of 3.4.1 contains Jackson of 2.10.5 >> >> (https://spark.apache.org/releases/spark-release-3-3-0.html) >> >> What am I missing? >> >> >> >> -- >> >> *Moshik Vitas* >> >> Senior Software Developer, Crossix >> >> Veeva Systems >> >> m: +972-54-5326-400 >> >> [email protected] >> >> >> >> > > -- > Bjørn Jørgensen > Vestre Aspehaug 4, 6010 Ålesund > Norge > > +47 480 94 297 > -- Bjørn Jørgensen Vestre Aspehaug 4, 6010 Ålesund Norge +47 480 94 297
