Thanks for replying,
The issue was import of spring-boot-dependencies on my dependencyManagement pom that forced invalid jar version. Removed this section and got valid spark dependencies. Regards, Moshik Vitas From: Bjørn Jørgensen <bjornjorgen...@gmail.com> Sent: Thursday, 2 November 2023 10:40 To: eab...@163.com Cc: user @spark <user@spark.apache.org>; Saar Barhoom <saar.barh...@veeva.com>; moshik.vi...@veeva.com Subject: Re: jackson-databind version mismatch [SPARK-43225][BUILD][SQL] Remove jackson-core-asl and jackson-mapper-asl from pre-built distribution <https://github.com/apache/spark/pull/40893> tor. 2. nov. 2023 kl. 09:15 skrev Bjørn Jørgensen <bjornjorgen...@gmail.com <mailto:bjornjorgen...@gmail.com> >: In spark 3.5.0 removed jackson-core-asl and jackson-mapper-asl those are with groupid org.codehaus.jackson. Those others jackson-* are with groupid com.fasterxml.jackson.core tor. 2. nov. 2023 kl. 01:43 skrev eab...@163.com <mailto:eab...@163.com> <eab...@163.com <mailto:eab...@163.com> >: Hi， Please check the versions of jar files starting with "jackson-". Make sure all versions are consistent. jackson jar list in spark-3.3.0: ================ 2022/06/10 04:37 75,714 jackson-annotations-2.13.3.jar 2022/06/10 04:37 374,895 jackson-core-2.13.3.jar 2022/06/10 04:37 232,248 jackson-core-asl-1.9.13.jar 2022/06/10 04:37 1,536,542 jackson-databind-2.13.3.jar 2022/06/10 04:37 52,020 jackson-dataformat-yaml-2.13.3.jar 2022/06/10 04:37 121,201 jackson-datatype-jsr310-2.13.3.jar 2022/06/10 04:37 780,664 jackson-mapper-asl-1.9.13.jar 2022/06/10 04:37 458,981 jackson-module-scala_2.12-2.13.3.jar ================ Spark 3.3.0 uses Jackson version 2.13.3, while Spark 3.5.0 uses Jackson version 2.15.2. I think you can remove the lower version of Jackson package to keep the versions consistent. eabour From: <mailto:moshik.vi...@veeva.com.INVALID> moshik.vi...@veeva.com.INVALID Date: 2023-11-01 15:03 To: <mailto:user@spark.apache.org> user@spark.apache.org CC: <mailto:saar.barh...@veeva.com> 'Saar Barhoom' Subject: jackson-databind version mismatch Hi Spark team, On upgrading spark version from 3.2.1 to 3.4.1 got the following issue: java.lang.NoSuchMethodError: 'com.fasterxml.jackson.core.JsonGenerator com.fasterxml.jackson.databind.ObjectMapper.createGenerator(java.io.OutputStream, com.fasterxml.jackson.core.JsonEncoding)' at org.apache.spark.util.JsonProtocol$.toJsonString(JsonProtocol.scala:75) at org.apache.spark.SparkThrowableHelper$.getMessage(SparkThrowableHelper.scala:74) at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$7(SQLExecution.scala:127) at scala.Option.map(Option.scala:230) at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$6(SQLExecution.scala:125) at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:195) at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:103) at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:827) at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:65) at org.apache.spark.sql.Dataset.withAction(Dataset.scala:4165) at org.apache.spark.sql.Dataset.head(Dataset.scala:3161) at org.apache.spark.sql.Dataset.take(Dataset.scala:3382) at org.apache.spark.sql.Dataset.takeAsList(Dataset.scala:3405) at com.crossix.safemine.cloud.utils.DebugRDDLogger.showDataset(DebugRDDLogger.java:84) at com.crossix.safemine.cloud.components.statistics.spark.StatisticsTransformer.getFillRateCountsWithSparkQuery(StatisticsTransformer.java:122) at com.crossix.safemine.cloud.components.statistics.spark.StatisticsTransformer.calculateStatistics(StatisticsTransformer.java:61) at com.crossix.safemine.cloud.components.statistics.spark.SparkFileStatistics.execute(SparkFileStatistics.java:102) at com.crossix.safemine.cloud.StatisticsFlow.calculateAllStatistics(StatisticsFlow.java:146) at com.crossix.safemine.cloud.StatisticsFlow.runStatistics(StatisticsFlow.java:119) at com.crossix.safemine.cloud.StatisticsFlow.initialFileStatistics(StatisticsFlow.java:77) at com.crossix.safemine.cloud.SMCFlow.process(SMCFlow.java:221) at com.crossix.safemine.cloud.SMCFlow.execute(SMCFlow.java:132) at com.crossix.safemine.cloud.SMCFlow.run(SMCFlow.java:91) I see that that spark package contains the dependency: com.fasterxml.jackson.core:jackson-databind:jar:2.10.5:compile But jackson-databind 2.10.5 does not contain ObjectMapper.createGenerator(java.io.OutputStream, com.fasterxml.jackson.core.JsonEncoding) It was added on 2.11.0 Trying to upgrade jackson-databind fails with: com.fasterxml.jackson.databind.JsonMappingException: Scala module 2.10.5 requires Jackson Databind version >= 2.10.0 and < 2.11.0 According to spark 3.3.0 release notes: "Upgrade Jackson to 2.13.3" but spark package of 3.4.1 contains Jackson of 2.10.5 ( <https://spark.apache.org/releases/spark-release-3-3-0.html> https://spark.apache.org/releases/spark-release-3-3-0.html) What am I missing? -- Moshik Vitas Senior Software Developer, Crossix Veeva Systems m: +972-54-5326-400 moshik.vitas@ <mailto:moshik.vi...@veeva.com> veeva.com -- Bjørn Jørgensen Vestre Aspehaug 4, 6010 Ålesund Norge +47 480 94 297 -- Bjørn Jørgensen Vestre Aspehaug 4, 6010 Ålesund Norge +47 480 94 297
