[jira] [Commented] (SPARK-8332) NoSuchMethodError: com.fasterxml.jackson.module.scala.deser.BigDecimalDeserializer
[ https://issues.apache.org/jira/browse/SPARK-8332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15350837#comment-15350837 ] Ian Hellstrom commented on SPARK-8332: -- Is there a solution for this? I'm running 1.4.1 on a Hortonworks cluster. One has Kerberos the other doesn't. On the non-Kerberos cluster the same job runs fine and on the one with Kerberos I get the same problem. AFAIK, both clusters are identical except for Kerberos. At least the /usr/hdp/ directories have the same library versions in them. > NoSuchMethodError: > com.fasterxml.jackson.module.scala.deser.BigDecimalDeserializer > -- > > Key: SPARK-8332 > URL: https://issues.apache.org/jira/browse/SPARK-8332 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 1.4.0 > Environment: spark 1.4 & hadoop 2.3.0-cdh5.0.0 >Reporter: Tao Li >Priority: Critical > Labels: 1.4.0, NoSuchMethodError, com.fasterxml.jackson > > I complied new spark 1.4.0 version. > But when I run a simple WordCount demo, it throws NoSuchMethodError > {code} > java.lang.NoSuchMethodError: > com.fasterxml.jackson.module.scala.deser.BigDecimalDeserializer > {code} > I found out that the default "fasterxml.jackson.version" is 2.4.4. > Is there any wrong or conflict with the jackson version? > Or is there possibly some project maven dependency containing the wrong > version of jackson? -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-8332) NoSuchMethodError: com.fasterxml.jackson.module.scala.deser.BigDecimalDeserializer
[ https://issues.apache.org/jira/browse/SPARK-8332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15006828#comment-15006828 ] Pedro Vilaça commented on SPARK-8332: - We're facing the same problem with a spark streaming job and I noticed that this issue was closed. Don't you have a plan to upgrade the jackson version that is being used? > NoSuchMethodError: > com.fasterxml.jackson.module.scala.deser.BigDecimalDeserializer > -- > > Key: SPARK-8332 > URL: https://issues.apache.org/jira/browse/SPARK-8332 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 1.4.0 > Environment: spark 1.4 & hadoop 2.3.0-cdh5.0.0 >Reporter: Tao Li >Priority: Critical > Labels: 1.4.0, NoSuchMethodError, com.fasterxml.jackson > > I complied new spark 1.4.0 version. > But when I run a simple WordCount demo, it throws NoSuchMethodError > {code} > java.lang.NoSuchMethodError: > com.fasterxml.jackson.module.scala.deser.BigDecimalDeserializer > {code} > I found out that the default "fasterxml.jackson.version" is 2.4.4. > Is there any wrong or conflict with the jackson version? > Or is there possibly some project maven dependency containing the wrong > version of jackson? -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-8332) NoSuchMethodError: com.fasterxml.jackson.module.scala.deser.BigDecimalDeserializer
[ https://issues.apache.org/jira/browse/SPARK-8332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14965900#comment-14965900 ] Jonathan Kelly commented on SPARK-8332: --- Sorry, yes, I should have checked to see what the latest version of Jackson 2.x is now. Looks like it's Jackson 2.6.2, so we'd probably want to try to upgrade to that if anything. And yes, Hadoop's Jackson can also collide, so we'd either need to upgrade it or shade it there as well. Classpath hell is hell. :( > NoSuchMethodError: > com.fasterxml.jackson.module.scala.deser.BigDecimalDeserializer > -- > > Key: SPARK-8332 > URL: https://issues.apache.org/jira/browse/SPARK-8332 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 1.4.0 > Environment: spark 1.4 & hadoop 2.3.0-cdh5.0.0 >Reporter: Tao Li >Priority: Critical > Labels: 1.4.0, NoSuchMethodError, com.fasterxml.jackson > > I complied new spark 1.4.0 version. > But when I run a simple WordCount demo, it throws NoSuchMethodError > {code} > java.lang.NoSuchMethodError: > com.fasterxml.jackson.module.scala.deser.BigDecimalDeserializer > {code} > I found out that the default "fasterxml.jackson.version" is 2.4.4. > Is there any wrong or conflict with the jackson version? > Or is there possibly some project maven dependency containing the wrong > version of jackson? -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-8332) NoSuchMethodError: com.fasterxml.jackson.module.scala.deser.BigDecimalDeserializer
[ https://issues.apache.org/jira/browse/SPARK-8332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14965829#comment-14965829 ] Sean Owen commented on SPARK-8332: -- How about 2.6, just out of curiosity? The other issue is colliding with Hadoop's Jackson, which I'm pretty sure is also not shaded. Of course, shading is an answer there, for Spark, also worth a try. It may be that it just pushes the problem around, and now your app still has to shade its Jackson to dodge Hadoop's copy, until Hadoop 3 at least. What a mess eh? > NoSuchMethodError: > com.fasterxml.jackson.module.scala.deser.BigDecimalDeserializer > -- > > Key: SPARK-8332 > URL: https://issues.apache.org/jira/browse/SPARK-8332 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 1.4.0 > Environment: spark 1.4 & hadoop 2.3.0-cdh5.0.0 >Reporter: Tao Li >Priority: Critical > Labels: 1.4.0, NoSuchMethodError, com.fasterxml.jackson > > I complied new spark 1.4.0 version. > But when I run a simple WordCount demo, it throws NoSuchMethodError > {code} > java.lang.NoSuchMethodError: > com.fasterxml.jackson.module.scala.deser.BigDecimalDeserializer > {code} > I found out that the default "fasterxml.jackson.version" is 2.4.4. > Is there any wrong or conflict with the jackson version? > Or is there possibly some project maven dependency containing the wrong > version of jackson? -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-8332) NoSuchMethodError: com.fasterxml.jackson.module.scala.deser.BigDecimalDeserializer
[ https://issues.apache.org/jira/browse/SPARK-8332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14965703#comment-14965703 ] Jonathan Kelly commented on SPARK-8332: --- Good news! I tried compiling Spark with Jackson 2.5.3 instead of 2.4.4, and it seems to work just fine (as long as this is the only version of Jackson 2.x in the classpath; the issue described above does indeed only occur when multiple versions are in the classpath). Would it be possible to upgrade to Jackson 2.5.3 in Spark 1.6.0? If so, should I cut an issue for this? > NoSuchMethodError: > com.fasterxml.jackson.module.scala.deser.BigDecimalDeserializer > -- > > Key: SPARK-8332 > URL: https://issues.apache.org/jira/browse/SPARK-8332 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 1.4.0 > Environment: spark 1.4 & hadoop 2.3.0-cdh5.0.0 >Reporter: Tao Li >Priority: Critical > Labels: 1.4.0, NoSuchMethodError, com.fasterxml.jackson > > I complied new spark 1.4.0 version. > But when I run a simple WordCount demo, it throws NoSuchMethodError > {code} > java.lang.NoSuchMethodError: > com.fasterxml.jackson.module.scala.deser.BigDecimalDeserializer > {code} > I found out that the default "fasterxml.jackson.version" is 2.4.4. > Is there any wrong or conflict with the jackson version? > Or is there possibly some project maven dependency containing the wrong > version of jackson? -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-8332) NoSuchMethodError: com.fasterxml.jackson.module.scala.deser.BigDecimalDeserializer
[ https://issues.apache.org/jira/browse/SPARK-8332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14965484#comment-14965484 ] Jonathan Kelly commented on SPARK-8332: --- Oh, you might be right. I was thinking that I had tried compiling Spark with a newer version of Jackson (2.5.3) and still ran into problems, but now I'm second guessing myself. It might have only been that my classpath had both versions. > NoSuchMethodError: > com.fasterxml.jackson.module.scala.deser.BigDecimalDeserializer > -- > > Key: SPARK-8332 > URL: https://issues.apache.org/jira/browse/SPARK-8332 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 1.4.0 > Environment: spark 1.4 & hadoop 2.3.0-cdh5.0.0 >Reporter: Tao Li >Priority: Critical > Labels: 1.4.0, NoSuchMethodError, com.fasterxml.jackson > > I complied new spark 1.4.0 version. > But when I run a simple WordCount demo, it throws NoSuchMethodError > {code} > java.lang.NoSuchMethodError: > com.fasterxml.jackson.module.scala.deser.BigDecimalDeserializer > {code} > I found out that the default "fasterxml.jackson.version" is 2.4.4. > Is there any wrong or conflict with the jackson version? > Or is there possibly some project maven dependency containing the wrong > version of jackson? -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-8332) NoSuchMethodError: com.fasterxml.jackson.module.scala.deser.BigDecimalDeserializer
[ https://issues.apache.org/jira/browse/SPARK-8332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14965477#comment-14965477 ] Sean Owen commented on SPARK-8332: -- I think Spark can use a later Jackson, though that's a different question. The question here is version mismatch. Shading might be possible too. I have this distant memory that it causes some problem, but it could be gone now. A first step may be to just see if shading works (i.e. tests pass). If so, this is likely the best way forward yes. > NoSuchMethodError: > com.fasterxml.jackson.module.scala.deser.BigDecimalDeserializer > -- > > Key: SPARK-8332 > URL: https://issues.apache.org/jira/browse/SPARK-8332 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 1.4.0 > Environment: spark 1.4 & hadoop 2.3.0-cdh5.0.0 >Reporter: Tao Li >Priority: Critical > Labels: 1.4.0, NoSuchMethodError, com.fasterxml.jackson > > I complied new spark 1.4.0 version. > But when I run a simple WordCount demo, it throws NoSuchMethodError > {code} > java.lang.NoSuchMethodError: > com.fasterxml.jackson.module.scala.deser.BigDecimalDeserializer > {code} > I found out that the default "fasterxml.jackson.version" is 2.4.4. > Is there any wrong or conflict with the jackson version? > Or is there possibly some project maven dependency containing the wrong > version of jackson? -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-8332) NoSuchMethodError: com.fasterxml.jackson.module.scala.deser.BigDecimalDeserializer
[ https://issues.apache.org/jira/browse/SPARK-8332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14965459#comment-14965459 ] Jonathan Kelly commented on SPARK-8332: --- [~srowen], isn't this still a problem? It seems odd that Spark cannot be upgraded beyond Jackson 2.4.4. It would at least be nice if Jackson were shadowed inside the Spark assembly so that users may use newer versions of Jackson in application code. > NoSuchMethodError: > com.fasterxml.jackson.module.scala.deser.BigDecimalDeserializer > -- > > Key: SPARK-8332 > URL: https://issues.apache.org/jira/browse/SPARK-8332 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 1.4.0 > Environment: spark 1.4 & hadoop 2.3.0-cdh5.0.0 >Reporter: Tao Li >Priority: Critical > Labels: 1.4.0, NoSuchMethodError, com.fasterxml.jackson > > I complied new spark 1.4.0 version. > But when I run a simple WordCount demo, it throws NoSuchMethodError > {code} > java.lang.NoSuchMethodError: > com.fasterxml.jackson.module.scala.deser.BigDecimalDeserializer > {code} > I found out that the default "fasterxml.jackson.version" is 2.4.4. > Is there any wrong or conflict with the jackson version? > Or is there possibly some project maven dependency containing the wrong > version of jackson? -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-8332) NoSuchMethodError: com.fasterxml.jackson.module.scala.deser.BigDecimalDeserializer
[ https://issues.apache.org/jira/browse/SPARK-8332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14964563#comment-14964563 ] Earthson Lu commented on SPARK-8332: SparkUI not works when upgrade fasterxml.jackson to 2.5.3 > NoSuchMethodError: > com.fasterxml.jackson.module.scala.deser.BigDecimalDeserializer > -- > > Key: SPARK-8332 > URL: https://issues.apache.org/jira/browse/SPARK-8332 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 1.4.0 > Environment: spark 1.4 & hadoop 2.3.0-cdh5.0.0 >Reporter: Tao Li >Priority: Critical > Labels: 1.4.0, NoSuchMethodError, com.fasterxml.jackson > > I complied new spark 1.4.0 version. > But when I run a simple WordCount demo, it throws NoSuchMethodError > {code} > java.lang.NoSuchMethodError: > com.fasterxml.jackson.module.scala.deser.BigDecimalDeserializer > {code} > I found out that the default "fasterxml.jackson.version" is 2.4.4. > Is there any wrong or conflict with the jackson version? > Or is there possibly some project maven dependency containing the wrong > version of jackson? -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-8332) NoSuchMethodError: com.fasterxml.jackson.module.scala.deser.BigDecimalDeserializer
[ https://issues.apache.org/jira/browse/SPARK-8332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14964562#comment-14964562 ] Earthson Lu commented on SPARK-8332: SparkUI not works when upgrade fasterxml.jackson to 2.5.3 > NoSuchMethodError: > com.fasterxml.jackson.module.scala.deser.BigDecimalDeserializer > -- > > Key: SPARK-8332 > URL: https://issues.apache.org/jira/browse/SPARK-8332 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 1.4.0 > Environment: spark 1.4 & hadoop 2.3.0-cdh5.0.0 >Reporter: Tao Li >Priority: Critical > Labels: 1.4.0, NoSuchMethodError, com.fasterxml.jackson > > I complied new spark 1.4.0 version. > But when I run a simple WordCount demo, it throws NoSuchMethodError > {code} > java.lang.NoSuchMethodError: > com.fasterxml.jackson.module.scala.deser.BigDecimalDeserializer > {code} > I found out that the default "fasterxml.jackson.version" is 2.4.4. > Is there any wrong or conflict with the jackson version? > Or is there possibly some project maven dependency containing the wrong > version of jackson? -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-8332) NoSuchMethodError: com.fasterxml.jackson.module.scala.deser.BigDecimalDeserializer
[ https://issues.apache.org/jira/browse/SPARK-8332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14636919#comment-14636919 ] Earthson Lu commented on SPARK-8332: I recompiled spark with fasterxml.jackson 2.5.3, it works with play-2.4.x I want to know: is this ok to use 2.5.3 instead of 2.4.4? > NoSuchMethodError: > com.fasterxml.jackson.module.scala.deser.BigDecimalDeserializer > -- > > Key: SPARK-8332 > URL: https://issues.apache.org/jira/browse/SPARK-8332 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 1.4.0 > Environment: spark 1.4 & hadoop 2.3.0-cdh5.0.0 >Reporter: Tao Li >Priority: Critical > Labels: 1.4.0, NoSuchMethodError, com.fasterxml.jackson > > I complied new spark 1.4.0 version. > But when I run a simple WordCount demo, it throws NoSuchMethodError > {code} > java.lang.NoSuchMethodError: > com.fasterxml.jackson.module.scala.deser.BigDecimalDeserializer > {code} > I found out that the default "fasterxml.jackson.version" is 2.4.4. > Is there any wrong or conflict with the jackson version? > Or is there possibly some project maven dependency containing the wrong > version of jackson? -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-8332) NoSuchMethodError: com.fasterxml.jackson.module.scala.deser.BigDecimalDeserializer
[ https://issues.apache.org/jira/browse/SPARK-8332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14621067#comment-14621067 ] Jonathan Kelly commented on SPARK-8332: --- Thanks, Sean. Is there any reason why Jackson can't be shaded in the Spark assembly? It's such a common package to use that it would really help for it to be shaded so that Spark apps aren't locked in to a particular Jackson version. > NoSuchMethodError: > com.fasterxml.jackson.module.scala.deser.BigDecimalDeserializer > -- > > Key: SPARK-8332 > URL: https://issues.apache.org/jira/browse/SPARK-8332 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 1.4.0 > Environment: spark 1.4 & hadoop 2.3.0-cdh5.0.0 >Reporter: Tao Li >Priority: Critical > Labels: 1.4.0, NoSuchMethodError, com.fasterxml.jackson > > I complied new spark 1.4.0 version. > But when I run a simple WordCount demo, it throws NoSuchMethodError > {code} > java.lang.NoSuchMethodError: > com.fasterxml.jackson.module.scala.deser.BigDecimalDeserializer > {code} > I found out that the default "fasterxml.jackson.version" is 2.4.4. > Is there any wrong or conflict with the jackson version? > Or is there possibly some project maven dependency containing the wrong > version of jackson? -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-8332) NoSuchMethodError: com.fasterxml.jackson.module.scala.deser.BigDecimalDeserializer
[ https://issues.apache.org/jira/browse/SPARK-8332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14621022#comment-14621022 ] Sean Owen commented on SPARK-8332: -- This has always happened to me when using Jackson > 2.5. Just use Jackson 2.4.x if you can. It's not shaded in Spark so you can't necessarily mix a different version in your app. > NoSuchMethodError: > com.fasterxml.jackson.module.scala.deser.BigDecimalDeserializer > -- > > Key: SPARK-8332 > URL: https://issues.apache.org/jira/browse/SPARK-8332 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 1.4.0 > Environment: spark 1.4 & hadoop 2.3.0-cdh5.0.0 >Reporter: Tao Li >Priority: Critical > Labels: 1.4.0, NoSuchMethodError, com.fasterxml.jackson > > I complied new spark 1.4.0 version. > But when I run a simple WordCount demo, it throws NoSuchMethodError > {code} > java.lang.NoSuchMethodError: > com.fasterxml.jackson.module.scala.deser.BigDecimalDeserializer > {code} > I found out that the default "fasterxml.jackson.version" is 2.4.4. > Is there any wrong or conflict with the jackson version? > Or is there possibly some project maven dependency containing the wrong > version of jackson? -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-8332) NoSuchMethodError: com.fasterxml.jackson.module.scala.deser.BigDecimalDeserializer
[ https://issues.apache.org/jira/browse/SPARK-8332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14620975#comment-14620975 ] Kevin Tham commented on SPARK-8332: --- I had no issue having play-json_2.11:2.4.0 (which depends on jackson 2.5.3) in my SBT build with Spark 1.3, and then upgrading to Spark 1.4 leads me to not be able to do an "sc.parallelize(someCollection)" where the collection is a Seq[(TreeMap[String, Double], Double)] and am able to reproduce the error Jonathan Kelly saw with (Exception in thread "main" com.fasterxml.jackson.databind.JsonMappingException: Could not find creator property with name 'id' (in class org.apache.spark.rdd.RDDOperationScope)) I would help when I have time to figure out why, but I'm interested to see what commit caused this and the actual cause of this error. This is causing us to stick with Spark 1.3 for now. > NoSuchMethodError: > com.fasterxml.jackson.module.scala.deser.BigDecimalDeserializer > -- > > Key: SPARK-8332 > URL: https://issues.apache.org/jira/browse/SPARK-8332 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 1.4.0 > Environment: spark 1.4 & hadoop 2.3.0-cdh5.0.0 >Reporter: Tao Li >Priority: Critical > Labels: 1.4.0, NoSuchMethodError, com.fasterxml.jackson > > I complied new spark 1.4.0 version. > But when I run a simple WordCount demo, it throws NoSuchMethodError > {code} > java.lang.NoSuchMethodError: > com.fasterxml.jackson.module.scala.deser.BigDecimalDeserializer > {code} > I found out that the default "fasterxml.jackson.version" is 2.4.4. > Is there any wrong or conflict with the jackson version? > Or is there possibly some project maven dependency containing the wrong > version of jackson? -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-8332) NoSuchMethodError: com.fasterxml.jackson.module.scala.deser.BigDecimalDeserializer
[ https://issues.apache.org/jira/browse/SPARK-8332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14604046#comment-14604046 ] Olivier Girardot commented on SPARK-8332: - Great [~jonathak] can you please post the configuration parameters you've been using (including the classpath) ? > NoSuchMethodError: > com.fasterxml.jackson.module.scala.deser.BigDecimalDeserializer > -- > > Key: SPARK-8332 > URL: https://issues.apache.org/jira/browse/SPARK-8332 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 1.4.0 > Environment: spark 1.4 & hadoop 2.3.0-cdh5.0.0 >Reporter: Tao Li >Priority: Critical > Labels: 1.4.0, NoSuchMethodError, com.fasterxml.jackson > > I complied new spark 1.4.0 version. > But when I run a simple WordCount demo, it throws NoSuchMethodError > {code} > java.lang.NoSuchMethodError: > com.fasterxml.jackson.module.scala.deser.BigDecimalDeserializer > {code} > I found out that the default "fasterxml.jackson.version" is 2.4.4. > Is there any wrong or conflict with the jackson version? > Or is there possibly some project maven dependency containing the wrong > version of jackson? -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-8332) NoSuchMethodError: com.fasterxml.jackson.module.scala.deser.BigDecimalDeserializer
[ https://issues.apache.org/jira/browse/SPARK-8332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14603881#comment-14603881 ] Jonathan Kelly commented on SPARK-8332: --- Actually, it seems like it may have been the fact that I was using Jackson 2.5.3 instead of Jackson 2.4.4. I downgraded and was able to run the SparkPi example successfully. > NoSuchMethodError: > com.fasterxml.jackson.module.scala.deser.BigDecimalDeserializer > -- > > Key: SPARK-8332 > URL: https://issues.apache.org/jira/browse/SPARK-8332 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 1.4.0 > Environment: spark 1.4 & hadoop 2.3.0-cdh5.0.0 >Reporter: Tao Li >Priority: Critical > Labels: 1.4.0, NoSuchMethodError, com.fasterxml.jackson > > I complied new spark 1.4.0 version. > But when I run a simple WordCount demo, it throws NoSuchMethodError > {code} > java.lang.NoSuchMethodError: > com.fasterxml.jackson.module.scala.deser.BigDecimalDeserializer > {code} > I found out that the default "fasterxml.jackson.version" is 2.4.4. > Is there any wrong or conflict with the jackson version? > Or is there possibly some project maven dependency containing the wrong > version of jackson? -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-8332) NoSuchMethodError: com.fasterxml.jackson.module.scala.deser.BigDecimalDeserializer
[ https://issues.apache.org/jira/browse/SPARK-8332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14603751#comment-14603751 ] Jonathan Kelly commented on SPARK-8332: --- I was running into the same issue, so I made sure to get rid of all Jackson 2.2 jars from my Hadoop classpath, but now I've run into a different issue: Exception in thread "main" com.fasterxml.jackson.databind.JsonMappingException: Could not find creator property with name 'id' (in class org.apache.spark.rdd.RDDOperationScope) at [Source: {"id":"0","name":"parallelize"}; line: 1, column: 1] at com.fasterxml.jackson.databind.JsonMappingException.from(JsonMappingException.java:148) at com.fasterxml.jackson.databind.DeserializationContext.mappingException(DeserializationContext.java:843) at com.fasterxml.jackson.databind.deser.BeanDeserializerFactory.addBeanProps(BeanDeserializerFactory.java:533) at com.fasterxml.jackson.databind.deser.BeanDeserializerFactory.buildBeanDeserializer(BeanDeserializerFactory.java:220) at com.fasterxml.jackson.databind.deser.BeanDeserializerFactory.createBeanDeserializer(BeanDeserializerFactory.java:143) at com.fasterxml.jackson.databind.deser.DeserializerCache._createDeserializer2(DeserializerCache.java:409) at com.fasterxml.jackson.databind.deser.DeserializerCache._createDeserializer(DeserializerCache.java:358) at com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCache2(DeserializerCache.java:265) at com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCacheValueDeserializer(DeserializerCache.java:245) at com.fasterxml.jackson.databind.deser.DeserializerCache.findValueDeserializer(DeserializerCache.java:143) at com.fasterxml.jackson.databind.DeserializationContext.findRootValueDeserializer(DeserializationContext.java:439) at com.fasterxml.jackson.databind.ObjectMapper._findRootDeserializer(ObjectMapper.java:3666) at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:3558) at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2578) at org.apache.spark.rdd.RDDOperationScope$.fromJson(RDDOperationScope.scala:82) at org.apache.spark.rdd.RDD$$anonfun$34.apply(RDD.scala:1486) at org.apache.spark.rdd.RDD$$anonfun$34.apply(RDD.scala:1486) at scala.Option.map(Option.scala:145) at org.apache.spark.rdd.RDD.(RDD.scala:1486) at org.apache.spark.rdd.ParallelCollectionRDD.(ParallelCollectionRDD.scala:85) at org.apache.spark.SparkContext$$anonfun$parallelize$1.apply(SparkContext.scala:697) at org.apache.spark.SparkContext$$anonfun$parallelize$1.apply(SparkContext.scala:695) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:148) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:109) at org.apache.spark.SparkContext.withScope(SparkContext.scala:681) at org.apache.spark.SparkContext.parallelize(SparkContext.scala:695) at org.apache.spark.examples.SparkPi$.main(SparkPi.scala:31) at org.apache.spark.examples.SparkPi.main(SparkPi.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:664) at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:169) at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:192) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:111) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) I'm just running: MASTER=yarn-client /usr/lib/spark/bin/run-example SparkPi 10 > NoSuchMethodError: > com.fasterxml.jackson.module.scala.deser.BigDecimalDeserializer > -- > > Key: SPARK-8332 > URL: https://issues.apache.org/jira/browse/SPARK-8332 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 1.4.0 > Environment: spark 1.4 & hadoop 2.3.0-cdh5.0.0 >Reporter: Tao Li >Priority: Critical > Labels: 1.4.0, NoSuchMethodError, com.fasterxml.jackson > > I complied new spark 1.4.0 version. > But when I run a simple WordCount demo, it throws NoSuchMethodError > {code} > java.lang.NoSuchMethodError: > com.fasterxml.jackson.module.scala.deser.BigDecimalDeserializer > {code} > I found out that the defaul
[jira] [Commented] (SPARK-8332) NoSuchMethodError: com.fasterxml.jackson.module.scala.deser.BigDecimalDeserializer
[ https://issues.apache.org/jira/browse/SPARK-8332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14602698#comment-14602698 ] Olivier Girardot commented on SPARK-8332: - Great Tao ! can you please send your SPARK_CLASSPATH ? > NoSuchMethodError: > com.fasterxml.jackson.module.scala.deser.BigDecimalDeserializer > -- > > Key: SPARK-8332 > URL: https://issues.apache.org/jira/browse/SPARK-8332 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 1.4.0 > Environment: spark 1.4 & hadoop 2.3.0-cdh5.0.0 >Reporter: Tao Li >Priority: Critical > Labels: 1.4.0, NoSuchMethodError, com.fasterxml.jackson > > I complied new spark 1.4.0 version. > But when I run a simple WordCount demo, it throws NoSuchMethodError > {code} > java.lang.NoSuchMethodError: > com.fasterxml.jackson.module.scala.deser.BigDecimalDeserializer > {code} > I found out that the default "fasterxml.jackson.version" is 2.4.4. > Is there any wrong or conflict with the jackson version? > Or is there possibly some project maven dependency containing the wrong > version of jackson? -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-8332) NoSuchMethodError: com.fasterxml.jackson.module.scala.deser.BigDecimalDeserializer
[ https://issues.apache.org/jira/browse/SPARK-8332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14602693#comment-14602693 ] Tao Li commented on SPARK-8332: --- Hi Olivier Girardot, you can see all the jars in your classpath at the web UI "Environment" tab. For this case, you can check if you have old version jackson related jars (under version 2.2). If you want to custom your own classpath, you can edit conf/spark-env.sh file and change SPARK_CLASSPATH variable. For me, I have add 'hadoop classpath' in my SPARK_CLASSPATH. But hadoop classpath contains 2.2 version jackson jars. So I have to exclude the jars in the SPARK_CLASSPATH. It help me to overcome this NoSuchMethodError. So I suggest you to check your SPARK_CLASSPATH carefully. > NoSuchMethodError: > com.fasterxml.jackson.module.scala.deser.BigDecimalDeserializer > -- > > Key: SPARK-8332 > URL: https://issues.apache.org/jira/browse/SPARK-8332 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 1.4.0 > Environment: spark 1.4 & hadoop 2.3.0-cdh5.0.0 >Reporter: Tao Li >Priority: Critical > Labels: 1.4.0, NoSuchMethodError, com.fasterxml.jackson > > I complied new spark 1.4.0 version. > But when I run a simple WordCount demo, it throws NoSuchMethodError > {code} > java.lang.NoSuchMethodError: > com.fasterxml.jackson.module.scala.deser.BigDecimalDeserializer > {code} > I found out that the default "fasterxml.jackson.version" is 2.4.4. > Is there any wrong or conflict with the jackson version? > Or is there possibly some project maven dependency containing the wrong > version of jackson? -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-8332) NoSuchMethodError: com.fasterxml.jackson.module.scala.deser.BigDecimalDeserializer
[ https://issues.apache.org/jira/browse/SPARK-8332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14602477#comment-14602477 ] Olivier Girardot commented on SPARK-8332: - Ok can you print the command line you're using to submit your job. My conclusion was that it's difficult to create classpath compatible between hive/hadoop from CDH5.x and Spark if you want to use hive tables... I'd like to see your classpath in order to better understand what is going on. > NoSuchMethodError: > com.fasterxml.jackson.module.scala.deser.BigDecimalDeserializer > -- > > Key: SPARK-8332 > URL: https://issues.apache.org/jira/browse/SPARK-8332 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 1.4.0 > Environment: spark 1.4 & hadoop 2.3.0-cdh5.0.0 >Reporter: Tao Li >Priority: Critical > Labels: 1.4.0, NoSuchMethodError, com.fasterxml.jackson > > I complied new spark 1.4.0 version. > But when I run a simple WordCount demo, it throws NoSuchMethodError > {code} > java.lang.NoSuchMethodError: > com.fasterxml.jackson.module.scala.deser.BigDecimalDeserializer > {code} > I found out that the default "fasterxml.jackson.version" is 2.4.4. > Is there any wrong or conflict with the jackson version? > Or is there possibly some project maven dependency containing the wrong > version of jackson? -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-8332) NoSuchMethodError: com.fasterxml.jackson.module.scala.deser.BigDecimalDeserializer
[ https://issues.apache.org/jira/browse/SPARK-8332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14602468#comment-14602468 ] Tao Li commented on SPARK-8332: --- I found that BigDecimalDeserializer will extend StdDeserializer in jackson-databind project. Above version jackson-databind-2.3, StdDeserializer have method handledType(). But under version jackson-databind-2.2, StdDeserializer don't have method handledType(). In my environment, hadoop 2.3.0-cdh5.0.0, the jackson-databind is /usr/lib/hadoop-mapreduce/.//jackson-databind-2.2.3.jar. So it don't have the method handledType() and throws NoSuchMethodError. > NoSuchMethodError: > com.fasterxml.jackson.module.scala.deser.BigDecimalDeserializer > -- > > Key: SPARK-8332 > URL: https://issues.apache.org/jira/browse/SPARK-8332 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 1.4.0 > Environment: spark 1.4 & hadoop 2.3.0-cdh5.0.0 >Reporter: Tao Li >Priority: Critical > Labels: 1.4.0, NoSuchMethodError, com.fasterxml.jackson > > I complied new spark 1.4.0 version. > But when I run a simple WordCount demo, it throws NoSuchMethodError > {code} > java.lang.NoSuchMethodError: > com.fasterxml.jackson.module.scala.deser.BigDecimalDeserializer > {code} > I found out that the default "fasterxml.jackson.version" is 2.4.4. > Is there any wrong or conflict with the jackson version? > Or is there possibly some project maven dependency containing the wrong > version of jackson? -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-8332) NoSuchMethodError: com.fasterxml.jackson.module.scala.deser.BigDecimalDeserializer
[ https://issues.apache.org/jira/browse/SPARK-8332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14593898#comment-14593898 ] Olivier Girardot commented on SPARK-8332: - You're right sorry > NoSuchMethodError: > com.fasterxml.jackson.module.scala.deser.BigDecimalDeserializer > -- > > Key: SPARK-8332 > URL: https://issues.apache.org/jira/browse/SPARK-8332 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 1.4.0 > Environment: spark 1.4 & hadoop 2.3.0-cdh5.0.0 >Reporter: Tao Li >Priority: Critical > Labels: 1.4.0, NoSuchMethodError, com.fasterxml.jackson > > I complied new spark 1.4.0 version. > But when I run a simple WordCount demo, it throws NoSuchMethodError > {code} > java.lang.NoSuchMethodError: > com.fasterxml.jackson.module.scala.deser.BigDecimalDeserializer > {code} > I found out that the default "fasterxml.jackson.version" is 2.4.4. > Is there any wrong or conflict with the jackson version? > Or is there possibly some project maven dependency containing the wrong > version of jackson? -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-8332) NoSuchMethodError: com.fasterxml.jackson.module.scala.deser.BigDecimalDeserializer
[ https://issues.apache.org/jira/browse/SPARK-8332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14589903#comment-14589903 ] Olivier Girardot commented on SPARK-8332: - I have the same issue on a CDH5 cluster > NoSuchMethodError: > com.fasterxml.jackson.module.scala.deser.BigDecimalDeserializer > -- > > Key: SPARK-8332 > URL: https://issues.apache.org/jira/browse/SPARK-8332 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 1.4.0 > Environment: spark 1.4 & hadoop 2.3.0-cdh5.0.0 >Reporter: Tao Li >Priority: Critical > Labels: 1.4.0, NoSuchMethodError, com.fasterxml.jackson > > I complied new spark 1.4.0 versio. But when I run a simple WordCount demo, it > throws NoSuchMethodError "java.lang.NoSuchMethodError: > com.fasterxml.jackson.module.scala.deser.BigDecimalDeserializer". > I found the default "fasterxml.jackson.version" is 2.4.4. It's there any > wrong or conflict with the jackson version? Or is there possible some project > maven dependency contains wrong version jackson? -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org