Steve, Thank you for the answer. How Hortonworks deal with this problem internally ? You have Spark 1.3.1 in HDP 2.3. Is it compilled with Jackson 2.2.3 ?
Regards, Maciek 2016-01-13 18:00 GMT+01:00 Steve Loughran <ste...@hortonworks.com>: > >> On 13 Jan 2016, at 03:23, Maciej Bryński <mac...@brynski.pl> wrote: >> >> Thanks. >> I successfully compiled Spark 1.6.0 with Jackson 2.2.3 from source. >> >> I'll try to using it. >> > > This is the eternal classpath version problem, with Jackson turning out to be > incredibly brittle. After one point update of the 1.x JAR broke things (it > removed a method), there's ~0 enthusiasm for incrementing the version > counters again. > > > 1. Hadoop, even Hadoop trunk, is built with : > <jackson2.version>2.2.3</jackson2.version> > > this means that right now, the Spark 1.6 shuffle isn't going to work in *any* > Hadoop cluster that hasn't been built with a compatible Jackson version: > either rebuild spark 1.6 or hadoop-core itself. > > > 2. There's a YARN JIRA for classpath isolation in aux services: > https://issues.apache.org/jira/browse/YARN-4577 , with a longer term one for > forking off the services entirely: > https://issues.apache.org/jira/browse/YARN-1593 . there's a patch for the > first one, which, if someone want's to apply and test locally, would be > valued. It wouldn't ship in Hadoop until 2.9. . Patch #2 got assigned to > someone last week, so maybe it'll surface in Hadoop 2.9 instead/as well. > > 3. I'm going to open a SPARK JIRA here, cross link it to the YARN ones -so at > least there'll be a central record. (Done: SPARK-12807) > > 4. I'll also add an "upgrade jackson" issue under HADOOP-9991, though like I > said: enthusiasm will be low. > > 5. You can D/L a version of spark 1.6 built against HDP 2.3: > http://hortonworks.com/hadoop-tutorial/apache-spark-1-6-technical-preview-with-hdp-2-3/ > > This isn't likely to work against HDP 2.2 BTW; later Hadoop JAR versions. > > -I suspect for things to work on CDH there'll be something similar. For ASF > Hadoop, rebuilding spark is all you have. > > 6. Looking @spark/master, it's been on jackson 2.5.3 since last month, from > SPARK-12269. Which is just going to make versioning even more traumatic. And > we know that amazon-aws has a back compatibility track record, so swapping > things around there is going to be fun. You'd probably need to rebuild > Hadoop-2.7.2+ with the later aws/s3 JARs to keep everything aligned > > on #4: has anyone found any compatibility problems if they swap out Jackson > 2.2.3 for Jackson 2.4.4 or 2.5.3 *without recompiling anything*? That's what > we need to know for Hadoop JAR updates. > > -Steve > > > > > >> 2016-01-13 11:25 GMT+01:00 Ted Yu <yuzhih...@gmail.com>: >>> I would suggest trying option #1 first. >>> >>> Thanks >>> >>>> On Jan 13, 2016, at 2:12 AM, Maciej Bryński <mac...@brynski.pl> wrote: >>>> >>>> Hi, >>>> I/m trying to run Spark 1.6.0 on HDP 2.2 >>>> Everything was fine until I tried to turn on dynamic allocation. >>>> According to instruction I need to add shuffle service to yarn classpath. >>>> The problem is that HDP 2.2 has jackson 2.2.3 and Spark is using 2.4.4. >>>> So connecting it gives error: >>>> >>>> 2016-01-11 16:56:51,222 INFO containermanager.AuxServices >>>> (AuxServices.java:addService(72)) - Adding auxiliary service >>>> spark_shuffle, "spark_shuffle" >>>> 2016-01-11 16:56:51,439 FATAL nodemanager.NodeManager >>>> (NodeManager.java:initAndStartNodeManager(465)) - Error starting >>>> NodeManager >>>> java.lang.NoSuchMethodError: >>>> com.fasterxml.jackson.core.JsonFactory.requiresPropertyOrdering()Z >>>> at >>>> com.fasterxml.jackson.databind.ObjectMapper.<init>(ObjectMapper.java:457) >>>> at >>>> com.fasterxml.jackson.databind.ObjectMapper.<init>(ObjectMapper.java:379) >>>> at >>>> org.apache.spark.network.shuffle.ExternalShuffleBlockResolver.<clinit>(ExternalShuffleBlockResolver.java:57) >>>> at >>>> org.apache.spark.network.shuffle.ExternalShuffleBlockHandler.<init>(ExternalShuffleBlockHandler.java:56) >>>> at >>>> org.apache.spark.network.yarn.YarnShuffleService.serviceInit(YarnShuffleService.java:128) >>>> at >>>> org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) >>>> at >>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices.serviceInit(AuxServices.java:143) >>>> at >>>> org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) >>>> at >>>> org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107) >>>> at >>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.serviceInit(ContainerManagerImpl.java:237) >>>> at >>>> org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) >>>> at >>>> org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107) >>>> at >>>> org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:253) >>>> at >>>> org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) >>>> at >>>> org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:462) >>>> at >>>> org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:509) >>>> >>>> >>>> What can I do ? >>>> I have following ideas: >>>> 1) Compile Spark 1.6.0 with modified pom.xml (change jackson version >>>> to 2.2.3). I'm not sure if this will be working >>>> 2) I tried to put shuffle service from different version of Spark. >>>> 1.4.1 works on HDP 2.2. >>>> Is it possible to run shuffle service from 1.4.1 with Spark 1.6.0 ? >>>> 3) Other ideas ? >>>> >>>> Regards, >>>> -- >>>> Maciek Bryński >>>> >>>> --------------------------------------------------------------------- >>>> To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org >>>> For additional commands, e-mail: dev-h...@spark.apache.org >>>> >> >> >> >> -- >> Maciek Bryński >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org >> For additional commands, e-mail: dev-h...@spark.apache.org > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org > For additional commands, e-mail: dev-h...@spark.apache.org > -- Maciek Bryński --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
