[ https://issues.apache.org/jira/browse/PIG-4693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14944494#comment-14944494 ]
Srikanth Sundarrajan commented on PIG-4693: ------------------------------------------- Running the following simple pig script {code} IN = LOAD 'test-data' USING PigStorage(''); G = GROUP IN BY $11; R = FOREACH G GENERATE group, SUM(IN.$10); STORE R INTO 'test-out' USING PigStorage(','); {code} results in {noformat} ERROR 2998: Unhandled internal error. com.esotericsoftware.kryo.Kryo.setInstantiatorStrategy(Lorg/objenesis/strategy/InstantiatorStrategy;)V java.lang.NoSuchMethodError: com.esotericsoftware.kryo.Kryo.setInstantiatorStrategy(Lorg/objenesis/strategy/InstantiatorStrategy;)V at com.twitter.chill.KryoBase.setInstantiatorStrategy(KryoBase.scala:86) at com.twitter.chill.EmptyScalaKryoInstantiator.newKryo(ScalaKryoInstantiator.scala:59) at org.apache.spark.serializer.KryoSerializer.newKryo(KryoSerializer.scala:80) at org.apache.spark.serializer.KryoSerializerInstance.borrowKryo(KryoSerializer.scala:227) at org.apache.spark.serializer.KryoSerializerInstance.<init>(KryoSerializer.scala:212) at org.apache.spark.serializer.KryoSerializer.newInstance(KryoSerializer.scala:128) at org.apache.spark.broadcast.TorrentBroadcast$.blockifyObject(TorrentBroadcast.scala:201) at org.apache.spark.broadcast.TorrentBroadcast.writeBlocks(TorrentBroadcast.scala:102) at org.apache.spark.broadcast.TorrentBroadcast.<init>(TorrentBroadcast.scala:85) at org.apache.spark.broadcast.TorrentBroadcastFactory.newBroadcast(TorrentBroadcastFactory.scala:34) at org.apache.spark.broadcast.BroadcastManager.newBroadcast(BroadcastManager.scala:62) at org.apache.spark.SparkContext.broadcast(SparkContext.scala:1291) at org.apache.spark.rdd.NewHadoopRDD.<init>(NewHadoopRDD.scala:77) at org.apache.spark.SparkContext$$anonfun$newAPIHadoopRDD$1.apply(SparkContext.scala:1099) at org.apache.spark.SparkContext$$anonfun$newAPIHadoopRDD$1.apply(SparkContext.scala:1094) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:147) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:108) at org.apache.spark.SparkContext.withScope(SparkContext.scala:681) at org.apache.spark.SparkContext.newAPIHadoopRDD(SparkContext.scala:1094) at org.apache.pig.backend.hadoop.executionengine.spark.converter.LoadConverter.convert(LoadConverter.java:88) at org.apache.pig.backend.hadoop.executionengine.spark.converter.LoadConverter.convert(LoadConverter.java:58) at org.apache.pig.backend.hadoop.executionengine.spark.SparkLauncher.physicalToRDD(SparkLauncher.java:636) at org.apache.pig.backend.hadoop.executionengine.spark.SparkLauncher.physicalToRDD(SparkLauncher.java:603) at org.apache.pig.backend.hadoop.executionengine.spark.SparkLauncher.physicalToRDD(SparkLauncher.java:603) at org.apache.pig.backend.hadoop.executionengine.spark.SparkLauncher.physicalToRDD(SparkLauncher.java:603) at org.apache.pig.backend.hadoop.executionengine.spark.SparkLauncher.physicalToRDD(SparkLauncher.java:603) at org.apache.pig.backend.hadoop.executionengine.spark.SparkLauncher.physicalToRDD(SparkLauncher.java:603) at org.apache.pig.backend.hadoop.executionengine.spark.SparkLauncher.sparkOperToRDD(SparkLauncher.java:555) at org.apache.pig.backend.hadoop.executionengine.spark.SparkLauncher.sparkPlanToRDD(SparkLauncher.java:504) at org.apache.pig.backend.hadoop.executionengine.spark.SparkLauncher.launchPig(SparkLauncher.java:206) at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.launchPig(HExecutionEngine.java:301) at org.apache.pig.PigServer.launchPlan(PigServer.java:1390) at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1375) at org.apache.pig.PigServer.execute(PigServer.java:1364) at org.apache.pig.PigServer.executeBatch(PigServer.java:415) at org.apache.pig.PigServer.executeBatch(PigServer.java:398) at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:171) at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:234) at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:205) at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:81) at org.apache.pig.Main.run(Main.java:624) at org.apache.pig.Main.main(Main.java:170) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.RunJar.run(RunJar.java:221) at org.apache.hadoop.util.RunJar.main(RunJar.java:136) ================================================================================ {noformat} Here is the logs from the class-loader: {noformat} [Loaded org.apache.spark.broadcast.TorrentBroadcast$$anonfun$5 from file:/data/d1/home/sriksun/pig/lib/spark/spark-assembly-1.4.1-hadoop2.2.0.jar] [Loaded org.apache.spark.serializer.KryoSerializationStream from file:/data/d1/home/sriksun/pig/lib/spark/spark-assembly-1.4.1-hadoop2.2.0.jar] [Loaded org.apache.spark.serializer.KryoDeserializationStream from file:/data/d1/home/sriksun/pig/lib/spark/spark-assembly-1.4.1-hadoop2.2.0.jar] [Loaded com.twitter.chill.KryoInstantiator from file:/data/d1/home/sriksun/pig/lib/spark/spark-assembly-1.4.1-hadoop2.2.0.jar] [Loaded com.twitter.chill.EmptyScalaKryoInstantiator from file:/data/d1/home/sriksun/pig/lib/spark/spark-assembly-1.4.1-hadoop2.2.0.jar] [Loaded com.twitter.chill.KryoInstantiator$3 from file:/data/d1/home/sriksun/pig/lib/spark/spark-assembly-1.4.1-hadoop2.2.0.jar] [Loaded com.twitter.chill.KryoInstantiator$2 from file:/data/d1/home/sriksun/pig/lib/spark/spark-assembly-1.4.1-hadoop2.2.0.jar] [Loaded com.twitter.chill.KryoInstantiator$1 from file:/data/d1/home/sriksun/pig/lib/spark/spark-assembly-1.4.1-hadoop2.2.0.jar] [Loaded com.twitter.chill.KryoInstantiator$4 from file:/data/d1/home/sriksun/pig/lib/spark/spark-assembly-1.4.1-hadoop2.2.0.jar] [Loaded org.objenesis.strategy.InstantiatorStrategy from file:/data/d1/home/sriksun/pig/lib/spark/spark-assembly-1.4.1-hadoop2.2.0.jar] [Loaded com.esotericsoftware.kryo.KryoException from file:/data/d1/home/sriksun/pig/lib/kryo-2.22.jar] [Loaded com.esotericsoftware.kryo.ClassResolver from file:/data/d1/home/sriksun/pig/lib/kryo-2.22.jar] [Loaded com.esotericsoftware.kryo.StreamFactory from file:/data/d1/home/sriksun/pig/lib/kryo-2.22.jar] [Loaded com.esotericsoftware.kryo.factories.SerializerFactory from file:/data/d1/home/sriksun/pig/lib/kryo-2.22.jar] [Loaded com.esotericsoftware.kryo.serializers.DefaultSerializers$IntSerializer from file:/data/d1/home/sriksun/pig/lib/kryo-2.22.jar] [Loaded com.esotericsoftware.kryo.serializers.DefaultSerializers$StringSerializer from file:/data/d1/home/sriksun/pig/lib/kryo-2.22.jar] [Loaded com.esotericsoftware.kryo.serializers.DefaultSerializers$FloatSerializer from file:/data/d1/home/sriksun/pig/lib/kryo-2.22.jar] [Loaded com.esotericsoftware.kryo.serializers.DefaultSerializers$BooleanSerializer from file:/data/d1/home/sriksun/pig/lib/kryo-2.22.jar] [Loaded com.esotericsoftware.kryo.serializers.DefaultSerializers$ByteSerializer from file:/data/d1/home/sriksun/pig/lib/kryo-2.22.jar] [Loaded com.esotericsoftware.kryo.serializers.DefaultSerializers$CharSerializer from file:/data/d1/home/sriksun/pig/lib/kryo-2.22.jar] [Loaded com.esotericsoftware.kryo.serializers.DefaultSerializers$ShortSerializer from file:/data/d1/home/sriksun/pig/lib/kryo-2.22.jar] [Loaded com.esotericsoftware.kryo.serializers.DefaultSerializers$LongSerializer from file:/data/d1/home/sriksun/pig/lib/kryo-2.22.jar] [Loaded com.esotericsoftware.kryo.serializers.DefaultSerializers$DoubleSerializer from file:/data/d1/home/sriksun/pig/lib/kryo-2.22.jar] [Loaded com.esotericsoftware.kryo.serializers.DefaultSerializers$VoidSerializer from file:/data/d1/home/sriksun/pig/lib/kryo-2.22.jar] [Loaded com.esotericsoftware.kryo.ReferenceResolver from file:/data/d1/home/sriksun/pig/lib/kryo-2.22.jar] [Loaded com.esotericsoftware.shaded.org.objenesis.instantiator.ObjectInstantiator from file:/data/d1/home/sriksun/pig/lib/kryo-2.22.jar] [Loaded com.twitter.chill.ObjectSerializer from file:/data/d1/home/sriksun/pig/lib/spark/spark-assembly-1.4.1-hadoop2.2.0.jar] [Loaded com.esotericsoftware.kryo.serializers.FieldSerializer from file:/data/d1/home/sriksun/pig/lib/kryo-2.22.jar] /hdp/2.2.0.0-2041/hadoop-hdfs/hadoop-hdfs-2.6.0.2.2.0.0-2041.jar] [Loaded com.esotericsoftware.kryo.util.DefaultClassResolver from file:/data/d1/home/sriksun/pig/lib/kryo-2.22.jar] {noformat} org.objenesis.strategy.InstantiatorStrategy is used to set the InstantiatorStrategy in Kryo, where Kryo in 2.22 requires shaded com.esotericsoftware.shaded.org.objenesis.strategy.InstantiatorStrategy.class and hence the error. I guess Kryo is pulled in as a dependency for OrcStorage. > Class conflicts: Kryo bundled in spark vs kryo bundled with pig > --------------------------------------------------------------- > > Key: PIG-4693 > URL: https://issues.apache.org/jira/browse/PIG-4693 > Project: Pig > Issue Type: Sub-task > Components: spark > Affects Versions: spark-branch > Reporter: Srikanth Sundarrajan > Assignee: Srikanth Sundarrajan > Labels: spork > Fix For: spark-branch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)