[
https://issues.apache.org/jira/browse/BEAM-2027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15979797#comment-15979797
]
Aviem Zur edited comment on BEAM-2027 at 4/22/17 7:22 AM:
----------------------------------------------------------
{{local class incompatible}} happens when trying to deserialize a Java class of
an older version than the one in your classpath.
The class in question is
{{org.apache.beam.runners.spark.coders.CoderHelpers$3}} which is a part of Beam.
In believe in your application it could be the case that the version of Beam in
the driver's classpath is different than the one in the executor's classpath.
Spark is serializing an instance of this class in the driver at one version and
deserializing it in an executor which expects a different version.
A few questions about your setup:
Did you package your application's jar as a fat jar containing Beam jars?
If not: do you distribute Beam jars to your executors in some other way?
Is this a streaming application? (from your description it didn't sound like
you used streaming).
If so, Spark streaming checkpoints certain serialized instances of classes.
When changing these in your application and resuming from checkpoint (for
example, upgrading Beam version) you can experience failures such as this. The
solution for this is deleting the checkpoint and starting fresh.
was (Author: aviemzur):
{{local class incompatible}} happens when trying to deserialize a Java class of
an older version than the one in your classpath.
The class in question is
{{org.apache.beam.runners.spark.coders.CoderHelpers$3}} which is a part of Beam.
In believe in your case the version of Beam in the driver's classpath is
different than the one in the executor's classpath.
Spark is serializing an instance of this class in the driver at one version and
deserializing it in an executor which expects a different version.
A few questions about your setup:
Did you package your application's jar as a fat jar containing Beam jars?
If not: do you distribute Beam jars to your executors in some other way?
Is this a streaming application? (from your description it didn't sound like
you used streaming).
If so, Spark streaming checkpoints certain serialized instances of classes.
When changing these in your application and resuming from checkpoint (for
example, upgrading Beam version) you can experience failures such as this. The
solution for this is deleting the checkpoint and starting fresh.
> get error sometimes while running the same code using beam0.6
> -------------------------------------------------------------
>
> Key: BEAM-2027
> URL: https://issues.apache.org/jira/browse/BEAM-2027
> Project: Beam
> Issue Type: Bug
> Components: runner-core, runner-spark
> Environment: spark-1.6.2-bin-hadoop2.6, hadoop-2.6.0, source:hdfs
> sink:hdfs
> Reporter: liyuntian
> Assignee: Aviem Zur
>
> run a yarn job using beam0.6.0, I get file from hdfs and write record to
> hdfs, I use spark-1.6.2-bin-hadoop2.6,hadoop-2.6.0. I get error sometime
> below,
> 17/04/20 21:10:45 WARN scheduler.TaskSetManager: Lost task 0.0 in stage 0.0
> (TID 0, etl-develop-003): java.io.InvalidClassException:
> org.apache.beam.runners.spark.coders.CoderHelpers$3; local class
> incompatible: stream classdesc serialVersionUID = 1334222146820528045, local
> class serialVersionUID = 5119956493581628999
> at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:617)
> at
> java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1622)
> at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1517)
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1771)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
> at
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
> at
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
> at
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
> at
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
> at scala.collection.immutable.$colon$colon.readObject(List.scala:362)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017)
> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893)
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
> at
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
> at
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
> at scala.collection.immutable.$colon$colon.readObject(List.scala:362)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017)
> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893)
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
> at
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
> at
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
> at
> org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:76)
> at
> org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:115)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
> at org.apache.spark.scheduler.Task.run(Task.scala:89)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:227)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> 17/04/20 21:10:45 INFO scheduler.TaskSetManager: Starting task 0.1 in stage
> 0.0 (TID 2, etl-develop-003, partition 0,PROCESS_LOCAL, 2080 bytes)
> 17/04/20 21:10:45 INFO scheduler.TaskSetManager: Lost task 0.1 in stage 0.0
> (TID 2) on executor etl-develop-003: java.io.InvalidClassException
> (org.apache.beam.runners.spark.coders.CoderHelpers$3; local class
> incompatible: stream classdesc serialVersionUID = 1334222146820528045, local
> class serialVersionUID = 5119956493581628999) [duplicate 1]
> 17/04/20 21:10:45 INFO scheduler.TaskSetManager: Starting task 0.2 in stage
> 0.0 (TID 3, etl-develop-003, partition 0,PROCESS_LOCAL, 2080 bytes)
> 17/04/20 21:10:45 INFO scheduler.TaskSetManager: Lost task 0.2 in stage 0.0
> (TID 3) on executor etl-develop-003: java.io.InvalidClassException
> (org.apache.beam.runners.spark.coders.CoderHelpers$3; local class
> incompatible: stream classdesc serialVersionUID = 1334222146820528045, local
> class serialVersionUID = 5119956493581628999) [duplicate 2]
> 17/04/20 21:10:45 INFO scheduler.TaskSetManager: Starting task 0.3 in stage
> 0.0 (TID 4, etl-develop-003, partition 0,PROCESS_LOCAL, 2080 bytes)
> 17/04/20 21:10:45 INFO scheduler.TaskSetManager: Lost task 0.3 in stage 0.0
> (TID 4) on executor etl-develop-003: java.io.InvalidClassException
> (org.apache.beam.runners.spark.coders.CoderHelpers$3; local class
> incompatible: stream classdesc serialVersionUID = 1334222146820528045, local
> class serialVersionUID = 5119956493581628999) [duplicate 3]
> 17/04/20 21:10:45 ERROR scheduler.TaskSetManager: Task 0 in stage 0.0 failed
> 4 times; aborting job
> 17/04/20 21:10:45 INFO cluster.YarnClusterScheduler: Cancelling stage 0
> 17/04/20 21:10:45 INFO cluster.YarnClusterScheduler: Stage 0 was cancelled
> 17/04/20 21:10:45 INFO scheduler.DAGScheduler: ResultStage 0 (collect at
> BoundedDataset.java:88) failed in 1.260 s
> 17/04/20 21:10:45 INFO scheduler.DAGScheduler: Job 0 failed: collect at
> BoundedDataset.java:88, took 1.424075 s
> 17/04/20 21:10:46 ERROR yarn.ApplicationMaster: User class threw exception:
> org.apache.beam.sdk.Pipeline$PipelineExecutionException:
> java.io.InvalidClassException:
> org.apache.beam.runners.spark.coders.CoderHelpers$3; local class
> incompatible: stream classdesc serialVersionUID = 1334222146820528045, local
> class serialVersionUID = 5119956493581628999
> org.apache.beam.sdk.Pipeline$PipelineExecutionException:
> java.io.InvalidClassException:
> org.apache.beam.runners.spark.coders.CoderHelpers$3; local class
> incompatible: stream classdesc serialVersionUID = 1334222146820528045, local
> class serialVersionUID = 5119956493581628999
> at
> org.apache.beam.runners.spark.SparkPipelineResult.beamExceptionFrom(SparkPipelineResult.java:73)
> at
> org.apache.beam.runners.spark.SparkPipelineResult.waitUntilFinish(SparkPipelineResult.java:113)
> at
> org.apache.beam.runners.spark.SparkPipelineResult.waitUntilFinish(SparkPipelineResult.java:102)
> at
> com.chinamobile.cmss.etl.beam.transform.server.SparkMain.main(SparkMain.java:96)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:558)
> Caused by: java.io.InvalidClassException:
> org.apache.beam.runners.spark.coders.CoderHelpers$3; local class
> incompatible: stream classdesc serialVersionUID = 1334222146820528045, local
> class serialVersionUID = 5119956493581628999
> at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:617)
> at
> java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1622)
> at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1517)
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1771)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
