Github user ch0ice commented on the issue:
https://github.com/apache/spark/pull/181
This problem arose again for me, and I reproduced it when I converted byte
into protoBuf after redis checked the data.
The following code in the deserialization (Utils deserialize (value, Utils
getContextOrSparkClassLoader ())) when there is an error
Map<byte[],byte[]> hResult =
jedisUtil.hgetAll((REDIS_LASTPOINT_PREFIX + serviceId));
hResult.forEach((key,value)->{
LocationPackage.LocationMessage lastLocationPackage =
Utils.deserialize(value,Utils.getContextOrSparkClassLoader());
if(lastLocationPackage.getLocation().getLocationTime() >=
sTime && lastLocationPackage.getLocation().getLocationTime() <= eTime){
String gpsRedisKey = REDIS_GPSLIST_PREFIX + serviceId +
"_" + lastLocationPackage.getLocation().getTerminalId();
redisKeys.add(gpsRedisKey);
}
});
ERROR:
java.lang.RuntimeException: Unable to find proto buffer class
at
com.google.protobuf.GeneratedMessageLite$SerializedForm.readResolve(GeneratedMessageLite.java:775)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at
java.io.ObjectStreamClass.invokeReadResolve(ObjectStreamClass.java:1104)
at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1810)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371)
at org.apache.spark.util.Utils$.deserialize(Utils.scala:161)
at org.apache.spark.util.Utils.deserialize(Utils.scala)
at
com.location.platform.analysis.util.AnalysisUtil.lambda$getRedisData$0(AnalysisUtil.java:310)
at
com.location.platform.analysis.util.AnalysisUtil$$Lambda$17/489334328.accept(Unknown
Source)
at java.util.Map.forEach(Map.java:630)
at
com.location.platform.analysis.util.AnalysisUtil.getRedisData(AnalysisUtil.java:307)
at
com.location.platform.analysis.util.AnalysisUtil.getMongoAndRedisDataset(AnalysisUtil.java:372)
at
com.location.platform.analysis.bi.TodayOdometerAnalysis.lambda$main$0(TodayOdometerAnalysis.java:96)
at
com.location.platform.analysis.bi.TodayOdometerAnalysis$$Lambda$1/1413730361.accept(Unknown
Source)
at java.util.ArrayList.forEach(ArrayList.java:1249)
at
com.location.platform.analysis.bi.TodayOdometerAnalysis.main(TodayOdometerAnalysis.java:76)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at
org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:775)
at
org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:119)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException:
com.location.platform.common.protobuf.LocationPackage$LocationMessage
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:264)
at
com.google.protobuf.GeneratedMessageLite$SerializedForm.readResolve(GeneratedMessageLite.java:768)
... 28 more
Only if there is a problem with the cluster, local mode is fine.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
