[
https://issues.apache.org/jira/browse/SPARK-5506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14299743#comment-14299743
]
Milad Khajavi commented on SPARK-5506:
--------------------------------------
The problem occurs with any framework that use Servlet.
> java.lang.ClassCastException using lambda expressions in combination of spark
> and Servlet
> -----------------------------------------------------------------------------------------
>
> Key: SPARK-5506
> URL: https://issues.apache.org/jira/browse/SPARK-5506
> Project: Spark
> Issue Type: Question
> Components: Spark Core
> Affects Versions: 1.2.0
> Environment: spark server: Ubuntu 14.04 amd64
> $ java -version
> java version "1.8.0_25"
> Java(TM) SE Runtime Environment (build 1.8.0_25-b17)
> Java HotSpot(TM) 64-Bit Server VM (build 25.25-b02, mixed mode)
> Reporter: Milad Khajavi
>
> I'm trying to build a web API for my Apache spark jobs using sparkjava.com
> framework. My code is:
> @Override
> public void init() {
> get("/hello",
> (req, res) -> {
> String sourcePath = "hdfs://spark:54310/input/*";
> SparkConf conf = new SparkConf().setAppName("LineCount");
> conf.setJars(new String[] {
> "/home/sam/resin-4.0.42/webapps/test.war" });
> File configFile = new File("config.properties");
> String sparkURI = "spark://hamrah:7077";
> conf.setMaster(sparkURI);
> conf.set("spark.driver.allowMultipleContexts", "true");
> JavaSparkContext sc = new JavaSparkContext(conf);
> @SuppressWarnings("resource")
> JavaRDD<String> log = sc.textFile(sourcePath);
> JavaRDD<String> lines = log.filter(x -> {
> return true;
> });
> return lines.count();
> });
> }
> If I remove the lambda expression or put it inside a simple jar rather than a
> web service (somehow a Servlet) it will run without any error. But using a
> lambda expression inside a Servlet will result this exception:
> 15/01/28 10:36:33 WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0,
> hamrah): java.lang.ClassCastException: cannot assign instance of
> java.lang.invoke.SerializedLambda to field
> org.apache.spark.api.java.JavaRDD$$anonfun$filter$1.f$1 of type
> org.apache.spark.api.java.function.Function in instance of
> org.apache.spark.api.java.JavaRDD$$anonfun$filter$1
> at
> java.io.ObjectStreamClass$FieldReflector.setObjFieldValues(ObjectStreamClass.java:2089)
> at java.io.ObjectStreamClass.setObjFieldValues(ObjectStreamClass.java:1261)
> at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1999)
> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918)
> at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
> at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993)
> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918)
> at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
> at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993)
> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918)
> at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371)
> at
> org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:62)
> at
> org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:87)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:57)
> at org.apache.spark.scheduler.Task.run(Task.scala:56)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:196)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> P.S: I tried combination of jersey and javaspark with jetty, tomcat and resin
> and all of them led me to the same result.
> Here the same issue:
> http://apache-spark-user-list.1001560.n3.nabble.com/Spark-on-YARN-java-lang-ClassCastException-SerializedLambda-to-org-apache-spark-api-java-function-Fu1-tt21261.html
> This is my colleague question in stackoverflow:
> http://stackoverflow.com/questions/28186607/java-lang-classcastexception-using-lambda-expressions-in-spark-job-on-remote-ser
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
