Getting exception on JavaSchemaRDD; org.apache.spark.SparkException: Task not serializable

2014-11-22 Thread vdiwakar.malladi
Hi I'm trying to load the parquet file for querying purpose from my web application. I could able to load it as JavaSchemaRDD. But at the time of using map function on the JavaSchemaRDD, I'm getting the following exception. The class in which I'm using this code implements Serializable class.

Re: Getting exception on JavaSchemaRDD; org.apache.spark.SparkException: Task not serializable

2014-11-22 Thread Akhil Das
You could be referring/sending the StandardSessionFacade inside your map function. You could bring the class StandardSessionFacade locally and Serialize it to get it fixed quickly. Thanks Best Regards On Sat, Nov 22, 2014 at 10:02 PM, vdiwakar.malladi vdiwakar.mall...@gmail.com wrote: Hi

Re: Getting exception on JavaSchemaRDD; org.apache.spark.SparkException: Task not serializable

2014-11-22 Thread vdiwakar.malladi
Thanks for your prompt response. I'm not using any thing in my map function. please see the below code. For sample purpose, I would like to using 'select * from '. This code worked for me in standalone mode. But when I integrated with my web application, it is throwing the specified exception.

Re: Getting exception on JavaSchemaRDD; org.apache.spark.SparkException: Task not serializable

2014-11-22 Thread Sean Owen
You are declaring an anonymous inner class here. It has a reference to the containing class even if you don't use it. If the closure cleaner can't determine it isn't used, this reference will cause everything in the outer class to serialize. Try rewriting this as a named static inner class . On

Re: Getting exception on JavaSchemaRDD; org.apache.spark.SparkException: Task not serializable

2014-11-22 Thread vdiwakar.malladi
Thanks. After writing it as static inner class, that exception not coming. But getting snappy related exception. I could see the corresponding dependency is in the spark assembly jar. Still getting the exception. Any quick suggestion on this? Here is the stack trace.