Re: Spark Job Failed - Class not serializable

2015-04-03 Thread Frank Austin Nothaft 
You’ll definitely want to use a Kryo-based serializer for Avro. We have a Kryo 
based serializer that wraps the Avro efficient serializer here.

Frank Austin Nothaft
fnoth...@berkeley.edu
fnoth...@eecs.berkeley.edu
202-340-0466

On Apr 3, 2015, at 5:41 AM, Akhil Das  wrote:

> Because, its throwing up serializable exceptions and kryo is a serializer to 
> serialize your objects.
> 
> Thanks
> Best Regards
> 
> On Fri, Apr 3, 2015 at 5:37 PM, Deepak Jain  wrote:
> I meant that I did not have to use kyro. Why will kyro help fix this issue 
> now ?
> 
> Sent from my iPhone
> 
> On 03-Apr-2015, at 5:36 pm, Deepak Jain  wrote:
> 
>> I was able to write record that extends specificrecord (avro) this class was 
>> not auto generated. Do we need to do something extra for auto generated 
>> classes 
>> 
>> Sent from my iPhone
>> 
>> On 03-Apr-2015, at 5:06 pm, Akhil Das  wrote:
>> 
>>> This thread might give you some insights 
>>> http://mail-archives.apache.org/mod_mbox/incubator-spark-user/201311.mbox/%3CCA+WVT8WXbEHac=N0GWxj-s9gqOkgG0VRL5B=ovjwexqm8ev...@mail.gmail.com%3E
>>> 
>>> Thanks
>>> Best Regards
>>> 
>>> On Fri, Apr 3, 2015 at 3:53 PM, ÐΞ€ρ@Ҝ (๏̯͡๏)  wrote:
>>> My Spark Job failed with
>>> 
>>> 
>>> 15/04/03 03:15:36 INFO scheduler.DAGScheduler: Job 0 failed: 
>>> saveAsNewAPIHadoopFile at AbstractInputHelper.scala:103, took 2.480175 s
>>> 15/04/03 03:15:36 ERROR yarn.ApplicationMaster: User class threw exception: 
>>> Job aborted due to stage failure: Task 0.0 in stage 2.0 (TID 0) had a not 
>>> serializable result: 
>>> com.ebay.ep.poc.spark.reporting.process.model.dw.SpsLevelMetricSum
>>> Serialization stack:
>>> - object not serializable (class: 
>>> com.ebay.ep.poc.spark.reporting.process.model.dw.SpsLevelMetricSum, value: 
>>> {"userId": 0, "spsPrgrmId": 0, "spsSlrLevelCd": 0, "spsSlrLevelSumStartDt": 
>>> null, "spsSlrLevelSumEndDt": null, "currPsLvlId": null})
>>> - field (class: scala.Tuple2, name: _2, type: class java.lang.Object)
>>> - object (class scala.Tuple2, (0,{"userId": 0, "spsPrgrmId": 0, 
>>> "spsSlrLevelCd": 0, "spsSlrLevelSumStartDt": null, "spsSlrLevelSumEndDt": 
>>> null, "currPsLvlId": null}))
>>> org.apache.spark.SparkException: Job aborted due to stage failure: Task 0.0 
>>> in stage 2.0 (TID 0) had a not serializable result: 
>>> com.ebay.ep.poc.spark.reporting.process.model.dw.SpsLevelMetricSum
>>> Serialization stack:
>>> - object not serializable (class: 
>>> com.ebay.ep.poc.spark.reporting.process.model.dw.SpsLevelMetricSum, value: 
>>> {"userId": 0, "spsPrgrmId": 0, "spsSlrLevelCd": 0, "spsSlrLevelSumStartDt": 
>>> null, "spsSlrLevelSumEndDt": null, "currPsLvlId": null})
>>> - field (class: scala.Tuple2, name: _2, type: class java.lang.Object)
>>> - object (class scala.Tuple2, (0,{"userId": 0, "spsPrgrmId": 0, 
>>> "spsSlrLevelCd": 0, "spsSlrLevelSumStartDt": null, "spsSlrLevelSumEndDt": 
>>> null, "currPsLvlId": null}))
>>> at 
>>> org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1203)
>>> at 
>>> org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1192)
>>> at 
>>> org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1191)
>>> at 
>>> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
>>> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
>>> at 
>>> org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1191)
>>> at 
>>> org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:693)
>>> at 
>>> org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:693)
>>> at scala.Option.foreach(Option.scala:236)
>>> at 
>>> org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:693)
>>> at 
>>> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1393)
>>> at 
>>> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1354)
>>> at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
>>> 15/04/03 03:15:36 INFO yarn.ApplicationMaster: Final app status: FAILED, 
>>> exitCode: 15, (reason: User class threw exception: Job aborted due to stage 
>>> failure: Task 0.0 in stage 2.0 (TID 0) had a not serializable result: 
>>> com.ebay.ep.poc.spark.reporting.process.model.dw.SpsLevelMetricSum
>>> Serialization stack:
>>> 
>>> 
>>> 
>>> com.ebay.ep.poc.spark.reporting.process.model.dw.SpsLevelMetricSum is auto 
>>> generated through avro schema using avro-generate-sources maven pulgin.
>>> 
>>> 
>>> package com.ebay.ep.poc.spark.reporting.process.model.dw;  
>>> 
>>> @SuppressWarnings("all")
>>> 
>>> @org.apache.avro.specific.AvroGenerated
>>> 
>>> public class SpsLevelMetricSum extends 
>>> org.apache.avro.specific.SpecificRecordBase i

Re: Spark Job Failed - Class not serializable

2015-04-03 Thread Akhil Das 
Because, its throwing up serializable exceptions and kryo is a serializer
to serialize your objects.

Thanks
Best Regards

On Fri, Apr 3, 2015 at 5:37 PM, Deepak Jain  wrote:

> I meant that I did not have to use kyro. Why will kyro help fix this issue
> now ?
>
> Sent from my iPhone
>
> On 03-Apr-2015, at 5:36 pm, Deepak Jain  wrote:
>
> I was able to write record that extends specificrecord (avro) this class
> was not auto generated. Do we need to do something extra for auto generated
> classes
>
> Sent from my iPhone
>
> On 03-Apr-2015, at 5:06 pm, Akhil Das  wrote:
>
> This thread might give you some insights
> http://mail-archives.apache.org/mod_mbox/incubator-spark-user/201311.mbox/%3CCA+WVT8WXbEHac=N0GWxj-s9gqOkgG0VRL5B=ovjwexqm8ev...@mail.gmail.com%3E
>
> Thanks
> Best Regards
>
> On Fri, Apr 3, 2015 at 3:53 PM, ÐΞ€ρ@Ҝ (๏̯͡๏)  wrote:
>
>> My Spark Job failed with
>>
>>
>> 15/04/03 03:15:36 INFO scheduler.DAGScheduler: Job 0 failed:
>> saveAsNewAPIHadoopFile at AbstractInputHelper.scala:103, took 2.480175 s
>> 15/04/03 03:15:36 ERROR yarn.ApplicationMaster: User class threw
>> exception: Job aborted due to stage failure: Task 0.0 in stage 2.0 (TID 0)
>> had a not serializable result:
>> com.ebay.ep.poc.spark.reporting.process.model.dw.SpsLevelMetricSum
>> Serialization stack:
>> - object not serializable (class:
>> com.ebay.ep.poc.spark.reporting.process.model.dw.SpsLevelMetricSum, value:
>> {"userId": 0, "spsPrgrmId": 0, "spsSlrLevelCd": 0, "spsSlrLevelSumStartDt":
>> null, "spsSlrLevelSumEndDt": null, "currPsLvlId": null})
>> - field (class: scala.Tuple2, name: _2, type: class java.lang.Object)
>> - object (class scala.Tuple2, (0,{"userId": 0, "spsPrgrmId": 0,
>> "spsSlrLevelCd": 0, "spsSlrLevelSumStartDt": null, "spsSlrLevelSumEndDt":
>> null, "currPsLvlId": null}))
>> org.apache.spark.SparkException: Job aborted due to stage failure: Task
>> 0.0 in stage 2.0 (TID 0) had a not serializable result:
>> com.ebay.ep.poc.spark.reporting.process.model.dw.SpsLevelMetricSum
>> Serialization stack:
>> - object not serializable (class:
>> com.ebay.ep.poc.spark.reporting.process.model.dw.SpsLevelMetricSum, value:
>> {"userId": 0, "spsPrgrmId": 0, "spsSlrLevelCd": 0, "spsSlrLevelSumStartDt":
>> null, "spsSlrLevelSumEndDt": null, "currPsLvlId": null})
>> - field (class: scala.Tuple2, name: _2, type: class java.lang.Object)
>> - object (class scala.Tuple2, (0,{"userId": 0, "spsPrgrmId": 0,
>> "spsSlrLevelCd": 0, "spsSlrLevelSumStartDt": null, "spsSlrLevelSumEndDt":
>> null, "currPsLvlId": null}))
>> at org.apache.spark.scheduler.DAGScheduler.org
>> $apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1203)
>> at
>> org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1192)
>> at
>> org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1191)
>> at
>> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
>> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
>> at
>> org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1191)
>> at
>> org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:693)
>> at
>> org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:693)
>> at scala.Option.foreach(Option.scala:236)
>> at
>> org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:693)
>> at
>> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1393)
>> at
>> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1354)
>> at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
>> 15/04/03 03:15:36 INFO yarn.ApplicationMaster: Final app status: FAILED,
>> exitCode: 15, (reason: User class threw exception: Job aborted due to stage
>> failure: Task 0.0 in stage 2.0 (TID 0) had a not serializable result:
>> com.ebay.ep.poc.spark.reporting.process.model.dw.SpsLevelMetricSum
>> Serialization stack:
>>
>> 
>>
>> com.ebay.ep.poc.spark.reporting.process.model.dw.SpsLevelMetricSum is
>> auto generated through avro schema using avro-generate-sources maven pulgin.
>>
>>
>> package com.ebay.ep.poc.spark.reporting.process.model.dw;
>>
>> @SuppressWarnings("all")
>>
>> @org.apache.avro.specific.AvroGenerated
>>
>> public class SpsLevelMetricSum extends
>> org.apache.avro.specific.SpecificRecordBase implements
>> org.apache.avro.specific.SpecificRecord {
>> ...
>> ...
>> }
>>
>> Can anyone suggest how to fix this ?
>>
>>
>>
>> --
>> Deepak
>>
>>
>

Re: Spark Job Failed - Class not serializable

2015-04-03 Thread Deepak Jain 
I meant that I did not have to use kyro. Why will kyro help fix this issue now ?

Sent from my iPhone

> On 03-Apr-2015, at 5:36 pm, Deepak Jain  wrote:
> 
> I was able to write record that extends specificrecord (avro) this class was 
> not auto generated. Do we need to do something extra for auto generated 
> classes 
> 
> Sent from my iPhone
> 
>> On 03-Apr-2015, at 5:06 pm, Akhil Das  wrote:
>> 
>> This thread might give you some insights 
>> http://mail-archives.apache.org/mod_mbox/incubator-spark-user/201311.mbox/%3CCA+WVT8WXbEHac=N0GWxj-s9gqOkgG0VRL5B=ovjwexqm8ev...@mail.gmail.com%3E
>> 
>> Thanks
>> Best Regards
>> 
>>> On Fri, Apr 3, 2015 at 3:53 PM, ÐΞ€ρ@Ҝ (๏̯͡๏)  wrote:
>>> My Spark Job failed with
>>> 
>>> 
>>> 15/04/03 03:15:36 INFO scheduler.DAGScheduler: Job 0 failed: 
>>> saveAsNewAPIHadoopFile at AbstractInputHelper.scala:103, took 2.480175 s
>>> 15/04/03 03:15:36 ERROR yarn.ApplicationMaster: User class threw exception: 
>>> Job aborted due to stage failure: Task 0.0 in stage 2.0 (TID 0) had a not 
>>> serializable result: 
>>> com.ebay.ep.poc.spark.reporting.process.model.dw.SpsLevelMetricSum
>>> Serialization stack:
>>> - object not serializable (class: 
>>> com.ebay.ep.poc.spark.reporting.process.model.dw.SpsLevelMetricSum, value: 
>>> {"userId": 0, "spsPrgrmId": 0, "spsSlrLevelCd": 0, "spsSlrLevelSumStartDt": 
>>> null, "spsSlrLevelSumEndDt": null, "currPsLvlId": null})
>>> - field (class: scala.Tuple2, name: _2, type: class java.lang.Object)
>>> - object (class scala.Tuple2, (0,{"userId": 0, "spsPrgrmId": 0, 
>>> "spsSlrLevelCd": 0, "spsSlrLevelSumStartDt": null, "spsSlrLevelSumEndDt": 
>>> null, "currPsLvlId": null}))
>>> org.apache.spark.SparkException: Job aborted due to stage failure: Task 0.0 
>>> in stage 2.0 (TID 0) had a not serializable result: 
>>> com.ebay.ep.poc.spark.reporting.process.model.dw.SpsLevelMetricSum
>>> Serialization stack:
>>> - object not serializable (class: 
>>> com.ebay.ep.poc.spark.reporting.process.model.dw.SpsLevelMetricSum, value: 
>>> {"userId": 0, "spsPrgrmId": 0, "spsSlrLevelCd": 0, "spsSlrLevelSumStartDt": 
>>> null, "spsSlrLevelSumEndDt": null, "currPsLvlId": null})
>>> - field (class: scala.Tuple2, name: _2, type: class java.lang.Object)
>>> - object (class scala.Tuple2, (0,{"userId": 0, "spsPrgrmId": 0, 
>>> "spsSlrLevelCd": 0, "spsSlrLevelSumStartDt": null, "spsSlrLevelSumEndDt": 
>>> null, "currPsLvlId": null}))
>>> at 
>>> org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1203)
>>> at 
>>> org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1192)
>>> at 
>>> org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1191)
>>> at 
>>> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
>>> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
>>> at 
>>> org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1191)
>>> at 
>>> org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:693)
>>> at 
>>> org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:693)
>>> at scala.Option.foreach(Option.scala:236)
>>> at 
>>> org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:693)
>>> at 
>>> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1393)
>>> at 
>>> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1354)
>>> at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
>>> 15/04/03 03:15:36 INFO yarn.ApplicationMaster: Final app status: FAILED, 
>>> exitCode: 15, (reason: User class threw exception: Job aborted due to stage 
>>> failure: Task 0.0 in stage 2.0 (TID 0) had a not serializable result: 
>>> com.ebay.ep.poc.spark.reporting.process.model.dw.SpsLevelMetricSum
>>> Serialization stack:
>>> 
>>> 
>>> 
>>> com.ebay.ep.poc.spark.reporting.process.model.dw.SpsLevelMetricSum is auto 
>>> generated through avro schema using avro-generate-sources maven pulgin.
>>> 
>>> 
>>> package com.ebay.ep.poc.spark.reporting.process.model.dw;  
>>> 
>>> @SuppressWarnings("all")
>>> 
>>> @org.apache.avro.specific.AvroGenerated
>>> 
>>> public class SpsLevelMetricSum extends 
>>> org.apache.avro.specific.SpecificRecordBase implements 
>>> org.apache.avro.specific.SpecificRecord {
>>> 
>>> ...
>>> ...
>>> }
>>> 
>>> Can anyone suggest how to fix this ?
>>> 
>>> 
>>> 
>>> -- 
>>> Deepak
>>

Re: Spark Job Failed - Class not serializable

2015-04-03 Thread Deepak Jain 
I was able to write record that extends specificrecord (avro) this class was 
not auto generated. Do we need to do something extra for auto generated classes 

Sent from my iPhone

> On 03-Apr-2015, at 5:06 pm, Akhil Das  wrote:
> 
> This thread might give you some insights 
> http://mail-archives.apache.org/mod_mbox/incubator-spark-user/201311.mbox/%3CCA+WVT8WXbEHac=N0GWxj-s9gqOkgG0VRL5B=ovjwexqm8ev...@mail.gmail.com%3E
> 
> Thanks
> Best Regards
> 
>> On Fri, Apr 3, 2015 at 3:53 PM, ÐΞ€ρ@Ҝ (๏̯͡๏)  wrote:
>> My Spark Job failed with
>> 
>> 
>> 15/04/03 03:15:36 INFO scheduler.DAGScheduler: Job 0 failed: 
>> saveAsNewAPIHadoopFile at AbstractInputHelper.scala:103, took 2.480175 s
>> 15/04/03 03:15:36 ERROR yarn.ApplicationMaster: User class threw exception: 
>> Job aborted due to stage failure: Task 0.0 in stage 2.0 (TID 0) had a not 
>> serializable result: 
>> com.ebay.ep.poc.spark.reporting.process.model.dw.SpsLevelMetricSum
>> Serialization stack:
>>  - object not serializable (class: 
>> com.ebay.ep.poc.spark.reporting.process.model.dw.SpsLevelMetricSum, value: 
>> {"userId": 0, "spsPrgrmId": 0, "spsSlrLevelCd": 0, "spsSlrLevelSumStartDt": 
>> null, "spsSlrLevelSumEndDt": null, "currPsLvlId": null})
>>  - field (class: scala.Tuple2, name: _2, type: class java.lang.Object)
>>  - object (class scala.Tuple2, (0,{"userId": 0, "spsPrgrmId": 0, 
>> "spsSlrLevelCd": 0, "spsSlrLevelSumStartDt": null, "spsSlrLevelSumEndDt": 
>> null, "currPsLvlId": null}))
>> org.apache.spark.SparkException: Job aborted due to stage failure: Task 0.0 
>> in stage 2.0 (TID 0) had a not serializable result: 
>> com.ebay.ep.poc.spark.reporting.process.model.dw.SpsLevelMetricSum
>> Serialization stack:
>>  - object not serializable (class: 
>> com.ebay.ep.poc.spark.reporting.process.model.dw.SpsLevelMetricSum, value: 
>> {"userId": 0, "spsPrgrmId": 0, "spsSlrLevelCd": 0, "spsSlrLevelSumStartDt": 
>> null, "spsSlrLevelSumEndDt": null, "currPsLvlId": null})
>>  - field (class: scala.Tuple2, name: _2, type: class java.lang.Object)
>>  - object (class scala.Tuple2, (0,{"userId": 0, "spsPrgrmId": 0, 
>> "spsSlrLevelCd": 0, "spsSlrLevelSumStartDt": null, "spsSlrLevelSumEndDt": 
>> null, "currPsLvlId": null}))
>>  at 
>> org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1203)
>>  at 
>> org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1192)
>>  at 
>> org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1191)
>>  at 
>> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
>>  at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
>>  at 
>> org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1191)
>>  at 
>> org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:693)
>>  at 
>> org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:693)
>>  at scala.Option.foreach(Option.scala:236)
>>  at 
>> org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:693)
>>  at 
>> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1393)
>>  at 
>> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1354)
>>  at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
>> 15/04/03 03:15:36 INFO yarn.ApplicationMaster: Final app status: FAILED, 
>> exitCode: 15, (reason: User class threw exception: Job aborted due to stage 
>> failure: Task 0.0 in stage 2.0 (TID 0) had a not serializable result: 
>> com.ebay.ep.poc.spark.reporting.process.model.dw.SpsLevelMetricSum
>> Serialization stack:
>> 
>> 
>> 
>> com.ebay.ep.poc.spark.reporting.process.model.dw.SpsLevelMetricSum is auto 
>> generated through avro schema using avro-generate-sources maven pulgin.
>> 
>> 
>> package com.ebay.ep.poc.spark.reporting.process.model.dw;  
>> 
>> @SuppressWarnings("all")
>> 
>> @org.apache.avro.specific.AvroGenerated
>> 
>> public class SpsLevelMetricSum extends 
>> org.apache.avro.specific.SpecificRecordBase implements 
>> org.apache.avro.specific.SpecificRecord {
>> 
>> ...
>> ...
>> }
>> 
>> Can anyone suggest how to fix this ?
>> 
>> 
>> 
>> -- 
>> Deepak
>

Re: Spark Job Failed - Class not serializable

2015-04-03 Thread Akhil Das 
This thread might give you some insights
http://mail-archives.apache.org/mod_mbox/incubator-spark-user/201311.mbox/%3CCA+WVT8WXbEHac=N0GWxj-s9gqOkgG0VRL5B=ovjwexqm8ev...@mail.gmail.com%3E

Thanks
Best Regards

On Fri, Apr 3, 2015 at 3:53 PM, ÐΞ€ρ@Ҝ (๏̯͡๏)  wrote:

> My Spark Job failed with
>
>
> 15/04/03 03:15:36 INFO scheduler.DAGScheduler: Job 0 failed:
> saveAsNewAPIHadoopFile at AbstractInputHelper.scala:103, took 2.480175 s
> 15/04/03 03:15:36 ERROR yarn.ApplicationMaster: User class threw
> exception: Job aborted due to stage failure: Task 0.0 in stage 2.0 (TID 0)
> had a not serializable result:
> com.ebay.ep.poc.spark.reporting.process.model.dw.SpsLevelMetricSum
> Serialization stack:
> - object not serializable (class:
> com.ebay.ep.poc.spark.reporting.process.model.dw.SpsLevelMetricSum, value:
> {"userId": 0, "spsPrgrmId": 0, "spsSlrLevelCd": 0, "spsSlrLevelSumStartDt":
> null, "spsSlrLevelSumEndDt": null, "currPsLvlId": null})
> - field (class: scala.Tuple2, name: _2, type: class java.lang.Object)
> - object (class scala.Tuple2, (0,{"userId": 0, "spsPrgrmId": 0,
> "spsSlrLevelCd": 0, "spsSlrLevelSumStartDt": null, "spsSlrLevelSumEndDt":
> null, "currPsLvlId": null}))
> org.apache.spark.SparkException: Job aborted due to stage failure: Task
> 0.0 in stage 2.0 (TID 0) had a not serializable result:
> com.ebay.ep.poc.spark.reporting.process.model.dw.SpsLevelMetricSum
> Serialization stack:
> - object not serializable (class:
> com.ebay.ep.poc.spark.reporting.process.model.dw.SpsLevelMetricSum, value:
> {"userId": 0, "spsPrgrmId": 0, "spsSlrLevelCd": 0, "spsSlrLevelSumStartDt":
> null, "spsSlrLevelSumEndDt": null, "currPsLvlId": null})
> - field (class: scala.Tuple2, name: _2, type: class java.lang.Object)
> - object (class scala.Tuple2, (0,{"userId": 0, "spsPrgrmId": 0,
> "spsSlrLevelCd": 0, "spsSlrLevelSumStartDt": null, "spsSlrLevelSumEndDt":
> null, "currPsLvlId": null}))
> at org.apache.spark.scheduler.DAGScheduler.org
> $apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1203)
> at
> org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1192)
> at
> org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1191)
> at
> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
> at
> org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1191)
> at
> org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:693)
> at
> org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:693)
> at scala.Option.foreach(Option.scala:236)
> at
> org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:693)
> at
> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1393)
> at
> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1354)
> at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
> 15/04/03 03:15:36 INFO yarn.ApplicationMaster: Final app status: FAILED,
> exitCode: 15, (reason: User class threw exception: Job aborted due to stage
> failure: Task 0.0 in stage 2.0 (TID 0) had a not serializable result:
> com.ebay.ep.poc.spark.reporting.process.model.dw.SpsLevelMetricSum
> Serialization stack:
>
> 
>
> com.ebay.ep.poc.spark.reporting.process.model.dw.SpsLevelMetricSum is auto
> generated through avro schema using avro-generate-sources maven pulgin.
>
>
> package com.ebay.ep.poc.spark.reporting.process.model.dw;
>
> @SuppressWarnings("all")
>
> @org.apache.avro.specific.AvroGenerated
>
> public class SpsLevelMetricSum extends
> org.apache.avro.specific.SpecificRecordBase implements
> org.apache.avro.specific.SpecificRecord {
> ...
> ...
> }
>
> Can anyone suggest how to fix this ?
>
>
>
> --
> Deepak
>
>

Spark Job Failed - Class not serializable

2015-04-03 Thread ๏̯͡๏ 
My Spark Job failed with


15/04/03 03:15:36 INFO scheduler.DAGScheduler: Job 0 failed:
saveAsNewAPIHadoopFile at AbstractInputHelper.scala:103, took 2.480175 s
15/04/03 03:15:36 ERROR yarn.ApplicationMaster: User class threw exception:
Job aborted due to stage failure: Task 0.0 in stage 2.0 (TID 0) had a not
serializable result:
com.ebay.ep.poc.spark.reporting.process.model.dw.SpsLevelMetricSum
Serialization stack:
- object not serializable (class:
com.ebay.ep.poc.spark.reporting.process.model.dw.SpsLevelMetricSum, value:
{"userId": 0, "spsPrgrmId": 0, "spsSlrLevelCd": 0, "spsSlrLevelSumStartDt":
null, "spsSlrLevelSumEndDt": null, "currPsLvlId": null})
- field (class: scala.Tuple2, name: _2, type: class java.lang.Object)
- object (class scala.Tuple2, (0,{"userId": 0, "spsPrgrmId": 0,
"spsSlrLevelCd": 0, "spsSlrLevelSumStartDt": null, "spsSlrLevelSumEndDt":
null, "currPsLvlId": null}))
org.apache.spark.SparkException: Job aborted due to stage failure: Task 0.0
in stage 2.0 (TID 0) had a not serializable result:
com.ebay.ep.poc.spark.reporting.process.model.dw.SpsLevelMetricSum
Serialization stack:
- object not serializable (class:
com.ebay.ep.poc.spark.reporting.process.model.dw.SpsLevelMetricSum, value:
{"userId": 0, "spsPrgrmId": 0, "spsSlrLevelCd": 0, "spsSlrLevelSumStartDt":
null, "spsSlrLevelSumEndDt": null, "currPsLvlId": null})
- field (class: scala.Tuple2, name: _2, type: class java.lang.Object)
- object (class scala.Tuple2, (0,{"userId": 0, "spsPrgrmId": 0,
"spsSlrLevelCd": 0, "spsSlrLevelSumStartDt": null, "spsSlrLevelSumEndDt":
null, "currPsLvlId": null}))
at org.apache.spark.scheduler.DAGScheduler.org
$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1203)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1192)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1191)
at
scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
at
org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1191)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:693)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:693)
at scala.Option.foreach(Option.scala:236)
at
org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:693)
at
org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1393)
at
org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1354)
at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
15/04/03 03:15:36 INFO yarn.ApplicationMaster: Final app status: FAILED,
exitCode: 15, (reason: User class threw exception: Job aborted due to stage
failure: Task 0.0 in stage 2.0 (TID 0) had a not serializable result:
com.ebay.ep.poc.spark.reporting.process.model.dw.SpsLevelMetricSum
Serialization stack:



com.ebay.ep.poc.spark.reporting.process.model.dw.SpsLevelMetricSum is auto
generated through avro schema using avro-generate-sources maven pulgin.


package com.ebay.ep.poc.spark.reporting.process.model.dw;

@SuppressWarnings("all")

@org.apache.avro.specific.AvroGenerated

public class SpsLevelMetricSum extends
org.apache.avro.specific.SpecificRecordBase implements
org.apache.avro.specific.SpecificRecord {
...
...
}

Can anyone suggest how to fix this ?



-- 
Deepak