Job aborted due to stage failure: Task not serializable:

2015-07-15 Thread Naveen Dabas
I am using the below code and using kryo serializer .when i run this code i got this error : Task not serializable in commented line2) how broadcast variables are treated in exceotu.are they local variables or can be used in any function defined as global variables. object

Spark executor memory information

2015-07-14 Thread Naveen Dabas
Hi, I am new to spark and need some guidance on below mentioned points: 1)I am using spark 1.2,is it possible to see how much memory is being allocated to an executor for web UI. If not how can we figure that out.2)    I am interested in source code of mlib,it is possible to get access to