Hi, I’ve tried to send this before but it didn’t come through.
I have a fairly simple set of commands which work in the spark shell. However, they do not work on zeppelin. The commands are: val wbIPR=Array(8.40, 8.09, 7.56, 6.83, 5.88, 4.72, 3.36, 1.78, 0.0) val pressureIPR=Array(0.00,10000000.00,20000000.00,30000000.00,40000000.00,50000000.00,60000000.00,70000000.00,80000000.00) case class IPR (w:Array[Double],p:Array[Double]) val iprInit=IPR(wbIPR,pressureIPR) val iprList=sc.parallelize((1.0 to 0.9 by -0.001).map(k => IPR(iprInit.w.map(_ * k), iprInit.p)).zipWithIndex,5) iprList.count() The error message is: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 3.0 failed 4 times, most recent failure: Lost task 0.3 in stage 3.0 (TID 54, ip-172-31-7-206.ec2.internal): ExecutorLostFailure (executor 14 lost) Driver stacktrace: at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1203) at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1192) at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1191) Any ideas why this could be failing? I don’t think it’s a compile issue, because if I run val iprList=sc.parallelize(pressureIPR) iprList.count() everything works. Thanks Dean
