Re: spark-shell -- running into ArrayIndexOutOfBoundsException
Turns out to be an issue with number of fields being read, one of the fields might be missing from the raw data file causing this error. Michael Ambrust pointed it out in another thread. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/spark-shell-running-into-ArrayIndexOutOfBoundsException-tp10480p10542.html Sent from the Apache Spark User List mailing list archive at Nabble.com.
Re: spark-shell -- running into ArrayIndexOutOfBoundsException
Just wanted to add more info.. I was using SparkSQL reading in the tab-delimited raw data files converting the timestamp to Date format: sc.textFile("rawdata/*").map(_.split("\t")).map(p => Point(df.format(new Date( p(0).trim.toLong*1000L )), p(1), p(2).trim.toInt ,p(3).trim.toInt, p(4).trim.toInt ,p(5))) Then I go about registering it as table and when I run simple query like select count(*) from , I get the ArrayIndexOutOfBoundsException. I bumped up the SPARK_DRIVER_MEMORY to 8g but still didn't help: export SPARK_DRIVER_MEMORY=8g Let me know if I'm missing any steps.. thanks! -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/spark-shell-running-into-ArrayIndexOutOfBoundsException-tp10480p10520.html Sent from the Apache Spark User List mailing list archive at Nabble.com.
spark-shell -- running into ArrayIndexOutOfBoundsException
tor$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386) at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) Thanks! -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/spark-shell-running-into-ArrayIndexOutOfBoundsException-tp10480.html Sent from the Apache Spark User List mailing list archive at Nabble.com.