Re: spark-shell -- running into ArrayIndexOutOfBoundsException

[buntu]

Turns out to be an issue with number of fields being read, one of the fields
might be missing from the raw data file causing this error. Michael Ambrust
pointed it out in another thread.



--
View this message in context: 
http://apache-spark-user-list.1001560.n3.nabble.com/spark-shell-running-into-ArrayIndexOutOfBoundsException-tp10480p10542.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

Re: spark-shell -- running into ArrayIndexOutOfBoundsException

[buntu]

Just wanted to add more info.. I was using SparkSQL reading in the
tab-delimited raw data files converting the timestamp to Date format:

  sc.textFile("rawdata/*").map(_.split("\t")).map(p => Point(df.format(new
Date( p(0).trim.toLong*1000L )), p(1), p(2).trim.toInt ,p(3).trim.toInt,
p(4).trim.toInt ,p(5)))

Then I go about registering it as table and when I run simple query like
select count(*) from 
, I get the ArrayIndexOutOfBoundsException.

I bumped up the SPARK_DRIVER_MEMORY to 8g but still didn't help: 
  export SPARK_DRIVER_MEMORY=8g

Let me know if I'm missing any steps.. thanks!



--
View this message in context: 
http://apache-spark-user-list.1001560.n3.nabble.com/spark-shell-running-into-ArrayIndexOutOfBoundsException-tp10480p10520.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.