[ https://issues.apache.org/jira/browse/SPARK-16225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15350718#comment-15350718 ]
Prabhu Joseph edited comment on SPARK-16225 at 6/27/16 10:00 AM: ----------------------------------------------------------------- Thanks [~srowen]. Changing code to use limit as Integer.MAX_VALUE in split(regex,limit) worked val pRowRDD = pTel.map(_.split(delimiter,Integer.MAX_VALUE)).map(p => Row(p(0),p(1),p(2),p(3),p(4),p(5))) was (Author: prabhu joseph): Thanks [~srowen] > Spark Sql throws ArrayIndexOutOfBoundsException on accessing Null Value > ----------------------------------------------------------------------- > > Key: SPARK-16225 > URL: https://issues.apache.org/jira/browse/SPARK-16225 > Project: Spark > Issue Type: Bug > Components: SQL > Affects Versions: 1.5.2 > Reporter: Prabhu Joseph > > Spark Sql throws ArrayIndexOutOfBoundsException on accessing Null value > Repro: > {code} > [root@spark1 spark]# cat repro > a,b,c,,, > ./bin/spark-shell > scala> val pTel = sc.textFile("file:///usr/hdp/2.3.4.7-4/spark/repro") > scala> pTel.take(1) > res0: Array[String] = Array(a,b,c,,,) > scala> val delimiter = "," > delimiter: String = , > scala> import org.apache.spark.sql._ > import org.apache.spark.sql._ > scala> val pRowRDD = pTel.map(_.split(delimiter)).map(p => > Row(p(0),p(1),p(2),p(3),p(4),p(5))) > pRowRDD: org.apache.spark.rdd.RDD[org.apache.spark.sql.Row] = > MapPartitionsRDD[3] at map at <console>:28 > scala> pRowRDD.take(1) > 16/06/27 08:21:40 ERROR Executor: Exception in task 0.0 in stage 1.0 (TID 1) > java.lang.ArrayIndexOutOfBoundsException: 3 > {code} > It works fine when the data is a,b,c,d,e,f -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org