Hi, When I'm working with csv files in R using SparkR, I got ClassCastException during the execution of SparkR methods. The below process works fine in 1.4.1, but it is broken from 1.5.0.
(I will use the flights csv file from the examples as a reference, but I can reproduce this with any csv file.) Steps to reproduce: 1. Init spark and sql contexts. Use spark package "com.databricks:spark-csv_2.11:1.0.3" for spark context initialization. 2. Init DataFrame as /df <- read.df(sqlContext, "path_to_flights.csv_file", source = "com.databricks.spark.csv", header="true")/ 3. Run command /head(df)/ 4. Exception is occurred: /ERROR CsvRelation$: Exception while parsing line: 2011-01-24 12:00:00,14,48,1448,1546,3,-1,"CO",1079,"SAT","N14214",0,37,191. java.lang.ClassCastException: java.lang.String cannot be cast to org.apache.spark.unsafe.types.UTF8String at org.apache.spark.sql.catalyst.expressions.BaseGenericInternalRow$class.getUTF8String(rows.scala:45) at org.apache.spark.sql.catalyst.expressions.GenericMutableRow.getUTF8String(rows.scala:247) .../ I'm using CentOS. On windows, the exception does not occur, but the DataFrame contains 0 row. Do I miss something? Thanks -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/SparkR-1-5-1-ClassCastException-when-working-with-CSV-files-tp25217.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
