Hi, I created a dataframe with 4 string columns (city, state, country, zipcode). I then applied the following nested schema to it by creating a custom StructType. When I run df.take(5), it gives the exception below as expected. The question is how I can convert the Rows in the dataframe to conform to this nested schema? Thanks!
root |-- ZipCode: struct (nullable = true) | |-- zip: string (nullable = true) |-- Address: struct (nullable = true) | |-- city: string (nullable = true) | |-- state: string (nullable = true) | |-- country: string (nullable = true) [info] org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 6.0 failed 1 times, most recent failure: Lost task 0.0 in stage 6.0 (TID 6, localhost): scala.MatchError: 95123 (of class java.lang.String) [info] at org.apache.spark.sql.catalyst.CatalystTypeConverters$$anonfun$createToCatalystConverter$4.apply(CatalystTypeConverters.scala:178) [info] at org.apache.spark.sql.catalyst.CatalystTypeConverters$.convertRowWithConverters(CatalystTypeConverters.scala:348) [info] at org.apache.spark.sql.catalyst.CatalystTypeConverters$$anonfun$createToCatalystConverter$4.apply(CatalystTypeConverters.scala:180) [info] at org.apache.spark.sql.SQLContext$$anonfun$9.apply(SQLContext.scala:488) [info] at org.apache.spark.sql.SQLContext$$anonfun$9.apply(SQLContext.scala:488) -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/How-to-convert-dataframe-to-a-nested-StructType-schema-tp24694.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
