Teng Yutong created SPARK-17290: ----------------------------------- Summary: Spark CSVInferSchema does not always respect nullValue settings Key: SPARK-17290 URL: https://issues.apache.org/jira/browse/SPARK-17290 Project: Spark Issue Type: Bug Components: SQL Affects Versions: 2.0.0 Reporter: Teng Yutong
When loading a csv-formated data file into a table which has boolean type column, if the boolean value is not given and the nullValue has been set, CSVInferSchema will fail to parse the data. e.g.: table schema: create table test(id varchar(10), flag boolean) USING com.databricks.spark.csv OPTIONS (path "test.csv", header "false", nullValue '') csv data example: aa, bb,true cc,false After some investigation, I found that CSVInferSchema will not check wether the current string match the nullValue or not if the target data type is Boolean、Timestamp、Date。 I am wondering that this logic is coded by purpose or not -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org