[
https://issues.apache.org/jira/browse/SPARK-25669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon resolved SPARK-25669.
----------------------------------
Resolution: Fixed
Fix Version/s: 2.4.0
Issue resolved by pull request 22656
[https://github.com/apache/spark/pull/22656]
> Check CSV header only when it exists
> ------------------------------------
>
> Key: SPARK-25669
> URL: https://issues.apache.org/jira/browse/SPARK-25669
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.4.0
> Reporter: Maxim Gekk
> Assignee: Maxim Gekk
> Priority: Minor
> Fix For: 2.4.0
>
>
> Currently, Spark checks the header in CSV files to fields names in provided
> or inferred schema. The check is bypassed if the header doesn't exists and
> CSV content is read from files. In the case, when input CSV comes as dataset
> of strings, Spark always compares the first row to the user specified or
> inferred schema. For example, parsing the following dataset:
> {code:scala}
> val input = Seq("1,2").toDS()
> spark.read.option("enforceSchema", false).csv(input)
> {code}
> throws the exception:
> {code:java}
> java.lang.IllegalArgumentException: CSV header does not conform to the schema.
> Header: 1, 2
> Schema: _c0, _c1
> Expected: _c0 but found: 1
> {code}
> Need to prevent comparison of the first row (if it is not a header) to
> specific or inferred schema.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
