[
https://issues.apache.org/jira/browse/SPARK-6591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18065581#comment-18065581
]
Malthe Borch commented on SPARK-6591:
-------------------------------------
The implementation doesn't handle `None` correctly; it becomes the string
"None", while really the option should just not be set.
> Python data source load options should auto convert common types into strings
> -----------------------------------------------------------------------------
>
> Key: SPARK-6591
> URL: https://issues.apache.org/jira/browse/SPARK-6591
> Project: Spark
> Issue Type: Improvement
> Components: PySpark, SQL
> Reporter: Reynold Xin
> Assignee: Yijie Shen
> Priority: Major
> Labels: DataFrame, DataSource
> Fix For: 1.4.2, 1.5.0
>
>
> See the discussion at : https://github.com/databricks/spark-csv/pull/39
> If the caller invokes
> {code}
> sqlContext.load("com.databricks.spark.csv", path = "cars.csv", header = True)
> {code}
> We should automatically turn header into "true" in string form.
> We should do this for booleans and numeric values.
> cc [~yhuai]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
