[ https://issues.apache.org/jira/browse/SPARK-22505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16250054#comment-16250054 ]
Ruslan Dautkhanov edited comment on SPARK-22505 at 11/13/17 7:47 PM: --------------------------------------------------------------------- Looks like we already discussed very similar topic 1.5 years ago on github ) https://github.com/databricks/spark-csv/issues/264#issuecomment-184943114 Any chance this can be added as a core Spark functionality? Not sure if we can even call that CsvParser().csvRdd from pySpark.. This is what I am asking exactly support for transforming dataframe to RDD[String] #188 https://github.com/databricks/spark-csv/commit/2eb90153a2d6a77b9cde4aee3f6e382df3da1746 I don't see CsvRdd from spark-csv module anywhere in Spark codebase https://github.com/databricks/spark-csv/commit/2eb90153a2d6a77b9cde4aee3f6e382df3da1746#diff-c6f09c5a3e6aedc2e6bfb1c16358e970R123 Did it make its way into Spark? What I see is only private class CSVInferSchema https://github.com/apache/spark/blob/v2.2.0/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVInferSchema.scala#L29 Any other way to achieve this? Thanks a lot for any leads. was (Author: tagar): Looks like we already discussed very similar topic 1.5 years ago on github ) https://github.com/databricks/spark-csv/issues/264#issuecomment-184943114 Any chance this can be added as a core Spark functionality? Not sure if we can even call that CsvParser().csvRdd from pySpark.. > toDF() / createDataFrame() type inference doesn't work as expected > ------------------------------------------------------------------ > > Key: SPARK-22505 > URL: https://issues.apache.org/jira/browse/SPARK-22505 > Project: Spark > Issue Type: Bug > Components: PySpark, Spark Core > Affects Versions: 2.2.0 > Reporter: Ruslan Dautkhanov > Labels: csvparser, inference, pyspark, schema, spark-sql > > {code} > df = > sc.parallelize([('1','a'),('2','b'),('3','c')]).toDF(['should_be_int','should_be_str']) > df.printSchema() > {code} > produces > {noformat} > root > |-- should_be_int: string (nullable = true) > |-- should_be_str: string (nullable = true) > {noformat} > Notice `should_be_int` has `string` datatype, according to documentation: > https://spark.apache.org/docs/latest/sql-programming-guide.html#inferring-the-schema-using-reflection > {quote} > Spark SQL can convert an RDD of Row objects to a DataFrame, inferring the > datatypes. Rows are constructed by passing a list of key/value pairs as > kwargs to the Row class. The keys of this list define the column names of the > table, *and the types are inferred by sampling the whole dataset*, similar to > the inference that is performed on JSON files. > {quote} > Schema inference works as expected when reading delimited files like > {code} > spark.read.format('csv').option('inferSchema', True)... > {code} > but not when using toDF() / createDataFrame() API calls. > Spark 2.2. -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org