[
https://issues.apache.org/jira/browse/SPARK-14141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15213199#comment-15213199
]
Luke Miner commented on SPARK-14141:
------------------------------------
If that's the case, it sounds like it is doable. One way would be to convert
each chunk into a dataframe using the `from_records` constructor, coercing
each column in the dataframe to the specified datatype, and then appending to
the final dataframe.
Or maybe it's worthwhile to add an issue to pandas allowing type coercion as
part of the `from_records` constructor...
> Let user specify datatypes of pandas dataframe in toPandas()
> ------------------------------------------------------------
>
> Key: SPARK-14141
> URL: https://issues.apache.org/jira/browse/SPARK-14141
> Project: Spark
> Issue Type: New Feature
> Components: Input/Output, PySpark, SQL
> Reporter: Luke Miner
> Priority: Minor
>
> Would be nice to specify the dtypes of the pandas dataframe during the
> toPandas() call. Something like:
> bq. pdf = df.toPandas(dtypes={'a': 'float64', 'b': 'datetime64', 'c': 'bool',
> 'd': 'category'})
> Since dtypes like `category` are more memory efficient, you could potentially
> load many more rows into a pandas dataframe with this option without running
> out of memory.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
