[
https://issues.apache.org/jira/browse/ARROW-1593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16175077#comment-16175077
]
Wes McKinney commented on ARROW-1593:
-------------------------------------
That seems reasonable. We could also address the range index by augmenting the
metadata in http://pandas-docs.github.io/pandas-docs-travis/developer.html
I have spent significantly less time optimizing {{RecordBatch.from_pandas}}
than {{RecordBatch/Table.to_pandas}}, so there's likely some perf improvements
that could be made on the ingest to Arrow path (e.g. parallel column
conversions)
> [PYTHON] serialize_pandas should pass through the preserve_index keyword
> ------------------------------------------------------------------------
>
> Key: ARROW-1593
> URL: https://issues.apache.org/jira/browse/ARROW-1593
> Project: Apache Arrow
> Issue Type: Improvement
> Components: Python
> Affects Versions: 0.7.0
> Reporter: Tom Augspurger
> Assignee: Tom Augspurger
> Priority: Minor
> Fix For: 0.8.0
>
>
> I'm doing some benchmarking of Arrow serialization for dask.distributed to
> serialize dataframes.
> Overall things look good compared to the current implementation (using
> pickle). The biggest difference was pickle's ability to use pandas'
> RangeIndex to avoid serializing the entire Index of values when possible.
> I suspect that a "range type" isn't in scope for arrow, but in the meantime
> applications using Arrow could detect the `RangeIndex`, and pass {{
> pyarrow.serialize_pandas(df, preserve_index=False) }}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
