[jira] [Commented] (SPARK-13534) Implement Apache Arrow serializer for Spark DataFrame for use in DataFrame.toPandas
[
https://issues.apache.org/jira/browse/SPARK-13534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16084886#comment-16084886
]
Ruslan Dautkhanov commented on SPARK-13534:
---
[~bryanc], thanks for the feedback. We sometimes have issues with caching
dataframes in Spark, so we wanted to see if Airflow could be a better fit in
PySpark than Pickle / cPickle for caching dataframes?
Thanks for the link - I will check that. On a separarate note, Arrow batched
columnar storage can still be iterated over for reads? For non-batched writes
PySpark serializer can fall back to non-Arrow format. So might be interesting
to explore if there are two serializers can be active at the same time -
batched Airflow and fall-back to cPickle if necessary?
> Implement Apache Arrow serializer for Spark DataFrame for use in
> DataFrame.toPandas
> ---
>
> Key: SPARK-13534
> URL: https://issues.apache.org/jira/browse/SPARK-13534
> Project: Spark
> Issue Type: Sub-task
> Components: PySpark
>Affects Versions: 2.1.0
>Reporter: Wes McKinney
>Assignee: Bryan Cutler
> Fix For: 2.3.0
>
> Attachments: benchmark.py
>
>
> The current code path for accessing Spark DataFrame data in Python using
> PySpark passes through an inefficient serialization-deserialiation process
> that I've examined at a high level here:
> https://gist.github.com/wesm/0cb5531b1c2e346a0007. Currently, RDD[Row]
> objects are being deserialized in pure Python as a list of tuples, which are
> then converted to pandas.DataFrame using its {{from_records}} alternate
> constructor. This also uses a large amount of memory.
> For flat (no nested types) schemas, the Apache Arrow memory layout
> (https://github.com/apache/arrow/tree/master/format) can be deserialized to
> {{pandas.DataFrame}} objects with comparatively small overhead compared with
> memcpy / system memory bandwidth -- Arrow's bitmasks must be examined,
> replacing the corresponding null values with pandas's sentinel values (None
> or NaN as appropriate).
> I will be contributing patches to Arrow in the coming weeks for converting
> between Arrow and pandas in the general case, so if Spark can send Arrow
> memory to PySpark, we will hopefully be able to increase the Python data
> access throughput by an order of magnitude or more. I propose to add an new
> serializer for Spark DataFrame and a new method that can be invoked from
> PySpark to request a Arrow memory-layout byte stream, prefixed by a data
> header indicating array buffer offsets and sizes.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-13534) Implement Apache Arrow serializer for Spark DataFrame for use in DataFrame.toPandas
[
https://issues.apache.org/jira/browse/SPARK-13534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16084390#comment-16084390
]
Bryan Cutler commented on SPARK-13534:
--
Hi [~tagar], the {{ArrowSerializer}} doesn't quite fit as a drop-in replacement
because the standard PySpark serializers use iterators over elements and Arrow
works on batches. Trying to iterate over the batches to get individual
elements would probably cancel out any performance gains. So then you would
need to operate on the data with an interface like Pandas. I proposed
something similar in my comment
[here|https://issues.apache.org/jira/browse/SPARK-21190?focusedCommentId=16077390=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16077390]
(some api
[details|https://gist.github.com/BryanCutler/2d2ae04e81fa96ba4b61dc095726419f]).
I'd like to hear what your use case is for working with Arrow data and what
you'd want to see in Spark to support this?
> Implement Apache Arrow serializer for Spark DataFrame for use in
> DataFrame.toPandas
> ---
>
> Key: SPARK-13534
> URL: https://issues.apache.org/jira/browse/SPARK-13534
> Project: Spark
> Issue Type: Sub-task
> Components: PySpark
>Affects Versions: 2.1.0
>Reporter: Wes McKinney
>Assignee: Bryan Cutler
> Fix For: 2.3.0
>
> Attachments: benchmark.py
>
>
> The current code path for accessing Spark DataFrame data in Python using
> PySpark passes through an inefficient serialization-deserialiation process
> that I've examined at a high level here:
> https://gist.github.com/wesm/0cb5531b1c2e346a0007. Currently, RDD[Row]
> objects are being deserialized in pure Python as a list of tuples, which are
> then converted to pandas.DataFrame using its {{from_records}} alternate
> constructor. This also uses a large amount of memory.
> For flat (no nested types) schemas, the Apache Arrow memory layout
> (https://github.com/apache/arrow/tree/master/format) can be deserialized to
> {{pandas.DataFrame}} objects with comparatively small overhead compared with
> memcpy / system memory bandwidth -- Arrow's bitmasks must be examined,
> replacing the corresponding null values with pandas's sentinel values (None
> or NaN as appropriate).
> I will be contributing patches to Arrow in the coming weeks for converting
> between Arrow and pandas in the general case, so if Spark can send Arrow
> memory to PySpark, we will hopefully be able to increase the Python data
> access throughput by an order of magnitude or more. I propose to add an new
> serializer for Spark DataFrame and a new method that can be invoked from
> PySpark to request a Arrow memory-layout byte stream, prefixed by a data
> header indicating array buffer offsets and sizes.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-13534) Implement Apache Arrow serializer for Spark DataFrame for use in DataFrame.toPandas
[
https://issues.apache.org/jira/browse/SPARK-13534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16083345#comment-16083345
]
Leif Walsh commented on SPARK-13534:
See SPARK-21190 for a case we're considering for using arrow to move data
between the executors and python workers.
> Implement Apache Arrow serializer for Spark DataFrame for use in
> DataFrame.toPandas
> ---
>
> Key: SPARK-13534
> URL: https://issues.apache.org/jira/browse/SPARK-13534
> Project: Spark
> Issue Type: Sub-task
> Components: PySpark
>Affects Versions: 2.1.0
>Reporter: Wes McKinney
>Assignee: Bryan Cutler
> Fix For: 2.3.0
>
> Attachments: benchmark.py
>
>
> The current code path for accessing Spark DataFrame data in Python using
> PySpark passes through an inefficient serialization-deserialiation process
> that I've examined at a high level here:
> https://gist.github.com/wesm/0cb5531b1c2e346a0007. Currently, RDD[Row]
> objects are being deserialized in pure Python as a list of tuples, which are
> then converted to pandas.DataFrame using its {{from_records}} alternate
> constructor. This also uses a large amount of memory.
> For flat (no nested types) schemas, the Apache Arrow memory layout
> (https://github.com/apache/arrow/tree/master/format) can be deserialized to
> {{pandas.DataFrame}} objects with comparatively small overhead compared with
> memcpy / system memory bandwidth -- Arrow's bitmasks must be examined,
> replacing the corresponding null values with pandas's sentinel values (None
> or NaN as appropriate).
> I will be contributing patches to Arrow in the coming weeks for converting
> between Arrow and pandas in the general case, so if Spark can send Arrow
> memory to PySpark, we will hopefully be able to increase the Python data
> access throughput by an order of magnitude or more. I propose to add an new
> serializer for Spark DataFrame and a new method that can be invoked from
> PySpark to request a Arrow memory-layout byte stream, prefixed by a data
> header indicating array buffer offsets and sizes.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-13534) Implement Apache Arrow serializer for Spark DataFrame for use in DataFrame.toPandas
[
https://issues.apache.org/jira/browse/SPARK-13534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16082900#comment-16082900
]
Ruslan Dautkhanov commented on SPARK-13534:
---
So Apache Arrow would currently be available only in a daraframe.toPandas()
call?
Do you have plans to extend that as a more generic pyspark serializer, like
https://github.com/apache/spark/blob/branch-2.2/python/pyspark/serializers.py#L22
for example
{code}
sc = SparkContext('local', 'test', serializer=ArrowSerializer())
{code}
> Implement Apache Arrow serializer for Spark DataFrame for use in
> DataFrame.toPandas
> ---
>
> Key: SPARK-13534
> URL: https://issues.apache.org/jira/browse/SPARK-13534
> Project: Spark
> Issue Type: Sub-task
> Components: PySpark
>Affects Versions: 2.1.0
>Reporter: Wes McKinney
>Assignee: Bryan Cutler
> Fix For: 2.3.0
>
> Attachments: benchmark.py
>
>
> The current code path for accessing Spark DataFrame data in Python using
> PySpark passes through an inefficient serialization-deserialiation process
> that I've examined at a high level here:
> https://gist.github.com/wesm/0cb5531b1c2e346a0007. Currently, RDD[Row]
> objects are being deserialized in pure Python as a list of tuples, which are
> then converted to pandas.DataFrame using its {{from_records}} alternate
> constructor. This also uses a large amount of memory.
> For flat (no nested types) schemas, the Apache Arrow memory layout
> (https://github.com/apache/arrow/tree/master/format) can be deserialized to
> {{pandas.DataFrame}} objects with comparatively small overhead compared with
> memcpy / system memory bandwidth -- Arrow's bitmasks must be examined,
> replacing the corresponding null values with pandas's sentinel values (None
> or NaN as appropriate).
> I will be contributing patches to Arrow in the coming weeks for converting
> between Arrow and pandas in the general case, so if Spark can send Arrow
> memory to PySpark, we will hopefully be able to increase the Python data
> access throughput by an order of magnitude or more. I propose to add an new
> serializer for Spark DataFrame and a new method that can be invoked from
> PySpark to request a Arrow memory-layout byte stream, prefixed by a data
> header indicating array buffer offsets and sizes.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-13534) Implement Apache Arrow serializer for Spark DataFrame for use in DataFrame.toPandas
[
https://issues.apache.org/jira/browse/SPARK-13534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16070475#comment-16070475
]
Bryan Cutler commented on SPARK-13534:
--
Hi [~jaise...@gmail.com], the DataFrameWriter API is for persisting to disk
which is not the intent for Arrow since in is an in-memory format. It would be
possible in the future to add an API to expose internal data from a Spark
Dataset as Arrow data that could be consumed by another process.
> Implement Apache Arrow serializer for Spark DataFrame for use in
> DataFrame.toPandas
> ---
>
> Key: SPARK-13534
> URL: https://issues.apache.org/jira/browse/SPARK-13534
> Project: Spark
> Issue Type: Sub-task
> Components: PySpark
>Affects Versions: 2.1.0
>Reporter: Wes McKinney
>Assignee: Bryan Cutler
> Fix For: 2.3.0
>
> Attachments: benchmark.py
>
>
> The current code path for accessing Spark DataFrame data in Python using
> PySpark passes through an inefficient serialization-deserialiation process
> that I've examined at a high level here:
> https://gist.github.com/wesm/0cb5531b1c2e346a0007. Currently, RDD[Row]
> objects are being deserialized in pure Python as a list of tuples, which are
> then converted to pandas.DataFrame using its {{from_records}} alternate
> constructor. This also uses a large amount of memory.
> For flat (no nested types) schemas, the Apache Arrow memory layout
> (https://github.com/apache/arrow/tree/master/format) can be deserialized to
> {{pandas.DataFrame}} objects with comparatively small overhead compared with
> memcpy / system memory bandwidth -- Arrow's bitmasks must be examined,
> replacing the corresponding null values with pandas's sentinel values (None
> or NaN as appropriate).
> I will be contributing patches to Arrow in the coming weeks for converting
> between Arrow and pandas in the general case, so if Spark can send Arrow
> memory to PySpark, we will hopefully be able to increase the Python data
> access throughput by an order of magnitude or more. I propose to add an new
> serializer for Spark DataFrame and a new method that can be invoked from
> PySpark to request a Arrow memory-layout byte stream, prefixed by a data
> header indicating array buffer offsets and sizes.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-13534) Implement Apache Arrow serializer for Spark DataFrame for use in DataFrame.toPandas
[
https://issues.apache.org/jira/browse/SPARK-13534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16070126#comment-16070126
]
Jais Sebastian commented on SPARK-13534:
Hi,
Do you have any plan to integrate Arrow format for DataFrameWriter for Java API
?
https://spark.apache.org/docs/latest/api/java/index.html?org/apache/spark/sql/DataFrameWriter.html
> Implement Apache Arrow serializer for Spark DataFrame for use in
> DataFrame.toPandas
> ---
>
> Key: SPARK-13534
> URL: https://issues.apache.org/jira/browse/SPARK-13534
> Project: Spark
> Issue Type: Sub-task
> Components: PySpark
>Affects Versions: 2.1.0
>Reporter: Wes McKinney
>Assignee: Bryan Cutler
> Fix For: 2.3.0
>
> Attachments: benchmark.py
>
>
> The current code path for accessing Spark DataFrame data in Python using
> PySpark passes through an inefficient serialization-deserialiation process
> that I've examined at a high level here:
> https://gist.github.com/wesm/0cb5531b1c2e346a0007. Currently, RDD[Row]
> objects are being deserialized in pure Python as a list of tuples, which are
> then converted to pandas.DataFrame using its {{from_records}} alternate
> constructor. This also uses a large amount of memory.
> For flat (no nested types) schemas, the Apache Arrow memory layout
> (https://github.com/apache/arrow/tree/master/format) can be deserialized to
> {{pandas.DataFrame}} objects with comparatively small overhead compared with
> memcpy / system memory bandwidth -- Arrow's bitmasks must be examined,
> replacing the corresponding null values with pandas's sentinel values (None
> or NaN as appropriate).
> I will be contributing patches to Arrow in the coming weeks for converting
> between Arrow and pandas in the general case, so if Spark can send Arrow
> memory to PySpark, we will hopefully be able to increase the Python data
> access throughput by an order of magnitude or more. I propose to add an new
> serializer for Spark DataFrame and a new method that can be invoked from
> PySpark to request a Arrow memory-layout byte stream, prefixed by a data
> header indicating array buffer offsets and sizes.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-13534) Implement Apache Arrow serializer for Spark DataFrame for use in DataFrame.toPandas
[
https://issues.apache.org/jira/browse/SPARK-13534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16066938#comment-16066938
]
Apache Spark commented on SPARK-13534:
--
User 'BryanCutler' has created a pull request for this issue:
https://github.com/apache/spark/pull/18459
> Implement Apache Arrow serializer for Spark DataFrame for use in
> DataFrame.toPandas
> ---
>
> Key: SPARK-13534
> URL: https://issues.apache.org/jira/browse/SPARK-13534
> Project: Spark
> Issue Type: Sub-task
> Components: PySpark
>Affects Versions: 2.1.0
>Reporter: Wes McKinney
>Assignee: Bryan Cutler
> Fix For: 2.3.0
>
> Attachments: benchmark.py
>
>
> The current code path for accessing Spark DataFrame data in Python using
> PySpark passes through an inefficient serialization-deserialiation process
> that I've examined at a high level here:
> https://gist.github.com/wesm/0cb5531b1c2e346a0007. Currently, RDD[Row]
> objects are being deserialized in pure Python as a list of tuples, which are
> then converted to pandas.DataFrame using its {{from_records}} alternate
> constructor. This also uses a large amount of memory.
> For flat (no nested types) schemas, the Apache Arrow memory layout
> (https://github.com/apache/arrow/tree/master/format) can be deserialized to
> {{pandas.DataFrame}} objects with comparatively small overhead compared with
> memcpy / system memory bandwidth -- Arrow's bitmasks must be examined,
> replacing the corresponding null values with pandas's sentinel values (None
> or NaN as appropriate).
> I will be contributing patches to Arrow in the coming weeks for converting
> between Arrow and pandas in the general case, so if Spark can send Arrow
> memory to PySpark, we will hopefully be able to increase the Python data
> access throughput by an order of magnitude or more. I propose to add an new
> serializer for Spark DataFrame and a new method that can be invoked from
> PySpark to request a Arrow memory-layout byte stream, prefixed by a data
> header indicating array buffer offsets and sizes.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-13534) Implement Apache Arrow serializer for Spark DataFrame for use in DataFrame.toPandas
[
https://issues.apache.org/jira/browse/SPARK-13534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16060413#comment-16060413
]
Bryan Cutler commented on SPARK-13534:
--
That is correct [~rxin], this did not have support for complex types or
date/timestamp. I created SPARK-21187 as an umbrella to track addition of all
remaining types.
> Implement Apache Arrow serializer for Spark DataFrame for use in
> DataFrame.toPandas
> ---
>
> Key: SPARK-13534
> URL: https://issues.apache.org/jira/browse/SPARK-13534
> Project: Spark
> Issue Type: New Feature
> Components: PySpark
>Affects Versions: 2.1.0
>Reporter: Wes McKinney
>Assignee: Bryan Cutler
> Fix For: 2.3.0
>
> Attachments: benchmark.py
>
>
> The current code path for accessing Spark DataFrame data in Python using
> PySpark passes through an inefficient serialization-deserialiation process
> that I've examined at a high level here:
> https://gist.github.com/wesm/0cb5531b1c2e346a0007. Currently, RDD[Row]
> objects are being deserialized in pure Python as a list of tuples, which are
> then converted to pandas.DataFrame using its {{from_records}} alternate
> constructor. This also uses a large amount of memory.
> For flat (no nested types) schemas, the Apache Arrow memory layout
> (https://github.com/apache/arrow/tree/master/format) can be deserialized to
> {{pandas.DataFrame}} objects with comparatively small overhead compared with
> memcpy / system memory bandwidth -- Arrow's bitmasks must be examined,
> replacing the corresponding null values with pandas's sentinel values (None
> or NaN as appropriate).
> I will be contributing patches to Arrow in the coming weeks for converting
> between Arrow and pandas in the general case, so if Spark can send Arrow
> memory to PySpark, we will hopefully be able to increase the Python data
> access throughput by an order of magnitude or more. I propose to add an new
> serializer for Spark DataFrame and a new method that can be invoked from
> PySpark to request a Arrow memory-layout byte stream, prefixed by a data
> header indicating array buffer offsets and sizes.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-13534) Implement Apache Arrow serializer for Spark DataFrame for use in DataFrame.toPandas
[
https://issues.apache.org/jira/browse/SPARK-13534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16060328#comment-16060328
]
Reynold Xin commented on SPARK-13534:
-
Was this done? I thought there are still other data types that are not
supported. We should either turn this into an umbrella ticket, or create a new
umbrella ticket.
> Implement Apache Arrow serializer for Spark DataFrame for use in
> DataFrame.toPandas
> ---
>
> Key: SPARK-13534
> URL: https://issues.apache.org/jira/browse/SPARK-13534
> Project: Spark
> Issue Type: New Feature
> Components: PySpark
>Affects Versions: 2.1.0
>Reporter: Wes McKinney
>Assignee: Bryan Cutler
> Fix For: 2.3.0
>
> Attachments: benchmark.py
>
>
> The current code path for accessing Spark DataFrame data in Python using
> PySpark passes through an inefficient serialization-deserialiation process
> that I've examined at a high level here:
> https://gist.github.com/wesm/0cb5531b1c2e346a0007. Currently, RDD[Row]
> objects are being deserialized in pure Python as a list of tuples, which are
> then converted to pandas.DataFrame using its {{from_records}} alternate
> constructor. This also uses a large amount of memory.
> For flat (no nested types) schemas, the Apache Arrow memory layout
> (https://github.com/apache/arrow/tree/master/format) can be deserialized to
> {{pandas.DataFrame}} objects with comparatively small overhead compared with
> memcpy / system memory bandwidth -- Arrow's bitmasks must be examined,
> replacing the corresponding null values with pandas's sentinel values (None
> or NaN as appropriate).
> I will be contributing patches to Arrow in the coming weeks for converting
> between Arrow and pandas in the general case, so if Spark can send Arrow
> memory to PySpark, we will hopefully be able to increase the Python data
> access throughput by an order of magnitude or more. I propose to add an new
> serializer for Spark DataFrame and a new method that can be invoked from
> PySpark to request a Arrow memory-layout byte stream, prefixed by a data
> header indicating array buffer offsets and sizes.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-13534) Implement Apache Arrow serializer for Spark DataFrame for use in DataFrame.toPandas
[
https://issues.apache.org/jira/browse/SPARK-13534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15966385#comment-15966385
]
Jacques Nadeau commented on SPARK-13534:
Great, thanks [~holdenk]!
> Implement Apache Arrow serializer for Spark DataFrame for use in
> DataFrame.toPandas
> ---
>
> Key: SPARK-13534
> URL: https://issues.apache.org/jira/browse/SPARK-13534
> Project: Spark
> Issue Type: New Feature
> Components: PySpark
>Reporter: Wes McKinney
> Attachments: benchmark.py
>
>
> The current code path for accessing Spark DataFrame data in Python using
> PySpark passes through an inefficient serialization-deserialiation process
> that I've examined at a high level here:
> https://gist.github.com/wesm/0cb5531b1c2e346a0007. Currently, RDD[Row]
> objects are being deserialized in pure Python as a list of tuples, which are
> then converted to pandas.DataFrame using its {{from_records}} alternate
> constructor. This also uses a large amount of memory.
> For flat (no nested types) schemas, the Apache Arrow memory layout
> (https://github.com/apache/arrow/tree/master/format) can be deserialized to
> {{pandas.DataFrame}} objects with comparatively small overhead compared with
> memcpy / system memory bandwidth -- Arrow's bitmasks must be examined,
> replacing the corresponding null values with pandas's sentinel values (None
> or NaN as appropriate).
> I will be contributing patches to Arrow in the coming weeks for converting
> between Arrow and pandas in the general case, so if Spark can send Arrow
> memory to PySpark, we will hopefully be able to increase the Python data
> access throughput by an order of magnitude or more. I propose to add an new
> serializer for Spark DataFrame and a new method that can be invoked from
> PySpark to request a Arrow memory-layout byte stream, prefixed by a data
> header indicating array buffer offsets and sizes.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-13534) Implement Apache Arrow serializer for Spark DataFrame for use in DataFrame.toPandas
[
https://issues.apache.org/jira/browse/SPARK-13534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15966339#comment-15966339
]
holdenk commented on SPARK-13534:
-
So I'm following along with the progress on this, I'll try and take a more
thorough look this Thursday.
> Implement Apache Arrow serializer for Spark DataFrame for use in
> DataFrame.toPandas
> ---
>
> Key: SPARK-13534
> URL: https://issues.apache.org/jira/browse/SPARK-13534
> Project: Spark
> Issue Type: New Feature
> Components: PySpark
>Reporter: Wes McKinney
> Attachments: benchmark.py
>
>
> The current code path for accessing Spark DataFrame data in Python using
> PySpark passes through an inefficient serialization-deserialiation process
> that I've examined at a high level here:
> https://gist.github.com/wesm/0cb5531b1c2e346a0007. Currently, RDD[Row]
> objects are being deserialized in pure Python as a list of tuples, which are
> then converted to pandas.DataFrame using its {{from_records}} alternate
> constructor. This also uses a large amount of memory.
> For flat (no nested types) schemas, the Apache Arrow memory layout
> (https://github.com/apache/arrow/tree/master/format) can be deserialized to
> {{pandas.DataFrame}} objects with comparatively small overhead compared with
> memcpy / system memory bandwidth -- Arrow's bitmasks must be examined,
> replacing the corresponding null values with pandas's sentinel values (None
> or NaN as appropriate).
> I will be contributing patches to Arrow in the coming weeks for converting
> between Arrow and pandas in the general case, so if Spark can send Arrow
> memory to PySpark, we will hopefully be able to increase the Python data
> access throughput by an order of magnitude or more. I propose to add an new
> serializer for Spark DataFrame and a new method that can be invoked from
> PySpark to request a Arrow memory-layout byte stream, prefixed by a data
> header indicating array buffer offsets and sizes.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-13534) Implement Apache Arrow serializer for Spark DataFrame for use in DataFrame.toPandas
[
https://issues.apache.org/jira/browse/SPARK-13534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15966231#comment-15966231
]
Jacques Nadeau commented on SPARK-13534:
Anybody know some committers we can get to look at this?
> Implement Apache Arrow serializer for Spark DataFrame for use in
> DataFrame.toPandas
> ---
>
> Key: SPARK-13534
> URL: https://issues.apache.org/jira/browse/SPARK-13534
> Project: Spark
> Issue Type: New Feature
> Components: PySpark
>Reporter: Wes McKinney
> Attachments: benchmark.py
>
>
> The current code path for accessing Spark DataFrame data in Python using
> PySpark passes through an inefficient serialization-deserialiation process
> that I've examined at a high level here:
> https://gist.github.com/wesm/0cb5531b1c2e346a0007. Currently, RDD[Row]
> objects are being deserialized in pure Python as a list of tuples, which are
> then converted to pandas.DataFrame using its {{from_records}} alternate
> constructor. This also uses a large amount of memory.
> For flat (no nested types) schemas, the Apache Arrow memory layout
> (https://github.com/apache/arrow/tree/master/format) can be deserialized to
> {{pandas.DataFrame}} objects with comparatively small overhead compared with
> memcpy / system memory bandwidth -- Arrow's bitmasks must be examined,
> replacing the corresponding null values with pandas's sentinel values (None
> or NaN as appropriate).
> I will be contributing patches to Arrow in the coming weeks for converting
> between Arrow and pandas in the general case, so if Spark can send Arrow
> memory to PySpark, we will hopefully be able to increase the Python data
> access throughput by an order of magnitude or more. I propose to add an new
> serializer for Spark DataFrame and a new method that can be invoked from
> PySpark to request a Arrow memory-layout byte stream, prefixed by a data
> header indicating array buffer offsets and sizes.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-13534) Implement Apache Arrow serializer for Spark DataFrame for use in DataFrame.toPandas
[
https://issues.apache.org/jira/browse/SPARK-13534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15713340#comment-15713340
]
Bryan Cutler commented on SPARK-13534:
--
Hi [~icexelloss], that sounds great! We could definitely use some help for
validation testing.
> Implement Apache Arrow serializer for Spark DataFrame for use in
> DataFrame.toPandas
> ---
>
> Key: SPARK-13534
> URL: https://issues.apache.org/jira/browse/SPARK-13534
> Project: Spark
> Issue Type: New Feature
> Components: PySpark
>Reporter: Wes McKinney
>
> The current code path for accessing Spark DataFrame data in Python using
> PySpark passes through an inefficient serialization-deserialiation process
> that I've examined at a high level here:
> https://gist.github.com/wesm/0cb5531b1c2e346a0007. Currently, RDD[Row]
> objects are being deserialized in pure Python as a list of tuples, which are
> then converted to pandas.DataFrame using its {{from_records}} alternate
> constructor. This also uses a large amount of memory.
> For flat (no nested types) schemas, the Apache Arrow memory layout
> (https://github.com/apache/arrow/tree/master/format) can be deserialized to
> {{pandas.DataFrame}} objects with comparatively small overhead compared with
> memcpy / system memory bandwidth -- Arrow's bitmasks must be examined,
> replacing the corresponding null values with pandas's sentinel values (None
> or NaN as appropriate).
> I will be contributing patches to Arrow in the coming weeks for converting
> between Arrow and pandas in the general case, so if Spark can send Arrow
> memory to PySpark, we will hopefully be able to increase the Python data
> access throughput by an order of magnitude or more. I propose to add an new
> serializer for Spark DataFrame and a new method that can be invoked from
> PySpark to request a Arrow memory-layout byte stream, prefixed by a data
> header indicating array buffer offsets and sizes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-13534) Implement Apache Arrow serializer for Spark DataFrame for use in DataFrame.toPandas
[
https://issues.apache.org/jira/browse/SPARK-13534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15712825#comment-15712825
]
Li Jin commented on SPARK-13534:
[~bryanc],
Allow me to introduce myself. I am Li Jin and I am working with [~wesmckinn] on
pyspark/arrow. I have been looking this issue recently and don't want to
duplicate effort. I think I can help by writing unit tests to validate the
arrow record batch created. What do you think?
> Implement Apache Arrow serializer for Spark DataFrame for use in
> DataFrame.toPandas
> ---
>
> Key: SPARK-13534
> URL: https://issues.apache.org/jira/browse/SPARK-13534
> Project: Spark
> Issue Type: New Feature
> Components: PySpark
>Reporter: Wes McKinney
>
> The current code path for accessing Spark DataFrame data in Python using
> PySpark passes through an inefficient serialization-deserialiation process
> that I've examined at a high level here:
> https://gist.github.com/wesm/0cb5531b1c2e346a0007. Currently, RDD[Row]
> objects are being deserialized in pure Python as a list of tuples, which are
> then converted to pandas.DataFrame using its {{from_records}} alternate
> constructor. This also uses a large amount of memory.
> For flat (no nested types) schemas, the Apache Arrow memory layout
> (https://github.com/apache/arrow/tree/master/format) can be deserialized to
> {{pandas.DataFrame}} objects with comparatively small overhead compared with
> memcpy / system memory bandwidth -- Arrow's bitmasks must be examined,
> replacing the corresponding null values with pandas's sentinel values (None
> or NaN as appropriate).
> I will be contributing patches to Arrow in the coming weeks for converting
> between Arrow and pandas in the general case, so if Spark can send Arrow
> memory to PySpark, we will hopefully be able to increase the Python data
> access throughput by an order of magnitude or more. I propose to add an new
> serializer for Spark DataFrame and a new method that can be invoked from
> PySpark to request a Arrow memory-layout byte stream, prefixed by a data
> header indicating array buffer offsets and sizes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-13534) Implement Apache Arrow serializer for Spark DataFrame for use in DataFrame.toPandas
[
https://issues.apache.org/jira/browse/SPARK-13534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15649436#comment-15649436
]
Apache Spark commented on SPARK-13534:
--
User 'BryanCutler' has created a pull request for this issue:
https://github.com/apache/spark/pull/15821
> Implement Apache Arrow serializer for Spark DataFrame for use in
> DataFrame.toPandas
> ---
>
> Key: SPARK-13534
> URL: https://issues.apache.org/jira/browse/SPARK-13534
> Project: Spark
> Issue Type: New Feature
> Components: PySpark
>Reporter: Wes McKinney
>
> The current code path for accessing Spark DataFrame data in Python using
> PySpark passes through an inefficient serialization-deserialiation process
> that I've examined at a high level here:
> https://gist.github.com/wesm/0cb5531b1c2e346a0007. Currently, RDD[Row]
> objects are being deserialized in pure Python as a list of tuples, which are
> then converted to pandas.DataFrame using its {{from_records}} alternate
> constructor. This also uses a large amount of memory.
> For flat (no nested types) schemas, the Apache Arrow memory layout
> (https://github.com/apache/arrow/tree/master/format) can be deserialized to
> {{pandas.DataFrame}} objects with comparatively small overhead compared with
> memcpy / system memory bandwidth -- Arrow's bitmasks must be examined,
> replacing the corresponding null values with pandas's sentinel values (None
> or NaN as appropriate).
> I will be contributing patches to Arrow in the coming weeks for converting
> between Arrow and pandas in the general case, so if Spark can send Arrow
> memory to PySpark, we will hopefully be able to increase the Python data
> access throughput by an order of magnitude or more. I propose to add an new
> serializer for Spark DataFrame and a new method that can be invoked from
> PySpark to request a Arrow memory-layout byte stream, prefixed by a data
> header indicating array buffer offsets and sizes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-13534) Implement Apache Arrow serializer for Spark DataFrame for use in DataFrame.toPandas
[
https://issues.apache.org/jira/browse/SPARK-13534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15649405#comment-15649405
]
Bryan Cutler commented on SPARK-13534:
--
I've been working on this with [~xusen]. We have a very rough WIP branch that
I'll link here in case others want to pitch in or review while we are working
out the kinks.
> Implement Apache Arrow serializer for Spark DataFrame for use in
> DataFrame.toPandas
> ---
>
> Key: SPARK-13534
> URL: https://issues.apache.org/jira/browse/SPARK-13534
> Project: Spark
> Issue Type: New Feature
> Components: PySpark
>Reporter: Wes McKinney
>
> The current code path for accessing Spark DataFrame data in Python using
> PySpark passes through an inefficient serialization-deserialiation process
> that I've examined at a high level here:
> https://gist.github.com/wesm/0cb5531b1c2e346a0007. Currently, RDD[Row]
> objects are being deserialized in pure Python as a list of tuples, which are
> then converted to pandas.DataFrame using its {{from_records}} alternate
> constructor. This also uses a large amount of memory.
> For flat (no nested types) schemas, the Apache Arrow memory layout
> (https://github.com/apache/arrow/tree/master/format) can be deserialized to
> {{pandas.DataFrame}} objects with comparatively small overhead compared with
> memcpy / system memory bandwidth -- Arrow's bitmasks must be examined,
> replacing the corresponding null values with pandas's sentinel values (None
> or NaN as appropriate).
> I will be contributing patches to Arrow in the coming weeks for converting
> between Arrow and pandas in the general case, so if Spark can send Arrow
> memory to PySpark, we will hopefully be able to increase the Python data
> access throughput by an order of magnitude or more. I propose to add an new
> serializer for Spark DataFrame and a new method that can be invoked from
> PySpark to request a Arrow memory-layout byte stream, prefixed by a data
> header indicating array buffer offsets and sizes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-13534) Implement Apache Arrow serializer for Spark DataFrame for use in DataFrame.toPandas
[
https://issues.apache.org/jira/browse/SPARK-13534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15596756#comment-15596756
]
Frederick Reiss commented on SPARK-13534:
-
We ([~bryanc], [~holdenk], [~yinxusen], and myself) are looking into this.
Here's a rough outline of the current planned approach:
- Add a dependency on Arrow 0.1's Java and Scala APIs to Spark.
- Add a new developer API method to Dataset, {{collectAsArrow()}}, that returns
an array of byte arrays, where each byte array contains a block of records in
Arrow format. The conversion to Arrow will be a streamlined version of the
Parquet conversion in {{ParquetWriteSupport}} (minus all the callbacks and
levels of indirection). Conversion of complex types (Struct, Array, Map) to
Arrow will not be supported in this version.
- modify Pyspark's {{DataFrame.toPandas}} method to use the following logic:
{noformat}
if (the schema of the DataFrame does not contain complex types)
Call collectAsArrow() on the underlying Scala Dataset.
Pull the resulting buffers of Arrow data over to the Python process.
Use Arrow's Python APIs to convert the buffers into a single Pandas
dataframe.
else
Use the existing code as a slow-path conversion.
{noformat}
> Implement Apache Arrow serializer for Spark DataFrame for use in
> DataFrame.toPandas
> ---
>
> Key: SPARK-13534
> URL: https://issues.apache.org/jira/browse/SPARK-13534
> Project: Spark
> Issue Type: New Feature
> Components: PySpark
>Reporter: Wes McKinney
>
> The current code path for accessing Spark DataFrame data in Python using
> PySpark passes through an inefficient serialization-deserialiation process
> that I've examined at a high level here:
> https://gist.github.com/wesm/0cb5531b1c2e346a0007. Currently, RDD[Row]
> objects are being deserialized in pure Python as a list of tuples, which are
> then converted to pandas.DataFrame using its {{from_records}} alternate
> constructor. This also uses a large amount of memory.
> For flat (no nested types) schemas, the Apache Arrow memory layout
> (https://github.com/apache/arrow/tree/master/format) can be deserialized to
> {{pandas.DataFrame}} objects with comparatively small overhead compared with
> memcpy / system memory bandwidth -- Arrow's bitmasks must be examined,
> replacing the corresponding null values with pandas's sentinel values (None
> or NaN as appropriate).
> I will be contributing patches to Arrow in the coming weeks for converting
> between Arrow and pandas in the general case, so if Spark can send Arrow
> memory to PySpark, we will hopefully be able to increase the Python data
> access throughput by an order of magnitude or more. I propose to add an new
> serializer for Spark DataFrame and a new method that can be invoked from
> PySpark to request a Arrow memory-layout byte stream, prefixed by a data
> header indicating array buffer offsets and sizes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-13534) Implement Apache Arrow serializer for Spark DataFrame for use in DataFrame.toPandas
[
https://issues.apache.org/jira/browse/SPARK-13534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15579946#comment-15579946
]
holdenk commented on SPARK-13534:
-
And now they have a release :) I'm not certain its at the stage where we can
use it - but I'll do some poking over the next few weeks :)
> Implement Apache Arrow serializer for Spark DataFrame for use in
> DataFrame.toPandas
> ---
>
> Key: SPARK-13534
> URL: https://issues.apache.org/jira/browse/SPARK-13534
> Project: Spark
> Issue Type: New Feature
> Components: PySpark
>Reporter: Wes McKinney
>
> The current code path for accessing Spark DataFrame data in Python using
> PySpark passes through an inefficient serialization-deserialiation process
> that I've examined at a high level here:
> https://gist.github.com/wesm/0cb5531b1c2e346a0007. Currently, RDD[Row]
> objects are being deserialized in pure Python as a list of tuples, which are
> then converted to pandas.DataFrame using its {{from_records}} alternate
> constructor. This also uses a large amount of memory.
> For flat (no nested types) schemas, the Apache Arrow memory layout
> (https://github.com/apache/arrow/tree/master/format) can be deserialized to
> {{pandas.DataFrame}} objects with comparatively small overhead compared with
> memcpy / system memory bandwidth -- Arrow's bitmasks must be examined,
> replacing the corresponding null values with pandas's sentinel values (None
> or NaN as appropriate).
> I will be contributing patches to Arrow in the coming weeks for converting
> between Arrow and pandas in the general case, so if Spark can send Arrow
> memory to PySpark, we will hopefully be able to increase the Python data
> access throughput by an order of magnitude or more. I propose to add an new
> serializer for Spark DataFrame and a new method that can be invoked from
> PySpark to request a Arrow memory-layout byte stream, prefixed by a data
> header indicating array buffer offsets and sizes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-13534) Implement Apache Arrow serializer for Spark DataFrame for use in DataFrame.toPandas
[
https://issues.apache.org/jira/browse/SPARK-13534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15557242#comment-15557242
]
holdenk commented on SPARK-13534:
-
For people following along arrow is in the middle of voting on its next
release, while its likely not yet at the point where we can start using it will
be good for those interested (like myself) to take a look once the release is
ready :)
> Implement Apache Arrow serializer for Spark DataFrame for use in
> DataFrame.toPandas
> ---
>
> Key: SPARK-13534
> URL: https://issues.apache.org/jira/browse/SPARK-13534
> Project: Spark
> Issue Type: New Feature
> Components: PySpark
>Reporter: Wes McKinney
>
> The current code path for accessing Spark DataFrame data in Python using
> PySpark passes through an inefficient serialization-deserialiation process
> that I've examined at a high level here:
> https://gist.github.com/wesm/0cb5531b1c2e346a0007. Currently, RDD[Row]
> objects are being deserialized in pure Python as a list of tuples, which are
> then converted to pandas.DataFrame using its {{from_records}} alternate
> constructor. This also uses a large amount of memory.
> For flat (no nested types) schemas, the Apache Arrow memory layout
> (https://github.com/apache/arrow/tree/master/format) can be deserialized to
> {{pandas.DataFrame}} objects with comparatively small overhead compared with
> memcpy / system memory bandwidth -- Arrow's bitmasks must be examined,
> replacing the corresponding null values with pandas's sentinel values (None
> or NaN as appropriate).
> I will be contributing patches to Arrow in the coming weeks for converting
> between Arrow and pandas in the general case, so if Spark can send Arrow
> memory to PySpark, we will hopefully be able to increase the Python data
> access throughput by an order of magnitude or more. I propose to add an new
> serializer for Spark DataFrame and a new method that can be invoked from
> PySpark to request a Arrow memory-layout byte stream, prefixed by a data
> header indicating array buffer offsets and sizes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-13534) Implement Apache Arrow serializer for Spark DataFrame for use in DataFrame.toPandas
[
https://issues.apache.org/jira/browse/SPARK-13534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15456990#comment-15456990
]
Frederick Reiss commented on SPARK-13534:
-
[~wesmckinn], are you planning to work on this issue soon?
> Implement Apache Arrow serializer for Spark DataFrame for use in
> DataFrame.toPandas
> ---
>
> Key: SPARK-13534
> URL: https://issues.apache.org/jira/browse/SPARK-13534
> Project: Spark
> Issue Type: New Feature
> Components: PySpark
>Reporter: Wes McKinney
>
> The current code path for accessing Spark DataFrame data in Python using
> PySpark passes through an inefficient serialization-deserialiation process
> that I've examined at a high level here:
> https://gist.github.com/wesm/0cb5531b1c2e346a0007. Currently, RDD[Row]
> objects are being deserialized in pure Python as a list of tuples, which are
> then converted to pandas.DataFrame using its {{from_records}} alternate
> constructor. This also uses a large amount of memory.
> For flat (no nested types) schemas, the Apache Arrow memory layout
> (https://github.com/apache/arrow/tree/master/format) can be deserialized to
> {{pandas.DataFrame}} objects with comparatively small overhead compared with
> memcpy / system memory bandwidth -- Arrow's bitmasks must be examined,
> replacing the corresponding null values with pandas's sentinel values (None
> or NaN as appropriate).
> I will be contributing patches to Arrow in the coming weeks for converting
> between Arrow and pandas in the general case, so if Spark can send Arrow
> memory to PySpark, we will hopefully be able to increase the Python data
> access throughput by an order of magnitude or more. I propose to add an new
> serializer for Spark DataFrame and a new method that can be invoked from
> PySpark to request a Arrow memory-layout byte stream, prefixed by a data
> header indicating array buffer offsets and sizes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-13534) Implement Apache Arrow serializer for Spark DataFrame for use in DataFrame.toPandas
[
https://issues.apache.org/jira/browse/SPARK-13534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15171273#comment-15171273
]
Wes McKinney commented on SPARK-13534:
--
SPARK-13391 would need to have its scope more narrowly defined. Perhaps we can
modify that issue scope to encompass UDF evaluation using Arrow for data
interchange? Otherwise we can close it.
This issue is a first step in that direction, but UDF evaluation is not in
scope for this issue (this is only about raw table data movement).
> Implement Apache Arrow serializer for Spark DataFrame for use in
> DataFrame.toPandas
> ---
>
> Key: SPARK-13534
> URL: https://issues.apache.org/jira/browse/SPARK-13534
> Project: Spark
> Issue Type: New Feature
> Components: PySpark
>Reporter: Wes McKinney
>
> The current code path for accessing Spark DataFrame data in Python using
> PySpark passes through an inefficient serialization-deserialiation process
> that I've examined at a high level here:
> https://gist.github.com/wesm/0cb5531b1c2e346a0007. Currently, RDD[Row]
> objects are being deserialized in pure Python as a list of tuples, which are
> then converted to pandas.DataFrame using its {{from_records}} alternate
> constructor. This also uses a large amount of memory.
> For flat (no nested types) schemas, the Apache Arrow memory layout
> (https://github.com/apache/arrow/tree/master/format) can be deserialized to
> {{pandas.DataFrame}} objects with comparatively small overhead compared with
> memcpy / system memory bandwidth -- Arrow's bitmasks must be examined,
> replacing the corresponding null values with pandas's sentinel values (None
> or NaN as appropriate).
> I will be contributing patches to Arrow in the coming weeks for converting
> between Arrow and pandas in the general case, so if Spark can send Arrow
> memory to PySpark, we will hopefully be able to increase the Python data
> access throughput by an order of magnitude or more. I propose to add an new
> serializer for Spark DataFrame and a new method that can be invoked from
> PySpark to request a Arrow memory-layout byte stream, prefixed by a data
> header indicating array buffer offsets and sizes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-13534) Implement Apache Arrow serializer for Spark DataFrame for use in DataFrame.toPandas
[
https://issues.apache.org/jira/browse/SPARK-13534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15171029#comment-15171029
]
Sean Owen commented on SPARK-13534:
---
[~wesmckinn] there's already https://issues.apache.org/jira/browse/SPARK-13391
-- is this meaningfully different? the other may be too broad to be useful and
we can close it.
> Implement Apache Arrow serializer for Spark DataFrame for use in
> DataFrame.toPandas
> ---
>
> Key: SPARK-13534
> URL: https://issues.apache.org/jira/browse/SPARK-13534
> Project: Spark
> Issue Type: New Feature
> Components: PySpark
>Reporter: Wes McKinney
>
> The current code path for accessing Spark DataFrame data in Python using
> PySpark passes through an inefficient serialization-deserialiation process
> that I've examined at a high level here:
> https://gist.github.com/wesm/0cb5531b1c2e346a0007. Currently, RDD[Row]
> objects are being deserialized in pure Python as a list of tuples, which are
> then converted to pandas.DataFrame using its {{from_records}} alternate
> constructor. This also uses a large amount of memory.
> For flat (no nested types) schemas, the Apache Arrow memory layout
> (https://github.com/apache/arrow/tree/master/format) can be deserialized to
> {{pandas.DataFrame}} objects with comparatively small overhead compared with
> memcpy / system memory bandwidth -- Arrow's bitmasks must be examined,
> replacing the corresponding null values with pandas's sentinel values (None
> or NaN as appropriate).
> I will be contributing patches to Arrow in the coming weeks for converting
> between Arrow and pandas in the general case, so if Spark can send Arrow
> memory to PySpark, we will hopefully be able to increase the Python data
> access throughput by an order of magnitude or more. I propose to add an new
> serializer for Spark DataFrame and a new method that can be invoked from
> PySpark to request a Arrow memory-layout byte stream, prefixed by a data
> header indicating array buffer offsets and sizes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
