ianmcook commented on code in PR #45481:
URL: https://github.com/apache/spark/pull/45481#discussion_r1592915751
##########
python/pyspark/sql/pandas/conversion.py:
##########
@@ -225,15 +225,68 @@ def toPandas(self) -> "PandasDataFrameLike":
else:
return pdf
- def _collect_as_arrow(self, split_batches: bool = False) ->
List["pa.RecordBatch"]:
+ def toArrowTable(self) -> "pa.Table":
"""
- Returns all records as a list of ArrowRecordBatches, pyarrow must be
installed
+ Returns the contents of this :class:`DataFrame` as PyArrow
``pyarrow.Table``.
+
+ This is only available if PyArrow is installed and available.
+
+ Notes
+ -----
+ This method should only be used if the resulting PyArrow
``pyarrow.Table`` is
+ expected to be small, as all the data is loaded into the driver's
memory.
+
+ Examples
+ --------
+ >>> df.toArrowTable() # doctest: +SKIP
+ pyarrow.Table
+ age: int64
+ name: string
+ ----
+ age: [[2,5]]
+ name: [["Alice","Bob"]]
+ """
+ from pyspark.sql.dataframe import DataFrame
+
+ assert isinstance(self, DataFrame)
+
+ jconf = self.sparkSession._jconf
+
+ from pyspark.sql.pandas.types import to_arrow_schema
+ from pyspark.sql.pandas.utils import require_minimum_pyarrow_version
+
+ require_minimum_pyarrow_version()
+ to_arrow_schema(self.schema)
+
+ import pyarrow as pa
+
+ self_destruct = jconf.arrowPySparkSelfDestructEnabled()
Review Comment:
Added in fd76fa3. Thank you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
