Peter Nguyen created SPARK-53053:
------------------------------------
Summary: Support Pandas Extension Properties
Key: SPARK-53053
URL: https://issues.apache.org/jira/browse/SPARK-53053
Project: Spark
Issue Type: Bug
Components: Pandas API on Spark
Affects Versions: 4.1.0
Reporter: Peter Nguyen
Original pandas supports properties like _constructor that are can be used to
easily override what datatype is returned by default when downstream libraries
are built to inherit from pandas classes (e.g Dask). By override the
_constructor method,
def _constructor(self):
return MyDataFrame
all functions like head() and getitem return the object using that type
(MyDataFrame) instead of a normal Pandas dataframe. To implement this, we would
have to manually modify functions like self.head() to return
self._constructor(result) instead of DataFrame(result)
https://pandas.pydata.org/docs/development/extending.html#override-constructor-properties
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
