[
https://issues.apache.org/jira/browse/SPARK-39199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xinrong Meng updated SPARK-39199:
---------------------------------
Description:
pandas API on Spark aims to achieve full pandas API coverage. Currently, most
pandas functions are supported in pandas API on Spark with parameters missing.
There are some common parameters missing:
- how to do with NAs: `skipna`, `dropna`
- filter data types: `numeric_only`, `bool_only`
- filter result length: `keep`
- reindex result: `ignore_index`
They support common use cases and should be prioritized.
was:
pandas API on Spark aims to achieve full pandas API coverage. Currently, most
pandas functions are supported in pandas API on Spark with parameters missing.
There are some common parameters missing:
> Implement pandas API missing parameters
> ---------------------------------------
>
> Key: SPARK-39199
> URL: https://issues.apache.org/jira/browse/SPARK-39199
> Project: Spark
> Issue Type: Umbrella
> Components: PySpark
> Affects Versions: 3.3.0, 3.4.0, 3.3.1
> Reporter: Xinrong Meng
> Priority: Major
>
> pandas API on Spark aims to achieve full pandas API coverage. Currently, most
> pandas functions are supported in pandas API on Spark with parameters missing.
> There are some common parameters missing:
> - how to do with NAs: `skipna`, `dropna`
> - filter data types: `numeric_only`, `bool_only`
> - filter result length: `keep`
> - reindex result: `ignore_index`
> They support common use cases and should be prioritized.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
