GitHub user icexelloss opened a pull request:
https://github.com/apache/spark/pull/18732
groupby().apply() with pandas udf
## What changes were proposed in this pull request?
This PR adds an apply() function on df.groupby(). apply() takes a pandas
udf that is a transformation on `pandas.DataFrame` -> `pandas.DataFrame`.
A quick example:
```
schema = df.schema
@pandas_udf(schema)
def normalize(pdf):
pdf.v1 = (pdf.v1 - pdf.v1.mean()) / pdf.v1.std()
return pdf
df.groupBy('id').apply(normalize(df))
```
This Patch consists of multiple parts which can be broken into small PRs:
* Arrow RecordBatch -> UnsafeRow conversions in ArrowConverters
* pandas_udf in pyspark.sql.functions
* FlatMapInPandas plan node to support groupby().apply()
The first two parts can be used to implement other pandas udf functions.
Design doc:
https://github.com/icexelloss/spark/blob/pandas-udf-doc/docs/pyspark-pandas-udf.md
## How was this patch tested?
* Arrow RecordBatch -> UnsafeRow conversions is tested with the existing
ArrowConverters Suite
* groupby().apply() is tested with a new pyspark test
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/icexelloss/spark groupby-apply-SPARK-20396
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/18732.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #18732
----
commit 8f38c15ea372c1ec9c2fefc36c0bfc3a22c3be14
Author: Li Jin <ice.xell...@gmail.com>
Date: 2017-07-25T15:37:39Z
Initial commit of groupby apply with pandas udf
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
