GitHub user BryanCutler opened a pull request:
https://github.com/apache/spark/pull/18659
[SPARK-21404][PYSPARK][WIP] Simple Python Vectorized UDFs
## What changes were proposed in this pull request?
WIP to demonstrate simple vectorized UDFs for Python as a preliminary step
to an expanded API from the SPIP at
https://issues.apache.org/jira/browse/SPARK-21190
## How was this patch tested?
(Please explain how this patch was tested. E.g. unit tests, integration
tests, manual tests)
(If this patch involves UI changes, please attach a screenshot; otherwise,
remove this)
Please review http://spark.apache.org/contributing.html before opening a
pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/BryanCutler/spark
arrow-vectorized-udfs-SPARK-21404
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/18659.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #18659
----
commit d2edd311b2f22d3cb9c3b2519835101727cbb127
Author: Bryan Cutler <[email protected]>
Date: 2017-07-14T00:29:23Z
vectorized udfs working but hardcoded for ArrowPandasSerializer
commit 028a58f1ef0cad71f361094f272033dc1a606bfb
Author: Bryan Cutler <[email protected]>
Date: 2017-07-14T21:20:51Z
Added conf for enabling vectorized UDFs, now working
commit c12f658d2df41d4cc66104a821954e5bc5976f8a
Author: Bryan Cutler <[email protected]>
Date: 2017-07-14T22:38:15Z
Columns for ArrowPandasSerializer need unique name
commit 11a7a87c92825b0d458780586dd6181450bd8e21
Author: Bryan Cutler <[email protected]>
Date: 2017-07-14T22:57:24Z
fixed style checks
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
