GitHub user gatorsmile opened a pull request: https://github.com/apache/spark/pull/20137
[SPARK-22939] [PySpark] Support Spark UDF in registerFunction [WIP] ## What changes were proposed in this pull request? ```Python import random from pyspark.sql.functions import udf from pyspark.sql.types import IntegerType, StringType random_udf = udf(lambda: int(random.random() * 100), IntegerType()).asNondeterministic() spark.catalog.registerFunction("random_udf", random_udf, StringType()) spark.sql("SELECT random_udf()").collect() ``` We will get the following error. ``` Py4JError: An error occurred while calling o29.__getnewargs__. Trace: py4j.Py4JException: Method __getnewargs__([]) does not exist at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:318) at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:326) at py4j.Gateway.invoke(Gateway.java:274) at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) at py4j.commands.CallCommand.execute(CallCommand.java:79) at py4j.GatewayConnection.run(GatewayConnection.java:214) at java.lang.Thread.run(Thread.java:745) ``` This PR is to support it. ## How was this patch tested? WIP You can merge this pull request into a Git repository by running: $ git pull https://github.com/gatorsmile/spark registerFunction Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/20137.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #20137 ---- commit 8216b6bb52082883fc9b212cd9ab21227f2b8491 Author: gatorsmile <gatorsmile@...> Date: 2018-01-02T15:28:19Z wip ---- --- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org