GitHub user e-dorigatti opened a pull request: https://github.com/apache/spark/pull/21467
[SPARK-23754][Python] Fix UDF hack ## What changes were proposed in this pull request? SPARK-23754 was fixed in #21383 by changing the UDF code to wrap the user function, but this required a hack to save its argspec. This PR reverts this change and fixes the `StopIteration` bug in the worker @HyukjinKwon ## How was this patch tested? Same tests, plus tests for pandas UDFs You can merge this pull request into a Git repository by running: $ git pull https://github.com/e-dorigatti/spark fix_udf_hack Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/21467.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #21467 ---- commit c4d2e681b9915a390cc77816de603ae6e0a58da9 Author: edorigatti <emilio.dorigatti@...> Date: 2018-05-31T11:42:58Z moved StopIteration guard from UDF to worker commit 81ae94602f6e3f9ef02bc3f6b9824a69aa0a2352 Author: edorigatti <emilio.dorigatti@...> Date: 2018-05-31T11:43:28Z test StopIteration for pandas UDFs commit b085fe9070ffa60c50d5e68ffaccc11cef25c443 Author: edorigatti <emilio.dorigatti@...> Date: 2018-05-31T12:10:15Z pep8 commit 55d5de534b3000c0cb48cfd2a9f1671e44913706 Author: edorigatti <emilio.dorigatti@...> Date: 2018-05-31T12:18:11Z removed argspec hack ---- --- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org