GitHub user BryanCutler opened a pull request: https://github.com/apache/spark/pull/16782
[SPARK-19348][PYTHON][WIP] PySpark keyword_only decorator is not thread-safe ## What changes were proposed in this pull request? The `@keyword_only` decorator in PySpark is not thread-safe. It writes kwargs to a static class variable in the decorator, which is then retrieved later in the class method as `_input_kwargs`. If multiple threads are constructing the same class with different kwargs, it becomes a race condition to read from the static class variable before it's overwritten. See [SPARK-19348](https://issues.apache.org/jira/browse/SPARK-19348) for reproduction code. This change will write the kwargs to a member variable so that multiple threads can operate on separate instances without the race condition. It does not protect against multiple threads operating on a single instance, but that is better left to the user to synchronize. ## How was this patch tested? WIP You can merge this pull request into a Git repository by running: $ git pull https://github.com/BryanCutler/spark pyspark-keyword_only-threadsafe-SPARK-19348 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/16782.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #16782 ---- commit a15e9b0a78d86685d6b27ac4fdfa37161376b288 Author: Bryan Cutler <cutl...@gmail.com> Date: 2017-02-02T23:01:45Z made a thread-safe addition to keyword_only decorator commit 83bcce0c2fc33184579224fca18066dbf6891d26 Author: Bryan Cutler <cutl...@gmail.com> Date: 2017-02-02T23:11:32Z remove blank line ---- --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org