HyukjinKwon opened a new pull request #28845:
URL: https://github.com/apache/spark/pull/28845
### What changes were proposed in this pull request?
This PR proposes to remove the warning about multi-thread in local
properties, and change the guide to use `collectWithJobGroup` for multi-threads
for now because:
- It is too noisy to users who don't use multiple threads - the number of
this single thread case is arguably more prevailing.
- There was a critical issue found about pin-thread mode SPARK-32010, which
will be fixed in Spark 3.1.
- To smoothly migrate, `RDD.collectWithJobGroup` was added, which will be
deprecated in Spark 3.1 with SPARK-32010 fixed.
I will target to deprecate `RDD.collectWithJobGroup`, and make this
pin-thread mode stable in Spark 3.1. In the future releases, I plan to make
this mode as a default mode, and remove `RDD.collectWithJobGroup` away.
### Why are the changes needed?
To avoid guiding users a feature with a critical issue, and provide a proper
workaround for now.
### Does this PR introduce _any_ user-facing change?
Yes, warning message and documentation.
### How was this patch tested?
Manually tested:
Before:
```
>>> spark.sparkContext.setLocalProperty("a", "b")
/.../spark/python/pyspark/util.py:141: UserWarning: Currently,
'setLocalProperty' (set to local
properties) with multiple threads does not properly work.
Internally threads on PVM and JVM are not synced, and JVM thread can be
reused for multiple
threads on PVM, which fails to isolate local properties for each thread on
PVM.
To work around this, you can set PYSPARK_PIN_THREAD to true (see
SPARK-22340). However,
note that it cannot inherit the local properties from the parent thread
although it isolates each
thread on PVM and JVM with its own local properties.
To work around this, you should manually copy and set the local properties
from the parent thread
to the child thread when you create another thread.
```
After:
```
>>> spark.sparkContext.setLocalProperty("a", "b")
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
