Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/2624#issuecomment-58143439
@mateiz Aside from restoring the `getThreadLocal` method in order to
preserve API compatibility, is this patch otherwise ready to merge?
---
If your project is set up
Github user davies commented on the pull request:
https://github.com/apache/spark/pull/2624#issuecomment-58143704
@mateiz @JoshRosen I had put getThreadLocal() back and deprecated it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2624#issuecomment-58143977
[QA tests have
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21368/consoleFull)
for PR 2624 at commit
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2624#issuecomment-58149038
[QA tests have
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21368/consoleFull)
for PR 2624 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/2624#issuecomment-58149042
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user mateiz commented on the pull request:
https://github.com/apache/spark/pull/2624#issuecomment-58241504
Yup, looks good! I'm going to merge it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/2624
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is
Github user mateiz commented on a diff in the pull request:
https://github.com/apache/spark/pull/2624#discussion_r18498163
--- Diff: core/src/main/scala/org/apache/spark/SparkEnv.scala ---
@@ -119,30 +118,20 @@ class SparkEnv (
}
object SparkEnv extends Logging {
Github user mateiz commented on a diff in the pull request:
https://github.com/apache/spark/pull/2624#discussion_r18498152
--- Diff: core/src/main/scala/org/apache/spark/SparkEnv.scala ---
@@ -119,30 +118,20 @@ class SparkEnv (
}
object SparkEnv extends Logging {
Github user mateiz commented on a diff in the pull request:
https://github.com/apache/spark/pull/2624#discussion_r18498170
--- Diff: core/src/main/scala/org/apache/spark/SparkEnv.scala ---
@@ -119,30 +118,20 @@ class SparkEnv (
}
object SparkEnv extends Logging {
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/2624#discussion_r18498596
--- Diff: core/src/main/scala/org/apache/spark/SparkEnv.scala ---
@@ -119,30 +118,20 @@ class SparkEnv (
}
object SparkEnv extends Logging {
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2624#issuecomment-57758615
[QA tests have
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21237/consoleFull)
for PR 2624 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/2624#issuecomment-57758618
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user tdas commented on the pull request:
https://github.com/apache/spark/pull/2624#issuecomment-57840767
Okay, this is a pretty significant change to remove the threadlocal object
completely. There are two things we can do
- Either we can do something minimal to just clear
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/2624#issuecomment-57850587
Either we can do something minimal to just clear the reference, so that
repeated sparkContext creation works from pySpark.
I'm not sure that there's an easy,
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/2624#issuecomment-57853572
It's worth noting that the ThreadLocals haven't seemed to cause problems in
any of the existing uses of Spark / PySpark. In PySpark Streaming, I think
we're running
Github user tdas commented on the pull request:
https://github.com/apache/spark/pull/2624#issuecomment-57892709
@JoshRosen the simple fix is to delete the threadlocal variable completely.
Then any access to the threadlocal variable from any thread (even threadpool in
Py4J) is going
GitHub user davies opened a pull request:
https://github.com/apache/spark/pull/2624
[SPARK-3762] clear reference of SparkEnv after stop
SparkEnv is cached in ThreadLocal object, so after stop and create a new
SparkContext, old SparkEnv is still used by some threads, it will trigger
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2624#issuecomment-57592595
[QA tests have
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21184/consoleFull)
for PR 2624 at commit
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2624#issuecomment-57598250
[QA tests have
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21184/consoleFull)
for PR 2624 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/2624#issuecomment-57598254
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/2624#discussion_r18371260
--- Diff: core/src/main/scala/org/apache/spark/SparkEnv.scala ---
@@ -130,6 +133,12 @@ object SparkEnv extends Logging {
env.set(e)
}
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/2624#issuecomment-57718816
It looks like SparkEnv has two getter methods: `get`, which returns
either the ThreadLocal value or the last SparkEnv set _by any thread_, and
`getThreadLocal`, which
Github user davies commented on the pull request:
https://github.com/apache/spark/pull/2624#issuecomment-57721319
@JoshRosen I'd like to move in this way, but getThreadLocal() is a public
API, I'm afraid to remove it.
In the past, I remembered that we can have two different
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/2624#issuecomment-57721877
@davies The SparkEnv class is marked `@developerAPI` and has this note in
its Scaladoc:
```
* NOTE: This is not intended for external use. This is exposed
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/2624#issuecomment-57722450
@davies:
In the past, I remembered that we can have two different SparkEnv in the
same JVM, a different SparkEnv for executor to hold different
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2624#issuecomment-57729754
[QA tests have
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21232/consoleFull)
for PR 2624 at commit
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/2624#issuecomment-57730092
We should probably update the docstrings to remove all references to
ThreadLocal, too.
---
If your project is set up for it, you can reply to this email and have your
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/2624#issuecomment-57733279
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2624#issuecomment-57733269
[QA tests have
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21232/consoleFull)
for PR 2624 at commit
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/2624#issuecomment-57739142
Hmm, strange test failure:
```
[info] - block generator throttling *** FAILED ***
[info] org.scalatest.exceptions.TestFailedException was thrown.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2624#issuecomment-57751658
[QA tests have
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21237/consoleFull)
for PR 2624 at commit
32 matches
Mail list logo
