Github user davies closed the pull request at:
https://github.com/apache/spark/pull/2935
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enab
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/2935#issuecomment-61337729
If we can't find a workaround to speed up the deserialization, do you mind
closing this PR for now?
---
If your project is set up for it, you can reply to this email a
Github user davies commented on the pull request:
https://github.com/apache/spark/pull/2935#issuecomment-60837619
The timeout of jenkins test is caused by performance regression introduced
in this PR. Deserialization of Hadoop Configuration is pretty slow, after
removing broadcast for
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/2935#issuecomment-60727942
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2935#issuecomment-60727933
**[Test build #22347 timed
out](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22347/consoleFull)**
for PR 2935 at commit
[`5329091`](https://git
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2935#issuecomment-60727880
**[Test build #482 timed
out](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/482/consoleFull)**
for PR 2935 at commit
[`63f2972`](https://gith
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2935#issuecomment-60717934
[Test build #22347 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22347/consoleFull)
for PR 2935 at commit
[`5329091`](https://githu
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2935#issuecomment-60717907
[Test build #482 has
started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/482/consoleFull)
for PR 2935 at commit
[`63f2972`](https://github
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/2935#issuecomment-60512756
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2935#issuecomment-60512753
**[Test build #22240 timed
out](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22240/consoleFull)**
for PR 2935 at commit
[`63f2972`](https://git
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2935#issuecomment-60512579
**[Test build #452 timed
out](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/452/consoleFull)**
for PR 2935 at commit
[`ae32e92`](https://gith
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2935#issuecomment-60509909
[Test build #22240 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22240/consoleFull)
for PR 2935 at commit
[`63f2972`](https://githu
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2935#issuecomment-60509760
[Test build #452 has
started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/452/consoleFull)
for PR 2935 at commit
[`ae32e92`](https://github
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2935#issuecomment-60500871
[Test build #2 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/2/consoleFull)
for PR 2935 at commit
[`ae32e92`](https://gith
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/2935#issuecomment-60500873
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2935#issuecomment-60500398
**[Test build #1 timed
out](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/1/consoleFull)**
for PR 2935 at commit
[`1fd70df`](https://git
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/2935#issuecomment-60500400
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2935#issuecomment-60497976
[Test build #2 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/2/consoleFull)
for PR 2935 at commit
[`ae32e92`](https://githu
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2935#issuecomment-60497275
[Test build #1 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/1/consoleFull)
for PR 2935 at commit
[`1fd70df`](https://githu
Github user davies commented on the pull request:
https://github.com/apache/spark/pull/2935#issuecomment-60497056
Jenkins, test this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have th
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2935#issuecomment-60495718
**[Test build #449 timed
out](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/449/consoleFull)**
for PR 2935 at commit
[`1fd70df`](https://gith
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2935#issuecomment-60495652
**[Test build #448 timed
out](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/448/consoleFull)**
for PR 2935 at commit
[`1fd70df`](https://gith
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2935#issuecomment-60491390
[Test build #449 has
started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/449/consoleFull)
for PR 2935 at commit
[`1fd70df`](https://github
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2935#issuecomment-60491324
[Test build #448 has
started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/448/consoleFull)
for PR 2935 at commit
[`1fd70df`](https://github
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2935#issuecomment-60476056
**[Test build #432 timed
out](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/432/consoleFull)**
for PR 2935 at commit
[`32bd815`](https://gith
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/2935#issuecomment-60476010
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2935#issuecomment-60476007
**[Test build #22201 timed
out](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22201/consoleFull)**
for PR 2935 at commit
[`1fd70df`](https://git
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2935#issuecomment-60474421
[Test build #22195 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22195/consoleFull)
for PR 2935 at commit
[`32bd815`](https://gith
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/2935#issuecomment-60474423
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2935#issuecomment-60474251
[Test build #22194 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22194/consoleFull)
for PR 2935 at commit
[`8694cb3`](https://gith
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/2935#issuecomment-60474253
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2935#issuecomment-60473598
[Test build #432 has
started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/432/consoleFull)
for PR 2935 at commit
[`32bd815`](https://github
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2935#issuecomment-60473554
[QA tests have
finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/425/consoleFull)
for PR 2935 at commit
[`bc46dda`](https://github.com/
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2935#issuecomment-60473566
[Test build #22201 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22201/consoleFull)
for PR 2935 at commit
[`1fd70df`](https://githu
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/2935#issuecomment-60473288
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2935#issuecomment-60473287
[QA tests have
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22192/consoleFull)
for PR 2935 at commit
[`bc46dda`](https://github.com/a
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/2935#issuecomment-60472516
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2935#issuecomment-60472470
[Test build #22195 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22195/consoleFull)
for PR 2935 at commit
[`32bd815`](https://githu
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2935#issuecomment-60472375
[QA tests have
started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/425/consoleFull)
for PR 2935 at commit
[`bc46dda`](https://github.com/a
Github user davies commented on the pull request:
https://github.com/apache/spark/pull/2935#issuecomment-60472321
@JoshRosen this PR is ready to review, thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your proj
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2935#issuecomment-60472283
[Test build #22194 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22194/consoleFull)
for PR 2935 at commit
[`8694cb3`](https://githu
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2935#issuecomment-60471977
[QA tests have
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22192/consoleFull)
for PR 2935 at commit
[`bc46dda`](https://github.com/ap
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/2935#discussion_r19373926
--- Diff: core/src/main/scala/org/apache/spark/SerializableWritable.scala
---
@@ -38,8 +38,10 @@ class SerializableWritable[T <: Writable](@transient var
t: T
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/2935#issuecomment-60468679
Yeah, we'll still need the synchronization there, but I was hoping to add
the `newConfiguration()` to address the use-cases in Streaming, where they
construct new confi
Github user davies commented on the pull request:
https://github.com/apache/spark/pull/2935#issuecomment-60468274
But inside readFields(), it may call new Configuration(), so we still need
to synchronize it here.
---
If your project is set up for it, you can reply to this email and
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/2935#issuecomment-60467588
Oh, one more comment: could you add a `newConfiguration()` method to
`SparkHadoopUtil` that synchronizes on the lock and returns `new
Configuration()`? This would help
46 matches
Mail list logo
