Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/3658
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/3658#issuecomment-71710576
Hey @pwendell, I'll try to get to this soon. But I wanted to get your
feedback on my idea for fixing the `network/` dependencies thing before I try
to implement it.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3658#issuecomment-71762783
[Test build #26199 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26199/consoleFull)
for PR 3658 at commit
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/3658#issuecomment-71765771
LGTM pending tests. @vanzin do you have more work you'd like to do on this
or are you happy with it?
---
If your project is set up for it, you can reply to this email
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3658#issuecomment-71769526
[Test build #26199 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26199/consoleFull)
for PR 3658 at commit
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/3658#issuecomment-71747360
@vanzin yeah this is a fair point, this would mean that network/common
would expose the (un-shaded) Guava classes... a bit clunky. If those classes
change in the future
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/3658#issuecomment-71773844
No, I'm done with it. Thanks for taking a look.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3658#issuecomment-71769535
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/3658#discussion_r23565016
--- Diff: pom.xml ---
@@ -1276,6 +1279,19 @@
includeorg.spark-project.spark:unused/include
/includes
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/3658#discussion_r23564938
--- Diff: core/pom.xml ---
@@ -408,6 +372,20 @@
/execution
/executions
/plugin
+ plugin
+
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/3658#discussion_r23566689
--- Diff: core/pom.xml ---
@@ -408,6 +372,20 @@
/execution
/executions
/plugin
+ plugin
+
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/3658#issuecomment-71544279
Hey so a couple questions about this. This does make things simpler overall
in terms of the build. Some higher level questions:
1. This adds 1,500 class files
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/3658#issuecomment-71545834
Does this suggest any limits on how many overall dependencies we can
shade in Spark longer term (i.e. in addition to guava)?
If we choose relocation as the
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3658#issuecomment-71547066
[Test build #26116 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26116/consoleFull)
for PR 3658 at commit
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/3658#issuecomment-71550179
Yeah, so I think it would be better to just shade guava in YARN as well.
The main dependency-related constraint was that we didn't want Spark's yarn
shuffle service to
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/3658#issuecomment-71575594
Hey @vanzin mind bringing this up to date now (I merged #3874).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3658#issuecomment-71557798
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3658#issuecomment-71557787
[Test build #26116 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26116/consoleFull)
for PR 3658 at commit
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/3658#issuecomment-71236254
Ping.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3658#issuecomment-69804680
[Test build #25477 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25477/consoleFull)
for PR 3658 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3658#issuecomment-69804693
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/3658#issuecomment-69827040
@pwendell would appreciate some input here. Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3658#issuecomment-69678894
[Test build #25442 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25442/consoleFull)
for PR 3658 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3658#issuecomment-69684171
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3658#issuecomment-69684164
[Test build #25442 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25442/consoleFull)
for PR 3658 at commit
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3658#issuecomment-68974879
[Test build #25133 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25133/consoleFull)
for PR 3658 at commit
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3658#issuecomment-68978591
[Test build #25133 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25133/consoleFull)
for PR 3658 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3658#issuecomment-68978595
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3658#issuecomment-68077871
[Test build #24792 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24792/consoleFull)
for PR 3658 at commit
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3658#issuecomment-68079908
[Test build #24792 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24792/consoleFull)
for PR 3658 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3658#issuecomment-68079909
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
GitHub user vanzin opened a pull request:
https://github.com/apache/spark/pull/3658
[SPARK-4809] Rework Guava library shading.
The current way of shading Guava is a little problematic. Code that
depends on spark-core does not see the transitive dependency, yet
classes in
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/3658#issuecomment-66393496
I ran the maven build + unit tests, and also audited the jar files in the
build using the tool I wrote:
https://gist.github.com/vanzin/bd9057fadf4a296220b7
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3658#issuecomment-66393963
[Test build #24285 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24285/consoleFull)
for PR 3658 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3658#issuecomment-66399893
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3658#issuecomment-66399889
[Test build #24285 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24285/consoleFull)
for PR 3658 at commit
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/3658#issuecomment-66404298
BTW, credit where credit is due, I got this idea from @arahuja
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as
37 matches
Mail list logo
