Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/3561#issuecomment-68465783
Alright, I've merged this to `master` (1.3.0), `branch-1.2` (1.2.1),
`branch-1.1` (1.1.2), and `branch-1.0` (1.0.3).
---
If your project is set up for it, you can repl
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/3561
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enab
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/3561#issuecomment-68465381
I finished my backports of the other patch, so I'm going to merge this now.
Thanks!
---
If your project is set up for it, you can reply to this email and have your
re
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3561#issuecomment-68462940
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3561#issuecomment-68462934
[Test build #24961 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24961/consoleFull)
for PR 3561 at commit
[`5e0fce1`](https://gith
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3561#issuecomment-68457422
[Test build #24961 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24961/consoleFull)
for PR 3561 at commit
[`5e0fce1`](https://githu
Github user brennonyork commented on the pull request:
https://github.com/apache/spark/pull/3561#issuecomment-68457321
@JoshRosen took care of the minor edits for ya!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If you
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/3561#issuecomment-68425921
Looks like that's some unrelated test flakiness, since all tests are now
passing. Since it doesn't seem like my PR broke any tests, let me go ahead and
finish backport
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3561#issuecomment-68425728
[Test build #24938 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24938/consoleFull)
for PR 3561 at commit
[`14daa20`](https://gith
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3561#issuecomment-68425731
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3561#issuecomment-68423483
[Test build #24938 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24938/consoleFull)
for PR 3561 at commit
[`14daa20`](https://githu
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/3561#issuecomment-68423373
Jenkins, retest this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not ha
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/3561#issuecomment-68423309
I'm still investigating; it might be caused by my PR, but it's not failing
deterministically in all builds so I'm not sure. I can dig in, but I'm sure
it's not caused
Github user brennonyork commented on the pull request:
https://github.com/apache/spark/pull/3561#issuecomment-68423286
Is `org.apache.spark.storage.BlockNotFoundException: Block taskresult_0 not
found` that a common timing-related test issue? If not I can definitely look
into what mig
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3561#issuecomment-68420950
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3561#issuecomment-68420947
[Test build #24924 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24924/consoleFull)
for PR 3561 at commit
[`14daa20`](https://gith
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3561#issuecomment-68418650
[Test build #24924 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24924/consoleFull)
for PR 3561 at commit
[`14daa20`](https://githu
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/3561#issuecomment-68418611
I've merged #3739 into `master`, so this should be ready to retest.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitH
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/3561#issuecomment-68418615
Jenkins, retest this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not ha
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/3561#discussion_r22264611
--- Diff:
core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala ---
@@ -123,14 +124,21 @@ private[spark] class SparkSubmitArguments(args:
Github user brennonyork commented on the pull request:
https://github.com/apache/spark/pull/3561#issuecomment-67673409
@JoshRosen great catch! Sounds like this can't be accepted until #3739 is
completed, but glad we have a resolution.
---
If your project is set up for it, you can rep
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/3561#issuecomment-67587796
I've opened #3739 to try to systematically clean up our explicit usages of
`System.setProperty` in the test code, as well as to address the implicit calls
that broke th
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/3561#issuecomment-67583605
Aha, I found the problem: it looks like tests in SparkSubmitSuite end up
calling SparkSubmit.main(), which sets system properties. I added some debug
logging:
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/3561#issuecomment-67548764
Maybe SparkSubmit threw an exception somewhere but it's swallowed in the
tests?
---
If your project is set up for it, you can reply to this email and have your
reply
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/3561#discussion_r22067557
--- Diff:
core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala ---
@@ -123,14 +124,21 @@ private[spark] class SparkSubmitArguments(args:
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/3561#discussion_r22067423
--- Diff:
core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala ---
@@ -123,14 +124,21 @@ private[spark] class SparkSubmitArguments(args:
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/3561#issuecomment-67547618
@brennonyork The pull request builder actually tests the result of merging
the upstream master into your branch, so you don't need to do anything to bring
this branch u
Github user brennonyork commented on the pull request:
https://github.com/apache/spark/pull/3561#issuecomment-67406314
@JoshRosen what if I merge the upstream master into this branch? Not sure
if I might've just branched at an odd time since I can't see any reason why
this little bit
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3561#issuecomment-67401688
[Test build #24554 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24554/consoleFull)
for PR 3561 at commit
[`14daa20`](https://gith
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3561#issuecomment-67401696
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3561#issuecomment-67392723
[Test build #24554 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24554/consoleFull)
for PR 3561 at commit
[`14daa20`](https://githu
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/3561#issuecomment-67391773
Jenkins, retest this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not ha
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/3561#issuecomment-67383788
I've seen this failure before. In the past it's caused by too many tests
running at once and contending for resources on the test machines. This may or
may not be rela
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/3561#issuecomment-67378920
This looks like a new issue, so I'm going to hold off on merging until I
figure it out. It looks like most of JavaAPISuite, DistributedSuite, and
FileServerSuite faile
Github user brennonyork commented on the pull request:
https://github.com/apache/spark/pull/3561#issuecomment-67366887
@JoshRosen this seems to be a new error with the test cases that wasn't
present last time (the Spark streaming issue). Any thoughts on why this code
would break this
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3561#issuecomment-67291599
[Test build #24538 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24538/consoleFull)
for PR 3561 at commit
[`14daa20`](https://gith
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3561#issuecomment-67291611
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3561#issuecomment-67286818
[Test build #24538 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24538/consoleFull)
for PR 3561 at commit
[`14daa20`](https://githu
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/3561#issuecomment-67286547
Jenkins, retest this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not ha
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3561#issuecomment-67224745
[Test build #24501 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24501/consoleFull)
for PR 3561 at commit
[`14daa20`](https://gith
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3561#issuecomment-67224757
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3561#issuecomment-67215790
[Test build #24501 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24501/consoleFull)
for PR 3561 at commit
[`14daa20`](https://githu
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/3561#issuecomment-67215577
Jenkins, retest this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not ha
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3561#issuecomment-67106502
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3561#issuecomment-67106493
[Test build #24480 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24480/consoleFull)
for PR 3561 at commit
[`14daa20`](https://gith
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3561#issuecomment-67102441
[Test build #24480 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24480/consoleFull)
for PR 3561 at commit
[`14daa20`](https://githu
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/3561#issuecomment-67102056
Jenkins, retest this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not ha
Github user brennonyork commented on a diff in the pull request:
https://github.com/apache/spark/pull/3561#discussion_r21871232
--- Diff:
core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala ---
@@ -123,14 +124,21 @@ private[spark] class SparkSubmitArguments(args:
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/3561#discussion_r21870538
--- Diff:
core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala ---
@@ -123,14 +124,21 @@ private[spark] class SparkSubmitArguments(args:
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/3561#issuecomment-67093951
This looks good to me. If this was a new feature, I'd probably say that
it's confusing that it works for `file://` URIs and not `hdfs://` ones, but
since we already ha
Github user brennonyork commented on the pull request:
https://github.com/apache/spark/pull/3561#issuecomment-67070428
Bump, @andrewor14 and @JoshRosen, any updates / issues with the modified
code?
---
If your project is set up for it, you can reply to this email and have your
reply
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3561#issuecomment-66707407
[Test build #24382 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24382/consoleFull)
for PR 3561 at commit
[`14daa20`](https://gith
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3561#issuecomment-66707412
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3561#issuecomment-66704418
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3561#issuecomment-66704409
[Test build #24380 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24380/consoleFull)
for PR 3561 at commit
[`c6dad68`](https://gith
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3561#issuecomment-66699836
[Test build #24382 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24382/consoleFull)
for PR 3561 at commit
[`14daa20`](https://githu
Github user brennonyork commented on the pull request:
https://github.com/apache/spark/pull/3561#issuecomment-66699387
@andrewor14 updated per all the relevant requests and retested everything!
:) Any other issues?
---
If your project is set up for it, you can reply to this email and
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/3561#discussion_r21712856
--- Diff:
core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala ---
@@ -123,14 +124,25 @@ private[spark] class SparkSubmitArguments(args:
Github user brennonyork commented on a diff in the pull request:
https://github.com/apache/spark/pull/3561#discussion_r21712580
--- Diff:
core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala ---
@@ -123,14 +124,25 @@ private[spark] class SparkSubmitArguments(args:
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/3561#discussion_r21712174
--- Diff:
core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala ---
@@ -123,14 +124,25 @@ private[spark] class SparkSubmitArguments(args:
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/3561#discussion_r21712139
--- Diff:
core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala ---
@@ -123,14 +124,25 @@ private[spark] class SparkSubmitArguments(args:
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/3561#discussion_r21712122
--- Diff:
core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala ---
@@ -123,14 +124,25 @@ private[spark] class SparkSubmitArguments(args:
Github user brennonyork commented on the pull request:
https://github.com/apache/spark/pull/3561#issuecomment-66692787
@JoshRosen and @andrewor14, updated to add a match statement for the URI
supporting the `file://` scheme. Figured a case would be best if we wanted to
look at support
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3561#issuecomment-66692751
[Test build #24380 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24380/consoleFull)
for PR 3561 at commit
[`c6dad68`](https://githu
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3561#issuecomment-66388765
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3561#issuecomment-66388758
[Test build #24272 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24272/consoleFull)
for PR 3561 at commit
[`8d20936`](https://gith
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/3561#issuecomment-66383770
Even if we're only going to support the `file://` scheme for now, we should
add a check that the URI actually uses the `file://` scheme before attempting
to strip it of
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3561#issuecomment-66380538
[Test build #24272 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24272/consoleFull)
for PR 3561 at commit
[`8d20936`](https://githu
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/3561#issuecomment-66379873
We could... but we'll need to re-run the tests anyway after you make your
changes. Retest this please
---
If your project is set up for it, you can reply to this emai
Github user brennonyork commented on the pull request:
https://github.com/apache/spark/pull/3561#issuecomment-66379089
Given that we're only supporting the `file` scheme then, should we rerun
the flaky streaming test to ensure everything is okay?
---
If your project is set up for it,
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/3561#issuecomment-66367737
Hey it seems that we can only support this feature when the jar is local
(i.e. with the `file:/` URI scheme). In general I think it's hard to support
this for all sche
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/3561#issuecomment-66168555
Hmm, it looks like there's already a JIRA for that particular test's
flakiness: [SPARK-1600](https://issues.apache.org/jira/browse/SPARK-1600).
---
If your project is
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/3561#issuecomment-66168296
It looks like that failure is due to a (known) flaky Spark Streaming test:
```
[info] CheckpointSuite:
[info] - basic rdd checkpoints + dstream graph che
Github user brennonyork commented on the pull request:
https://github.com/apache/spark/pull/3561#issuecomment-66163505
@JoshRosen I'm pretty sure we can definitely support the `hdfs://` URI
model. I'll look and see if, given an `hdfs://` URI, Spark would already have
some sort of Hado
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3561#issuecomment-65732928
[Test build #24150 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24150/consoleFull)
for PR 3561 at commit
[`8d20936`](https://gith
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3561#issuecomment-65732932
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/3561#issuecomment-65729038
*EDIT*: mistakenly had `file://` instead of `hdfs://` in earlier comment.
---
If your project is set up for it, you can reply to this email and have your
reply appear o
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/3561#issuecomment-65728956
This sounds good to me, although there's one edge case that I'm curious
about: what if my main JAR is hosted in a shared cluster filesystem and has a
URI like `file://p
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3561#issuecomment-65727285
[Test build #24150 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24150/consoleFull)
for PR 3561 at commit
[`8d20936`](https://githu
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/3561#issuecomment-65726913
Jenkins, this is ok to test.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not ha
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3561#issuecomment-65292887
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your pro
GitHub user brennonyork opened a pull request:
https://github.com/apache/spark/pull/3561
[SPARK-4298][Core] - The spark-submit cannot read Main-Class from Manifest.
Resolves a bug where the `Main-Class` from a .jar file wasn't being read in
properly. This was caused by the fact that
82 matches
Mail list logo
