Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/19717
The SparkR tests seem flaky all of a sudden; other PRs are also failing. So
I'm just going to ignore that failure.
Merging to master.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19717
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19717
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84726/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19717
**[Test build #84726 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84726/testReport)**
for PR 19717 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19717
**[Test build #84726 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84726/testReport)**
for PR 19717 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19717
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84701/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19717
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19717
**[Test build #84701 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84701/testReport)**
for PR 19717 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19717
**[Test build #84701 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84701/testReport)**
for PR 19717 at commit
Github user felixcheung commented on the issue:
https://github.com/apache/spark/pull/19717
sounds like a good to go from @vanzin and a couple of others too.
any other comment?
---
-
To unsubscribe, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19717
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84672/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19717
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19717
**[Test build #84672 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84672/testReport)**
for PR 19717 at commit
Github user liyinan926 commented on the issue:
https://github.com/apache/spark/pull/19717
@vanzin Created https://issues.apache.org/jira/browse/SPARK-22743 to track
the work on consolidating the common logic for handling driver and executor
memory overhead. Addressed other comments
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19717
**[Test build #84672 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84672/testReport)**
for PR 19717 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19717
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19717
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84658/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19717
**[Test build #84658 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84658/testReport)**
for PR 19717 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19717
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19717
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84656/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19717
**[Test build #84656 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84656/testReport)**
for PR 19717 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19717
**[Test build #84658 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84658/testReport)**
for PR 19717 at commit
Github user liyinan926 commented on the issue:
https://github.com/apache/spark/pull/19717
@vanzin addressed your latest comments in
https://github.com/apache/spark/pull/19717/commits/7d2b30373b2e4d8d5311e10c3f9a62a2d900d568.
PTAL. Thanks!
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19717
**[Test build #84656 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84656/testReport)**
for PR 19717 at commit
Github user mccheah commented on the issue:
https://github.com/apache/spark/pull/19717
@foxish the headless service should be created by the submission client
because of permissioning. Our design allows for different sets of credentials
to be used for first submitting the application
Github user liyinan926 commented on the issue:
https://github.com/apache/spark/pull/19717
> I can't load this review anymore. Seems like github needs to work on
supporting long comment threads...
Yes, I had to try reloading many times...
---
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/19717
I can't load this review anymore. Seems like github needs to work on
supporting long comment threads...
--
Marcelo
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19717
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84646/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19717
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19717
**[Test build #84646 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84646/testReport)**
for PR 19717 at commit
Github user foxish commented on the issue:
https://github.com/apache/spark/pull/19717
> Yes, I agree that we eventually should allow client mode including use
cases that directly create SparkContext
Sorry, in your previous comment, it seemed like you were suggesting
Github user liyinan926 commented on the issue:
https://github.com/apache/spark/pull/19717
> I think we should move the headless service creation into the backend
code - anything essential for the backend to run shouldn't depend on the
submission client/steps.
I agree service
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19717
**[Test build #84646 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84646/testReport)**
for PR 19717 at commit
Github user foxish commented on the issue:
https://github.com/apache/spark/pull/19717
@liyinan926:
> Actually given that in our implementations the executors talk to the
driver through a Kubernetes headless service that is only created by the
submission client. None of the
Github user liyinan926 commented on the issue:
https://github.com/apache/spark/pull/19717
@vanzin I implemented the enforcement in
https://github.com/apache/spark/pull/19717/commits/44c40b1b7d546e4d214aff78c282c768aa0e3d42.
PTAL. Thanks!
---
Github user liyinan926 commented on the issue:
https://github.com/apache/spark/pull/19717
Checking the presence of `kubernetes.default.svc` is not sufficient as the
driver service must exist. When the submission client is not involved, the
driver itself must create that service
Github user foxish commented on the issue:
https://github.com/apache/spark/pull/19717
I understand the concern here. I think we can guard against accidental use
in client mode to post a nicer error message. @liyinan926, how about testing if
`kubernetes.default.svc` is accessible?
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/19717
All existing clusters work with client mode, so you're on your own here...
maybe setting an environment variable when starting the driver and checking
that it's set in your code. Or something.
---
Github user liyinan926 commented on the issue:
https://github.com/apache/spark/pull/19717
Actually given that in our implementations the executors talk to the driver
through a Kubernetes headless service that is only created by the submission
client. I don't think any of the two
Github user liyinan926 commented on the issue:
https://github.com/apache/spark/pull/19717
There are two cases to consider here. 1) If the client that creates the
`SparkContext` is running in a Kubernetes pod, it should work the same way as
the driver pod of an application submitted
Github user liyinan926 commented on the issue:
https://github.com/apache/spark/pull/19717
We do enforce that by:
```
case (KUBERNETES, CLIENT) =>
printErrorAndExit("Client mode is currently not supported for
Kubernetes.")
```
---
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/19717
It's more than a docs issue though, it should be enforced in the code, and
as far as I've seen the code path I mentioned does not generate an error (or at
least not an obvious one).
---
Github user liyinan926 commented on the issue:
https://github.com/apache/spark/pull/19717
@vanzin Sorry I didn't see your question. We don't yet support client mode
and no one has ever tried a true client mode, i.e., the driver runs on the
client machine outside a Kubernetes cluster.
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/19717
I don't think anyone answered my previous question:
> One question though: what happens if someone does new
SparkContext(myConf) in their code with a > k8s URL? That's a valid way of
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19717
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84613/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19717
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19717
**[Test build #84613 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84613/testReport)**
for PR 19717 at commit
Github user liyinan926 commented on the issue:
https://github.com/apache/spark/pull/19717
With some bug fixes in
https://github.com/apache/spark/pull/19717/commits/83d0b9cc29eec2654308344ac8b2208e5e3f08df,
I was able to run `SparkPi` on a GKE cluster.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19717
**[Test build #84613 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84613/testReport)**
for PR 19717 at commit
Github user foxish commented on the issue:
https://github.com/apache/spark/pull/19717
Thanks to all the reviewers, @liyinan926 and @mccheah for the comments &
fixes. This looks real close!
@ifilonenko - if we could get our integration tests to run once against
this PR this week
Github user jiangxb1987 commented on the issue:
https://github.com/apache/spark/pull/19717
LGTM too, thanks for the awesome work!
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user mridulm commented on the issue:
https://github.com/apache/spark/pull/19717
@liyinan926 I dont have any other comments. Given that I am not as familiar
with this part of the codebase as I would have liked to be, I will defer to
others on whether it is ready for merge or
Github user liyinan926 commented on the issue:
https://github.com/apache/spark/pull/19717
@vanzin @jerryshao @felixcheung @viirya @mridulm @ueshin @jiangxb1987
Really appreciate your time reviewing this! Do you have more comments? Given
that we have at least one more PR (most likely
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19717
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19717
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84568/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19717
**[Test build #84568 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84568/testReport)**
for PR 19717 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19717
**[Test build #84568 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84568/testReport)**
for PR 19717 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19717
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84542/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19717
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19717
**[Test build #84542 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84542/testReport)**
for PR 19717 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19717
**[Test build #84542 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84542/testReport)**
for PR 19717 at commit
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/19717
retest this please.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19717
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19717
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84530/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19717
**[Test build #84530 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84530/testReport)**
for PR 19717 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19717
**[Test build #84530 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84530/testReport)**
for PR 19717 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19717
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19717
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84513/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19717
**[Test build #84513 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84513/testReport)**
for PR 19717 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19717
**[Test build #84513 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84513/testReport)**
for PR 19717 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19717
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84503/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19717
**[Test build #84503 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84503/testReport)**
for PR 19717 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19717
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19717
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84500/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19717
**[Test build #84500 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84500/testReport)**
for PR 19717 at commit
Github user liyinan926 commented on the issue:
https://github.com/apache/spark/pull/19717
@vanzin @jerryshao @felixcheung @viirya @mridulm @ueshin @jiangxb1987 all
comments have been addressed/answered. PTAL. Thanks!
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19717
**[Test build #84503 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84503/testReport)**
for PR 19717 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19717
**[Test build #84500 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84500/testReport)**
for PR 19717 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19717
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84453/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19717
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19717
**[Test build #84453 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84453/testReport)**
for PR 19717 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19717
**[Test build #84453 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84453/testReport)**
for PR 19717 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19717
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19717
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84450/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19717
**[Test build #84450 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84450/testReport)**
for PR 19717 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19717
**[Test build #84450 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84450/testReport)**
for PR 19717 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19717
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19717
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84440/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19717
**[Test build #84440 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84440/testReport)**
for PR 19717 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19717
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19717
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84438/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19717
**[Test build #84438 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84438/testReport)**
for PR 19717 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19717
**[Test build #84440 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84440/testReport)**
for PR 19717 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19717
**[Test build #84438 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84438/testReport)**
for PR 19717 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19717
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19717
**[Test build #84437 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84437/testReport)**
for PR 19717 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19717
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84437/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19717
**[Test build #84437 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84437/testReport)**
for PR 19717 at commit
Github user felixcheung commented on the issue:
https://github.com/apache/spark/pull/19717
@vanzin @mridulm @jerryshao would love your comment.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For
Github user felixcheung commented on the issue:
https://github.com/apache/spark/pull/19717
will do a pass on the latest over the weekend
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For
1 - 100 of 127 matches
Mail list logo
