Github user markgrover commented on the pull request:
https://github.com/apache/spark/pull/9118#issuecomment-148409085
LGTM too, the tests seem to have passed as well.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If
Github user markgrover commented on a diff in the pull request:
https://github.com/apache/spark/pull/9118#discussion_r42047771
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ---
@@ -340,6 +340,14 @@ private[spark] class Client(
&quo
Github user markgrover commented on a diff in the pull request:
https://github.com/apache/spark/pull/9118#discussion_r42042756
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ---
@@ -340,6 +340,14 @@ private[spark] class Client(
&quo
Github user markgrover commented on a diff in the pull request:
https://github.com/apache/spark/pull/9118#discussion_r42042383
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ---
@@ -340,6 +340,14 @@ private[spark] class Client(
&quo
Github user markgrover commented on a diff in the pull request:
https://github.com/apache/spark/pull/9118#discussion_r42042421
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ---
@@ -340,6 +340,14 @@ private[spark] class Client(
&quo
Github user markgrover commented on the pull request:
https://github.com/apache/spark/pull/8998#issuecomment-146006957
Pointing out the obvious here, but we should document this new property so
someone doesn't have to read the source code to figure out what's the name of
th
Github user markgrover commented on a diff in the pull request:
https://github.com/apache/spark/pull/8007#discussion_r37793701
--- Diff:
yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala ---
@@ -213,8 +216,11 @@ private[yarn] class YarnAllocator(
* Deal
Github user markgrover commented on the pull request:
https://github.com/apache/spark/pull/7739#issuecomment-133203764
Thanks Hari! The YarnClusterSuite change is rather benign and not
absolutely necessary. So, based on this I believe this request is ready to be
merged.
And
Github user markgrover commented on the pull request:
https://github.com/apache/spark/pull/8007#issuecomment-133203614
Ah, makes sense, thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user markgrover commented on the pull request:
https://github.com/apache/spark/pull/8007#issuecomment-133196307
> Your assumption probably holds for the preemption case, since it's YARN
killing the container. But I can imagine that if the container exits by itself,
it
Github user markgrover commented on the pull request:
https://github.com/apache/spark/pull/7739#issuecomment-133154333
Jenkins, test this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user markgrover commented on the pull request:
https://github.com/apache/spark/pull/7739#issuecomment-132752506
Hey @harishreedharan about
> Could you also add some tests to each of the components (YARN/standalone
and mesos to ensure the correct classpaths get passed in
Github user markgrover commented on a diff in the pull request:
https://github.com/apache/spark/pull/7739#discussion_r37457145
--- Diff: core/src/main/scala/org/apache/spark/SparkConf.scala ---
@@ -479,6 +484,23 @@ class SparkConf(loadDefaults: Boolean) extends
Cloneable with
Github user markgrover commented on a diff in the pull request:
https://github.com/apache/spark/pull/7739#discussion_r37456912
--- Diff: core/src/main/scala/org/apache/spark/SparkConf.scala ---
@@ -479,6 +484,23 @@ class SparkConf(loadDefaults: Boolean) extends
Cloneable with
Github user markgrover commented on a diff in the pull request:
https://github.com/apache/spark/pull/8007#discussion_r37367588
--- Diff:
yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala ---
@@ -207,6 +211,17 @@ private[yarn] class YarnAllocator
Github user markgrover commented on the pull request:
https://github.com/apache/spark/pull/8007#issuecomment-132367263
>Actually @markgrover can you describe in more detail how you were trying
to use GetExecutorLossReason?
So, I uploaded my
[YARN](https://gist.github.
Github user markgrover commented on the pull request:
https://github.com/apache/spark/pull/8007#issuecomment-132292001
Yeah, for sure, let me upload the logs on gist and explain what I am
talking about. Will do that soon.
---
If your project is set up for it, you can reply to this
Github user markgrover commented on the pull request:
https://github.com/apache/spark/pull/8093#issuecomment-132291876
Thanks Marcelo, the code more or less works but GetExecutorLossReason is
getting called twice. So, I need to work that out with @mccheah. Once that's
resolv
Github user markgrover commented on a diff in the pull request:
https://github.com/apache/spark/pull/8007#discussion_r37108635
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/YarnSchedulerBackend.scala
---
@@ -91,6 +92,66 @@ private[spark] abstract class
Github user markgrover commented on a diff in the pull request:
https://github.com/apache/spark/pull/8007#discussion_r37101551
--- Diff:
yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala ---
@@ -207,6 +211,17 @@ private[yarn] class YarnAllocator
Github user markgrover commented on the pull request:
https://github.com/apache/spark/pull/8093#issuecomment-131185171
I am have ported (aka stolen) some relevant ideas from #8007 here, just
pushed another commit. I still have some concerns regarding the testing
thereof, so I am
Github user markgrover commented on a diff in the pull request:
https://github.com/apache/spark/pull/8007#discussion_r37096380
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/YarnSchedulerBackend.scala
---
@@ -91,6 +92,66 @@ private[spark] abstract class
Github user markgrover commented on the pull request:
https://github.com/apache/spark/pull/8007#issuecomment-131174401
Also, I am borrowing some of your ideas in my pull request and
indadvertedly validating some of your work:-) So far, so good but I am getting
quite a few exceptions
Github user markgrover commented on a diff in the pull request:
https://github.com/apache/spark/pull/8007#discussion_r37093692
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala
---
@@ -239,10 +240,10 @@ class
Github user markgrover commented on a diff in the pull request:
https://github.com/apache/spark/pull/8007#discussion_r37034816
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedClusterMessage.scala
---
@@ -92,6 +94,11 @@ private[spark] object
Github user markgrover commented on the pull request:
https://github.com/apache/spark/pull/8093#issuecomment-130865695
Thanks @vanzin. I looked at #8007 and some pieces of it are definitely very
relevant. So, I will use some of the ideas from there.
In full disclosure, I did
Github user markgrover commented on the pull request:
https://github.com/apache/spark/pull/8007#issuecomment-130863421
Hey @mccheah, good stuff. Turns out I have a similar pull request
lingering: #8093
In that pull request, I discuss the same race condition you are tackling
Github user markgrover commented on a diff in the pull request:
https://github.com/apache/spark/pull/8007#discussion_r37029918
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/YarnSchedulerBackend.scala
---
@@ -91,6 +92,66 @@ private[spark] abstract class
Github user markgrover commented on a diff in the pull request:
https://github.com/apache/spark/pull/8007#discussion_r37029882
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/YarnSchedulerBackend.scala
---
@@ -91,6 +92,66 @@ private[spark] abstract class
Github user markgrover commented on the pull request:
https://github.com/apache/spark/pull/8093#issuecomment-130408846
Updated the review incorporating all the review feedback. In the meanwhile,
let me poke a little more into the race condition and get back with what I find.
---
If
Github user markgrover commented on the pull request:
https://github.com/apache/spark/pull/8093#issuecomment-130167519
> guess we could do that. My concern is that the race is probably always
going to be won by the executor disconnect message (instead of the explicit
RemoveExecu
Github user markgrover commented on the pull request:
https://github.com/apache/spark/pull/7739#issuecomment-130161669
Thanks @harishreedharan and @squito for your reviews!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well
Github user markgrover commented on a diff in the pull request:
https://github.com/apache/spark/pull/7739#discussion_r36826616
--- Diff: core/src/main/scala/org/apache/spark/deploy/Client.scala ---
@@ -68,7 +68,12 @@ private class ClientEndpoint(
val mainClass
Github user markgrover commented on a diff in the pull request:
https://github.com/apache/spark/pull/7739#discussion_r36826563
--- Diff:
core/src/main/scala/org/apache/spark/deploy/rest/StandaloneRestServer.scala ---
@@ -135,7 +135,10 @@ private[rest] class
Github user markgrover commented on a diff in the pull request:
https://github.com/apache/spark/pull/7739#discussion_r36826589
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerBackend.scala
---
@@ -88,7 +88,10 @@ private[spark] class
Github user markgrover commented on a diff in the pull request:
https://github.com/apache/spark/pull/7739#discussion_r36826585
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/SparkDeploySchedulerBackend.scala
---
@@ -61,8 +61,10 @@ private[spark] class
Github user markgrover commented on a diff in the pull request:
https://github.com/apache/spark/pull/7739#discussion_r36826580
--- Diff:
core/src/main/scala/org/apache/spark/deploy/rest/mesos/MesosRestServer.scala ---
@@ -88,7 +88,10 @@ private[mesos] class
Github user markgrover commented on a diff in the pull request:
https://github.com/apache/spark/pull/7739#discussion_r36826569
--- Diff:
core/src/main/scala/org/apache/spark/deploy/rest/SubmitRestProtocolRequest.scala
---
@@ -17,6 +17,8 @@
package
Github user markgrover commented on a diff in the pull request:
https://github.com/apache/spark/pull/7739#discussion_r36826550
--- Diff: core/src/main/scala/org/apache/spark/SparkConf.scala ---
@@ -479,6 +484,23 @@ class SparkConf(loadDefaults: Boolean) extends
Cloneable with
Github user markgrover commented on the pull request:
https://github.com/apache/spark/pull/8093#issuecomment-130159464
Thanks for the review, @squito. I did consider that as well. The UI listens
to whatever is sent as `SparkListenerTaskEnd` message by the
DAGScheduler
Github user markgrover commented on a diff in the pull request:
https://github.com/apache/spark/pull/8093#discussion_r36825910
--- Diff:
yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala ---
@@ -435,19 +437,24 @@ private[yarn] class YarnAllocator
Github user markgrover commented on a diff in the pull request:
https://github.com/apache/spark/pull/8093#discussion_r36825911
--- Diff:
yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala ---
@@ -423,6 +424,7 @@ private[yarn] class YarnAllocator(
for
Github user markgrover commented on the pull request:
https://github.com/apache/spark/pull/8093#issuecomment-130084493
@andrewor14
[here](https://issues.apache.org/jira/secure/attachment/12749771/error_showing_in_UI.png)'s
the link from the JIRA.
---
If your project is set u
Github user markgrover commented on a diff in the pull request:
https://github.com/apache/spark/pull/8093#discussion_r36794461
--- Diff: core/src/main/scala/org/apache/spark/util/JsonProtocol.scala ---
@@ -795,7 +796,8 @@ private[spark] object JsonProtocol {
case
Github user markgrover commented on the pull request:
https://github.com/apache/spark/pull/8093#issuecomment-130053497
Thanks for the review, @vanzin! Indeed, the race is tricky. Yup, I like
that solution but there are cases where the RemoveExecutor event may never
come, right (in
Github user markgrover commented on a diff in the pull request:
https://github.com/apache/spark/pull/8093#discussion_r36793126
--- Diff: core/src/main/scala/org/apache/spark/util/JsonProtocol.scala ---
@@ -795,7 +796,8 @@ private[spark] object JsonProtocol {
case
Github user markgrover commented on a diff in the pull request:
https://github.com/apache/spark/pull/8093#discussion_r36792140
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/YarnSchedulerBackend.scala
---
@@ -110,6 +114,7 @@ private[spark] abstract class
Github user markgrover commented on a diff in the pull request:
https://github.com/apache/spark/pull/8093#discussion_r36783358
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/YarnSchedulerBackend.scala
---
@@ -110,6 +114,7 @@ private[spark] abstract class
Github user markgrover commented on a diff in the pull request:
https://github.com/apache/spark/pull/8093#discussion_r36782545
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/YarnSchedulerBackend.scala
---
@@ -17,6 +17,10 @@
package
Github user markgrover commented on a diff in the pull request:
https://github.com/apache/spark/pull/8093#discussion_r36782462
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala
---
@@ -19,6 +19,9 @@ package
Github user markgrover commented on the pull request:
https://github.com/apache/spark/pull/7739#issuecomment-129715625
Jenkins, retest this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user markgrover commented on the pull request:
https://github.com/apache/spark/pull/8093#issuecomment-129712836
There's one thing I'd really appreciate people's thoughts on:
There seems to be a race condition related to displaying the error message
in th
Github user markgrover commented on the pull request:
https://github.com/apache/spark/pull/8093#issuecomment-129710600
Jenkins, please test this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user markgrover commented on the pull request:
https://github.com/apache/spark/pull/8093#issuecomment-129709754
This pull request is meant to achieve two goals:
1. Show in driver logs, primarily in yarn client mode, if YARN is killing
containers because of one or more of
GitHub user markgrover opened a pull request:
https://github.com/apache/spark/pull/8093
SPARK-9790: [YARN] Expose in WebUI if NodeManager is the reason why
executors were killed.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com
Github user markgrover commented on the pull request:
https://github.com/apache/spark/pull/7739#issuecomment-128778513
Hmm, looks like PYTHONPATH is not being set up properly by some python
tests. It turns out I am not the only suffering from this - the build before
mine
(https
Github user markgrover commented on the pull request:
https://github.com/apache/spark/pull/7739#issuecomment-128773200
I am taking a look.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user markgrover commented on the pull request:
https://github.com/apache/spark/pull/7739#issuecomment-128019886
Thanks @squito !
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this
Github user markgrover commented on the pull request:
https://github.com/apache/spark/pull/7739#issuecomment-127762510
If that doesn't work and some can retrigger the test run on my behalf, I'd
appreciate that. cc @squito @sarutak
---
If your project is set up for it, you
Github user markgrover commented on the pull request:
https://github.com/apache/spark/pull/7739#issuecomment-127762398
Seems unrelated failure. I don't think I am on the whitelist but I will try
this:
Jenkins, test this please.
---
If your project is set up for it, yo
Github user markgrover commented on the pull request:
https://github.com/apache/spark/pull/7739#issuecomment-127634581
Hi @sarutak, thanks again for taking a look, I've rebased and fixed up the
history. I have also update the PR with some more info.
---
If your project is set u
Github user markgrover commented on the pull request:
https://github.com/apache/spark/pull/7739#issuecomment-127482812
Sure thing. Will do!
On Aug 3, 2015 10:18 PM, "Kousuke Saruta" wrote:
> Hey @markgrover <https://github.com/markgrover> , co
Github user markgrover commented on the pull request:
https://github.com/apache/spark/pull/7739#issuecomment-126101858
Still looks like a file missing in Jenkins workspace:
```
13:55:21.583 ERROR org.apache.spark.executor.Executor: Exception in task
0.0 in stage 11926.0 (TID
Github user markgrover commented on the pull request:
https://github.com/apache/spark/pull/7739#issuecomment-125977059
I will take a look at the failures.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
GitHub user markgrover opened a pull request:
https://github.com/apache/spark/pull/7739
[WIP] SPARK-9384: Easier setting of executor and driver classpath
Adding new property spark.common.extraClassPath for adding to both the
driver and executor classpath.
More testing
Github user markgrover commented on a diff in the pull request:
https://github.com/apache/spark/pull/2982#discussion_r20687473
--- Diff: bin/compute-classpath.cmd ---
@@ -1,3 +1,4 @@
+<<<<<<< HEAD
--- End diff --
This looks like a left behind m
Github user markgrover commented on the pull request:
https://github.com/apache/spark/pull/2477#issuecomment-58454665
Thanks for including me, @JoshRosen!
I agree with @markhamstra here.
If I *had* to make this work, I wouldn't do the symlink. In fact, I'
201 - 267 of 267 matches
Mail list logo
