Github user brndnmtthws commented on the pull request:
https://github.com/apache/spark/pull/2401#issuecomment-56126568
I've cleaned up the patch again.
I spent about an hour trying to apply this to the YARN code, but it was
pretty difficult to follow so I gave up.
---
If you
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2401#issuecomment-56125354
[QA tests have
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20564/consoleFull)
for PR 2401 at commit
[`d51b74f`](https://github.com/a
Github user brndnmtthws commented on the pull request:
https://github.com/apache/spark/pull/2401#issuecomment-56123833
I can emulate the YARN behaviour, but it seems better to just do the same
thing with both Mesos and YARN. Thoughts? I can refactor this (including the
YARN code) to
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/2401#issuecomment-56123690
Yes @willb I have the same concern. I think in other contexts "overhead"
refers to the additional space, not the total space, so an "overhead fraction"
of 0.15 means w
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/2401#issuecomment-56122505
@willb in the case of Yarn, the parameter is called "overhead" because it
actually sets the amount of overhead you want to add to the requested heap
memory. The PR being r
Github user willb commented on the pull request:
https://github.com/apache/spark/pull/2401#issuecomment-56121260
@andrewor14 This bothers me too, but in a slightly different way: calling
the parameter âoverheadâ when it really refers to how to scale requested
memory to accommodat
Github user brndnmtthws commented on a diff in the pull request:
https://github.com/apache/spark/pull/2401#discussion_r17763516
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/CommonProps.scala
---
@@ -0,0 +1,27 @@
+/*
+ * Licensed to the Apache Soft
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/2401#discussion_r17763285
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/CommonProps.scala
---
@@ -0,0 +1,27 @@
+/*
+ * Licensed to the Apache Softw
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2401#issuecomment-56120575
[QA tests have
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20564/consoleFull)
for PR 2401 at commit
[`d51b74f`](https://github.com/ap
Github user brndnmtthws commented on the pull request:
https://github.com/apache/spark/pull/2401#issuecomment-56120347
Forgot to mention: I also set the executor CPUs correctly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as w
Github user brndnmtthws commented on the pull request:
https://github.com/apache/spark/pull/2401#issuecomment-56120329
Updated as per @andrewor14's suggestions.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your proj
Github user brndnmtthws commented on a diff in the pull request:
https://github.com/apache/spark/pull/2401#discussion_r17763087
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/CommonProps.scala
---
@@ -0,0 +1,27 @@
+/*
+ * Licensed to the Apache Soft
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/2401#issuecomment-56119651
@brndnmtthws Yes it would be good to keep the approach used in #1391 in
mind. It would be ideal if we could use an approach as similar as what we
currently (or plan to
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/2401#discussion_r17762547
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/CommonProps.scala
---
@@ -0,0 +1,27 @@
+/*
+ * Licensed to the Apache Softw
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/2401#discussion_r17762483
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/CommonProps.scala
---
@@ -0,0 +1,27 @@
+/*
+ * Licensed to the Apache Softw
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/2401#discussion_r17762446
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/CommonProps.scala
---
@@ -0,0 +1,27 @@
+/*
+ * Licensed to the Apache Softw
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/2401#issuecomment-55922361
Might want to take a look at #1391 also; that adds a similar thing for Yarn
(a multiplier for the JVM overhead), it might be worth to merge the two code
paths (or add this
Github user brndnmtthws commented on the pull request:
https://github.com/apache/spark/pull/2401#issuecomment-55921158
When you need to provide guarantees for other services, it's better to
stick to hard limits. Having other tasks get randomly OOM killed is a bad
experience.
---
If
Github user erikerlandson commented on the pull request:
https://github.com/apache/spark/pull/2401#issuecomment-55920577
Applying soft memory limits via cgroups is one option, for environments
supporting cgroups. Reserve a certain amount of memory, and if it is exceeded
then swapping
Github user brndnmtthws commented on the pull request:
https://github.com/apache/spark/pull/2401#issuecomment-55903894
Oh, and one more thing you may want to think about is the OS filesystem
buffers. Again, as you scale up the heap, you may want to proportionally
reserve a slice of o
Github user brndnmtthws commented on the pull request:
https://github.com/apache/spark/pull/2401#issuecomment-55903580
That implies that as you grow the heap, you're not adding threads (or other
things that use off-heap memory). I'm not familiar with Spark's execution
model, but I wo
Github user willb commented on the pull request:
https://github.com/apache/spark/pull/2401#issuecomment-55902476
Here's a somewhat-related concern: it seems like JVM overhead is unlikely
to scale linearly with requested heap size, so maybe a straight-up 15% isn't a
great default? (I
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2401#issuecomment-55827647
[QA tests have
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20428/consoleFull)
for PR 2401 at commit
[`298ca44`](https://github.com/a
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2401#issuecomment-55827491
[QA tests have
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20427/consoleFull)
for PR 2401 at commit
[`8cbde19`](https://github.com/a
Github user brndnmtthws commented on the pull request:
https://github.com/apache/spark/pull/2401#issuecomment-55821735
YARN has a similar strategy, from what I can tell. There's a separate
config value, `spark.yarn.executor.memoryOverhead`.
We could go the other way, where it
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/2401#issuecomment-55821489
Hey will this have compatbility issues for existing deployments? I know
many clusters where they just have Spark request the entire amount of memory on
the node. With th
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2401#issuecomment-55821123
[QA tests have
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20428/consoleFull)
for PR 2401 at commit
[`298ca44`](https://github.com/ap
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2401#issuecomment-55820562
[QA tests have
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20427/consoleFull)
for PR 2401 at commit
[`8cbde19`](https://github.com/ap
Github user brndnmtthws commented on the pull request:
https://github.com/apache/spark/pull/2401#issuecomment-55820092
Updated diff based on coding.
I've also added the same thing to the coarse scheduler, and updated the
documentation accordingly.
---
If your project is set
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2401#issuecomment-55814769
[QA tests have
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20421/consoleFull)
for PR 2401 at commit
[`6532d47`](https://github.com/a
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2401#issuecomment-55814658
[QA tests have
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20421/consoleFull)
for PR 2401 at commit
[`6532d47`](https://github.com/ap
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/2401#issuecomment-55814086
ok to test
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
e
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2401#issuecomment-55662990
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project d
GitHub user brndnmtthws opened a pull request:
https://github.com/apache/spark/pull/2401
[SPARK-3535][Mesos] Add 15% task memory overhead.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/brndnmtthws/spark master
Alternatively yo
34 matches
Mail list logo
