Github user galv commented on a diff in the pull request:
https://github.com/apache/spark/pull/21511#discussion_r198653002
--- Diff:
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala
---
@@ -104,6 +104,20 @@ private[spark] object Config
Github user galv commented on the issue:
https://github.com/apache/spark/pull/20761
I agree that there is apparently unnecessary complexity in the validator.
I'll try to take a look again at the code today.
On Fri, Jun 15, 2018 at 9:15 AM, Marcelo Vanzin
wrote
Github user galv commented on the issue:
https://github.com/apache/spark/pull/20761
I would like to see this merged, though I got derailed by spark summit last
week and other things. I will look this patch over again soon @szyszy If you're
busy lately, perhaps I can take over
Github user galv commented on a diff in the pull request:
https://github.com/apache/spark/pull/21494#discussion_r193953345
--- Diff: core/src/main/scala/org/apache/spark/util/RpcUtils.scala ---
@@ -44,7 +44,7 @@ private[spark] object RpcUtils {
/** Returns the default
Github user galv commented on a diff in the pull request:
https://github.com/apache/spark/pull/21494#discussion_r193953432
--- Diff: core/src/main/scala/org/apache/spark/barrier/BarrierRDD.scala ---
@@ -0,0 +1,43 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF
Github user galv commented on a diff in the pull request:
https://github.com/apache/spark/pull/21511#discussion_r193945410
--- Diff:
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/BasicExecutorFeatureStep.scala
---
@@ -150,6 +152,16
Github user galv commented on a diff in the pull request:
https://github.com/apache/spark/pull/21511#discussion_r193944223
--- Diff:
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala
---
@@ -104,6 +104,20 @@ private[spark] object Config
Github user galv commented on a diff in the pull request:
https://github.com/apache/spark/pull/21511#discussion_r193945389
--- Diff:
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/BasicExecutorFeatureStep.scala
---
@@ -172,7 +184,7
Github user galv commented on a diff in the pull request:
https://github.com/apache/spark/pull/21511#discussion_r193945237
--- Diff:
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala
---
@@ -104,6 +104,20 @@ private[spark] object Config
Github user galv commented on a diff in the pull request:
https://github.com/apache/spark/pull/21494#discussion_r193290266
--- Diff: python/pyspark/worker.py ---
@@ -232,6 +236,13 @@ def main(infile, outfile):
shuffle.DiskBytesSpilled = 0
Github user galv commented on a diff in the pull request:
https://github.com/apache/spark/pull/21494#discussion_r193269255
--- Diff: core/src/test/scala/org/apache/spark/SparkContextSuite.scala ---
@@ -627,6 +627,52 @@ class SparkContextSuite extends SparkFunSuite
Github user galv commented on a diff in the pull request:
https://github.com/apache/spark/pull/21494#discussion_r193289530
--- Diff: python/pyspark/worker.py ---
@@ -232,6 +236,13 @@ def main(infile, outfile):
shuffle.DiskBytesSpilled = 0
Github user galv commented on a diff in the pull request:
https://github.com/apache/spark/pull/21494#discussion_r193291076
--- Diff: python/pyspark/worker.py ---
@@ -232,6 +236,13 @@ def main(infile, outfile):
shuffle.DiskBytesSpilled = 0
Github user galv commented on a diff in the pull request:
https://github.com/apache/spark/pull/21494#discussion_r193555968
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala ---
@@ -123,6 +124,21 @@ private[spark] class TaskSetManager(
// TODO: We
Github user galv commented on a diff in the pull request:
https://github.com/apache/spark/pull/21494#discussion_r193269297
--- Diff: core/src/test/scala/org/apache/spark/SparkContextSuite.scala ---
@@ -627,6 +627,52 @@ class SparkContextSuite extends SparkFunSuite
Github user galv commented on a diff in the pull request:
https://github.com/apache/spark/pull/20761#discussion_r190107244
--- Diff:
resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala
---
@@ -306,9 +318,14 @@ private[yarn] class YarnAllocator
Github user galv commented on a diff in the pull request:
https://github.com/apache/spark/pull/20761#discussion_r190106824
--- Diff:
resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ResourceTypeValidator.scala
---
@@ -0,0 +1,185 @@
+/*
+ * Licensed
Github user galv commented on a diff in the pull request:
https://github.com/apache/spark/pull/20761#discussion_r190105256
--- Diff:
resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ResourceTypeHelper.scala
---
@@ -0,0 +1,180 @@
+/*
+ * Licensed
Github user galv commented on a diff in the pull request:
https://github.com/apache/spark/pull/20761#discussion_r190105917
--- Diff:
resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ResourceTypeValidator.scala
---
@@ -0,0 +1,185 @@
+/*
+ * Licensed
Github user galv commented on a diff in the pull request:
https://github.com/apache/spark/pull/20761#discussion_r190106056
--- Diff:
resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ResourceTypeValidator.scala
---
@@ -0,0 +1,185 @@
+/*
+ * Licensed
Github user galv commented on a diff in the pull request:
https://github.com/apache/spark/pull/20761#discussion_r190106859
--- Diff:
resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ResourceTypeValidator.scala
---
@@ -0,0 +1,185 @@
+/*
+ * Licensed
Github user galv commented on a diff in the pull request:
https://github.com/apache/spark/pull/20761#discussion_r190107444
--- Diff:
resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala
---
@@ -306,9 +318,14 @@ private[yarn] class YarnAllocator
Github user galv commented on a diff in the pull request:
https://github.com/apache/spark/pull/20761#discussion_r190104166
--- Diff:
resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ResourceTypeHelper.scala
---
@@ -0,0 +1,150 @@
+/*
+ * Licensed
Github user galv commented on a diff in the pull request:
https://github.com/apache/spark/pull/20761#discussion_r190107502
--- Diff:
resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala
---
@@ -306,9 +318,14 @@ private[yarn] class YarnAllocator
Github user galv commented on a diff in the pull request:
https://github.com/apache/spark/pull/20761#discussion_r190104076
--- Diff:
resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala
---
@@ -249,6 +259,10 @@ private[spark] class Client(
val
Github user galv commented on a diff in the pull request:
https://github.com/apache/spark/pull/20761#discussion_r190103364
--- Diff:
resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ResourceTypeHelper.scala
---
@@ -0,0 +1,180 @@
+/*
+ * Licensed
Github user galv commented on a diff in the pull request:
https://github.com/apache/spark/pull/20761#discussion_r189949493
--- Diff:
resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ResourceTypeHelper.scala
---
@@ -0,0 +1,180 @@
+/*
+ * Licensed
Github user galv commented on a diff in the pull request:
https://github.com/apache/spark/pull/20761#discussion_r189473608
--- Diff:
resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/ClientSuite.scala
---
@@ -199,6 +204,88 @@ class ClientSuite extends
Github user galv commented on a diff in the pull request:
https://github.com/apache/spark/pull/20761#discussion_r189451540
--- Diff:
resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ResourceTypeHelper.scala
---
@@ -0,0 +1,180 @@
+/*
+ * Licensed
Github user galv commented on a diff in the pull request:
https://github.com/apache/spark/pull/20761#discussion_r189449512
--- Diff:
resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ResourceTypeHelper.scala
---
@@ -0,0 +1,180 @@
+/*
+ * Licensed
Github user galv commented on a diff in the pull request:
https://github.com/apache/spark/pull/20761#discussion_r189449967
--- Diff:
resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ResourceTypeHelper.scala
---
@@ -0,0 +1,180 @@
+/*
+ * Licensed
Github user galv commented on a diff in the pull request:
https://github.com/apache/spark/pull/20761#discussion_r189450284
--- Diff:
resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ResourceTypeHelper.scala
---
@@ -0,0 +1,180 @@
+/*
+ * Licensed
Github user galv commented on a diff in the pull request:
https://github.com/apache/spark/pull/20761#discussion_r189449462
--- Diff:
resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/ResourceTypeHelperSuite.scala
---
@@ -0,0 +1,153 @@
+/*
+ * Licensed
Github user galv commented on a diff in the pull request:
https://github.com/apache/spark/pull/20761#discussion_r189449133
--- Diff:
resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/ResourceTypeHelperSuite.scala
---
@@ -0,0 +1,153 @@
+/*
+ * Licensed
Github user galv commented on the issue:
https://github.com/apache/spark/pull/20761
I may be missing something, but doesn't this change depend on Spark's yarn
client being made hadoop 3.0 compatible
Github user galv commented on a diff in the pull request:
https://github.com/apache/spark/pull/20761#discussion_r189426179
--- Diff: docs/running-on-yarn.md ---
@@ -121,6 +121,28 @@ To use a custom metrics.properties for the application
master and executors, upd
Use lower
Github user galv commented on a diff in the pull request:
https://github.com/apache/spark/pull/20761#discussion_r189426163
--- Diff: docs/running-on-yarn.md ---
@@ -121,6 +121,28 @@ To use a custom metrics.properties for the application
master and executors, upd
Use lower
37 matches
Mail list logo
