Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/22612#discussion_r239185785
--- Diff:
core/src/main/scala/org/apache/spark/executor/ProcfsMetricsGetter.scala ---
@@ -0,0 +1,229 @@
+/*
+ * Licensed to the Apache Software
Github user squito commented on the issue:
https://github.com/apache/spark/pull/23236
lgtm
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user squito commented on the issue:
https://github.com/apache/spark/pull/23223
@attilapiros
> I mean if node blacklisting in Spark would be perfectly aligned to YARN
then it would be just redundant to have it in Spark in the first place.
I'm not su
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/23223#discussion_r239174670
--- Diff:
resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala
---
@@ -612,11 +612,14 @@ private[yarn] class
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/23223#discussion_r239173889
--- Diff:
resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/YarnAllocatorSuite.scala
---
@@ -417,4 +426,59 @@ class YarnAllocatorSuite
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/23092#discussion_r238426468
--- Diff: core/src/main/scala/org/apache/spark/deploy/SparkHadoopUtil.scala
---
@@ -471,7 +471,11 @@ object SparkHadoopUtil {
try
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/22612#discussion_r238424342
--- Diff:
core/src/main/scala/org/apache/spark/executor/ProcfsMetricsGetter.scala ---
@@ -0,0 +1,223 @@
+/*
+ * Licensed to the Apache Software
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/22612#discussion_r238388633
--- Diff:
core/src/main/scala/org/apache/spark/executor/ProcfsMetricsGetter.scala ---
@@ -0,0 +1,223 @@
+/*
+ * Licensed to the Apache Software
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/22612#discussion_r238371374
--- Diff:
core/src/main/scala/org/apache/spark/executor/ProcfsMetricsGetter.scala ---
@@ -0,0 +1,223 @@
+/*
+ * Licensed to the Apache Software
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/22612#discussion_r238363825
--- Diff:
core/src/main/scala/org/apache/spark/executor/ProcfsMetricsGetter.scala ---
@@ -0,0 +1,223 @@
+/*
+ * Licensed to the Apache Software
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/22612#discussion_r238366634
--- Diff:
core/src/main/scala/org/apache/spark/executor/ProcfsMetricsGetter.scala ---
@@ -0,0 +1,223 @@
+/*
+ * Licensed to the Apache Software
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/22612#discussion_r238363142
--- Diff:
core/src/main/scala/org/apache/spark/executor/ProcfsMetricsGetter.scala ---
@@ -0,0 +1,223 @@
+/*
+ * Licensed to the Apache Software
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/22612#discussion_r237992560
--- Diff:
core/src/main/scala/org/apache/spark/executor/ProcfsMetricsGetter.scala ---
@@ -0,0 +1,228 @@
+/*
+ * Licensed to the Apache Software
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/22612#discussion_r237982013
--- Diff:
core/src/main/scala/org/apache/spark/executor/ProcfsMetricsGetter.scala ---
@@ -0,0 +1,228 @@
+/*
+ * Licensed to the Apache Software
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/22612#discussion_r237975861
--- Diff:
core/src/main/scala/org/apache/spark/executor/ProcfsMetricsGetter.scala ---
@@ -138,19 +138,22 @@ private[spark] class ProcfsMetricsGetter
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/22612#discussion_r237981353
--- Diff:
core/src/main/scala/org/apache/spark/executor/ProcfsMetricsGetter.scala ---
@@ -0,0 +1,228 @@
+/*
+ * Licensed to the Apache Software
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/22612#discussion_r237978529
--- Diff:
core/src/main/scala/org/apache/spark/executor/ProcfsMetricsGetter.scala ---
@@ -0,0 +1,228 @@
+/*
+ * Licensed to the Apache Software
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/22612#discussion_r237979881
--- Diff:
core/src/main/scala/org/apache/spark/executor/ProcfsMetricsGetter.scala ---
@@ -0,0 +1,228 @@
+/*
+ * Licensed to the Apache Software
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/22612#discussion_r237977304
--- Diff:
core/src/main/scala/org/apache/spark/executor/ExecutorMetrics.scala ---
@@ -49,14 +47,14 @@ class ExecutorMetrics private[spark] extends
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/22612#discussion_r237982177
--- Diff:
core/src/main/scala/org/apache/spark/executor/ProcfsMetricsGetter.scala ---
@@ -0,0 +1,228 @@
+/*
+ * Licensed to the Apache Software
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/23166#discussion_r237886286
--- Diff: core/src/main/scala/org/apache/spark/api/python/PythonRDD.scala
---
@@ -708,16 +709,36 @@ private[spark] class PythonBroadcast(@transient var
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/23166#discussion_r237738802
--- Diff: core/src/main/scala/org/apache/spark/api/python/PythonRDD.scala
---
@@ -708,16 +709,36 @@ private[spark] class PythonBroadcast(@transient var
Github user squito commented on the issue:
https://github.com/apache/spark/pull/23058
merged to master, thanks @wypoon
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/23166#discussion_r237535187
--- Diff: core/src/main/scala/org/apache/spark/api/python/PythonRDD.scala
---
@@ -708,16 +709,36 @@ private[spark] class PythonBroadcast(@transient var
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/23058#discussion_r237268983
--- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala
---
@@ -789,21 +785,31 @@ private[spark] class BlockManager
Github user squito commented on the issue:
https://github.com/apache/spark/pull/23058
> causes any performance degradation compared to memory mapping
@ankuriitg good question, though if you look at what the old code was
doing, it wasn't memory mapping the file, it was read
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/23166#discussion_r237228157
--- Diff: python/pyspark/broadcast.py ---
@@ -118,8 +121,16 @@ def dump(self, value, f):
f.close()
def load_from_path(self, path
Github user squito commented on the issue:
https://github.com/apache/spark/pull/23166
> The only question I have is does this cause more memory usage on the
driver because it has a reference to that broadcast value or is something else
already holding on to it?
yeah g
Github user squito commented on the issue:
https://github.com/apache/spark/pull/23166
maybe a dumb question -- couldn't you just add
```python
self._value = value
```
[inside `__init__()` when on the
driver](https://github.com/apache/spark/blob/master/python
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/22612#discussion_r236863378
--- Diff:
core/src/main/scala/org/apache/spark/executor/ExecutorMetrics.scala ---
@@ -28,16 +28,14 @@ import org.apache.spark.metrics.ExecutorMetricType
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/22612#discussion_r236856360
--- Diff:
core/src/main/scala/org/apache/spark/executor/ProcfsMetricsGetter.scala ---
@@ -0,0 +1,231 @@
+/*
+ * Licensed to the Apache Software
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/22612#discussion_r236858045
--- Diff:
core/src/main/scala/org/apache/spark/executor/ProcfsMetricsGetter.scala ---
@@ -0,0 +1,231 @@
+/*
+ * Licensed to the Apache Software
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/22612#discussion_r236856995
--- Diff:
core/src/main/scala/org/apache/spark/executor/ProcfsMetricsGetter.scala ---
@@ -0,0 +1,231 @@
+/*
+ * Licensed to the Apache Software
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/22612#discussion_r236860940
--- Diff:
core/src/test/scala/org/apache/spark/executor/ProcfsMetricsGetterSuite.scala ---
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/22612#discussion_r236857465
--- Diff:
core/src/main/scala/org/apache/spark/executor/ProcfsMetricsGetter.scala ---
@@ -0,0 +1,231 @@
+/*
+ * Licensed to the Apache Software
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/22612#discussion_r236855952
--- Diff:
core/src/main/scala/org/apache/spark/executor/ProcfsMetricsGetter.scala ---
@@ -0,0 +1,231 @@
+/*
+ * Licensed to the Apache Software
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/22612#discussion_r236860104
--- Diff:
core/src/main/scala/org/apache/spark/metrics/ExecutorMetricType.scala ---
@@ -95,10 +136,22 @@ private[spark] object ExecutorMetricType
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/22612#discussion_r236855451
--- Diff:
core/src/main/scala/org/apache/spark/executor/ProcfsMetricsGetter.scala ---
@@ -0,0 +1,231 @@
+/*
+ * Licensed to the Apache Software
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/22612#discussion_r236848338
--- Diff:
core/src/main/scala/org/apache/spark/executor/ProcfsMetricsGetter.scala ---
@@ -0,0 +1,231 @@
+/*
+ * Licensed to the Apache Software
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/22612#discussion_r236847493
--- Diff:
core/src/main/scala/org/apache/spark/executor/ProcfsMetricsGetter.scala ---
@@ -0,0 +1,231 @@
+/*
+ * Licensed to the Apache Software
Github user squito commented on the issue:
https://github.com/apache/spark/pull/23058
@mridulm @jerryshao @Ngone51 @vanzin just checking if you want to look at
this before I merge, will leave open a bit
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/22612#discussion_r236769432
--- Diff:
core/src/main/scala/org/apache/spark/metrics/ExecutorMetricType.scala ---
@@ -84,6 +122,8 @@ case object MappedPoolMemory extends
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/22612#discussion_r236766605
--- Diff:
core/src/main/scala/org/apache/spark/executor/ProcfsBasedSystems.scala ---
@@ -0,0 +1,228 @@
+/*
+ * Licensed to the Apache Software
Github user squito commented on the issue:
https://github.com/apache/spark/pull/23058
@attilapiros yes, something like that would be possible. I was thinking
you'd just use the existing serializer methods to do it, soemthing like:
```scala
val buffer
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/23117#discussion_r236431048
--- Diff: dev/run-tests.py ---
@@ -434,6 +434,63 @@ def run_python_tests(test_modules, parallelism):
run_cmd(command)
+def
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/23117#discussion_r236431733
--- Diff: dev/run-tests.py ---
@@ -434,6 +434,63 @@ def run_python_tests(test_modules, parallelism):
run_cmd(command)
+def
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/23106#discussion_r236418843
--- Diff:
core/src/main/java/org/apache/spark/shuffle/sort/ShuffleExternalSorter.java ---
@@ -242,8 +243,13 @@ private void writeSortedFile(boolean
Github user squito commented on the issue:
https://github.com/apache/spark/pull/23111
we might need to be careful that this doesn't un-intentionally overload the
jenkins workers so that we end up hitting more timeouts from too many things
running concurrently (I dunno how isolated
Github user squito commented on the issue:
https://github.com/apache/spark/pull/23111
wow, thats great! glad there is a big speedup.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user squito commented on the issue:
https://github.com/apache/spark/pull/23109
lte review, but lgtm anyway
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user squito commented on the issue:
https://github.com/apache/spark/pull/23058
lgtm
I looked more into the lifecycle of the buffers and when they get
`disposed`, and it looks fine to me. (In fact I think there is no need for the
`dispose` in the first place
Github user squito commented on the issue:
https://github.com/apache/spark/pull/23113
I'm not sure if I follow the entire discussion here yet. I did spend a
while trying to figure out if there is a race condition, and I'm pretty sure
there is not. From your last comments, it looks
Github user squito commented on the issue:
https://github.com/apache/spark/pull/23090
btw I agree this was bad judgement on my part to only backport to 2.4,
sorry abotu that and thanks for catching @dongjoon-hyun . I do think this
fixes a bug (the ability to use the UI when you
Github user squito commented on the issue:
https://github.com/apache/spark/pull/23090
I think the default is pretty reasonable in most cases, in that this is the
first time we've heard of someone hitting this limit. I'm not sure how high we
would make it to get around this problem
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/22612#discussion_r235124501
--- Diff:
core/src/main/scala/org/apache/spark/executor/ProcfsBasedSystems.scala ---
@@ -0,0 +1,228 @@
+/*
+ * Licensed to the Apache Software
Github user squito commented on the issue:
https://github.com/apache/spark/pull/23000
lgtm
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user squito commented on the issue:
https://github.com/apache/spark/pull/23090
merged to master / 2.4
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
GitHub user squito opened a pull request:
https://github.com/apache/spark/pull/23092
[SPARK-26094][CORE][STREAMING] createNonEcFile creates parent dirs.
## What changes were proposed in this pull request?
We explicitly avoid files with hdfs erasure coding for the streaming
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/23000#discussion_r234807971
--- Diff:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/util/DateTimeUtilsSuite.scala
---
@@ -410,6 +410,30 @@ class DateTimeUtilsSuite extends
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/23090#discussion_r234799817
--- Diff: docs/configuration.md ---
@@ -973,6 +973,14 @@ Apart from these, the following properties are also
available, and may be useful
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/23058#discussion_r234779890
--- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala
---
@@ -693,9 +693,9 @@ private[spark] class BlockManager
Github user squito commented on the issue:
https://github.com/apache/spark/pull/23058
can we also make the same change to `TaskResultGetter`? We could avoid the
same inefficiency for large task results that also get fetched to disk. (That
would actually get us closer to removing
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/23058#discussion_r234326643
--- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala
---
@@ -693,9 +693,9 @@ private[spark] class BlockManager
Github user squito commented on the issue:
https://github.com/apache/spark/pull/23058
@attilapiros can you review this please?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/22612#discussion_r234340260
--- Diff:
core/src/main/scala/org/apache/spark/executor/ProcfsBasedSystems.scala ---
@@ -0,0 +1,228 @@
+/*
+ * Licensed to the Apache Software
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/22612#discussion_r234340168
--- Diff:
core/src/main/scala/org/apache/spark/executor/ProcfsBasedSystems.scala ---
@@ -0,0 +1,228 @@
+/*
+ * Licensed to the Apache Software
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/22612#discussion_r234343451
--- Diff:
core/src/main/scala/org/apache/spark/executor/ProcfsBasedSystems.scala ---
@@ -0,0 +1,228 @@
+/*
+ * Licensed to the Apache Software
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/22612#discussion_r234342703
--- Diff:
core/src/main/scala/org/apache/spark/executor/ProcfsBasedSystems.scala ---
@@ -0,0 +1,228 @@
+/*
+ * Licensed to the Apache Software
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/22612#discussion_r234337409
--- Diff:
core/src/main/scala/org/apache/spark/executor/ProcfsBasedSystems.scala ---
@@ -0,0 +1,228 @@
+/*
+ * Licensed to the Apache Software
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/22612#discussion_r234344505
--- Diff:
core/src/main/scala/org/apache/spark/executor/ProcfsBasedSystems.scala ---
@@ -0,0 +1,228 @@
+/*
+ * Licensed to the Apache Software
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/22612#discussion_r234341965
--- Diff:
core/src/main/scala/org/apache/spark/executor/ProcfsBasedSystems.scala ---
@@ -0,0 +1,228 @@
+/*
+ * Licensed to the Apache Software
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/22612#discussion_r234344631
--- Diff:
core/src/main/scala/org/apache/spark/metrics/ExecutorMetricType.scala ---
@@ -19,25 +19,43 @@ package org.apache.spark.metrics
import
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/22612#discussion_r234347780
--- Diff:
core/src/main/scala/org/apache/spark/metrics/ExecutorMetricType.scala ---
@@ -84,6 +122,8 @@ case object MappedPoolMemory extends
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/22612#discussion_r234343988
--- Diff:
core/src/main/scala/org/apache/spark/executor/ProcfsBasedSystems.scala ---
@@ -0,0 +1,268 @@
+/*
+ * Licensed to the Apache Software
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/22612#discussion_r234334930
--- Diff:
core/src/main/scala/org/apache/spark/executor/ExecutorMetrics.scala ---
@@ -28,16 +28,14 @@ import org.apache.spark.metrics.ExecutorMetricType
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/22612#discussion_r234339484
--- Diff:
core/src/main/scala/org/apache/spark/executor/ProcfsBasedSystems.scala ---
@@ -0,0 +1,228 @@
+/*
+ * Licensed to the Apache Software
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/22612#discussion_r234340136
--- Diff:
core/src/main/scala/org/apache/spark/executor/ProcfsBasedSystems.scala ---
@@ -0,0 +1,228 @@
+/*
+ * Licensed to the Apache Software
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/22612#discussion_r234337238
--- Diff:
core/src/main/scala/org/apache/spark/executor/ExecutorMetrics.scala ---
@@ -69,9 +67,8 @@ class ExecutorMetrics private[spark] extends
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/22612#discussion_r234339856
--- Diff:
core/src/main/scala/org/apache/spark/executor/ProcfsBasedSystems.scala ---
@@ -0,0 +1,228 @@
+/*
+ * Licensed to the Apache Software
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/22612#discussion_r234344821
--- Diff:
core/src/main/scala/org/apache/spark/metrics/ExecutorMetricType.scala ---
@@ -19,25 +19,43 @@ package org.apache.spark.metrics
import
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/22612#discussion_r234342892
--- Diff:
core/src/main/scala/org/apache/spark/executor/ProcfsBasedSystems.scala ---
@@ -0,0 +1,228 @@
+/*
+ * Licensed to the Apache Software
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/22612#discussion_r234338058
--- Diff:
core/src/main/scala/org/apache/spark/executor/ProcfsBasedSystems.scala ---
@@ -0,0 +1,228 @@
+/*
+ * Licensed to the Apache Software
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/22612#discussion_r234334582
--- Diff: core/src/main/scala/org/apache/spark/Heartbeater.scala ---
@@ -60,11 +60,16 @@ private[spark] class Heartbeater
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/22612#discussion_r234339667
--- Diff:
core/src/main/scala/org/apache/spark/executor/ProcfsBasedSystems.scala ---
@@ -0,0 +1,228 @@
+/*
+ * Licensed to the Apache Software
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/22612#discussion_r234348523
--- Diff:
core/src/main/scala/org/apache/spark/executor/ProcfsBasedSystems.scala ---
@@ -0,0 +1,228 @@
+/*
+ * Licensed to the Apache Software
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/22612#discussion_r234342770
--- Diff:
core/src/main/scala/org/apache/spark/executor/ProcfsBasedSystems.scala ---
@@ -0,0 +1,228 @@
+/*
+ * Licensed to the Apache Software
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/22612#discussion_r234343203
--- Diff:
core/src/main/scala/org/apache/spark/executor/ProcfsBasedSystems.scala ---
@@ -0,0 +1,228 @@
+/*
+ * Licensed to the Apache Software
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/22612#discussion_r234344763
--- Diff:
core/src/main/scala/org/apache/spark/metrics/ExecutorMetricType.scala ---
@@ -19,25 +19,43 @@ package org.apache.spark.metrics
import
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/22612#discussion_r234347011
--- Diff:
core/src/main/scala/org/apache/spark/metrics/ExecutorMetricType.scala ---
@@ -47,18 +65,39 @@ private[spark] abstract class
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/22612#discussion_r234340033
--- Diff:
core/src/main/scala/org/apache/spark/executor/ProcfsBasedSystems.scala ---
@@ -0,0 +1,228 @@
+/*
+ * Licensed to the Apache Software
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/22612#discussion_r234337384
--- Diff:
core/src/main/scala/org/apache/spark/executor/ProcfsBasedSystems.scala ---
@@ -0,0 +1,228 @@
+/*
+ * Licensed to the Apache Software
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/22612#discussion_r234342135
--- Diff:
core/src/main/scala/org/apache/spark/executor/ProcfsBasedSystems.scala ---
@@ -0,0 +1,228 @@
+/*
+ * Licensed to the Apache Software
Github user squito commented on the issue:
https://github.com/apache/spark/pull/23041
failure from another flaky, SPARK-24153
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user squito commented on the issue:
https://github.com/apache/spark/pull/23041
lgtm
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/22612#discussion_r233255758
--- Diff:
core/src/main/scala/org/apache/spark/metrics/ExecutorMetricType.scala ---
@@ -19,18 +19,31 @@ package org.apache.spark.metrics
import
Github user squito commented on the issue:
https://github.com/apache/spark/pull/23021
thanks for working on this @HyukjinKwon . Did you test on python3 as well?
I'm surprised elapsed time is the same -- I would have expected it to be
faster as the tests could run in parallel
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/22612#discussion_r231921807
--- Diff:
core/src/main/scala/org/apache/spark/executor/ExecutorMetrics.scala ---
@@ -69,11 +67,10 @@ class ExecutorMetrics private[spark] extends
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/22612#discussion_r232224322
--- Diff:
core/src/main/scala/org/apache/spark/metrics/ExecutorMetricType.scala ---
@@ -95,10 +148,18 @@ private[spark] object ExecutorMetricType
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/22612#discussion_r232221648
--- Diff:
core/src/main/scala/org/apache/spark/metrics/ExecutorMetricType.scala ---
@@ -19,18 +19,31 @@ package org.apache.spark.metrics
import
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/22612#discussion_r232220878
--- Diff:
core/src/main/scala/org/apache/spark/executor/ProcfsBasedSystems.scala ---
@@ -0,0 +1,222 @@
+/*
+ * Licensed to the Apache Software
1 - 100 of 3521 matches
Mail list logo
