Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/3120#discussion_r23066142
--- Diff: core/src/main/scala/org/apache/spark/executor/TaskMetrics.scala
---
@@ -146,6 +185,10 @@ class TaskMetrics extends Serializable {
}
_
Github user ksakellis commented on a diff in the pull request:
https://github.com/apache/spark/pull/3120#discussion_r23066065
--- Diff: core/src/main/scala/org/apache/spark/executor/TaskMetrics.scala
---
@@ -146,6 +185,10 @@ class TaskMetrics extends Serializable {
}
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/3120#discussion_r23065867
--- Diff: core/src/main/scala/org/apache/spark/executor/TaskMetrics.scala
---
@@ -146,6 +185,10 @@ class TaskMetrics extends Serializable {
}
_
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/3120
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enab
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3120#issuecomment-70198056
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3120#issuecomment-70198053
[Test build #25623 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25623/consoleFull)
for PR 3120 at commit
[`54e6658`](https://gith
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3120#issuecomment-70192426
[Test build #25623 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25623/consoleFull)
for PR 3120 at commit
[`54e6658`](https://githu
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/3120#issuecomment-70192111
Jenkins, test this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/3120#issuecomment-70192150
LGTM pending tests
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this fea
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/3120#discussion_r22822398
--- Diff: core/src/main/scala/org/apache/spark/CacheManager.scala ---
@@ -44,7 +44,14 @@ private[spark] class CacheManager(blockManager:
BlockManager) e
Github user ksakellis commented on a diff in the pull request:
https://github.com/apache/spark/pull/3120#discussion_r22822258
--- Diff: core/src/main/scala/org/apache/spark/CacheManager.scala ---
@@ -44,7 +44,14 @@ private[spark] class CacheManager(blockManager:
BlockManager) exten
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/3120#discussion_r22814472
--- Diff: core/src/main/scala/org/apache/spark/rdd/NewHadoopRDD.scala ---
@@ -153,34 +157,19 @@ class NewHadoopRDD[K, V](
throw new java.util.N
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/3120#discussion_r22809066
--- Diff: core/src/main/scala/org/apache/spark/CacheManager.scala ---
@@ -44,7 +44,14 @@ private[spark] class CacheManager(blockManager:
BlockManager) e
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/3120#discussion_r22808807
--- Diff:
core/src/test/scala/org/apache/spark/metrics/InputOutputMetricsSuite.scala ---
@@ -17,66 +17,147 @@
package org.apache.spark.metric
Github user ksakellis commented on a diff in the pull request:
https://github.com/apache/spark/pull/3120#discussion_r22807972
--- Diff: core/src/main/scala/org/apache/spark/rdd/NewHadoopRDD.scala ---
@@ -153,34 +157,19 @@ class NewHadoopRDD[K, V](
throw new java.util.
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/3120#discussion_r22778563
--- Diff: core/src/main/scala/org/apache/spark/rdd/NewHadoopRDD.scala ---
@@ -109,18 +109,22 @@ class NewHadoopRDD[K, V](
logInfo("Input split: " +
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/3120#discussion_r22778546
--- Diff: core/src/main/scala/org/apache/spark/CacheManager.scala ---
@@ -44,7 +44,14 @@ private[spark] class CacheManager(blockManager:
BlockManager) extend
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/3120#discussion_r22778368
--- Diff: core/src/main/scala/org/apache/spark/rdd/NewHadoopRDD.scala ---
@@ -153,34 +157,19 @@ class NewHadoopRDD[K, V](
throw new java.util.N
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3120#issuecomment-69538897
[Test build #25404 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25404/consoleFull)
for PR 3120 at commit
[`a2ca793`](https://gith
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3120#issuecomment-69538904
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/3120#discussion_r22777883
--- Diff: core/src/main/scala/org/apache/spark/CacheManager.scala ---
@@ -44,7 +44,14 @@ private[spark] class CacheManager(blockManager:
BlockManager) extend
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3120#issuecomment-69535993
[Test build #25404 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25404/consoleFull)
for PR 3120 at commit
[`a2ca793`](https://githu
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/3120#issuecomment-69535751
Jenkins, test this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user ksakellis commented on the pull request:
https://github.com/apache/spark/pull/3120#issuecomment-68778832
@kayousterhout @pwendell ping?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user ksakellis commented on the pull request:
https://github.com/apache/spark/pull/3120#issuecomment-67213648
@kayousterhout The test you pointed out:
sc.parallelize(1 to 2).saveAsTextFile("file:tester1")
val a = sc.textFile("file:tester1")
sc.parallelize(1 to
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/3120#discussion_r20829599
--- Diff: core/src/main/scala/org/apache/spark/rdd/HadoopRDD.scala ---
@@ -252,7 +258,7 @@ class HadoopRDD[K, V](
&& bytesReadCallback.isDefi
Github user ksakellis commented on the pull request:
https://github.com/apache/spark/pull/3120#issuecomment-64050322
@pwendell Can you please comment on kay's suggestion?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. I
Github user ksakellis commented on the pull request:
https://github.com/apache/spark/pull/3120#issuecomment-63108892
@ash211 Just updated the pr.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not ha
Github user ash211 commented on the pull request:
https://github.com/apache/spark/pull/3120#issuecomment-62996573
@ksakellis it looks like this has a merge conflict now -- would you mind
updating this PR?
---
If your project is set up for it, you can reply to this email and have your
Github user ksakellis commented on a diff in the pull request:
https://github.com/apache/spark/pull/3120#discussion_r20331209
--- Diff: core/src/main/scala/org/apache/spark/rdd/HadoopRDD.scala ---
@@ -252,7 +258,7 @@ class HadoopRDD[K, V](
&& bytesReadCallback.isDef
Github user ksakellis commented on a diff in the pull request:
https://github.com/apache/spark/pull/3120#discussion_r20273844
--- Diff:
core/src/test/scala/org/apache/spark/metrics/InputMetricsSuite.scala ---
@@ -27,50 +30,92 @@ import scala.collection.mutable.ArrayBuffer
impo
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/3120#discussion_r20265190
--- Diff: core/src/main/scala/org/apache/spark/rdd/HadoopRDD.scala ---
@@ -252,7 +258,7 @@ class HadoopRDD[K, V](
&& bytesReadCallback.i
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/3120#discussion_r20263365
--- Diff: core/src/main/scala/org/apache/spark/rdd/HadoopRDD.scala ---
@@ -252,7 +258,7 @@ class HadoopRDD[K, V](
&& bytesReadCallback.i
Github user sryza commented on a diff in the pull request:
https://github.com/apache/spark/pull/3120#discussion_r20263076
--- Diff: core/src/main/scala/org/apache/spark/rdd/HadoopRDD.scala ---
@@ -252,7 +258,7 @@ class HadoopRDD[K, V](
&& bytesReadCallback.isDefined
Github user sryza commented on a diff in the pull request:
https://github.com/apache/spark/pull/3120#discussion_r20262822
--- Diff: core/src/main/scala/org/apache/spark/rdd/HadoopRDD.scala ---
@@ -252,7 +258,7 @@ class HadoopRDD[K, V](
&& bytesReadCallback.isDefined
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/3120#discussion_r20261253
--- Diff: core/src/main/scala/org/apache/spark/rdd/HadoopRDD.scala ---
@@ -252,7 +258,7 @@ class HadoopRDD[K, V](
&& bytesReadCallback.i
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/3120#discussion_r20261117
--- Diff: core/src/main/scala/org/apache/spark/rdd/HadoopRDD.scala ---
@@ -252,7 +258,7 @@ class HadoopRDD[K, V](
&& bytesReadCallback.i
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/3120#discussion_r20260893
--- Diff: core/src/main/scala/org/apache/spark/rdd/HadoopRDD.scala ---
@@ -252,7 +258,7 @@ class HadoopRDD[K, V](
&& bytesReadCallback.i
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/3120#discussion_r20257665
--- Diff: core/src/main/scala/org/apache/spark/CacheManager.scala ---
@@ -44,7 +44,14 @@ private[spark] class CacheManager(blockManager:
BlockManager) e
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/3120#discussion_r20257286
--- Diff:
core/src/test/scala/org/apache/spark/metrics/InputMetricsSuite.scala ---
@@ -27,50 +30,92 @@ import scala.collection.mutable.ArrayBuffer
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/3120#discussion_r20256875
--- Diff:
core/src/test/scala/org/apache/spark/metrics/InputMetricsSuite.scala ---
@@ -27,50 +30,92 @@ import scala.collection.mutable.ArrayBuffer
Github user ksakellis commented on a diff in the pull request:
https://github.com/apache/spark/pull/3120#discussion_r20248211
--- Diff: core/src/main/scala/org/apache/spark/rdd/HadoopRDD.scala ---
@@ -252,7 +258,7 @@ class HadoopRDD[K, V](
&& bytesReadCallback.isDef
Github user ksakellis commented on a diff in the pull request:
https://github.com/apache/spark/pull/3120#discussion_r20247680
--- Diff:
core/src/test/scala/org/apache/spark/metrics/InputMetricsSuite.scala ---
@@ -27,50 +30,92 @@ import scala.collection.mutable.ArrayBuffer
impo
Github user ksakellis commented on a diff in the pull request:
https://github.com/apache/spark/pull/3120#discussion_r20247614
--- Diff:
core/src/test/scala/org/apache/spark/metrics/InputMetricsSuite.scala ---
@@ -27,50 +30,92 @@ import scala.collection.mutable.ArrayBuffer
impo
Github user ksakellis commented on a diff in the pull request:
https://github.com/apache/spark/pull/3120#discussion_r20247484
--- Diff: core/src/main/scala/org/apache/spark/CacheManager.scala ---
@@ -44,7 +44,14 @@ private[spark] class CacheManager(blockManager:
BlockManager) exten
Github user ksakellis commented on a diff in the pull request:
https://github.com/apache/spark/pull/3120#discussion_r20205181
--- Diff:
core/src/test/scala/org/apache/spark/metrics/InputMetricsSuite.scala ---
@@ -27,50 +30,92 @@ import scala.collection.mutable.ArrayBuffer
impo
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3120#issuecomment-62345370
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3120#issuecomment-62345369
[Test build #23136 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23136/consoleFull)
for PR 3120 at commit
[`f1a615f`](https://gith
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3120#issuecomment-62345268
[Test build #23136 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23136/consoleFull)
for PR 3120 at commit
[`f1a615f`](https://githu
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/3120#issuecomment-62344992
Jenkins, test this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user kayousterhout commented on the pull request:
https://github.com/apache/spark/pull/3120#issuecomment-62228520
Jenkins, this is OK to test
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3120#issuecomment-62226591
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your pro
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/3120#discussion_r19979783
--- Diff:
core/src/test/scala/org/apache/spark/metrics/InputMetricsSuite.scala ---
@@ -27,50 +30,92 @@ import scala.collection.mutable.ArrayBuffer
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/3120#discussion_r19979756
--- Diff:
core/src/test/scala/org/apache/spark/metrics/InputMetricsSuite.scala ---
@@ -27,50 +30,92 @@ import scala.collection.mutable.ArrayBuffer
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/3120#discussion_r19979659
--- Diff: core/src/main/scala/org/apache/spark/rdd/HadoopRDD.scala ---
@@ -252,7 +258,7 @@ class HadoopRDD[K, V](
&& bytesReadCallback.i
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/3120#discussion_r19978898
--- Diff:
core/src/test/scala/org/apache/spark/metrics/InputMetricsSuite.scala ---
@@ -27,50 +30,92 @@ import scala.collection.mutable.ArrayBuffer
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/3120#discussion_r19978714
--- Diff: core/src/main/scala/org/apache/spark/CacheManager.scala ---
@@ -44,7 +44,14 @@ private[spark] class CacheManager(blockManager:
BlockManager) e
Github user kayousterhout commented on the pull request:
https://github.com/apache/spark/pull/3120#issuecomment-62058201
Jenkins, test this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not h
Github user sryza commented on the pull request:
https://github.com/apache/spark/pull/3120#issuecomment-61917343
Had a few nitpicks. Otherwise, this looks good to me.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If yo
Github user sryza commented on a diff in the pull request:
https://github.com/apache/spark/pull/3120#discussion_r19922345
--- Diff: core/src/main/scala/org/apache/spark/rdd/NewHadoopRDD.scala ---
@@ -173,12 +179,12 @@ class NewHadoopRDD[K, V](
// Update metrics with f
Github user sryza commented on a diff in the pull request:
https://github.com/apache/spark/pull/3120#discussion_r19922310
--- Diff: core/src/main/scala/org/apache/spark/CacheManager.scala ---
@@ -44,7 +44,14 @@ private[spark] class CacheManager(blockManager:
BlockManager) extends L
Github user sryza commented on a diff in the pull request:
https://github.com/apache/spark/pull/3120#discussion_r19922230
--- Diff:
core/src/test/scala/org/apache/spark/metrics/InputMetricsSuite.scala ---
@@ -17,8 +17,11 @@
package org.apache.spark.metrics
+imp
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3120#issuecomment-61888558
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your pro
GitHub user ksakellis opened a pull request:
https://github.com/apache/spark/pull/3120
[SPARK-4092] [CORE] Fix InputMetrics for coalesce'd Rdds
When calculating the input metrics there was an assumption that one task
only reads from one block - this is not true for some operations i
64 matches
Mail list logo
