Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/2696#issuecomment-60002148
Hey @JoshRosen is this still WIP?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/2696#discussion_r19180655
--- Diff: core/src/main/scala/org/apache/spark/ui/SparkUI.scala ---
@@ -21,47 +21,30 @@ import org.apache.spark.{Logging, SecurityManager,
SparkConf, Spark
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/2696#discussion_r19180607
--- Diff: core/src/main/scala/org/apache/spark/ui/SparkUI.scala ---
@@ -116,4 +94,40 @@ private[spark] object SparkUI {
def getUIPort(conf: SparkConf
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/2696#issuecomment-59875888
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2696#issuecomment-59875884
[QA tests have
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21956/consoleFull)
for PR 2696 at commit
[`787444c`](https://github.com/a
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2696#issuecomment-59871647
[QA tests have
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21956/consoleFull)
for PR 2696 at commit
[`787444c`](https://github.com/ap
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/2696#discussion_r19127119
--- Diff:
core/src/main/scala/org/apache/spark/ui/jobs/JobProgressListener.scala ---
@@ -40,17 +40,25 @@ class JobProgressListener(conf: SparkConf) extends
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/2696#discussion_r19126684
--- Diff:
core/src/main/scala/org/apache/spark/ui/jobs/JobProgressListener.scala ---
@@ -40,17 +40,25 @@ class JobProgressListener(conf: SparkConf) extends
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/2696#issuecomment-59449668
This looks pretty good to me. As discussed offline, it's worth considering
putting the new logic in SparkContext into a trait and mix it in. This will
mark the first s
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/2696#discussion_r18994206
--- Diff: core/src/main/scala/org/apache/spark/ui/jobs/UIData.scala ---
@@ -65,4 +66,10 @@ private[jobs] object UIData {
var taskInfo: TaskInfo,
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/2696#discussion_r18994137
--- Diff:
core/src/main/scala/org/apache/spark/ui/jobs/JobProgressListener.scala ---
@@ -61,8 +69,30 @@ class JobProgressListener(conf: SparkConf) extends
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/2696#discussion_r18993986
--- Diff:
core/src/main/scala/org/apache/spark/ui/jobs/JobProgressListener.scala ---
@@ -61,8 +69,30 @@ class JobProgressListener(conf: SparkConf) extends
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/2696#discussion_r18993968
--- Diff:
core/src/main/scala/org/apache/spark/ui/jobs/JobProgressListener.scala ---
@@ -40,17 +40,25 @@ class JobProgressListener(conf: SparkConf) extends
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/2696#discussion_r18993930
--- Diff: core/src/main/scala/org/apache/spark/ui/SparkUI.scala ---
@@ -116,4 +99,23 @@ private[spark] object SparkUI {
def getUIPort(conf: SparkConf
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/2696#discussion_r18993758
--- Diff: core/src/main/scala/org/apache/spark/ui/SparkUI.scala ---
@@ -116,4 +99,23 @@ private[spark] object SparkUI {
def getUIPort(conf: SparkConf
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/2696#discussion_r18993665
--- Diff: core/src/main/java/org/apache/spark/ExecutionStatus.java ---
@@ -0,0 +1,24 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) und
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2696#issuecomment-59145855
[QA tests have
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21755/consoleFull)
for PR 2696 at commit
[`249ca16`](https://github.com/a
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/2696#issuecomment-59145858
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2696#issuecomment-59139518
[QA tests have
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21755/consoleFull)
for PR 2696 at commit
[`249ca16`](https://github.com/ap
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2696#discussion_r18868548
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -229,10 +233,26 @@ class SparkContext(config: SparkConf) extends Logging
{
priv
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2696#discussion_r18868330
--- Diff: core/src/main/java/org/apache/spark/SparkJobInfo.java ---
@@ -0,0 +1,30 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/2696#discussion_r18867651
--- Diff: core/src/main/java/org/apache/spark/SparkJobInfo.java ---
@@ -0,0 +1,30 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under o
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/2696#discussion_r18864694
--- Diff: core/src/main/scala/org/apache/spark/StatusAPIImpl.scala ---
@@ -0,0 +1,68 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) unde
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/2696#issuecomment-59129325
@vanzin Yeah, maybe we should split SPARK-2321; this PR is only concerned
with progress monitoring; I'm not proposing any stabilization of the listener
APIs here.
---
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/2696#discussion_r18864112
--- Diff: core/src/main/scala/org/apache/spark/StatusAPIImpl.scala ---
@@ -0,0 +1,68 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) unde
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/2696#discussion_r18864057
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -853,6 +873,12 @@ class SparkContext(config: SparkConf) extends Logging {
/**
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/2696#discussion_r18863745
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -229,10 +233,26 @@ class SparkContext(config: SparkConf) extends Logging
{
p
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/2696#discussion_r18863507
--- Diff: core/src/main/scala/org/apache/spark/StatusAPIImpl.scala ---
@@ -0,0 +1,68 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) unde
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2696#discussion_r18863409
--- Diff:
core/src/main/scala/org/apache/spark/api/java/JavaSparkContext.scala ---
@@ -132,6 +132,12 @@ class JavaSparkContext(val sc: SparkContext)
/** D
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/2696#discussion_r18863321
--- Diff:
core/src/main/scala/org/apache/spark/api/java/JavaSparkContext.scala ---
@@ -132,6 +132,12 @@ class JavaSparkContext(val sc: SparkContext)
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/2696#issuecomment-59112305
Hi @JoshRosen,
I think this is going in the right direction and is definitely an
improvement over the status quo. I left some comments about semantics of
certain
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2696#discussion_r18856537
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -853,6 +873,12 @@ class SparkContext(config: SparkConf) extends Logging {
/** Th
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2696#discussion_r18856178
--- Diff: core/src/main/java/org/apache/spark/SparkJobInfo.java ---
@@ -0,0 +1,30 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2696#discussion_r18856060
--- Diff: core/src/main/scala/org/apache/spark/StatusAPIImpl.scala ---
@@ -0,0 +1,68 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under o
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2696#discussion_r18855937
--- Diff: core/src/main/scala/org/apache/spark/StatusAPIImpl.scala ---
@@ -0,0 +1,68 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under o
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2696#discussion_r18855753
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -853,6 +873,12 @@ class SparkContext(config: SparkConf) extends Logging {
/** Th
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2696#discussion_r18855660
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -229,10 +233,26 @@ class SparkContext(config: SparkConf) extends Logging
{
priv
Github user ChengXiangLi commented on a diff in the pull request:
https://github.com/apache/spark/pull/2696#discussion_r18806856
--- Diff: core/src/main/java/org/apache/spark/SparkJobInfo.java ---
@@ -0,0 +1,30 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) unde
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2696#discussion_r18796584
--- Diff: core/src/main/java/org/apache/spark/SparkStageInfo.java ---
@@ -0,0 +1,33 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2696#discussion_r18796556
--- Diff: core/src/main/java/org/apache/spark/SparkJobInfo.java ---
@@ -0,0 +1,30 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2696#discussion_r18796034
--- Diff: core/src/main/scala/org/apache/spark/StatusAPIImpl.scala ---
@@ -0,0 +1,68 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2696#discussion_r18796050
--- Diff: core/src/main/scala/org/apache/spark/StatusAPIImpl.scala ---
@@ -0,0 +1,68 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2696#discussion_r18794207
--- Diff: core/src/main/scala/org/apache/spark/StatusAPIImpl.scala ---
@@ -0,0 +1,68 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2696#discussion_r18794000
--- Diff:
core/src/main/scala/org/apache/spark/api/java/JavaSparkContext.scala ---
@@ -132,6 +132,12 @@ class JavaSparkContext(val sc: SparkContext)
/** D
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/2696#issuecomment-58451983
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/2
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/2696#issuecomment-58276464
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/2
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2696#issuecomment-58276452
[QA tests have
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21407/consoleFull)
for PR 2696 at commit
[`08cbec9`](https://github.com/a
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2696#issuecomment-58269497
[QA tests have
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21407/consoleFull)
for PR 2696 at commit
[`08cbec9`](https://github.com/ap
GitHub user JoshRosen opened a pull request:
https://github.com/apache/spark/pull/2696
[SPARK-2321] [WIP] Stable pull-based progress / status API
This is a WIP pull request that proposes a stable, pull-based progress /
status API for Spark (see
[SPARK-2321](https://issues.apache.or
49 matches
Mail list logo
