Github user squito commented on the pull request:
https://github.com/apache/spark/pull/3029#issuecomment-63149772
Sorry for my delay in responding ...
(a) I think this DOES add a lot of value over the std INFO logging. One
log line per task completion is *much* noisier than w
Github user davies commented on the pull request:
https://github.com/apache/spark/pull/3029#issuecomment-62503514
> But if it doesn't play nicely with logging & multiple stages, this seems
like a very small improvement for the initial user experience, but a big
headache for serious us
Github user squito commented on the pull request:
https://github.com/apache/spark/pull/3029#issuecomment-62499404
I totally see the appeal of the one-progress bar (hence my initial
excitement when I tried this out). But if it doesn't play nicely with logging
& multiple stages, this s
Github user davies commented on the pull request:
https://github.com/apache/spark/pull/3029#issuecomment-62480890
@kayousterhout @squito The motivation of this PR is to allow user to see
the progress without flushing away their input/output, so it's better to have a
one-line progress
Github user kayousterhout commented on the pull request:
https://github.com/apache/spark/pull/3029#issuecomment-62476320
@squito that's a good idea with the minimum time interval to avoid
unnecessary clutter!
---
If your project is set up for it, you can reply to this email and have
Github user squito commented on the pull request:
https://github.com/apache/spark/pull/3029#issuecomment-62475673
I was just about to suggest the same thing . So I admit it seemed a lot
cooler to have the console keep updating, but I agree with their concerns.
As a slight mod
Github user kayousterhout commented on the pull request:
https://github.com/apache/spark/pull/3029#issuecomment-62474857
I'm not sure if this is a good idea, but what if the progress bar printed a
new line each time it advanced, and just used the normal logging
infrastructure? So the
Github user davies commented on the pull request:
https://github.com/apache/spark/pull/3029#issuecomment-62473806
@pwendell @kayousterhout , Thanks for review this.
Console is a character device, all the graph are implemented as an stream
of characters. It's very hard to show
Github user kayousterhout commented on the pull request:
https://github.com/apache/spark/pull/3029#issuecomment-62458897
This is definitely a cool/useful feature! A few things:
(1) As Patrick alluded to, I think it could and should be able to
implemented as a SparkListener, w
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/3029#discussion_r20112906
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala ---
@@ -528,7 +601,7 @@ private[spark] class TaskSetManager(
sche
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/3029#discussion_r20112968
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala ---
@@ -569,6 +641,11 @@ private[spark] class TaskSetManager(
l
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/3029#issuecomment-62454596
This is a very nifty feature. However, it's not great to have modifications
to the TaskSetManager and other scheduler internals for this
presentation-related logic. It w
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/3029#discussion_r20112947
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala ---
@@ -554,12 +627,11 @@ private[spark] class TaskSetManager(
va
Github user squito commented on the pull request:
https://github.com/apache/spark/pull/3029#issuecomment-62330615
this is awesome!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3029#issuecomment-61589367
[Test build #22850 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22850/consoleFull)
for PR 3029 at commit
[`6fd30ff`](https://gith
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3029#issuecomment-61589372
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3029#issuecomment-61588061
[Test build #22845 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22845/consoleFull)
for PR 3029 at commit
[`a60477c`](https://gith
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3029#issuecomment-61588063
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3029#issuecomment-61587948
[Test build #22846 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22846/consoleFull)
for PR 3029 at commit
[`e1f524d`](https://gith
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3029#issuecomment-61587953
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3029#issuecomment-61583786
[Test build #22850 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22850/consoleFull)
for PR 3029 at commit
[`6fd30ff`](https://githu
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3029#issuecomment-61581788
[Test build #22846 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22846/consoleFull)
for PR 3029 at commit
[`e1f524d`](https://githu
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3029#issuecomment-61581372
[Test build #22845 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22845/consoleFull)
for PR 3029 at commit
[`a60477c`](https://githu
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3029#issuecomment-61394842
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3029#issuecomment-61394840
[Test build #22739 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22739/consoleFull)
for PR 3029 at commit
[`5cae3f2`](https://gith
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3029#issuecomment-61393558
[Test build #22739 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22739/consoleFull)
for PR 3029 at commit
[`5cae3f2`](https://githu
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3029#issuecomment-61393376
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3029#issuecomment-61393375
[Test build #22737 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22737/consoleFull)
for PR 3029 at commit
[`ea49fe0`](https://gith
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3029#issuecomment-61393355
[Test build #22737 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22737/consoleFull)
for PR 3029 at commit
[`ea49fe0`](https://githu
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/3029#discussion_r19709497
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala ---
@@ -521,6 +529,51 @@ private[spark] class TaskSetManager(
sched.dagS
Github user aarondav commented on a diff in the pull request:
https://github.com/apache/spark/pull/3029#discussion_r19702297
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala ---
@@ -521,6 +529,51 @@ private[spark] class TaskSetManager(
sched.da
Github user aarondav commented on a diff in the pull request:
https://github.com/apache/spark/pull/3029#discussion_r19702293
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala ---
@@ -521,6 +529,51 @@ private[spark] class TaskSetManager(
sched.da
Github user aarondav commented on a diff in the pull request:
https://github.com/apache/spark/pull/3029#discussion_r19702287
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala ---
@@ -521,6 +529,51 @@ private[spark] class TaskSetManager(
sched.da
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3029#issuecomment-61360812
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3029#issuecomment-61360811
[Test build #22685 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22685/consoleFull)
for PR 3029 at commit
[`bc53d99`](https://gith
Github user davies commented on the pull request:
https://github.com/apache/spark/pull/3029#issuecomment-61359713
@JoshRosen I had make some improvements: 1) finished the bar before print
the result in scala shell 2) can interwave with logging better(will not
overwrite each other) 3)
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3029#issuecomment-61359587
[Test build #22685 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22685/consoleFull)
for PR 3029 at commit
[`bc53d99`](https://githu
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3029#issuecomment-61357827
[Test build #503 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/503/consoleFull)
for PR 3029 at commit
[`e6bb189`](https://githu
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3029#issuecomment-61356828
[Test build #503 has
started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/503/consoleFull)
for PR 3029 at commit
[`e6bb189`](https://github
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3029#issuecomment-61353655
[Test build #22669 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22669/consoleFull)
for PR 3029 at commit
[`e6bb189`](https://gith
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3029#issuecomment-61353658
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3029#issuecomment-61350386
[Test build #22669 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22669/consoleFull)
for PR 3029 at commit
[`e6bb189`](https://githu
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3029#issuecomment-61349718
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3029#issuecomment-61349714
**[Test build #22656 timed
out](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22656/consoleFull)**
for PR 3029 at commit
[`7e7d4e7`](https://git
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3029#issuecomment-61340878
[Test build #22656 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22656/consoleFull)
for PR 3029 at commit
[`7e7d4e7`](https://githu
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/3029#discussion_r19697163
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala ---
@@ -521,6 +528,47 @@ private[spark] class TaskSetManager(
sched.d
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3029#issuecomment-61211030
[Test build #22589 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22589/consoleFull)
for PR 3029 at commit
[`5df26bb`](https://gith
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3029#issuecomment-61211033
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3029#issuecomment-61204861
[Test build #22589 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22589/consoleFull)
for PR 3029 at commit
[`5df26bb`](https://githu
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3029#issuecomment-61199091
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3029#issuecomment-61199090
[Test build #22584 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22584/consoleFull)
for PR 3029 at commit
[`9e42208`](https://gith
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3029#issuecomment-61198910
[Test build #22584 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22584/consoleFull)
for PR 3029 at commit
[`9e42208`](https://githu
Github user davies commented on the pull request:
https://github.com/apache/spark/pull/3029#issuecomment-61198507
cc @JoshRosen @mateiz
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this f
GitHub user davies opened a pull request:
https://github.com/apache/spark/pull/3029
[SPARK-4017] show progress bar in console and title
The progress bar will look like this:
