Github user davies commented on the pull request:
https://github.com/apache/spark/pull/3029#issuecomment-62503514
> But if it doesn't play nicely with logging & multiple stages, this seems
like a very small improvement for the initial user experience, but a big
headache for serious users.
If the logging level is DEBUG/INFO, progress bar can not contribute more
value than logging, so it's fine to disable it. In WARN level, the logging
should be much less (even nothing), in case there is some logging, they will
look like this
```
[Stage 2] ==============> ] 10 +
5/100 xxx
[WARN] xxxxxxxx
[Stage 2] =================> ] 15 +
5/100 xxx
```
I think it's not bad, especially it's not common cases.
If there are multiple stages, which are not running concurrently (this is
common cases), the progress bar will be showed stage by stage, looks like
```
[Stage 1] Finished in xxx seconds. (med/avg/xxx)
[Stage 2] =================> ] 15 +
5/100 xxx
```
We can improve the case in which multiple stages run concurrently, looks
like
```
[Stage 1/2/3] [=============> ]
140+39/340 xxxx
```
So, I can not agree with you that the current approach does not play nicely
with logging/multiple stages.
> In this case its just a couple more lines. If the stage took longer, than
it would be even more lines, but that seems ok, since its not that much noise
per unit time.
Progress bar is useful for slow jobs, so it's expected to take long time
(maybe more than 1 minute) to finish a stage, then the progress bar will occupy
the whole screen (more than 40 lines), use need to scroll the screen to see
previous output (results).
If we use INFO for progress bar, we need to special trick to enable
progress bar also disable others. If we can do this, we can filter out the
progress logging without all others right now. It will not so easy to use to
users, special for user who is not familiar with log4j.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
