Github user davies commented on the pull request: https://github.com/apache/spark/pull/3029#issuecomment-62503514 > But if it doesn't play nicely with logging & multiple stages, this seems like a very small improvement for the initial user experience, but a big headache for serious users. If the logging level is DEBUG/INFO, progress bar can not contribute more value than logging, so it's fine to disable it. In WARN level, the logging should be much less (even nothing), in case there is some logging, they will look like this ``` [Stage 2] ==============> ] 10 + 5/100 xxx [WARN] xxxxxxxx [Stage 2] =================> ] 15 + 5/100 xxx ``` I think it's not bad, especially it's not common cases. If there are multiple stages, which are not running concurrently (this is common cases), the progress bar will be showed stage by stage, looks like ``` [Stage 1] Finished in xxx seconds. (med/avg/xxx) [Stage 2] =================> ] 15 + 5/100 xxx ``` We can improve the case in which multiple stages run concurrently, looks like ``` [Stage 1/2/3] [=============> ] 140+39/340 xxxx ``` So, I can not agree with you that the current approach does not play nicely with logging/multiple stages. > In this case its just a couple more lines. If the stage took longer, than it would be even more lines, but that seems ok, since its not that much noise per unit time. Progress bar is useful for slow jobs, so it's expected to take long time (maybe more than 1 minute) to finish a stage, then the progress bar will occupy the whole screen (more than 40 lines), use need to scroll the screen to see previous output (results). If we use INFO for progress bar, we need to special trick to enable progress bar also disable others. If we can do this, we can filter out the progress logging without all others right now. It will not so easy to use to users, special for user who is not familiar with log4j.
--- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org