[GitHub] spark pull request: [SPARK-4858] Add an option to turn off a progr...
Github user maropu commented on the pull request: https://github.com/apache/spark/pull/3709#issuecomment-69450446 Ok. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4858] Add an option to turn off a progr...
Github user maropu closed the pull request at: https://github.com/apache/spark/pull/3709 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4858] Add an option to turn off a progr...
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/3709#issuecomment-69236555 Yeah that's a good point. It seems that `--conf` is sufficient for this use case. @maropu would you mind closing this issue? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4858] Add an option to turn off a progr...
Github user davies commented on the pull request: https://github.com/apache/spark/pull/3709#issuecomment-69144559 @pwendell This is a better solution, we can not continue to add more flags for every useful user-cases. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4858] Add an option to turn off a progr...
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/3709#issuecomment-69143775 Just wondering, it seems like a fairly esoteric option for adding a built-in flag. Why not just do this? ``` ./bin/spark-shell --conf spark.ui.showConsoleProgress=false ``` Is it a common thing to disable this progress bar? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4858] Add an option to turn off a progr...
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/3709#discussion_r22625797
--- Diff: bin/utils.sh ---
@@ -32,10 +32,11 @@ function gatherSparkSubmitOpts() {
APPLICATION_OPTS=()
while (($#)); do
case "$1" in
- --master | --deploy-mode | --class | --name | --jars | --py-files |
--files | \
- --conf | --properties-file | --driver-memory | --driver-java-options
| \
- --driver-library-path | --driver-class-path | --executor-memory |
--driver-cores | \
- --total-executor-cores | --executor-cores | --queue |
--num-executors | --archives)
+ --master | --deploy-mode | --class | --name | --jars |
--no-progress-bar | \
--- End diff --
The diff will be minimized, easy to review and blame (in future).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4858] Add an option to turn off a progr...
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/3709#issuecomment-69110931 @davies any other comments? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4858] Add an option to turn off a progr...
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/3709#discussion_r22624919
--- Diff: bin/utils.sh ---
@@ -32,10 +32,11 @@ function gatherSparkSubmitOpts() {
APPLICATION_OPTS=()
while (($#)); do
case "$1" in
- --master | --deploy-mode | --class | --name | --jars | --py-files |
--files | \
- --conf | --properties-file | --driver-memory | --driver-java-options
| \
- --driver-library-path | --driver-class-path | --executor-memory |
--driver-cores | \
- --total-executor-cores | --executor-cores | --queue |
--num-executors | --archives)
+ --master | --deploy-mode | --class | --name | --jars |
--no-progress-bar | \
--- End diff --
please also update `bin/windows-utils.cmd`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4858] Add an option to turn off a progr...
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/3709#discussion_r22624759 --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala --- @@ -380,6 +386,8 @@ private[spark] class SparkSubmitArguments(args: Seq[String], env: Map[String, St | | --executor-memory MEM Memory per executor (e.g. 1000M, 2G) (Default: 1G). | +| --no-progress-bar Turn off a console progress bar --- End diff -- Turn off the --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4858] Add an option to turn off a progr...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3709#issuecomment-68055152 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24785/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4858] Add an option to turn off a progr...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3709#issuecomment-68055149 [Test build #24785 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24785/consoleFull) for PR 3709 at commit [`0681403`](https://github.com/apache/spark/commit/06814035d454a9b7e444b0dc657a572a6ae2f899). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4858] Add an option to turn off a progr...
Github user maropu commented on the pull request: https://github.com/apache/spark/pull/3709#issuecomment-68050965 Fixed, please test it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4858] Add an option to turn off a progr...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3709#issuecomment-68050938 [Test build #24785 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24785/consoleFull) for PR 3709 at commit [`0681403`](https://github.com/apache/spark/commit/06814035d454a9b7e444b0dc657a572a6ae2f899). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4858] Add an option to turn off a progr...
Github user davies commented on the pull request: https://github.com/apache/spark/pull/3709#issuecomment-68016895 @maropu Thank you for explain this, it sounds good to me. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4858] Add an option to turn off a progr...
Github user maropu commented on the pull request: https://github.com/apache/spark/pull/3709#issuecomment-68016680 @andrewor14 OK, I'll re-check it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4858] Add an option to turn off a progr...
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/3709#issuecomment-67896317 Looks like this one has a legitimate test failure --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4858] Add an option to turn off a progr...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3709#issuecomment-67891795 [Test build #24713 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24713/consoleFull) for PR 3709 at commit [`e17e88f`](https://github.com/apache/spark/commit/e17e88f86e6f02cd9848b83e2832611790c138b2). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4858] Add an option to turn off a progr...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3709#issuecomment-67891805 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24713/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4858] Add an option to turn off a progr...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3709#issuecomment-67884964 [Test build #24713 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24713/consoleFull) for PR 3709 at commit [`e17e88f`](https://github.com/apache/spark/commit/e17e88f86e6f02cd9848b83e2832611790c138b2). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4858] Add an option to turn off a progr...
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/3709#issuecomment-67884407 ok to test --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4858] Add an option to turn off a progr...
Github user maropu commented on a diff in the pull request:
https://github.com/apache/spark/pull/3709#discussion_r21958440
--- Diff: bin/utils.sh ---
@@ -32,10 +32,11 @@ function gatherSparkSubmitOpts() {
APPLICATION_OPTS=()
while (($#)); do
case "$1" in
- --master | --deploy-mode | --class | --name | --jars | --py-files |
--files | \
- --conf | --properties-file | --driver-memory | --driver-java-options
| \
- --driver-library-path | --driver-class-path | --executor-memory |
--driver-cores | \
- --total-executor-cores | --executor-cores | --queue |
--num-executors | --archives)
+ --master | --deploy-mode | --class | --name | --jars |
--no-progress-bar | \
--- End diff --
No reason. Is it better to place the end?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4858] Add an option to turn off a progr...
Github user maropu commented on the pull request: https://github.com/apache/spark/pull/3709#issuecomment-67291895 @davies Thx for your comment :) IMHO, it'd be better that switching on/off the bar is independent of log4j log level. This is useful for users to look into some user-defined logs in WARNING levels when user-written codes in Spark cause some exceptions. Currently, the bar slightly makes me confused when I'd like to check some interested logs, so I think easily switching-off is better in a spark-shell option. Thanks, takeshi --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4858] Add an option to turn off a progr...
Github user davies commented on the pull request: https://github.com/apache/spark/pull/3709#issuecomment-67118160 @maropu Thank you for work in this. I'm just wondering what's the user case that his argument will be useful? Because progress bar is disabled if logging level is DEBUG or INFO. This is useful when you have customized log4j configs, such as having default logger level as WARNING, but having INFO/DEBUG in some components. In these cases, the command line arguments will work better than put `spark.ui.showConsoleProgress = false` in spark-default.conf? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4858] Add an option to turn off a progr...
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/3709#discussion_r21879847
--- Diff: bin/utils.sh ---
@@ -32,10 +32,11 @@ function gatherSparkSubmitOpts() {
APPLICATION_OPTS=()
while (($#)); do
case "$1" in
- --master | --deploy-mode | --class | --name | --jars | --py-files |
--files | \
- --conf | --properties-file | --driver-memory | --driver-java-options
| \
- --driver-library-path | --driver-class-path | --executor-memory |
--driver-cores | \
- --total-executor-cores | --executor-cores | --queue |
--num-executors | --archives)
+ --master | --deploy-mode | --class | --name | --jars |
--no-progress-bar | \
--- End diff --
Is there any reason to put `--no-progress-bar` here(instead of in the end)?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4858] Add an option to turn off a progr...
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/3709#issuecomment-67117052 /cc @davies --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4858] Add an option to turn off a progr...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3709#issuecomment-67110752 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4858] Add an option to turn off a progr...
GitHub user maropu opened a pull request: https://github.com/apache/spark/pull/3709 [SPARK-4858] Add an option to turn off a progress bar in spark-shell Add an '--no-progress-bar' option to easily turn off a progress bar in spark-shell for users who'd like to look into debug logs or something. You can merge this pull request into a Git repository by running: $ git pull https://github.com/maropu/spark ShowConsoleProgressBarOptionSpike Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/3709.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3709 commit e17e88f86e6f02cd9848b83e2832611790c138b2 Author: Takeshi Yamamuro Date: 2014-12-10T13:59:12Z Add an option to turn off a progress bar in spark-shell --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
