[GitHub] spark issue #21784: [SPARK-24182][YARN][FOLLOW-UP] Turn off noisy log output
Github user wangyum commented on the issue: https://github.com/apache/spark/pull/21784 cc @vanzin, @jerryshao --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21784: [SPARK-24182][YARN][FOLLOW-UP] Turn off noisy log output
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/21784 Hm, yea. I don't think find this super noisy though to be honest. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21784: [SPARK-24182][YARN][FOLLOW-UP] Turn off noisy log output
Github user wangyum commented on the issue: https://github.com/apache/spark/pull/21784 It's noisy when type something:  --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21784: [SPARK-24182][YARN][FOLLOW-UP] Turn off noisy log output
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/21784
OK, but you can set sc.setLogLevel in the shell. For instance, if I run
`spark.range(10).show()` with INFO, I got something like this:
```
scala> 18/07/18 07:58:47 INFO yarn.Client: Application report for
application_1531383843352_0013 (state: RUNNING)
spark.r18/07/18 07:58:48 INFO yarn.Client: Application report for
application_1531383843352_0013 (state: RUNNING)
ange(10)18/07/18 07:58:49 INFO yarn.Client: Application report for
application_1531383843352_0013 (state: RUNNING)
.show()18/07/18 07:58:50 INFO yarn.Client: Application report for
application_1531383843352_0013 (state: RUNNING)
18/07/18 07:58:51 INFO internal.SharedState: loading hive config file:
file:/home/spark/spark/conf/hive-site.xml
18/07/18 07:58:51 INFO internal.SharedState: Setting
hive.metastore.warehouse.dir ('null') to the value of spark.sql.warehouse.dir
('file:/home/spark/spark/spark-warehouse').
18/07/18 07:58:51 INFO internal.SharedState: Warehouse path is
'file:/home/spark/spark/spark-warehouse'.
18/07/18 07:58:51 INFO ui.JettyUtils: Adding filter
org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter to /SQL.
18/07/18 07:58:51 INFO handler.ContextHandler: Started
o.e.j.s.ServletContextHandler@7a68818c{/SQL,null,AVAILABLE,@Spark}
18/07/18 07:58:51 INFO ui.JettyUtils: Adding filter
org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter to /SQL/json.
18/07/18 07:58:51 INFO handler.ContextHandler: Started
o.e.j.s.ServletContextHandler@5f745970{/SQL/json,null,AVAILABLE,@Spark}
18/07/18 07:58:51 INFO ui.JettyUtils: Adding filter
org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter to /SQL/execution.
18/07/18 07:58:51 INFO handler.ContextHandler: Started
o.e.j.s.ServletContextHandler@2afd8972{/SQL/execution,null,AVAILABLE,@Spark}
18/07/18 07:58:51 INFO ui.JettyUtils: Adding filter
org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter to
/SQL/execution/json.
18/07/18 07:58:51 INFO handler.ContextHandler: Started
o.e.j.s.ServletContextHandler@5784f6b9{/SQL/execution/json,null,AVAILABLE,@Spark}
18/07/18 07:58:51 INFO ui.JettyUtils: Adding filter
org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter to /static/sql.
18/07/18 07:58:51 INFO handler.ContextHandler: Started
o.e.j.s.ServletContextHandler@6ccf06f1{/static/sql,null,AVAILABLE,@Spark}
18/07/18 07:58:51 INFO yarn.Client: Application report for
application_1531383843352_0013 (state: RUNNING)
18/07/18 07:58:52 INFO state.StateStoreCoordinatorRef: Registered
StateStoreCoordinator endpoint
18/07/18 07:58:52 INFO yarn.Client: Application report for
application_1531383843352_0013 (state: RUNNING)
18/07/18 07:58:53 INFO codegen.CodeGenerator: Code generated in 254.142542
ms
18/07/18 07:58:53 INFO codegen.CodeGenerator: Code generated in 65.397101 ms
18/07/18 07:58:53 INFO spark.SparkContext: Starting job: show at
:24
18/07/18 07:58:53 INFO scheduler.DAGScheduler: Got job 0 (show at
:24) with 1 output partitions
18/07/18 07:58:53 INFO scheduler.DAGScheduler: Final stage: ResultStage 0
(show at :24)
18/07/18 07:58:53 INFO scheduler.DAGScheduler: Parents of final stage:
List()
18/07/18 07:58:53 INFO scheduler.DAGScheduler: Missing parents: List()
18/07/18 07:58:53 INFO scheduler.DAGScheduler: Submitting ResultStage 0
(MapPartitionsRDD[4] at show at :24), which has no missing parents
18/07/18 07:58:53 INFO yarn.Client: Application report for
application_1531383843352_0013 (state: RUNNING)
18/07/18 07:58:53 INFO memory.MemoryStore: Block broadcast_0 stored as
values in memory (estimated size 8.1 KB, free 408.9 MB)
18/07/18 07:58:53 INFO memory.MemoryStore: Block broadcast_0_piece0 stored
as bytes in memory (estimated size 4.0 KB, free 408.9 MB)
18/07/18 07:58:53 INFO storage.BlockManagerInfo: Added broadcast_0_piece0
in memory on ctr-e138-1518143905142-411342-01-02.hwx.site:41073 (size: 4.0
KB, free: 408.9 MB)
18/07/18 07:58:53 INFO spark.SparkContext: Created broadcast 0 from
broadcast at DAGScheduler.scala:1039
18/07/18 07:58:54 INFO scheduler.DAGScheduler: Submitting 1 missing tasks
from ResultStage 0 (MapPartitionsRDD[4] at show at :24) (first 15
tasks are for partitions Vector(0))
18/07/18 07:58:54 INFO cluster.YarnScheduler: Adding task set 0.0 with 1
tasks
18/07/18 07:58:54 INFO scheduler.TaskSetManager: Starting task 0.0 in stage
0.0 (TID 0, ctr-e138-1518143905142-411342-01-08.hwx.site, executor 2,
partition 0, PROCESS_LOCAL, 7864 bytes)
[Stage 0:> (0 + 1)
/ 1]18/07/18 07:58:54 INFO storage.BlockManagerInfo: Added broadcast_0_piece0
in memory on ctr-e138-1518143905142-411342-01-08.hwx.site:37671 (size: 4.0
KB, free: 366.3 MB)
18/07/18 07:58:54 INFO yarn.Client: Application report for
appl
[GitHub] spark issue #21784: [SPARK-24182][YARN][FOLLOW-UP] Turn off noisy log output
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/21784 @vanzin WDYT about this? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21784: [SPARK-24182][YARN][FOLLOW-UP] Turn off noisy log output
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/21784 @wangyum, mind adding Closes #21784 here? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21784: [SPARK-24182][YARN][FOLLOW-UP] Turn off noisy log output
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/21784 Also, mind adding `[SPARK-24873]` in the PR title since the JIRA happened to be open anyway. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21784: [SPARK-24182][YARN][FOLLOW-UP] Turn off noisy log output
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21784 **[Test build #93111 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93111/testReport)** for PR 21784 at commit [`d6563ec`](https://github.com/apache/spark/commit/d6563ecd4708088b5e87fa62629c3c5048c91b6c). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21784: [SPARK-24182][YARN][FOLLOW-UP] Turn off noisy log output
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21784 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/1012/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21784: [SPARK-24182][YARN][FOLLOW-UP] Turn off noisy log output
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21784 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21784: [SPARK-24182][YARN][FOLLOW-UP] Turn off noisy log output
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21784 **[Test build #93111 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93111/testReport)** for PR 21784 at commit [`d6563ec`](https://github.com/apache/spark/commit/d6563ecd4708088b5e87fa62629c3c5048c91b6c). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21784: [SPARK-24182][YARN][FOLLOW-UP] Turn off noisy log output
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21784 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21784: [SPARK-24182][YARN][FOLLOW-UP] Turn off noisy log output
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21784 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93111/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
