Re: [PR] [SPARK-48124][CORE] Disable structured logging for Interpreters by default [spark]
gengliangwang commented on PR #46383: URL: https://github.com/apache/spark/pull/46383#issuecomment-2094365561 Thank you, @dongjoon-hyun ! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
Re: [PR] [SPARK-48124][CORE] Disable structured logging for Interpreters by default [spark]
dongjoon-hyun commented on PR #46383: URL: https://github.com/apache/spark/pull/46383#issuecomment-2094345099 Merged to master for Apache Spark 4.0.0-preview. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
Re: [PR] [SPARK-48124][CORE] Disable structured logging for Interpreters by default [spark]
dongjoon-hyun closed pull request #46383: [SPARK-48124][CORE] Disable structured logging for Interpreters by default URL: https://github.com/apache/spark/pull/46383 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
Re: [PR] [SPARK-48124][Core] Disable structured logging for Interpreters by default [spark]
gengliangwang commented on code in PR #46383: URL: https://github.com/apache/spark/pull/46383#discussion_r1589920680 ## docs/core-migration-guide.md: ## @@ -42,7 +42,7 @@ license: | - Since Spark 4.0, Spark uses the external shuffle service for deleting shuffle blocks for deallocated executors when the shuffle is no longer needed. To restore the legacy behavior, you can set `spark.shuffle.service.removeShuffle` to `false`. -- Since Spark 4.0, the default log4j output has shifted from plain text to JSON lines to enhance analyzability. To revert to plain text output, you can either set `spark.log.structuredLogging.enabled` to `false`, or use a custom log4j configuration. Review Comment: Note: the flag `spark.log.structuredLogging.enabled` is not taking effect as expected. I created https://issues.apache.org/jira/browse/SPARK-48126 as a follow-up. Let's not mention it in the Spark 4.0 preview release. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[PR] [SPARK-48124][Core] Disable structured logging for Interpreters by default [spark]
gengliangwang opened a new pull request, #46383:
URL: https://github.com/apache/spark/pull/46383
### What changes were proposed in this pull request?
For interpreters, structured logging should be disabled by default to avoid
generating mixed plain text and structured logs on the same console.
spark-shell output with mixed plain text and structured logs:
```
Using Scala version 2.13.13 (OpenJDK 64-Bit Server VM, Java 17.0.9)
Type in expressions to have them evaluated.
Type :help for more information.
{"ts":"2024-05-04T01:11:03.797Z","level":"WARN","msg":"Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable","logger":"NativeCodeLoader"}
{"ts":"2024-05-04T01:11:04.104Z","level":"WARN","msg":"Service 'SparkUI' could
not bind on port 4040. Attempting port 4041.","logger":"Utils"}
Spark context Web UI available at http://10.10.114.155:4041/
Spark context available as 'sc' (master = local[*], app id =
local-1714785064155).
Spark session available as 'spark'.
```
After changes, all the output are plain text:
```
Type :help for more information.
24/05/03 18:11:35 WARN NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
24/05/03 18:11:35 WARN Utils: Service 'SparkUI' could not bind on port 4040.
Attempting port 4041.
Spark context Web UI available at http://10.10.114.155:4041/
Spark context available as 'sc' (master = local[*], app id =
local-1714785095892).
Spark session available as 'spark'.
```
Note that submitting a spark application using `spark-submit` will still
generates structured logs.
### Why are the changes needed?
To avoid tgenerating mixed plain text and structured logs on the same
console when using the Interpreters.
### Does this PR introduce _any_ user-facing change?
No, this reverts to the behavior of Spark 3.5
### How was this patch tested?
Manual test
### Was this patch authored or co-authored using generative AI tooling?
No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
