[
https://issues.apache.org/jira/browse/AMBARI-13740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Drazen Zubovic updated AMBARI-13740:
------------------------------------
Description:
It seems like the 'spark-log4j-properties' configuration passed from Ambari |
Services | Spark | Configs is getting corrupt and reformatted into one line
without LF. The log4j config file is getting compressed into a field, with all
LF characters dropped out. This configuration apparently cannot be changed into
normal/standard multi line configuration that would be meaningful to Spark.
This way it seems like the whole log4j configuration is being ignored
completely. If not due to the syntactical sanity, then at least due to the
first character is '#', which applies to the whole line. The default Spark
loglevel is set to INFO, so consequently console prints whole number of INFO
messages for each statement passed to the console. Consequently further, the
loglevel cannot be changed to WARN or ERROR and INFO lines cannot be suppressed
from the console.
--> The log4j.properties.template configuration looks like this:
# Set everything to be logged to the console
log4j.rootCategory=INFO, console
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.target=System.err
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{1}:
%m%n
# Settings to quiet third party logs that are too verbose
log4j.logger.org.eclipse.jetty=WARN
log4j.logger.org.eclipse.jetty.util.component.AbstractLifeCycle=ERROR
log4j.logger.org.apache.spark.repl.SparkIMain$exprTyper=INFO
log4j.logger.org.apache.spark.repl.SparkILoop$SparkILoopInterpreter=INFO
--> The log4j.properties configuration looks like this:
# Set everything to be logged to the consolelog4j.rootCategory=WARN,
consolelog4j.appender.console=org.apache.log4j.ConsoleAppenderlog4j.appender.console.target=System.errlog4j.appender.console.layout=org.apache.log4j.PatternLayoutlog4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd
HH:mm:ss} %p %c{1}: %m%n# Settings to quiet third party logs that are too
verboselog4j.logger.org.eclipse.jetty=WARNlog4j.logger.org.eclipse.jetty.util.component.AbstractLifeCycle=ERRORlog4j.logger.org.apache.spark.repl.SparkIMain$exprTyper=INFOlog4j.logger.org.apache.spark.repl.SparkILoop$SparkILoopInterpreter=INFO
--> This issue leads to the ### INFO messages that log4j prints in Spark
console, without possibility to change the loglevel:
[root@sandbox ~]# spark-shell --master yarn-client --driver-memory 512m
--executor-memory 512m
log4j:WARN No appenders could be found for logger
(org.apache.hadoop.metrics2.lib.MutableMetricsFactory).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more
info.
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
15/11/05 12:34:41 INFO SecurityManager: Changing view acls to: root
15/11/05 12:34:41 INFO SecurityManager: Changing modify acls to: root
15/11/05 12:34:41 INFO SecurityManager: SecurityManager: authentication
disabled; ui acls disabled; users with view permissions: Set(root); users with
modify permissions: Set(root)
15/11/05 12:34:41 INFO HttpServer: Starting HTTP Server
15/11/05 12:34:41 INFO Server: jetty-8.y.z-SNAPSHOT
15/11/05 12:34:41 INFO AbstractConnector: Started [email protected]:54564
15/11/05 12:34:41 INFO Utils: Successfully started service 'HTTP class server'
on port 54564.
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 1.3.1
/_/
Using Scala version 2.10.4 (OpenJDK 64-Bit Server VM, Java 1.7.0_85)
Type in expressions to have them evaluated.
Type :help for more information.
15/11/05 12:34:44 INFO SparkContext: Running Spark version 1.3.1
15/11/05 12:34:44 INFO SecurityManager: Changing view acls to: root
15/11/05 12:34:44 INFO SecurityManager: Changing modify acls to: root
15/11/05 12:34:44 INFO SecurityManager: SecurityManager: authentication
disabled; ui acls disabled; users with view permissions: Set(root); users with
modify permissions: Set(root)
15/11/05 12:34:44 INFO Slf4jLogger: Slf4jLogger started
15/11/05 12:34:44 INFO Remoting: Starting remoting
15/11/05 12:34:44 INFO Remoting: Remoting started; listening on addresses
:[akka.tcp://[email protected]:35020]
15/11/05 12:34:44 INFO Utils: Successfully started service 'sparkDriver' on
port 35020.
15/11/05 12:34:44 INFO SparkEnv: Registering MapOutputTracker
15/11/05 12:34:44 INFO SparkEnv: Registering BlockManagerMaster
15/11/05 12:34:44 INFO DiskBlockManager: Created local directory at
/tmp/spark-3b14e9fc-73b1-45e7-96f3-f0ee48f3a238/blockmgr-ffd0619a-5fbc-4ccc-bcef-ce8560e03cd7
15/11/05 12:34:44 INFO MemoryStore: MemoryStore started with capacity 265.4 MB
15/11/05 12:34:45 INFO HttpFileServer: HTTP File server directory is
/tmp/spark-7648f575-a8e3-4b25-945f-b4ded8631592/httpd-ec5526bb-4de6-42d6-83aa-a1ba0a2ca5dd
15/11/05 12:34:45 INFO HttpServer: Starting HTTP Server
15/11/05 12:34:45 INFO Server: jetty-8.y.z-SNAPSHOT
15/11/05 12:34:45 INFO AbstractConnector: Started [email protected]:58853
15/11/05 12:34:45 INFO Utils: Successfully started service 'HTTP file server'
on port 58853.
15/11/05 12:34:45 INFO SparkEnv: Registering OutputCommitCoordinator
1
...
...
was:
It seems like spark-log4j-properties is getting corrupt and reformatted into
one line without LF. Like the log4j config file is compressed into a field,
with LF chars masked out. The configuration apparently cannot be changed into a
multi line configuration that would be meaningful to Spark. This way it seems
that whole log4j configuration is being ignored and takes default values.
Default loglevel is INFO, so consequently console prints number of INFO
messages for each statement. Consequently further, the loglevel cannot be
changed to WARN or ERROR. Since all lines from file are compressed into single
line with '#' character at the beginning it looks like the log4j configuration
for Spark is completely ignored.
--> The log4j.properties.template configuration looks like this:
# Set everything to be logged to the console
log4j.rootCategory=INFO, console
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.target=System.err
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{1}:
%m%n
# Settings to quiet third party logs that are too verbose
log4j.logger.org.eclipse.jetty=WARN
log4j.logger.org.eclipse.jetty.util.component.AbstractLifeCycle=ERROR
log4j.logger.org.apache.spark.repl.SparkIMain$exprTyper=INFO
log4j.logger.org.apache.spark.repl.SparkILoop$SparkILoopInterpreter=INFO
--> The log4j.properties configuration looks like this:
# Set everything to be logged to the consolelog4j.rootCategory=WARN,
consolelog4j.appender.console=org.apache.log4j.ConsoleAppenderlog4j.appender.console.target=System.errlog4j.appender.console.layout=org.apache.log4j.PatternLayoutlog4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd
HH:mm:ss} %p %c{1}: %m%n# Settings to quiet third party logs that are too
verboselog4j.logger.org.eclipse.jetty=WARNlog4j.logger.org.eclipse.jetty.util.component.AbstractLifeCycle=ERRORlog4j.logger.org.apache.spark.repl.SparkIMain$exprTyper=INFOlog4j.logger.org.apache.spark.repl.SparkILoop$SparkILoopInterpreter=INFO
--> This issue leads to the ### INFO messages that log4j prints in Spark
console, without possibility to change the loglevel:
[root@sandbox ~]# spark-shell --master yarn-client --driver-memory 512m
--executor-memory 512m
log4j:WARN No appenders could be found for logger
(org.apache.hadoop.metrics2.lib.MutableMetricsFactory).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more
info.
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
15/11/05 12:34:41 INFO SecurityManager: Changing view acls to: root
15/11/05 12:34:41 INFO SecurityManager: Changing modify acls to: root
15/11/05 12:34:41 INFO SecurityManager: SecurityManager: authentication
disabled; ui acls disabled; users with view permissions: Set(root); users with
modify permissions: Set(root)
15/11/05 12:34:41 INFO HttpServer: Starting HTTP Server
15/11/05 12:34:41 INFO Server: jetty-8.y.z-SNAPSHOT
15/11/05 12:34:41 INFO AbstractConnector: Started [email protected]:54564
15/11/05 12:34:41 INFO Utils: Successfully started service 'HTTP class server'
on port 54564.
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 1.3.1
/_/
Using Scala version 2.10.4 (OpenJDK 64-Bit Server VM, Java 1.7.0_85)
Type in expressions to have them evaluated.
Type :help for more information.
15/11/05 12:34:44 INFO SparkContext: Running Spark version 1.3.1
15/11/05 12:34:44 INFO SecurityManager: Changing view acls to: root
15/11/05 12:34:44 INFO SecurityManager: Changing modify acls to: root
15/11/05 12:34:44 INFO SecurityManager: SecurityManager: authentication
disabled; ui acls disabled; users with view permissions: Set(root); users with
modify permissions: Set(root)
15/11/05 12:34:44 INFO Slf4jLogger: Slf4jLogger started
15/11/05 12:34:44 INFO Remoting: Starting remoting
15/11/05 12:34:44 INFO Remoting: Remoting started; listening on addresses
:[akka.tcp://[email protected]:35020]
15/11/05 12:34:44 INFO Utils: Successfully started service 'sparkDriver' on
port 35020.
15/11/05 12:34:44 INFO SparkEnv: Registering MapOutputTracker
15/11/05 12:34:44 INFO SparkEnv: Registering BlockManagerMaster
15/11/05 12:34:44 INFO DiskBlockManager: Created local directory at
/tmp/spark-3b14e9fc-73b1-45e7-96f3-f0ee48f3a238/blockmgr-ffd0619a-5fbc-4ccc-bcef-ce8560e03cd7
15/11/05 12:34:44 INFO MemoryStore: MemoryStore started with capacity 265.4 MB
15/11/05 12:34:45 INFO HttpFileServer: HTTP File server directory is
/tmp/spark-7648f575-a8e3-4b25-945f-b4ded8631592/httpd-ec5526bb-4de6-42d6-83aa-a1ba0a2ca5dd
15/11/05 12:34:45 INFO HttpServer: Starting HTTP Server
15/11/05 12:34:45 INFO Server: jetty-8.y.z-SNAPSHOT
15/11/05 12:34:45 INFO AbstractConnector: Started [email protected]:58853
15/11/05 12:34:45 INFO Utils: Successfully started service 'HTTP file server'
on port 58853.
15/11/05 12:34:45 INFO SparkEnv: Registering OutputCommitCoordinator
1
...
...
> Corrupt 'spark-log4j-properties' for Spark service
> --------------------------------------------------
>
> Key: AMBARI-13740
> URL: https://issues.apache.org/jira/browse/AMBARI-13740
> Project: Ambari
> Issue Type: Bug
> Components: ambari-admin, ambari-client
> Affects Versions: 2.1.0
> Environment: Hortonworks sandbox 2.3.1; HDP-2.3.0.0-2557
> Reporter: Drazen Zubovic
> Priority: Minor
>
> It seems like the 'spark-log4j-properties' configuration passed from Ambari |
> Services | Spark | Configs is getting corrupt and reformatted into one line
> without LF. The log4j config file is getting compressed into a field, with
> all LF characters dropped out. This configuration apparently cannot be
> changed into normal/standard multi line configuration that would be
> meaningful to Spark.
> This way it seems like the whole log4j configuration is being ignored
> completely. If not due to the syntactical sanity, then at least due to the
> first character is '#', which applies to the whole line. The default Spark
> loglevel is set to INFO, so consequently console prints whole number of INFO
> messages for each statement passed to the console. Consequently further, the
> loglevel cannot be changed to WARN or ERROR and INFO lines cannot be
> suppressed from the console.
> --> The log4j.properties.template configuration looks like this:
> # Set everything to be logged to the console
> log4j.rootCategory=INFO, console
> log4j.appender.console=org.apache.log4j.ConsoleAppender
> log4j.appender.console.target=System.err
> log4j.appender.console.layout=org.apache.log4j.PatternLayout
> log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p
> %c{1}: %m%n
> # Settings to quiet third party logs that are too verbose
> log4j.logger.org.eclipse.jetty=WARN
> log4j.logger.org.eclipse.jetty.util.component.AbstractLifeCycle=ERROR
> log4j.logger.org.apache.spark.repl.SparkIMain$exprTyper=INFO
> log4j.logger.org.apache.spark.repl.SparkILoop$SparkILoopInterpreter=INFO
> --> The log4j.properties configuration looks like this:
> # Set everything to be logged to the consolelog4j.rootCategory=WARN,
> consolelog4j.appender.console=org.apache.log4j.ConsoleAppenderlog4j.appender.console.target=System.errlog4j.appender.console.layout=org.apache.log4j.PatternLayoutlog4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd
> HH:mm:ss} %p %c{1}: %m%n# Settings to quiet third party logs that are too
> verboselog4j.logger.org.eclipse.jetty=WARNlog4j.logger.org.eclipse.jetty.util.component.AbstractLifeCycle=ERRORlog4j.logger.org.apache.spark.repl.SparkIMain$exprTyper=INFOlog4j.logger.org.apache.spark.repl.SparkILoop$SparkILoopInterpreter=INFO
> --> This issue leads to the ### INFO messages that log4j prints in Spark
> console, without possibility to change the loglevel:
> [root@sandbox ~]# spark-shell --master yarn-client --driver-memory 512m
> --executor-memory 512m
> log4j:WARN No appenders could be found for logger
> (org.apache.hadoop.metrics2.lib.MutableMetricsFactory).
> log4j:WARN Please initialize the log4j system properly.
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more
> info.
> Using Spark's default log4j profile:
> org/apache/spark/log4j-defaults.properties
> 15/11/05 12:34:41 INFO SecurityManager: Changing view acls to: root
> 15/11/05 12:34:41 INFO SecurityManager: Changing modify acls to: root
> 15/11/05 12:34:41 INFO SecurityManager: SecurityManager: authentication
> disabled; ui acls disabled; users with view permissions: Set(root); users
> with modify permissions: Set(root)
> 15/11/05 12:34:41 INFO HttpServer: Starting HTTP Server
> 15/11/05 12:34:41 INFO Server: jetty-8.y.z-SNAPSHOT
> 15/11/05 12:34:41 INFO AbstractConnector: Started
> [email protected]:54564
> 15/11/05 12:34:41 INFO Utils: Successfully started service 'HTTP class
> server' on port 54564.
> Welcome to
> ____ __
> / __/__ ___ _____/ /__
> _\ \/ _ \/ _ `/ __/ '_/
> /___/ .__/\_,_/_/ /_/\_\ version 1.3.1
> /_/
> Using Scala version 2.10.4 (OpenJDK 64-Bit Server VM, Java 1.7.0_85)
> Type in expressions to have them evaluated.
> Type :help for more information.
> 15/11/05 12:34:44 INFO SparkContext: Running Spark version 1.3.1
> 15/11/05 12:34:44 INFO SecurityManager: Changing view acls to: root
> 15/11/05 12:34:44 INFO SecurityManager: Changing modify acls to: root
> 15/11/05 12:34:44 INFO SecurityManager: SecurityManager: authentication
> disabled; ui acls disabled; users with view permissions: Set(root); users
> with modify permissions: Set(root)
> 15/11/05 12:34:44 INFO Slf4jLogger: Slf4jLogger started
> 15/11/05 12:34:44 INFO Remoting: Starting remoting
> 15/11/05 12:34:44 INFO Remoting: Remoting started; listening on addresses
> :[akka.tcp://[email protected]:35020]
> 15/11/05 12:34:44 INFO Utils: Successfully started service 'sparkDriver' on
> port 35020.
> 15/11/05 12:34:44 INFO SparkEnv: Registering MapOutputTracker
> 15/11/05 12:34:44 INFO SparkEnv: Registering BlockManagerMaster
> 15/11/05 12:34:44 INFO DiskBlockManager: Created local directory at
> /tmp/spark-3b14e9fc-73b1-45e7-96f3-f0ee48f3a238/blockmgr-ffd0619a-5fbc-4ccc-bcef-ce8560e03cd7
> 15/11/05 12:34:44 INFO MemoryStore: MemoryStore started with capacity 265.4 MB
> 15/11/05 12:34:45 INFO HttpFileServer: HTTP File server directory is
> /tmp/spark-7648f575-a8e3-4b25-945f-b4ded8631592/httpd-ec5526bb-4de6-42d6-83aa-a1ba0a2ca5dd
> 15/11/05 12:34:45 INFO HttpServer: Starting HTTP Server
> 15/11/05 12:34:45 INFO Server: jetty-8.y.z-SNAPSHOT
> 15/11/05 12:34:45 INFO AbstractConnector: Started
> [email protected]:58853
> 15/11/05 12:34:45 INFO Utils: Successfully started service 'HTTP file server'
> on port 58853.
> 15/11/05 12:34:45 INFO SparkEnv: Registering OutputCommitCoordinator
> 1
> ...
> ...
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
