Hi,
How can the spark log be saved into file instead of showing them on console?
Below is my conf/log4j.properties
conf/log4j.properties
###
# Root logger option
log4j.rootLogger=INFO, file
# Direct log messages to a log file
log4j.appender.file=org.apache.log4j.RollingFileAppender
#Redirect to Tomcat logs folder
log4j.appender.file.File=/hadoop_logs/spark/spark_logging.log
log4j.appender.file.MaxFileSize=10MB
log4j.appender.file.MaxBackupIndex=10
log4j.appender.file.layout=org.apache.log4j.PatternLayout
log4j.appender.file.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss} %-5p
%c{1}:%L - %m%n
###
I tried to stop and start spark again, it still shows INFO WARN log on console.
Any ideas?
Regards
Arthur
scala> hiveContext.hql("show tables")
warning: there were 1 deprecation warning(s); re-run with -deprecation for
details
2014-10-03 19:35:01,554 INFO [main] parse.ParseDriver
(ParseDriver.java:parse(179)) - Parsing command: show tables
2014-10-03 19:35:01,715 INFO [main] parse.ParseDriver
(ParseDriver.java:parse(197)) - Parse Completed
2014-10-03 19:35:01,845 INFO [main] Configuration.deprecation
(Configuration.java:warnOnceIfDeprecated(1009)) - mapred.input.dir.recursive is
deprecated. Instead, use mapreduce.input.fileinputformat.input.dir.recursive
2014-10-03 19:35:01,847 INFO [main] ql.Driver
(PerfLogger.java:PerfLogBegin(97)) - <PERFLOG method=Driver.run>
2014-10-03 19:35:01,847 INFO [main] ql.Driver
(PerfLogger.java:PerfLogBegin(97)) - <PERFLOG method=TimeToSubmit>
2014-10-03 19:35:01,847 INFO [main] ql.Driver
(PerfLogger.java:PerfLogBegin(97)) - <PERFLOG method=compile>
2014-10-03 19:35:01,863 INFO [main] ql.Driver
(PerfLogger.java:PerfLogBegin(97)) - <PERFLOG method=parse>
2014-10-03 19:35:01,863 INFO [main] parse.ParseDriver
(ParseDriver.java:parse(179)) - Parsing command: show tables
2014-10-03 19:35:01,863 INFO [main] parse.ParseDriver
(ParseDriver.java:parse(197)) - Parse Completed
2014-10-03 19:35:01,863 INFO [main] ql.Driver
(PerfLogger.java:PerfLogEnd(124)) - </PERFLOG method=parse start=1412336101863
end=1412336101863 duration=0>
2014-10-03 19:35:01,863 INFO [main] ql.Driver
(PerfLogger.java:PerfLogBegin(97)) - <PERFLOG method=semanticAnalyze>
2014-10-03 19:35:01,941 INFO [main] ql.Driver (Driver.java:compile(450)) -
Semantic Analysis Completed
2014-10-03 19:35:01,941 INFO [main] ql.Driver
(PerfLogger.java:PerfLogEnd(124)) - </PERFLOG method=semanticAnalyze
start=1412336101863 end=1412336101941 duration=78>
2014-10-03 19:35:01,979 INFO [main] exec.ListSinkOperator
(Operator.java:initialize(338)) - Initializing Self 0 OP
2014-10-03 19:35:01,980 INFO [main] exec.ListSinkOperator
(Operator.java:initializeChildren(403)) - Operator 0 OP initialized
2014-10-03 19:35:01,980 INFO [main] exec.ListSinkOperator
(Operator.java:initialize(378)) - Initialization Done 0 OP
2014-10-03 19:35:01,985 INFO [main] ql.Driver (Driver.java:getSchema(264)) -
Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:tab_name,
type:string, comment:from deserializer)], properties:null)
2014-10-03 19:35:01,985 INFO [main] ql.Driver
(PerfLogger.java:PerfLogEnd(124)) - </PERFLOG method=compile
start=1412336101847 end=1412336101985 duration=138>
2014-10-03 19:35:01,985 INFO [main] ql.Driver
(PerfLogger.java:PerfLogBegin(97)) - <PERFLOG method=Driver.execute>
2014-10-03 19:35:01,985 INFO [main] Configuration.deprecation
(Configuration.java:warnOnceIfDeprecated(1009)) - mapred.job.name is
deprecated. Instead, use mapreduce.job.name
2014-10-03 19:35:01,986 INFO [main] ql.Driver (Driver.java:execute(1117)) -
Starting command: show tables
2014-10-03 19:35:01,994 INFO [main] ql.Driver
(PerfLogger.java:PerfLogEnd(124)) - </PERFLOG method=TimeToSubmit
start=1412336101847 end=1412336101994 duration=147>
2014-10-03 19:35:01,994 INFO [main] ql.Driver
(PerfLogger.java:PerfLogBegin(97)) - <PERFLOG method=runTasks>
2014-10-03 19:35:01,994 INFO [main] ql.Driver
(PerfLogger.java:PerfLogBegin(97)) - <PERFLOG method=task.DDL.Stage-0>
2014-10-03 19:35:02,019 INFO [main] metastore.HiveMetaStore
(HiveMetaStore.java:newRawStore(411)) - 0: Opening raw store with implemenation
class:org.apache.hadoop.hive.metastore.ObjectStore
2014-10-03 19:35:02,034 INFO [main] metastore.ObjectStore
(ObjectStore.java:initialize(232)) - ObjectStore, initialize called
2014-10-03 19:35:02,084 WARN [main] DataNucleus.General
(Log4JLogger.java:warn(96)) - Plugin (Bundle) "org.datanucleus.api.jdo" is
already registered. Ensure you dont have multiple JAR versions of the same
plugin in the classpath. The URL
"file://hadoop/spark/lib_managed/jars/datanucleus-api-jdo-3.2.1.jar" is already
registered, and you are trying to register an identical plugin located at URL
"file://hadoop/spark-1.1.0_patched/lib_managed/jars/datanucleus-api-jdo-3.2.1.jar."
2014-10-03 19:35:02,108 WARN [main] DataNucleus.General
(Log4JLogger.java:warn(96)) - Plugin (Bundle) "org.datanucleus" is already
registered. Ensure you dont have multiple JAR versions of the same plugin in
the classpath. The URL
"file://hadoop/spark/lib_managed/jars/datanucleus-core-3.2.2.jar" is already
registered, and you are trying to register an identical plugin located at URL
"file://hadoop/spark-1.1.0_patched/lib_managed/jars/datanucleus-core-3.2.2.jar."
2014-10-03 19:35:02,137 WARN [main] DataNucleus.General
(Log4JLogger.java:warn(96)) - Plugin (Bundle) "org.datanucleus.store.rdbms" is
already registered. Ensure you dont have multiple JAR versions of the same
plugin in the classpath. The URL
"file://hadoop/spark-1.1.0_patched/lib_managed/jars/datanucleus-rdbms-3.2.1.jar"
is already registered, and you are trying to register an identical plugin
located at URL
"file://hadoop/spark/lib_managed/jars/datanucleus-rdbms-3.2.1.jar."
2014-10-03 19:35:02,174 INFO [main] DataNucleus.Persistence
(Log4JLogger.java:info(77)) - Property datanucleus.cache.level2 unknown - will
be ignored
2014-10-03 19:35:02,300 WARN [main] bonecp.BoneCPConfig
(BoneCPConfig.java:sanitize(1537)) - Max Connections < 1. Setting to 20
2014-10-03 19:35:02,769 INFO [main] metastore.ObjectStore
(ObjectStore.java:getPMF(280)) - Setting MetaStore object pin classes with
hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
2014-10-03 19:35:02,791 INFO [main] metastore.ObjectStore
(ObjectStore.java:setConf(215)) - Initialized ObjectStore
2014-10-03 19:35:03,268 WARN [main] bonecp.BoneCPConfig
(BoneCPConfig.java:sanitize(1537)) - Max Connections < 1. Setting to 20
2014-10-03 19:35:03,400 INFO [main] metastore.HiveMetaStore
(HiveMetaStore.java:logInfo(454)) - 0: get_database: default
2014-10-03 19:35:03,400 INFO [main] HiveMetaStore.audit
(HiveMetaStore.java:logAuditEvent(239)) - ugi=edhuser ip=unknown-ip-addr
cmd=get_database: default
2014-10-03 19:35:03,405 INFO [main] metastore.HiveMetaStore
(HiveMetaStore.java:logInfo(454)) - 0: get_tables: db=default pat=.*
2014-10-03 19:35:03,405 INFO [main] HiveMetaStore.audit
(HiveMetaStore.java:logAuditEvent(239)) - ugi=edhuser ip=unknown-ip-addr
cmd=get_tables: db=default pat=.*
2014-10-03 19:35:03,408 INFO [main] DataNucleus.Datastore
(Log4JLogger.java:info(77)) - The class
"org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as
"embedded-only" so does not have its own datastore table.
2014-10-03 19:35:03,409 INFO [main] DataNucleus.Datastore
(Log4JLogger.java:info(77)) - The class
"org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so
does not have its own datastore table.
2014-10-03 19:35:03,493 INFO [main] ql.Driver
(PerfLogger.java:PerfLogEnd(124)) - </PERFLOG method=task.DDL.Stage-0
start=1412336101994 end=1412336103493 duration=1499>
2014-10-03 19:35:03,493 INFO [main] ql.Driver
(PerfLogger.java:PerfLogEnd(124)) - </PERFLOG method=runTasks
start=1412336101994 end=1412336103493 duration=1499>
2014-10-03 19:35:03,493 INFO [main] ql.Driver
(PerfLogger.java:PerfLogEnd(124)) - </PERFLOG method=Driver.execute
start=1412336101985 end=1412336103493 duration=1508>
2014-10-03 19:35:03,496 INFO [main] ql.Driver
(SessionState.java:printInfo(410)) - OK
2014-10-03 19:35:03,496 INFO [main] ql.Driver
(PerfLogger.java:PerfLogBegin(97)) - <PERFLOG method=releaseLocks>
2014-10-03 19:35:03,496 INFO [main] ql.Driver
(PerfLogger.java:PerfLogEnd(124)) - </PERFLOG method=releaseLocks
start=1412336103496 end=1412336103496 duration=0>
2014-10-03 19:35:03,496 INFO [main] ql.Driver
(PerfLogger.java:PerfLogEnd(124)) - </PERFLOG method=Driver.run
start=1412336101847 end=1412336103496 duration=1649>
2014-10-03 19:35:03,497 INFO [main] Configuration.deprecation
(Configuration.java:warnOnceIfDeprecated(1009)) - mapred.input.dir.recursive is
deprecated. Instead, use mapreduce.input.fileinputformat.input.dir.recursive
2014-10-03 19:35:03,497 INFO [main] Configuration.deprecation
(Configuration.java:warnOnceIfDeprecated(1009)) - mapred.input.dir is
deprecated. Instead, use mapreduce.input.fileinputformat.inputdir
2014-10-03 19:35:03,508 INFO [main] mapred.FileInputFormat
(FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2014-10-03 19:35:03,517 INFO [main] ql.Driver
(PerfLogger.java:PerfLogBegin(97)) - <PERFLOG method=releaseLocks>
2014-10-03 19:35:03,518 INFO [main] ql.Driver
(PerfLogger.java:PerfLogEnd(124)) - </PERFLOG method=releaseLocks
start=1412336103517 end=1412336103518 duration=1>
