Hi, I want to do log aggregation in a Spark standalone mode cluster, using Apache Flume. But something weird happended. Here are my operations:
(1) Start a flume agent, listening on port 33333. ( flume-conf.properties <http://apache-spark-user-list.1001560.n3.nabble.com/file/n19140/flume-conf.properties> ) (2) In log4j.properties, I added a Flume Log4jAppender to make Spark able to send its log to a flume agent. ( log4j.properties <http://apache-spark-user-list.1001560.n3.nabble.com/file/n19140/log4j.properties> ) (3) sbin/start-all.sh. In this stage, all logs were aggregated into the flume agent and no exception happened. (flume-ng-log4jappender version: 1.5.0.1) (4) bin/spark-submit --class org.apache.examples.SparkPi ../lib/spark-examples-1.1.0-hadoop2.4.0.jar 1 At the beginning, logs were aggregated normally. However, when the driver need to stop SparkContext sc, it threw some exceptions and the driver exit abnormally: sparkPi.log <http://apache-spark-user-list.1001560.n3.nabble.com/file/n19140/sparkPi.log> /==== ... 14/11/18 23:06:24 INFO SparkDeploySchedulerBackend:org.apache.spark.Logging$class.logInfo(Logging.scala:59) - Asking each executor to shut down 14/11/18 23:06:25 INFO MapOutputTrackerMasterActor:org.apache.spark.Logging$class.logInfo(Logging.scala:59) - MapOutputTrackerActor stopped! 14/11/18 23:06:25 INFO ConnectionManager:org.apache.spark.Logging$class.logInfo(Logging.scala:59) - Selector thread was interrupted! log4j:ERROR Flume append() failed. 14/11/18 23:06:25 ERROR ConnectionManager:org.apache.spark.Logging$class.logError(Logging.scala:96) - Error in select loop org.apache.flume.FlumeException: Flume append() failed. Exception follows. ... ====/ I also tried setting log4j.appender.flume.unsafeMode property to true. This time, the drive finished the job but didn't exit: sparkPi_snd.log <http://apache-spark-user-list.1001560.n3.nabble.com/file/n19140/sparkPi_snd.log> /==== .... 14/11/18 23:36:49 WARN NettyAvroRpcClient:org.apache.spark.Logging$class.logInfo(Logging.scala:59) - Using default maxIOWorkers log4j:ERROR Cannot Append to Appender! Appender either closed or not setup correctly! 14/11/18 23:36:49 INFO ConnectionManager:org.apache.spark.Logging$class.logInfo(Logging.scala:59) - Removing ReceivingConnection to ConnectionManagerId(pc215,59272) 14/11/18 23:36:49 ERROR ConnectionManager:org.apache.spark.Logging$class.logError(Logging.scala:75) - Corresponding SendingConnection to ConnectionManagerId(pc215,59272) not found 14/11/18 23:36:49 WARN ConnectionManager:org.apache.spark.Logging$class.logWarning(Logging.scala:71) - All connections not cleaned up 14/11/18 23:36:49 INFO ConnectionManager:org.apache.spark.Logging$class.logInfo(Logging.scala:59) - ConnectionManager stopped 14/11/18 23:36:49 INFO MemoryStore:org.apache.spark.Logging$class.logInfo(Logging.scala:59) - MemoryStore cleared 14/11/18 23:36:49 INFO BlockManager:org.apache.spark.Logging$class.logInfo(Logging.scala:59) - BlockManager stopped 14/11/18 23:36:49 INFO BlockManagerMaster:org.apache.spark.Logging$class.logInfo(Logging.scala:59) - BlockManagerMaster stopped 14/11/18 23:36:49 INFO RemoteActorRefProvider$RemotingTerminator:akka.event.slf4j.Slf4jLogger$$anonfun$receive$1$$anonfun$applyOrElse$3.apply$mcV$sp(Slf4jLogger.scala:74) - Shutting down remote daemon. 14/11/18 23:36:49 INFO RemoteActorRefProvider$RemotingTerminator:akka.event.slf4j.Slf4jLogger$$anonfun$receive$1$$anonfun$applyOrElse$3.apply$mcV$sp(Slf4jLogger.scala:74) - Remote daemon shut down; proceeding with flushing remote transports. 14/11/18 23:36:49 INFO Remoting:akka.event.slf4j.Slf4jLogger$$anonfun$receive$1$$anonfun$applyOrElse$3.apply$mcV$sp(Slf4jLogger.scala:74) - Remoting shut down 14/11/18 23:36:49 INFO SparkContext:org.apache.spark.Logging$class.logInfo(Logging.scala:59) - Successfully stopped SparkContext ====/ I have searched Google with keywords like "Spark log aggregation", "Spark Flume Log4jAppender", "ERROR Flume append() failed", but I can't figure out how to solve this problem. Does anybody has an idea? -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Logging-problem-in-Spark-when-using-Flume-Log4jAppender-tp19140.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org