[ https://issues.apache.org/jira/browse/SPARK-21321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thomas Graves resolved SPARK-21321. ----------------------------------- Resolution: Fixed Assignee: Jong Yoon Lee Fix Version/s: 2.3.0 2.2.1 > Spark very verbose on shutdown confusing users > ---------------------------------------------- > > Key: SPARK-21321 > URL: https://issues.apache.org/jira/browse/SPARK-21321 > Project: Spark > Issue Type: Improvement > Components: Spark Core > Affects Versions: 2.1.1 > Reporter: Jong Yoon Lee > Assignee: Jong Yoon Lee > Priority: Minor > Fix For: 2.2.1, 2.3.0 > > Original Estimate: 24h > Remaining Estimate: 24h > > On shutdown, spark can be very verbose and can spit out errors that causes > the user to be confused. > If possible, we should not print those out and just ignore them by changing > the changing the log level from WARNING to DEBUG. > Also shutdown of spark can take a long time because of the backlog of events > in the event queue ( warnings like: Message $message dropped. > ${e.getMessage}). > This happens more with dynamic allocation on. > I am suggesting to > 1. change the log level when the shutdown is happening and the RPC > connections are closed(RpcEnvStoppedException). > 2. Clear the event queue when RPC module is stopped and spark is shutting > down. -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org