Marcelo Vanzin created SPARK-26414:
--------------------------------------
Summary: Race between SparkContext and YARN AM can cause NPE in UI
setup code
Key: SPARK-26414
URL: https://issues.apache.org/jira/browse/SPARK-26414
Project: Spark
Issue Type: Bug
Components: YARN
Affects Versions: 2.4.0
Reporter: Marcelo Vanzin
There's a super narrow race between the SparkContext and the AM startup code:
- SC starts the AM and waits for it to go into running state
- AM goes into running state, unblocking SC
- AM sends AmIpFilter config to SC, adds the filter to the list and then the
filter configs
- unblocked SC is in the middle of setting up the UI and sees only the filter,
but not the configs
Then you get this:
{noformat}
ERROR org.apache.spark.SparkContext - Error initializing SparkContext.
java.lang.NullPointerException
at
org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter.init(AmIpFilter.java:81)
at
org.spark_project.jetty.servlet.FilterHolder.initialize(FilterHolder.java:139)
at
org.spark_project.jetty.servlet.ServletHandler.initialize(ServletHandler.java:881)
at
org.spark_project.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:349)
at
org.spark_project.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:778)
at
org.spark_project.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:262)
at
org.spark_project.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.apache.spark.ui.ServerInfo.addHandler(JettyUtils.scala:520)
at
org.apache.spark.ui.WebUI$$anonfun$attachHandler$1.apply(WebUI.scala:96)
at
org.apache.spark.ui.WebUI$$anonfun$attachHandler$1.apply(WebUI.scala:96)
at scala.Option.foreach(Option.scala:257)
at org.apache.spark.ui.WebUI.attachHandler(WebUI.scala:96)
at
org.apache.spark.SparkContext$$anonfun$22$$anonfun$apply$8.apply(SparkContext.scala:522)
at
org.apache.spark.SparkContext$$anonfun$22$$anonfun$apply$8.apply(SparkContext.scala:522)
at scala.Option.foreach(Option.scala:257)
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
