[jira] [Commented] (SPARK-29765) Monitoring UI throws IndexOutOfBoundsException when accessing metrics of attempt in stage
[
https://issues.apache.org/jira/browse/SPARK-29765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17156429#comment-17156429
]
Jacob Niebloom commented on SPARK-29765:
What was the resolution to this? Thanks! [~shahid] [~Tradunsky]
> Monitoring UI throws IndexOutOfBoundsException when accessing metrics of
> attempt in stage
> -
>
> Key: SPARK-29765
> URL: https://issues.apache.org/jira/browse/SPARK-29765
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 2.4.4
> Environment: Amazon EMR 5.27
>Reporter: Viacheslav Tradunsky
>Priority: Major
>
> When clicking on one of the largest tasks by input, I get to
> [http://:20888/proxy/application_1572992299050_0001/stages/stage/?id=74=0|http://10.207.110.207:20888/proxy/application_1572992299050_0001/stages/stage/?id=74=0]
> with 500 error
> {code:java}
> java.lang.IndexOutOfBoundsException: 95745 at
> scala.collection.immutable.Vector.checkRangeConvert(Vector.scala:132) at
> scala.collection.immutable.Vector.apply(Vector.scala:122) at
> org.apache.spark.status.AppStatusStore$$anonfun$scanTasks$1$1.apply$mcDJ$sp(AppStatusStore.scala:255)
> at
> org.apache.spark.status.AppStatusStore$$anonfun$scanTasks$1$1.apply(AppStatusStore.scala:254)
> at
> org.apache.spark.status.AppStatusStore$$anonfun$scanTasks$1$1.apply(AppStatusStore.scala:254)
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
> at
> scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
> at scala.collection.mutable.ArrayOps$ofLong.foreach(ArrayOps.scala:246) at
> scala.collection.TraversableLike$class.map(TraversableLike.scala:234) at
> scala.collection.mutable.ArrayOps$ofLong.map(ArrayOps.scala:246) at
> org.apache.spark.status.AppStatusStore.scanTasks$1(AppStatusStore.scala:254)
> at
> org.apache.spark.status.AppStatusStore.taskSummary(AppStatusStore.scala:287)
> at org.apache.spark.ui.jobs.StagePage.render(StagePage.scala:321) at
> org.apache.spark.ui.WebUI$$anonfun$2.apply(WebUI.scala:84) at
> org.apache.spark.ui.WebUI$$anonfun$2.apply(WebUI.scala:84) at
> org.apache.spark.ui.JettyUtils$$anon$3.doGet(JettyUtils.scala:90) at
> javax.servlet.http.HttpServlet.service(HttpServlet.java:687) at
> javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at
> org.spark_project.jetty.servlet.ServletHolder.handle(ServletHolder.java:848)
> at
> org.spark_project.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1772)
> at
> org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter.doFilter(AmIpFilter.java:166)
> at
> org.spark_project.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
> at
> org.spark_project.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
> at
> org.spark_project.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
> at
> org.spark_project.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
> at
> org.spark_project.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
> at
> org.spark_project.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> at
> org.spark_project.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:493)
> at
> org.spark_project.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
> at
> org.spark_project.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
> at org.spark_project.jetty.server.Server.handle(Server.java:539) at
> org.spark_project.jetty.server.HttpChannel.handle(HttpChannel.java:333) at
> org.spark_project.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
> at
> org.spark_project.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
> at org.spark_project.jetty.io.FillInterest.fillable(FillInterest.java:108)
> at
> org.spark_project.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
> at
> org.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
> at
> org.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
> at
> org.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
> at
> org.spark_project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
> at
> org.spark_project.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
> at java.lang.Thread.run(Thread.java:748){code}
--
This message was sent by Atlassian Jira
[jira] [Commented] (SPARK-29765) Monitoring UI throws IndexOutOfBoundsException when accessing metrics of attempt in stage
[
https://issues.apache.org/jira/browse/SPARK-29765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16968760#comment-16968760
]
shahid commented on SPARK-29765:
Event drop can also happen that if we don't provide sufficient memory. That
case increasing the configs won't help I think.
If the event drop happens the entire UI behaves weirdly.
> Monitoring UI throws IndexOutOfBoundsException when accessing metrics of
> attempt in stage
> -
>
> Key: SPARK-29765
> URL: https://issues.apache.org/jira/browse/SPARK-29765
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 2.4.4
> Environment: Amazon EMR 5.27
>Reporter: Viacheslav Tradunsky
>Priority: Major
>
> When clicking on one of the largest tasks by input, I get to
> [http://:20888/proxy/application_1572992299050_0001/stages/stage/?id=74=0|http://10.207.110.207:20888/proxy/application_1572992299050_0001/stages/stage/?id=74=0]
> with 500 error
> {code:java}
> java.lang.IndexOutOfBoundsException: 95745 at
> scala.collection.immutable.Vector.checkRangeConvert(Vector.scala:132) at
> scala.collection.immutable.Vector.apply(Vector.scala:122) at
> org.apache.spark.status.AppStatusStore$$anonfun$scanTasks$1$1.apply$mcDJ$sp(AppStatusStore.scala:255)
> at
> org.apache.spark.status.AppStatusStore$$anonfun$scanTasks$1$1.apply(AppStatusStore.scala:254)
> at
> org.apache.spark.status.AppStatusStore$$anonfun$scanTasks$1$1.apply(AppStatusStore.scala:254)
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
> at
> scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
> at scala.collection.mutable.ArrayOps$ofLong.foreach(ArrayOps.scala:246) at
> scala.collection.TraversableLike$class.map(TraversableLike.scala:234) at
> scala.collection.mutable.ArrayOps$ofLong.map(ArrayOps.scala:246) at
> org.apache.spark.status.AppStatusStore.scanTasks$1(AppStatusStore.scala:254)
> at
> org.apache.spark.status.AppStatusStore.taskSummary(AppStatusStore.scala:287)
> at org.apache.spark.ui.jobs.StagePage.render(StagePage.scala:321) at
> org.apache.spark.ui.WebUI$$anonfun$2.apply(WebUI.scala:84) at
> org.apache.spark.ui.WebUI$$anonfun$2.apply(WebUI.scala:84) at
> org.apache.spark.ui.JettyUtils$$anon$3.doGet(JettyUtils.scala:90) at
> javax.servlet.http.HttpServlet.service(HttpServlet.java:687) at
> javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at
> org.spark_project.jetty.servlet.ServletHolder.handle(ServletHolder.java:848)
> at
> org.spark_project.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1772)
> at
> org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter.doFilter(AmIpFilter.java:166)
> at
> org.spark_project.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
> at
> org.spark_project.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
> at
> org.spark_project.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
> at
> org.spark_project.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
> at
> org.spark_project.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
> at
> org.spark_project.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> at
> org.spark_project.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:493)
> at
> org.spark_project.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
> at
> org.spark_project.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
> at org.spark_project.jetty.server.Server.handle(Server.java:539) at
> org.spark_project.jetty.server.HttpChannel.handle(HttpChannel.java:333) at
> org.spark_project.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
> at
> org.spark_project.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
> at org.spark_project.jetty.io.FillInterest.fillable(FillInterest.java:108)
> at
> org.spark_project.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
> at
> org.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
> at
> org.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
> at
> org.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
> at
> org.spark_project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
> at
> org.spark_project.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
>
[jira] [Commented] (SPARK-29765) Monitoring UI throws IndexOutOfBoundsException when accessing metrics of attempt in stage
[
https://issues.apache.org/jira/browse/SPARK-29765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16968753#comment-16968753
]
Viacheslav Tradunsky commented on SPARK-29765:
--
What's in particularly interesting, that the error doesn't reproduce when this
stage is in completed list.
> Monitoring UI throws IndexOutOfBoundsException when accessing metrics of
> attempt in stage
> -
>
> Key: SPARK-29765
> URL: https://issues.apache.org/jira/browse/SPARK-29765
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 2.4.4
> Environment: Amazon EMR 5.27
>Reporter: Viacheslav Tradunsky
>Priority: Major
>
> When clicking on one of the largest tasks by input, I get to
> [http://:20888/proxy/application_1572992299050_0001/stages/stage/?id=74=0|http://10.207.110.207:20888/proxy/application_1572992299050_0001/stages/stage/?id=74=0]
> with 500 error
> {code:java}
> java.lang.IndexOutOfBoundsException: 95745 at
> scala.collection.immutable.Vector.checkRangeConvert(Vector.scala:132) at
> scala.collection.immutable.Vector.apply(Vector.scala:122) at
> org.apache.spark.status.AppStatusStore$$anonfun$scanTasks$1$1.apply$mcDJ$sp(AppStatusStore.scala:255)
> at
> org.apache.spark.status.AppStatusStore$$anonfun$scanTasks$1$1.apply(AppStatusStore.scala:254)
> at
> org.apache.spark.status.AppStatusStore$$anonfun$scanTasks$1$1.apply(AppStatusStore.scala:254)
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
> at
> scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
> at scala.collection.mutable.ArrayOps$ofLong.foreach(ArrayOps.scala:246) at
> scala.collection.TraversableLike$class.map(TraversableLike.scala:234) at
> scala.collection.mutable.ArrayOps$ofLong.map(ArrayOps.scala:246) at
> org.apache.spark.status.AppStatusStore.scanTasks$1(AppStatusStore.scala:254)
> at
> org.apache.spark.status.AppStatusStore.taskSummary(AppStatusStore.scala:287)
> at org.apache.spark.ui.jobs.StagePage.render(StagePage.scala:321) at
> org.apache.spark.ui.WebUI$$anonfun$2.apply(WebUI.scala:84) at
> org.apache.spark.ui.WebUI$$anonfun$2.apply(WebUI.scala:84) at
> org.apache.spark.ui.JettyUtils$$anon$3.doGet(JettyUtils.scala:90) at
> javax.servlet.http.HttpServlet.service(HttpServlet.java:687) at
> javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at
> org.spark_project.jetty.servlet.ServletHolder.handle(ServletHolder.java:848)
> at
> org.spark_project.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1772)
> at
> org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter.doFilter(AmIpFilter.java:166)
> at
> org.spark_project.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
> at
> org.spark_project.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
> at
> org.spark_project.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
> at
> org.spark_project.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
> at
> org.spark_project.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
> at
> org.spark_project.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> at
> org.spark_project.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:493)
> at
> org.spark_project.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
> at
> org.spark_project.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
> at org.spark_project.jetty.server.Server.handle(Server.java:539) at
> org.spark_project.jetty.server.HttpChannel.handle(HttpChannel.java:333) at
> org.spark_project.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
> at
> org.spark_project.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
> at org.spark_project.jetty.io.FillInterest.fillable(FillInterest.java:108)
> at
> org.spark_project.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
> at
> org.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
> at
> org.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
> at
> org.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
> at
> org.spark_project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
> at
> org.spark_project.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
> at java.lang.Thread.run(Thread.java:748){code}
[jira] [Commented] (SPARK-29765) Monitoring UI throws IndexOutOfBoundsException when accessing metrics of attempt in stage
[
https://issues.apache.org/jira/browse/SPARK-29765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16968751#comment-16968751
]
Viacheslav Tradunsky commented on SPARK-29765:
--
Increased the capacity to 2 as pointed out in docs:
[https://people.apache.org/~pwendell/spark-nightly/spark-master-docs/latest/configuration.html#scheduling]
But the error still happens.
> Monitoring UI throws IndexOutOfBoundsException when accessing metrics of
> attempt in stage
> -
>
> Key: SPARK-29765
> URL: https://issues.apache.org/jira/browse/SPARK-29765
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 2.4.4
> Environment: Amazon EMR 5.27
>Reporter: Viacheslav Tradunsky
>Priority: Major
>
> When clicking on one of the largest tasks by input, I get to
> [http://:20888/proxy/application_1572992299050_0001/stages/stage/?id=74=0|http://10.207.110.207:20888/proxy/application_1572992299050_0001/stages/stage/?id=74=0]
> with 500 error
> {code:java}
> java.lang.IndexOutOfBoundsException: 95745 at
> scala.collection.immutable.Vector.checkRangeConvert(Vector.scala:132) at
> scala.collection.immutable.Vector.apply(Vector.scala:122) at
> org.apache.spark.status.AppStatusStore$$anonfun$scanTasks$1$1.apply$mcDJ$sp(AppStatusStore.scala:255)
> at
> org.apache.spark.status.AppStatusStore$$anonfun$scanTasks$1$1.apply(AppStatusStore.scala:254)
> at
> org.apache.spark.status.AppStatusStore$$anonfun$scanTasks$1$1.apply(AppStatusStore.scala:254)
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
> at
> scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
> at scala.collection.mutable.ArrayOps$ofLong.foreach(ArrayOps.scala:246) at
> scala.collection.TraversableLike$class.map(TraversableLike.scala:234) at
> scala.collection.mutable.ArrayOps$ofLong.map(ArrayOps.scala:246) at
> org.apache.spark.status.AppStatusStore.scanTasks$1(AppStatusStore.scala:254)
> at
> org.apache.spark.status.AppStatusStore.taskSummary(AppStatusStore.scala:287)
> at org.apache.spark.ui.jobs.StagePage.render(StagePage.scala:321) at
> org.apache.spark.ui.WebUI$$anonfun$2.apply(WebUI.scala:84) at
> org.apache.spark.ui.WebUI$$anonfun$2.apply(WebUI.scala:84) at
> org.apache.spark.ui.JettyUtils$$anon$3.doGet(JettyUtils.scala:90) at
> javax.servlet.http.HttpServlet.service(HttpServlet.java:687) at
> javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at
> org.spark_project.jetty.servlet.ServletHolder.handle(ServletHolder.java:848)
> at
> org.spark_project.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1772)
> at
> org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter.doFilter(AmIpFilter.java:166)
> at
> org.spark_project.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
> at
> org.spark_project.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
> at
> org.spark_project.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
> at
> org.spark_project.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
> at
> org.spark_project.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
> at
> org.spark_project.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> at
> org.spark_project.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:493)
> at
> org.spark_project.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
> at
> org.spark_project.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
> at org.spark_project.jetty.server.Server.handle(Server.java:539) at
> org.spark_project.jetty.server.HttpChannel.handle(HttpChannel.java:333) at
> org.spark_project.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
> at
> org.spark_project.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
> at org.spark_project.jetty.io.FillInterest.fillable(FillInterest.java:108)
> at
> org.spark_project.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
> at
> org.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
> at
> org.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
> at
> org.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
> at
> org.spark_project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
> at
>
[jira] [Commented] (SPARK-29765) Monitoring UI throws IndexOutOfBoundsException when accessing metrics of attempt in stage
[
https://issues.apache.org/jira/browse/SPARK-29765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16968739#comment-16968739
]
Viacheslav Tradunsky commented on SPARK-29765:
--
I think it is capacity after code review ;)
spark.scheduler.listenerbus.eventqueue.capacity
Thanks a lot!
> Monitoring UI throws IndexOutOfBoundsException when accessing metrics of
> attempt in stage
> -
>
> Key: SPARK-29765
> URL: https://issues.apache.org/jira/browse/SPARK-29765
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 2.4.4
> Environment: Amazon EMR 5.27
>Reporter: Viacheslav Tradunsky
>Priority: Major
>
> When clicking on one of the largest tasks by input, I get to
> [http://:20888/proxy/application_1572992299050_0001/stages/stage/?id=74=0|http://10.207.110.207:20888/proxy/application_1572992299050_0001/stages/stage/?id=74=0]
> with 500 error
> {code:java}
> java.lang.IndexOutOfBoundsException: 95745 at
> scala.collection.immutable.Vector.checkRangeConvert(Vector.scala:132) at
> scala.collection.immutable.Vector.apply(Vector.scala:122) at
> org.apache.spark.status.AppStatusStore$$anonfun$scanTasks$1$1.apply$mcDJ$sp(AppStatusStore.scala:255)
> at
> org.apache.spark.status.AppStatusStore$$anonfun$scanTasks$1$1.apply(AppStatusStore.scala:254)
> at
> org.apache.spark.status.AppStatusStore$$anonfun$scanTasks$1$1.apply(AppStatusStore.scala:254)
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
> at
> scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
> at scala.collection.mutable.ArrayOps$ofLong.foreach(ArrayOps.scala:246) at
> scala.collection.TraversableLike$class.map(TraversableLike.scala:234) at
> scala.collection.mutable.ArrayOps$ofLong.map(ArrayOps.scala:246) at
> org.apache.spark.status.AppStatusStore.scanTasks$1(AppStatusStore.scala:254)
> at
> org.apache.spark.status.AppStatusStore.taskSummary(AppStatusStore.scala:287)
> at org.apache.spark.ui.jobs.StagePage.render(StagePage.scala:321) at
> org.apache.spark.ui.WebUI$$anonfun$2.apply(WebUI.scala:84) at
> org.apache.spark.ui.WebUI$$anonfun$2.apply(WebUI.scala:84) at
> org.apache.spark.ui.JettyUtils$$anon$3.doGet(JettyUtils.scala:90) at
> javax.servlet.http.HttpServlet.service(HttpServlet.java:687) at
> javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at
> org.spark_project.jetty.servlet.ServletHolder.handle(ServletHolder.java:848)
> at
> org.spark_project.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1772)
> at
> org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter.doFilter(AmIpFilter.java:166)
> at
> org.spark_project.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
> at
> org.spark_project.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
> at
> org.spark_project.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
> at
> org.spark_project.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
> at
> org.spark_project.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
> at
> org.spark_project.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> at
> org.spark_project.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:493)
> at
> org.spark_project.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
> at
> org.spark_project.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
> at org.spark_project.jetty.server.Server.handle(Server.java:539) at
> org.spark_project.jetty.server.HttpChannel.handle(HttpChannel.java:333) at
> org.spark_project.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
> at
> org.spark_project.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
> at org.spark_project.jetty.io.FillInterest.fillable(FillInterest.java:108)
> at
> org.spark_project.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
> at
> org.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
> at
> org.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
> at
> org.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
> at
> org.spark_project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
> at
> org.spark_project.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
> at
[jira] [Commented] (SPARK-29765) Monitoring UI throws IndexOutOfBoundsException when accessing metrics of attempt in stage
[
https://issues.apache.org/jira/browse/SPARK-29765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16968737#comment-16968737
]
shahid commented on SPARK-29765:
Thanks. Yes, you can try increasing
`spark.scheduler.listenerbus.eventqueue.size`.
> Monitoring UI throws IndexOutOfBoundsException when accessing metrics of
> attempt in stage
> -
>
> Key: SPARK-29765
> URL: https://issues.apache.org/jira/browse/SPARK-29765
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 2.4.4
> Environment: Amazon EMR 5.27
>Reporter: Viacheslav Tradunsky
>Priority: Major
>
> When clicking on one of the largest tasks by input, I get to
> [http://:20888/proxy/application_1572992299050_0001/stages/stage/?id=74=0|http://10.207.110.207:20888/proxy/application_1572992299050_0001/stages/stage/?id=74=0]
> with 500 error
> {code:java}
> java.lang.IndexOutOfBoundsException: 95745 at
> scala.collection.immutable.Vector.checkRangeConvert(Vector.scala:132) at
> scala.collection.immutable.Vector.apply(Vector.scala:122) at
> org.apache.spark.status.AppStatusStore$$anonfun$scanTasks$1$1.apply$mcDJ$sp(AppStatusStore.scala:255)
> at
> org.apache.spark.status.AppStatusStore$$anonfun$scanTasks$1$1.apply(AppStatusStore.scala:254)
> at
> org.apache.spark.status.AppStatusStore$$anonfun$scanTasks$1$1.apply(AppStatusStore.scala:254)
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
> at
> scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
> at scala.collection.mutable.ArrayOps$ofLong.foreach(ArrayOps.scala:246) at
> scala.collection.TraversableLike$class.map(TraversableLike.scala:234) at
> scala.collection.mutable.ArrayOps$ofLong.map(ArrayOps.scala:246) at
> org.apache.spark.status.AppStatusStore.scanTasks$1(AppStatusStore.scala:254)
> at
> org.apache.spark.status.AppStatusStore.taskSummary(AppStatusStore.scala:287)
> at org.apache.spark.ui.jobs.StagePage.render(StagePage.scala:321) at
> org.apache.spark.ui.WebUI$$anonfun$2.apply(WebUI.scala:84) at
> org.apache.spark.ui.WebUI$$anonfun$2.apply(WebUI.scala:84) at
> org.apache.spark.ui.JettyUtils$$anon$3.doGet(JettyUtils.scala:90) at
> javax.servlet.http.HttpServlet.service(HttpServlet.java:687) at
> javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at
> org.spark_project.jetty.servlet.ServletHolder.handle(ServletHolder.java:848)
> at
> org.spark_project.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1772)
> at
> org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter.doFilter(AmIpFilter.java:166)
> at
> org.spark_project.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
> at
> org.spark_project.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
> at
> org.spark_project.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
> at
> org.spark_project.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
> at
> org.spark_project.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
> at
> org.spark_project.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> at
> org.spark_project.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:493)
> at
> org.spark_project.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
> at
> org.spark_project.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
> at org.spark_project.jetty.server.Server.handle(Server.java:539) at
> org.spark_project.jetty.server.HttpChannel.handle(HttpChannel.java:333) at
> org.spark_project.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
> at
> org.spark_project.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
> at org.spark_project.jetty.io.FillInterest.fillable(FillInterest.java:108)
> at
> org.spark_project.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
> at
> org.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
> at
> org.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
> at
> org.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
> at
> org.spark_project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
> at
> org.spark_project.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
> at java.lang.Thread.run(Thread.java:748){code}
--
This message was sent by Atlassian Jira
[jira] [Commented] (SPARK-29765) Monitoring UI throws IndexOutOfBoundsException when accessing metrics of attempt in stage
[
https://issues.apache.org/jira/browse/SPARK-29765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16968720#comment-16968720
]
Viacheslav Tradunsky commented on SPARK-29765:
--
Exactly as you said:
{code:java}
ERROR AsyncEventQueue: Dropping event from queue appStatus. This likely means
one of the listeners is too slow and cannot keep up with the rate at which
tasks are being started by the scheduler{code}
Will increase the capacity and go back with results. Thanks!
> Monitoring UI throws IndexOutOfBoundsException when accessing metrics of
> attempt in stage
> -
>
> Key: SPARK-29765
> URL: https://issues.apache.org/jira/browse/SPARK-29765
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 2.4.4
> Environment: Amazon EMR 5.27
>Reporter: Viacheslav Tradunsky
>Priority: Major
>
> When clicking on one of the largest tasks by input, I get to
> [http://:20888/proxy/application_1572992299050_0001/stages/stage/?id=74=0|http://10.207.110.207:20888/proxy/application_1572992299050_0001/stages/stage/?id=74=0]
> with 500 error
> {code:java}
> java.lang.IndexOutOfBoundsException: 95745 at
> scala.collection.immutable.Vector.checkRangeConvert(Vector.scala:132) at
> scala.collection.immutable.Vector.apply(Vector.scala:122) at
> org.apache.spark.status.AppStatusStore$$anonfun$scanTasks$1$1.apply$mcDJ$sp(AppStatusStore.scala:255)
> at
> org.apache.spark.status.AppStatusStore$$anonfun$scanTasks$1$1.apply(AppStatusStore.scala:254)
> at
> org.apache.spark.status.AppStatusStore$$anonfun$scanTasks$1$1.apply(AppStatusStore.scala:254)
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
> at
> scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
> at scala.collection.mutable.ArrayOps$ofLong.foreach(ArrayOps.scala:246) at
> scala.collection.TraversableLike$class.map(TraversableLike.scala:234) at
> scala.collection.mutable.ArrayOps$ofLong.map(ArrayOps.scala:246) at
> org.apache.spark.status.AppStatusStore.scanTasks$1(AppStatusStore.scala:254)
> at
> org.apache.spark.status.AppStatusStore.taskSummary(AppStatusStore.scala:287)
> at org.apache.spark.ui.jobs.StagePage.render(StagePage.scala:321) at
> org.apache.spark.ui.WebUI$$anonfun$2.apply(WebUI.scala:84) at
> org.apache.spark.ui.WebUI$$anonfun$2.apply(WebUI.scala:84) at
> org.apache.spark.ui.JettyUtils$$anon$3.doGet(JettyUtils.scala:90) at
> javax.servlet.http.HttpServlet.service(HttpServlet.java:687) at
> javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at
> org.spark_project.jetty.servlet.ServletHolder.handle(ServletHolder.java:848)
> at
> org.spark_project.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1772)
> at
> org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter.doFilter(AmIpFilter.java:166)
> at
> org.spark_project.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
> at
> org.spark_project.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
> at
> org.spark_project.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
> at
> org.spark_project.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
> at
> org.spark_project.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
> at
> org.spark_project.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> at
> org.spark_project.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:493)
> at
> org.spark_project.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
> at
> org.spark_project.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
> at org.spark_project.jetty.server.Server.handle(Server.java:539) at
> org.spark_project.jetty.server.HttpChannel.handle(HttpChannel.java:333) at
> org.spark_project.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
> at
> org.spark_project.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
> at org.spark_project.jetty.io.FillInterest.fillable(FillInterest.java:108)
> at
> org.spark_project.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
> at
> org.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
> at
> org.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
> at
> org.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
> at
>
[jira] [Commented] (SPARK-29765) Monitoring UI throws IndexOutOfBoundsException when accessing metrics of attempt in stage
[
https://issues.apache.org/jira/browse/SPARK-29765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16968707#comment-16968707
]
shahid commented on SPARK-29765:
Yes, then it seems some event drop has happened. Could you check the logs
related to the event drop? To prevent this, I think you can increase the Event
queue capacity which is by default 1000 I think
> Monitoring UI throws IndexOutOfBoundsException when accessing metrics of
> attempt in stage
> -
>
> Key: SPARK-29765
> URL: https://issues.apache.org/jira/browse/SPARK-29765
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 2.4.4
> Environment: Amazon EMR 5.27
>Reporter: Viacheslav Tradunsky
>Priority: Major
>
> When clicking on one of the largest tasks by input, I get to
> [http://:20888/proxy/application_1572992299050_0001/stages/stage/?id=74=0|http://10.207.110.207:20888/proxy/application_1572992299050_0001/stages/stage/?id=74=0]
> with 500 error
> {code:java}
> java.lang.IndexOutOfBoundsException: 95745 at
> scala.collection.immutable.Vector.checkRangeConvert(Vector.scala:132) at
> scala.collection.immutable.Vector.apply(Vector.scala:122) at
> org.apache.spark.status.AppStatusStore$$anonfun$scanTasks$1$1.apply$mcDJ$sp(AppStatusStore.scala:255)
> at
> org.apache.spark.status.AppStatusStore$$anonfun$scanTasks$1$1.apply(AppStatusStore.scala:254)
> at
> org.apache.spark.status.AppStatusStore$$anonfun$scanTasks$1$1.apply(AppStatusStore.scala:254)
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
> at
> scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
> at scala.collection.mutable.ArrayOps$ofLong.foreach(ArrayOps.scala:246) at
> scala.collection.TraversableLike$class.map(TraversableLike.scala:234) at
> scala.collection.mutable.ArrayOps$ofLong.map(ArrayOps.scala:246) at
> org.apache.spark.status.AppStatusStore.scanTasks$1(AppStatusStore.scala:254)
> at
> org.apache.spark.status.AppStatusStore.taskSummary(AppStatusStore.scala:287)
> at org.apache.spark.ui.jobs.StagePage.render(StagePage.scala:321) at
> org.apache.spark.ui.WebUI$$anonfun$2.apply(WebUI.scala:84) at
> org.apache.spark.ui.WebUI$$anonfun$2.apply(WebUI.scala:84) at
> org.apache.spark.ui.JettyUtils$$anon$3.doGet(JettyUtils.scala:90) at
> javax.servlet.http.HttpServlet.service(HttpServlet.java:687) at
> javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at
> org.spark_project.jetty.servlet.ServletHolder.handle(ServletHolder.java:848)
> at
> org.spark_project.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1772)
> at
> org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter.doFilter(AmIpFilter.java:166)
> at
> org.spark_project.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
> at
> org.spark_project.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
> at
> org.spark_project.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
> at
> org.spark_project.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
> at
> org.spark_project.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
> at
> org.spark_project.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> at
> org.spark_project.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:493)
> at
> org.spark_project.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
> at
> org.spark_project.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
> at org.spark_project.jetty.server.Server.handle(Server.java:539) at
> org.spark_project.jetty.server.HttpChannel.handle(HttpChannel.java:333) at
> org.spark_project.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
> at
> org.spark_project.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
> at org.spark_project.jetty.io.FillInterest.fillable(FillInterest.java:108)
> at
> org.spark_project.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
> at
> org.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
> at
> org.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
> at
> org.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
> at
> org.spark_project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
> at
>
[jira] [Commented] (SPARK-29765) Monitoring UI throws IndexOutOfBoundsException when accessing metrics of attempt in stage
[
https://issues.apache.org/jira/browse/SPARK-29765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16968704#comment-16968704
]
Viacheslav Tradunsky commented on SPARK-29765:
--
[~shahid]
Reproduced.
An interesting thing is that despite the stage was marked as completed, on UI
it showed tasks 106632/109889 (3 running).
> Monitoring UI throws IndexOutOfBoundsException when accessing metrics of
> attempt in stage
> -
>
> Key: SPARK-29765
> URL: https://issues.apache.org/jira/browse/SPARK-29765
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 2.4.4
> Environment: Amazon EMR 5.27
>Reporter: Viacheslav Tradunsky
>Priority: Major
>
> When clicking on one of the largest tasks by input, I get to
> [http://:20888/proxy/application_1572992299050_0001/stages/stage/?id=74=0|http://10.207.110.207:20888/proxy/application_1572992299050_0001/stages/stage/?id=74=0]
> with 500 error
> {code:java}
> java.lang.IndexOutOfBoundsException: 95745 at
> scala.collection.immutable.Vector.checkRangeConvert(Vector.scala:132) at
> scala.collection.immutable.Vector.apply(Vector.scala:122) at
> org.apache.spark.status.AppStatusStore$$anonfun$scanTasks$1$1.apply$mcDJ$sp(AppStatusStore.scala:255)
> at
> org.apache.spark.status.AppStatusStore$$anonfun$scanTasks$1$1.apply(AppStatusStore.scala:254)
> at
> org.apache.spark.status.AppStatusStore$$anonfun$scanTasks$1$1.apply(AppStatusStore.scala:254)
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
> at
> scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
> at scala.collection.mutable.ArrayOps$ofLong.foreach(ArrayOps.scala:246) at
> scala.collection.TraversableLike$class.map(TraversableLike.scala:234) at
> scala.collection.mutable.ArrayOps$ofLong.map(ArrayOps.scala:246) at
> org.apache.spark.status.AppStatusStore.scanTasks$1(AppStatusStore.scala:254)
> at
> org.apache.spark.status.AppStatusStore.taskSummary(AppStatusStore.scala:287)
> at org.apache.spark.ui.jobs.StagePage.render(StagePage.scala:321) at
> org.apache.spark.ui.WebUI$$anonfun$2.apply(WebUI.scala:84) at
> org.apache.spark.ui.WebUI$$anonfun$2.apply(WebUI.scala:84) at
> org.apache.spark.ui.JettyUtils$$anon$3.doGet(JettyUtils.scala:90) at
> javax.servlet.http.HttpServlet.service(HttpServlet.java:687) at
> javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at
> org.spark_project.jetty.servlet.ServletHolder.handle(ServletHolder.java:848)
> at
> org.spark_project.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1772)
> at
> org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter.doFilter(AmIpFilter.java:166)
> at
> org.spark_project.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
> at
> org.spark_project.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
> at
> org.spark_project.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
> at
> org.spark_project.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
> at
> org.spark_project.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
> at
> org.spark_project.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> at
> org.spark_project.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:493)
> at
> org.spark_project.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
> at
> org.spark_project.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
> at org.spark_project.jetty.server.Server.handle(Server.java:539) at
> org.spark_project.jetty.server.HttpChannel.handle(HttpChannel.java:333) at
> org.spark_project.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
> at
> org.spark_project.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
> at org.spark_project.jetty.io.FillInterest.fillable(FillInterest.java:108)
> at
> org.spark_project.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
> at
> org.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
> at
> org.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
> at
> org.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
> at
> org.spark_project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
> at
> org.spark_project.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
> at
[jira] [Commented] (SPARK-29765) Monitoring UI throws IndexOutOfBoundsException when accessing metrics of attempt in stage
[
https://issues.apache.org/jira/browse/SPARK-29765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16968652#comment-16968652
]
shahid commented on SPARK-29765:
I still not sure about the root cause, as I am not able to reproduce with small
data. From the number I can see that it is something related to cleaning up the
store, when the number of tasks exceeds the threshold. If you can still
reproduce with the same data even after increasing the threshold, then it might
be due to some other issue.
> Monitoring UI throws IndexOutOfBoundsException when accessing metrics of
> attempt in stage
> -
>
> Key: SPARK-29765
> URL: https://issues.apache.org/jira/browse/SPARK-29765
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 2.4.4
> Environment: Amazon EMR 5.27
>Reporter: Viacheslav Tradunsky
>Priority: Major
>
> When clicking on one of the largest tasks by input, I get to
> [http://:20888/proxy/application_1572992299050_0001/stages/stage/?id=74=0|http://10.207.110.207:20888/proxy/application_1572992299050_0001/stages/stage/?id=74=0]
> with 500 error
> {code:java}
> java.lang.IndexOutOfBoundsException: 95745 at
> scala.collection.immutable.Vector.checkRangeConvert(Vector.scala:132) at
> scala.collection.immutable.Vector.apply(Vector.scala:122) at
> org.apache.spark.status.AppStatusStore$$anonfun$scanTasks$1$1.apply$mcDJ$sp(AppStatusStore.scala:255)
> at
> org.apache.spark.status.AppStatusStore$$anonfun$scanTasks$1$1.apply(AppStatusStore.scala:254)
> at
> org.apache.spark.status.AppStatusStore$$anonfun$scanTasks$1$1.apply(AppStatusStore.scala:254)
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
> at
> scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
> at scala.collection.mutable.ArrayOps$ofLong.foreach(ArrayOps.scala:246) at
> scala.collection.TraversableLike$class.map(TraversableLike.scala:234) at
> scala.collection.mutable.ArrayOps$ofLong.map(ArrayOps.scala:246) at
> org.apache.spark.status.AppStatusStore.scanTasks$1(AppStatusStore.scala:254)
> at
> org.apache.spark.status.AppStatusStore.taskSummary(AppStatusStore.scala:287)
> at org.apache.spark.ui.jobs.StagePage.render(StagePage.scala:321) at
> org.apache.spark.ui.WebUI$$anonfun$2.apply(WebUI.scala:84) at
> org.apache.spark.ui.WebUI$$anonfun$2.apply(WebUI.scala:84) at
> org.apache.spark.ui.JettyUtils$$anon$3.doGet(JettyUtils.scala:90) at
> javax.servlet.http.HttpServlet.service(HttpServlet.java:687) at
> javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at
> org.spark_project.jetty.servlet.ServletHolder.handle(ServletHolder.java:848)
> at
> org.spark_project.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1772)
> at
> org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter.doFilter(AmIpFilter.java:166)
> at
> org.spark_project.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
> at
> org.spark_project.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
> at
> org.spark_project.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
> at
> org.spark_project.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
> at
> org.spark_project.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
> at
> org.spark_project.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> at
> org.spark_project.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:493)
> at
> org.spark_project.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
> at
> org.spark_project.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
> at org.spark_project.jetty.server.Server.handle(Server.java:539) at
> org.spark_project.jetty.server.HttpChannel.handle(HttpChannel.java:333) at
> org.spark_project.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
> at
> org.spark_project.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
> at org.spark_project.jetty.io.FillInterest.fillable(FillInterest.java:108)
> at
> org.spark_project.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
> at
> org.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
> at
> org.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
> at
> org.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
> at
>
[jira] [Commented] (SPARK-29765) Monitoring UI throws IndexOutOfBoundsException when accessing metrics of attempt in stage
[
https://issues.apache.org/jira/browse/SPARK-29765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16968649#comment-16968649
]
Viacheslav Tradunsky commented on SPARK-29765:
--
[~shahid]
sure, in case we have more than 200 000 tasks shall we set this to our maximum?
Just trying to understand how GC of objects could influence access in immutable
collection. Do you allow somehow the elements to be collected when they are
referenced by that vector?
> Monitoring UI throws IndexOutOfBoundsException when accessing metrics of
> attempt in stage
> -
>
> Key: SPARK-29765
> URL: https://issues.apache.org/jira/browse/SPARK-29765
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 2.4.4
> Environment: Amazon EMR 5.27
>Reporter: Viacheslav Tradunsky
>Priority: Major
>
> When clicking on one of the largest tasks by input, I get to
> [http://:20888/proxy/application_1572992299050_0001/stages/stage/?id=74=0|http://10.207.110.207:20888/proxy/application_1572992299050_0001/stages/stage/?id=74=0]
> with 500 error
> {code:java}
> java.lang.IndexOutOfBoundsException: 95745 at
> scala.collection.immutable.Vector.checkRangeConvert(Vector.scala:132) at
> scala.collection.immutable.Vector.apply(Vector.scala:122) at
> org.apache.spark.status.AppStatusStore$$anonfun$scanTasks$1$1.apply$mcDJ$sp(AppStatusStore.scala:255)
> at
> org.apache.spark.status.AppStatusStore$$anonfun$scanTasks$1$1.apply(AppStatusStore.scala:254)
> at
> org.apache.spark.status.AppStatusStore$$anonfun$scanTasks$1$1.apply(AppStatusStore.scala:254)
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
> at
> scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
> at scala.collection.mutable.ArrayOps$ofLong.foreach(ArrayOps.scala:246) at
> scala.collection.TraversableLike$class.map(TraversableLike.scala:234) at
> scala.collection.mutable.ArrayOps$ofLong.map(ArrayOps.scala:246) at
> org.apache.spark.status.AppStatusStore.scanTasks$1(AppStatusStore.scala:254)
> at
> org.apache.spark.status.AppStatusStore.taskSummary(AppStatusStore.scala:287)
> at org.apache.spark.ui.jobs.StagePage.render(StagePage.scala:321) at
> org.apache.spark.ui.WebUI$$anonfun$2.apply(WebUI.scala:84) at
> org.apache.spark.ui.WebUI$$anonfun$2.apply(WebUI.scala:84) at
> org.apache.spark.ui.JettyUtils$$anon$3.doGet(JettyUtils.scala:90) at
> javax.servlet.http.HttpServlet.service(HttpServlet.java:687) at
> javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at
> org.spark_project.jetty.servlet.ServletHolder.handle(ServletHolder.java:848)
> at
> org.spark_project.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1772)
> at
> org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter.doFilter(AmIpFilter.java:166)
> at
> org.spark_project.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
> at
> org.spark_project.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
> at
> org.spark_project.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
> at
> org.spark_project.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
> at
> org.spark_project.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
> at
> org.spark_project.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> at
> org.spark_project.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:493)
> at
> org.spark_project.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
> at
> org.spark_project.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
> at org.spark_project.jetty.server.Server.handle(Server.java:539) at
> org.spark_project.jetty.server.HttpChannel.handle(HttpChannel.java:333) at
> org.spark_project.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
> at
> org.spark_project.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
> at org.spark_project.jetty.io.FillInterest.fillable(FillInterest.java:108)
> at
> org.spark_project.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
> at
> org.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
> at
> org.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
> at
> org.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
> at
>
[jira] [Commented] (SPARK-29765) Monitoring UI throws IndexOutOfBoundsException when accessing metrics of attempt in stage
[
https://issues.apache.org/jira/browse/SPARK-29765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16968642#comment-16968642
]
shahid commented on SPARK-29765:
Just to narrow down the problem, can you please increase
`spark.ui.retainedTasks` from 10 to 20 ? Thanks
> Monitoring UI throws IndexOutOfBoundsException when accessing metrics of
> attempt in stage
> -
>
> Key: SPARK-29765
> URL: https://issues.apache.org/jira/browse/SPARK-29765
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 2.4.4
> Environment: Amazon EMR 5.27
>Reporter: Viacheslav Tradunsky
>Priority: Major
>
> When clicking on one of the largest tasks by input, I get to
> [http://:20888/proxy/application_1572992299050_0001/stages/stage/?id=74=0|http://10.207.110.207:20888/proxy/application_1572992299050_0001/stages/stage/?id=74=0]
> with 500 error
> {code:java}
> java.lang.IndexOutOfBoundsException: 95745 at
> scala.collection.immutable.Vector.checkRangeConvert(Vector.scala:132) at
> scala.collection.immutable.Vector.apply(Vector.scala:122) at
> org.apache.spark.status.AppStatusStore$$anonfun$scanTasks$1$1.apply$mcDJ$sp(AppStatusStore.scala:255)
> at
> org.apache.spark.status.AppStatusStore$$anonfun$scanTasks$1$1.apply(AppStatusStore.scala:254)
> at
> org.apache.spark.status.AppStatusStore$$anonfun$scanTasks$1$1.apply(AppStatusStore.scala:254)
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
> at
> scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
> at scala.collection.mutable.ArrayOps$ofLong.foreach(ArrayOps.scala:246) at
> scala.collection.TraversableLike$class.map(TraversableLike.scala:234) at
> scala.collection.mutable.ArrayOps$ofLong.map(ArrayOps.scala:246) at
> org.apache.spark.status.AppStatusStore.scanTasks$1(AppStatusStore.scala:254)
> at
> org.apache.spark.status.AppStatusStore.taskSummary(AppStatusStore.scala:287)
> at org.apache.spark.ui.jobs.StagePage.render(StagePage.scala:321) at
> org.apache.spark.ui.WebUI$$anonfun$2.apply(WebUI.scala:84) at
> org.apache.spark.ui.WebUI$$anonfun$2.apply(WebUI.scala:84) at
> org.apache.spark.ui.JettyUtils$$anon$3.doGet(JettyUtils.scala:90) at
> javax.servlet.http.HttpServlet.service(HttpServlet.java:687) at
> javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at
> org.spark_project.jetty.servlet.ServletHolder.handle(ServletHolder.java:848)
> at
> org.spark_project.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1772)
> at
> org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter.doFilter(AmIpFilter.java:166)
> at
> org.spark_project.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
> at
> org.spark_project.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
> at
> org.spark_project.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
> at
> org.spark_project.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
> at
> org.spark_project.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
> at
> org.spark_project.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> at
> org.spark_project.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:493)
> at
> org.spark_project.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
> at
> org.spark_project.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
> at org.spark_project.jetty.server.Server.handle(Server.java:539) at
> org.spark_project.jetty.server.HttpChannel.handle(HttpChannel.java:333) at
> org.spark_project.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
> at
> org.spark_project.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
> at org.spark_project.jetty.io.FillInterest.fillable(FillInterest.java:108)
> at
> org.spark_project.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
> at
> org.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
> at
> org.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
> at
> org.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
> at
> org.spark_project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
> at
> org.spark_project.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
> at java.lang.Thread.run(Thread.java:748){code}
--
This message was
[jira] [Commented] (SPARK-29765) Monitoring UI throws IndexOutOfBoundsException when accessing metrics of attempt in stage
[
https://issues.apache.org/jira/browse/SPARK-29765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16968318#comment-16968318
]
Viacheslav Tradunsky commented on SPARK-29765:
--
ok, got it.
> Monitoring UI throws IndexOutOfBoundsException when accessing metrics of
> attempt in stage
> -
>
> Key: SPARK-29765
> URL: https://issues.apache.org/jira/browse/SPARK-29765
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 2.4.4
> Environment: Amazon EMR 5.27
>Reporter: Viacheslav Tradunsky
>Priority: Major
>
> When clicking on one of the largest tasks by input, I get to
> [http://:20888/proxy/application_1572992299050_0001/stages/stage/?id=74=0|http://10.207.110.207:20888/proxy/application_1572992299050_0001/stages/stage/?id=74=0]
> with 500 error
> {code:java}
> java.lang.IndexOutOfBoundsException: 95745 at
> scala.collection.immutable.Vector.checkRangeConvert(Vector.scala:132) at
> scala.collection.immutable.Vector.apply(Vector.scala:122) at
> org.apache.spark.status.AppStatusStore$$anonfun$scanTasks$1$1.apply$mcDJ$sp(AppStatusStore.scala:255)
> at
> org.apache.spark.status.AppStatusStore$$anonfun$scanTasks$1$1.apply(AppStatusStore.scala:254)
> at
> org.apache.spark.status.AppStatusStore$$anonfun$scanTasks$1$1.apply(AppStatusStore.scala:254)
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
> at
> scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
> at scala.collection.mutable.ArrayOps$ofLong.foreach(ArrayOps.scala:246) at
> scala.collection.TraversableLike$class.map(TraversableLike.scala:234) at
> scala.collection.mutable.ArrayOps$ofLong.map(ArrayOps.scala:246) at
> org.apache.spark.status.AppStatusStore.scanTasks$1(AppStatusStore.scala:254)
> at
> org.apache.spark.status.AppStatusStore.taskSummary(AppStatusStore.scala:287)
> at org.apache.spark.ui.jobs.StagePage.render(StagePage.scala:321) at
> org.apache.spark.ui.WebUI$$anonfun$2.apply(WebUI.scala:84) at
> org.apache.spark.ui.WebUI$$anonfun$2.apply(WebUI.scala:84) at
> org.apache.spark.ui.JettyUtils$$anon$3.doGet(JettyUtils.scala:90) at
> javax.servlet.http.HttpServlet.service(HttpServlet.java:687) at
> javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at
> org.spark_project.jetty.servlet.ServletHolder.handle(ServletHolder.java:848)
> at
> org.spark_project.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1772)
> at
> org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter.doFilter(AmIpFilter.java:166)
> at
> org.spark_project.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
> at
> org.spark_project.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
> at
> org.spark_project.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
> at
> org.spark_project.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
> at
> org.spark_project.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
> at
> org.spark_project.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> at
> org.spark_project.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:493)
> at
> org.spark_project.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
> at
> org.spark_project.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
> at org.spark_project.jetty.server.Server.handle(Server.java:539) at
> org.spark_project.jetty.server.HttpChannel.handle(HttpChannel.java:333) at
> org.spark_project.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
> at
> org.spark_project.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
> at org.spark_project.jetty.io.FillInterest.fillable(FillInterest.java:108)
> at
> org.spark_project.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
> at
> org.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
> at
> org.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
> at
> org.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
> at
> org.spark_project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
> at
> org.spark_project.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
> at java.lang.Thread.run(Thread.java:748){code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (SPARK-29765) Monitoring UI throws IndexOutOfBoundsException when accessing metrics of attempt in stage
[
https://issues.apache.org/jira/browse/SPARK-29765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16968314#comment-16968314
]
Hyukjin Kwon commented on SPARK-29765:
--
Please avoid to set Critical+ which is usually reserved for committers.
> Monitoring UI throws IndexOutOfBoundsException when accessing metrics of
> attempt in stage
> -
>
> Key: SPARK-29765
> URL: https://issues.apache.org/jira/browse/SPARK-29765
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 2.4.4
> Environment: Amazon EMR 5.27
>Reporter: Viacheslav Tradunsky
>Priority: Major
>
> When clicking on one of the largest tasks by input, I get to
> [http://:20888/proxy/application_1572992299050_0001/stages/stage/?id=74=0|http://10.207.110.207:20888/proxy/application_1572992299050_0001/stages/stage/?id=74=0]
> with 500 error
> {code:java}
> java.lang.IndexOutOfBoundsException: 95745 at
> scala.collection.immutable.Vector.checkRangeConvert(Vector.scala:132) at
> scala.collection.immutable.Vector.apply(Vector.scala:122) at
> org.apache.spark.status.AppStatusStore$$anonfun$scanTasks$1$1.apply$mcDJ$sp(AppStatusStore.scala:255)
> at
> org.apache.spark.status.AppStatusStore$$anonfun$scanTasks$1$1.apply(AppStatusStore.scala:254)
> at
> org.apache.spark.status.AppStatusStore$$anonfun$scanTasks$1$1.apply(AppStatusStore.scala:254)
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
> at
> scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
> at scala.collection.mutable.ArrayOps$ofLong.foreach(ArrayOps.scala:246) at
> scala.collection.TraversableLike$class.map(TraversableLike.scala:234) at
> scala.collection.mutable.ArrayOps$ofLong.map(ArrayOps.scala:246) at
> org.apache.spark.status.AppStatusStore.scanTasks$1(AppStatusStore.scala:254)
> at
> org.apache.spark.status.AppStatusStore.taskSummary(AppStatusStore.scala:287)
> at org.apache.spark.ui.jobs.StagePage.render(StagePage.scala:321) at
> org.apache.spark.ui.WebUI$$anonfun$2.apply(WebUI.scala:84) at
> org.apache.spark.ui.WebUI$$anonfun$2.apply(WebUI.scala:84) at
> org.apache.spark.ui.JettyUtils$$anon$3.doGet(JettyUtils.scala:90) at
> javax.servlet.http.HttpServlet.service(HttpServlet.java:687) at
> javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at
> org.spark_project.jetty.servlet.ServletHolder.handle(ServletHolder.java:848)
> at
> org.spark_project.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1772)
> at
> org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter.doFilter(AmIpFilter.java:166)
> at
> org.spark_project.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
> at
> org.spark_project.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
> at
> org.spark_project.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
> at
> org.spark_project.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
> at
> org.spark_project.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
> at
> org.spark_project.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> at
> org.spark_project.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:493)
> at
> org.spark_project.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
> at
> org.spark_project.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
> at org.spark_project.jetty.server.Server.handle(Server.java:539) at
> org.spark_project.jetty.server.HttpChannel.handle(HttpChannel.java:333) at
> org.spark_project.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
> at
> org.spark_project.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
> at org.spark_project.jetty.io.FillInterest.fillable(FillInterest.java:108)
> at
> org.spark_project.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
> at
> org.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
> at
> org.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
> at
> org.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
> at
> org.spark_project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
> at
> org.spark_project.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
> at java.lang.Thread.run(Thread.java:748){code}
--
This message was sent by Atlassian Jira
