[jira] [Updated] (SPARK-35914) Driver can't distribute task to executor because NullPointerException
[
https://issues.apache.org/jira/browse/SPARK-35914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Helt Long updated SPARK-35914:
--
Environment:
hadoop 2.6.0-cdh5.7.1
Spark 3.0.1, 3.1.1, 3.1.2
was:
CDH 5.7.1: Hadoop 2.6.5
Spark 3.0.1, 3.1.1, 3.1.2
> Driver can't distribute task to executor because NullPointerException
> -
>
> Key: SPARK-35914
> URL: https://issues.apache.org/jira/browse/SPARK-35914
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 3.0.1, 3.1.1, 3.1.2
> Environment: hadoop 2.6.0-cdh5.7.1
> Spark 3.0.1, 3.1.1, 3.1.2
>Reporter: Helt Long
>Priority: Major
> Attachments: stuck log.png, webui stuck.png
>
>
> When use spark3 submit a spark job to yarn cluster, I get a problem. Once in
> a while, driver can't distribute any tasks to any executors, and the stage
> will stuck , the total spark job will stuck. Check driver log, I found
> NullPointerException. It's like a netty problem, I can confirm this problem
> only exist in spark3, because I use spark2 never happend.
>
> {code:java}
> // Error message
> 21/06/28 14:42:43 INFO TaskSetManager: Starting task 2592.0 in stage 1.0 (TID
> 3494) (worker39.hadoop, executor 84, partition 2592, RACK_LOCAL, 5006 bytes)
> taskResourceAssignments Map()
> 21/06/28 14:42:43 INFO TaskSetManager: Finished task 4155.0 in stage 1.0 (TID
> 3367) in 36670 ms on worker39.hadoop (executor 84) (3278/4249)
> 21/06/28 14:42:43 INFO TaskSetManager: Finished task 2283.0 in stage 1.0 (TID
> 3422) in 22371 ms on worker15.hadoop (executor 109) (3279/4249)
> 21/06/28 14:42:43 ERROR Inbox: Ignoring error
> java.lang.NullPointerException
> at java.lang.String.length(String.java:623)
> at
> java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:420)
> at java.lang.StringBuilder.append(StringBuilder.java:136)
> at
> org.apache.spark.scheduler.TaskSetManager.$anonfun$resourceOffer$5(TaskSetManager.scala:483)
> at org.apache.spark.internal.Logging.logInfo(Logging.scala:57)
> at org.apache.spark.internal.Logging.logInfo$(Logging.scala:56)
> at
> org.apache.spark.scheduler.TaskSetManager.logInfo(TaskSetManager.scala:54)
> at
> org.apache.spark.scheduler.TaskSetManager.$anonfun$resourceOffer$2(TaskSetManager.scala:484)
> at scala.Option.map(Option.scala:230)
> at
> org.apache.spark.scheduler.TaskSetManager.resourceOffer(TaskSetManager.scala:444)
> at
> org.apache.spark.scheduler.TaskSchedulerImpl.$anonfun$resourceOfferSingleTaskSet$2(TaskSchedulerImpl.scala:397)
> at
> org.apache.spark.scheduler.TaskSchedulerImpl.$anonfun$resourceOfferSingleTaskSet$2$adapted(TaskSchedulerImpl.scala:392)
> at scala.Option.foreach(Option.scala:407)
> at
> org.apache.spark.scheduler.TaskSchedulerImpl.$anonfun$resourceOfferSingleTaskSet$1(TaskSchedulerImpl.scala:392)
> at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:158)
> at
> org.apache.spark.scheduler.TaskSchedulerImpl.resourceOfferSingleTaskSet(TaskSchedulerImpl.scala:383)
> at
> org.apache.spark.scheduler.TaskSchedulerImpl.$anonfun$resourceOffers$20(TaskSchedulerImpl.scala:581)
> at
> org.apache.spark.scheduler.TaskSchedulerImpl.$anonfun$resourceOffers$20$adapted(TaskSchedulerImpl.scala:576)
> at
> scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)
> at
> scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33)
> at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:198)
> at
> org.apache.spark.scheduler.TaskSchedulerImpl.$anonfun$resourceOffers$16(TaskSchedulerImpl.scala:576)
> at
> org.apache.spark.scheduler.TaskSchedulerImpl.$anonfun$resourceOffers$16$adapted(TaskSchedulerImpl.scala:547)
> at
> scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
> at
> scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
> at
> org.apache.spark.scheduler.TaskSchedulerImpl.resourceOffers(TaskSchedulerImpl.scala:547)
> at
> org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend$DriverEndpoint.$anonfun$makeOffers$5(CoarseGrainedSchedulerBackend.scala:340)
> at
> org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend.org$apache$spark$scheduler$cluster$CoarseGrainedSchedulerBackend$$withLock(CoarseGrainedSchedulerBackend.scala:904)
> at
> org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend$DriverEndpoint.org$apache$spark$scheduler$cluster$CoarseGrainedSchedulerBackend$DriverEndpoint$$makeOffers(CoarseGrainedSchedulerBackend.scala:332)
> at
> org.apache.spark.schedu
[jira] [Commented] (SPARK-35914) Driver can't distribute task to executor because NullPointerException
[
https://issues.apache.org/jira/browse/SPARK-35914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17371758#comment-17371758
]
Helt Long commented on SPARK-35914:
---
Do what you wanna do, [~code_kr_dev_s], but this problem is difficult to
recurrent, it's random occurrence. If you need any help, tell me!
> Driver can't distribute task to executor because NullPointerException
> -
>
> Key: SPARK-35914
> URL: https://issues.apache.org/jira/browse/SPARK-35914
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 3.0.1, 3.1.1, 3.1.2
> Environment: CDH 5.7.1: Hadoop 2.6.5
> Spark 3.0.1, 3.1.1, 3.1.2
>Reporter: Helt Long
>Priority: Major
> Attachments: stuck log.png, webui stuck.png
>
>
> When use spark3 submit a spark job to yarn cluster, I get a problem. Once in
> a while, driver can't distribute any tasks to any executors, and the stage
> will stuck , the total spark job will stuck. Check driver log, I found
> NullPointerException. It's like a netty problem, I can confirm this problem
> only exist in spark3, because I use spark2 never happend.
>
> {code:java}
> // Error message
> 21/06/28 14:42:43 INFO TaskSetManager: Starting task 2592.0 in stage 1.0 (TID
> 3494) (worker39.hadoop, executor 84, partition 2592, RACK_LOCAL, 5006 bytes)
> taskResourceAssignments Map()
> 21/06/28 14:42:43 INFO TaskSetManager: Finished task 4155.0 in stage 1.0 (TID
> 3367) in 36670 ms on worker39.hadoop (executor 84) (3278/4249)
> 21/06/28 14:42:43 INFO TaskSetManager: Finished task 2283.0 in stage 1.0 (TID
> 3422) in 22371 ms on worker15.hadoop (executor 109) (3279/4249)
> 21/06/28 14:42:43 ERROR Inbox: Ignoring error
> java.lang.NullPointerException
> at java.lang.String.length(String.java:623)
> at
> java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:420)
> at java.lang.StringBuilder.append(StringBuilder.java:136)
> at
> org.apache.spark.scheduler.TaskSetManager.$anonfun$resourceOffer$5(TaskSetManager.scala:483)
> at org.apache.spark.internal.Logging.logInfo(Logging.scala:57)
> at org.apache.spark.internal.Logging.logInfo$(Logging.scala:56)
> at
> org.apache.spark.scheduler.TaskSetManager.logInfo(TaskSetManager.scala:54)
> at
> org.apache.spark.scheduler.TaskSetManager.$anonfun$resourceOffer$2(TaskSetManager.scala:484)
> at scala.Option.map(Option.scala:230)
> at
> org.apache.spark.scheduler.TaskSetManager.resourceOffer(TaskSetManager.scala:444)
> at
> org.apache.spark.scheduler.TaskSchedulerImpl.$anonfun$resourceOfferSingleTaskSet$2(TaskSchedulerImpl.scala:397)
> at
> org.apache.spark.scheduler.TaskSchedulerImpl.$anonfun$resourceOfferSingleTaskSet$2$adapted(TaskSchedulerImpl.scala:392)
> at scala.Option.foreach(Option.scala:407)
> at
> org.apache.spark.scheduler.TaskSchedulerImpl.$anonfun$resourceOfferSingleTaskSet$1(TaskSchedulerImpl.scala:392)
> at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:158)
> at
> org.apache.spark.scheduler.TaskSchedulerImpl.resourceOfferSingleTaskSet(TaskSchedulerImpl.scala:383)
> at
> org.apache.spark.scheduler.TaskSchedulerImpl.$anonfun$resourceOffers$20(TaskSchedulerImpl.scala:581)
> at
> org.apache.spark.scheduler.TaskSchedulerImpl.$anonfun$resourceOffers$20$adapted(TaskSchedulerImpl.scala:576)
> at
> scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)
> at
> scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33)
> at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:198)
> at
> org.apache.spark.scheduler.TaskSchedulerImpl.$anonfun$resourceOffers$16(TaskSchedulerImpl.scala:576)
> at
> org.apache.spark.scheduler.TaskSchedulerImpl.$anonfun$resourceOffers$16$adapted(TaskSchedulerImpl.scala:547)
> at
> scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
> at
> scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
> at
> org.apache.spark.scheduler.TaskSchedulerImpl.resourceOffers(TaskSchedulerImpl.scala:547)
> at
> org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend$DriverEndpoint.$anonfun$makeOffers$5(CoarseGrainedSchedulerBackend.scala:340)
> at
> org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend.org$apache$spark$scheduler$cluster$CoarseGrainedSchedulerBackend$$withLock(CoarseGrainedSchedulerBackend.scala:904)
> at
> org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend$DriverEndpoint.org$apache$spark$scheduler$cluster$CoarseGrainedSchedulerBackend$DriverEndpoint$$makeOffers(Coars
[jira] [Commented] (SPARK-35914) Driver can't distribute task to executor because NullPointerException
[
https://issues.apache.org/jira/browse/SPARK-35914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17371009#comment-17371009
]
Helt Long commented on SPARK-35914:
---
I guess this problem is related to hadoop-version, I use
CDH-5.7.1:hadoop-2.6.5, spark3 use hadoop-2.7. Because the other problem I
found in spark about webui, it's caused by the version. I will try higher
hadoop-version to confirm it.
[SPARK-35802] Error loading the stages/stage/ page in spark UI - ASF JIRA
(apache.org)
> Driver can't distribute task to executor because NullPointerException
> -
>
> Key: SPARK-35914
> URL: https://issues.apache.org/jira/browse/SPARK-35914
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 3.0.1, 3.1.1, 3.1.2
> Environment: CDH 5.7.1: Hadoop 2.6.5
> Spark 3.0.1, 3.1.1, 3.1.2
>Reporter: Helt Long
>Priority: Major
> Attachments: stuck log.png, webui stuck.png
>
>
> When use spark3 submit a spark job to yarn cluster, I get a problem. Once in
> a while, driver can't distribute any tasks to any executors, and the stage
> will stuck , the total spark job will stuck. Check driver log, I found
> NullPointerException. It's like a netty problem, I can confirm this problem
> only exist in spark3, because I use spark2 never happend.
>
> {code:java}
> // Error message
> 21/06/28 14:42:43 INFO TaskSetManager: Starting task 2592.0 in stage 1.0 (TID
> 3494) (worker39.hadoop, executor 84, partition 2592, RACK_LOCAL, 5006 bytes)
> taskResourceAssignments Map()
> 21/06/28 14:42:43 INFO TaskSetManager: Finished task 4155.0 in stage 1.0 (TID
> 3367) in 36670 ms on worker39.hadoop (executor 84) (3278/4249)
> 21/06/28 14:42:43 INFO TaskSetManager: Finished task 2283.0 in stage 1.0 (TID
> 3422) in 22371 ms on worker15.hadoop (executor 109) (3279/4249)
> 21/06/28 14:42:43 ERROR Inbox: Ignoring error
> java.lang.NullPointerException
> at java.lang.String.length(String.java:623)
> at
> java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:420)
> at java.lang.StringBuilder.append(StringBuilder.java:136)
> at
> org.apache.spark.scheduler.TaskSetManager.$anonfun$resourceOffer$5(TaskSetManager.scala:483)
> at org.apache.spark.internal.Logging.logInfo(Logging.scala:57)
> at org.apache.spark.internal.Logging.logInfo$(Logging.scala:56)
> at
> org.apache.spark.scheduler.TaskSetManager.logInfo(TaskSetManager.scala:54)
> at
> org.apache.spark.scheduler.TaskSetManager.$anonfun$resourceOffer$2(TaskSetManager.scala:484)
> at scala.Option.map(Option.scala:230)
> at
> org.apache.spark.scheduler.TaskSetManager.resourceOffer(TaskSetManager.scala:444)
> at
> org.apache.spark.scheduler.TaskSchedulerImpl.$anonfun$resourceOfferSingleTaskSet$2(TaskSchedulerImpl.scala:397)
> at
> org.apache.spark.scheduler.TaskSchedulerImpl.$anonfun$resourceOfferSingleTaskSet$2$adapted(TaskSchedulerImpl.scala:392)
> at scala.Option.foreach(Option.scala:407)
> at
> org.apache.spark.scheduler.TaskSchedulerImpl.$anonfun$resourceOfferSingleTaskSet$1(TaskSchedulerImpl.scala:392)
> at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:158)
> at
> org.apache.spark.scheduler.TaskSchedulerImpl.resourceOfferSingleTaskSet(TaskSchedulerImpl.scala:383)
> at
> org.apache.spark.scheduler.TaskSchedulerImpl.$anonfun$resourceOffers$20(TaskSchedulerImpl.scala:581)
> at
> org.apache.spark.scheduler.TaskSchedulerImpl.$anonfun$resourceOffers$20$adapted(TaskSchedulerImpl.scala:576)
> at
> scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)
> at
> scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33)
> at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:198)
> at
> org.apache.spark.scheduler.TaskSchedulerImpl.$anonfun$resourceOffers$16(TaskSchedulerImpl.scala:576)
> at
> org.apache.spark.scheduler.TaskSchedulerImpl.$anonfun$resourceOffers$16$adapted(TaskSchedulerImpl.scala:547)
> at
> scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
> at
> scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
> at
> org.apache.spark.scheduler.TaskSchedulerImpl.resourceOffers(TaskSchedulerImpl.scala:547)
> at
> org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend$DriverEndpoint.$anonfun$makeOffers$5(CoarseGrainedSchedulerBackend.scala:340)
> at
> org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend.org$apache$spark$scheduler$cluster$CoarseGrainedSchedulerBackend$$withLock(CoarseGrainedSchedulerBackend.scala:904)
>
[jira] [Resolved] (SPARK-35802) Error loading the stages/stage/ page in spark UI
[
https://issues.apache.org/jira/browse/SPARK-35802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Helt Long resolved SPARK-35802.
---
Resolution: Not A Bug
I try hadoop-2.7.5, the problem is not existed. So I confirm it's because I
used hadoop-2.6.5. Sorry for this, I closed It.
> Error loading the stages/stage/ page in spark UI
>
>
> Key: SPARK-35802
> URL: https://issues.apache.org/jira/browse/SPARK-35802
> Project: Spark
> Issue Type: Bug
> Components: Web UI
>Affects Versions: 3.0.0, 3.0.1, 3.1.1, 3.1.2
> Environment: CDH 5.7.1： Hadoop 2.6.5
> Spark on yarn cluster mode
>Reporter: Helt Long
>Priority: Major
> Attachments: spark3.1.2-request-20210628093538.png,
> spark3.1.2-stage-20210628093549.png, spark3.1.2-webui-20210628093559.png
>
>
> I try to load the sparkUI page for a specific stage, I get the following
> error:
> {quote}Unable to connect to the server. Looks like the Spark application must
> have ended. Please Switch to the history UI.
> {quote}
> Obviously the server is still alive and process new messages.
> Looking at the network tab shows one of the requests fails:
>
> {{curl
> 'http://:8080/proxy/app-20201008130147-0001/api/v1/applications/app-20201008130147-0001/stages/11/0/taskTable'
>
>
>
> Error 500 Request failed.
>
> HTTP ERROR 500
> Problem accessing
> /api/v1/applications/app-20201008130147-0001/stages/11/0/taskTable. Reason:
> Request failed. href="http://eclipse.org/jetty";>Powered by Jetty:// 9.4.z-SNAPSHOT
>
> }}
> requests to any other object that I've tested seem to work, for example
>
> {{curl
> 'http://:8080/proxy/app-20201008130147-0001/api/v1/applications/app-20201008130147-0001/stages/11/0/taskSummary'}}
>
> The exception is:
> {{/api/v1/applications/app-20201008130147-0001/stages/11/0/taskTable
> javax.servlet.ServletException: java.lang.NullPointerException
> at
> org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:410)
> at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346)
> at
> org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:366)
> at
> org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:319)
> at
> org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)
> at
> org.sparkproject.jetty.servlet.ServletHolder.handle(ServletHolder.java:873)
> at
> org.sparkproject.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1623)
> at
> org.apache.spark.ui.HttpSecurityFilter.doFilter(HttpSecurityFilter.scala:95)
> at
> org.sparkproject.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
> at
> org.sparkproject.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
> at
> org.sparkproject.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
> at
> org.sparkproject.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)
> at
> org.sparkproject.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
> at
> org.sparkproject.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
> at
> org.sparkproject.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
> at
> org.sparkproject.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)
> at
> org.sparkproject.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
> at
> org.sparkproject.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:753)
> at
> org.sparkproject.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220)
> at
> org.sparkproject.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
> at org.sparkproject.jetty.server.Server.handle(Server.java:505)
> at org.sparkproject.jetty.server.HttpChannel.handle(HttpChannel.java:370)
> at
> org.sparkproject.jetty.server.HttpConnection.onFillable(HttpConnection.java:267)
> at
> org.sparkproject.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
> at org.sparkproject.jetty.io.FillInterest.fillable(FillInterest.java:103)
> at org.sparkproject.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
> at
> org.sparkproject.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
> at
> org.sparkproject.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
> at
> org.sparkproject.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
> at
> org.sparkproject.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
> at
> org.sparkproject.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
> at
> org.sparkproject.jetty.util.th
[jira] [Commented] (SPARK-35802) Error loading the stages/stage/ page in spark UI
[
https://issues.apache.org/jira/browse/SPARK-35802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17370626#comment-17370626
]
Helt Long commented on SPARK-35802:
---
Sound like it's key point, I will try higher hadoop version and close it
myself. Thanks [~sarutak]!
> Error loading the stages/stage/ page in spark UI
>
>
> Key: SPARK-35802
> URL: https://issues.apache.org/jira/browse/SPARK-35802
> Project: Spark
> Issue Type: Bug
> Components: Web UI
>Affects Versions: 3.0.0, 3.0.1, 3.1.1, 3.1.2
> Environment: CDH 5.7.1： Hadoop 2.6.5
> Spark on yarn cluster mode
>Reporter: Helt Long
>Priority: Major
> Attachments: spark3.1.2-request-20210628093538.png,
> spark3.1.2-stage-20210628093549.png, spark3.1.2-webui-20210628093559.png
>
>
> I try to load the sparkUI page for a specific stage, I get the following
> error:
> {quote}Unable to connect to the server. Looks like the Spark application must
> have ended. Please Switch to the history UI.
> {quote}
> Obviously the server is still alive and process new messages.
> Looking at the network tab shows one of the requests fails:
>
> {{curl
> 'http://:8080/proxy/app-20201008130147-0001/api/v1/applications/app-20201008130147-0001/stages/11/0/taskTable'
>
>
>
> Error 500 Request failed.
>
> HTTP ERROR 500
> Problem accessing
> /api/v1/applications/app-20201008130147-0001/stages/11/0/taskTable. Reason:
> Request failed. href="http://eclipse.org/jetty";>Powered by Jetty:// 9.4.z-SNAPSHOT
>
> }}
> requests to any other object that I've tested seem to work, for example
>
> {{curl
> 'http://:8080/proxy/app-20201008130147-0001/api/v1/applications/app-20201008130147-0001/stages/11/0/taskSummary'}}
>
> The exception is:
> {{/api/v1/applications/app-20201008130147-0001/stages/11/0/taskTable
> javax.servlet.ServletException: java.lang.NullPointerException
> at
> org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:410)
> at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346)
> at
> org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:366)
> at
> org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:319)
> at
> org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)
> at
> org.sparkproject.jetty.servlet.ServletHolder.handle(ServletHolder.java:873)
> at
> org.sparkproject.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1623)
> at
> org.apache.spark.ui.HttpSecurityFilter.doFilter(HttpSecurityFilter.scala:95)
> at
> org.sparkproject.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
> at
> org.sparkproject.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
> at
> org.sparkproject.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
> at
> org.sparkproject.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)
> at
> org.sparkproject.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
> at
> org.sparkproject.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
> at
> org.sparkproject.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
> at
> org.sparkproject.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)
> at
> org.sparkproject.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
> at
> org.sparkproject.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:753)
> at
> org.sparkproject.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220)
> at
> org.sparkproject.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
> at org.sparkproject.jetty.server.Server.handle(Server.java:505)
> at org.sparkproject.jetty.server.HttpChannel.handle(HttpChannel.java:370)
> at
> org.sparkproject.jetty.server.HttpConnection.onFillable(HttpConnection.java:267)
> at
> org.sparkproject.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
> at org.sparkproject.jetty.io.FillInterest.fillable(FillInterest.java:103)
> at org.sparkproject.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
> at
> org.sparkproject.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
> at
> org.sparkproject.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
> at
> org.sparkproject.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
> at
> org.sparkproject.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
> at
> org.sparkproject.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
> at
> org.sparkproject.jetty.ut
[jira] [Updated] (SPARK-35914) Driver can't distribute task to executor because NullPointerException
[
https://issues.apache.org/jira/browse/SPARK-35914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Helt Long updated SPARK-35914:
--
Attachment: webui stuck.png
> Driver can't distribute task to executor because NullPointerException
> -
>
> Key: SPARK-35914
> URL: https://issues.apache.org/jira/browse/SPARK-35914
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 3.0.1, 3.1.1, 3.1.2
> Environment: CDH 5.7.1: Hadoop 2.6.5
> Spark 3.0.1, 3.1.1, 3.1.2
>Reporter: Helt Long
>Priority: Major
> Attachments: stuck log.png, webui stuck.png
>
>
> When use spark3 submit a spark job to yarn cluster, I get a problem. Once in
> a while, driver can't distribute any tasks to any executors, and the stage
> will stuck , the total spark job will stuck. Check driver log, I found
> NullPointerException. It's like a netty problem, I can confirm this problem
> only exist in spark3, because I use spark2 never happend.
>
> {code:java}
> // Error message
> 21/06/28 14:42:43 INFO TaskSetManager: Starting task 2592.0 in stage 1.0 (TID
> 3494) (worker39.hadoop, executor 84, partition 2592, RACK_LOCAL, 5006 bytes)
> taskResourceAssignments Map()
> 21/06/28 14:42:43 INFO TaskSetManager: Finished task 4155.0 in stage 1.0 (TID
> 3367) in 36670 ms on worker39.hadoop (executor 84) (3278/4249)
> 21/06/28 14:42:43 INFO TaskSetManager: Finished task 2283.0 in stage 1.0 (TID
> 3422) in 22371 ms on worker15.hadoop (executor 109) (3279/4249)
> 21/06/28 14:42:43 ERROR Inbox: Ignoring error
> java.lang.NullPointerException
> at java.lang.String.length(String.java:623)
> at
> java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:420)
> at java.lang.StringBuilder.append(StringBuilder.java:136)
> at
> org.apache.spark.scheduler.TaskSetManager.$anonfun$resourceOffer$5(TaskSetManager.scala:483)
> at org.apache.spark.internal.Logging.logInfo(Logging.scala:57)
> at org.apache.spark.internal.Logging.logInfo$(Logging.scala:56)
> at
> org.apache.spark.scheduler.TaskSetManager.logInfo(TaskSetManager.scala:54)
> at
> org.apache.spark.scheduler.TaskSetManager.$anonfun$resourceOffer$2(TaskSetManager.scala:484)
> at scala.Option.map(Option.scala:230)
> at
> org.apache.spark.scheduler.TaskSetManager.resourceOffer(TaskSetManager.scala:444)
> at
> org.apache.spark.scheduler.TaskSchedulerImpl.$anonfun$resourceOfferSingleTaskSet$2(TaskSchedulerImpl.scala:397)
> at
> org.apache.spark.scheduler.TaskSchedulerImpl.$anonfun$resourceOfferSingleTaskSet$2$adapted(TaskSchedulerImpl.scala:392)
> at scala.Option.foreach(Option.scala:407)
> at
> org.apache.spark.scheduler.TaskSchedulerImpl.$anonfun$resourceOfferSingleTaskSet$1(TaskSchedulerImpl.scala:392)
> at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:158)
> at
> org.apache.spark.scheduler.TaskSchedulerImpl.resourceOfferSingleTaskSet(TaskSchedulerImpl.scala:383)
> at
> org.apache.spark.scheduler.TaskSchedulerImpl.$anonfun$resourceOffers$20(TaskSchedulerImpl.scala:581)
> at
> org.apache.spark.scheduler.TaskSchedulerImpl.$anonfun$resourceOffers$20$adapted(TaskSchedulerImpl.scala:576)
> at
> scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)
> at
> scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33)
> at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:198)
> at
> org.apache.spark.scheduler.TaskSchedulerImpl.$anonfun$resourceOffers$16(TaskSchedulerImpl.scala:576)
> at
> org.apache.spark.scheduler.TaskSchedulerImpl.$anonfun$resourceOffers$16$adapted(TaskSchedulerImpl.scala:547)
> at
> scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
> at
> scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
> at
> org.apache.spark.scheduler.TaskSchedulerImpl.resourceOffers(TaskSchedulerImpl.scala:547)
> at
> org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend$DriverEndpoint.$anonfun$makeOffers$5(CoarseGrainedSchedulerBackend.scala:340)
> at
> org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend.org$apache$spark$scheduler$cluster$CoarseGrainedSchedulerBackend$$withLock(CoarseGrainedSchedulerBackend.scala:904)
> at
> org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend$DriverEndpoint.org$apache$spark$scheduler$cluster$CoarseGrainedSchedulerBackend$DriverEndpoint$$makeOffers(CoarseGrainedSchedulerBackend.scala:332)
> at
> org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend$DriverEndpoint$$anonfun$receive$1.applyOrElse(Coars
[jira] [Updated] (SPARK-35914) Driver can't distribute task to executor because NullPointerException
[
https://issues.apache.org/jira/browse/SPARK-35914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Helt Long updated SPARK-35914:
--
Attachment: stuck log.png
> Driver can't distribute task to executor because NullPointerException
> -
>
> Key: SPARK-35914
> URL: https://issues.apache.org/jira/browse/SPARK-35914
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 3.0.1, 3.1.1, 3.1.2
> Environment: CDH 5.7.1: Hadoop 2.6.5
> Spark 3.0.1, 3.1.1, 3.1.2
>Reporter: Helt Long
>Priority: Major
> Attachments: stuck log.png, webui stuck.png
>
>
> When use spark3 submit a spark job to yarn cluster, I get a problem. Once in
> a while, driver can't distribute any tasks to any executors, and the stage
> will stuck , the total spark job will stuck. Check driver log, I found
> NullPointerException. It's like a netty problem, I can confirm this problem
> only exist in spark3, because I use spark2 never happend.
>
> {code:java}
> // Error message
> 21/06/28 14:42:43 INFO TaskSetManager: Starting task 2592.0 in stage 1.0 (TID
> 3494) (worker39.hadoop, executor 84, partition 2592, RACK_LOCAL, 5006 bytes)
> taskResourceAssignments Map()
> 21/06/28 14:42:43 INFO TaskSetManager: Finished task 4155.0 in stage 1.0 (TID
> 3367) in 36670 ms on worker39.hadoop (executor 84) (3278/4249)
> 21/06/28 14:42:43 INFO TaskSetManager: Finished task 2283.0 in stage 1.0 (TID
> 3422) in 22371 ms on worker15.hadoop (executor 109) (3279/4249)
> 21/06/28 14:42:43 ERROR Inbox: Ignoring error
> java.lang.NullPointerException
> at java.lang.String.length(String.java:623)
> at
> java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:420)
> at java.lang.StringBuilder.append(StringBuilder.java:136)
> at
> org.apache.spark.scheduler.TaskSetManager.$anonfun$resourceOffer$5(TaskSetManager.scala:483)
> at org.apache.spark.internal.Logging.logInfo(Logging.scala:57)
> at org.apache.spark.internal.Logging.logInfo$(Logging.scala:56)
> at
> org.apache.spark.scheduler.TaskSetManager.logInfo(TaskSetManager.scala:54)
> at
> org.apache.spark.scheduler.TaskSetManager.$anonfun$resourceOffer$2(TaskSetManager.scala:484)
> at scala.Option.map(Option.scala:230)
> at
> org.apache.spark.scheduler.TaskSetManager.resourceOffer(TaskSetManager.scala:444)
> at
> org.apache.spark.scheduler.TaskSchedulerImpl.$anonfun$resourceOfferSingleTaskSet$2(TaskSchedulerImpl.scala:397)
> at
> org.apache.spark.scheduler.TaskSchedulerImpl.$anonfun$resourceOfferSingleTaskSet$2$adapted(TaskSchedulerImpl.scala:392)
> at scala.Option.foreach(Option.scala:407)
> at
> org.apache.spark.scheduler.TaskSchedulerImpl.$anonfun$resourceOfferSingleTaskSet$1(TaskSchedulerImpl.scala:392)
> at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:158)
> at
> org.apache.spark.scheduler.TaskSchedulerImpl.resourceOfferSingleTaskSet(TaskSchedulerImpl.scala:383)
> at
> org.apache.spark.scheduler.TaskSchedulerImpl.$anonfun$resourceOffers$20(TaskSchedulerImpl.scala:581)
> at
> org.apache.spark.scheduler.TaskSchedulerImpl.$anonfun$resourceOffers$20$adapted(TaskSchedulerImpl.scala:576)
> at
> scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)
> at
> scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33)
> at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:198)
> at
> org.apache.spark.scheduler.TaskSchedulerImpl.$anonfun$resourceOffers$16(TaskSchedulerImpl.scala:576)
> at
> org.apache.spark.scheduler.TaskSchedulerImpl.$anonfun$resourceOffers$16$adapted(TaskSchedulerImpl.scala:547)
> at
> scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
> at
> scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
> at
> org.apache.spark.scheduler.TaskSchedulerImpl.resourceOffers(TaskSchedulerImpl.scala:547)
> at
> org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend$DriverEndpoint.$anonfun$makeOffers$5(CoarseGrainedSchedulerBackend.scala:340)
> at
> org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend.org$apache$spark$scheduler$cluster$CoarseGrainedSchedulerBackend$$withLock(CoarseGrainedSchedulerBackend.scala:904)
> at
> org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend$DriverEndpoint.org$apache$spark$scheduler$cluster$CoarseGrainedSchedulerBackend$DriverEndpoint$$makeOffers(CoarseGrainedSchedulerBackend.scala:332)
> at
> org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend$DriverEndpoint$$anonfun$receive$1.applyOrElse(CoarseG
[jira] [Commented] (SPARK-35802) Error loading the stages/stage/ page in spark UI
[
https://issues.apache.org/jira/browse/SPARK-35802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17370491#comment-17370491
]
Helt Long commented on SPARK-35802:
---
[~sarutak]
I used CDH 5.7.1, and used Spark on Yarn cluster mode, this problem happend all
the time. When I google thhis problem, I found the same problem on
stackoverflow, so I move the problem there
[Error loading the stages/stage/ page in spark UI - Stack
Overflow|https://stackoverflow.com/questions/64265444/error-loading-the-stages-stage-id-page-in-spark-ui]
I can 100% recurrence problem
I add some message about env like blow:
CDH 5.7.1： Hadoop 2.6.5
Spark on yarn cluster mode
> Error loading the stages/stage/ page in spark UI
>
>
> Key: SPARK-35802
> URL: https://issues.apache.org/jira/browse/SPARK-35802
> Project: Spark
> Issue Type: Bug
> Components: Web UI
>Affects Versions: 3.0.0, 3.0.1, 3.1.1, 3.1.2
> Environment: CDH 5.7.1： Hadoop 2.6.5
> Spark on yarn cluster mode
>Reporter: Helt Long
>Priority: Major
> Attachments: spark3.1.2-request-20210628093538.png,
> spark3.1.2-stage-20210628093549.png, spark3.1.2-webui-20210628093559.png
>
>
> I try to load the sparkUI page for a specific stage, I get the following
> error:
> {quote}Unable to connect to the server. Looks like the Spark application must
> have ended. Please Switch to the history UI.
> {quote}
> Obviously the server is still alive and process new messages.
> Looking at the network tab shows one of the requests fails:
>
> {{curl
> 'http://:8080/proxy/app-20201008130147-0001/api/v1/applications/app-20201008130147-0001/stages/11/0/taskTable'
>
>
>
> Error 500 Request failed.
>
> HTTP ERROR 500
> Problem accessing
> /api/v1/applications/app-20201008130147-0001/stages/11/0/taskTable. Reason:
> Request failed. href="http://eclipse.org/jetty";>Powered by Jetty:// 9.4.z-SNAPSHOT
>
> }}
> requests to any other object that I've tested seem to work, for example
>
> {{curl
> 'http://:8080/proxy/app-20201008130147-0001/api/v1/applications/app-20201008130147-0001/stages/11/0/taskSummary'}}
>
> The exception is:
> {{/api/v1/applications/app-20201008130147-0001/stages/11/0/taskTable
> javax.servlet.ServletException: java.lang.NullPointerException
> at
> org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:410)
> at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346)
> at
> org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:366)
> at
> org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:319)
> at
> org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)
> at
> org.sparkproject.jetty.servlet.ServletHolder.handle(ServletHolder.java:873)
> at
> org.sparkproject.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1623)
> at
> org.apache.spark.ui.HttpSecurityFilter.doFilter(HttpSecurityFilter.scala:95)
> at
> org.sparkproject.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
> at
> org.sparkproject.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
> at
> org.sparkproject.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
> at
> org.sparkproject.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)
> at
> org.sparkproject.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
> at
> org.sparkproject.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
> at
> org.sparkproject.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
> at
> org.sparkproject.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)
> at
> org.sparkproject.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
> at
> org.sparkproject.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:753)
> at
> org.sparkproject.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220)
> at
> org.sparkproject.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
> at org.sparkproject.jetty.server.Server.handle(Server.java:505)
> at org.sparkproject.jetty.server.HttpChannel.handle(HttpChannel.java:370)
> at
> org.sparkproject.jetty.server.HttpConnection.onFillable(HttpConnection.java:267)
> at
> org.sparkproject.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
> at org.sparkproject.jetty.io.FillInterest.fillable(FillInterest.java:103)
> at org.sparkproject.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
> at
> org.sparkproject.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
> at
> org.sparkproject.jetty.util.thread.strategy.EatWhatYo
[jira] [Comment Edited] (SPARK-35802) Error loading the stages/stage/ page in spark UI
[
https://issues.apache.org/jira/browse/SPARK-35802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17370491#comment-17370491
]
Helt Long edited comment on SPARK-35802 at 6/28/21, 8:33 AM:
-
[~sarutak]
I used CDH 5.7.1, and used Spark on Yarn cluster mode, this problem happend all
the time. When I google this problem, I found the same problem on
stackoverflow, so I move the problem there
[Error loading the stages/stage/ page in spark UI - Stack
Overflow|https://stackoverflow.com/questions/64265444/error-loading-the-stages-stage-id-page-in-spark-ui]
I can 100% recurrence problem
I add some message about env like blow:
CDH 5.7.1： Hadoop 2.6.5
Spark on yarn cluster mode
was (Author: heltman):
[~sarutak]
I used CDH 5.7.1, and used Spark on Yarn cluster mode, this problem happend all
the time. When I google thhis problem, I found the same problem on
stackoverflow, so I move the problem there
[Error loading the stages/stage/ page in spark UI - Stack
Overflow|https://stackoverflow.com/questions/64265444/error-loading-the-stages-stage-id-page-in-spark-ui]
I can 100% recurrence problem
I add some message about env like blow:
CDH 5.7.1： Hadoop 2.6.5
Spark on yarn cluster mode
> Error loading the stages/stage/ page in spark UI
>
>
> Key: SPARK-35802
> URL: https://issues.apache.org/jira/browse/SPARK-35802
> Project: Spark
> Issue Type: Bug
> Components: Web UI
>Affects Versions: 3.0.0, 3.0.1, 3.1.1, 3.1.2
> Environment: CDH 5.7.1： Hadoop 2.6.5
> Spark on yarn cluster mode
>Reporter: Helt Long
>Priority: Major
> Attachments: spark3.1.2-request-20210628093538.png,
> spark3.1.2-stage-20210628093549.png, spark3.1.2-webui-20210628093559.png
>
>
> I try to load the sparkUI page for a specific stage, I get the following
> error:
> {quote}Unable to connect to the server. Looks like the Spark application must
> have ended. Please Switch to the history UI.
> {quote}
> Obviously the server is still alive and process new messages.
> Looking at the network tab shows one of the requests fails:
>
> {{curl
> 'http://:8080/proxy/app-20201008130147-0001/api/v1/applications/app-20201008130147-0001/stages/11/0/taskTable'
>
>
>
> Error 500 Request failed.
>
> HTTP ERROR 500
> Problem accessing
> /api/v1/applications/app-20201008130147-0001/stages/11/0/taskTable. Reason:
> Request failed. href="http://eclipse.org/jetty";>Powered by Jetty:// 9.4.z-SNAPSHOT
>
> }}
> requests to any other object that I've tested seem to work, for example
>
> {{curl
> 'http://:8080/proxy/app-20201008130147-0001/api/v1/applications/app-20201008130147-0001/stages/11/0/taskSummary'}}
>
> The exception is:
> {{/api/v1/applications/app-20201008130147-0001/stages/11/0/taskTable
> javax.servlet.ServletException: java.lang.NullPointerException
> at
> org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:410)
> at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346)
> at
> org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:366)
> at
> org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:319)
> at
> org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)
> at
> org.sparkproject.jetty.servlet.ServletHolder.handle(ServletHolder.java:873)
> at
> org.sparkproject.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1623)
> at
> org.apache.spark.ui.HttpSecurityFilter.doFilter(HttpSecurityFilter.scala:95)
> at
> org.sparkproject.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
> at
> org.sparkproject.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
> at
> org.sparkproject.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
> at
> org.sparkproject.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)
> at
> org.sparkproject.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
> at
> org.sparkproject.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
> at
> org.sparkproject.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
> at
> org.sparkproject.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)
> at
> org.sparkproject.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
> at
> org.sparkproject.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:753)
> at
> org.sparkproject.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220)
> at
> org.sparkproject.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
> at org.sparkproject.jetty.server.Server.handle(Server.java:505)
> at org.sparkproject
[jira] [Updated] (SPARK-35802) Error loading the stages/stage/ page in spark UI
[
https://issues.apache.org/jira/browse/SPARK-35802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Helt Long updated SPARK-35802:
--
Description:
I try to load the sparkUI page for a specific stage, I get the following error:
{quote}Unable to connect to the server. Looks like the Spark application must
have ended. Please Switch to the history UI.
{quote}
Obviously the server is still alive and process new messages.
Looking at the network tab shows one of the requests fails:
{{curl
'http://:8080/proxy/app-20201008130147-0001/api/v1/applications/app-20201008130147-0001/stages/11/0/taskTable'
Error 500 Request failed.
HTTP ERROR 500
Problem accessing
/api/v1/applications/app-20201008130147-0001/stages/11/0/taskTable. Reason:
Request failed.http://eclipse.org/jetty";>Powered
by Jetty:// 9.4.z-SNAPSHOT
}}
requests to any other object that I've tested seem to work, for example
{{curl
'http://:8080/proxy/app-20201008130147-0001/api/v1/applications/app-20201008130147-0001/stages/11/0/taskSummary'}}
The exception is:
{{/api/v1/applications/app-20201008130147-0001/stages/11/0/taskTable
javax.servlet.ServletException: java.lang.NullPointerException
at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:410)
at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346)
at
org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:366)
at
org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:319)
at
org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)
at org.sparkproject.jetty.servlet.ServletHolder.handle(ServletHolder.java:873)
at
org.sparkproject.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1623)
at org.apache.spark.ui.HttpSecurityFilter.doFilter(HttpSecurityFilter.scala:95)
at
org.sparkproject.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
at
org.sparkproject.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
at
org.sparkproject.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
at
org.sparkproject.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)
at
org.sparkproject.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
at
org.sparkproject.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
at
org.sparkproject.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
at
org.sparkproject.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)
at
org.sparkproject.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
at
org.sparkproject.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:753)
at
org.sparkproject.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220)
at
org.sparkproject.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at org.sparkproject.jetty.server.Server.handle(Server.java:505)
at org.sparkproject.jetty.server.HttpChannel.handle(HttpChannel.java:370)
at
org.sparkproject.jetty.server.HttpConnection.onFillable(HttpConnection.java:267)
at
org.sparkproject.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
at org.sparkproject.jetty.io.FillInterest.fillable(FillInterest.java:103)
at org.sparkproject.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
at
org.sparkproject.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
at
org.sparkproject.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
at
org.sparkproject.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at
org.sparkproject.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
at
org.sparkproject.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
at
org.sparkproject.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:698)
at
org.sparkproject.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:804)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NullPointerException
at
org.apache.spark.status.api.v1.StagesResource.$anonfun$doPagination$1(StagesResource.scala:175)
at
org.apache.spark.status.api.v1.BaseAppResource.$anonfun$withUI$1(ApiRootResource.scala:140)
at org.apache.spark.ui.SparkUI.withSparkUI(SparkUI.scala:107)
at
org.apache.spark.status.api.v1.BaseAppResource.withUI(ApiRootResource.scala:135)
at
org.apache.spark.status.api.v1.BaseAppResource.withUI$(ApiRootResource.scala:133)
at
org.apache.spark.status.api.v1.StagesResource.withUI(StagesResource.scala:28)
at
org.apache.spark.status.api.v1.StagesResource.doPagination(StagesResource.scala:174)
at
org.apache.spark.status.api.v1.StagesResource.$anonfun$taskTable$1(StagesResource.scala:129)
at
[jira] [Created] (SPARK-35914) Driver can't distribute task to executor because NullPointerException
Helt Long created SPARK-35914:
-
Summary: Driver can't distribute task to executor because
NullPointerException
Key: SPARK-35914
URL: https://issues.apache.org/jira/browse/SPARK-35914
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 3.1.2, 3.1.1, 3.0.1
Environment: CDH 5.7.1: Hadoop 2.6.5
Spark 3.0.1, 3.1.1, 3.1.2
Reporter: Helt Long
When use spark3 submit a spark job to yarn cluster, I get a problem. Once in a
while, driver can't distribute any tasks to any executors, and the stage will
stuck , the total spark job will stuck. Check driver log, I found
NullPointerException. It's like a netty problem, I can confirm this problem
only exist in spark3, because I use spark2 never happend.
{code:java}
// Error message
21/06/28 14:42:43 INFO TaskSetManager: Starting task 2592.0 in stage 1.0 (TID
3494) (worker39.hadoop, executor 84, partition 2592, RACK_LOCAL, 5006 bytes)
taskResourceAssignments Map()
21/06/28 14:42:43 INFO TaskSetManager: Finished task 4155.0 in stage 1.0 (TID
3367) in 36670 ms on worker39.hadoop (executor 84) (3278/4249)
21/06/28 14:42:43 INFO TaskSetManager: Finished task 2283.0 in stage 1.0 (TID
3422) in 22371 ms on worker15.hadoop (executor 109) (3279/4249)
21/06/28 14:42:43 ERROR Inbox: Ignoring error
java.lang.NullPointerException
at java.lang.String.length(String.java:623)
at
java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:420)
at java.lang.StringBuilder.append(StringBuilder.java:136)
at
org.apache.spark.scheduler.TaskSetManager.$anonfun$resourceOffer$5(TaskSetManager.scala:483)
at org.apache.spark.internal.Logging.logInfo(Logging.scala:57)
at org.apache.spark.internal.Logging.logInfo$(Logging.scala:56)
at
org.apache.spark.scheduler.TaskSetManager.logInfo(TaskSetManager.scala:54)
at
org.apache.spark.scheduler.TaskSetManager.$anonfun$resourceOffer$2(TaskSetManager.scala:484)
at scala.Option.map(Option.scala:230)
at
org.apache.spark.scheduler.TaskSetManager.resourceOffer(TaskSetManager.scala:444)
at
org.apache.spark.scheduler.TaskSchedulerImpl.$anonfun$resourceOfferSingleTaskSet$2(TaskSchedulerImpl.scala:397)
at
org.apache.spark.scheduler.TaskSchedulerImpl.$anonfun$resourceOfferSingleTaskSet$2$adapted(TaskSchedulerImpl.scala:392)
at scala.Option.foreach(Option.scala:407)
at
org.apache.spark.scheduler.TaskSchedulerImpl.$anonfun$resourceOfferSingleTaskSet$1(TaskSchedulerImpl.scala:392)
at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:158)
at
org.apache.spark.scheduler.TaskSchedulerImpl.resourceOfferSingleTaskSet(TaskSchedulerImpl.scala:383)
at
org.apache.spark.scheduler.TaskSchedulerImpl.$anonfun$resourceOffers$20(TaskSchedulerImpl.scala:581)
at
org.apache.spark.scheduler.TaskSchedulerImpl.$anonfun$resourceOffers$20$adapted(TaskSchedulerImpl.scala:576)
at
scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)
at
scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:198)
at
org.apache.spark.scheduler.TaskSchedulerImpl.$anonfun$resourceOffers$16(TaskSchedulerImpl.scala:576)
at
org.apache.spark.scheduler.TaskSchedulerImpl.$anonfun$resourceOffers$16$adapted(TaskSchedulerImpl.scala:547)
at
scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
at
scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
at
org.apache.spark.scheduler.TaskSchedulerImpl.resourceOffers(TaskSchedulerImpl.scala:547)
at
org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend$DriverEndpoint.$anonfun$makeOffers$5(CoarseGrainedSchedulerBackend.scala:340)
at
org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend.org$apache$spark$scheduler$cluster$CoarseGrainedSchedulerBackend$$withLock(CoarseGrainedSchedulerBackend.scala:904)
at
org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend$DriverEndpoint.org$apache$spark$scheduler$cluster$CoarseGrainedSchedulerBackend$DriverEndpoint$$makeOffers(CoarseGrainedSchedulerBackend.scala:332)
at
org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend$DriverEndpoint$$anonfun$receive$1.applyOrElse(CoarseGrainedSchedulerBackend.scala:157)
at org.apache.spark.rpc.netty.Inbox.$anonfun$process$1(Inbox.scala:115)
at org.apache.spark.rpc.netty.Inbox.safelyCall(Inbox.scala:213)
at org.apache.spark.rpc.netty.Inbox.process(Inbox.scala:100)
at
org.apache.spark.rpc.netty.MessageLoop.org$apache$spark$rpc$netty$MessageLoop$$receiveLoop(MessageLoop.scala:75)
[jira] [Updated] (SPARK-35802) Error loading the stages/stage/ page in spark UI
[
https://issues.apache.org/jira/browse/SPARK-35802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Helt Long updated SPARK-35802:
--
Attachment: spark3.1.2-webui-20210628093559.png
> Error loading the stages/stage/ page in spark UI
>
>
> Key: SPARK-35802
> URL: https://issues.apache.org/jira/browse/SPARK-35802
> Project: Spark
> Issue Type: Bug
> Components: Web UI
>Affects Versions: 3.0.0, 3.0.1, 3.1.1, 3.1.2
>Reporter: Helt Long
>Priority: Major
> Attachments: spark3.1.2-request-20210628093538.png,
> spark3.1.2-stage-20210628093549.png, spark3.1.2-webui-20210628093559.png
>
>
> I try to load the sparkUI page for a specific stage, I get the following
> error:
> {quote}Unable to connect to the server. Looks like the Spark application must
> have ended. Please Switch to the history UI.
> {quote}
> Obviously the server is still alive and process new messages.
> Looking at the network tab shows one of the requests fails:
>
> {{curl
> 'http://:8080/proxy/app-20201008130147-0001/api/v1/applications/app-20201008130147-0001/stages/11/0/taskTable'
>
>
>
> Error 500 Request failed.
>
> HTTP ERROR 500
> Problem accessing
> /api/v1/applications/app-20201008130147-0001/stages/11/0/taskTable. Reason:
> Request failed. href="http://eclipse.org/jetty";>Powered by Jetty:// 9.4.z-SNAPSHOT
>
> }}
> requests to any other object that I've tested seem to work, for example
>
> {{curl
> 'http://:8080/proxy/app-20201008130147-0001/api/v1/applications/app-20201008130147-0001/stages/11/0/taskSummary'}}
>
> The exception is:
> {{/api/v1/applications/app-20201008130147-0001/stages/11/0/taskTable
> javax.servlet.ServletException: java.lang.NullPointerException
> at
> org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:410)
> at
> org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346)
> at
> org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:366)
> at
> org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:319)
> at
> org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)
> at
> org.sparkproject.jetty.servlet.ServletHolder.handle(ServletHolder.java:873)
> at
> org.sparkproject.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1623)
> at
> org.apache.spark.ui.HttpSecurityFilter.doFilter(HttpSecurityFilter.scala:95)
> at
> org.sparkproject.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
> at
> org.sparkproject.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
> at
> org.sparkproject.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
> at
> org.sparkproject.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)
> at
> org.sparkproject.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
> at
> org.sparkproject.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
> at
> org.sparkproject.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
> at
> org.sparkproject.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)
> at
> org.sparkproject.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
> at
> org.sparkproject.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:753)
> at
> org.sparkproject.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220)
> at
> org.sparkproject.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
> at org.sparkproject.jetty.server.Server.handle(Server.java:505)
> at org.sparkproject.jetty.server.HttpChannel.handle(HttpChannel.java:370)
> at
> org.sparkproject.jetty.server.HttpConnection.onFillable(HttpConnection.java:267)
> at
> org.sparkproject.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
> at org.sparkproject.jetty.io.FillInterest.fillable(FillInterest.java:103)
> at
> org.sparkproject.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
> at
> org.sparkproject.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
> at
> org.sparkproject.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
> at
> org.sparkproject.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
> at
> org.sparkproject.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
> at
> org.sparkproject.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
> at
> org.sparkproject.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:698)
> at
> org.sparkproje
[jira] [Updated] (SPARK-35802) Error loading the stages/stage/ page in spark UI
[
https://issues.apache.org/jira/browse/SPARK-35802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Helt Long updated SPARK-35802:
--
Attachment: spark3.1.2-request-20210628093538.png
> Error loading the stages/stage/ page in spark UI
>
>
> Key: SPARK-35802
> URL: https://issues.apache.org/jira/browse/SPARK-35802
> Project: Spark
> Issue Type: Bug
> Components: Web UI
>Affects Versions: 3.0.0, 3.0.1, 3.1.1, 3.1.2
>Reporter: Helt Long
>Priority: Major
> Attachments: spark3.1.2-request-20210628093538.png,
> spark3.1.2-stage-20210628093549.png, spark3.1.2-webui-20210628093559.png
>
>
> I try to load the sparkUI page for a specific stage, I get the following
> error:
> {quote}Unable to connect to the server. Looks like the Spark application must
> have ended. Please Switch to the history UI.
> {quote}
> Obviously the server is still alive and process new messages.
> Looking at the network tab shows one of the requests fails:
>
> {{curl
> 'http://:8080/proxy/app-20201008130147-0001/api/v1/applications/app-20201008130147-0001/stages/11/0/taskTable'
>
>
>
> Error 500 Request failed.
>
> HTTP ERROR 500
> Problem accessing
> /api/v1/applications/app-20201008130147-0001/stages/11/0/taskTable. Reason:
> Request failed. href="http://eclipse.org/jetty";>Powered by Jetty:// 9.4.z-SNAPSHOT
>
> }}
> requests to any other object that I've tested seem to work, for example
>
> {{curl
> 'http://:8080/proxy/app-20201008130147-0001/api/v1/applications/app-20201008130147-0001/stages/11/0/taskSummary'}}
>
> The exception is:
> {{/api/v1/applications/app-20201008130147-0001/stages/11/0/taskTable
> javax.servlet.ServletException: java.lang.NullPointerException
> at
> org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:410)
> at
> org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346)
> at
> org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:366)
> at
> org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:319)
> at
> org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)
> at
> org.sparkproject.jetty.servlet.ServletHolder.handle(ServletHolder.java:873)
> at
> org.sparkproject.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1623)
> at
> org.apache.spark.ui.HttpSecurityFilter.doFilter(HttpSecurityFilter.scala:95)
> at
> org.sparkproject.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
> at
> org.sparkproject.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
> at
> org.sparkproject.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
> at
> org.sparkproject.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)
> at
> org.sparkproject.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
> at
> org.sparkproject.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
> at
> org.sparkproject.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
> at
> org.sparkproject.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)
> at
> org.sparkproject.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
> at
> org.sparkproject.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:753)
> at
> org.sparkproject.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220)
> at
> org.sparkproject.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
> at org.sparkproject.jetty.server.Server.handle(Server.java:505)
> at org.sparkproject.jetty.server.HttpChannel.handle(HttpChannel.java:370)
> at
> org.sparkproject.jetty.server.HttpConnection.onFillable(HttpConnection.java:267)
> at
> org.sparkproject.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
> at org.sparkproject.jetty.io.FillInterest.fillable(FillInterest.java:103)
> at
> org.sparkproject.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
> at
> org.sparkproject.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
> at
> org.sparkproject.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
> at
> org.sparkproject.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
> at
> org.sparkproject.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
> at
> org.sparkproject.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
> at
> org.sparkproject.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:698)
> at
> org.sparkpro
[jira] [Updated] (SPARK-35802) Error loading the stages/stage/ page in spark UI
[
https://issues.apache.org/jira/browse/SPARK-35802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Helt Long updated SPARK-35802:
--
Attachment: spark3.1.2-stage-20210628093549.png
> Error loading the stages/stage/ page in spark UI
>
>
> Key: SPARK-35802
> URL: https://issues.apache.org/jira/browse/SPARK-35802
> Project: Spark
> Issue Type: Bug
> Components: Web UI
>Affects Versions: 3.0.0, 3.0.1, 3.1.1, 3.1.2
>Reporter: Helt Long
>Priority: Major
> Attachments: spark3.1.2-request-20210628093538.png,
> spark3.1.2-stage-20210628093549.png, spark3.1.2-webui-20210628093559.png
>
>
> I try to load the sparkUI page for a specific stage, I get the following
> error:
> {quote}Unable to connect to the server. Looks like the Spark application must
> have ended. Please Switch to the history UI.
> {quote}
> Obviously the server is still alive and process new messages.
> Looking at the network tab shows one of the requests fails:
>
> {{curl
> 'http://:8080/proxy/app-20201008130147-0001/api/v1/applications/app-20201008130147-0001/stages/11/0/taskTable'
>
>
>
> Error 500 Request failed.
>
> HTTP ERROR 500
> Problem accessing
> /api/v1/applications/app-20201008130147-0001/stages/11/0/taskTable. Reason:
> Request failed. href="http://eclipse.org/jetty";>Powered by Jetty:// 9.4.z-SNAPSHOT
>
> }}
> requests to any other object that I've tested seem to work, for example
>
> {{curl
> 'http://:8080/proxy/app-20201008130147-0001/api/v1/applications/app-20201008130147-0001/stages/11/0/taskSummary'}}
>
> The exception is:
> {{/api/v1/applications/app-20201008130147-0001/stages/11/0/taskTable
> javax.servlet.ServletException: java.lang.NullPointerException
> at
> org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:410)
> at
> org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346)
> at
> org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:366)
> at
> org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:319)
> at
> org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)
> at
> org.sparkproject.jetty.servlet.ServletHolder.handle(ServletHolder.java:873)
> at
> org.sparkproject.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1623)
> at
> org.apache.spark.ui.HttpSecurityFilter.doFilter(HttpSecurityFilter.scala:95)
> at
> org.sparkproject.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
> at
> org.sparkproject.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
> at
> org.sparkproject.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
> at
> org.sparkproject.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)
> at
> org.sparkproject.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
> at
> org.sparkproject.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
> at
> org.sparkproject.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
> at
> org.sparkproject.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)
> at
> org.sparkproject.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
> at
> org.sparkproject.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:753)
> at
> org.sparkproject.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220)
> at
> org.sparkproject.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
> at org.sparkproject.jetty.server.Server.handle(Server.java:505)
> at org.sparkproject.jetty.server.HttpChannel.handle(HttpChannel.java:370)
> at
> org.sparkproject.jetty.server.HttpConnection.onFillable(HttpConnection.java:267)
> at
> org.sparkproject.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
> at org.sparkproject.jetty.io.FillInterest.fillable(FillInterest.java:103)
> at
> org.sparkproject.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
> at
> org.sparkproject.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
> at
> org.sparkproject.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
> at
> org.sparkproject.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
> at
> org.sparkproject.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
> at
> org.sparkproject.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
> at
> org.sparkproject.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:698)
> at
> org.sparkproje
[jira] [Commented] (SPARK-35802) Error loading the stages/stage/ page in spark UI
[
https://issues.apache.org/jira/browse/SPARK-35802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17370372#comment-17370372
]
Helt Long commented on SPARK-35802:
---
I'am sorry for my unclear question. But it is the webUI's request, not by
myself. I add some picture to desc it. [~sarutak], [~Gengliang.Wang]
> Error loading the stages/stage/ page in spark UI
>
>
> Key: SPARK-35802
> URL: https://issues.apache.org/jira/browse/SPARK-35802
> Project: Spark
> Issue Type: Bug
> Components: Web UI
>Affects Versions: 3.0.0, 3.0.1, 3.1.1, 3.1.2
>Reporter: Helt Long
>Priority: Major
>
> I try to load the sparkUI page for a specific stage, I get the following
> error:
> {quote}Unable to connect to the server. Looks like the Spark application must
> have ended. Please Switch to the history UI.
> {quote}
> Obviously the server is still alive and process new messages.
> Looking at the network tab shows one of the requests fails:
>
> {{curl
> 'http://:8080/proxy/app-20201008130147-0001/api/v1/applications/app-20201008130147-0001/stages/11/0/taskTable'
>
>
>
> Error 500 Request failed.
>
> HTTP ERROR 500
> Problem accessing
> /api/v1/applications/app-20201008130147-0001/stages/11/0/taskTable. Reason:
> Request failed. href="http://eclipse.org/jetty";>Powered by Jetty:// 9.4.z-SNAPSHOT
>
> }}
> requests to any other object that I've tested seem to work, for example
>
> {{curl
> 'http://:8080/proxy/app-20201008130147-0001/api/v1/applications/app-20201008130147-0001/stages/11/0/taskSummary'}}
>
> The exception is:
> {{/api/v1/applications/app-20201008130147-0001/stages/11/0/taskTable
> javax.servlet.ServletException: java.lang.NullPointerException
> at
> org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:410)
> at
> org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346)
> at
> org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:366)
> at
> org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:319)
> at
> org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)
> at
> org.sparkproject.jetty.servlet.ServletHolder.handle(ServletHolder.java:873)
> at
> org.sparkproject.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1623)
> at
> org.apache.spark.ui.HttpSecurityFilter.doFilter(HttpSecurityFilter.scala:95)
> at
> org.sparkproject.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
> at
> org.sparkproject.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
> at
> org.sparkproject.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
> at
> org.sparkproject.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)
> at
> org.sparkproject.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
> at
> org.sparkproject.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
> at
> org.sparkproject.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
> at
> org.sparkproject.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)
> at
> org.sparkproject.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
> at
> org.sparkproject.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:753)
> at
> org.sparkproject.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220)
> at
> org.sparkproject.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
> at org.sparkproject.jetty.server.Server.handle(Server.java:505)
> at org.sparkproject.jetty.server.HttpChannel.handle(HttpChannel.java:370)
> at
> org.sparkproject.jetty.server.HttpConnection.onFillable(HttpConnection.java:267)
> at
> org.sparkproject.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
> at org.sparkproject.jetty.io.FillInterest.fillable(FillInterest.java:103)
> at
> org.sparkproject.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
> at
> org.sparkproject.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
> at
> org.sparkproject.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
> at
> org.sparkproject.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
> at
> org.sparkproject.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
> at
> org.sparkproject.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
> at
> org.sparkproject.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:698)
> at
> org.
[jira] [Created] (SPARK-35802) Error loading the stages/stage/ page in spark UI
Helt Long created SPARK-35802:
-
Summary: Error loading the stages/stage/ page in spark UI
Key: SPARK-35802
URL: https://issues.apache.org/jira/browse/SPARK-35802
Project: Spark
Issue Type: Bug
Components: Web UI
Affects Versions: 3.1.2, 3.1.1, 3.0.1, 3.0.0
Reporter: Helt Long
I try to load the sparkUI page for a specific stage, I get the following error:
{quote}Unable to connect to the server. Looks like the Spark application must
have ended. Please Switch to the history UI.
{quote}
Obviously the server is still alive and process new messages.
Looking at the network tab shows one of the requests fails:
{{curl
'http://:8080/proxy/app-20201008130147-0001/api/v1/applications/app-20201008130147-0001/stages/11/0/taskTable'
Error 500 Request failed.
HTTP ERROR 500
Problem accessing
/api/v1/applications/app-20201008130147-0001/stages/11/0/taskTable. Reason:
Request failed.http://eclipse.org/jetty";>Powered by Jetty:// 9.4.z-SNAPSHOT
}}
requests to any other object that I've tested seem to work, for example
{{curl
'http://:8080/proxy/app-20201008130147-0001/api/v1/applications/app-20201008130147-0001/stages/11/0/taskSummary'}}
The exception is:
{{/api/v1/applications/app-20201008130147-0001/stages/11/0/taskTable
javax.servlet.ServletException: java.lang.NullPointerException
at
org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:410)
at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346)
at
org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:366)
at
org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:319)
at
org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)
at
org.sparkproject.jetty.servlet.ServletHolder.handle(ServletHolder.java:873)
at
org.sparkproject.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1623)
at
org.apache.spark.ui.HttpSecurityFilter.doFilter(HttpSecurityFilter.scala:95)
at
org.sparkproject.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
at
org.sparkproject.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
at
org.sparkproject.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
at
org.sparkproject.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)
at
org.sparkproject.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
at
org.sparkproject.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
at
org.sparkproject.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
at
org.sparkproject.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)
at
org.sparkproject.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
at
org.sparkproject.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:753)
at
org.sparkproject.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220)
at
org.sparkproject.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at org.sparkproject.jetty.server.Server.handle(Server.java:505)
at org.sparkproject.jetty.server.HttpChannel.handle(HttpChannel.java:370)
at
org.sparkproject.jetty.server.HttpConnection.onFillable(HttpConnection.java:267)
at
org.sparkproject.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
at org.sparkproject.jetty.io.FillInterest.fillable(FillInterest.java:103)
at org.sparkproject.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
at
org.sparkproject.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
at
org.sparkproject.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
at
org.sparkproject.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at
org.sparkproject.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
at
org.sparkproject.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
at
org.sparkproject.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:698)
at
org.sparkproject.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:804)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NullPointerException
at
org.apache.spark.status.api.v1.StagesResource.$anonfun$doPagination$1(StagesResource.scala:175)
at
org.apache.spark.status.api.v1.BaseAppResource.$anonfun$withUI$1(ApiRootResource.scala:140)
at org.apache.spark.ui.SparkUI.withSparkUI(SparkUI.scala:107)
at
org.apache.spark.status.api.v1.BaseAppResource.withUI(ApiRootResource.scala:135)
at
org.apache.spark.status.api.v1.BaseAppResour
