[jira] [Commented] (SPARK-27122) YARN test failures in Java 9+
[ https://issues.apache.org/jira/browse/SPARK-27122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16930019#comment-16930019 ] Dongjoon Hyun commented on SPARK-27122: --- This is backported to `branch-2.4` via https://github.com/apache/spark/pull/25793 . Although this helps JDK11, but this is a general issue with JDK8+. So, I'll remove this from this JDK11 umbrella like SPARK-29087 . > YARN test failures in Java 9+ > - > > Key: SPARK-27122 > URL: https://issues.apache.org/jira/browse/SPARK-27122 > Project: Spark > Issue Type: Sub-task > Components: YARN >Affects Versions: 3.0.0 >Reporter: Sean Owen >Assignee: Ajith S >Priority: Major > Fix For: 2.4.5, 3.0.0 > > Attachments: image-2019-03-14-09-34-20-592.png, > image-2019-03-14-09-35-23-046.png > > > Currently on Java 11: > {code} > YarnSchedulerBackendSuite: > - RequestExecutors reflects node blacklist and is serializable > - Respect user filters when adding AM IP filter *** FAILED *** > java.lang.ClassCastException: > org.spark_project.jetty.servlet.ServletContextHandler cannot be cast to > org.eclipse.jetty.servlet.ServletContextHandler > at > scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:237) > at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62) > at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55) > at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49) > at scala.collection.TraversableLike.map(TraversableLike.scala:237) > at scala.collection.TraversableLike.map$(TraversableLike.scala:230) > at scala.collection.AbstractTraversable.map(Traversable.scala:108) > at > org.apache.spark.scheduler.cluster.YarnSchedulerBackend.$anonfun$addWebUIFilter$2(YarnSchedulerBackend.scala:183) > at > org.apache.spark.scheduler.cluster.YarnSchedulerBackend.$anonfun$addWebUIFilter$2$adapted(YarnSchedulerBackend.scala:174) > at scala.Option.foreach(Option.scala:274) > ... > {code} > This looks like a classpath issue, probably ultimately related to the same > classloader issues in https://issues.apache.org/jira/browse/SPARK-26839 -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-27122) YARN test failures in Java 9+
[ https://issues.apache.org/jira/browse/SPARK-27122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16792355#comment-16792355 ] Ajith S commented on SPARK-27122: - ping [~srowen] [~dongjoon] [~Gengliang.Wang] > YARN test failures in Java 9+ > - > > Key: SPARK-27122 > URL: https://issues.apache.org/jira/browse/SPARK-27122 > Project: Spark > Issue Type: Sub-task > Components: YARN >Affects Versions: 3.0.0 >Reporter: Sean Owen >Priority: Major > Attachments: image-2019-03-14-09-34-20-592.png, > image-2019-03-14-09-35-23-046.png > > > Currently on Java 11: > {code} > YarnSchedulerBackendSuite: > - RequestExecutors reflects node blacklist and is serializable > - Respect user filters when adding AM IP filter *** FAILED *** > java.lang.ClassCastException: > org.spark_project.jetty.servlet.ServletContextHandler cannot be cast to > org.eclipse.jetty.servlet.ServletContextHandler > at > scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:237) > at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62) > at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55) > at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49) > at scala.collection.TraversableLike.map(TraversableLike.scala:237) > at scala.collection.TraversableLike.map$(TraversableLike.scala:230) > at scala.collection.AbstractTraversable.map(Traversable.scala:108) > at > org.apache.spark.scheduler.cluster.YarnSchedulerBackend.$anonfun$addWebUIFilter$2(YarnSchedulerBackend.scala:183) > at > org.apache.spark.scheduler.cluster.YarnSchedulerBackend.$anonfun$addWebUIFilter$2$adapted(YarnSchedulerBackend.scala:174) > at scala.Option.foreach(Option.scala:274) > ... > {code} > This looks like a classpath issue, probably ultimately related to the same > classloader issues in https://issues.apache.org/jira/browse/SPARK-26839 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-27122) YARN test failures in Java 9+
[ https://issues.apache.org/jira/browse/SPARK-27122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16792318#comment-16792318 ] Ajith S commented on SPARK-27122: - The problem seems to be shading of jetty package. When we run test, the class path seems to be made from the classes folder(resource-managers/yarn/target/scala-2.12/classes) instead of jar (resource-managers/yarn/target/spark-yarn_2.12-3.0.0-SNAPSHOT.jar) Here is test classpath info: !image-2019-03-14-09-34-20-592.png! And here is the javap command which shows the difference between org.apache.spark.scheduler.cluster.YarnSchedulerBackend present in jar folder and classes folder !image-2019-03-14-09-35-23-046.png! > YARN test failures in Java 9+ > - > > Key: SPARK-27122 > URL: https://issues.apache.org/jira/browse/SPARK-27122 > Project: Spark > Issue Type: Sub-task > Components: YARN >Affects Versions: 3.0.0 >Reporter: Sean Owen >Priority: Major > Attachments: image-2019-03-14-09-34-20-592.png, > image-2019-03-14-09-35-23-046.png > > > Currently on Java 11: > {code} > YarnSchedulerBackendSuite: > - RequestExecutors reflects node blacklist and is serializable > - Respect user filters when adding AM IP filter *** FAILED *** > java.lang.ClassCastException: > org.spark_project.jetty.servlet.ServletContextHandler cannot be cast to > org.eclipse.jetty.servlet.ServletContextHandler > at > scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:237) > at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62) > at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55) > at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49) > at scala.collection.TraversableLike.map(TraversableLike.scala:237) > at scala.collection.TraversableLike.map$(TraversableLike.scala:230) > at scala.collection.AbstractTraversable.map(Traversable.scala:108) > at > org.apache.spark.scheduler.cluster.YarnSchedulerBackend.$anonfun$addWebUIFilter$2(YarnSchedulerBackend.scala:183) > at > org.apache.spark.scheduler.cluster.YarnSchedulerBackend.$anonfun$addWebUIFilter$2$adapted(YarnSchedulerBackend.scala:174) > at scala.Option.foreach(Option.scala:274) > ... > {code} > This looks like a classpath issue, probably ultimately related to the same > classloader issues in https://issues.apache.org/jira/browse/SPARK-26839 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-27122) YARN test failures in Java 9+
[ https://issues.apache.org/jira/browse/SPARK-27122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16792299#comment-16792299 ] Ajith S commented on SPARK-27122: - I can reproduce this issue even in Java8. I would like to work on this. > YARN test failures in Java 9+ > - > > Key: SPARK-27122 > URL: https://issues.apache.org/jira/browse/SPARK-27122 > Project: Spark > Issue Type: Sub-task > Components: YARN >Affects Versions: 3.0.0 >Reporter: Sean Owen >Priority: Major > > Currently on Java 11: > {code} > YarnSchedulerBackendSuite: > - RequestExecutors reflects node blacklist and is serializable > - Respect user filters when adding AM IP filter *** FAILED *** > java.lang.ClassCastException: > org.spark_project.jetty.servlet.ServletContextHandler cannot be cast to > org.eclipse.jetty.servlet.ServletContextHandler > at > scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:237) > at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62) > at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55) > at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49) > at scala.collection.TraversableLike.map(TraversableLike.scala:237) > at scala.collection.TraversableLike.map$(TraversableLike.scala:230) > at scala.collection.AbstractTraversable.map(Traversable.scala:108) > at > org.apache.spark.scheduler.cluster.YarnSchedulerBackend.$anonfun$addWebUIFilter$2(YarnSchedulerBackend.scala:183) > at > org.apache.spark.scheduler.cluster.YarnSchedulerBackend.$anonfun$addWebUIFilter$2$adapted(YarnSchedulerBackend.scala:174) > at scala.Option.foreach(Option.scala:274) > ... > {code} > This looks like a classpath issue, probably ultimately related to the same > classloader issues in https://issues.apache.org/jira/browse/SPARK-26839 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org