[jira] [Comment Edited] (SPARK-48238) Spark fail to start due to class o.a.h.yarn.server.webproxy.amfilter.AmIpFilter is not a jakarta.servlet.Filter
[
https://issues.apache.org/jira/browse/SPARK-48238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17846764#comment-17846764
]
HiuFung Kwok edited comment on SPARK-48238 at 5/15/24 10:06 PM:
[~dongjoon] I'm still new to the codebase, I will need to check in exact how we
currently provide backward support for Hadoop and Hive, before commenting
further.
was (Author: hf):
[~dongjoon] I'm still new to the codebase, I will need to check in exactly how
we currently provide backward support for Hadoop and Hive, before commenting
further.
> Spark fail to start due to class
> o.a.h.yarn.server.webproxy.amfilter.AmIpFilter is not a jakarta.servlet.Filter
> ---
>
> Key: SPARK-48238
> URL: https://issues.apache.org/jira/browse/SPARK-48238
> Project: Spark
> Issue Type: Sub-task
> Components: Build
>Affects Versions: 4.0.0
>Reporter: Cheng Pan
>Priority: Blocker
>
> I tested the latest master branch, it failed to start on YARN mode
> {code:java}
> dev/make-distribution.sh --tgz -Phive,hive-thriftserver,yarn{code}
>
> {code:java}
> $ bin/spark-sql --master yarn
> WARNING: Using incubator modules: jdk.incubator.vector
> Setting default log level to "WARN".
> To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use
> setLogLevel(newLevel).
> 2024-05-10 17:58:17 WARN NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 2024-05-10 17:58:18 WARN Client: Neither spark.yarn.jars nor
> spark.yarn.archive} is set, falling back to uploading libraries under
> SPARK_HOME.
> 2024-05-10 17:58:25 ERROR SparkContext: Error initializing SparkContext.
> org.sparkproject.jetty.util.MultiException: Multiple exceptions
> at
> org.sparkproject.jetty.util.MultiException.ifExceptionThrow(MultiException.java:117)
> ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.sparkproject.jetty.servlet.ServletHandler.initialize(ServletHandler.java:751)
> ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.sparkproject.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:392)
> ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.sparkproject.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:902)
> ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.sparkproject.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:306)
> ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.sparkproject.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:93)
> ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.spark.ui.ServerInfo.addHandler(JettyUtils.scala:514)
> ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.spark.ui.SparkUI.$anonfun$attachAllHandlers$2(SparkUI.scala:81)
> ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.spark.ui.SparkUI.$anonfun$attachAllHandlers$2$adapted(SparkUI.scala:81)
> ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at scala.collection.IterableOnceOps.foreach(IterableOnce.scala:619)
> ~[scala-library-2.13.13.jar:?]
> at scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:617)
> ~[scala-library-2.13.13.jar:?]
> at scala.collection.AbstractIterable.foreach(Iterable.scala:935)
> ~[scala-library-2.13.13.jar:?]
> at
> org.apache.spark.ui.SparkUI.$anonfun$attachAllHandlers$1(SparkUI.scala:81)
> ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.spark.ui.SparkUI.$anonfun$attachAllHandlers$1$adapted(SparkUI.scala:79)
> ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at scala.Option.foreach(Option.scala:437) ~[scala-library-2.13.13.jar:?]
> at org.apache.spark.ui.SparkUI.attachAllHandlers(SparkUI.scala:79)
> ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.spark.SparkContext.$anonfun$new$31(SparkContext.scala:690)
> ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.spark.SparkContext.$anonfun$new$31$adapted(SparkContext.scala:690)
> ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at scala.Option.foreach(Option.scala:437) ~[scala-library-2.13.13.jar:?]
> at org.apache.spark.SparkContext.(SparkContext.scala:690)
> ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2963)
> ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$2(SparkSession.scala:1118)
> ~[spark-sql_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
[jira] [Commented] (SPARK-48238) Spark fail to start due to class o.a.h.yarn.server.webproxy.amfilter.AmIpFilter is not a jakarta.servlet.Filter
[
https://issues.apache.org/jira/browse/SPARK-48238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17846764#comment-17846764
]
HiuFung Kwok commented on SPARK-48238:
--
[~dongjoon] I'm still new to the codebase, I will need to check in exactly how
we currently provide backward support for Hadoop and Hive, before commenting
further.
> Spark fail to start due to class
> o.a.h.yarn.server.webproxy.amfilter.AmIpFilter is not a jakarta.servlet.Filter
> ---
>
> Key: SPARK-48238
> URL: https://issues.apache.org/jira/browse/SPARK-48238
> Project: Spark
> Issue Type: Sub-task
> Components: Build
>Affects Versions: 4.0.0
>Reporter: Cheng Pan
>Priority: Blocker
>
> I tested the latest master branch, it failed to start on YARN mode
> {code:java}
> dev/make-distribution.sh --tgz -Phive,hive-thriftserver,yarn{code}
>
> {code:java}
> $ bin/spark-sql --master yarn
> WARNING: Using incubator modules: jdk.incubator.vector
> Setting default log level to "WARN".
> To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use
> setLogLevel(newLevel).
> 2024-05-10 17:58:17 WARN NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 2024-05-10 17:58:18 WARN Client: Neither spark.yarn.jars nor
> spark.yarn.archive} is set, falling back to uploading libraries under
> SPARK_HOME.
> 2024-05-10 17:58:25 ERROR SparkContext: Error initializing SparkContext.
> org.sparkproject.jetty.util.MultiException: Multiple exceptions
> at
> org.sparkproject.jetty.util.MultiException.ifExceptionThrow(MultiException.java:117)
> ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.sparkproject.jetty.servlet.ServletHandler.initialize(ServletHandler.java:751)
> ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.sparkproject.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:392)
> ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.sparkproject.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:902)
> ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.sparkproject.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:306)
> ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.sparkproject.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:93)
> ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.spark.ui.ServerInfo.addHandler(JettyUtils.scala:514)
> ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.spark.ui.SparkUI.$anonfun$attachAllHandlers$2(SparkUI.scala:81)
> ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.spark.ui.SparkUI.$anonfun$attachAllHandlers$2$adapted(SparkUI.scala:81)
> ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at scala.collection.IterableOnceOps.foreach(IterableOnce.scala:619)
> ~[scala-library-2.13.13.jar:?]
> at scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:617)
> ~[scala-library-2.13.13.jar:?]
> at scala.collection.AbstractIterable.foreach(Iterable.scala:935)
> ~[scala-library-2.13.13.jar:?]
> at
> org.apache.spark.ui.SparkUI.$anonfun$attachAllHandlers$1(SparkUI.scala:81)
> ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.spark.ui.SparkUI.$anonfun$attachAllHandlers$1$adapted(SparkUI.scala:79)
> ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at scala.Option.foreach(Option.scala:437) ~[scala-library-2.13.13.jar:?]
> at org.apache.spark.ui.SparkUI.attachAllHandlers(SparkUI.scala:79)
> ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.spark.SparkContext.$anonfun$new$31(SparkContext.scala:690)
> ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.spark.SparkContext.$anonfun$new$31$adapted(SparkContext.scala:690)
> ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at scala.Option.foreach(Option.scala:437) ~[scala-library-2.13.13.jar:?]
> at org.apache.spark.SparkContext.(SparkContext.scala:690)
> ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2963)
> ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$2(SparkSession.scala:1118)
> ~[spark-sql_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at scala.Option.getOrElse(Option.scala:201) [scala-library-2.13.13.jar:?]
> at
> org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:1112)
> [spark-sql_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
>
[jira] [Commented] (SPARK-44265) Built-in XML data source support
[ https://issues.apache.org/jira/browse/SPARK-44265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17845755#comment-17845755 ] HiuFung Kwok commented on SPARK-44265: -- Thanks! Let me have a look. > Built-in XML data source support > > > Key: SPARK-44265 > URL: https://issues.apache.org/jira/browse/SPARK-44265 > Project: Spark > Issue Type: Umbrella > Components: SQL >Affects Versions: 4.0.0 >Reporter: Sandip Agarwala >Priority: Critical > Labels: pull-request-available > > XML is a widely used data format. An external spark-xml package > ([https://github.com/databricks/spark-xml)] is available to read and write > XML data in spark. Making spark-xml built-in will provide a better user > experience for Spark SQL and structured streaming. The proposal is to inline > code from spark-xml package. > > Here is the link to > [SPIP|https://docs.google.com/document/d/1ZaOBT4-YFtN58UCx2cdFhlsKbie1ugAn-Fgz_Dddz-Q/edit] -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-48238) Spark fail to start due to class o.a.h.yarn.server.webproxy.amfilter.AmIpFilter is not a jakarta.servlet.Filter
[
https://issues.apache.org/jira/browse/SPARK-48238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17845750#comment-17845750
]
HiuFung Kwok commented on SPARK-48238:
--
I will open a few tickets on the Hadoop side to follow up on the deps bump, and
link those back to this ticket.
However, given the proposed timeline of Spark 4, I won't have the bandwidth to
work on it in the coming weeks.
If anyone takes the lead on this, feel free.
> Spark fail to start due to class
> o.a.h.yarn.server.webproxy.amfilter.AmIpFilter is not a jakarta.servlet.Filter
> ---
>
> Key: SPARK-48238
> URL: https://issues.apache.org/jira/browse/SPARK-48238
> Project: Spark
> Issue Type: Bug
> Components: Build
>Affects Versions: 4.0.0
>Reporter: Cheng Pan
>Priority: Blocker
>
> I tested the latest master branch, it failed to start on YARN mode
> {code:java}
> dev/make-distribution.sh --tgz -Phive,hive-thriftserver,yarn{code}
>
> {code:java}
> $ bin/spark-sql --master yarn
> WARNING: Using incubator modules: jdk.incubator.vector
> Setting default log level to "WARN".
> To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use
> setLogLevel(newLevel).
> 2024-05-10 17:58:17 WARN NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 2024-05-10 17:58:18 WARN Client: Neither spark.yarn.jars nor
> spark.yarn.archive} is set, falling back to uploading libraries under
> SPARK_HOME.
> 2024-05-10 17:58:25 ERROR SparkContext: Error initializing SparkContext.
> org.sparkproject.jetty.util.MultiException: Multiple exceptions
> at
> org.sparkproject.jetty.util.MultiException.ifExceptionThrow(MultiException.java:117)
> ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.sparkproject.jetty.servlet.ServletHandler.initialize(ServletHandler.java:751)
> ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.sparkproject.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:392)
> ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.sparkproject.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:902)
> ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.sparkproject.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:306)
> ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.sparkproject.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:93)
> ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.spark.ui.ServerInfo.addHandler(JettyUtils.scala:514)
> ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.spark.ui.SparkUI.$anonfun$attachAllHandlers$2(SparkUI.scala:81)
> ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.spark.ui.SparkUI.$anonfun$attachAllHandlers$2$adapted(SparkUI.scala:81)
> ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at scala.collection.IterableOnceOps.foreach(IterableOnce.scala:619)
> ~[scala-library-2.13.13.jar:?]
> at scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:617)
> ~[scala-library-2.13.13.jar:?]
> at scala.collection.AbstractIterable.foreach(Iterable.scala:935)
> ~[scala-library-2.13.13.jar:?]
> at
> org.apache.spark.ui.SparkUI.$anonfun$attachAllHandlers$1(SparkUI.scala:81)
> ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.spark.ui.SparkUI.$anonfun$attachAllHandlers$1$adapted(SparkUI.scala:79)
> ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at scala.Option.foreach(Option.scala:437) ~[scala-library-2.13.13.jar:?]
> at org.apache.spark.ui.SparkUI.attachAllHandlers(SparkUI.scala:79)
> ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.spark.SparkContext.$anonfun$new$31(SparkContext.scala:690)
> ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.spark.SparkContext.$anonfun$new$31$adapted(SparkContext.scala:690)
> ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at scala.Option.foreach(Option.scala:437) ~[scala-library-2.13.13.jar:?]
> at org.apache.spark.SparkContext.(SparkContext.scala:690)
> ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2963)
> ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$2(SparkSession.scala:1118)
> ~[spark-sql_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at scala.Option.getOrElse(Option.scala:201) [scala-library-2.13.13.jar:?]
> at
>
[jira] [Commented] (SPARK-48238) Spark fail to start due to class o.a.h.yarn.server.webproxy.amfilter.AmIpFilter is not a jakarta.servlet.Filter
[
https://issues.apache.org/jira/browse/SPARK-48238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17845749#comment-17845749
]
HiuFung Kwok commented on SPARK-48238:
--
[~chengpan] It's sad but you are right, we need to revert to Jetty 10 (The
last version of Javax), until Hadoop bump their dep to Jakarta compatible
version.
> Spark fail to start due to class
> o.a.h.yarn.server.webproxy.amfilter.AmIpFilter is not a jakarta.servlet.Filter
> ---
>
> Key: SPARK-48238
> URL: https://issues.apache.org/jira/browse/SPARK-48238
> Project: Spark
> Issue Type: Bug
> Components: Build
>Affects Versions: 4.0.0
>Reporter: Cheng Pan
>Priority: Blocker
>
> I tested the latest master branch, it failed to start on YARN mode
> {code:java}
> dev/make-distribution.sh --tgz -Phive,hive-thriftserver,yarn{code}
>
> {code:java}
> $ bin/spark-sql --master yarn
> WARNING: Using incubator modules: jdk.incubator.vector
> Setting default log level to "WARN".
> To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use
> setLogLevel(newLevel).
> 2024-05-10 17:58:17 WARN NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 2024-05-10 17:58:18 WARN Client: Neither spark.yarn.jars nor
> spark.yarn.archive} is set, falling back to uploading libraries under
> SPARK_HOME.
> 2024-05-10 17:58:25 ERROR SparkContext: Error initializing SparkContext.
> org.sparkproject.jetty.util.MultiException: Multiple exceptions
> at
> org.sparkproject.jetty.util.MultiException.ifExceptionThrow(MultiException.java:117)
> ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.sparkproject.jetty.servlet.ServletHandler.initialize(ServletHandler.java:751)
> ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.sparkproject.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:392)
> ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.sparkproject.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:902)
> ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.sparkproject.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:306)
> ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.sparkproject.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:93)
> ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.spark.ui.ServerInfo.addHandler(JettyUtils.scala:514)
> ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.spark.ui.SparkUI.$anonfun$attachAllHandlers$2(SparkUI.scala:81)
> ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.spark.ui.SparkUI.$anonfun$attachAllHandlers$2$adapted(SparkUI.scala:81)
> ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at scala.collection.IterableOnceOps.foreach(IterableOnce.scala:619)
> ~[scala-library-2.13.13.jar:?]
> at scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:617)
> ~[scala-library-2.13.13.jar:?]
> at scala.collection.AbstractIterable.foreach(Iterable.scala:935)
> ~[scala-library-2.13.13.jar:?]
> at
> org.apache.spark.ui.SparkUI.$anonfun$attachAllHandlers$1(SparkUI.scala:81)
> ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.spark.ui.SparkUI.$anonfun$attachAllHandlers$1$adapted(SparkUI.scala:79)
> ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at scala.Option.foreach(Option.scala:437) ~[scala-library-2.13.13.jar:?]
> at org.apache.spark.ui.SparkUI.attachAllHandlers(SparkUI.scala:79)
> ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.spark.SparkContext.$anonfun$new$31(SparkContext.scala:690)
> ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.spark.SparkContext.$anonfun$new$31$adapted(SparkContext.scala:690)
> ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at scala.Option.foreach(Option.scala:437) ~[scala-library-2.13.13.jar:?]
> at org.apache.spark.SparkContext.(SparkContext.scala:690)
> ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2963)
> ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$2(SparkSession.scala:1118)
> ~[spark-sql_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at scala.Option.getOrElse(Option.scala:201) [scala-library-2.13.13.jar:?]
> at
> org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:1112)
> [spark-sql_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
>
[jira] [Commented] (SPARK-48238) Spark fail to start due to class o.a.h.yarn.server.webproxy.amfilter.AmIpFilter is not a jakarta.servlet.Filter
[
https://issues.apache.org/jira/browse/SPARK-48238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17845748#comment-17845748
]
HiuFung Kwok commented on SPARK-48238:
--
Let me have a look.
> Spark fail to start due to class
> o.a.h.yarn.server.webproxy.amfilter.AmIpFilter is not a jakarta.servlet.Filter
> ---
>
> Key: SPARK-48238
> URL: https://issues.apache.org/jira/browse/SPARK-48238
> Project: Spark
> Issue Type: Bug
> Components: Build
>Affects Versions: 4.0.0
>Reporter: Cheng Pan
>Priority: Blocker
>
> I tested the latest master branch, it failed to start on YARN mode
> {code:java}
> dev/make-distribution.sh --tgz -Phive,hive-thriftserver,yarn{code}
>
> {code:java}
> $ bin/spark-sql --master yarn
> WARNING: Using incubator modules: jdk.incubator.vector
> Setting default log level to "WARN".
> To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use
> setLogLevel(newLevel).
> 2024-05-10 17:58:17 WARN NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 2024-05-10 17:58:18 WARN Client: Neither spark.yarn.jars nor
> spark.yarn.archive} is set, falling back to uploading libraries under
> SPARK_HOME.
> 2024-05-10 17:58:25 ERROR SparkContext: Error initializing SparkContext.
> org.sparkproject.jetty.util.MultiException: Multiple exceptions
> at
> org.sparkproject.jetty.util.MultiException.ifExceptionThrow(MultiException.java:117)
> ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.sparkproject.jetty.servlet.ServletHandler.initialize(ServletHandler.java:751)
> ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.sparkproject.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:392)
> ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.sparkproject.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:902)
> ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.sparkproject.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:306)
> ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.sparkproject.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:93)
> ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.spark.ui.ServerInfo.addHandler(JettyUtils.scala:514)
> ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.spark.ui.SparkUI.$anonfun$attachAllHandlers$2(SparkUI.scala:81)
> ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.spark.ui.SparkUI.$anonfun$attachAllHandlers$2$adapted(SparkUI.scala:81)
> ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at scala.collection.IterableOnceOps.foreach(IterableOnce.scala:619)
> ~[scala-library-2.13.13.jar:?]
> at scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:617)
> ~[scala-library-2.13.13.jar:?]
> at scala.collection.AbstractIterable.foreach(Iterable.scala:935)
> ~[scala-library-2.13.13.jar:?]
> at
> org.apache.spark.ui.SparkUI.$anonfun$attachAllHandlers$1(SparkUI.scala:81)
> ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.spark.ui.SparkUI.$anonfun$attachAllHandlers$1$adapted(SparkUI.scala:79)
> ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at scala.Option.foreach(Option.scala:437) ~[scala-library-2.13.13.jar:?]
> at org.apache.spark.ui.SparkUI.attachAllHandlers(SparkUI.scala:79)
> ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.spark.SparkContext.$anonfun$new$31(SparkContext.scala:690)
> ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.spark.SparkContext.$anonfun$new$31$adapted(SparkContext.scala:690)
> ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at scala.Option.foreach(Option.scala:437) ~[scala-library-2.13.13.jar:?]
> at org.apache.spark.SparkContext.(SparkContext.scala:690)
> ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2963)
> ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$2(SparkSession.scala:1118)
> ~[spark-sql_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at scala.Option.getOrElse(Option.scala:201) [scala-library-2.13.13.jar:?]
> at
> org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:1112)
> [spark-sql_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.spark.sql.hive.thriftserver.SparkSQLEnv$.init(SparkSQLEnv.scala:64)
> [spark-hive-thriftserver_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
>
[jira] [Commented] (SPARK-44265) Built-in XML data source support
[ https://issues.apache.org/jira/browse/SPARK-44265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17845650#comment-17845650 ] HiuFung Kwok commented on SPARK-44265: -- [~gurwls223] Hi, I have double-checked all remaining tickets, these are all duplicated tickets and the functionality already exists on the master branch of Spark. Perhaps we can go ahead to mark all sub-tasks as duplicated and mark this umbrella task as resolved? > Built-in XML data source support > > > Key: SPARK-44265 > URL: https://issues.apache.org/jira/browse/SPARK-44265 > Project: Spark > Issue Type: Umbrella > Components: SQL >Affects Versions: 4.0.0 >Reporter: Sandip Agarwala >Priority: Critical > Labels: pull-request-available > > XML is a widely used data format. An external spark-xml package > ([https://github.com/databricks/spark-xml)] is available to read and write > XML data in spark. Making spark-xml built-in will provide a better user > experience for Spark SQL and structured streaming. The proposal is to inline > code from spark-xml package. > > Here is the link to > [SPIP|https://docs.google.com/document/d/1ZaOBT4-YFtN58UCx2cdFhlsKbie1ugAn-Fgz_Dddz-Q/edit] -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-44810) XML: ArrayType and MapType support in from_xml
[ https://issues.apache.org/jira/browse/SPARK-44810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17845125#comment-17845125 ] HiuFung Kwok edited comment on SPARK-44810 at 5/9/24 8:49 PM: -- [~sandip.agarwala] Should this be closed also, it seems the functionality has already been ported, and relevant tests exist under `XmlSuite.scala`. and same for the other sub-tasks under this EPIC. was (Author: hf): [~sandip.agarwala] Should this be closed also, it seems the functionality has already been ported, and relevant tests exist under `XmlSuite.scala`. > XML: ArrayType and MapType support in from_xml > -- > > Key: SPARK-44810 > URL: https://issues.apache.org/jira/browse/SPARK-44810 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 4.0.0 >Reporter: Sandip Agarwala >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-44810) XML: ArrayType and MapType support in from_xml
[ https://issues.apache.org/jira/browse/SPARK-44810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17845125#comment-17845125 ] HiuFung Kwok commented on SPARK-44810: -- [~sandip.agarwala] Should this be closed also, it seems the functionality has already been ported, and relevant tests exist under `XmlSuite.scala`. > XML: ArrayType and MapType support in from_xml > -- > > Key: SPARK-44810 > URL: https://issues.apache.org/jira/browse/SPARK-44810 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 4.0.0 >Reporter: Sandip Agarwala >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-47275) XML: Change to not support DROPMALFORMED parse mode
[ https://issues.apache.org/jira/browse/SPARK-47275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17842463#comment-17842463 ] HiuFung Kwok commented on SPARK-47275: -- [~yhosny] I thought, this is already done under SPARK-47218, is that correct? https://github.com/apache/spark/pull/45379/files#diff-dc59a51cc9cc5b40a0969c1b9bd0ed81d14f71473ccf9a16cbc63e09d6aa1843R193 > XML: Change to not support DROPMALFORMED parse mode > --- > > Key: SPARK-47275 > URL: https://issues.apache.org/jira/browse/SPARK-47275 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 4.0.0 >Reporter: Yousof Hosny >Priority: Minor > > Change XML expressions to not support DROPMALFORMED parse mode. This matches > JSON expressions which also do not support it. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-47219) XML: Ignore commented row tags in XML tokenizer
[ https://issues.apache.org/jira/browse/SPARK-47219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17841351#comment-17841351 ] HiuFung Kwok commented on SPARK-47219: -- [~sandip.agarwala] Do we need this task, it seems to be duplicate of SPARK-47218 . > XML: Ignore commented row tags in XML tokenizer > --- > > Key: SPARK-47219 > URL: https://issues.apache.org/jira/browse/SPARK-47219 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 4.0.0 >Reporter: Sandip Agarwala >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-42846) Assign a name to the error class _LEGACY_ERROR_TEMP_2011
[
https://issues.apache.org/jira/browse/SPARK-42846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17841347#comment-17841347
]
HiuFung Kwok commented on SPARK-42846:
--
Hi [~maxgekk] , I have submitted an MR for this, would you mind having a look?
Thx.
> Assign a name to the error class _LEGACY_ERROR_TEMP_2011
>
>
> Key: SPARK-42846
> URL: https://issues.apache.org/jira/browse/SPARK-42846
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 3.5.0
>Reporter: Max Gekk
>Priority: Minor
> Labels: pull-request-available, starter
>
> Choose a proper name for the error class *_LEGACY_ERROR_TEMP_2011* defined in
> {*}core/src/main/resources/error/error-classes.json{*}. The name should be
> short but complete (look at the example in error-classes.json).
> Add a test which triggers the error from user code if such test still doesn't
> exist. Check exception fields by using {*}checkError(){*}. The last function
> checks valuable error fields only, and avoids dependencies from error text
> message. In this way, tech editors can modify error format in
> error-classes.json, and don't worry of Spark's internal tests. Migrate other
> tests that might trigger the error onto checkError().
> If you cannot reproduce the error from user space (using SQL query), replace
> the error by an internal error, see {*}SparkException.internalError(){*}.
> Improve the error message format in error-classes.json if the current is not
> clear. Propose a solution to users how to avoid and fix such kind of errors.
> Please, look at the PR below as examples:
> * [https://github.com/apache/spark/pull/38685]
> * [https://github.com/apache/spark/pull/38656]
> * [https://github.com/apache/spark/pull/38490]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-47086) Upgrade Jetty to 12
[ https://issues.apache.org/jira/browse/SPARK-47086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17829372#comment-17829372 ] HiuFung Kwok edited comment on SPARK-47086 at 3/21/24 4:10 AM: --- Due to the Jetty handler structure change, additional effort will be required for the Jetty 11 to 12 upgrade to rewrite the existing `redirect` handler, in order to comply with the new structure. Hence I have stopped the work, and below is my WIP branch. [https://github.com/apache/spark/pull/45500] If other wishes to continue the work, feel free. was (Author: hf): Due to the Jetty handler structure change, Jetty 11 to 12 additional effort will be required to rewrite the existing `redirect` handler to comply with the new structure. Hence I have stopped the work, and below is my WIP branch. [https://github.com/apache/spark/pull/45500] If other wishes to continue the work, feel free. > Upgrade Jetty to 12 > --- > > Key: SPARK-47086 > URL: https://issues.apache.org/jira/browse/SPARK-47086 > Project: Spark > Issue Type: Sub-task > Components: Build >Affects Versions: 4.0.0 >Reporter: HiuFung Kwok >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-47086) Upgrade Jetty to 12
[ https://issues.apache.org/jira/browse/SPARK-47086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17829372#comment-17829372 ] HiuFung Kwok commented on SPARK-47086: -- Due to the Jetty handler structure change, Jetty 11 to 12 additional effort will be required to rewrite the existing `redirect` handler to comply with the new structure. Hence I have stopped the work, and below is my WIP branch. [https://github.com/apache/spark/pull/45500] If other wishes to continue the work, feel free. > Upgrade Jetty to 12 > --- > > Key: SPARK-47086 > URL: https://issues.apache.org/jira/browse/SPARK-47086 > Project: Spark > Issue Type: Sub-task > Components: Build >Affects Versions: 4.0.0 >Reporter: HiuFung Kwok >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-47170) Remove redundant scope identifier for `jakarta.servlet-api` and `javax.servlet-api`
HiuFung Kwok created SPARK-47170: Summary: Remove redundant scope identifier for `jakarta.servlet-api` and `javax.servlet-api` Key: SPARK-47170 URL: https://issues.apache.org/jira/browse/SPARK-47170 Project: Spark Issue Type: Sub-task Components: Build Affects Versions: 4.0.0 Reporter: HiuFung Kwok This is a follow-up ticket for SPRARK-47046 to remove the redundant `scope` XML clause - compile. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-44319) Migrate jersey 2 to jersey 3
[ https://issues.apache.org/jira/browse/SPARK-44319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17820268#comment-17820268 ] HiuFung Kwok commented on SPARK-44319: -- [~dongjoon] FYI I marked this as resolved also. > Migrate jersey 2 to jersey 3 > > > Key: SPARK-44319 > URL: https://issues.apache.org/jira/browse/SPARK-44319 > Project: Spark > Issue Type: Sub-task > Components: Build, Spark Core >Affects Versions: 4.0.0 >Reporter: Yang Jie >Priority: Minor > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-44319) Migrate jersey 2 to jersey 3
[ https://issues.apache.org/jira/browse/SPARK-44319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] HiuFung Kwok resolved SPARK-44319. -- Fix Version/s: 4.0.0 Resolution: Fixed The work is done under the scope of SPARK-47118. > Migrate jersey 2 to jersey 3 > > > Key: SPARK-44319 > URL: https://issues.apache.org/jira/browse/SPARK-44319 > Project: Spark > Issue Type: Sub-task > Components: Build, Spark Core >Affects Versions: 4.0.0 >Reporter: Yang Jie >Priority: Minor > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-47086) Migrate jetty 11 to jetty 12
HiuFung Kwok created SPARK-47086: Summary: Migrate jetty 11 to jetty 12 Key: SPARK-47086 URL: https://issues.apache.org/jira/browse/SPARK-47086 Project: Spark Issue Type: Sub-task Components: Build Affects Versions: 4.0.0 Reporter: HiuFung Kwok -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-46938) Migrate jetty 10 to jetty 11
[ https://issues.apache.org/jira/browse/SPARK-46938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17817612#comment-17817612 ] HiuFung Kwok commented on SPARK-46938: -- To update on this ticket, the Hive SPARK-44114 is blocked due to a classloader issue on Hive issue, and they don't have a concrete date on release v3.2. In the meantime, I'm checking whether is possible to update the logic on Tservlet to get around this. Thanks, > Migrate jetty 10 to jetty 11 > > > Key: SPARK-46938 > URL: https://issues.apache.org/jira/browse/SPARK-46938 > Project: Spark > Issue Type: Sub-task > Components: Build >Affects Versions: 4.0.0 >Reporter: Yang Jie >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-44114) Upgrade built-in Hive to 3+
[ https://issues.apache.org/jira/browse/SPARK-44114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17817215#comment-17817215 ] HiuFung Kwok edited comment on SPARK-44114 at 2/14/24 5:58 AM: --- hi [~dongjoon] It turns out that is blocked by HIVE-27508, which is the issue presented on the classloader when Hive package above JDK8, the changes seem to have landed on their hotfix, but the work on this ticket can only be resumed when the Hive community releases v3.1.4. Hence would you mind linking HIVE-27508 as the blocker for this ticket? Thanks, was (Author: hf): hi [~dongjoon] It turns out that is blocked by HIVE-27508, which is the issue presented on the classloader when Hive is above JDK8, the changes seem to have landed on their hotfix, but the work on this ticket can only be resumed when the Hive community releases v3.1.4. Hence would you mind linking HIVE-27508 as the blocker for this ticket? Thanks, > Upgrade built-in Hive to 3+ > --- > > Key: SPARK-44114 > URL: https://issues.apache.org/jira/browse/SPARK-44114 > Project: Spark > Issue Type: Sub-task > Components: Build, SQL >Affects Versions: 4.0.0 >Reporter: Dongjoon Hyun >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-44114) Upgrade built-in Hive to 3+
[ https://issues.apache.org/jira/browse/SPARK-44114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17817215#comment-17817215 ] HiuFung Kwok commented on SPARK-44114: -- hi [~dongjoon] It turns out that is blocked by HIVE-27508, which is the issue presented on the classloader when Hive is above JDK8, the changes seem to have landed on their hotfix, but the work on this ticket can only be resumed when the Hive community releases v3.1.4. Hence would you mind linking HIVE-27508 as the blocker for this ticket? Thanks, > Upgrade built-in Hive to 3+ > --- > > Key: SPARK-44114 > URL: https://issues.apache.org/jira/browse/SPARK-44114 > Project: Spark > Issue Type: Sub-task > Components: Build, SQL >Affects Versions: 4.0.0 >Reporter: Dongjoon Hyun >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-46938) Migrate jetty 10 to jetty 11
[ https://issues.apache.org/jira/browse/SPARK-46938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17815117#comment-17815117 ] HiuFung Kwok commented on SPARK-46938: -- [~LuciferYang] The Hive community ended up not releasing v2.3.10, maybe I have overlooked. If that is the case, the alternative would be Hive 3.0 and hence should we mark the below ticket as a dependency? https://issues.apache.org/jira/browse/SPARK-44114 I can pick up SPARK-44114 instead, but if there is any concern, please raise it. > Migrate jetty 10 to jetty 11 > > > Key: SPARK-46938 > URL: https://issues.apache.org/jira/browse/SPARK-46938 > Project: Spark > Issue Type: Sub-task > Components: Build >Affects Versions: 4.0.0 >Reporter: Yang Jie >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-46938) Migrate jetty 10 to jetty 11
[ https://issues.apache.org/jira/browse/SPARK-46938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17814173#comment-17814173 ] HiuFung Kwok commented on SPARK-46938: -- [~LuciferYang] Hi, it turns out, lib THRIFT will also need to bump to v0.19 as there are a few Servlet references on their end. Do we need a separate ticket for this? But I don't think separate MR is possible, all Jersey, lib Thrfit and Jetty 11 upgrades need to happen within one MR due to the Jakrata package name change. https://issues.apache.org/jira/browse/THRIFT-5700?jql=project%20%3D%20THRIFT%20AND%20text%20~%20%22Jakarta%22%20ORDER%20BY%20created%20DESC > Migrate jetty 10 to jetty 11 > > > Key: SPARK-46938 > URL: https://issues.apache.org/jira/browse/SPARK-46938 > Project: Spark > Issue Type: Sub-task > Components: Build >Affects Versions: 4.0.0 >Reporter: Yang Jie >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
