[
https://issues.apache.org/jira/browse/SPARK-45897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arnaud Nauwynck updated SPARK-45897:
------------------------------------
Description:
updating springboot-dependencies 2.7 to 3.1 breaks spark with springboot
compatibilities.
{noformat}
<dependencyManagement>
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-dependencies</artifactId>
<version>2.7.14</version> <!-- OK 2.7.* works fine, but 3.*
ERROR -->
<type>pom</type>
<scope>import</scope>
</dependency>
</dependencies>
</dependencyManagement>
{noformat}
Internally, everything compile ok, but fail at runtime while creating the
SparkSession, with SparkUI:
{noformat}
Caused by: java.lang.NoClassDefFoundError: javax/servlet/Servlet
at org.apache.spark.ui.SparkUI$.create(SparkUI.scala:239)
~[spark-core_2.13-3.5.0.jar:3.5.0]
at org.apache.spark.SparkContext.<init>(SparkContext.scala:503)
~[spark-core_2.13-3.5.0.jar:3.5.0]
at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2888)
~[spark-core_2.13-3.5.0.jar:3.5.0]
at
org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$2(SparkSession.scala:1099)
~[spark-sql_2.13-3.5.0.jar:3.5.0]
at scala.Option.getOrElse(Option.scala:201)
~[scala-library-2.13.8.jar:na]
at
org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:1093)
~[spark-sql_2.13-3.5.0.jar:3.5.0]
....
Caused by: java.lang.ClassNotFoundException: javax.servlet.Servlet
at
java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641)
~[na:na]
at
java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
~[na:na]
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
~[na:na]
... 49 common frames omitted
{noformat}
Root cause: major change with springboot 3: 'javax.servlet.Servlet' class is NO
more defined in package 'javax.servlet.', but migrated to 'jakarta.servlet.'
The compiled dependency from spark-core to javax-servlet-api is replaced by
maven dependencyManagement
It "might be" possible to get rid of maven dependencyManagement or to override
several artifact versions, but it is NOT PRACTICAL.
Here is a version that work, much more complex with springboot3 than
springboot2.
It requires few of forced versions, and exclusions in maven.
{noformat}
<properties>
<spark.version>3.5.0</spark.version>
<scala.version>2.13</scala.version>
</properties>
<dependencyManagement>
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-dependencies</artifactId>
<version>3.1.2</version>
<type>pom</type>
<scope>import</scope>
</dependency>
<dependency>
<groupId>org.glassfish.jersey.containers</groupId>
<artifactId>jersey-container-servlet-core</artifactId>
<version>2.41</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
<version>2.0.9</version>
</dependency>
</dependencies>
</dependencyManagement>
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_${scala.version}</artifactId>
<version>${spark.version}</version>
<exclusions>
<exclusion>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-slf4j2-impl</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>jakarta.servlet</groupId>
<artifactId>jakarta.servlet-api</artifactId>
<version>4.0.4</version>
</dependency>
</dependencies>
{noformat}
there are several examples showing errors and fixed for servlet, log4j,
glassfish incompatibilities here:
https://github.com/Arnaud-Nauwynck/test-snippets/tree/master/test-springboot-spark
A possible long term solution is to migrate to servlet in package
"jakarta.servlet." instead of "javax.servlet." as springboot already did, so
there would "not (?)" be incompatibilities between very old and much more
recent jars.
Any roadmap for this?
was:
updating springboot-dependencies 2.7 to 3.1 breaks spark with springboot
compatibilities.
{noformat}
<dependencyManagement>
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-dependencies</artifactId>
<version>2.7.14</version> <!-- OK 2.7.* works fine, but 3.*
ERROR -->
<type>pom</type>
<scope>import</scope>
</dependency>
</dependencies>
</dependencyManagement>
{noformat}
Internally, everything compile ok, but fail at runtime while creating the
SparkSession, with SparkUI:
{noformat}
Caused by: java.lang.NoClassDefFoundError: javax/servlet/Servlet
at org.apache.spark.ui.SparkUI$.create(SparkUI.scala:239)
~[spark-core_2.13-3.5.0.jar:3.5.0]
at org.apache.spark.SparkContext.<init>(SparkContext.scala:503)
~[spark-core_2.13-3.5.0.jar:3.5.0]
at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2888)
~[spark-core_2.13-3.5.0.jar:3.5.0]
at
org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$2(SparkSession.scala:1099)
~[spark-sql_2.13-3.5.0.jar:3.5.0]
at scala.Option.getOrElse(Option.scala:201)
~[scala-library-2.13.8.jar:na]
at
org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:1093)
~[spark-sql_2.13-3.5.0.jar:3.5.0]
....
Caused by: java.lang.ClassNotFoundException: javax.servlet.Servlet
at
java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641)
~[na:na]
at
java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
~[na:na]
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
~[na:na]
... 49 common frames omitted
{noformat}
Root cause: major change with springboot 3: 'javax.servlet.Servlet' class is NO
more defined in package 'javax.*', but migrated to 'jakarta.*'
The compiled dependency from spark-core to javax-servlet-api is replaced by
maven dependencyManagement
It "might be" possible to get rid of maven dependencyManagement or to override
several artifact versions, but it is NOT PRACTICAL.
Here is a version that work, much more complex with springboot3 than
springboot2.
It requires few of forced versions, and exclusions in maven.
{noformat}
<properties>
<spark.version>3.5.0</spark.version>
<scala.version>2.13</scala.version>
</properties>
<dependencyManagement>
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-dependencies</artifactId>
<version>3.1.2</version>
<type>pom</type>
<scope>import</scope>
</dependency>
<dependency>
<groupId>org.glassfish.jersey.containers</groupId>
<artifactId>jersey-container-servlet-core</artifactId>
<version>2.41</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
<version>2.0.9</version>
</dependency>
</dependencies>
</dependencyManagement>
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_${scala.version}</artifactId>
<version>${spark.version}</version>
<exclusions>
<exclusion>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-slf4j2-impl</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>jakarta.servlet</groupId>
<artifactId>jakarta.servlet-api</artifactId>
<version>4.0.4</version>
</dependency>
</dependencies>
{noformat}
there are several examples showing errors and fixed for servlet, log4j,
glassfish incompatibilities here:
https://github.com/Arnaud-Nauwynck/test-snippets/tree/master/test-springboot-spark
A possible long term solution is to migrate to servlet in package "jakarta.*"
instead of "javax.*" as springboot already did, so there would "not (?)" be
incompatibilities between very old and much more recent jars.
Any roadmap for this?
> java.lang.NoClassDefFoundError: javax/servlet/Servlet incompatibilities
> upgrading springboot-dependencies 2.7 to 3.* (package javax.* -> jakarta.*)
> ---------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-45897
> URL: https://issues.apache.org/jira/browse/SPARK-45897
> Project: Spark
> Issue Type: Dependency upgrade
> Components: Web UI
> Affects Versions: 3.2.4, 3.3.3, 3.4.0, 3.4.1, 3.5.0
> Reporter: Arnaud Nauwynck
> Priority: Major
>
> updating springboot-dependencies 2.7 to 3.1 breaks spark with springboot
> compatibilities.
> {noformat}
> <dependencyManagement>
> <dependencies>
> <dependency>
> <groupId>org.springframework.boot</groupId>
> <artifactId>spring-boot-dependencies</artifactId>
> <version>2.7.14</version> <!-- OK 2.7.* works fine, but 3.*
> ERROR -->
> <type>pom</type>
> <scope>import</scope>
> </dependency>
> </dependencies>
> </dependencyManagement>
> {noformat}
> Internally, everything compile ok, but fail at runtime while creating the
> SparkSession, with SparkUI:
> {noformat}
> Caused by: java.lang.NoClassDefFoundError: javax/servlet/Servlet
> at org.apache.spark.ui.SparkUI$.create(SparkUI.scala:239)
> ~[spark-core_2.13-3.5.0.jar:3.5.0]
> at org.apache.spark.SparkContext.<init>(SparkContext.scala:503)
> ~[spark-core_2.13-3.5.0.jar:3.5.0]
> at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2888)
> ~[spark-core_2.13-3.5.0.jar:3.5.0]
> at
> org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$2(SparkSession.scala:1099)
> ~[spark-sql_2.13-3.5.0.jar:3.5.0]
> at scala.Option.getOrElse(Option.scala:201)
> ~[scala-library-2.13.8.jar:na]
> at
> org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:1093)
> ~[spark-sql_2.13-3.5.0.jar:3.5.0]
> ....
> Caused by: java.lang.ClassNotFoundException: javax.servlet.Servlet
> at
> java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641)
> ~[na:na]
> at
> java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
> ~[na:na]
> at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
> ~[na:na]
> ... 49 common frames omitted
> {noformat}
> Root cause: major change with springboot 3: 'javax.servlet.Servlet' class is
> NO more defined in package 'javax.servlet.', but migrated to
> 'jakarta.servlet.'
> The compiled dependency from spark-core to javax-servlet-api is replaced by
> maven dependencyManagement
> It "might be" possible to get rid of maven dependencyManagement or to
> override several artifact versions, but it is NOT PRACTICAL.
> Here is a version that work, much more complex with springboot3 than
> springboot2.
> It requires few of forced versions, and exclusions in maven.
>
> {noformat}
> <properties>
> <spark.version>3.5.0</spark.version>
> <scala.version>2.13</scala.version>
> </properties>
> <dependencyManagement>
> <dependencies>
> <dependency>
> <groupId>org.springframework.boot</groupId>
> <artifactId>spring-boot-dependencies</artifactId>
> <version>3.1.2</version>
> <type>pom</type>
> <scope>import</scope>
> </dependency>
> <dependency>
> <groupId>org.glassfish.jersey.containers</groupId>
> <artifactId>jersey-container-servlet-core</artifactId>
> <version>2.41</version>
> </dependency>
> <dependency>
> <groupId>org.slf4j</groupId>
> <artifactId>slf4j-api</artifactId>
> <version>2.0.9</version>
> </dependency>
> </dependencies>
> </dependencyManagement>
> <dependencies>
> <dependency>
> <groupId>org.springframework.boot</groupId>
> <artifactId>spring-boot-starter-web</artifactId>
> </dependency>
> <dependency>
> <groupId>org.apache.spark</groupId>
> <artifactId>spark-core_${scala.version}</artifactId>
> <version>${spark.version}</version>
> <exclusions>
> <exclusion>
> <groupId>org.apache.logging.log4j</groupId>
> <artifactId>log4j-slf4j2-impl</artifactId>
> </exclusion>
> </exclusions>
> </dependency>
> <dependency>
> <groupId>jakarta.servlet</groupId>
> <artifactId>jakarta.servlet-api</artifactId>
> <version>4.0.4</version>
> </dependency>
> </dependencies>
> {noformat}
> there are several examples showing errors and fixed for servlet, log4j,
> glassfish incompatibilities here:
> https://github.com/Arnaud-Nauwynck/test-snippets/tree/master/test-springboot-spark
> A possible long term solution is to migrate to servlet in package
> "jakarta.servlet." instead of "javax.servlet." as springboot already did, so
> there would "not (?)" be incompatibilities between very old and much more
> recent jars.
> Any roadmap for this?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
