[jira] [Created] (SPARK-45897) java.lang.NoClassDefFoundError: javax/servlet/Servlet incompatibilities upgrading springboot-dependencies 2.7 to 3.* (package javax.* -> jakarta.*)

[Arnaud Nauwynck (Jira)]

Arnaud Nauwynck created SPARK-45897:
---------------------------------------

             Summary: java.lang.NoClassDefFoundError: javax/servlet/Servlet 
incompatibilities upgrading springboot-dependencies 2.7 to 3.* (package javax.* 
-> jakarta.*)
                 Key: SPARK-45897
                 URL: https://issues.apache.org/jira/browse/SPARK-45897
             Project: Spark
          Issue Type: Dependency upgrade
          Components: Web UI
    Affects Versions: 3.5.0, 3.4.1, 3.4.0, 3.3.3, 3.2.4
            Reporter: Arnaud Nauwynck


updating springboot-dependencies 2.7 to 3.1 breaks spark with springboot 
compatibilities.


{noformat}
    <dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-dependencies</artifactId>
                <version>2.7.14</version> <!-- OK 2.7.* works fine, but  3.* 
ERROR --> 
                <type>pom</type>
                <scope>import</scope>
            </dependency>
        </dependencies>
    </dependencyManagement>
{noformat}

Internally, everything compile ok, but fail at runtime while creating the 
SparkSession, with SparkUI:


{noformat}
Caused by: java.lang.NoClassDefFoundError: javax/servlet/Servlet
        at org.apache.spark.ui.SparkUI$.create(SparkUI.scala:239) 
~[spark-core_2.13-3.5.0.jar:3.5.0]
        at org.apache.spark.SparkContext.<init>(SparkContext.scala:503) 
~[spark-core_2.13-3.5.0.jar:3.5.0]
        at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2888) 
~[spark-core_2.13-3.5.0.jar:3.5.0]
        at 
org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$2(SparkSession.scala:1099)
 ~[spark-sql_2.13-3.5.0.jar:3.5.0]
        at scala.Option.getOrElse(Option.scala:201) 
~[scala-library-2.13.8.jar:na]
        at 
org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:1093) 
~[spark-sql_2.13-3.5.0.jar:3.5.0]
....

Caused by: java.lang.ClassNotFoundException: javax.servlet.Servlet
        at 
java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641)
 ~[na:na]
        at 
java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
 ~[na:na]
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521) 
~[na:na]
        ... 49 common frames omitted

{noformat}

Root cause: major change with springboot 3: 'javax.servlet.Servlet' class is NO 
more defined in package 'javax.*', but migrated to 'jakarta.*'

The compiled dependency from spark-core to javax-servlet-api is replaced by 
maven dependencyManagement


It "might be" possible to get rid of maven dependencyManagement or to override 
several artifact versions, but it is NOT PRACTICAL.

Here is a version that work, much more complex  with springboot3 than 
springboot2.

It requires few of forced versions, and exclusions in maven.

    
{noformat}
<properties>
        <spark.version>3.5.0</spark.version>
        <scala.version>2.13</scala.version>
    </properties>

    <dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-dependencies</artifactId>
                <version>3.1.2</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
            <dependency>
                <groupId>org.glassfish.jersey.containers</groupId>
                <artifactId>jersey-container-servlet-core</artifactId>
                <version>2.41</version>
            </dependency>
            <dependency>
                <groupId>org.slf4j</groupId>
                <artifactId>slf4j-api</artifactId>
                <version>2.0.9</version>
            </dependency>
        </dependencies>
    </dependencyManagement>

    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>

        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-core_${scala.version}</artifactId>
            <version>${spark.version}</version>
            <exclusions>
                <exclusion>
                    <groupId>org.apache.logging.log4j</groupId>
                    <artifactId>log4j-slf4j2-impl</artifactId>
                </exclusion>
            </exclusions>
        </dependency>

        <dependency>
            <groupId>jakarta.servlet</groupId>
            <artifactId>jakarta.servlet-api</artifactId>
            <version>4.0.4</version>
        </dependency>

    </dependencies>
{noformat}


there are several examples showing errors and fixed for servlet, log4j, 
glassfish incompatibilities here: 
https://github.com/Arnaud-Nauwynck/test-snippets/tree/master/test-springboot-spark



A possible long term solution is to migrate to servlet in package "jakarta.*" 
instead of "javax.*" as springboot already did, so there would "not (?)" be 
incompatibilities between very old and much more recent jars.

Any roadmap for this?








--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org