It already does. I think that's not the same idea? On Mon, Dec 4, 2023, 8:12 PM Almog Tavor <almogta...@gmail.com> wrote:
> I think Spark should start shading it’s problematic deps similar to how > it’s done in Flink > > On Mon, 4 Dec 2023 at 2:57 Sean Owen <sro...@gmail.com> wrote: > >> I am not sure we can control that - the Scala _x.y suffix has particular >> meaning in the Scala ecosystem for artifacts and thus the naming of .jar >> files. And we need to work with the Scala ecosystem. >> >> What can't handle these files, Spring Boot? does it somehow assume the >> .jar file name relates to Java modules? >> >> By the by, Spark 4 is already moving to the jakarta.* packages for >> similar reasons. >> >> I don't think Spark does or can really leverage Java modules. It started >> waaay before that and expect that it has some structural issues that are >> incompatible with Java modules, like multiple places declaring code in the >> same Java package. >> >> As in all things, if there's a change that doesn't harm anything else and >> helps support for Java modules, sure, suggest it. If it has the conflicts I >> think it will, probably not possible and not really a goal I think. >> >> >> On Sun, Dec 3, 2023 at 11:30 AM Marc Le Bihan <mlebiha...@gmail.com> >> wrote: >> >>> Hello, >>> >>> Last month, I've attempted the experience of upgrading my >>> Spring-Boot 2 Java project, that relies heavily on Spark 3.4.2, to >>> Spring-Boot 3. It didn't succeed yet, but was informative. >>> >>> Spring-Boot 2 → 3 means especially javax.* becoming jakarka.* : >>> javax.activation, javax.ws.rs, javax.persistence, javax.validation, >>> javax.servlet... all of these have to change their packages and >>> dependencies. >>> Apart of that, they were some trouble with ANTLR 4 against ANTLR 3, >>> and few things with SFL4 and Log4J. >>> >>> It was not easy, and I guessed that going into modules could be a >>> key. But when I'm near the Spark submodules of my project, it fail with >>> messages such as: >>> package org.apache.spark.sql.types is declared in the unnamed >>> module, but module fr.ecoemploi.outbound.spark.core does not read it >>> >>> But I can't handle the spark dependencies easily, because they have >>> an "invalid name" for Java. It's a matter that it doesn't want the "_" that >>> is in the "_2.13" suffix of the jars. >>> [WARNING] Can't extract module name from >>> breeze-macros_2.13-2.1.0.jar: breeze.macros.2.13: Invalid module name: '2' >>> is not a Java identifier >>> [WARNING] Can't extract module name from >>> spark-tags_2.13-3.4.2.jar: spark.tags.2.13: Invalid module name: '2' is not >>> a Java identifier >>> [WARNING] Can't extract module name from >>> spark-unsafe_2.13-3.4.2.jar: spark.unsafe.2.13: Invalid module name: '2' is >>> not a Java identifier >>> [WARNING] Can't extract module name from >>> spark-mllib_2.13-3.4.2.jar: spark.mllib.2.13: Invalid module name: '2' is >>> not a Java identifier >>> [... around 30 ...] >>> >>> I think that changing the naming pattern of the Spark jars for the >>> 4.x could be a good idea, >>> but beyond that, what about attempting to integrate Spark into >>> modules, it's submodules defining module-info.java? >>> >>> Is it something that you think that [must | should | might | should >>> not | must not] be done? >>> >>> Regards, >>> >>> Marc Le Bihan >>> >>
