Agreed with Kyle. An artifact name of Spark3.0 like iceberg-spark-runtime-3.0_2.12-0.13.1.jar is more accurate and consistent, less confusing for users.
On Sun, Feb 20, 2022 at 10:47 PM Kyle Bendickson <k...@tabular.io> wrote: > Thanks for bringing this up Jeff! > > Normally I agree, it’s not a good practice to change artifact name. > However, in this case, the artifact has changed already. The > “spark3-runtime” used to be for all versions of Spark 3 (at the time Spark > 3.0 and 3.1). It no longer is, as it’s only tested / used with Spark 3.0. > > I encounter many users who have upgraded to newer versions of Spark, but > have not upgraded the artifact to the newly versioned by Spark name system > as “spark3-runtime” sounds like it encompasses all versions. And they > encounter subtle bugs and it’s not a great user experience to solve > upgrading that way. > > These users are, however, updating the Iceberg artifact to the new > versions. > > So I think in this case, breaking naming has benefits. As users who go to > upgrade when new Iceberg version are released, and their dependency is not > found, they will hopefully check maven and see the new naming convention / > artifacts. > > So I support option 2 also, with naming with Spark and Scala versions. > Otherwise, we continue to see people using the old “spark3-runtime” as they > upgrade Spark versions and encounter subtle errors (class not found, wrong > type signatures due to version mismatch). > > Users eventually have to upgrade their pom if / when they upgrade Spark, > due to incompatibility. This way at least, breaking will be loud as there’s > won’t be a new Iceberg version, > > Is it possible to mark to the old spark3-runtime / spark-runtime as > deprecated or otherwise point to the new artifacts in Maven? > > - Kyle > > On Sun, Feb 20, 2022 at 9:41 PM Jeff Zhang <zjf...@gmail.com> wrote: > >> I don't think it is best practice to just change the artifact name of >> published jars. Unless we publish a new version with the new naming >> convention. >> >> On Mon, Feb 21, 2022 at 12:36 PM Jack Ye <yezhao...@gmail.com> wrote: >> >>> I think option 2 is ideal, but I don't know if there is any hard >>> requirement from ASF/Maven Central side for us to keep backwards >>> compatibility of package names published in maven. If there is a >>> requirement then we cannot change it. >>> >>> As a mitigation, I stated in >>> https://iceberg.apache.org/multi-engine-support that Spark 2.4 and 3.0 >>> jar names do not follow the naming convention of newer versions for >>> backwards compatibility. >>> >>> Best, >>> Jack Ye >>> >>> On Sun, Feb 20, 2022 at 7:03 PM OpenInx <open...@gmail.com> wrote: >>> >>>> Hi everyone >>>> >>>> The current spark2.4, spark3.0 have the following unaligned runtime >>>> artifact names: >>>> >>>> # Spark 2.4 >>>> iceberg-spark-runtime-0.13.1.jar >>>> # Spark 3.0 >>>> iceberg-spark3-runtime-0.13.1.jar >>>> # Spark 3.1 >>>> iceberg-spark-runtime-3.1_2.12-0.13.1.jar >>>> # Spark 3.2 >>>> iceberg-spark-runtime-3.2_2.12-0.13.1.jar >>>> >>>> From the spark 3.1 and spark 3.2's runtime artifact names, we can >>>> easily recognize: >>>> 1. What's the spark major version that the runtime jar is attached to >>>> 2. What's the spark scala version that the runtime jar is compiled with >>>> >>>> But for spark 3.0 and spark 2.4, it's not easy to understand what's >>>> the above information. I think we kept those legacy names because they >>>> were introduced in older iceberg releases and we wanted to avoid changing >>>> the modules that users depend on and opted not to rename, but they are >>>> indeed causing confusion for the new community users. >>>> >>>> In general, we have two options: >>>> >>>> Option#1: keep the current artifact names, that mean spark 2.4 & spark >>>> 3.0 will always use the iceberg-spark-runtime-<iceberg-version>.jar and >>>> iceberg-spark3-runtime-<iceberg-version>.jar until them get retired in the >>>> apache iceberg official repo. >>>> Option#2: Change the spark2.4 & spark3.0's artifact names to the >>>> generic name format: >>>> iceberg-spark-runtime-<spark-major.minor>_<scala-version>-<iceberg-version>.jar. >>>> It makes sharing all the consistent name format between all the spark >>>> versions. >>>> >>>> Personally, I'd prefer option#2 because that looks more friendly for >>>> new community users (although it will require the old users to change their >>>> pom.xml to the new version). >>>> >>>> What is your preference ? >>>> >>>> Reference: >>>> 1. Created a PR to change the artifact names and we had few >>>> discussions there. https://github.com/apache/iceberg/pull/4158 >>>> 2. >>>> https://github.com/apache/iceberg-docs/pull/27#discussion_r800297155 >>>> >>> >> >> -- >> Best Regards >> >> Jeff Zhang >> > -- Best, Yufei `This is not a contribution`
