Hi JB, We've already excluded Spark's parquet dependency from iceberg-spark-runtime[1], and I don't think we should go back and block variant/geometry support. On the other hand, it's unlikely for Spark 4.0.x to bump to Parquet 1.16.x in a bug-fix release[2]. Hence, `useClassPathFirst=true` is the best solution I can see. We can add it to our documents.
1. https://github.com/apache/iceberg/blob/main/spark/v4.0/build.gradle#L80 2. https://github.com/apache/spark/pull/52165#issuecomment-3240831583 Regards, Manu On Fri, Nov 21, 2025 at 2:23 AM Jean-Baptiste Onofré <[email protected]> wrote: > Hi, > > While testing the 1.10.x and main branches, I encountered an issue > regarding Parquet dependency versions that needs clarification. > > I noticed a mismatch in the Parquet versions used by Spark itself and > the Iceberg Spark extension: > > - Spark 4.0.1 uses Parquet version 1.15.2. > - Iceberg Spark 1.10.0 uses Parquet version 1.16.0. > > If I set spark.executor.userClassPathFirst=true, the execution is > fine. However, with the default setting (useClassPathFirst=false), > running table maintenance actions (such as expireSnapshots) results in > a java.lang.NoSuchMethodError: > > org.apache.parquet.schema.LogicalTypeAnnotation$VariantLogicalTypeAnnotation > org.apache.parquet.schema.LogicalTypeAnnotation.variantType(byte). > This error originates within the Iceberg Spark > ParquetWithSparkSchemaVisitor. So, I suspect a change in variant (and > variant schema) in Parquet. > > This issue suggests there may be an incompatible change between > Parquet 1.15.2 and 1.16.0. Since shading does not seem to resolve > this, I wonder if we should enforce a consistent Parquet version > across both Spark and the Iceberg extension to prevent such conflicts. > > Do you have any thoughts on how to best address this dependency > mismatch for the upcoming 1.10.1 release? > > Regards, > JB > > On Mon, Sep 22, 2025 at 10:29 AM Amogh Jahagirdar <[email protected]> > wrote: > > > > Hey folks, > > > > Iceberg 1.10 was released 2 weeks ago and there was one issue around > incorrect variant filtering reported that I think meets the criteria for a > patch release. The fix PR is in (thank you Drew). I wanted to kick this > discussion thread off in case folks had other issues in the 1.10 release > that they think warrant a patch release. > > > > I also think this PR is a good candidate for a patch release; this is > for addressing a long-standing issue where closing the S3FileIO during an > event like moving broadcast variables from memory to disk leads to an > unexpected closing of the http client. There's still some discussion on the > approach of the fix but there's general recognition that it's a legitimate > issue, so I think it'd be ideal to get this in for a patch release as well. > > > > I've also created a milestone here. > > > > Thanks, > > > > Amogh Jahagirdar >
