Hi, While testing the 1.10.x and main branches, I encountered an issue regarding Parquet dependency versions that needs clarification.
I noticed a mismatch in the Parquet versions used by Spark itself and the Iceberg Spark extension: - Spark 4.0.1 uses Parquet version 1.15.2. - Iceberg Spark 1.10.0 uses Parquet version 1.16.0. If I set spark.executor.userClassPathFirst=true, the execution is fine. However, with the default setting (useClassPathFirst=false), running table maintenance actions (such as expireSnapshots) results in a java.lang.NoSuchMethodError: org.apache.parquet.schema.LogicalTypeAnnotation$VariantLogicalTypeAnnotation org.apache.parquet.schema.LogicalTypeAnnotation.variantType(byte). This error originates within the Iceberg Spark ParquetWithSparkSchemaVisitor. So, I suspect a change in variant (and variant schema) in Parquet. This issue suggests there may be an incompatible change between Parquet 1.15.2 and 1.16.0. Since shading does not seem to resolve this, I wonder if we should enforce a consistent Parquet version across both Spark and the Iceberg extension to prevent such conflicts. Do you have any thoughts on how to best address this dependency mismatch for the upcoming 1.10.1 release? Regards, JB On Mon, Sep 22, 2025 at 10:29 AM Amogh Jahagirdar <[email protected]> wrote: > > Hey folks, > > Iceberg 1.10 was released 2 weeks ago and there was one issue around > incorrect variant filtering reported that I think meets the criteria for a > patch release. The fix PR is in (thank you Drew). I wanted to kick this > discussion thread off in case folks had other issues in the 1.10 release that > they think warrant a patch release. > > I also think this PR is a good candidate for a patch release; this is for > addressing a long-standing issue where closing the S3FileIO during an event > like moving broadcast variables from memory to disk leads to an unexpected > closing of the http client. There's still some discussion on the approach of > the fix but there's general recognition that it's a legitimate issue, so I > think it'd be ideal to get this in for a patch release as well. > > I've also created a milestone here. > > Thanks, > > Amogh Jahagirdar
