andygrove opened a new issue, #5691:
URL: https://github.com/apache/datafusion-comet/issues/5691
### Describe the bug
Writing to a table whose partition spec has been evolved (a partition field
and its source column dropped) panics inside iceberg-rust while Comet encodes
the manifest:
```
Comet native panic: panicked at itertools-0.13.0/src/zip_eq_impl.rs:49:17:
itertools: .zip_eq() reached end of one iterator before the other
at std::panicking::begin_panic::<&str>
at
<iceberg::spec::manifest::writer::ManifestWriter>::construct_partition_summaries
at
comet::execution::operators::iceberg_write::encode_data_files_as_manifest::{closure#0}
at <comet::execution::operators::iceberg_write::IcebergWriteExec as
...ExecutionPlan>::execute::{closure#0}
```
surfacing on the JVM side as `org.apache.comet.CometNativeException:
itertools: .zip_eq() reached end of one iterator before the other`, raised
through `CometIcebergWriteExec.drainAvroPayload`.
`construct_partition_summaries` zips the partition values of each data file
against the fields of a partition spec. After the spec has been evolved, those
two lengths disagree and `zip_eq` panics rather than returning an error.
Two problems here: the write fails at all, and it fails by panicking across
the JNI boundary rather than by returning a `Result`.
### Steps to reproduce
Spark 4.1.3, Iceberg 1.11.0, `spark.comet.iceberg.write.enabled=true` plus
the Iceberg Spark SQL test setup from `dev/diffs/iceberg/1.11.0.diff`:
```
./gradlew -DsparkVersions=4.1 -DscalaVersion=2.13 -DflinkVersions=
-DkafkaVersions= \
:iceberg-spark:iceberg-spark-extensions-4.1_2.13:test \
--tests '*TestAlterTablePartitionFields*' -Pquick=true -x javadoc
```
Failing cases, all of which drop a partition field and then its source
column before writing again:
- `testDropPartitionAndSourceColumnLong`
- `testDropPartitionAndSourceColumnTimestamp`
- `deleteAfterDroppingPartitionAndSourceColumn`
### Expected behavior
Writing to a table with an evolved partition spec succeeds and produces the
same manifests as iceberg-java. Failing that, the native writer returns an
error that Comet can turn into a fallback or a clean Spark exception, and never
panics.
### Additional context
Found by turning the two Iceberg write flags on by default in #5677.
Shares a trigger with the sibling failure in the same test class filed
separately: `testReaddColumnAfterIdentityPartitionDrop` fails with `No column
with source column id 2 in schema` from the same
`encode_data_files_as_manifest` phase. A single fix to how the writer resolves
the partition spec for already-written data files may close both.
Part of #5649.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
