jordepic opened a new pull request, #5298:
URL: https://github.com/apache/datafusion-comet/pull/5298
## Which issue does this PR close?
(Partially, second of three) Closes #4322.
## Rationale for this change
The first part of this work (#4658) split the Iceberg V2 write into a
"writer" operator (inside AQE) and a "committer" operator, with the writer
still running through iceberg-java. The end goal is to replace the writer's
per-task parquet write with iceberg-rust — but only when the native write would
produce the same outcome as iceberg-java: the same parquet features,
statistics, and manifest metadata. If we can't reproduce a write exactly, we
should fall back and say why.
This PR adds that detection layer. It contains no native writing (that's
part three); it decides eligibility and reports the reason for every fall-back,
so the compatibility contract is reviewable and pinned by tests before the
native writer lands.
## What changes are included in this PR?
1) A `CometIcebergNativeWrite` operator serde, registered in `CometExecRule`
and gated by a new `spark.comet.write.iceberg.nativeAcceleration.enabled`
config (off by default). Its `getSupportLevel` inspects the write and either
accepts it or reports the first failing check via Comet's extended EXPLAIN;
conversion is stubbed ("not yet implemented") until part three.
2) The eligibility rules follow an allowlist philosophy: most Iceberg write
settings are NOT supported. Only vetted `write.parquet.*` keys are accepted (a
fixed set that the native writer will translate exactly, plus a few
value-checked ones like `page-version=v1` and row-group-check counts at their
defaults); any other `write.parquet.*` or `parquet.*` key — including keys
added by future Iceberg versions — falls back. Checks run on the effective
configuration (table properties overlaid with `SparkWrite.writeProperties`, so
per-write options and `spark.sql.iceberg.*` session overrides are honoured),
and also cover the resolved write format, format version, encryption, metrics
modes, custom FileIO/location providers, storage schemes, and `parquet.*` keys
in the session Hadoop configuration.
3) Documentation: `iceberg-writes.md` now states the supported configuration
as a whitelist table, documents the unconditional divergences accepted behind
the toggle (footer key-value metadata, page CRCs, encoding labels,
partition-path escaping, row-group/file-rolling cadence), and records the plan
to assemble manifest `DataFile` metrics on the JVM at commit time using
Iceberg's own `MetricsConfig`, which will let the remaining `counts`/`none`
metrics restrictions be lifted in part three.
4) Reflection helpers in `IcebergReflection` for reading `SparkWrite`
internals (format, table, writeProperties, data location), with every
reflection failure treated as a fall-back rather than a pass.
## How are these changes tested?
A new `CometIcebergWriteDetectionSuite` (32 tests) pins every rule: one
negative test per fall-back trigger asserting the reason string, positive tests
for the boundary cases (defaults, explicit metrics modes, session-conf codec
overrides, s3 data locations, partitioned tables), a smoke test that every
reflection accessor resolves against the Iceberg runtime on the classpath, and
a test that the planner registration records a fall-back reason on the write
exec. The suite passes on the spark-3.4 (Iceberg 1.5.2, Scala 2.12) and
spark-3.5 (Iceberg 1.8.1) profiles.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
