jackylee-ch opened a new issue, #12698:
URL: https://github.com/apache/gluten/issues/12698
### Description
Gluten currently has **no Iceberg-specific configuration entry at all** —
neither for read nor for
write. Searching the whole repository for `buildConf` / `buildStaticConf` /
`registerConf` keys
containing `iceberg` returns nothing.
As a result, operators who need to turn Iceberg offload off (bad data, a
validation gap, a
regression after an upgrade) have to reach for a switch that is far broader
than Iceberg:
| Path | Only way to disable today | Problem |
|---|---|---|
| Read | `spark.gluten.sql.columnar.batchscan` | Generic DSv2 switch.
Turning it off also falls back Paimon, Hudi and every other batch scan. |
| Write | `spark.gluten.sql.enable.enhancedFeatures` | A bundled flag
(`VeloxConfig.scala`), documented as "iceberg native write **and other
features**". Default `true`. |
#### Details
**Read.** `OffloadIcebergScan`
(`gluten-iceberg/.../extension/OffloadIcebergScan.scala`) is gated
only by `Validators.FallbackByUserOptions`:
```scala
case p: BatchScanExec if !glutenConf.enableColumnarBatchScan => fail(p)
```
There is no way to express "fall back Iceberg scans but keep other DSv2
scans offloaded".
**Write.** The five `OffloadIcebergWrite` rules
(`backends-velox/src-iceberg/.../extension/OffloadIcebergWrite.scala`) are
gated by
`VeloxBackend.supportAppendDataExec()` and friends, all of which return
`VeloxConfig.get.enableEnhancedFeatures()`. Two consequences:
1. The `enhancedFeatures` flag conflates "was the native lib built with
enhanced features" with
"does the user want Iceberg native write". Today Iceberg write happens to
be its only member, but
the doc string already says "and other features" — a second member
removes the ability to target
Iceberg alone.
2. The build-time half of that gate is already gone:
`isEnhancedFeaturesEnabled` in
`cpp/velox/jni/VeloxJniWrapper.cc` now unconditionally returns `true` (the
`#ifdef GLUTEN_ENABLE_ENHANCED_FEATURES` was dropped). Iceberg native
write is therefore on by
default, with the bundled SQL conf as the only remaining gate. #12660
similarly notes "Iceberg is
not dependent on the enhanced tag anymore".
3. `supportAppendDataExec()` is also read once at **rule-injection time**
(`VeloxRuleApi.scala`, to decide whether to inject
`RewriteCreateTableAsSelect`), so flipping
`enhancedFeatures` at runtime only affects the validator half — it is not
a clean kill switch.
Iceberg write validation carries a long list of restrictions (Parquet only,
no sort order, no
UUID/FIXED, no merge schema, no accept-any-schema, no float/double or nested
partition columns,
no incompatible column names — see `IcebergWriteExec.doValidateInternal`). A
write-path problem
means bad files rather than a slow query, so a switch that disables *only*
Iceberg write, without
touching anything else, is worth having on its own.
#### Proposal
Add one switch per direction, both defaulting to `true` so behaviour is
unchanged:
```
spark.gluten.sql.columnar.iceberg.enableNativeRead
spark.gluten.sql.columnar.backend.velox.iceberg.enableNativeWrite
```
The asymmetry in the key names is deliberate and follows where the code
lives:
- `OffloadIcebergScan` is in `gluten-iceberg/` and is shared by **both**
backends
(`VeloxIcebergComponent` and `CHIcebergComponent` both call it), so the
read key must not carry a
`backend.velox` prefix.
- `OffloadIcebergWrite` is in `backends-velox/src-iceberg/` and is
Velox-only. The write key mirrors
the existing Delta precedent,
`spark.gluten.sql.columnar.backend.velox.delta.enableNativeWrite`
(`VeloxDeltaConfig`), which is checked with an early return in
`OffloadDeltaCommand`.
Checks belong inside the offload rules (not at injection time), so the
switches stay dynamic.
The write switch should be AND-ed with `enhancedFeatures` rather than
replacing it.
One known gap to decide on: `AllVeloxConfiguration` only walks
`VeloxConfig.allEntries`, so
component-level config objects are not picked up by the docs generator —
which is why
`VeloxDeltaConfig`'s key is absent from `docs/velox-configuration.md` today.
Either accept the gap
or extend the generator to cover component configs.
Related but distinct: #11703 (mapping Iceberg *table properties* to Velox
write configs), #8953
(Iceberg support tracker).
This enhancement description was written with the assistance of AI tooling.
### Gluten version
main branch
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
