jackylee-ch opened a new pull request, #12699:
URL: https://github.com/apache/gluten/pull/12699
## What changes are proposed in this pull request?
Gluten has no Iceberg-specific configuration entry today, so disabling
Iceberg offload means
reaching for a much broader switch:
| Path | Only way to disable before this PR | Problem |
|---|---|---|
| Read | `spark.gluten.sql.columnar.batchscan` | Generic DSv2 switch — also
falls back Paimon, Hudi and every other batch scan. |
| Write | `spark.gluten.sql.enable.enhancedFeatures` | Bundled flag,
documented as "iceberg native write **and other features**". |
This PR adds two backend-agnostic switches, both defaulting to `true` so
behaviour is unchanged:
```
spark.gluten.sql.columnar.iceberg.enableNativeRead
spark.gluten.sql.columnar.iceberg.enableNativeWrite
```
They are declared in a new `GlutenIcebergConfig` in `gluten-iceberg` rather
than in a backend
module, because `OffloadIcebergScan` is shared by `VeloxIcebergComponent`
and `CHIcebergComponent`,
and a backend that later gains Iceberg write support should honour the same
write key instead of
introducing its own.
Both switches are checked inside the offload rules (`OffloadIcebergScanBase`
/
`OffloadIcebergWriteBase`) rather than at rule-injection time, so they stay
modifiable per session.
The write switch is AND-ed with the existing `enhancedFeatures` gate rather
than replacing it.
Also documents both keys in `docs/get-started/VeloxIceberg.md`. Note they do
not appear in
`docs/velox-configuration.md`, because `AllVeloxConfiguration` only walks
`VeloxConfig.allEntries` —
the same reason `VeloxDeltaConfig`'s key is absent there today. Extending
that generator to cover
component-level configs is left as a follow-up.
## How was this patch tested?
New unit suites covering the gate itself, verified by inverting the
condition and watching both
assertions fail before restoring it:
- `gluten-iceberg`: `GlutenIcebergConfigSuite` (key names, defaults, runtime
modifiability,
reading from the active `SQLConf`), `OffloadIcebergScanSuite` (gate blocks
offload, gate is
consulted per call rather than captured at rule construction).
- `backends-velox`: `OffloadIcebergWriteSuite` (same for the write gate).
New end-to-end tests:
- `IcebergSuite` — Iceberg scan falls back to `BatchScanExec` with the read
switch off, offload
resumes when it is restored, and a Parquet scan in the same session stays
offloaded
(`FileSourceScanExecTransformer`), which is the behaviour the generic
batch-scan switch cannot
express.
- `VeloxIcebergSuite` (enhanced) — Iceberg write falls back to vanilla
`AppendDataExec` with the
write switch off while reads stay offloaded, and offload resumes when it
is restored.
Ran locally on macOS with `-Pspark-3.5 -Pscala-2.12 -Pbackends-velox
-Piceberg`: the three unit
suites pass (7 tests). `spotless:check`, `scalastyle:check` and
`checkstyle:check` pass for both
modules, and `./dev/format-scala-code.sh` is applied. The two end-to-end
suites need a native
build and are left to CI.
## Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude claude-opus-4-8
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
