zhangfengcdt commented on code in PR #7027: URL: https://github.com/apache/datafusion-comet/pull/7027#discussion_r4634403597
##########
docs/source/user-guide/latest/iceberg.md:
##########
@@ -535,26 +535,26 @@ iceberg-java resolves per-write options and
`spark.sql.iceberg.*` session overri
A write is eligible only when ALL of the following hold:
-| Setting
| Supported
values
|
-|
-------------------------------------------------------------------------------------------------------------------------------------------
|
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|
-| resolved write format (`write-format` option overlaid on
`write.format.default`)
| `parquet`
|
-| `format-version`
| `1`, `2` or
`3` (on `3`, writes that carry row lineage fall back; see below)
|
-| `write.parquet.compression-codec` / `compression-level` /
`row-group-size-bytes` / `page-size-bytes` / `page-row-limit` /
`dict-size-bytes` | the sizes/limit must be positive Java ints
(`Integer.parseInt` semantics — no trimming, no values past `Int.MaxValue` —
matching iceberg-java, whose writer fails on anything else);
`compression-level` must be a Java int within the native writer's per-codec
range (zstd 1–22, gzip 0–9, brotli 0–11; ignored for snappy/lz4/none), which
iceberg-java never validates, so other levels fall back; `brotli` needs
Hadoop's `BrotliCodec` class on the classpath (see below) |
-| `write.parquet.row-group-check-min-record-count`
| unset or `100`
(the default)
|
-| `write.parquet.row-group-check-max-record-count`
| unset or
`10000` (the default)
|
-| `write.parquet.page-version`
| unset or `v1`
|
-| `write.parquet.shred-variants`
| any value
(iceberg-java shreds only top-level `variant` columns, and a `variant` column
falls back on its type)
|
-| `write.parquet.variant-inference-buffer-size`
| any value
(only meaningful when shredding a `variant` column, which falls back)
|
-| `write.parquet.bloom-filter-enabled.column.<col>`
| unset or
`false`
|
-| `write.metadata.metrics.*`
| any value
(manifest metrics are re-derived on the JVM with Iceberg's own logic)
|
-| `write.spark.fanout.enabled`
| any value (the
native writer implements both clustered and fanout modes)
|
-| `write.target-file-size-bytes`
| any value (the
two writers can choose different roll points; see accepted divergences)
|
-| data location URI scheme
| `file`,
`memory`, `s3`, `s3a`, `gs`, matched case-sensitively (`S3://` falls back).
`s3`, `s3a` and `gs` need a bucket in the authority (`s3://bucket/...`), so a
hostless form such as `s3:/bucket/key` falls back. `gs` only when the `FileIO`
opening the data location is a `GCSFileIO`; see below
|
-| resolved `table.locationProvider()`
| Iceberg's
built-in `DefaultLocationProvider`
|
-| Hadoop S3A settings for an `s3` / `s3a` data location
|
`fs.s3a.access.key`, `secret.key`, `session.token`, `endpoint`,
`endpoint.region`, and `path.style.access`, plus the ignored, encryption and
credential settings described below, including their
`fs.s3a.bucket.<data-bucket>.*` forms; any other effective `fs.s3a.*` setting
falls back
|
-| Iceberg `FileIO` S3 settings for an `s3` / `s3a` data location
| the S3
endpoint, region, static/session credentials, path-style, SSE (`none`, `s3`,
`kms`, or `custom`; not `dsse-kms`), assume-role, anonymous/config-chain
settings parsed by the pinned iceberg-rust version, plus Comet's
credential-provider class and built-in web-identity properties. When a custom
provider is configured, its vendor-owned `s3.*` / `client.*` properties are
also forwarded; unsupported Iceberg-defined S3 settings still fall back
|
-| partition spec
| any, except a
`void` field whose source column was dropped beside a live field, or one
sourced from an `unknown` column (see below)
|
-| column types
| any except
`uuid` (Spark plans it as a string; no Arrow cast reaches `fixed(16)`), the v3
types `variant`, `timestamp_ns`, `geometry` and `geography`, `unknown` inside a
nested type, and a table whose columns are all `unknown`
|
+| Setting
| Supported
values
|
+|
-------------------------------------------------------------------------------------------------------------------------------------------
|
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|
+| resolved write format (`write-format` option overlaid on
`write.format.default`)
| `parquet`
|
+| `format-version`
| `1`, `2` or
`3` (on `3`, writes that carry row lineage fall back; see below)
|
+| `write.parquet.compression-codec` / `compression-level` /
`row-group-size-bytes` / `page-size-bytes` / `page-row-limit` /
`dict-size-bytes` | the codec must be a name iceberg-java accepts
(`uncompressed`, `snappy`, `gzip`, `lz4`, `zstd`, `brotli`, matched
case-insensitively and without trimming; `none` is ORC's spelling and falls
back, as do `lzo` and `lz4_raw`, which iceberg-java accepts but the native
writer cannot write); the sizes/limit must be positive Java ints
(`Integer.parseInt` semantics — no trimming, no values past `Int.MaxValue` —
matching iceberg-java, whose writer fails on anything else);
`compression-level` must be a Java int within the native writer's per-codec
range (zstd 1–22, gzip 0–9, brotli 0–11; ignored by both writers for
snappy/lz4/uncompressed) — iceberg-java never validates the level, so an
out-of-range value falls back rather than becoming a native task failure;
`brotli` needs Hadoop's `BrotliCodec` class on the classpath (see below) |
+| `write.parquet.row-group-check-min-record-count`
| unset or `100`
(the default); a value iceberg-java cannot parse as a Java int, such as one
with whitespace, falls back
|
+| `write.parquet.row-group-check-max-record-count`
| unset or
`10000` (the default); a value iceberg-java cannot parse as a Java int, such as
one with whitespace, falls back
|
+| `write.parquet.page-version`
| unset or `v1`,
spelled exactly (iceberg-java rejects `V1`)
|
Review Comment:
It looks like only v1.11 will reject "V1" value, but v1.10 actually accept
it and other values than "v1" that fallbacks to spark. Maybe reword it to be
more accurate.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
