[
https://issues.apache.org/jira/browse/HIVE-30129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denys Kuzmenko updated HIVE-30129:
----------------------------------
Description:
HiveIcebergOutputFormat.setWriterLevelConfiguration is meant to bridge HS2's
parquet.compression into Iceberg's write.parquet.compression-codec for any
table that sets no codec of its own:
{code}
if (table.properties().get(TableProperties.PARQUET_COMPRESSION) == null &&
jc.get(ParquetOutputFormat.COMPRESSION) != null) {
table.properties().put(TableProperties.PARQUET_COMPRESSION,
jc.get(ParquetOutputFormat.COMPRESSION));
}
{code}
In practice jc.get(ParquetOutputFormat.COMPRESSION) is null in the actual
Tez/LLAP write-task JobConf even when parquet.compression is set cluster-wide —
it isn't a registered HiveConf.ConfVars, so nothing guarantees it's copied into
a task's JobConf the way the ordinary (non-Iceberg) Parquet writer's config is.
The bridge silently never fires, and the write falls through to Iceberg's own
internal default (zstd since Iceberg 1.4.0), with no error and nothing visible
in SHOW TBLPROPERTIES.
*Impact*: any Iceberg table whose data is written fresh through Hive
(CTAS/INSERT, not a metadata-only register of pre-existing files) and that
doesn't set its own write.parquet.compression-codec, silently ignores the
cluster-wide codec setting
was:
HiveIcebergOutputFormat.setWriterLevelConfiguration is meant to bridge HS2's
parquet.compression into Iceberg's write.parquet.compression-codec for any
table that sets no codec of its own:
{code}
if (table.properties().get(TableProperties.PARQUET_COMPRESSION) == null &&
jc.get(ParquetOutputFormat.COMPRESSION) != null) {
table.properties().put(TableProperties.PARQUET_COMPRESSION,
jc.get(ParquetOutputFormat.COMPRESSION));
}
{code}
In practice jc.get(ParquetOutputFormat.COMPRESSION) (the raw parquet-mr key
parquet.compression) is null in the actual Tez/LLAP write-task JobConf, even
when it's set cluster-wide in hive-site.xml — it isn't a registered
HiveConf.ConfVars, so nothing in Hive's plan-serialization path guarantees it's
copied into a task's JobConf the way the ordinary (non-Iceberg) Parquet
writer's config is. The bridge silently never fires, and the write falls
through to Iceberg's own internal default (zstd since Iceberg 1.4.0 — confirmed
via TableProperties.PARQUET_COMPRESSION_DEFAULT_SINCE_1_4_0), with no error
and no trace in SHOW TBLPROPERTIES.
*Impact*: any Iceberg table that doesn't set its own
write.parquet.compression-codec TBLPROPERTIES silently ignores the cluster-wide
codec setting and gets Iceberg's default instead — defeating the only mechanism
Hive has for pinning one codec across Parquet and Iceberg tables.
> Iceberg: HiveIcebergOutputFormat silently drops the cluster's
> parquet.compression override, writing Iceberg's own default codec instead
> ---------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HIVE-30129
> URL: https://issues.apache.org/jira/browse/HIVE-30129
> Project: Hive
> Issue Type: Bug
> Components: Iceberg integration
> Reporter: Denys Kuzmenko
> Priority: Major
>
> HiveIcebergOutputFormat.setWriterLevelConfiguration is meant to bridge HS2's
> parquet.compression into Iceberg's write.parquet.compression-codec for any
> table that sets no codec of its own:
>
> {code}
>
>
> if (table.properties().get(TableProperties.PARQUET_COMPRESSION) == null &&
>
>
> jc.get(ParquetOutputFormat.COMPRESSION) != null) {
>
>
> table.properties().put(TableProperties.PARQUET_COMPRESSION,
> jc.get(ParquetOutputFormat.COMPRESSION));
>
> }
> {code}
>
>
>
>
>
> In practice jc.get(ParquetOutputFormat.COMPRESSION) is null in the actual
> Tez/LLAP write-task JobConf even when parquet.compression is set cluster-wide
> — it isn't a registered HiveConf.ConfVars, so nothing guarantees it's copied
> into a task's JobConf the way the ordinary (non-Iceberg) Parquet writer's
> config is. The bridge silently never fires, and the write falls through to
> Iceberg's own internal default (zstd since Iceberg 1.4.0), with no error and
> nothing visible in SHOW TBLPROPERTIES.
> *Impact*: any Iceberg table whose data is written fresh through Hive
> (CTAS/INSERT, not a metadata-only register of pre-existing files) and that
> doesn't set its own write.parquet.compression-codec, silently ignores the
> cluster-wide codec setting
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
