[
https://issues.apache.org/jira/browse/HIVE-30129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denys Kuzmenko updated HIVE-30129:
----------------------------------
Description:
HiveIcebergOutputFormat.setWriterLevelConfiguration is meant to bridge HS2's
parquet.compression into Iceberg's write.parquet.compression-codec for any
table that sets no codec of its own:
{code}
if (table.properties().get(TableProperties.PARQUET_COMPRESSION) == null &&
jc.get(ParquetOutputFormat.COMPRESSION) != null) {
table.properties().put(TableProperties.PARQUET_COMPRESSION,
jc.get(ParquetOutputFormat.COMPRESSION));
}
{code}
In practice jc.get(ParquetOutputFormat.COMPRESSION) (the raw parquet-mr key
parquet.compression) is null in the actual Tez/LLAP write-task JobConf, even
when it's set cluster-wide in hive-site.xml — it isn't a registered
HiveConf.ConfVars, so nothing in Hive's plan-serialization path guarantees it's
copied into a task's JobConf the way the ordinary (non-Iceberg) Parquet
writer's config is. The bridge silently never fires, and the write falls
through to Iceberg's own internal default (zstd since Iceberg 1.4.0 — confirmed
via TableProperties.PARQUET_COMPRESSION_DEFAULT_SINCE_1_4_0), with no error
and no trace in SHOW TBLPROPERTIES.
*Impact*: any Iceberg table that doesn't set its own
write.parquet.compression-codec TBLPROPERTIES silently ignores the cluster-wide
codec setting and gets Iceberg's default instead — defeating the only mechanism
Hive has for pinning one codec across Parquet and Iceberg tables.
was:
HiveIcebergOutputFormat.setWriterLevelConfiguration is meant to bridge HS2's
parquet.compression into Iceberg's write.parquet.compression-codec for any
table that sets no codec of its own:
{code}
if (table.properties().get(TableProperties.PARQUET_COMPRESSION) == null &&
jc.get(ParquetOutputFormat.COMPRESSION) != null) {
table.properties().put(TableProperties.PARQUET_COMPRESSION,
jc.get(ParquetOutputFormat.COMPRESSION));
}
{code}
In practice jc.get(ParquetOutputFormat.COMPRESSION) (the raw parquet-mr key
parquet.compression) is null in the actual Tez/LLAP write-task JobConf, even
when it's set cluster-wide in hive-site.xml — it isn't a registered
HiveConf.ConfVars, so nothing in Hive's plan-serialization path guarantees it's
copied into a task's JobConf the way the ordinary (non-Iceberg) Parquet
writer's config is. The bridge silently never fires, and the write falls
through to Iceberg's own internal default (zstd since Iceberg 1.4.0 — confirmed
via TableProperties.PARQUET_COMPRESSION_DEFAULT_SINCE_1_4_0), with no error
and no trace in SHOW TBLPROPERTIES.
> Iceberg: HiveIcebergOutputFormat silently drops the cluster's
> parquet.compression override, writing Iceberg's own default codec instead
> ---------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HIVE-30129
> URL: https://issues.apache.org/jira/browse/HIVE-30129
> Project: Hive
> Issue Type: Bug
> Reporter: Denys Kuzmenko
> Priority: Major
>
> HiveIcebergOutputFormat.setWriterLevelConfiguration is meant to bridge HS2's
> parquet.compression into Iceberg's write.parquet.compression-codec for any
> table that sets no codec of its own:
>
> {code}
>
>
> if (table.properties().get(TableProperties.PARQUET_COMPRESSION) == null &&
>
>
> jc.get(ParquetOutputFormat.COMPRESSION) != null) {
>
>
> table.properties().put(TableProperties.PARQUET_COMPRESSION,
> jc.get(ParquetOutputFormat.COMPRESSION));
>
> }
> {code}
>
>
>
>
>
> In practice jc.get(ParquetOutputFormat.COMPRESSION) (the raw parquet-mr key
> parquet.compression) is null in the actual Tez/LLAP write-task JobConf, even
> when it's set cluster-wide in hive-site.xml — it isn't a registered
> HiveConf.ConfVars, so nothing in Hive's plan-serialization path guarantees
> it's copied into a task's JobConf the way the ordinary (non-Iceberg) Parquet
> writer's config is. The bridge silently never fires, and the write falls
> through to Iceberg's own internal default (zstd since Iceberg 1.4.0 —
> confirmed via TableProperties.PARQUET_COMPRESSION_DEFAULT_SINCE_1_4_0), with
> no error and no trace in SHOW TBLPROPERTIES.
> *Impact*: any Iceberg table that doesn't set its own
> write.parquet.compression-codec TBLPROPERTIES silently ignores the
> cluster-wide codec setting and gets Iceberg's default instead — defeating the
> only mechanism Hive has for pinning one codec across Parquet and Iceberg
> tables.
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
