[
https://issues.apache.org/jira/browse/FLINK-39342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ericzeng resolved FLINK-39342.
------------------------------
Fix Version/s: cdc-3.7.0
Resolution: Fixed
> [Iceberg] Support hadoop.conf.* prefix to pass Hadoop configuration
> properties
> -------------------------------------------------------------------------------
>
> Key: FLINK-39342
> URL: https://issues.apache.org/jira/browse/FLINK-39342
> Project: Flink
> Issue Type: New Feature
> Components: Flink CDC
> Affects Versions: cdc-3.5.0
> Reporter: ericzeng
> Priority: Major
> Labels: pull-request-available
> Fix For: cdc-3.7.0
>
>
> The Iceberg pipeline connector currently relies on classpath-based Hadoop
> configuration files (core-site.xml, hdfs-site.xml) to configure Hadoop
> settings. There is no way to pass Hadoop configuration properties (e.g., S3
>
> credentials, HDFS endpoint, Kerberos settings) directly through the
> connector's job configuration.
>
>
> *Motivation*
> In many deployment environments (containerized, cloud-native, or multi-tenant
> clusters), users cannot easily place Hadoop XML config files on the
> classpath. They need a way to set Hadoop properties programmatically via the
>
> pipeline job configuration — similar to how catalog.properties.* is already
> supported.
> *Description*
> Add support for a new hadoop.conf.* prefix in the Iceberg pipeline sink
> connector. Any property with this prefix will be stripped of the prefix and
> applied to the underlying Hadoop Configuration object before it is passed to
> CatalogUtil.buildIcebergCatalog(), IcebergWriter, IcebergCommitter, and
> CompactionOperator.
>
> *Example usage:*
> sink:
> type: iceberg
> catalog.properties.type: hadoop
> catalog.properties.warehouse: s3a://my-bucket/warehouse
> hadoop.conf.fs.s3a.access.key: xxxxxx
> hadoop.conf.fs.s3a.secret.key: xxx
> hadoop.conf.fs.s3a.endpoint: s3.us-east-1.amazonaws.com
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
