mmehrten commented on issue #15425: URL: https://github.com/apache/iceberg/issues/15425#issuecomment-4084178728
I'm able to get Kafka Connect to sink to S3 tables both with the REST catalog and with a modified version of the S3TablesCatalog ([updated for Iceberg 1.10](https://github.com/awslabs/s3-tables-catalog/pull/61)). I'm using Debezium as a source with one topic per-table, so I wonder if your transforms are causing an issue in this case. Here's an example of the config I'm using with Debezium in case it helps: ``` connector.class=org.apache.iceberg.connect.IcebergSinkConnector iceberg.catalog.io-impl=org.apache.iceberg.aws.s3.S3FileIO iceberg.catalog.client.region=us-east-1 iceberg.catalog.type=rest iceberg.catalog.uri=https://s3tables.us-east-1.amazonaws.com/iceberg iceberg.catalog.warehouse=arn:aws:s3tables:us-east-1:XXXXXX:bucket/mdm-table-example iceberg.catalog.rest.auth.type=sigv4 iceberg.catalog.rest.signing-name=s3tables iceberg.catalog.rest.signing-region=us-east-1 iceberg.catalog.rest.sigv4-enabled=true iceberg.kafka.session.timeout.ms=240000 iceberg.kafka.heartbeat.interval.ms=30000 iceberg.control.commit.timeout-ms=180000 iceberg.control.commit.interval-ms=600000 tasks.max=2 topics=salesdb.salesdb.CUSTOMER,salesdb.salesdb.CUSTOMER_SITE,salesdb.salesdb.SALES_ORDER,salesdb.salesdb.PRODUCT,salesdb.salesdb.SUPPLIER transforms=debezium transforms.debezium.type=org.apache.iceberg.connect.transforms.DebeziumTransform iceberg.tables.evolve-schema-enabled=true iceberg.tables.auto-create-enabled=true iceberg.tables.dynamic-enabled=true iceberg.tables.route-field=_cdc.source iceberg.table.write-format=parquet iceberg.tables.cdc-field=_cdc.op iceberg.tables.salesdb.customer-id-columns=cust_id iceberg.tables.salesdb.customer-site-id-columns=site_id iceberg.tables.salesdb.sales-order-id-columns=order_id iceberg.tables.salesdb.product-id-columns=product_id iceberg.tables.salesdb.supplier-id-columns=supplier_id iceberg.table.upsert-mode-enabled=true iceberg.table.write.upserts.enabled=true iceberg.table.write.delete.enabled=true iceberg.table.schema-evolution.enabled=true iceberg.table.commit.timeout-ms=300000 iceberg.control.commit.interval-ms=30000 iceberg.table.commit.retry.num-retries=3 iceberg.control.topic=control-iceberg value.converter=org.apache.kafka.connect.json.JsonConverter key.converter=org.apache.kafka.connect.json.JsonConverter key.converter.schemas.enable=false value.converter.schemas.enable=false ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
