xiasongh commented on code in PR #11313:
URL: https://github.com/apache/iceberg/pull/11313#discussion_r1798594599
##########
kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/data/SinkWriter.java:
##########
@@ -133,6 +145,20 @@ private String extractRouteValue(Object recordValue,
String routeField) {
return routeValue == null ? null : routeValue.toString();
}
+ private String formatRoutePattern(SinkRecord record, String routePattern) {
+ if (routePattern == null) {
+ return null;
+ }
+
+ String topicName = record.topic();
+ if (topicName == null) {
+ return null;
+ }
+
+ // replace topic namespace separator
+ return routePattern.replace("{topic}", topicName.replace(".", "_"));
Review Comment:
> topicName.replace(".", "_")
I use AWS Glue catalog, which doesn't support nested namespaces. Topic names
with more than 1 `.` would be invalid table names
The Debezium JDBC sink connector [0] also does this, so it's not totally
unheard of
It probably makes most sense to turn this into it's own config, maybe
something like `iceberg.tables.route-pattern.namespace-separator`?
[0]
https://debezium.io/documentation/reference/stable/connectors/jdbc.html#jdbc-property-table-naming-strategy
##########
kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/data/SinkWriter.java:
##########
@@ -133,6 +145,20 @@ private String extractRouteValue(Object recordValue,
String routeField) {
return routeValue == null ? null : routeValue.toString();
}
+ private String formatRoutePattern(SinkRecord record, String routePattern) {
+ if (routePattern == null) {
+ return null;
+ }
+
+ String topicName = record.topic();
+ if (topicName == null) {
+ return null;
+ }
+
+ // replace topic namespace separator
+ return routePattern.replace("{topic}", topicName.replace(".", "_"));
Review Comment:
> topicName.replace(".", "_")
I use AWS Glue catalog, which doesn't support nested namespaces. Topic names
with more than 1 `.` would be invalid table names
The Debezium JDBC sink connector [0] also does this, so it's not totally
unheard of
It probably makes most sense to turn this into it's own config, maybe
something like `iceberg.tables.route-pattern.namespace-separator`?
[0]
https://debezium.io/documentation/reference/stable/connectors/jdbc.html#jdbc-property-table-naming-strategy
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
