This is an automated email from the ASF dual-hosted git repository.
JNSimba pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 5c50da3a3b1 [fix](streaming-job) Encode CDC stream load records as
UTF-8 (#66771)
5c50da3a3b1 is described below
commit 5c50da3a3b120baef8329c01a5f4a0e0b805944b
Author: wudi <[email protected]>
AuthorDate: Fri Aug 21 20:34:21 2026 +0800
[fix](streaming-job) Encode CDC stream load records as UTF-8 (#66771)
### What problem does this PR solve?
Problem Summary:
CDC streaming jobs encoded JSON records with `String.getBytes()`, so the
stream-load payload depended on the CDC Client JVM default charset. On
JVMs using a non-UTF-8 default charset, characters such as Chinese text
and emoji were replaced with question marks before reaching Doris.
Encode records explicitly as UTF-8, matching the existing CDC fetch path
and the stream-load JSON contract.
Before the change, `MySqlCharsetITCase` running with
`-Dfile.encoding=US-ASCII` failed with `expected: "测试数据😀" but was:
"?????"`. After the change, the same test passes without requiring the
CDC Client JVM default charset to be UTF-8.
---
.../java/org/apache/doris/cdcclient/service/PipelineCoordinator.java | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git
a/fs_brokers/cdc_client/src/main/java/org/apache/doris/cdcclient/service/PipelineCoordinator.java
b/fs_brokers/cdc_client/src/main/java/org/apache/doris/cdcclient/service/PipelineCoordinator.java
index 7e96a030df7..8f3de9fbf5c 100644
---
a/fs_brokers/cdc_client/src/main/java/org/apache/doris/cdcclient/service/PipelineCoordinator.java
+++
b/fs_brokers/cdc_client/src/main/java/org/apache/doris/cdcclient/service/PipelineCoordinator.java
@@ -621,7 +621,8 @@ public class PipelineCoordinator {
String dorisTable =
targetTableMappings.getOrDefault(table, table);
for (String record : result.getRecords()) {
scannedRows++;
- batchStreamLoad.writeRecord(targetDb, dorisTable,
record.getBytes());
+ batchStreamLoad.writeRecord(
+ targetDb, dorisTable,
record.getBytes(StandardCharsets.UTF_8));
}
// Mark last message as data (not heartbeat)
lastMessageIsHeartbeat = false;
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
