This is an automated email from the ASF dual-hosted git repository.
voonhous pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hudi.git
The following commit(s) were added to refs/heads/master by this push:
new a5b240368874 fix(core): decode strings read from storage as UTF-8
(#19454)
a5b240368874 is described below
commit a5b2403688749f49416ffbb90e427f33c8299dbe
Author: Lin Liu <[email protected]>
AuthorDate: Fri Aug 14 05:24:36 2026 -0700
fix(core): decode strings read from storage as UTF-8 (#19454)
* [MINOR] Decode log block header/footer metadata as UTF-8
HoodieLogBlock.getLogMetadata() read each metadata entry with new
String(byte[]),
which applies the platform default charset. The write side
(getLogMetadataBytes) serializes those values with
StringUtils.getUTF8Bytes(),
so the two sides disagree on any JVM whose default charset is not UTF-8.
The visible consequence is a corrupted header value on read. When the value
is
the writer schema and it contains non-ASCII field names, the corrupted
string
then fails Avro parsing with "Illegal initial character".
Decode with StringUtils.fromUTF8Bytes() so the read matches the write. That
method already lives in the same StringUtils class this file imports
getUTF8Bytes from, so no new dependency is introduced.
Adds a round-trip test asserting a schema with a non-ASCII field name
survives
getHeaderMetadataBytes -> getHeaderMetadata and still parses as Avro.
Note on the test's scope: the old and new code agree whenever the JVM
default
charset is already UTF-8, so this test passes before and after the change
on a
UTF-8 JVM. It documents the invariant rather than reproducing the failure;
reproducing that requires running with a non-UTF-8 default charset, e.g.
-Dfile.encoding=ISO-8859-1, under which new String(utf8Bytes) demonstrably
mojibakes.
* Sweep the decode side onto UTF-8 and add a checkstyle guard
HUDI-6825 (#9634) moved every encode site to StringUtils.getUTF8Bytes but
left the matching decode sites on the platform default charset, so any
non-ASCII value written to storage comes back corrupted on a JVM whose
default charset is not UTF-8. StringUtils.fromUTF8Bytes did not exist
until #10574, four months later, which is why no decode sweep followed.
Beyond the log block header this fixes:
- OrcUtils.readSchema, the other half of the pair #9634 created. The
writer (HoodieAvroOrcWriter) was converted, the reader was not, so a
MOR table with ORC base files and non-ASCII field names still failed.
- RocksDBDAO prefix search and delete, where the decode is a loop
condition, so a non-ASCII partition path silently returned no file
groups instead of mojibake.
- HoodieTableMetaClient index metadata, UtilHelpers schema files,
JdbcSource password files, LockAuditingCommand config, and several
toString/exception paths.
StringUtils.toHexString moves to String.valueOf: it takes the char[]
constructor, not the byte[] one, so it was never a charset bug.
A RegexpSinglelineJava rule now fails the build on a charset-less
new String(...) in main sources, which is what actually stops this from
regressing again. Test sources are suppressed since their bytes are
built in-process.
The log block test folds the non-ASCII case into testHeaderMetadata and
replaces the round-trip assertion with one on the bytes on the wire. The
round trip cannot fail on a UTF-8 JVM, where the old and new code are
the same call, so it was never the regression guard it looked like.
* Fix checkstyle suppression against the 1.0 suppressions DTD
The suppression used an id attribute, which the 1.0 suppressions DTD
does not declare, and omitted checks, which that DTD marks #REQUIRED.
Checkstyle therefore failed to parse checkstyle-suppressions.xml and
every Maven module failed at the compile phase. Only the trino plugin
passed, since it uses its own checkstyle config.
Suppress by check name instead, which the 1.0 DTD does allow. There is
one RegexpSinglelineJava module in the config, so this is equivalent
without having to bump the DTD version for the whole file.
Validated both config files with xmllint against the DTDs bundled in
checkstyle 9.3: the previous revision reports "No declaration for
attribute id" plus "does not carry attribute checks", and this one
validates clean.
---------
Co-authored-by: Lokesh Jain <[email protected]>
Co-authored-by: voon <[email protected]>
---
.../hudi/cli/commands/LockAuditingCommand.java | 3 ++-
.../common/model/SerializableIndexedRecord.java | 3 ++-
.../hudi/common/table/HoodieTableMetaClient.java | 3 ++-
.../common/table/log/block/HoodieLogBlock.java | 6 ++++-
.../hudi/common/util/collection/RocksDBDAO.java | 13 +++++-----
.../hudi/common/util/hash/ColumnIndexID.java | 3 ++-
.../apache/hudi/common/util/hash/FileIndexID.java | 3 ++-
.../hudi/common/util/hash/PartitionIndexID.java | 3 ++-
.../common/table/log/block/TestHoodieLogBlock.java | 29 ++++++++++++++++++++--
.../java/org/apache/hudi/common/util/OrcUtils.java | 4 ++-
.../org/apache/hudi/common/util/StringUtils.java | 2 +-
.../org/apache/hudi/io/hfile/HFileBlockType.java | 2 +-
.../org/apache/hudi/io/hfile/UTF8StringKey.java | 4 ++-
.../org/apache/hudi/utilities/UtilHelpers.java | 2 +-
.../apache/hudi/utilities/sources/JdbcSource.java | 2 +-
.../hudi/utilities/util/BloomFilterData.java | 4 ++-
style/checkstyle-suppressions.xml | 3 +++
style/checkstyle.xml | 9 +++++++
18 files changed, 76 insertions(+), 22 deletions(-)
diff --git
a/hudi-cli/src/main/java/org/apache/hudi/cli/commands/LockAuditingCommand.java
b/hudi-cli/src/main/java/org/apache/hudi/cli/commands/LockAuditingCommand.java
index 3a0ebe694da6..b06786a6ad5a 100644
---
a/hudi-cli/src/main/java/org/apache/hudi/cli/commands/LockAuditingCommand.java
+++
b/hudi-cli/src/main/java/org/apache/hudi/cli/commands/LockAuditingCommand.java
@@ -21,6 +21,7 @@ package org.apache.hudi.cli.commands;
import org.apache.hudi.cli.HoodieCLI;
import
org.apache.hudi.client.transaction.lock.audit.StorageLockProviderAuditService;
import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.StringUtils;
import org.apache.hudi.io.util.FileIOUtils;
import org.apache.hudi.storage.StoragePath;
import org.apache.hudi.storage.StoragePathInfo;
@@ -278,7 +279,7 @@ public class LockAuditingCommand {
// Read and parse the configuration
String configContent;
try (InputStream inputStream = HoodieCLI.storage.open(configPath)) {
- configContent = new String(FileIOUtils.readAsByteArray(inputStream));
+ configContent =
StringUtils.fromUTF8Bytes(FileIOUtils.readAsByteArray(inputStream));
} catch (FileNotFoundException e) {
return String.format("Lock Audit Status: DISABLED\n"
+ "Table: %s\n"
diff --git
a/hudi-common/src/main/java/org/apache/hudi/common/model/SerializableIndexedRecord.java
b/hudi-common/src/main/java/org/apache/hudi/common/model/SerializableIndexedRecord.java
index 351fd13a06e9..d1f3cd44b96f 100644
---
a/hudi-common/src/main/java/org/apache/hudi/common/model/SerializableIndexedRecord.java
+++
b/hudi-common/src/main/java/org/apache/hudi/common/model/SerializableIndexedRecord.java
@@ -20,6 +20,7 @@
package org.apache.hudi.common.model;
import org.apache.hudi.common.avro.HoodieAvroUtils;
+import org.apache.hudi.common.util.StringUtils;
import org.apache.hudi.common.util.ValidationUtils;
import org.apache.hudi.common.util.VisibleForTesting;
import org.apache.hudi.exception.HoodieIOException;
@@ -189,6 +190,6 @@ public class SerializableIndexedRecord implements
GenericRecord, KryoSerializabl
@Override
public String toString() {
- return record == null ? "SERIALIZED: " + new String(recordBytes) :
record.toString();
+ return record == null ? "SERIALIZED: " +
StringUtils.fromUTF8Bytes(recordBytes) : record.toString();
}
}
diff --git
a/hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableMetaClient.java
b/hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableMetaClient.java
index 1aaca2519282..fbeb62fe0158 100644
---
a/hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableMetaClient.java
+++
b/hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableMetaClient.java
@@ -103,6 +103,7 @@ import static
org.apache.hudi.common.table.HoodieTableConfig.inferMergingConfigs
import static
org.apache.hudi.common.table.HoodieTableConfig.inferMergingConfigsForV9TableCreation;
import static org.apache.hudi.common.util.ConfigUtils.containsConfigProperty;
import static org.apache.hudi.common.util.ConfigUtils.getStringWithAltKeys;
+import static org.apache.hudi.common.util.StringUtils.fromUTF8Bytes;
import static org.apache.hudi.common.util.StringUtils.getUTF8Bytes;
import static org.apache.hudi.common.util.ValidationUtils.checkArgument;
import static org.apache.hudi.common.util.ValidationUtils.checkState;
@@ -355,7 +356,7 @@ public class HoodieTableMetaClient implements Serializable {
try {
Option<byte[]> bytesOpt = FileIOUtils.readDataFromPath(storage,
indexDefinitionPath, true);
if (bytesOpt.isPresent()) {
- return Option.of(HoodieIndexMetadata.fromJson(new
String(bytesOpt.get())));
+ return
Option.of(HoodieIndexMetadata.fromJson(fromUTF8Bytes(bytesOpt.get())));
} else {
return Option.of(new HoodieIndexMetadata());
}
diff --git
a/hudi-common/src/main/java/org/apache/hudi/common/table/log/block/HoodieLogBlock.java
b/hudi-common/src/main/java/org/apache/hudi/common/table/log/block/HoodieLogBlock.java
index c6cb16d10dee..5ca0db25c13a 100644
---
a/hudi-common/src/main/java/org/apache/hudi/common/table/log/block/HoodieLogBlock.java
+++
b/hudi-common/src/main/java/org/apache/hudi/common/table/log/block/HoodieLogBlock.java
@@ -56,6 +56,7 @@ import java.util.function.Supplier;
import java.util.stream.Collectors;
import static
org.apache.hudi.common.model.HoodieRecordLocation.isPositionValid;
+import static org.apache.hudi.common.util.StringUtils.fromUTF8Bytes;
import static org.apache.hudi.common.util.StringUtils.getUTF8Bytes;
import static org.apache.hudi.common.util.ValidationUtils.checkState;
@@ -468,7 +469,10 @@ public abstract class HoodieLogBlock {
int metadataEntrySize = dis.readInt();
byte[] metadataEntry = new byte[metadataEntrySize];
dis.readFully(metadataEntry, 0, metadataEntrySize);
- metadata.put(typeMapper.apply(metadataEntryIndex), new
String(metadataEntry));
+ // Must match getLogMetadataBytes(), which writes these values as
UTF-8. Decoding with the
+ // platform default charset corrupts non-ASCII values: loudly for a
schema name, which Avro
+ // then rejects, and silently for a doc, a default or a prop.
+ metadata.put(typeMapper.apply(metadataEntryIndex),
fromUTF8Bytes(metadataEntry));
metadataCount--;
}
return metadata;
diff --git
a/hudi-common/src/main/java/org/apache/hudi/common/util/collection/RocksDBDAO.java
b/hudi-common/src/main/java/org/apache/hudi/common/util/collection/RocksDBDAO.java
index c9d5f83babfb..a36f3dc711db 100644
---
a/hudi-common/src/main/java/org/apache/hudi/common/util/collection/RocksDBDAO.java
+++
b/hudi-common/src/main/java/org/apache/hudi/common/util/collection/RocksDBDAO.java
@@ -59,6 +59,7 @@ import java.util.concurrent.ConcurrentHashMap;
import java.util.stream.Collectors;
import java.util.stream.Stream;
+import static org.apache.hudi.common.util.StringUtils.fromUTF8Bytes;
import static org.apache.hudi.common.util.StringUtils.getUTF8Bytes;
/**
@@ -153,8 +154,8 @@ public class RocksDBDAO {
for (int index = 0; index < managedHandles.size(); index++) {
ColumnFamilyHandle handle = managedHandles.get(index);
ColumnFamilyDescriptor descriptor = managedColumnFamilies.get(index);
- String familyNameFromHandle = new String(handle.getName());
- String familyNameFromDescriptor = new String(descriptor.getName());
+ String familyNameFromHandle = fromUTF8Bytes(handle.getName());
+ String familyNameFromDescriptor = fromUTF8Bytes(descriptor.getName());
ValidationUtils.checkArgument(familyNameFromDescriptor.equals(familyNameFromHandle),
"Family Handles not in order with descriptors");
@@ -385,11 +386,11 @@ public class RocksDBDAO {
List<Pair<String, T>> results = new LinkedList<>();
try (final RocksIterator it =
getRocksDB().newIterator(managedHandlesMap.get(columnFamilyName))) {
it.seek(getUTF8Bytes(prefix));
- while (it.isValid() && new String(it.key()).startsWith(prefix)) {
+ while (it.isValid() && fromUTF8Bytes(it.key()).startsWith(prefix)) {
long beginTs = System.nanoTime();
T val = deserializePayload(columnFamilyName, it.value());
timeTakenMicro += ((System.nanoTime() - beginTs) / 1000);
- results.add(Pair.of(new String(it.key()), val));
+ results.add(Pair.of(fromUTF8Bytes(it.key()), val));
it.next();
}
}
@@ -425,8 +426,8 @@ public class RocksDBDAO {
// Find first and last keys to be deleted
String firstEntry = null;
String lastEntry = null;
- while (it.isValid() && new String(it.key()).startsWith(prefix)) {
- String result = new String(it.key());
+ while (it.isValid() && fromUTF8Bytes(it.key()).startsWith(prefix)) {
+ String result = fromUTF8Bytes(it.key());
it.next();
if (firstEntry == null) {
firstEntry = result;
diff --git
a/hudi-common/src/main/java/org/apache/hudi/common/util/hash/ColumnIndexID.java
b/hudi-common/src/main/java/org/apache/hudi/common/util/hash/ColumnIndexID.java
index 92e60b30a311..375ddc7ae84e 100644
---
a/hudi-common/src/main/java/org/apache/hudi/common/util/hash/ColumnIndexID.java
+++
b/hudi-common/src/main/java/org/apache/hudi/common/util/hash/ColumnIndexID.java
@@ -20,6 +20,7 @@
package org.apache.hudi.common.util.hash;
import org.apache.hudi.common.util.Base64CodecUtil;
+import org.apache.hudi.common.util.StringUtils;
/**
* A stateful Hoodie object ID representing any table column.
@@ -58,7 +59,7 @@ public class ColumnIndexID extends HoodieIndexID {
@Override
public String toString() {
- return new String(this.hash);
+ return StringUtils.fromUTF8Bytes(this.hash);
}
@Override
diff --git
a/hudi-common/src/main/java/org/apache/hudi/common/util/hash/FileIndexID.java
b/hudi-common/src/main/java/org/apache/hudi/common/util/hash/FileIndexID.java
index 3f9616908bb3..dfcf9c2ca388 100644
---
a/hudi-common/src/main/java/org/apache/hudi/common/util/hash/FileIndexID.java
+++
b/hudi-common/src/main/java/org/apache/hudi/common/util/hash/FileIndexID.java
@@ -20,6 +20,7 @@
package org.apache.hudi.common.util.hash;
import org.apache.hudi.common.util.Base64CodecUtil;
+import org.apache.hudi.common.util.StringUtils;
/**
* Hoodie object ID representing any file.
@@ -58,7 +59,7 @@ public class FileIndexID extends HoodieIndexID {
@Override
public String toString() {
- return new String(this.hash);
+ return StringUtils.fromUTF8Bytes(this.hash);
}
@Override
diff --git
a/hudi-common/src/main/java/org/apache/hudi/common/util/hash/PartitionIndexID.java
b/hudi-common/src/main/java/org/apache/hudi/common/util/hash/PartitionIndexID.java
index 0fbae27b80de..a5d23f4af265 100644
---
a/hudi-common/src/main/java/org/apache/hudi/common/util/hash/PartitionIndexID.java
+++
b/hudi-common/src/main/java/org/apache/hudi/common/util/hash/PartitionIndexID.java
@@ -20,6 +20,7 @@
package org.apache.hudi.common.util.hash;
import org.apache.hudi.common.util.Base64CodecUtil;
+import org.apache.hudi.common.util.StringUtils;
/**
* Hoodie object ID representing any partition.
@@ -58,7 +59,7 @@ public class PartitionIndexID extends HoodieIndexID {
@Override
public String toString() {
- return new String(this.hash);
+ return StringUtils.fromUTF8Bytes(this.hash);
}
@Override
diff --git
a/hudi-common/src/test/java/org/apache/hudi/common/table/log/block/TestHoodieLogBlock.java
b/hudi-common/src/test/java/org/apache/hudi/common/table/log/block/TestHoodieLogBlock.java
index 3f900dd96bce..1433c0bd7222 100644
---
a/hudi-common/src/test/java/org/apache/hudi/common/table/log/block/TestHoodieLogBlock.java
+++
b/hudi-common/src/test/java/org/apache/hudi/common/table/log/block/TestHoodieLogBlock.java
@@ -35,12 +35,15 @@ import org.mockito.stubbing.Answer;
import java.io.IOException;
import java.io.InputStream;
+import java.nio.ByteBuffer;
+import java.util.Arrays;
import java.util.Collections;
import java.util.HashMap;
import java.util.Map;
import java.util.function.Supplier;
import java.util.stream.IntStream;
+import static org.apache.hudi.common.util.StringUtils.getUTF8Bytes;
import static org.junit.jupiter.api.Assertions.assertArrayEquals;
import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertTrue;
@@ -82,10 +85,15 @@ public class TestHoodieLogBlock {
@Test
public void testHeaderMetadata() throws IOException {
+ // SCHEMA carries a non-ASCII field name and a non-ASCII default. Both are
written as UTF-8, so
+ // decoding with the platform default charset corrupts them: loudly for
the name, which Avro
+ // then rejects, and silently for the default.
+ String schema = "{\"type\":\"record\",\"name\":\"r\",\"fields\":["
+ + "{\"name\":\"名字\",\"type\":\"string\",\"default\":\"李明\"}]}";
Map<HoodieLogBlock.HeaderMetadataType, String> a = new HashMap<>();
a.put(HoodieLogBlock.HeaderMetadataType.INSTANT_TIME, "100");
a.put(HoodieLogBlock.HeaderMetadataType.TARGET_INSTANT_TIME, "1");
- a.put(HoodieLogBlock.HeaderMetadataType.SCHEMA, "{}");
+ a.put(HoodieLogBlock.HeaderMetadataType.SCHEMA, schema);
a.put(HoodieLogBlock.HeaderMetadataType.COMMAND_BLOCK_TYPE, "rollback");
a.put(HoodieLogBlock.HeaderMetadataType.COMPACTED_BLOCK_TIMES, "1");
a.put(HoodieLogBlock.HeaderMetadataType.RECORD_POSITIONS, "");
@@ -96,7 +104,7 @@ public class TestHoodieLogBlock {
Map<HoodieLogBlock.HeaderMetadataType, String> b =
HoodieLogBlock.getHeaderMetadata(new ByteArraySeekableDataInputStream(new
ByteBufferBackedInputStream(bytes)));
Assertions.assertEquals("100",
b.get(HoodieLogBlock.HeaderMetadataType.INSTANT_TIME));
Assertions.assertEquals("1",
b.get(HoodieLogBlock.HeaderMetadataType.TARGET_INSTANT_TIME));
- Assertions.assertEquals("{}",
b.get(HoodieLogBlock.HeaderMetadataType.SCHEMA));
+ Assertions.assertEquals(schema,
b.get(HoodieLogBlock.HeaderMetadataType.SCHEMA));
Assertions.assertEquals("rollback",
b.get(HoodieLogBlock.HeaderMetadataType.COMMAND_BLOCK_TYPE));
Assertions.assertEquals("1",
b.get(HoodieLogBlock.HeaderMetadataType.COMPACTED_BLOCK_TIMES));
Assertions.assertEquals("",
b.get(HoodieLogBlock.HeaderMetadataType.RECORD_POSITIONS));
@@ -104,6 +112,23 @@ public class TestHoodieLogBlock {
Assertions.assertEquals("true",
b.get(HoodieLogBlock.HeaderMetadataType.IS_PARTIAL));
}
+ @Test
+ public void testHeaderMetadataIsWrittenAsUtf8() throws IOException {
+ // The read side decodes with StringUtils.fromUTF8Bytes, which is only
correct because the write
+ // side encodes with StringUtils.getUTF8Bytes. Assert that contract
against the bytes on the
+ // wire: unlike the round-trip in testHeaderMetadata, this holds whatever
the JVM default
+ // charset happens to be, so it still fails if the write side regresses to
String.getBytes().
+ String value = "名字";
+ Map<HoodieLogBlock.HeaderMetadataType, String> a = new HashMap<>();
+ a.put(HoodieLogBlock.HeaderMetadataType.SCHEMA, value);
+ byte[] bytes = HoodieLogBlock.getHeaderMetadataBytes(a);
+
+ // Layout for a single entry: int entryCount, int ordinal, int
valueLength, then the value bytes.
+ byte[] expected = getUTF8Bytes(value);
+ Assertions.assertEquals(expected.length, ByteBuffer.wrap(bytes).getInt(8));
+ assertArrayEquals(expected, Arrays.copyOfRange(bytes, 12, bytes.length));
+ }
+
private SeekableDataInputStream prepareMockedLogInputStream(int contentSize,
int
numReadFailTimes) throws IOException {
IOException exception = new IOException("Read content from log file
fails");
diff --git
a/hudi-hadoop-common/src/main/java/org/apache/hudi/common/util/OrcUtils.java
b/hudi-hadoop-common/src/main/java/org/apache/hudi/common/util/OrcUtils.java
index b8abcde4eb32..d80d7829894f 100644
--- a/hudi-hadoop-common/src/main/java/org/apache/hudi/common/util/OrcUtils.java
+++ b/hudi-hadoop-common/src/main/java/org/apache/hudi/common/util/OrcUtils.java
@@ -67,6 +67,7 @@ import java.util.concurrent.atomic.AtomicLong;
import java.util.stream.Collectors;
import static org.apache.hudi.common.util.BinaryUtil.toBytes;
+import static org.apache.hudi.common.util.StringUtils.fromUTF8Bytes;
import static org.apache.hudi.common.util.StringUtils.getUTF8Bytes;
import static org.apache.hudi.hadoop.fs.HadoopFSUtils.convertToHadoopPath;
@@ -231,7 +232,8 @@ public class OrcUtils extends FileFormatUtils {
if (reader.hasMetadataValue("orc.avro.schema")) {
ByteBuffer metadataValue = reader.getMetadataValue("orc.avro.schema");
byte[] bytes = toBytes(metadataValue);
- return HoodieSchema.parse(new String(bytes));
+ // Must match HoodieAvroOrcWriter, which writes this value as UTF-8.
+ return HoodieSchema.parse(fromUTF8Bytes(bytes));
} else {
TypeDescription orcSchema = reader.getSchema();
return AvroOrcUtils.createSchema(orcSchema);
diff --git a/hudi-io/src/main/java/org/apache/hudi/common/util/StringUtils.java
b/hudi-io/src/main/java/org/apache/hudi/common/util/StringUtils.java
index cacbc3854371..efe8967525ec 100644
--- a/hudi-io/src/main/java/org/apache/hudi/common/util/StringUtils.java
+++ b/hudi-io/src/main/java/org/apache/hudi/common/util/StringUtils.java
@@ -102,7 +102,7 @@ public class StringUtils {
}
public static String toHexString(byte[] bytes) {
- return new String(encodeHex(bytes));
+ return String.valueOf(encodeHex(bytes));
}
public static char[] encodeHex(byte[] data) {
diff --git a/hudi-io/src/main/java/org/apache/hudi/io/hfile/HFileBlockType.java
b/hudi-io/src/main/java/org/apache/hudi/io/hfile/HFileBlockType.java
index e2f711c29e81..795ae06e214d 100644
--- a/hudi-io/src/main/java/org/apache/hudi/io/hfile/HFileBlockType.java
+++ b/hudi-io/src/main/java/org/apache/hudi/io/hfile/HFileBlockType.java
@@ -168,7 +168,7 @@ public enum HFileBlockType {
in.readFully(buf);
if (IOUtils.compareTo(buf, magic) != 0) {
throw new IOException("Invalid magic: expected "
- + new String(magic) + ", got " + new String(buf));
+ + new String(magic, UTF_8) + ", got " + new String(buf, UTF_8));
}
}
}
diff --git a/hudi-io/src/main/java/org/apache/hudi/io/hfile/UTF8StringKey.java
b/hudi-io/src/main/java/org/apache/hudi/io/hfile/UTF8StringKey.java
index b1d6592eea2a..efb28224cb76 100644
--- a/hudi-io/src/main/java/org/apache/hudi/io/hfile/UTF8StringKey.java
+++ b/hudi-io/src/main/java/org/apache/hudi/io/hfile/UTF8StringKey.java
@@ -21,6 +21,8 @@ package org.apache.hudi.io.hfile;
import java.nio.charset.StandardCharsets;
+import static org.apache.hudi.common.util.StringUtils.fromUTF8Bytes;
+
/**
* Represent a UTF8 String key only, with no length information encoded.
*/
@@ -46,7 +48,7 @@ public class UTF8StringKey extends Key {
@Override
public String toString() {
return "UTF8StringKey{"
- + new String(getBytes())
+ + fromUTF8Bytes(getBytes())
+ "}";
}
}
diff --git
a/hudi-utilities/src/main/java/org/apache/hudi/utilities/UtilHelpers.java
b/hudi-utilities/src/main/java/org/apache/hudi/utilities/UtilHelpers.java
index 22bf3f4b4dcb..a416bb82bb37 100644
--- a/hudi-utilities/src/main/java/org/apache/hudi/utilities/UtilHelpers.java
+++ b/hudi-utilities/src/main/java/org/apache/hudi/utilities/UtilHelpers.java
@@ -322,7 +322,7 @@ public class UtilHelpers {
try (FSDataInputStream inputStream = fs.open(p)) {
inputStream.readFully(0, buf.array(), 0, buf.array().length);
}
- return new String(buf.array());
+ return StringUtils.fromUTF8Bytes(buf.array());
}
public static SparkConf buildSparkConf(String appName, String defaultMaster)
{
diff --git
a/hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/JdbcSource.java
b/hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/JdbcSource.java
index 0f5cd6fa68e9..0807e909923c 100644
---
a/hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/JdbcSource.java
+++
b/hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/JdbcSource.java
@@ -108,7 +108,7 @@ public class JdbcSource extends RowSource {
passwordFileStream = fileSystem.open(new
Path(getStringWithAltKeys(properties, JdbcSourceConfig.PASSWORD_FILE)));
byte[] bytes = new byte[passwordFileStream.available()];
passwordFileStream.read(bytes);
- dataFrameReader = dataFrameReader.option(Config.PASSWORD_PROP, new
String(bytes));
+ dataFrameReader = dataFrameReader.option(Config.PASSWORD_PROP,
StringUtils.fromUTF8Bytes(bytes));
} else {
throw new IllegalArgumentException(String.format("JDBCSource needs
either a %s or %s to connect to RDBMS "
+ "datasource", JdbcSourceConfig.PASSWORD_FILE.key(),
JdbcSourceConfig.PASSWORD.key()));
diff --git
a/hudi-utilities/src/main/java/org/apache/hudi/utilities/util/BloomFilterData.java
b/hudi-utilities/src/main/java/org/apache/hudi/utilities/util/BloomFilterData.java
index c3635b12979f..d221f6167fb2 100644
---
a/hudi-utilities/src/main/java/org/apache/hudi/utilities/util/BloomFilterData.java
+++
b/hudi-utilities/src/main/java/org/apache/hudi/utilities/util/BloomFilterData.java
@@ -19,6 +19,8 @@
package org.apache.hudi.utilities.util;
+import org.apache.hudi.common.util.StringUtils;
+
import javax.annotation.Nonnull;
import java.nio.ByteBuffer;
@@ -68,7 +70,7 @@ public class BloomFilterData implements
Comparable<BloomFilterData> {
@Override
public String toString() {
- String bloomFilterString = new String(bloomFilter.array());
+ String bloomFilterString = StringUtils.fromUTF8Bytes(bloomFilter.array());
return "BloomFilterData{"
+ "partitionPath='" + partitionPath + '\''
+ ", filename='" + filename + '\''
diff --git a/style/checkstyle-suppressions.xml
b/style/checkstyle-suppressions.xml
index 39ccbf1aa460..92681fa3063a 100644
--- a/style/checkstyle-suppressions.xml
+++ b/style/checkstyle-suppressions.xml
@@ -33,6 +33,9 @@
<suppress checks="IllegalTokenText" files="PartitionPathEncodeUtils.java"/>
<!-- SQL is longer than 200 characters -->
<suppress checks="LineLength" files="sql-transformer.properties"/>
+ <!-- Charset-correctness of the decode side only matters for bytes that came
from storage, so the
+ rule is scoped to main sources; test fixtures build their own bytes
in-process. -->
+ <suppress checks="RegexpSinglelineJava" files="src[\\/]test[\\/].*"/>
<!-- Ignore hudi-trino as this module has its own checkstyle rules -->
<suppress checks=".*" files="hudi-trino[\\/].*"/>
<suppress checks=".*" files="target[\\/].*"/>
diff --git a/style/checkstyle.xml b/style/checkstyle.xml
index dcc3d0aac00a..8fa5515150f0 100644
--- a/style/checkstyle.xml
+++ b/style/checkstyle.xml
@@ -59,6 +59,15 @@
</module>
<module name="SuppressWarningsHolder"/>
<module name="OuterTypeFilename"/>
+ <!-- HUDI-6825 converted the encode side to StringUtils.getUTF8Bytes
but left the decode
+ side on the platform default charset, which corrupts non-ASCII
values read back from
+ storage. Keep the two sides symmetric. -->
+ <module name="RegexpSinglelineJava">
+ <property name="id" value="DefaultCharsetDecode"/>
+ <property name="format" value="new
String\((?![^;]*(?:UTF_8|StandardCharsets|Charset))"/>
+ <property name="message" value="Decode bytes with an explicit
charset (StringUtils.fromUTF8Bytes); new String(byte[]) uses the platform
default charset. For a char[] use String.valueOf."/>
+ <property name="ignoreComments" value="true"/>
+ </module>
<module name="IllegalTokenText">
<property name="tokens" value="STRING_LITERAL, CHAR_LITERAL"/>
<property name="format"
value="\\u00(09|0(a|A)|0(c|C)|0(d|D)|22|27|5(C|c))|\\(0(10|11|12|14|15|42|47)|134)"/>
