This is an automated email from the ASF dual-hosted git repository.
davidzollo pushed a commit to branch dev
in repository https://gitbox.apache.org/repos/asf/seatunnel.git
The following commit(s) were added to refs/heads/dev by this push:
new 99c7295ff0 [Fix][Connector-V2] Preserve ORC nested field case (#11428)
99c7295ff0 is described below
commit 99c7295ff0cd00cd9b6652f9499d17561247731e
Author: Jast <[email protected]>
AuthorDate: Fri Aug 21 10:42:20 2026 +0800
[Fix][Connector-V2] Preserve ORC nested field case (#11428)
Co-authored-by: zhangshenghang <[email protected]>
Co-authored-by: zhangshenghang <[email protected]>
Co-authored-by: davidzollo <[email protected]>
---
.../introduction/concepts/incompatible-changes.md | 8 +++
.../introduction/concepts/incompatible-changes.md | 8 +++
.../file/sink/writer/OrcWriteStrategy.java | 2 +-
.../file/writer/OrcWriteStrategyTest.java | 79 ++++++++++++++++++++++
4 files changed, 96 insertions(+), 1 deletion(-)
diff --git a/docs/en/introduction/concepts/incompatible-changes.md
b/docs/en/introduction/concepts/incompatible-changes.md
index ae63461640..b8c8226f4f 100644
--- a/docs/en/introduction/concepts/incompatible-changes.md
+++ b/docs/en/introduction/concepts/incompatible-changes.md
@@ -93,6 +93,14 @@ You need to check this document before you upgrade to
related version.
### Connector Changes
+- **Breaking Change: ORC file sink preserves case of nested struct field
names**
+ - **Affected component**:
`seatunnel-connectors-v2/connector-file/connector-file-base` (used by all
File/HDFS/S3/OSS ORC sinks that share `OrcWriteStrategy`)
+ - **Description**: Previously, `OrcWriteStrategy.buildFieldWithRowType(...)`
forced every nested `ROW` (struct) field name to lowercase when building the
ORC schema, so a nested field declared as `MD5` was persisted as `md5` in the
file footer. Downstream consumers that read the column by its declared
original-case name received null/missing values. The `.toLowerCase()` call has
been removed from the recursive nested-field branch, so nested struct field
names are now written verbatim i [...]
+ - **Impact**: ORC files written by SeaTunnel after this change embed the
original-case nested field names in their schema footer. Users that adapted to
the old behavior (for example, case-sensitive ORC readers with
`orc.schema.evolution.case.sensitive=true`, Spark with
`spark.sql.caseSensitive=true`, or pipelines that expected `md5` rather than
`MD5`) will see the inverse problem: null values or schema mismatches when
reading new files. Directories that mix pre-upgrade files (lowercase [...]
+ - **Migration Guide**:
+ - **Mixed-version directories**: Re-materialize the directory so every
file is produced by the new version, or write pre- and post-upgrade files into
separate directories and read them independently.
+ - **Case-sensitive consumers**: Configure the reader for case-insensitive
schema evolution where supported, or remap the column at read time.
+ - **Case-only sibling fields** (for example `MD5` and `md5` in the same
struct): now representable; case-insensitive downstream consumers (such as
Hive) may treat them as ambiguous — disambiguate at the source if needed.
- **Breaking Change: Iceberg Connector — source table primary key is no longer
silently inherited**
- **Affected component**: `seatunnel-connectors-v2/connector-iceberg`
- **Description**: `SchemaUtils.toIcebergSchema()` previously fell back to
the CDC source
diff --git a/docs/zh/introduction/concepts/incompatible-changes.md
b/docs/zh/introduction/concepts/incompatible-changes.md
index fb7373adf3..967d917247 100644
--- a/docs/zh/introduction/concepts/incompatible-changes.md
+++ b/docs/zh/introduction/concepts/incompatible-changes.md
@@ -92,6 +92,14 @@
### 连接器变更
+- **破坏性变更：ORC 文件 Sink 保留嵌套 Struct 字段名的大小写**
+ - **影响范围**：`seatunnel-connectors-v2/connector-file/connector-file-base`（所有共享
`OrcWriteStrategy` 的 File/HDFS/S3/OSS ORC Sink）
+ - **变更说明**：此前，`OrcWriteStrategy.buildFieldWithRowType(...)` 在构建 ORC Schema
时，会将每个嵌套 `ROW`（struct）字段名强制转为小写，因此声明为 `MD5` 的嵌套字段在文件 footer 中被持久化为
`md5`。下游消费者按原始大小写名称读取该列时会得到 null/缺失值。本次移除了递归嵌套字段分支上的 `.toLowerCase()` 调用，嵌套
struct 字段名将按原始大小写写入文件 Schema。
+ - **影响**：升级后由 SeaTunnel 写入的 ORC 文件，其 Schema footer
中的嵌套字段名保留原始大小写。已经适配旧行为的用户（例如使用 `orc.schema.evolution.case.sensitive=true` 的 ORC
Reader、设置 `spark.sql.caseSensitive=true` 的 Spark、或预期读到 `md5` 而非 `MD5`
的下游管道）将面临相反的问题：读取新文件时出现空值或 Schema
不匹配。同一目录下混合旧版本（小写嵌套字段名）和新版本（原始大小写）的文件时，相同逻辑列对应的嵌套 Schema 形态不一致，大小写敏感的 Schema
合并无法调和。
+ - **迁移指南**：
+ - **混合版本目录**：将目录重新物化，使所有文件都由新版本写入；或将旧版本与新版本文件分别写入不同目录，独立读取。
+ - **大小写敏感的下游**：在支持的情况下将 Reader 配置为大小写不敏感的 Schema Evolution，或在读取时重映射该列。
+ - **仅大小写不同的同名兄弟字段**（例如同一 struct 中同时存在 `MD5` 和 `md5`）：现在可被表达；大小写不敏感的下游（如
Hive）可能将其视为歧义字段，如需保留请在源头消歧。
- **破坏性变更：Iceberg 连接器 — 不再自动继承源表主键**
- **影响范围**：`seatunnel-connectors-v2/connector-iceberg`
- **变更说明**：当未显式配置 `iceberg.table.primary-keys`
时，`SchemaUtils.toIcebergSchema()`
diff --git
a/seatunnel-connectors-v2/connector-file/connector-file-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/sink/writer/OrcWriteStrategy.java
b/seatunnel-connectors-v2/connector-file/connector-file-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/sink/writer/OrcWriteStrategy.java
index 48424d66ed..6dfc04d537 100644
---
a/seatunnel-connectors-v2/connector-file/connector-file-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/sink/writer/OrcWriteStrategy.java
+++
b/seatunnel-connectors-v2/connector-file/connector-file-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/sink/writer/OrcWriteStrategy.java
@@ -213,7 +213,7 @@ public class OrcWriteStrategy extends
AbstractWriteStrategy<Writer> {
SeaTunnelDataType<?>[] fieldTypes = ((SeaTunnelRowType)
type).getFieldTypes();
for (int i = 0; i < fieldTypes.length; i++) {
struct.addField(
- ((SeaTunnelRowType)
type).getFieldName(i).toLowerCase(),
+ ((SeaTunnelRowType) type).getFieldName(i),
buildFieldWithRowType(fieldTypes[i]));
}
return struct;
diff --git
a/seatunnel-connectors-v2/connector-file/connector-file-base/src/test/java/org/apache/seatunnel/connectors/seatunnel/file/writer/OrcWriteStrategyTest.java
b/seatunnel-connectors-v2/connector-file/connector-file-base/src/test/java/org/apache/seatunnel/connectors/seatunnel/file/writer/OrcWriteStrategyTest.java
index aaca9476c3..90320af004 100644
---
a/seatunnel-connectors-v2/connector-file/connector-file-base/src/test/java/org/apache/seatunnel/connectors/seatunnel/file/writer/OrcWriteStrategyTest.java
+++
b/seatunnel-connectors-v2/connector-file/connector-file-base/src/test/java/org/apache/seatunnel/connectors/seatunnel/file/writer/OrcWriteStrategyTest.java
@@ -49,6 +49,85 @@ public class OrcWriteStrategyTest {
private static final String TMP_PATH =
"file:///tmp/seatunnel/orc/batch/test";
private static final int ORC_WRITE_NUMBER = 2000;
+ @Test
+ public void testOrcStructSchemaPreservesFieldCase() {
+ SeaTunnelRowType structType =
+ new SeaTunnelRowType(
+ new String[] {"MD5"}, new SeaTunnelDataType[]
{BasicType.STRING_TYPE});
+
+ org.apache.orc.TypeDescription structSchema =
+ OrcWriteStrategy.buildFieldWithRowType(structType);
+
+ Assertions.assertEquals("MD5", structSchema.getFieldNames().get(0));
+ }
+
+ @DisabledOnOs(OS.WINDOWS)
+ @Test
+ public void testOrcNestedFieldCaseRoundTrip() throws Exception {
+ String tmpPath = "file:///tmp/seatunnel/orc/nested/test";
+ Map<String, Object> writeConfig = new HashMap<>();
+ writeConfig.put("tmp_path", tmpPath);
+ writeConfig.put("path", "file:///tmp/seatunnel/orc/nested");
+ writeConfig.put("file_format_type", FileFormat.ORC.name());
+
+ SeaTunnelRowType nestedFieldType =
+ new SeaTunnelRowType(
+ new String[] {"MD5"}, new SeaTunnelDataType[]
{BasicType.STRING_TYPE});
+ SeaTunnelRowType writeRowType =
+ new SeaTunnelRowType(
+ new String[] {"payload"}, new SeaTunnelDataType[]
{nestedFieldType});
+ FileSinkConfig writeSinkConfig =
+ new FileSinkConfig(ReadonlyConfig.fromMap(writeConfig),
writeRowType);
+ OrcWriteStrategy writeStrategy = new OrcWriteStrategy(writeSinkConfig);
+
+ LocalFileSystemConf.LocalConf hadoopConf =
+ new LocalFileSystemConf.LocalConf(FS_DEFAULT_NAME_DEFAULT);
+ writeStrategy.setCatalogTable(
+ CatalogTableUtil.getCatalogTable("test", null, null, "test",
writeRowType));
+ writeStrategy.init(hadoopConf, "test1", "test1", 0);
+ writeStrategy.beginTransaction(1L);
+
+ SeaTunnelRow nestedValue = new SeaTunnelRow(new Object[] {"abc123"});
+ writeStrategy.write(new SeaTunnelRow(new Object[] {nestedValue}));
+ writeStrategy.finishAndCloseFile();
+ writeStrategy.close();
+
+ OrcReadStrategy readStrategy = new OrcReadStrategy();
+ readStrategy.init(hadoopConf);
+ List<String> readFiles = readStrategy.getFileNamesByPath(tmpPath);
+ Assertions.assertEquals(1, readFiles.size());
+ String readFilePath = readFiles.get(0);
+
+ SeaTunnelRowType readRowType =
readStrategy.getSeaTunnelRowTypeInfo(readFilePath);
+ Assertions.assertEquals(1, readRowType.getTotalFields());
+ SeaTunnelDataType<?> nestedReadType = readRowType.getFieldType(0);
+ Assertions.assertTrue(nestedReadType instanceof SeaTunnelRowType);
+ SeaTunnelRowType nestedReadRowType = (SeaTunnelRowType) nestedReadType;
+ Assertions.assertArrayEquals(new String[] {"MD5"},
nestedReadRowType.getFieldNames());
+
+ List<SeaTunnelRow> readRows = new ArrayList<>();
+ Collector<SeaTunnelRow> readCollector =
+ new Collector<SeaTunnelRow>() {
+ @Override
+ public void collect(SeaTunnelRow record) {
+ readRows.add(record);
+ }
+
+ @Override
+ public Object getCheckpointLock() {
+ return null;
+ }
+ };
+ readStrategy.read(readFilePath, "test", readCollector);
+ Assertions.assertEquals(1, readRows.size());
+ SeaTunnelRow readRow = readRows.get(0);
+ Object nestedObj = readRow.getField(0);
+ Assertions.assertTrue(nestedObj instanceof SeaTunnelRow);
+ SeaTunnelRow nestedRead = (SeaTunnelRow) nestedObj;
+ Assertions.assertEquals("abc123", nestedRead.getField(0));
+ readStrategy.close();
+ }
+
@DisabledOnOs(OS.WINDOWS)
@Test
public void testCloseWriterWhenRollingFile() throws Exception {
