Re: [PR] [fix](iceberg) Preserve nested field case in created schemas [doris]
github-actions[bot] commented on PR #67166: URL: https://github.com/apache/doris/pull/67166#issuecomment-5525498728 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](iceberg) Preserve nested field case in created schemas [doris]
Gabriel39 merged PR #67166: URL: https://github.com/apache/doris/pull/67166 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](iceberg) Preserve nested field case in created schemas [doris]
hello-stephen commented on PR #67166: URL: https://github.com/apache/doris/pull/67166#issuecomment-5524936319 # FE Regression Coverage Report Increment line coverage ` 56.32% (49/87)` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/67166_458e4651d95aad2ae1da42b665082ce4791e0462_merge_fe/increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/67166_458e4651d95aad2ae1da42b665082ce4791e0462_merge_fe/report/index.html) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](iceberg) Preserve nested field case in created schemas [doris]
hello-stephen commented on PR #67166: URL: https://github.com/apache/doris/pull/67166#issuecomment-5523764152 ClickBench: Total hot run time: 14.71 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit 458e4651d95aad2ae1da42b665082ce4791e0462, data reload: false query1 0.000.000.00 query2 0.080.030.03 query3 0.260.110.11 query4 1.610.100.10 query5 0.180.170.17 query6 1.250.690.69 query7 0.030.000.01 query8 0.050.030.04 query9 0.290.210.21 query10 0.360.340.34 query11 0.160.110.12 query12 0.150.130.12 query13 0.320.300.31 query14 0.460.440.45 query15 0.360.350.35 query16 0.220.230.22 query17 0.700.720.71 query18 0.180.170.17 query19 1.141.171.14 query20 0.010.020.02 query21 15.45 0.150.11 query22 5.090.040.05 query23 16.18 0.260.11 query24 3.080.320.26 query25 0.120.050.04 query26 0.720.170.11 query27 0.030.030.02 query28 3.710.520.29 query29 12.48 3.182.61 query30 0.250.100.12 query31 2.750.370.18 query32 3.510.320.22 query33 1.341.421.39 query34 15.35 2.191.76 query35 1.711.701.71 query36 0.450.280.28 query37 0.060.040.04 query38 0.050.030.03 query39 0.030.020.02 query40 0.110.070.08 query41 0.070.020.02 query42 0.030.020.02 query43 0.040.030.03 Total cold run time: 90.42 s Total hot run time: 14.71 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](iceberg) Preserve nested field case in created schemas [doris]
hello-stephen commented on PR #67166: URL: https://github.com/apache/doris/pull/67166#issuecomment-5523713148 TPC-DS: Total hot run time: 82056 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 458e4651d95aad2ae1da42b665082ce4791e0462, data reload: false query5 4243410 332 332 query6 389 132 126 126 query7 4958416 208 208 query8 286 127 121 121 query9 8690290029142900 query10 411 221 187 187 query11 53761041916 916 query12 115 70 71 70 query13 1188435 327 327 query14 6070220620762076 query14_11994193719631937 query15 177 123 119 119 query16 920 370 363 363 query17 800 466 370 370 query18 2327318 242 242 query19 162 142 118 118 query20 72 69 71 69 query21 205 100 88 88 query22 5379543053735373 query23 6781623959615961 query23_15884602959835983 query24 73071102800 800 query24_1768 783 787 783 query25 438 299 260 260 query26 1238239 130 130 query27 2779448 268 268 query28 4657150515261505 query29 941 444 347 347 query30 258 160 129 129 query31 814 396 336 336 query32 154 81 78 78 query33 481 227 176 176 query34 1013822 491 491 query35 418 409 346 346 query36 563 582 558 558 query37 124 82 72 72 query38 1009863 822 822 query39 511 488 477 477 query39_1467 461 443 443 query40 214 94 78 78 query41 60 57 57 57 query42 78 73 73 73 query43 244 247 218 218 query44 1039551 576 551 query45 112 131 94 94 query46 767 855 543 543 query47 752 750 727 727 query48 321 306 220 220 query49 522 233 193 193 query50 727 249 196 196 query51 8292810281898102 query52 79 74 60 60 query53 200 212 144 144 query54 208 181 153 153 query55 78 57 64 57 query56 214 313 155 155 query57 687 660 653 653 query58 220 159 168 159 query59 1244122411121112 query60 238 184 185 184 query61 135 113 124 113 query62 385 204 181 181 query63 169 142 143 142 query64 2734761 585 585 query65 1642161015741574 query66 1816258 200 200 query67 9555965095599559 query68 30291150726 726 query69 333 225 183 183 query70 681 641 615 615 query71 266 178 164 164 query72 2318182015881588 query73 671 557 330 330 query74 1986121211331133 query75 11561094951 951 query76 2352717 535 535 query77 257 245 212 212 query78 3867374831923192 query79 2320821 605 605 query80 1589325 278 278 query81 496 154 136 136 query82 1015120 98 98 query83 280 212 193 193 query84 294 115 87 87 query85 782 362 296 296 query86 392 185 174 174 query87 1031964 897 897 query88 2767210120932093 query89 292 196 175 175 query90 1973131 131 131 query91 131 115 98 98 query92 79 70 62 62 query93 1595727 727 query94 661 260 245 245 query95 509 313 228 228 query96 800 594 264 264 query97 1058105810271027 query98 167 139 131 131 query99 427 346 331 331 Total cold run time: 178090 ms Total hot run time: 82056 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific
Re: [PR] [fix](iceberg) Preserve nested field case in created schemas [doris]
hello-stephen commented on PR #67166: URL: https://github.com/apache/doris/pull/67166#issuecomment-5523633809 TPC-H: Total hot run time: 17027 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 458e4651d95aad2ae1da42b665082ce4791e0462, data reload: false -- Round 1 -- q1 17627 313831143114 q2 2103278 246 246 q3 10203 863 523 523 q4 4670254 208 208 q5 7678538 385 385 q6 140 115 95 95 q7 542 496 384 384 q8 9246875 864 864 q9 3430239324052393 q10 6524864 728 728 q11 416 196 186 186 q12 685 265 207 207 q13 18242 153811541154 q14 163 148 143 143 q15 q16 443 389 369 369 q17 1408834 831 831 q18 2985223522552235 q19 1136900 742 742 q20 368 280 203 203 q21 5473179318721793 q22 327 268 224 224 Total cold run time: 93809 ms Total hot run time: 17027 ms - Round 2, with runtime_filter_mode=off - q1 3493342933793379 q2 509 412 366 366 q3 2208229821522152 q4 12031169887 887 q5 2187208621002086 q6 171 117 88 88 q7 1038916 884 884 q8 1613143714061406 q9 3134315031353135 q10 1860179716041604 q11 361 271 252 252 q12 453 433 349 349 q13 1478152911731173 q14 179 179 154 154 q15 q16 397 392 358 358 q17 3537335031563156 q18 4807442047224420 q19 882 921 896 896 q20 1014972 844 844 q21 3881326032243224 q22 399 359 316 316 Total cold run time: 34804 ms Total hot run time: 31129 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](iceberg) Preserve nested field case in created schemas [doris]
hello-stephen commented on PR #67166: URL: https://github.com/apache/doris/pull/67166#issuecomment-5522945578 # FE UT Coverage Report Increment line coverage ` 95.00% (57/60)` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/67166_458e4651d95aad2ae1da42b665082ce4791e0462/fe_increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/67166_458e4651d95aad2ae1da42b665082ce4791e0462/fe_report/index.html) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](iceberg) Preserve nested field case in created schemas [doris]
github-actions[bot] commented on code in PR #67166:
URL: https://github.com/apache/doris/pull/67166#discussion_r3922039767
##
fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergCatalogOps.java:
##
@@ -555,24 +559,25 @@ public void modifyColumn(String dbName, String tableName,
IcebergColumnChange co
throw new DorisConnectorException("Modify column type
from complex to primitive is not"
+ " supported: " + column.getName());
}
-updateSchema.updateColumn(column.getName(),
newType.asPrimitiveType(), targetComment);
+updateSchema.updateColumn(currentName,
newType.asPrimitiveType(), targetComment);
} else {
// A complex (STRUCT/ARRAY/MAP) modify diffs the new type
against the current one field-by-field
// (IcebergComplexTypeDiff); the top-level column doc is
updated separately, as in legacy.
if (current.type().isPrimitiveType()) {
throw new DorisConnectorException("Modify column type
from non-complex to complex is not"
+ " supported: " + column.getName());
}
-IcebergComplexTypeDiff.apply(updateSchema,
column.getName(), current.type(), newType,
+IcebergComplexTypeDiff.apply(updateSchema, currentName,
current.type(), newType,
column.getSourceType());
if (!Objects.equals(current.doc(), targetComment)) {
-updateSchema.updateColumnDoc(column.getName(),
targetComment);
+updateSchema.updateColumnDoc(currentName,
targetComment);
}
}
if (column.isNullable()) {
-updateSchema.makeColumnOptional(column.getName());
+updateSchema.makeColumnOptional(currentName);
}
-applyPosition(updateSchema, position, column.getName());
+IcebergNestedColumnEvolution.applyTopLevelPosition(
Review Comment:
[P1] Canonicalize the other top-level positioning paths too. This fixes
MODIFY by resolving both the target and AFTER sibling to persisted Iceberg
spellings, but ADD still routes through `applyPosition`, which passes
`position.getAfterColumn()` verbatim, and `reorderColumns` still passes every
caller-provided name directly to the case-sensitive `moveFirst`/`moveAfter`
APIs. On a table with stored fields `Id`/`Label`, `ADD ... AFTER id` and a
reorder using `label`/`id` therefore still fail even though Doris resolves
external columns case-insensitively. Please reuse schema-backed
canonicalization for ADD and resolve the complete reorder list before staging
it, with mixed-case catalog-backed tests.
##
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/ExpressionAnalyzer.java:
##
@@ -622,6 +630,20 @@ public Expression visitBoundFunction(BoundFunction
boundFunction, ExpressionRewr
return TypeCoercionUtils.processBoundFunction(boundFunction);
}
+private ElementAt canonicalizeStructSelector(ElementAt elementAt) {
+Expression left = elementAt.left();
+Expression right = elementAt.right();
+if (left.getDataType() instanceof StructType && right instanceof
StringLikeLiteral) {
Review Comment:
[P1] Unwrap bracket-expression variables before canonicalizing. The parser
builds `payload[@field]` directly as `ElementAt`, and `ElementAt` legality/type
resolution uses `getArgument(1)`, so a user variable whose value is displayed
Unicode `Σ` is accepted. This helper checks the raw right child, sees
`Variable`, and skips the rewrite; `VariableToLiteral` later exposes `Σ`, but
only integer selectors are normalized afterward. BE then receives `Σ` against
the thrift runtime name `σ` and fails lookup. Please canonicalize the
wrapper-aware argument while preserving variable-use/SQL-cache accounting, and
add a variable-backed bracket execution test. Function syntax is already
unwrapped by `constructUnboundFunctionArguments`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [fix](iceberg) Preserve nested field case in created schemas [doris]
hello-stephen commented on PR #67166: URL: https://github.com/apache/doris/pull/67166#issuecomment-5521779099 # FE Regression Coverage Report Increment line coverage ` 87.27% (48/55)` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/67166_6f1193d7f63f92948be69f0c62d6a643ad8be848_merge_fe/increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/67166_6f1193d7f63f92948be69f0c62d6a643ad8be848_merge_fe/report/index.html) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](iceberg) Preserve nested field case in created schemas [doris]
Gabriel39 commented on PR #67166: URL: https://github.com/apache/doris/pull/67166#issuecomment-5521529060 @924060929 Thanks for the compatibility review. I checked all four points against the current PR head: 1. The Catalog `StructType.getField` compatibility fallback was already present. However, the review exposed a real edge case: a pre-ROOT image can contain two legacy runtime names that both match one selector under broad Unicode case folding. Commit 458e4651d9 fixes this by accepting a legacy fallback only when it is unique; exact persisted runtime spellings still work, and ambiguous selectors are rejected instead of returning the wrong sibling. I did not add `readObject`: these types are replayed through Gson, and rebuilding `fieldMap` cannot recover the original locale or external spelling that old images never persisted. 2. `StructElement` no longer exists on current master; it was merged into `ElementAt`. Both `element_at` and the `struct_element` alias therefore use the same `StructType.getField` path, so no separate fix is needed. 3. `IcebergScanNode` no longer exists on current master. Identity partition columns are obtained through `IcebergPartitionUtils.getIdentityPartitionColumns`, which intentionally preserves source-column case and already has focused coverage, so changing partition-key case handling is outside this PR. 4. `StructType.addField` already uses `toLowerCase(Locale.ROOT)` on the current PR head, so no additional change is needed there. I also fixed the separate computed-base dereference gap in 458e4651d9 and added regression coverage for both valid issues. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](iceberg) Preserve nested field case in created schemas [doris]
Gabriel39 commented on code in PR #67166:
URL: https://github.com/apache/doris/pull/67166#discussion_r3921704554
##
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/ExpressionAnalyzer.java:
##
@@ -1277,7 +1285,8 @@ private Optional bindNestedFields(UnboundSlot
unboundSlot, Slot slot
throw new AnalysisException("No such struct field '" +
fieldName + "' in '" + lastFieldName + "'");
}
lastFieldName = fieldName;
-expression = new ElementAt(expression, new
StringLiteral(fieldName));
+// Dereference-created selectors also cross the thrift
boundary and must use runtime identity.
+expression = new ElementAt(expression, new
StringLiteral(field.getName()));
Review Comment:
Fixed in 458e4651d9. The struct branch in `visitDereferenceExpression` now
canonicalizes the newly constructed `ElementAt` before returning, because that
node is not revisited by `visitElementAt`. A regression test covers dotted
access on a computed `CAST` base with a Unicode selector.
##
fe/fe-type/src/main/java/org/apache/doris/catalog/StructType.java:
##
@@ -122,15 +123,15 @@ public boolean supportSubType(Type subType) {
public void addField(StructField field) {
field.setPosition(fields.size());
fields.add(field);
-fieldMap.put(field.getName().toLowerCase(), field);
+fieldMap.put(field.getName().toLowerCase(Locale.ROOT), field);
}
public ArrayList getFields() {
return fields;
}
public StructField getField(String fieldName) {
-return fieldMap.get(fieldName.toLowerCase());
+return fieldMap.get(fieldName.toLowerCase(Locale.ROOT));
Review Comment:
Fixed in 458e4651d9. Legacy locale-dependent names now use broad case
folding only when it produces a unique candidate. Exact persisted runtime
spellings remain accessible, while an ambiguous selector is rejected instead of
silently choosing the wrong sibling. The new tests cover Gson replay,
Catalog/Nereids conversion, round-trip conversion, and analyzer rejection.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [fix](iceberg) Preserve nested field case in created schemas [doris]
Gabriel39 commented on PR #67166: URL: https://github.com/apache/doris/pull/67166#issuecomment-5521517193 /review -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](iceberg) Preserve nested field case in created schemas [doris]
Gabriel39 commented on PR #67166: URL: https://github.com/apache/doris/pull/67166#issuecomment-5521515929 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](iceberg) Preserve nested field case in created schemas [doris]
924060929 commented on PR #67166:
URL: https://github.com/apache/doris/pull/67166#issuecomment-5520315128
## Compatibility Review: Struct Field Name Lookup Risks
Thanks for the detailed work on preserving nested field case. I ran targeted
compatibility tests simulating locale-dependent metadata persistence and
replay. Below are findings from code tracing and Java-level verification.
### Issue 1 — `catalog.StructType.getField()` has no legacy fallback (P1)
The PR adds a `legacyLocaleDependentName` fallback in
`nereids.types.StructType.getField()`, but the **catalog**
`StructType.getField()` (which is the persistent type, deserialized via Gson
with `@SerializedName` on `fieldMap`) does **not** get a similar fallback.
After Gson deserialization, `fieldMap` keys are whatever the old FE
persisted — they are **not** rebuilt by the constructor. On a non-ROOT locale
(e.g. Turkish `tr-TR`):
```
Old FE (Turkish): "I".toLowerCase() = "ı" (U+0131) → fieldMap key = "ı"
New FE (ROOT):getField("I") → fieldMap.get("I".toLowerCase(ROOT)) =
fieldMap.get("i") → MISS
```
I verified this with a standalone Java test — the catalog `getField()`
returns `null` for fields persisted under a different locale. Any code path
that goes through the catalog `StructType.getField()` (e.g. `FunctionCallExpr`
in the old analyzer, connector metadata paths) will fail to find the field.
**Suggestion**: Either add the same legacy fallback to
`catalog.StructType.getField()`, or add a `readObject()` that rebuilds
`fieldMap` with `Locale.ROOT` keys after Gson deserialization:
```java
private void readObject(ObjectInputStream in) throws IOException,
ClassNotFoundException {
in.defaultReadObject();
fieldMap.clear();
for (StructField field : fields) {
fieldMap.put(field.getName().toLowerCase(Locale.ROOT), field);
}
}
```
### Issue 2 — `StructElement.getSignatures()` uses raw name for map lookup
(P2)
`StructElement.java:88` looks up the field name directly in `nameToFields`
**without any `toLowerCase()`**:
```java
String name = ((StringLikeLiteral) child(1)).getStringValue();
if (!structArgType.getNameToFields().containsKey(name)) { // raw name vs
lowered keys
```
The map keys are lowered (via `StructField` constructor), so
`struct_element(s, 'FieldA')` looks up `"FieldA"` against key `"fielda"` →
**always misses** for mixed-case names.
This is a pre-existing bug, but the PR's `canonicalizeStructSelector()` in
`ExpressionAnalyzer` only handles `ElementAt`, not `StructElement`. I confirmed
via grep that there is no `visitStructElement` in `ExpressionAnalyzer` and no
rewrite rule that converts `StructElement` to `ElementAt` — so `StructElement`
goes through its own binding path and never gets canonicalized.
**Suggestion**: Add `StructElement` canonicalization in
`ExpressionAnalyzer.visitUnboundFunction()` alongside the existing `ElementAt`
handling, or fix `getSignatures()` to lower the lookup key.
### Issue 3 — `IcebergScanNode.getPathPartitionKeys()` uses
`String::toLowerCase` without `Locale.ROOT` (P2)
```java
// IcebergScanNode.java:385
return icebergTable.spec().fields().stream()
.map(PartitionField::name).map(String::toLowerCase) // ← no Locale.ROOT
.collect(Collectors.toList());
```
This file is not in the PR's changed files. After the PR, struct field names
in the Nereids layer use `toLowerCase(Locale.ROOT)`, but partition key names
from this path still use locale-dependent `toLowerCase()`. On a Turkish JVM,
partition key `"I"` becomes `"ı"` while the corresponding struct field name
becomes `"i"` — they won't match.
### Issue 4 — `catalog.StructType.addField()` key generation may be
inconsistent (P2)
```java
public void addField(StructField field) {
fieldMap.put(field.getName().toLowerCase(), field); // ← no Locale.ROOT
}
```
If the PR changes `getField()` to use `toLowerCase(Locale.ROOT)` but
`addField()` still uses `toLowerCase()` (default locale), then after an `ALTER
TABLE ADD COLUMN` on a struct, the new field's key and the lookup key could
diverge on non-ROOT locales.
### What works correctly
- ✅ Nereids `StructType.getField()` legacy fallback — verified it correctly
resolves Turkish-persisted fields via `equalsIgnoreCase`
- ✅ Pure ASCII field names — `toLowerCase()` and `toLowerCase(Locale.ROOT)`
produce identical results for ASCII, so the vast majority of users are
unaffected
- ✅ `ElementAt` canonicalization in `ExpressionAnalyzer` — covers
`element_at()` and dotted dereference paths
- ✅ `NestedColumnPruning` / `AccessPathExpressionCollector` `Locale.ROOT`
changes — consistent with Nereids struct keys
### Test methodology
I wrote a standalone Java test that simulates:
1. `StructElement` raw name lookup
Re: [PR] [fix](iceberg) Preserve nested field case in created schemas [doris]
hello-stephen commented on PR #67166: URL: https://github.com/apache/doris/pull/67166#issuecomment-5519765864 ClickBench: Total hot run time: 14.68 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit 6f1193d7f63f92948be69f0c62d6a643ad8be848, data reload: false query1 0.010.000.01 query2 0.070.030.03 query3 0.250.110.11 query4 1.610.100.10 query5 0.180.160.15 query6 1.250.680.68 query7 0.030.000.01 query8 0.040.030.04 query9 0.290.220.21 query10 0.340.340.35 query11 0.170.120.12 query12 0.150.120.11 query13 0.300.310.30 query14 0.470.440.45 query15 0.360.330.33 query16 0.210.230.21 query17 0.670.720.73 query18 0.190.170.18 query19 1.131.201.18 query20 0.010.010.01 query21 15.44 0.160.12 query22 5.110.040.04 query23 16.19 0.240.10 query24 3.000.310.24 query25 0.110.030.04 query26 0.750.160.12 query27 0.040.040.04 query28 3.620.540.28 query29 12.42 3.162.54 query30 0.270.120.13 query31 2.750.360.17 query32 3.570.330.24 query33 1.421.501.42 query34 15.33 2.211.77 query35 1.731.751.71 query36 0.450.280.27 query37 0.060.040.04 query38 0.040.030.03 query39 0.030.020.02 query40 0.120.080.07 query41 0.080.020.02 query42 0.030.020.02 query43 0.030.030.03 Total cold run time: 90.32 s Total hot run time: 14.68 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](iceberg) Preserve nested field case in created schemas [doris]
hello-stephen commented on PR #67166: URL: https://github.com/apache/doris/pull/67166#issuecomment-5519731515 TPC-DS: Total hot run time: 80910 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 6f1193d7f63f92948be69f0c62d6a643ad8be848, data reload: false query5 4281421 321 321 query6 411 129 123 123 query7 4941427 235 235 query8 309 123 114 114 query9 8695290828872887 query10 407 224 173 173 query11 53601027912 912 query12 134 70 72 70 query13 1204436 339 339 query14 6138218120852085 query14_11968195319431943 query15 179 113 109 109 query16 943 373 360 360 query17 833 450 369 369 query18 2336332 236 236 query19 169 139 110 110 query20 69 71 68 68 query21 203 99 91 91 query22 5383537752445244 query23 6741624858125812 query23_16043582961365829 query24 72801107765 765 query24_1777 810 776 776 query25 424 302 269 269 query26 1221231 131 131 query27 2801413 245 245 query28 4707150814871487 query29 968 446 354 354 query30 267 159 130 130 query31 840 398 334 334 query32 133 76 72 72 query33 475 227 194 194 query34 993 819 508 508 query35 412 402 336 336 query36 590 578 522 522 query37 115 84 71 71 query38 997 884 806 806 query39 494 461 475 461 query39_1437 466 485 466 query40 196 87 78 78 query41 52 50 50 50 query42 71 69 72 69 query43 242 239 206 206 query44 1019544 546 544 query45 107 102 97 97 query46 801 869 557 557 query47 762 757 729 729 query48 308 316 231 231 query49 530 232 176 176 query50 722 265 195 195 query51 7919802878827882 query52 69 68 58 58 query53 196 191 150 150 query54 206 162 147 147 query55 81 64 55 55 query56 193 175 184 175 query57 686 740 663 663 query58 199 162 151 151 query59 1216122510761076 query60 244 168 169 168 query61 123 136 113 113 query62 361 206 177 177 query63 170 138 139 138 query64 2875674 598 598 query65 1606153616211536 query66 1839267 213 213 query67 9826964094079407 query68 27621218768 768 query69 327 233 201 201 query70 679 596 636 596 query71 246 180 165 165 query72 2328170815601560 query73 645 623 344 344 query74 1557120911281128 query75 11801089932 932 query76 2286735 532 532 query77 250 260 219 219 query78 3889382031973197 query79 2288803 572 572 query80 1577324 266 266 query81 489 156 130 130 query82 622 124 93 93 query83 274 204 188 188 query84 296 107 89 89 query85 792 353 289 289 query86 393 175 165 165 query87 1000954 878 878 query88 2754210920952095 query89 272 191 172 172 query90 1979128 129 128 query91 130 119 100 100 query92 79 67 62 62 query93 14231080731 731 query94 630 273 205 205 query95 523 324 223 223 query96 802 582 294 294 query97 1038103610051005 query98 144 138 128 128 query99 446 340 307 307 Total cold run time: 176852 ms Total hot run time: 80910 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific
Re: [PR] [fix](iceberg) Preserve nested field case in created schemas [doris]
hello-stephen commented on PR #67166: URL: https://github.com/apache/doris/pull/67166#issuecomment-5519679901 TPC-H: Total hot run time: 16804 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 6f1193d7f63f92948be69f0c62d6a643ad8be848, data reload: false -- Round 1 -- q1 17607 307230583058 q2 2080270 234 234 q3 10230 852 506 506 q4 4666247 201 201 q5 7682548 389 389 q6 136 112 92 92 q7 521 489 379 379 q8 9227840 940 840 q9 3469239923772377 q10 6526840 687 687 q11 407 196 176 176 q12 633 253 201 201 q13 18128 150011511151 q14 155 149 137 137 q15 q16 445 399 369 369 q17 1410948 845 845 q18 3088224722382238 q19 1275858 781 781 q20 388 287 192 192 q21 5610172618691726 q22 326 268 225 225 Total cold run time: 94009 ms Total hot run time: 16804 ms - Round 2, with runtime_filter_mode=off - q1 3446336033473347 q2 520 396 370 370 q3 2199239522742274 q4 11671161886 886 q5 2135213320832083 q6 164 124 88 88 q7 1032955 874 874 q8 1589140414241404 q9 3110312130463046 q10 1846177616141614 q11 359 269 250 250 q12 461 426 349 349 q13 1482153311571157 q14 183 171 161 161 q15 q16 393 387 364 364 q17 3570330432143214 q18 4800442047234420 q19 868 861 859 859 q20 982 960 831 831 q21 3779304132263041 q22 409 347 332 332 Total cold run time: 34494 ms Total hot run time: 30964 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](iceberg) Preserve nested field case in created schemas [doris]
github-actions[bot] commented on code in PR #67166:
URL: https://github.com/apache/doris/pull/67166#discussion_r3920379758
##
fe/fe-type/src/main/java/org/apache/doris/catalog/StructType.java:
##
@@ -122,15 +123,15 @@ public boolean supportSubType(Type subType) {
public void addField(StructField field) {
field.setPosition(fields.size());
fields.add(field);
-fieldMap.put(field.getName().toLowerCase(), field);
+fieldMap.put(field.getName().toLowerCase(Locale.ROOT), field);
}
public ArrayList getFields() {
return fields;
}
public StructField getField(String fieldName) {
-return fieldMap.get(fieldName.toLowerCase());
+return fieldMap.get(fieldName.toLowerCase(Locale.ROOT));
Review Comment:
The exact-miss fallback still leaves a silent wrong-sibling case. A
pre-change FE under `tr-TR` could persist two source fields `I` and `i` as
distinct runtime/map keys `ı` and `i`. Before upgrade, lookup of selector `I`
lowercased to `ı` and selected the first field; on this head it ROOT-lowercases
to `i` and returns the second sibling as an exact hit, so the legacy fallback
is never reached. The added single-field replay test exercises only an exact
miss and does not cover this collision. Please handle or reject ambiguous
legacy sets before exact-hit precedence, and add a two-field pre-change replay
test through catalog lookup and Nereids analysis/pruning.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [fix](iceberg) Preserve nested field case in created schemas [doris]
Gabriel39 commented on PR #67166: URL: https://github.com/apache/doris/pull/67166#issuecomment-5519054146 /review -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](iceberg) Preserve nested field case in created schemas [doris]
Gabriel39 commented on PR #67166: URL: https://github.com/apache/doris/pull/67166#issuecomment-5519053355 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](iceberg) Preserve nested field case in created schemas [doris]
hello-stephen commented on PR #67166: URL: https://github.com/apache/doris/pull/67166#issuecomment-5510073113 TPC-H: Total hot run time: 16669 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit d6f98ebafd895ddc0fba1d9b6b74ae321cbcfbb8, data reload: false -- Round 1 -- q1 17559 302630183018 q2 2155264 225 225 q3 10190 902 512 512 q4 4677249 201 201 q5 7673560 409 409 q6 135 113 94 94 q7 531 497 387 387 q8 9233892 896 892 q9 3421239723732373 q10 6529843 707 707 q11 400 199 178 178 q12 612 254 193 193 q13 18152 150611551155 q14 156 150 136 136 q15 q16 431 395 368 368 q17 1391922 793 793 q18 3073222422382224 q19 1261860 763 763 q20 383 289 201 201 q21 5604161518721615 q22 323 265 225 225 Total cold run time: 93889 ms Total hot run time: 16669 ms - Round 2, with runtime_filter_mode=off - q1 3395331332943294 q2 514 394 359 359 q3 2191229021742174 q4 11801162892 892 q5 2186211521112111 q6 166 119 87 87 q7 1028942 884 884 q8 1584141013951395 q9 3127310530943094 q10 1851179616121612 q11 353 268 257 257 q12 452 430 349 349 q13 1464153211481148 q14 169 164 159 159 q15 q16 394 389 354 354 q17 3614335332553255 q18 4783442747254427 q19 836 808 933 808 q20 1012958 868 868 q21 3860305333233053 q22 407 342 326 326 Total cold run time: 34566 ms Total hot run time: 30906 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](iceberg) Preserve nested field case in created schemas [doris]
github-actions[bot] commented on code in PR #67166:
URL: https://github.com/apache/doris/pull/67166#discussion_r3916289666
##
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/ExpressionAnalyzer.java:
##
@@ -1277,7 +1285,8 @@ private Optional bindNestedFields(UnboundSlot
unboundSlot, Slot slot
throw new AnalysisException("No such struct field '" +
fieldName + "' in '" + lastFieldName + "'");
}
lastFieldName = fieldName;
-expression = new ElementAt(expression, new
StringLiteral(fieldName));
+// Dereference-created selectors also cross the thrift
boundary and must use runtime identity.
+expression = new ElementAt(expression, new
StringLiteral(field.getName()));
Review Comment:
[P1] Canonicalize computed dotted dereferences too
This line fixes only the `UnboundSlot` path handled by `bindNestedFields`. A
computed base such as ``(CAST(payload AS STRUCT<`Σ`:BIGINT>)).`Σ` `` is parsed
as `DereferenceExpression`; `visitDereferenceExpression` resolves the field but
returns `new ElementAt(expression, dereferenceExpression.child(1))`, and that
newly created node is not revisited by `visitElementAt`. The selector therefore
remains `Σ` while the cast/thrift struct field is `σ`, so BE's bytewise
`iequal` lookup still reports the field missing. This is a separate analyzer
branch from the existing direct-selector thread. Please canonicalize the
resolved field in `visitDereferenceExpression` too (or share one helper) and
add a computed-base dotted-access test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [fix](iceberg) Preserve nested field case in created schemas [doris]
Gabriel39 commented on PR #67166: URL: https://github.com/apache/doris/pull/67166#issuecomment-5512107849 /review -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](iceberg) Preserve nested field case in created schemas [doris]
hello-stephen commented on PR #67166: URL: https://github.com/apache/doris/pull/67166#issuecomment-5510216965 ClickBench: Total hot run time: 14.66 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit d6f98ebafd895ddc0fba1d9b6b74ae321cbcfbb8, data reload: false query1 0.000.000.01 query2 0.080.040.04 query3 0.250.110.12 query4 1.610.110.10 query5 0.170.160.16 query6 1.240.670.70 query7 0.040.010.00 query8 0.050.030.03 query9 0.290.220.21 query10 0.350.340.35 query11 0.160.110.12 query12 0.150.120.12 query13 0.310.300.31 query14 0.440.450.45 query15 0.360.350.35 query16 0.210.240.21 query17 0.740.690.67 query18 0.160.170.18 query19 1.161.141.16 query20 0.020.010.01 query21 15.46 0.170.11 query22 5.050.040.04 query23 16.15 0.240.10 query24 3.050.350.25 query25 0.110.040.04 query26 0.740.150.12 query27 0.040.040.03 query28 3.650.520.28 query29 12.45 3.212.55 query30 0.260.100.12 query31 2.770.360.17 query32 3.520.320.22 query33 1.571.511.45 query34 15.41 2.171.77 query35 1.751.741.74 query36 0.460.290.28 query37 0.060.040.03 query38 0.040.030.03 query39 0.030.030.03 query40 0.110.080.07 query41 0.070.020.03 query42 0.030.020.03 query43 0.030.030.02 Total cold run time: 90.6 s Total hot run time: 14.66 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](iceberg) Preserve nested field case in created schemas [doris]
Gabriel39 commented on code in PR #67166:
URL: https://github.com/apache/doris/pull/67166#discussion_r3915838603
##
fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadata.java:
##
@@ -1336,7 +1336,8 @@ private DorisConnectorException
upgradeNestedModifyError(IcebergTableHandle hand
try {
Types.NestedField current = executeAuthenticated(() ->
catalogOps.withTable(handle.getDbName(),
handle.getTableName(),
-table ->
table.schema().findField(column.getName(;
+table ->
IcebergNestedColumnEvolution.findTopLevelField(
Review Comment:
Fixed. Nested MODIFY now passes the full ConnectorColumnPath to the
parity-error upgrader and resolves the canonical nested target before comparing
types. Missing or invalid targets remain best-effort and keep the original
build error. Added an InMemoryCatalog regression with Root.Leaf ARRAY plus
a top-level LEAF ARRAY decoy.
##
fe/fe-type/src/main/java/org/apache/doris/catalog/Type.java:
##
@@ -526,7 +526,7 @@ public String hideVersionForVersionColumn(
StructType structType = (StructType) this;
for (int i = 0; i < structType.getFields().size(); i++) {
StructField field = structType.getFields().get(i);
-StringBuilder desc = new
StringBuilder(field.getName()).append(":")
+StringBuilder desc = new
StringBuilder(field.getOriginalName()).append(":")
Review Comment:
Fixed. Nereids now rewrites a resolved STRUCT string selector to the field
normalized runtime name before thrift/BE translation, for both element_at and
dotted dereference, while metadata display keeps the original spelling. Added
analyzer coverage for Σ to σ and ẞ to ß, plus an external Iceberg regression
that DESCRIBEs and queries the exact displayed spellings.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [fix](iceberg) Preserve nested field case in created schemas [doris]
Gabriel39 commented on PR #67166: URL: https://github.com/apache/doris/pull/67166#issuecomment-5512105961 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](iceberg) Preserve nested field case in created schemas [doris]
hello-stephen commented on PR #67166: URL: https://github.com/apache/doris/pull/67166#issuecomment-5510160489 TPC-DS: Total hot run time: 82362 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit d6f98ebafd895ddc0fba1d9b6b74ae321cbcfbb8, data reload: false query5 4278402 333 333 query6 396 139 134 134 query7 4945408 252 252 query8 296 120 117 117 query9 8658288328902883 query10 399 225 188 188 query11 54101053935 935 query12 151 71 70 70 query13 1206445 331 331 query14 6121219320742074 query14_11980196619641964 query15 172 118 112 112 query16 945 389 373 373 query17 817 461 388 388 query18 2358337 248 248 query19 172 146 117 117 query20 71 71 71 71 query21 197 102 88 88 query22 5495531053855310 query23 6611631459945994 query23_16104597161085971 query24 72981120802 802 query24_1795 786 794 786 query25 446 304 263 263 query26 1212232 137 137 query27 2800434 265 265 query28 4668150615081506 query29 947 446 363 363 query30 249 154 131 131 query31 827 401 333 333 query32 129 77 79 77 query33 477 225 196 196 query34 1000804 495 495 query35 425 407 343 343 query36 565 588 535 535 query37 121 81 81 81 query38 1015867 826 826 query39 532 503 460 460 query39_1458 466 459 459 query40 207 96 80 80 query41 59 60 56 56 query42 80 75 73 73 query43 246 241 212 212 query44 1025575 558 558 query45 108 101 97 97 query46 767 885 564 564 query47 770 775 709 709 query48 303 308 233 233 query49 528 233 170 170 query50 775 267 196 196 query51 8225802382308023 query52 73 78 64 64 query53 194 213 146 146 query54 234 272 156 156 query55 83 64 59 59 query56 196 177 154 154 query57 681 661 641 641 query58 202 171 179 171 query59 1204122711001100 query60 269 186 180 180 query61 110 113 118 113 query62 345 219 171 171 query63 167 139 145 139 query64 2775744 553 553 query65 1597160015951595 query66 1832270 201 201 query67 9696969696309630 query68 27661159750 750 query69 345 228 203 203 query70 648 623 624 623 query71 244 182 167 167 query72 2297171917341719 query73 632 628 346 346 query74 1575121311611161 query75 11671108955 955 query76 2291742 525 525 query77 241 263 223 223 query78 4084383932833283 query79 2827820 589 589 query80 1593328 293 293 query81 490 160 131 131 query82 649 124 102 102 query83 284 220 194 194 query84 320 110 92 92 query85 792 352 315 315 query86 393 205 161 161 query87 1014972 903 903 query88 2769211120882088 query89 285 198 178 178 query90 1966128 129 128 query91 134 121 100 100 query92 83 70 64 64 query93 15191050685 685 query94 638 247 178 178 query95 522 269 303 269 query96 864 601 266 266 query97 10631079997 997 query98 162 141 131 131 query99 422 341 319 319 Total cold run time: 177988 ms Total hot run time: 82362 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific
Re: [PR] [fix](iceberg) Preserve nested field case in created schemas [doris]
hello-stephen commented on PR #67166: URL: https://github.com/apache/doris/pull/67166#issuecomment-5508526396 # FE UT Coverage Report Increment line coverage ` 92.68% (38/41)` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/67166_d6f98ebafd895ddc0fba1d9b6b74ae321cbcfbb8/fe_increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/67166_d6f98ebafd895ddc0fba1d9b6b74ae321cbcfbb8/fe_report/index.html) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](iceberg) Preserve nested field case in created schemas [doris]
github-actions[bot] commented on code in PR #67166:
URL: https://github.com/apache/doris/pull/67166#discussion_r3913179864
##
fe/fe-type/src/main/java/org/apache/doris/catalog/Type.java:
##
@@ -526,7 +526,7 @@ public String hideVersionForVersionColumn(
StructType structType = (StructType) this;
for (int i = 0; i < structType.getFields().size(); i++) {
StructField field = structType.getFields().get(i);
-StringBuilder desc = new
StringBuilder(field.getName()).append(":")
+StringBuilder desc = new
StringBuilder(field.getOriginalName()).append(":")
Review Comment:
[P1] Keep the displayed Unicode field name executable
For a current Iceberg `STRUCT<Σ:INT>`, this line now shows `Σ` while the
runtime/thrift child name is ROOT-lowered `σ`. Nereids accepts
`element_at(payload, 'Σ')`, but translation preserves that literal and BE
`DataTypeStruct::try_get_position_by_name` uses bytewise `iequal`, so execution
raises `Element not found` (length-changing pairs such as `ẞ`/`ß` fail too).
Base displayed `σ`, which executed. Canonicalize a resolved struct selector to
its runtime name before BE translation, or align BE with the same Unicode
identity, and add an end-to-end query using the exact displayed Unicode
spelling.
##
fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadata.java:
##
@@ -1336,7 +1336,8 @@ private DorisConnectorException
upgradeNestedModifyError(IcebergTableHandle hand
try {
Types.NestedField current = executeAuthenticated(() ->
catalogOps.withTable(handle.getDbName(),
handle.getTableName(),
-table ->
table.schema().findField(column.getName(;
+table ->
IcebergNestedColumnEvolution.findTopLevelField(
Review Comment:
[P2] Resolve the nested path before upgrading this error
This helper is also called by `modifyNestedColumn`, but it receives only
`ConnectorColumn`, whose name is the leaf. For `MODIFY COLUMN Root.leaf
ARRAY` with actual `Root.Leaf ARRAY` and unrelated top-level
`LEAF ARRAY`, `buildColumnType` fails, this new case-insensitive
top-level lookup binds `LEAF`, and the error reports float-to-smallint instead
of the requested int-to-smallint. Pass the full `ConnectorColumnPath` on the
nested call and resolve that path before deriving the parity error; please add
a same-leaf top-level decoy test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [fix](iceberg) Preserve nested field case in created schemas [doris]
Gabriel39 commented on code in PR #67166:
URL: https://github.com/apache/doris/pull/67166#discussion_r3912706267
##
fe/fe-type/src/main/java/org/apache/doris/catalog/StructType.java:
##
@@ -122,15 +123,15 @@ public boolean supportSubType(Type subType) {
public void addField(StructField field) {
field.setPosition(fields.size());
fields.add(field);
-fieldMap.put(field.getName().toLowerCase(), field);
+fieldMap.put(field.getName().toLowerCase(Locale.ROOT), field);
}
public ArrayList getFields() {
return fields;
}
public StructField getField(String fieldName) {
-return fieldMap.get(fieldName.toLowerCase());
+return fieldMap.get(fieldName.toLowerCase(Locale.ROOT));
Review Comment:
Fixed in d6f98ebafd. Exact ROOT-key lookup remains the primary path. A
broader compatibility lookup is now used only after an exact miss and only for
replayed fields whose pre-change metadata has no originalName; that legacy
marker is preserved through Catalog-to-Nereids conversion and back. Newly
created ROOT-distinct names remain separate. Added a Turkish pre-ROOT JSON
replay test plus a guard for current dotless-i fields. The PR description now
defines the supported new-FE-reading-old-Master rolling-upgrade direction and
the metadata-format boundary.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [fix](iceberg) Preserve nested field case in created schemas [doris]
Gabriel39 commented on code in PR #67166:
URL: https://github.com/apache/doris/pull/67166#discussion_r3912709994
##
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/NestedColumnPruning.java:
##
@@ -662,7 +663,7 @@ public void setAccessByPath(List path, int
accessIndex, ColumnAccessPath
accessPartialChild = true;
if (this.type.isStructType()) {
-String fieldName = path.get(accessIndex).toLowerCase();
+String fieldName =
path.get(accessIndex).toLowerCase(Locale.ROOT);
Review Comment:
Fixed in d6f98ebafd. Cast path remapping now compares the already
ROOT-normalized runtime keys exactly, matching the access-tree identity rules.
Added a cast-plus-pruning regression with `i` and dotless `ı`; before the fix
it selected the first source field, and after the fix it preserves and prunes
the requested second sibling.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [fix](iceberg) Preserve nested field case in created schemas [doris]
Gabriel39 commented on PR #67166: URL: https://github.com/apache/doris/pull/67166#issuecomment-5507461810 /review -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](iceberg) Preserve nested field case in created schemas [doris]
Gabriel39 commented on PR #67166: URL: https://github.com/apache/doris/pull/67166#issuecomment-5507459206 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](iceberg) Preserve nested field case in created schemas [doris]
github-actions[bot] commented on code in PR #67166:
URL: https://github.com/apache/doris/pull/67166#discussion_r3912357211
##
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/NestedColumnPruning.java:
##
@@ -662,7 +663,7 @@ public void setAccessByPath(List path, int
accessIndex, ColumnAccessPath
accessPartialChild = true;
if (this.type.isStructType()) {
-String fieldName = path.get(accessIndex).toLowerCase();
+String fieldName =
path.get(accessIndex).toLowerCase(Locale.ROOT);
Review Comment:
[P1] Keep cast pruning on the same ROOT-key identity
The surrounding access-tree lookups now use ROOT-lowercase keys, but
`replacePathByAnotherTree` still matches cast fields with `equalsIgnoreCase`.
Java treats `i`/dotless `ı` (and `σ`/final `ς`) as equal even though the new
`StructType` maps permit them as distinct ROOT keys. For `element_at(CAST(s AS
STRUCT), 'ı')`, the first field can therefore win and the
scan is pruned to `i`, leaving the requested sibling absent or reading the
wrong one. Please compare the canonical keys exactly in the cast remapper and
add a cast-plus-pruning regression for a ROOT-distinct sibling pair.
##
fe/fe-type/src/main/java/org/apache/doris/catalog/StructType.java:
##
@@ -122,15 +123,15 @@ public boolean supportSubType(Type subType) {
public void addField(StructField field) {
field.setPosition(fields.size());
fields.add(field);
-fieldMap.put(field.getName().toLowerCase(), field);
+fieldMap.put(field.getName().toLowerCase(Locale.ROOT), field);
}
public ArrayList getFields() {
return fields;
}
public StructField getField(String fieldName) {
-return fieldMap.get(fieldName.toLowerCase());
+return fieldMap.get(fieldName.toLowerCase(Locale.ROOT));
Review Comment:
[P1] Preserve lookup for pre-ROOT serialized structs
This lookup now uses ROOT keys, but both `StructField.name` and
`StructType.fieldMap` are persisted. An older FE running under `tr-TR`
serialized `STRUCT` with the runtime name/key `ı`; after upgrade this
method looks for ROOT `i` and returns null, so a previously valid
`element_at(s, 'I')` fails. The fallback for missing `originalName` does not
repair either legacy runtime value. Please migrate/rebuild old serialized
struct maps and names (or provide a compatible legacy lookup), add a pre-change
Turkish JSON/image replay test, and define the old-FE rolling direction for
newly written ROOT keys.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [fix](iceberg) Preserve nested field case in created schemas [doris]
Gabriel39 commented on PR #67166: URL: https://github.com/apache/doris/pull/67166#issuecomment-5506429047 /review -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](iceberg) Preserve nested field case in created schemas [doris]
Gabriel39 commented on code in PR #67166:
URL: https://github.com/apache/doris/pull/67166#discussion_r3911999185
##
fe/fe-core/src/main/java/org/apache/doris/nereids/types/StructField.java:
##
@@ -49,7 +50,23 @@ public StructField(String name, DataType dataType, boolean
nullable, String comm
public StructField(String name, DataType dataType, boolean nullable,
String comment,
boolean commentSpecified) {
+this(name, name, dataType, nullable, comment, commentSpecified);
+}
+
+/**
+ * Creates a field with separate names for case-insensitive runtime lookup
and external schema spelling.
+ *
+ * @param name field name normalized internally for runtime lookup
+ * @param originalName field spelling preserved for external schema
metadata
+ * @param dataType field data type
+ * @param nullable whether the field accepts null values
+ * @param comment field comment
+ * @param commentSpecified whether the comment was explicitly specified
+ */
+public StructField(String name, String originalName, DataType dataType,
boolean nullable, String comment,
+boolean commentSpecified) {
this.name = Objects.requireNonNull(name, "name should not be
null").toLowerCase();
Review Comment:
Fixed in a3845bb524. The catalog and Nereids StructField/StructType runtime
keys, named_struct validation, access-path collection, and nested-column
pruning now all use Locale.ROOT. The new Turkish-locale tests cover distinct
uppercase I and dotless ı fields through conversion, expression analysis,
lookup, and pruning.
##
fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergComplexTypeDiff.java:
##
@@ -172,10 +172,11 @@ private static void applyStructChange(UpdateSchema
updateSchema, String path,
Types.NestedField oldField = oldFields.get(i);
Types.NestedField newField = newFields.get(i);
String fieldPath = path + "." + oldField.name();
-existingNames.add(oldField.name());
+existingNames.add(lowercaseName(oldField.name()));
-// Legacy ColumnType rule: existing fields are matched by position
and may not be renamed.
-if (!oldField.name().equals(newField.name())) {
+// Iceberg defines case-insensitive identity with ROOT-lowercase
keys. Java equalsIgnoreCase is
+// broader for some Unicode characters and could otherwise route
an update to the wrong field.
+if
(!lowercaseName(oldField.name()).equals(lowercaseName(newField.name( {
Review Comment:
Fixed in a3845bb524. Flat Iceberg MODIFY now resolves the top-level field
case-insensitively, uses current.name() for every staged type, doc,
nullability, and position update, and resolves AFTER references to their
persisted spelling. The catalog-backed test covers both primitive and complex
updates with mismatched root casing and a mismatched AFTER sibling.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [fix](iceberg) Preserve nested field case in created schemas [doris]
Gabriel39 commented on PR #67166: URL: https://github.com/apache/doris/pull/67166#issuecomment-5506427813 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](iceberg) Preserve nested field case in created schemas [doris]
hello-stephen commented on PR #67166: URL: https://github.com/apache/doris/pull/67166#issuecomment-5434157967 ClickBench: Total hot run time: 14.76 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit 02f366aec0ef4c420bd40acdcdedb2a80d87b715, data reload: false query1 0.010.000.00 query2 0.080.040.04 query3 0.260.110.11 query4 1.620.100.11 query5 0.180.160.16 query6 1.190.660.70 query7 0.030.000.00 query8 0.050.040.03 query9 0.290.230.22 query10 0.340.340.35 query11 0.170.120.11 query12 0.150.120.12 query13 0.290.300.29 query14 0.440.450.44 query15 0.370.340.34 query16 0.210.240.22 query17 0.640.660.66 query18 0.180.170.16 query19 1.131.091.20 query20 0.020.010.01 query21 15.45 0.150.13 query22 5.080.040.05 query23 16.13 0.250.11 query24 3.000.320.27 query25 0.100.050.03 query26 0.730.170.13 query27 0.030.040.02 query28 3.630.570.28 query29 12.46 3.192.55 query30 0.270.120.12 query31 2.780.390.18 query32 3.500.320.23 query33 1.461.701.59 query34 15.34 2.171.77 query35 1.761.751.71 query36 0.470.290.28 query37 0.060.040.03 query38 0.040.030.03 query39 0.030.030.02 query40 0.120.080.08 query41 0.080.020.02 query42 0.040.020.02 query43 0.030.020.02 Total cold run time: 90.24 s Total hot run time: 14.76 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](iceberg) Preserve nested field case in created schemas [doris]
hello-stephen commented on PR #67166: URL: https://github.com/apache/doris/pull/67166#issuecomment-5434130691 TPC-DS: Total hot run time: 81927 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 02f366aec0ef4c420bd40acdcdedb2a80d87b715, data reload: false query5 4300403 320 320 query6 383 139 129 129 query7 4980437 238 238 query8 303 121 132 121 query9 8684287328652865 query10 364 222 198 198 query11 53991028907 907 query12 130 75 69 69 query13 1187455 324 324 query14 6052217320942094 query14_12000202919681968 query15 181 119 108 108 query16 946 380 360 360 query17 932 458 379 379 query18 2328336 241 241 query19 170 142 109 109 query20 70 69 71 69 query21 211 99 86 86 query22 5472544953375337 query23 6609610159715971 query23_16144621260056005 query24 73111109783 783 query24_1785 781 787 781 query25 441 311 264 264 query26 1243231 131 131 query27 2766421 253 253 query28 4703149614821482 query29 975 447 361 361 query30 255 157 133 133 query31 833 403 329 329 query32 131 76 75 75 query33 468 234 180 180 query34 997 801 479 479 query35 408 404 337 337 query36 583 575 515 515 query37 132 80 70 70 query38 1008850 826 826 query39 508 510 471 471 query39_1466 454 466 454 query40 208 94 79 79 query41 59 58 55 55 query42 74 72 73 72 query43 240 242 210 210 query44 1018547 569 547 query45 109 104 101 101 query46 806 858 554 554 query47 765 778 714 714 query48 323 308 267 267 query49 526 246 181 181 query50 772 264 187 187 query51 8165813081218121 query52 69 66 63 63 query53 185 214 142 142 query54 234 181 251 181 query55 71 59 54 54 query56 221 170 163 163 query57 691 653 655 653 query58 202 161 156 156 query59 1222125110801080 query60 236 177 168 168 query61 115 118 117 117 query62 361 207 180 180 query63 173 143 139 139 query64 2638707 583 583 query65 1599159716021597 query66 1874285 199 199 query67 10188 968694329432 query68 27851142753 753 query69 346 228 195 195 query70 674 624 643 624 query71 249 169 163 163 query72 2342172315981598 query73 667 595 345 345 query74 1575122611251125 query75 11731091958 958 query76 2290701 551 551 query77 257 256 200 200 query78 3863373632883288 query79 2500822 573 573 query80 1594338 280 280 query81 498 153 130 130 query82 658 126 103 103 query83 295 216 189 189 query84 268 108 91 91 query85 800 357 296 296 query86 399 175 165 165 query87 1017964 881 881 query88 2771213720962096 query89 298 195 177 177 query90 1968129 130 129 query91 133 120 100 100 query92 74 72 71 71 query93 15131109646 646 query94 683 250 201 201 query95 527 336 259 259 query96 776 571 274 274 query97 1065106510381038 query98 166 140 129 129 query99 429 345 304 304 Total cold run time: 178057 ms Total hot run time: 81927 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific
Re: [PR] [fix](iceberg) Preserve nested field case in created schemas [doris]
hello-stephen commented on PR #67166: URL: https://github.com/apache/doris/pull/67166#issuecomment-5434089626 TPC-H: Total hot run time: 16895 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 02f366aec0ef4c420bd40acdcdedb2a80d87b715, data reload: false -- Round 1 -- q1 17592 305530753055 q2 2168270 223 223 q3 10184 874 500 500 q4 4669242 196 196 q5 7687565 382 382 q6 139 119 94 94 q7 530 498 384 384 q8 9230864 890 864 q9 3450238623862386 q10 6517852 704 704 q11 390 205 179 179 q12 614 260 197 197 q13 18146 155211941194 q14 163 156 141 141 q15 q16 441 400 366 366 q17 1347893 813 813 q18 3011224022082208 q19 1118890 774 774 q20 355 282 202 202 q21 5220180518871805 q22 330 271 228 228 Total cold run time: 93301 ms Total hot run time: 16895 ms - Round 2, with runtime_filter_mode=off - q1 3410333133653331 q2 510 416 371 371 q3 2224231021882188 q4 11881171883 883 q5 2163208321142083 q6 168 121 85 85 q7 1032937 848 848 q8 1605139114211391 q9 3105309730953095 q10 1851182016771677 q11 357 269 250 250 q12 451 435 345 345 q13 1493156011521152 q14 177 168 162 162 q15 q16 390 400 350 350 q17 3577320431373137 q18 4821438447534384 q19 2864889 873 873 q20 964 941 811 811 q21 3729303932463039 q22 389 325 307 307 Total cold run time: 36468 ms Total hot run time: 30762 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](iceberg) Preserve nested field case in created schemas [doris]
github-actions[bot] commented on code in PR #67166:
URL: https://github.com/apache/doris/pull/67166#discussion_r3868407634
##
fe/fe-core/src/main/java/org/apache/doris/nereids/types/StructField.java:
##
@@ -49,7 +50,23 @@ public StructField(String name, DataType dataType, boolean
nullable, String comm
public StructField(String name, DataType dataType, boolean nullable,
String comment,
boolean commentSpecified) {
+this(name, name, dataType, nullable, comment, commentSpecified);
+}
+
+/**
+ * Creates a field with separate names for case-insensitive runtime lookup
and external schema spelling.
+ *
+ * @param name field name normalized internally for runtime lookup
+ * @param originalName field spelling preserved for external schema
metadata
+ * @param dataType field data type
+ * @param nullable whether the field accepts null values
+ * @param comment field comment
+ * @param commentSpecified whether the comment was explicitly specified
+ */
+public StructField(String name, String originalName, DataType dataType,
boolean nullable, String comment,
+boolean commentSpecified) {
this.name = Objects.requireNonNull(name, "name should not be
null").toLowerCase();
Review Comment:
[P1] Normalize the runtime key with Locale.ROOT
The new split-name contract is still locale-dependent on its runtime half:
both changed `StructField` constructors call the no-argument `toLowerCase()`,
and both `StructType` maps/lookups do the same. On an FE started with `tr-TR`,
`STRUCT` stores the runtime key as dotless `ı`, while this PR exports
the preserved `I` to Iceberg and its new identity checks normalize that
spelling to ROOT `i`; `element_at(payload, 'i')` therefore cannot find the
field, and ROOT-distinct `I`/`ı` fields collapse internally. This is upstream
of, and distinct from, the fixed `equalsIgnoreCase` MODIFY thread. Please use
one `Locale.ROOT` key consistently across both `StructField`/`StructType` pairs
and the same normalization in `CreateNamedStruct`,
`AccessPathExpressionCollector`, and `NestedColumnPruning`; otherwise scan
pruning can still select a locale-dependent sibling after analysis succeeds.
Cover uppercase `I` plus dotless `ı` under a Turkish default locale.
##
fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergComplexTypeDiff.java:
##
@@ -172,10 +172,11 @@ private static void applyStructChange(UpdateSchema
updateSchema, String path,
Types.NestedField oldField = oldFields.get(i);
Types.NestedField newField = newFields.get(i);
String fieldPath = path + "." + oldField.name();
-existingNames.add(oldField.name());
+existingNames.add(lowercaseName(oldField.name()));
-// Legacy ColumnType rule: existing fields are matched by position
and may not be renamed.
-if (!oldField.name().equals(newField.name())) {
+// Iceberg defines case-insensitive identity with ROOT-lowercase
keys. Java equalsIgnoreCase is
+// broader for some Unicode characters and could otherwise route
an update to the wrong field.
+if
(!lowercaseName(oldField.name()).equals(lowercaseName(newField.name( {
Review Comment:
[P1] Canonicalize the top-level MODIFY path too
This ROOT-based child match is still unreachable when the existing complex
column itself has mixed-case spelling. For an Iceberg field `Payload
STRUCT`, `MODIFY COLUMN payload
STRUCT` reaches `IcebergCatalogOps.modifyColumn` with
`payload`; that method calls the case-sensitive
`schema.findField(column.getName())` and throws before this diff runs. It also
reuses the requested spelling for the diff, doc, nullability, and position
paths. The nested MODIFY and flat rename paths already resolve
case-insensitively and retain the canonical stored path. Please do the same for
flat MODIFY, use `current.name()` for every staged update, resolve any `AFTER
` reference to its stored spelling too, and add a catalog-backed test
with a case-mismatched mixed-case root.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [fix](iceberg) Preserve nested field case in created schemas [doris]
Gabriel39 commented on code in PR #67166:
URL: https://github.com/apache/doris/pull/67166#discussion_r3868228046
##
fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergComplexTypeDiff.java:
##
@@ -172,10 +172,11 @@ private static void applyStructChange(UpdateSchema
updateSchema, String path,
Types.NestedField oldField = oldFields.get(i);
Types.NestedField newField = newFields.get(i);
String fieldPath = path + "." + oldField.name();
-existingNames.add(oldField.name());
+existingNames.add(oldField.name().toLowerCase(Locale.ROOT));
-// Legacy ColumnType rule: existing fields are matched by position
and may not be renamed.
-if (!oldField.name().equals(newField.name())) {
+// A full-type MODIFY is case-insensitive and must preserve
existing Iceberg spelling; only the
+// dedicated RENAME operation is allowed to change a field name.
+if (!oldField.name().equalsIgnoreCase(newField.name())) {
Review Comment:
Fixed. Complex MODIFY now uses the same Locale.ROOT lowercase identity as
Iceberg and the sibling collision set. Added an InMemoryCatalog regression for
the Unicode sigma boundary and verified that a rejected positional swap leaves
both fields unchanged.
##
fe/fe-type/src/main/java/org/apache/doris/catalog/StructField.java:
##
@@ -51,7 +54,25 @@ public StructField(String name, Type type, String comment,
boolean containsNull)
public StructField(String name, Type type, String comment, boolean
containsNull,
boolean commentSpecified) {
+this(name, name, type, comment, containsNull, commentSpecified);
+}
+
+/**
+ * Creates a field with separate names for case-insensitive runtime lookup
and external schema spelling.
+ *
+ * @param name field name normalized internally for runtime lookup
+ * @param originalName field spelling preserved for external schema
metadata
+ * @param type field type
+ * @param comment field comment
+ * @param containsNull whether the field accepts null values
+ * @param commentSpecified whether the comment was explicitly specified
+ */
+public StructField(String name, String originalName, Type type, String
comment, boolean containsNull,
+boolean commentSpecified) {
this.name = name.toLowerCase();
+// Runtime struct lookup is case-insensitive, but external schemas
such as Iceberg must preserve the
+// original spelling. Keep both so metadata writes never leak the
normalized lookup key.
+this.originalName = originalName;
Review Comment:
Fixed. FILE TVF now preserves the raw PStructField spelling as originalName
while using the Locale.ROOT-lowercase name only for runtime lookup and
duplicate detection. Added a protobuf schema unit test and an end-to-end
Parquet FILE TVF to Iceberg CTAS regression covering direct STRUCT fields plus
STRUCT fields nested under ARRAY and MAP.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [fix](iceberg) Preserve nested field case in created schemas [doris]
Gabriel39 commented on PR #67166: URL: https://github.com/apache/doris/pull/67166#issuecomment-5433551359 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](iceberg) Preserve nested field case in created schemas [doris]
Gabriel39 commented on PR #67166: URL: https://github.com/apache/doris/pull/67166#issuecomment-5433552016 /review -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](iceberg) Preserve nested field case in created schemas [doris]
Gabriel39 commented on PR #67166: URL: https://github.com/apache/doris/pull/67166#issuecomment-5427446276 /review -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](iceberg) Preserve nested field case in created schemas [doris]
Gabriel39 commented on PR #67166: URL: https://github.com/apache/doris/pull/67166#issuecomment-5427446029 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](iceberg) Preserve nested field case in created schemas [doris]
hello-stephen commented on PR #67166: URL: https://github.com/apache/doris/pull/67166#issuecomment-5425974127 # FE UT Coverage Report Increment line coverage ` 90.00% (9/10)` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/67166_1a56c153b87d8d2ff2ae0330924514a240705b0f/fe_increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/67166_1a56c153b87d8d2ff2ae0330924514a240705b0f/fe_report/index.html) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](iceberg) Preserve nested field case in created schemas [doris]
hello-stephen commented on PR #67166: URL: https://github.com/apache/doris/pull/67166#issuecomment-5425871308 ClickBench: Total hot run time: 14.59 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit 1a56c153b87d8d2ff2ae0330924514a240705b0f, data reload: false query1 0.010.000.00 query2 0.070.040.03 query3 0.240.110.12 query4 1.610.100.10 query5 0.180.160.17 query6 1.230.670.69 query7 0.040.010.00 query8 0.040.030.03 query9 0.290.220.21 query10 0.350.340.35 query11 0.160.120.11 query12 0.150.120.12 query13 0.300.310.31 query14 0.440.450.44 query15 0.360.340.34 query16 0.240.230.24 query17 0.680.690.65 query18 0.170.170.18 query19 1.231.141.13 query20 0.020.020.01 query21 15.43 0.180.11 query22 5.060.040.05 query23 16.18 0.260.10 query24 3.030.300.27 query25 0.110.030.03 query26 0.810.170.12 query27 0.040.030.03 query28 3.620.520.29 query29 12.56 3.082.55 query30 0.250.120.12 query31 2.750.380.17 query32 3.530.320.22 query33 1.431.521.44 query34 15.30 2.281.75 query35 1.721.721.68 query36 0.460.300.29 query37 0.060.040.04 query38 0.040.030.03 query39 0.040.030.02 query40 0.110.070.07 query41 0.080.030.02 query42 0.040.020.02 query43 0.030.020.02 Total cold run time: 90.49 s Total hot run time: 14.59 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](iceberg) Preserve nested field case in created schemas [doris]
hello-stephen commented on PR #67166: URL: https://github.com/apache/doris/pull/67166#issuecomment-5425818131 TPC-DS: Total hot run time: 81641 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 1a56c153b87d8d2ff2ae0330924514a240705b0f, data reload: false query5 4264404 328 328 query6 442 130 121 121 query7 4930419 229 229 query8 305 123 116 116 query9 8661285828772858 query10 389 220 182 182 query11 53841038910 910 query12 125 71 77 71 query13 1211456 329 329 query14 6086219720712071 query14_11953195219181918 query15 181 119 110 110 query16 929 375 355 355 query17 821 448 383 383 query18 2331314 231 231 query19 171 131 105 105 query20 70 67 69 67 query21 212 103 85 85 query22 5354521652085208 query23 6642603960396039 query23_15901595660655956 query24 72671098784 784 query24_1754 755 786 755 query25 547 284 226 226 query26 1226237 134 134 query27 2787419 259 259 query28 4689150414811481 query29 926 419 340 340 query30 252 156 123 123 query31 850 390 321 321 query32 132 72 82 72 query33 471 214 167 167 query34 1035823 472 472 query35 401 431 336 336 query36 576 579 531 531 query37 141 76 71 71 query38 1001840 817 817 query39 483 493 470 470 query39_1451 434 446 434 query40 204 89 72 72 query41 52 49 49 49 query42 70 69 74 69 query43 243 237 208 208 query44 1029541 550 541 query45 118 102 96 96 query46 782 828 506 506 query47 743 744 712 712 query48 320 315 235 235 query49 554 243 185 185 query50 731 271 186 186 query51 8298823282738232 query52 68 77 57 57 query53 200 209 151 151 query54 232 188 170 170 query55 84 60 53 53 query56 236 196 157 157 query57 711 661 671 661 query58 209 173 184 173 query59 1204123110891089 query60 243 179 177 177 query61 135 134 133 133 query62 359 215 187 187 query63 168 136 138 136 query64 2740738 643 643 query65 1583150915161509 query66 1793269 211 211 query67 9594983496289628 query68 27881168760 760 query69 372 217 203 203 query70 665 632 619 619 query71 263 186 175 175 query72 2397170415531553 query73 680 529 336 336 query74 1565122211201120 query75 11731088957 957 query76 2292748 553 553 query77 252 258 205 205 query78 3784357431823182 query79 2808846 592 592 query80 1555316 279 279 query81 489 159 132 132 query82 699 137 94 94 query83 279 208 188 188 query84 293 113 92 92 query85 813 344 287 287 query86 407 198 171 171 query87 1003971 881 881 query88 2777209220902090 query89 291 196 178 178 query90 1940120 128 120 query91 128 120 102 102 query92 81 81 69 69 query93 14611089703 703 query94 641 256 225 225 query95 534 325 224 224 query96 803 609 258 258 query97 10871031981 981 query98 167 137 137 137 query99 423 344 320 320 Total cold run time: 177420 ms Total hot run time: 81641 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific
Re: [PR] [fix](iceberg) Preserve nested field case in created schemas [doris]
hello-stephen commented on PR #67166: URL: https://github.com/apache/doris/pull/67166#issuecomment-5425736597 TPC-H: Total hot run time: 17122 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 1a56c153b87d8d2ff2ae0330924514a240705b0f, data reload: false -- Round 1 -- q1 17581 306130753061 q2 2334254 218 218 q3 1 896 534 534 q4 4690255 201 201 q5 7707583 390 390 q6 139 120 93 93 q7 533 508 387 387 q8 9329977 978 977 q9 3474239124042391 q10 6532882 705 705 q11 457 200 181 181 q12 732 266 204 204 q13 18052 155111781178 q14 166 142 135 135 q15 q16 435 399 369 369 q17 1457875 873 873 q18 3073224522532245 q19 1108915 786 786 q20 367 282 210 210 q21 5257174718291747 q22 345 269 237 237 Total cold run time: 93768 ms Total hot run time: 17122 ms - Round 2, with runtime_filter_mode=off - q1 3505336133123312 q2 518 400 375 375 q3 239221402140 q4 12161183893 893 q5 2198210921072107 q6 181 130 88 88 q7 1072909 835 835 q8 1633142714401427 q9 3153310630793079 q10 1931184916581658 q11 362 273 255 255 q12 471 439 344 344 q13 1504157511511151 q14 169 161 155 155 q15 q16 395 404 360 360 q17 3601329232503250 q18 4907438748294387 q19 5338875 850 850 q20 997 968 852 852 q21 3808310431883104 q22 416 342 331 331 Total cold run time: 39597 ms Total hot run time: 30953 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](iceberg) Preserve nested field case in created schemas [doris]
github-actions[bot] commented on code in PR #67166:
URL: https://github.com/apache/doris/pull/67166#discussion_r3862726983
##
fe/fe-type/src/main/java/org/apache/doris/catalog/StructField.java:
##
@@ -51,7 +54,25 @@ public StructField(String name, Type type, String comment,
boolean containsNull)
public StructField(String name, Type type, String comment, boolean
containsNull,
boolean commentSpecified) {
+this(name, name, type, comment, containsNull, commentSpecified);
+}
+
+/**
+ * Creates a field with separate names for case-insensitive runtime lookup
and external schema spelling.
+ *
+ * @param name field name normalized internally for runtime lookup
+ * @param originalName field spelling preserved for external schema
metadata
+ * @param type field type
+ * @param comment field comment
+ * @param containsNull whether the field accepts null values
+ * @param commentSpecified whether the comment was explicitly specified
+ */
+public StructField(String name, String originalName, Type type, String
comment, boolean containsNull,
+boolean commentSpecified) {
this.name = name.toLowerCase();
+// Runtime struct lookup is case-insensitive, but external schemas
such as Iceberg must preserve the
+// original spelling. Keep both so metadata writes never leak the
normalized lookup key.
+this.originalName = originalName;
Review Comment:
[P1] Preserve FILE-TVF spelling before constructing this field
This can preserve only the constructor argument. In `CREATE TABLE
iceberg_catalog... AS SELECT payload FROM S3(...)`, BE sends the raw Parquet
child spelling in `PStructField`, but
`ExternalFileTableValuedFunction.getColumnType()` immediately lowercases
`structField.getName()` and passes that value to the legacy constructor. Both
`name` and `originalName` are therefore already lowercase; the structured CTAS
path reaches `ConnectorColumnConverter` and `IcebergSchemaBuilder` with no raw
spelling to recover, so `CaseSensitive` is silently persisted as
`casesensitive` (also for STRUCT under ARRAY/MAP). Please construct the field
with the raw protobuf spelling, use the lowercase form only for duplicate
detection, and cover FILE TVF to external Iceberg CTAS.
##
fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergComplexTypeDiff.java:
##
@@ -172,10 +172,11 @@ private static void applyStructChange(UpdateSchema
updateSchema, String path,
Types.NestedField oldField = oldFields.get(i);
Types.NestedField newField = newFields.get(i);
String fieldPath = path + "." + oldField.name();
-existingNames.add(oldField.name());
+existingNames.add(oldField.name().toLowerCase(Locale.ROOT));
-// Legacy ColumnType rule: existing fields are matched by position
and may not be renamed.
-if (!oldField.name().equals(newField.name())) {
+// A full-type MODIFY is case-insensitive and must preserve
existing Iceberg spelling; only the
+// dedicated RENAME operation is allowed to change a field name.
+if (!oldField.name().equalsIgnoreCase(newField.name())) {
Review Comment:
[P1] Match using Iceberg's lowercase-key identity
`equalsIgnoreCase` is broader than the `toLowerCase(Locale.ROOT)` identity
used by Iceberg and by `existingNames`. For example, Java treats `Σ` and `ς` as
equal ignoring case even though their ROOT-lowercase keys (`σ` and `ς`) are
distinct and Iceberg permits them as sibling fields. With existing
`STRUCT<Σ:INT, ς:INT>` and requested `STRUCT<ς:BIGINT, Σ:INT>`, both positional
checks pass, so the canonical old path makes `UpdateSchema` commit BIGINT to
`Σ` instead of rejecting the apparent swap; comment and nullability changes can
be misapplied the same way. Please compare the same ROOT-lowercase key used by
Iceberg and the collision set, and add an InMemoryCatalog regression for this
boundary.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [fix](iceberg) Preserve nested field case in created schemas [doris]
Gabriel39 commented on code in PR #67166:
URL: https://github.com/apache/doris/pull/67166#discussion_r3862439968
##
fe/fe-core/src/main/java/org/apache/doris/datasource/connector/converter/ConnectorColumnConverter.java:
##
@@ -238,7 +238,7 @@ public static ConnectorType toConnectorType(Type dorisType)
{
// isCommentSpecified() so the diff can tell an omitted COMMENT
(preserve the current doc) from
// COMMENT '' (clear it) — the comment string is "" for both
(#65329 omit-preserves-metadata).
for (StructField f : struct.getFields()) {
-names.add(f.getName());
+names.add(f.getOriginalName());
Review Comment:
Fixed. Existing struct fields are now matched case-insensitively while every
update uses the existing Iceberg field path, so only the dedicated rename
operation can change persisted spelling. Appended-field collision checks are
also case-insensitive. Added InMemoryCatalog regression coverage for type,
comment, and nullability changes, ARRAY/MAP nesting, and duplicate-case
collisions.
##
fe/fe-core/src/main/java/org/apache/doris/nereids/types/StructField.java:
##
@@ -49,7 +50,13 @@ public StructField(String name, DataType dataType, boolean
nullable, String comm
public StructField(String name, DataType dataType, boolean nullable,
String comment,
boolean commentSpecified) {
+this(name, name, dataType, nullable, comment, commentSpecified);
+}
+
+public StructField(String name, String originalName, DataType dataType,
boolean nullable, String comment,
Review Comment:
Fixed. Added Javadoc that documents the normalized runtime name versus
preserved external schema spelling contract. The CI-equivalent `mvn clean
checkstyle:check` now passes across all 74 FE modules with zero violations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [fix](iceberg) Preserve nested field case in created schemas [doris]
Gabriel39 commented on PR #67166: URL: https://github.com/apache/doris/pull/67166#issuecomment-5424879135 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](iceberg) Preserve nested field case in created schemas [doris]
Gabriel39 commented on PR #67166: URL: https://github.com/apache/doris/pull/67166#issuecomment-5424880347 /review -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](iceberg) Preserve nested field case in created schemas [doris]
github-actions[bot] commented on code in PR #67166:
URL: https://github.com/apache/doris/pull/67166#discussion_r3862201732
##
fe/fe-core/src/main/java/org/apache/doris/nereids/types/StructField.java:
##
@@ -49,7 +50,13 @@ public StructField(String name, DataType dataType, boolean
nullable, String comm
public StructField(String name, DataType dataType, boolean nullable,
String comment,
boolean commentSpecified) {
+this(name, name, dataType, nullable, comment, commentSpecified);
+}
+
+public StructField(String name, String originalName, DataType dataType,
boolean nullable, String comment,
Review Comment:
[P1] Add the required Javadoc for this public constructor
The PR's CheckStyle job fails on this exact declaration with
`MissingJavadocMethod`, so the FE build cannot pass until this new public
overload has Javadoc. Please document it (especially the normalized `name`
versus `originalName` contract) and rerun checkstyle.
##
fe/fe-core/src/main/java/org/apache/doris/datasource/connector/converter/ConnectorColumnConverter.java:
##
@@ -238,7 +238,7 @@ public static ConnectorType toConnectorType(Type dorisType)
{
// isCommentSpecified() so the diff can tell an omitted COMMENT
(preserve the current doc) from
// COMMENT '' (clear it) — the comment string is "" for both
(#65329 omit-preserves-metadata).
for (StructField f : struct.getFields()) {
-names.add(f.getName());
+names.add(f.getOriginalName());
Review Comment:
[P1] Preserve the existing Iceberg spelling during complex MODIFY
This now exports the spelling from the new DDL verbatim. For an existing
Iceberg field `CaseSensitive`, `MODIFY COLUMN payload
STRUCT` reaches
`IcebergComplexTypeDiff.applyStructChange`, whose case-sensitive
`String.equals` treats it as a forbidden rename, even though Doris and the
legacy `ColumnType` path compare the normalized field name and accept this
case-insensitive match. This makes a legal type widening depend on repeating
the remote case exactly (also under ARRAY/MAP). Please match existing fields
case-insensitively and retain `oldField.name()` unless the explicit rename
operation is used.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [fix](iceberg) Preserve nested field case in created schemas [doris]
Gabriel39 commented on PR #67166: URL: https://github.com/apache/doris/pull/67166#issuecomment-5423839880 /review -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](iceberg) Preserve nested field case in created schemas [doris]
hello-stephen commented on PR #67166: URL: https://github.com/apache/doris/pull/67166#issuecomment-5423838878 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR). Please clearly describe your PR: 1. What problem was fixed (it's best to include specific error reporting information). How it was fixed. 2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be. 3. What features were added. Why was this function added? 4. Which code was refactored and why was this part of the code refactored? 5. Which functions were optimized and what is the difference before and after the optimization? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](iceberg) Preserve nested field case in created schemas [doris]
Gabriel39 commented on PR #67166: URL: https://github.com/apache/doris/pull/67166#issuecomment-5423838815 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
