Gabriel39 commented on code in PR #67784:
URL: https://github.com/apache/doris/pull/67784#discussion_r4003258234
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergUtils.java:
##########
@@ -716,14 +716,16 @@ private static Type
icebergPrimitiveTypeToDorisType(org.apache.iceberg.types.Typ
case STRING:
return Type.STRING;
case UUID:
- return enableMappingVarbinary ?
ScalarType.createVarbinaryType(16) : Type.STRING;
+ return ScalarType.createVarbinaryType(16);
case BINARY:
- return enableMappingVarbinary ?
ScalarType.createVarbinaryType(VarBinaryType.MAX_VARBINARY_LENGTH)
- : Type.STRING;
+ // Arbitrary binary payloads are not valid UTF-8 in general,
so exposing them as
+ // STRING makes Arrow clients reject otherwise valid Iceberg
values.
+ return
ScalarType.createVarbinaryType(VarBinaryType.MAX_VARBINARY_LENGTH);
Review Comment:
Fixed in 049705dd44. Iceberg partition transforms now accept VARBINARY:
bucket hashes the raw bytes and truncate slices bytes rather than UTF-8
characters, preserving nullable inputs. Dynamic identity extraction also
accepts the binary column layout. Commit values use typed 0x-prefixed hex;
paths render binary/fixed as base64 and UUID as canonical text before escaping.
FE commit conversion reconstructs ByteBuffer/UUID and validates fixed/UUID
lengths. Tests cover non-UTF-8 bytes, empty values, nullable transforms, writer
routing, and BINARY/FIXED/UUID DataFile commit values. Validation: 90 FE tests,
27 BE tests, non-UT object compilation with strict narrowing diagnostics,
Checkstyle, and clang-format 16 passed. The external regression run is pending
CI.
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/hive/HiveMetaStoreClientHelper.java:
##########
@@ -648,8 +648,7 @@ public static Type hiveTypeToDorisType(String hiveType, int
timeScale, boolean e
case "string":
return ScalarType.createStringType();
case "binary":
- return enableMappingVarbinary ?
ScalarType.createVarbinaryType(VarBinaryType.MAX_VARBINARY_LENGTH)
- : ScalarType.createStringType();
+ return
ScalarType.createVarbinaryType(VarBinaryType.MAX_VARBINARY_LENGTH);
Review Comment:
Fixed in 049705dd44 using the partition-specific compatible mapping option.
HMSExternalTable keeps BINARY partition columns as VARCHAR, matching the
existing HMS text partition-name contract and the supported
metadata-literal/writer-routing paths. Non-partition BINARY columns remain
VARBINARY. This deliberately does not introduce a new encoding for existing
Hive partition directory names. The regression test checks partition-column
normalization, materialization of an existing text partition value, and
preservation of VARBINARY for the data column. All 90 selected FE tests and
Checkstyle passed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
