JingsongLi commented on code in PR #1144:
URL: https://github.com/apache/paimon-rust/pull/1144#discussion_r4301943673
##########
crates/paimon/src/table/write_batch_normalize.rs:
##########
@@ -230,6 +230,12 @@ fn normalize_visible_array(
}
match (array.data_type(), expected) {
+ // Java has one STRING value representation. Arrow offset widths do
+ // not change the logical values, including in keys and nested fields.
+ // This specific cast checks offset overflow without enabling value
casts.
+ (DataType::LargeUtf8, DataType::Utf8) => {
Review Comment:
Fixed in 1a0d6217. The conversion now subtracts the first visible offset and
slices the shared value buffer before narrowing offsets to i32. This accepts a
small slice starting beyond 2 GiB without copying its payload, while still
rejecting a visible byte range that exceeds the 32-bit limit.
Added a sparse-mmap regression covering the high starting offset, Unicode,
NULLs, empty slices, buffer sharing and actual range overflow. The 23
normalization tests, 2 core upsert tests, Clippy and rustfmt checks pass.
##########
crates/paimon/src/table/write_batch_normalize.rs:
##########
@@ -230,6 +230,12 @@ fn normalize_visible_array(
}
match (array.data_type(), expected) {
+ // Java has one STRING value representation. Arrow offset widths do
Review Comment:
Keeping LargeBinary/Binary conversions out of this PR. PyPaimon maps binary
to BYTES and large_binary to BLOB; accepting them interchangeably would require
coordinated handling across schema validation and the write/update paths,
expanding the scope considerably.
The existing rules remain in place, with regression assertions that neither
direction is implicitly accepted by write normalization.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
