github-actions[bot] commented on code in PR #68297:
URL: https://github.com/apache/doris/pull/68297#discussion_r4061082470
##########
be/src/core/data_type_serde/data_type_varbinary_serde.cpp:
##########
@@ -301,6 +305,82 @@ Status
DataTypeVarbinarySerDe::deserialize_one_cell_from_json(IColumn& column, S
return Status::OK();
}
+Status DataTypeVarbinarySerDe::from_string(StringRef& str, IColumn& column,
+ const FormatOptions& options) const
{
+ // Partition structs use the same hex representation as nested VARBINARY
output. Decode it
+ // before appending so arbitrary bytes survive JSON transport instead of
becoming NULL.
+ if (str.size < 2 || str.data[0] != '0' || str.data[1] != 'x' || (str.size
- 2) % 2 != 0 ||
+ str.size - 2 > std::numeric_limits<int>::max()) {
+ return Status::InvalidArgument("Invalid VARBINARY hex representation");
+ }
+ // The INT_MAX guard also makes narrowing to the decoder's 32-bit offset
type safe.
+ const auto hex_size = cast_set<ColumnString::Offset>(str.size - 2);
+ std::string bytes(hex_size / 2, '\0');
+ if (string_hex::hex_decode(str.data + 2, hex_size, bytes.data()) !=
bytes.size()) {
+ return Status::InvalidArgument("Invalid VARBINARY hex representation");
+ }
+ assert_cast<ColumnVarbinary&>(column).insert_data(bytes.data(),
bytes.size());
+ return Status::OK();
+}
+
+Status DataTypeVarbinarySerDe::deserialize_one_cell_from_hive_text(
+ IColumn& column, Slice& slice, const FormatOptions& options,
+ int hive_text_complex_type_delimiter_level) const {
+ // Hive LazyBinary uses lenient Base64 (including URL-safe letters and
whitespace),
Review Comment:
[P1] Preserve Hive TEXTFILE binary rows across upgrades
This unconditionally changes Doris's persisted Hive TEXTFILE representation
from raw bytes to Base64. VHivePartitionWriter writes this method's output
through VCSVTransformer, and later scans feed those file bytes back through
TextReader to this decoder. After an upgrade, a legacy raw value such as
dGVzdDI= is therefore silently returned as test2; in a mixed-version cluster,
an old reader returns a new writer's Base64 text as the value. The raw fallback
cannot disambiguate legacy payloads that happen to use the Base64 alphabet.
Please add an explicit table/catalog compatibility or migration mode for legacy
raw files, and cover old-writer/new-reader plus new-writer/old-reader scalar
and nested cases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
