HappenLee commented on code in PR #70470:
URL: https://github.com/apache/doris/pull/70470#discussion_r4545956037
##########
be/src/core/data_type_serde/data_type_jsonb_serde.cpp:
##########
@@ -390,4 +392,20 @@ Status DataTypeJsonbSerDe::decode_bson_value(IColumn&
column, const BsonValueVie
: Status::InvalidArgument("BSON Extended JSON exceeds
JSONB limits");
}
+Status DataTypeJsonbSerDe::deserialize_one_cell_from_simdjson(
+ IColumn& column, simdjson::ondemand::value& input, const
FormatOptions& options,
+ SimdJsonReadContext& context, const std::string& column_name) const {
+ const auto type = input.type().value();
+ if (type != simdjson::ondemand::json_type::object &&
+ type != simdjson::ondemand::json_type::array) {
+ return DataTypeSerDe::deserialize_one_cell_from_simdjson(column,
input, options, context,
+ column_name);
+ }
+ auto& writer = context.jsonb_writer;
+ RETURN_IF_ERROR(JsonbParser::parse_value(input, writer, context.cache()));
+ const auto size = writer.getOutput()->getSize();
+ column.insert_data(writer.getOutput()->getBuffer(), size);
Review Comment:
[P2] Preserve the existing 1 GiB JSONB value limit
The previous path calls `deserialize_one_cell_from_json()` ->
`JsonBinaryValue::from_json_string()`, which rejects an encoded JSONB value
larger than `JsonBinaryValue::MAX_LENGTH` (1 GiB) with `InternalError` before
inserting it into the column. This new object/array path bypasses that
validation and inserts the writer output directly, while scalar inputs still
use the old checked path.
Consequently, an object/array whose encoded JSONB size exceeds 1 GiB can now
be accepted where it was previously rejected. The generic ColumnString
total-byte limit is not an equivalent per-value JSONB check. This changes
compatibility for a refactor intended to preserve existing behavior.
Please retain the same `MAX_LENGTH` validation and error behavior before
`insert_data()`, or share that validation between both entry points. This
finding is based on the code paths; I have not run a >1 GiB input reproduction.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
