github-actions[bot] commented on code in PR #70470:
URL: https://github.com/apache/doris/pull/70470#discussion_r4547177441
##########
be/src/core/data_type_serde/data_type_array_serde.cpp:
##########
@@ -847,4 +848,22 @@ Status DataTypeArraySerDe::decode_bson_value(IColumn&
column, const BsonValueVie
return Status::OK();
}
+Status DataTypeArraySerDe::deserialize_one_cell_from_simdjson(
+ IColumn& column, simdjson::ondemand::value& input, const
FormatOptions& options,
+ SimdJsonReadContext& context, const std::string& column_name) const {
+ if (input.type().value() != simdjson::ondemand::json_type::array) {
+ return Status::DataQualityError("Json value isn't array, but the
column `{}` is array.",
+ column_name);
+ }
+ auto& target = assert_cast<ColumnArray&>(column);
+ auto& data = target.get_data();
+
+ for (simdjson::ondemand::value element : input.get_array()) {
+ RETURN_IF_ERROR(nested_serde->deserialize_one_cell_from_simdjson(
+ data, element, options, context, column_name + ".element"));
+ }
+ target.get_offsets().push_back(data.size());
Review Comment:
[P1] Roll back partial children before deriving the next array offset
For a Hive/OpenX JSON table with `ignore.malformed.json=true` and nullable
`a ARRAY<ARRAY<INT>>`, read `{"a":[[1],2]}` followed by `{"a":[]}` in the same
batch. The first inner `[1]` is appended before `2` fails the nested-array type
check with `DataQualityError`. The reader truncates only outer rows, then
appends the malformed row as NULL; that repeats outer offset 0 while leaving
one child behind. This line now appends `data.size()==1` for the next empty
array, so it becomes `[[1]]` instead of `[]`.
The old `offsets.back() + field_count` kept that empty row's offset at 0.
Same-type query mapping passes the column through, so the newly exposed value
reaches the query result. Restore child sizes on failure before reusing the
column, and add a reader test for an ignored malformed nested array followed by
a valid empty array. This is verified by static control-flow inspection; no
runtime reproduction was run.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
