mrhhsg opened a new pull request, #68484:
URL: https://github.com/apache/doris/pull/68484
### What problem does this PR solve?
Issue Number: None
Problem Summary:
`JSON_PARSE`, `JSON_PARSE_ERROR_TO_NULL`, `JSON_PARSE_ERROR_TO_VALUE`,
`CAST(... AS JSON)`, JSON column loads and every other path that goes through
`JsonbParser` accepted invalid JSON numbers such as `01`, `00`, `-01`, `1.`,
`01.5` or `1.e5` and silently turned them into a different valid value: `01`
became `0`, `1.` became `1`, `1.e5` became `100000`, and `1e400` became
`inf`.
This happened at the top level as well as inside arrays and objects, so
malformed source data was normalized instead of rejected.
Root cause: `JsonbParser::parse_number_success` treated simdjson's
`NUMBER_ERROR` as success in order to support integers just above the uint64
range (simdjson reports `18446744073709551616` as `NUMBER_ERROR` and longer
integers as `BIGINT_ERROR`). However, simdjson also returns `NUMBER_ERROR`
for
malformed tokens (leading zeros, a trailing `.`, an incomplete exponent,
trailing garbage) and for values beyond the double range.
`simdjson_result::get`
leaves the caller's `simdjson::ondemand::number` untouched on error, so the
writer emitted its zero-initialized payload for integer-looking tokens; for
float-looking tokens the `number == 0` branch re-parsed the raw token with
`StringParser::string_to_float`, which accepts `1.`, `01.5` and `1e400`.
Fix: `write_number` only falls back to the raw token for `NUMBER_ERROR` and
`BIGINT_ERROR`; any other error is returned as `InvalidArgument`. The raw
token is validated against the JSON number grammar
`-?(0|[1-9][0-9]*)(\.[0-9]+)?([eE][+-]?[0-9]+)?` (after trimming the trailing
JSON whitespace that `raw_json_token()` may include); a digits-only token is
then parsed as int128, anything else as a finite double. A root number must
additionally reach the end of the document, because simdjson reports
`NUMBER_ERROR` for `18446744073709551616 0` before it checks for trailing
content and the raw token stops at the first token. Malformed input now fails
with `InvalidArgument`, so `JSON_PARSE` raises an error and
`JSON_PARSE_ERROR_TO_NULL` returns NULL. The successful path takes the number
type from the parsed `number` object and no longer re-parses zero values with
`StringParser`, since a value simdjson parsed successfully is already exact.
Before:
```
SELECT json_parse('01'), json_parse('[1.]'), json_parse('{"k":1.e5}'),
json_parse('1e400');
-- 0 [1] {"k":100000} inf
```
After:
```
SELECT json_parse('01');
-- ERROR: Parse json document failed at row 0, error:
[INVALID_ARGUMENT]simdjson get_number failed: NUMBER_ERROR: Problem while
parsing a number, raw string is: 01
SELECT json_parse_error_to_null('01');
-- NULL
```
Valid large numbers keep working: `18446744073709551616` and
`-9223372036854775809` are still stored as int128, and integers beyond int128
still fall back to double. As a side effect, a root number longer than
simdjson's root scalar buffer (1082 bytes) that contains a fraction or an
exponent is now accepted through the same raw-token path instead of failing;
the nested form of such numbers was already accepted.
### Release note
JSON text parsing (`JSON_PARSE` and its variants, `CAST(... AS JSON)`,
`JSON_VALID`, and loading into JSON columns) now rejects malformed JSON
number
tokens such as `01`, `00`, `1.` or `1.e5`, and numbers beyond the double
range
such as `1e400`, instead of silently converting them to a different value.
### Check List (For Author)
- Test:
- Unit Test: `JsonbParserTest.*` in
`be/test/util/jsonb_parser_simd_test.cpp`
- Regression test: `datatype_p0/json/test_json_parse_invalid_number`
- Behavior changed: Yes (invalid JSON number tokens and numbers beyond the
double range are rejected on every JSON text parsing path; previously they were
normalized to a different value)
- Does this need documentation: No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
