[
https://issues.apache.org/jira/browse/FLINK-41298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ramin Gharib reassigned FLINK-41298:
------------------------------------
Assignee: Ramin Gharib
> PARSE_JSON, TRY_PARSE_JSON and the raw format drop content after the first
> JSON value
> -------------------------------------------------------------------------------------
>
> Key: FLINK-41298
> URL: https://issues.apache.org/jira/browse/FLINK-41298
> Project: Flink
> Issue Type: Bug
> Components: API / Core, Table SQL / Runtime
> Affects Versions: 2.1.0, 2.2.0, 2.3.0
> Reporter: Ramin Gharib
> Assignee: Ramin Gharib
> Priority: Major
>
> {{PARSE_JSON}} reads the first JSON value in its input and stops. It never
> checks that the input ends there, so anything after that value is dropped
> without an error. Two JSON documents in one string come back as the first
> one, and the second is lost.
> {noformat}
> SELECT JSON_STRING(PARSE_JSON('{"a":1} xyz')); -- {"a":1}, expected an
> error
> SELECT JSON_STRING(PARSE_JSON('[1]]')); -- [1], expected an error
> SELECT JSON_STRING(PARSE_JSON('{"a":1}{"a":2}')); -- {"a":1}, expected an
> error
> SELECT JSON_STRING(TRY_PARSE_JSON('{"a":1} xyz')); -- {"a":1}, expected NULL
> {noformat}
> Output of {{BinaryVariantInternalBuilder.parseJson(input, false)}} on the
> 2.3.0 jars:
> {noformat}
> input result
> {"a":1} xyz {"a":1}
> [1]] [1]
> {"a":1}{"a":2} {"a":1}
> {"a":1}\n{"a":2} {"a":1}
> 1 2 1
> hello JsonParseException: Unrecognized token 'hello'
> {noformat}
> Two entry points on master reach this code:
> ||Entry point||Builder method||
> |{{PARSE_JSON}}, {{TRY_PARSE_JSON}}|{{parseJson(byte[], boolean)}}, through
> {{SqlJsonUtils.parseJsonToVariant}}|
> |{{raw}} format, VARIANT column|{{parseJson(String, boolean)}}, in
> {{RawFormatDeserializationSchema}}|
> Both overloads end the same way:
> {code:java}
> try (JsonParser parser = JSON_FACTORY.createParser(bytes)) {
> parser.nextToken();
> return parseJson(parser, allowDuplicateKeys); // returns after one value
> }
> {code}
> The same class already rejects content glued to a number, with "Unexpected
> content after the JSON number". It does not check what follows the whole
> value.
> *Proposal*
> In {{parseJson(String, boolean)}} and {{parseJson(byte[], boolean)}}, call
> {{parser.nextToken()}} once more after the value is built, and throw a
> {{JsonParseException}} if it is not null. Then {{PARSE_JSON}} fails,
> {{TRY_PARSE_JSON}} returns NULL, and the {{raw}} format reports a
> deserialization error for the record. Whitespace after the value stays
> allowed, because Jackson skips it.
> {{parseJson(JsonParser, boolean)}} stays as it is. Its caller owns the parser
> and decides what may follow.
> *Relation to FLINK-40225*
> FLINK-40225 asked for the same check in {{IS JSON}}, {{JSON_VALUE}} and the
> other JSON functions. It was closed as Won't Fix: it breaks running jobs, and
> lenient parsing is the default in Jackson and Spark. This case differs in two
> ways:
> # {{PARSE_JSON}} stores what it parsed. The dropped text is lost data, not a
> looser validity check. A {{raw}} record that holds newline-delimited JSON
> documents keeps only the first one.
> # The {{raw}} format's VARIANT support is new in 2.4.0, which is not released
> yet. Making it strict now breaks no running job.
> *Open question*
> {{PARSE_JSON}} has been out since 2.1.0, so rejecting trailing content turns
> input that parses today into an error. Should that part go to the dev@ list
> first, as FLINK-40225 suggests, while the {{raw}} format gets the check for
> 2.4.0?
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
