[
https://issues.apache.org/jira/browse/FLINK-40218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-40218:
-----------------------------------
Labels: pull-request-available (was: )
> Allow building a Variant from a token source, without a String round-trip
> -------------------------------------------------------------------------
>
> Key: FLINK-40218
> URL: https://issues.apache.org/jira/browse/FLINK-40218
> Project: Flink
> Issue Type: Improvement
> Components: API / Type Serialization System
> Reporter: Ramin Gharib
> Assignee: Ramin Gharib
> Priority: Major
> Labels: pull-request-available
>
> *Description*
> The only way to build a {{Variant}} from JSON today is from a {{{}String{}}}.
> {{PARSE_JSON}} and the internal
> {{BinaryVariantInternalBuilder.parseJson(String)}} both take a string, create
> a Jackson {{JsonParser}} over it, and walk the tokens in {{{}buildJson{}}}.
> A consumer that already holds parsed JSON cannot use that work. It must
> serialize its data back to a {{String}} and hand it to Flink, which parses it
> a second time. This shows up wherever a format or connector wants a VARIANT
> column: the format has already parsed the record into a tree or a parser of
> its own, yet it is forced to {{toString()}} and let flink-core re-parse.
> Two things drive this:
> # *A redundant pass.* The value is parsed by the caller, serialized to a
> {{{}String{}}}, then parsed again by flink-core.
> # *The Jackson type is not portable.* {{buildJson}} is written against
> Flink's shaded {{{}org.apache.flink.shaded.jackson2...JsonParser{}}}. A
> caller that uses a differently relocated or unshaded Jackson cannot pass its
> own parser in. A {{String}} is the only type that crosses that boundary,
> which is why the round-trip exists.
> Concretely, for one VARIANT value read through a JSON-based format:
> {code:java}
> Current:
> bytes ──(caller parses)──▶ tree ──(toString)──▶ String ──(flink
> re-parses)──▶ tokens ──▶ Variant
> [needed] [wasted] [wasted] {code}
> *Proposed change*
> Introduce a small token-source abstraction in
> {{org.apache.flink.types.variant}} and drive {{buildJson}} off it. The
> concrete Jackson parser becomes one implementation of that abstraction rather
> than a hard dependency.
> {code:java}
> // flink-core: org.apache.flink.types.variant
> @Internal
> public interface VariantJsonSource {
>
> /** Advance to the next token and return it. Returns END_INPUT once input
> is exhausted. */
> Token next() throws IOException;
> /** Field name of the current FIELD_NAME token. */
> String fieldName() throws IOException;
> /** Text of the current STRING token. */
> String stringValue() throws IOException;
> /**
> * Raw literal of the current NUMBER token, e.g. "1e5", "100000", "3.14".
> * The builder decides long vs decimal vs double from this text, so
> numeric
> * classification stays in one place and every source behaves identically.
> */
> String numberText() throws IOException;
> enum Token {
> START_OBJECT, END_OBJECT,
> START_ARRAY, END_ARRAY,
> FIELD_NAME,
> STRING, NUMBER, TRUE, FALSE, NULL,
> END_INPUT
> }
> } {code}
> {code:java}
> // BinaryVariantInternalBuilder
> public static BinaryVariant parseJson(VariantJsonSource source, boolean
> allowDuplicateKeys) throws IOException {
> // buildJson, expressed purely against VariantJsonSource
> }
> {code}
> With the interface a JSON-based format wraps the value it already parsed and
> builds the Variant in a single walk:
> {code:java}
> Proposed:
> bytes ──(caller parses)──▶ tree ──(wrap as source)──▶ tokens ──▶ Variant
> [needed] [no serialize] [no re-parse] {code}
> *Design notes*
> * Number classification stays in the builder. The source exposes the raw
> number literal through {{{}numberText(){}}}, so the long-vs-decimal-vs-double
> decision, including the rule that scientific notation goes to {{{}double{}}},
> lives in one place. Sources do not re-implement it and cannot diverge.
> * Boolean values are carried by the {{TRUE}} / {{FALSE}} tokens, so no
> separate value accessor is needed.
> * The interface uses only JDK types plus its own enum. It names no Jackson
> type, shaded or otherwise, so any caller can implement it regardless of how
> its Jackson is relocated.
> * The exact signature is open. {{numberText}} versus typed number accessors
> is the main choice, and the text form is preferred because it is the only
> shape that preserves the current numeric routing without leaking policy into
> each source. This can be finalized in the PR.
> *Compatibility*
> Additive and {{{}@Internal{}}}. {{parseJson(String)}} stays and is
> reimplemented on top of the new method by adapting the existing Jackson
> parser to {{{}VariantJsonSource{}}}. {{PARSE_JSON}} and {{TRY_PARSE_JSON}}
> behavior is unchanged.
> *Verifying this change*
> * Existing {{PARSE_JSON}} / {{TRY_PARSE_JSON}} and
> {{BinaryVariantInternalBuilder}} tests pass unchanged, proving the String
> path still behaves identically.
> * Add a test that builds a {{Variant}} from a non-Jackson
> {{VariantJsonSource}} implementation and asserts it is byte-for-byte equal to
> the Variant produced by {{parseJson(String)}} for the same document, across
> objects, arrays, and all scalar and numeric forms.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
