[
https://issues.apache.org/jira/browse/FLINK-40318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ramin Gharib updated FLINK-40318:
---------------------------------
Description:
Flink has no way to check that a variant binary is well formed before using it.
{{BinaryVariant}} validates only the metadata version and the 16 MiB size
limit. Everything else relies on {{checkIndex}} failing lazily during access.
Spark's {{VariantUtil}}, which {{BinaryVariantUtil}} is derived from, provides
{{isValidVariant(value, metadata)}} backed by a recursive {{validateImpl}}. It
checks the metadata version, structural well formedness of every value, access
bounds, primitive type-info validity, dictionary-id existence for object
fields, and recurses through nested objects and arrays. Neither method was
carried over.
This matters once variant bytes arrive from outside the job, for example from a
Parquet or Avro source or from restored state.
h3. Scope
Port {{isValidVariant}} and {{validateImpl}} as the implementation of the
planned {{IS_VALID_VARIANT}} SQL function.
Do not call the check automatically when a {{BinaryVariant}} is constructed or
read. That walks every value in full, even when a query reads one field. Users
decide where to pay for the check by calling {{IS_VALID_VARIANT}}.
The check answers "can every part of the value be read", the same as Spark:
* Key order and duplicate keys are not checked. Reading works without them, and
Spark does not check them either.
* A type id that Flink cannot read makes the value invalid. Until the new
primitive codes 17-20 are supported, this includes a UUID, a TIME or a
nanosecond timestamp written by another engine.
was:
Flink has no way to check that a variant binary is well formed before using it.
\{{BinaryVariant}} validates only the metadata version, and everything else
relies on \{{checkIndex}} failing lazily during access.
Spark's \{{VariantUtil}}, which \{{BinaryVariantUtil}} is derived from,
provides \{{isValidVariant(value, metadata)}} backed by a recursive
\{{validateImpl}}. It checks the metadata version, structural well formedness
of every value, access bounds, primitive type-info validity, dictionary-id
existence for object fields, and recurses through nested objects and arrays.
Neither method was carried over.
This matters once variant bytes arrive from outside the job, for example from a
Parquet or Avro source or from restored state, where malformed input should be
rejected up front with a clear error instead of surfacing as an index failure
somewhere deep in a query.
h3. Scope
Port \{{isValidVariant}} and \{{validateImpl}}, and decide where to call them,
most likely when constructing a \{{BinaryVariant}} from external bytes rather
than on every access.
> Add a validation entry point for a VARIANT binary
> -------------------------------------------------
>
> Key: FLINK-40318
> URL: https://issues.apache.org/jira/browse/FLINK-40318
> Project: Flink
> Issue Type: Improvement
> Components: API / Core
> Reporter: Ramin Gharib
> Assignee: Ramin Gharib
> Priority: Minor
>
> Flink has no way to check that a variant binary is well formed before using
> it. {{BinaryVariant}} validates only the metadata version and the 16 MiB size
> limit. Everything else relies on {{checkIndex}} failing lazily during access.
> Spark's {{VariantUtil}}, which {{BinaryVariantUtil}} is derived from,
> provides {{isValidVariant(value, metadata)}} backed by a recursive
> {{validateImpl}}. It checks the metadata version, structural well formedness
> of every value, access bounds, primitive type-info validity, dictionary-id
> existence for object fields, and recurses through nested objects and arrays.
> Neither method was carried over.
> This matters once variant bytes arrive from outside the job, for example from
> a Parquet or Avro source or from restored state.
> h3. Scope
> Port {{isValidVariant}} and {{validateImpl}} as the implementation of the
> planned {{IS_VALID_VARIANT}} SQL function.
> Do not call the check automatically when a {{BinaryVariant}} is constructed
> or read. That walks every value in full, even when a query reads one field.
> Users decide where to pay for the check by calling {{IS_VALID_VARIANT}}.
> The check answers "can every part of the value be read", the same as Spark:
> * Key order and duplicate keys are not checked. Reading works without them,
> and Spark does not check them either.
> * A type id that Flink cannot read makes the value invalid. Until the new
> primitive codes 17-20 are supported, this includes a UUID, a TIME or a
> nanosecond timestamp written by another engine.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
