twalthr commented on code in PR #28758:
URL: https://github.com/apache/flink/pull/28758#discussion_r3622420298
##########
docs/content.zh/docs/sql/reference/data-types.md:
##########
@@ -1507,19 +1507,62 @@ close to the semantics of JSON. Compared to `ROW` and
`STRUCTURED` type, `VARIAN
flexibility to support highly nested and evolving schema.
`VARIANT` allows for deeply nested data structures, such as arrays within
arrays, maps within maps,
-or combinations of both.This capability makes `VARIANT` ideal for scenarios
where data complexity
+or combinations of both. This capability makes `VARIANT` ideal for scenarios
where data complexity
and nesting are significant.
`VARIANT` allows schema evolution, enabling the storage of data with changing
or unknown schemas
without requiring upfront schema definition. For example, if a new field is
added to the data, it
can be directly incorporated into the `VARIANT` data without modifying the
table schema. This is
particularly useful in dynamic environments where schemas may evolve over time.
+A `VARIANT` stores a single value of one of the following kinds: `NULL`,
`BOOLEAN`, `TINYINT`,
+`SMALLINT`, `INT`, `BIGINT`, `FLOAT`, `DOUBLE`, `DECIMAL` (up to precision
38), `STRING`, `DATE`,
+`TIMESTAMP`, `TIMESTAMP_LTZ`, `BYTES`, or a nested array or object.
`TIMESTAMP` and `TIMESTAMP_LTZ`
+are stored with microsecond precision and `DATE` as a day count. There is no
`TIME` kind.
+
+The `PARSE_JSON` function produces only the kinds that JSON syntax can express:
+
+| JSON input | Stored `VARIANT` kind
|
+|-----------------------------------------------------|-------------------------------------------------|
+| `null` | `NULL`
|
+| `true` / `false` | `BOOLEAN`
|
+| Integer within the 64-bit signed range | smallest of
`TINYINT`/`SMALLINT`/`INT`/`BIGINT` |
+| Integer beyond 64-bit, up to 38 significant digits | `DECIMAL`
|
+| Decimal in plain notation, up to precision/scale 38 | `DECIMAL`
|
+| Number in scientific notation, e.g. `1.5e3` | `DOUBLE`
|
+| Number exceeding 38 digits of precision or scale | `DOUBLE`
|
+| String | `STRING`
|
+| Array | array (elements
encoded by the same rules) |
+| Object | object (values encoded
by the same rules) |
+
+Because JSON has no literal for float, date, timestamp, or binary values,
`PARSE_JSON` never
+produces `FLOAT`, `DATE`, `TIMESTAMP`, `TIMESTAMP_LTZ`, or `BYTES`. A
`VARIANT` can still hold
+those kinds when built by other producers, such as the programmatic
`VariantBuilder` API or format
+conversions like Avro.
+
+The JSON specification has no `NaN` or infinity literals, so
`PARSE_JSON('NaN')`,
+`PARSE_JSON('Infinity')`, and `PARSE_JSON('-Infinity')` fail, and
`TRY_PARSE_JSON` returns `NULL`.
+A `VARIANT` has no dedicated kind for these values. To keep one, store it as a
JSON string and cast
+it back out, for example `CAST(CAST(PARSE_JSON('"Infinity"') AS STRING) AS
FLOAT)`.
+
A primitive-valued `VARIANT` can be converted to a scalar type with `CAST` or
`TRY_CAST`. Numeric
-targets are lenient: a variant holding any numeric value casts to any numeric
type, so a JSON integer
-such as `PARSE_JSON('42')` casts to `INT` or `BIGINT`. Other targets require
the stored value to be of
-the matching kind. When the value cannot be converted, `CAST` fails the job
and `TRY_CAST` returns
-`NULL`. Use the `JSON_STRING` function to obtain the JSON string
representation of a `VARIANT`.
+targets accept any numeric value, so `PARSE_JSON('42')` casts to `INT`,
`BIGINT`, or `DOUBLE`. A
+value outside an integer or `DECIMAL` target's range fails `CAST` and returns
`NULL` for
+`TRY_CAST`. Other targets require the stored value to be of the matching kind,
and a mismatch is
+handled the same way. Casting a `VARIANT` to `CHAR`/`VARCHAR` extracts the
scalar value, so a
+stored string is returned unquoted (for example `foo`), while objects and
arrays return their JSON
Review Comment:
did you double check this behavior with other vendors? sounds incorrect to
me.
```
while objects and arrays return their JSON
```
##########
docs/content.zh/docs/sql/reference/data-types.md:
##########
@@ -1507,19 +1507,62 @@ close to the semantics of JSON. Compared to `ROW` and
`STRUCTURED` type, `VARIAN
flexibility to support highly nested and evolving schema.
`VARIANT` allows for deeply nested data structures, such as arrays within
arrays, maps within maps,
-or combinations of both.This capability makes `VARIANT` ideal for scenarios
where data complexity
+or combinations of both. This capability makes `VARIANT` ideal for scenarios
where data complexity
and nesting are significant.
`VARIANT` allows schema evolution, enabling the storage of data with changing
or unknown schemas
without requiring upfront schema definition. For example, if a new field is
added to the data, it
can be directly incorporated into the `VARIANT` data without modifying the
table schema. This is
particularly useful in dynamic environments where schemas may evolve over time.
+A `VARIANT` stores a single value of one of the following kinds: `NULL`,
`BOOLEAN`, `TINYINT`,
+`SMALLINT`, `INT`, `BIGINT`, `FLOAT`, `DOUBLE`, `DECIMAL` (up to precision
38), `STRING`, `DATE`,
+`TIMESTAMP`, `TIMESTAMP_LTZ`, `BYTES`, or a nested array or object.
`TIMESTAMP` and `TIMESTAMP_LTZ`
+are stored with microsecond precision and `DATE` as a day count. There is no
`TIME` kind.
+
+The `PARSE_JSON` function produces only the kinds that JSON syntax can express:
+
+| JSON input | Stored `VARIANT` kind
|
+|-----------------------------------------------------|-------------------------------------------------|
+| `null` | `NULL`
|
+| `true` / `false` | `BOOLEAN`
|
+| Integer within the 64-bit signed range | smallest of
`TINYINT`/`SMALLINT`/`INT`/`BIGINT` |
+| Integer beyond 64-bit, up to 38 significant digits | `DECIMAL`
|
+| Decimal in plain notation, up to precision/scale 38 | `DECIMAL`
|
+| Number in scientific notation, e.g. `1.5e3` | `DOUBLE`
|
+| Number exceeding 38 digits of precision or scale | `DOUBLE`
|
+| String | `STRING`
|
+| Array | array (elements
encoded by the same rules) |
+| Object | object (values encoded
by the same rules) |
+
+Because JSON has no literal for float, date, timestamp, or binary values,
`PARSE_JSON` never
+produces `FLOAT`, `DATE`, `TIMESTAMP`, `TIMESTAMP_LTZ`, or `BYTES`. A
`VARIANT` can still hold
+those kinds when built by other producers, such as the programmatic
`VariantBuilder` API or format
+conversions like Avro.
+
+The JSON specification has no `NaN` or infinity literals, so
`PARSE_JSON('NaN')`,
+`PARSE_JSON('Infinity')`, and `PARSE_JSON('-Infinity')` fail, and
`TRY_PARSE_JSON` returns `NULL`.
+A `VARIANT` has no dedicated kind for these values. To keep one, store it as a
JSON string and cast
+it back out, for example `CAST(CAST(PARSE_JSON('"Infinity"') AS STRING) AS
FLOAT)`.
+
A primitive-valued `VARIANT` can be converted to a scalar type with `CAST` or
`TRY_CAST`. Numeric
-targets are lenient: a variant holding any numeric value casts to any numeric
type, so a JSON integer
-such as `PARSE_JSON('42')` casts to `INT` or `BIGINT`. Other targets require
the stored value to be of
-the matching kind. When the value cannot be converted, `CAST` fails the job
and `TRY_CAST` returns
-`NULL`. Use the `JSON_STRING` function to obtain the JSON string
representation of a `VARIANT`.
+targets accept any numeric value, so `PARSE_JSON('42')` casts to `INT`,
`BIGINT`, or `DOUBLE`. A
+value outside an integer or `DECIMAL` target's range fails `CAST` and returns
`NULL` for
+`TRY_CAST`. Other targets require the stored value to be of the matching kind,
and a mismatch is
+handled the same way. Casting a `VARIANT` to `CHAR`/`VARCHAR` extracts the
scalar value, so a
+stored string is returned unquoted (for example `foo`), while objects and
arrays return their JSON
+representation. Use `JSON_STRING` for the JSON representation, where a string
stays quoted (for
+example `"foo"`). Casting to `TIME` is not supported.
+
+{{< hint info >}}
+Unlike a regular numeric cast, casting a numeric `VARIANT` never wraps on
overflow. To narrow a
Review Comment:
```suggestion
Unlike a regular numeric cast, casting a numeric `VARIANT` never overflows.
To narrow a
```
##########
docs/content.zh/docs/sql/reference/data-types.md:
##########
@@ -1507,19 +1507,62 @@ close to the semantics of JSON. Compared to `ROW` and
`STRUCTURED` type, `VARIAN
flexibility to support highly nested and evolving schema.
`VARIANT` allows for deeply nested data structures, such as arrays within
arrays, maps within maps,
-or combinations of both.This capability makes `VARIANT` ideal for scenarios
where data complexity
+or combinations of both. This capability makes `VARIANT` ideal for scenarios
where data complexity
and nesting are significant.
`VARIANT` allows schema evolution, enabling the storage of data with changing
or unknown schemas
without requiring upfront schema definition. For example, if a new field is
added to the data, it
can be directly incorporated into the `VARIANT` data without modifying the
table schema. This is
particularly useful in dynamic environments where schemas may evolve over time.
+A `VARIANT` stores a single value of one of the following kinds: `NULL`,
`BOOLEAN`, `TINYINT`,
+`SMALLINT`, `INT`, `BIGINT`, `FLOAT`, `DOUBLE`, `DECIMAL` (up to precision
38), `STRING`, `DATE`,
+`TIMESTAMP`, `TIMESTAMP_LTZ`, `BYTES`, or a nested array or object.
`TIMESTAMP` and `TIMESTAMP_LTZ`
+are stored with microsecond precision and `DATE` as a day count. There is no
`TIME` kind.
+
+The `PARSE_JSON` function produces only the kinds that JSON syntax can express:
+
+| JSON input | Stored `VARIANT` kind
|
+|-----------------------------------------------------|-------------------------------------------------|
+| `null` | `NULL`
|
+| `true` / `false` | `BOOLEAN`
|
+| Integer within the 64-bit signed range | smallest of
`TINYINT`/`SMALLINT`/`INT`/`BIGINT` |
+| Integer beyond 64-bit, up to 38 significant digits | `DECIMAL`
|
+| Decimal in plain notation, up to precision/scale 38 | `DECIMAL`
|
+| Number in scientific notation, e.g. `1.5e3` | `DOUBLE`
|
+| Number exceeding 38 digits of precision or scale | `DOUBLE`
|
+| String | `STRING`
|
+| Array | array (elements
encoded by the same rules) |
+| Object | object (values encoded
by the same rules) |
+
+Because JSON has no literal for float, date, timestamp, or binary values,
`PARSE_JSON` never
+produces `FLOAT`, `DATE`, `TIMESTAMP`, `TIMESTAMP_LTZ`, or `BYTES`. A
`VARIANT` can still hold
+those kinds when built by other producers, such as the programmatic
`VariantBuilder` API or format
+conversions like Avro.
+
+The JSON specification has no `NaN` or infinity literals, so
`PARSE_JSON('NaN')`,
+`PARSE_JSON('Infinity')`, and `PARSE_JSON('-Infinity')` fail, and
`TRY_PARSE_JSON` returns `NULL`.
+A `VARIANT` has no dedicated kind for these values. To keep one, store it as a
JSON string and cast
+it back out, for example `CAST(CAST(PARSE_JSON('"Infinity"') AS STRING) AS
FLOAT)`.
+
A primitive-valued `VARIANT` can be converted to a scalar type with `CAST` or
`TRY_CAST`. Numeric
-targets are lenient: a variant holding any numeric value casts to any numeric
type, so a JSON integer
-such as `PARSE_JSON('42')` casts to `INT` or `BIGINT`. Other targets require
the stored value to be of
-the matching kind. When the value cannot be converted, `CAST` fails the job
and `TRY_CAST` returns
-`NULL`. Use the `JSON_STRING` function to obtain the JSON string
representation of a `VARIANT`.
+targets accept any numeric value, so `PARSE_JSON('42')` casts to `INT`,
`BIGINT`, or `DOUBLE`. A
+value outside an integer or `DECIMAL` target's range fails `CAST` and returns
`NULL` for
+`TRY_CAST`. Other targets require the stored value to be of the matching kind,
and a mismatch is
+handled the same way. Casting a `VARIANT` to `CHAR`/`VARCHAR` extracts the
scalar value, so a
+stored string is returned unquoted (for example `foo`), while objects and
arrays return their JSON
+representation. Use `JSON_STRING` for the JSON representation, where a string
stays quoted (for
+example `"foo"`). Casting to `TIME` is not supported.
Review Comment:
incomplete list anyway.
```suggestion
example `"foo"`).
```
##########
docs/content.zh/docs/sql/reference/data-types.md:
##########
@@ -1507,19 +1507,62 @@ close to the semantics of JSON. Compared to `ROW` and
`STRUCTURED` type, `VARIAN
flexibility to support highly nested and evolving schema.
`VARIANT` allows for deeply nested data structures, such as arrays within
arrays, maps within maps,
-or combinations of both.This capability makes `VARIANT` ideal for scenarios
where data complexity
+or combinations of both. This capability makes `VARIANT` ideal for scenarios
where data complexity
and nesting are significant.
`VARIANT` allows schema evolution, enabling the storage of data with changing
or unknown schemas
without requiring upfront schema definition. For example, if a new field is
added to the data, it
can be directly incorporated into the `VARIANT` data without modifying the
table schema. This is
particularly useful in dynamic environments where schemas may evolve over time.
+A `VARIANT` stores a single value of one of the following kinds: `NULL`,
`BOOLEAN`, `TINYINT`,
+`SMALLINT`, `INT`, `BIGINT`, `FLOAT`, `DOUBLE`, `DECIMAL` (up to precision
38), `STRING`, `DATE`,
+`TIMESTAMP`, `TIMESTAMP_LTZ`, `BYTES`, or a nested array or object.
`TIMESTAMP` and `TIMESTAMP_LTZ`
+are stored with microsecond precision and `DATE` as a day count. There is no
`TIME` kind.
+
+The `PARSE_JSON` function produces only the kinds that JSON syntax can express:
+
+| JSON input | Stored `VARIANT` kind
|
+|-----------------------------------------------------|-------------------------------------------------|
+| `null` | `NULL`
|
+| `true` / `false` | `BOOLEAN`
|
+| Integer within the 64-bit signed range | smallest of
`TINYINT`/`SMALLINT`/`INT`/`BIGINT` |
+| Integer beyond 64-bit, up to 38 significant digits | `DECIMAL`
|
+| Decimal in plain notation, up to precision/scale 38 | `DECIMAL`
|
+| Number in scientific notation, e.g. `1.5e3` | `DOUBLE`
|
+| Number exceeding 38 digits of precision or scale | `DOUBLE`
|
+| String | `STRING`
|
+| Array | array (elements
encoded by the same rules) |
+| Object | object (values encoded
by the same rules) |
+
+Because JSON has no literal for float, date, timestamp, or binary values,
`PARSE_JSON` never
+produces `FLOAT`, `DATE`, `TIMESTAMP`, `TIMESTAMP_LTZ`, or `BYTES`. A
`VARIANT` can still hold
+those kinds when built by other producers, such as the programmatic
`VariantBuilder` API or format
+conversions like Avro.
+
+The JSON specification has no `NaN` or infinity literals, so
`PARSE_JSON('NaN')`,
+`PARSE_JSON('Infinity')`, and `PARSE_JSON('-Infinity')` fail, and
`TRY_PARSE_JSON` returns `NULL`.
+A `VARIANT` has no dedicated kind for these values. To keep one, store it as a
JSON string and cast
+it back out, for example `CAST(CAST(PARSE_JSON('"Infinity"') AS STRING) AS
FLOAT)`.
+
A primitive-valued `VARIANT` can be converted to a scalar type with `CAST` or
`TRY_CAST`. Numeric
-targets are lenient: a variant holding any numeric value casts to any numeric
type, so a JSON integer
-such as `PARSE_JSON('42')` casts to `INT` or `BIGINT`. Other targets require
the stored value to be of
-the matching kind. When the value cannot be converted, `CAST` fails the job
and `TRY_CAST` returns
-`NULL`. Use the `JSON_STRING` function to obtain the JSON string
representation of a `VARIANT`.
+targets accept any numeric value, so `PARSE_JSON('42')` casts to `INT`,
`BIGINT`, or `DOUBLE`. A
+value outside an integer or `DECIMAL` target's range fails `CAST` and returns
`NULL` for
+`TRY_CAST`. Other targets require the stored value to be of the matching kind,
and a mismatch is
+handled the same way. Casting a `VARIANT` to `CHAR`/`VARCHAR` extracts the
scalar value, so a
+stored string is returned unquoted (for example `foo`), while objects and
arrays return their JSON
+representation. Use `JSON_STRING` for the JSON representation, where a string
stays quoted (for
+example `"foo"`). Casting to `TIME` is not supported.
+
+{{< hint info >}}
+Unlike a regular numeric cast, casting a numeric `VARIANT` never wraps on
overflow. To narrow a
+`VARIANT` with wrap-around semantics, first cast it to a type that fits the
stored value, then apply a
Review Comment:
```suggestion
`VARIANT`, first cast it to a type that fits the stored value, then apply a
```
##########
docs/content.zh/docs/sql/reference/data-types.md:
##########
@@ -1507,19 +1507,62 @@ close to the semantics of JSON. Compared to `ROW` and
`STRUCTURED` type, `VARIAN
flexibility to support highly nested and evolving schema.
`VARIANT` allows for deeply nested data structures, such as arrays within
arrays, maps within maps,
-or combinations of both.This capability makes `VARIANT` ideal for scenarios
where data complexity
+or combinations of both. This capability makes `VARIANT` ideal for scenarios
where data complexity
and nesting are significant.
`VARIANT` allows schema evolution, enabling the storage of data with changing
or unknown schemas
without requiring upfront schema definition. For example, if a new field is
added to the data, it
can be directly incorporated into the `VARIANT` data without modifying the
table schema. This is
particularly useful in dynamic environments where schemas may evolve over time.
+A `VARIANT` stores a single value of one of the following kinds: `NULL`,
`BOOLEAN`, `TINYINT`,
+`SMALLINT`, `INT`, `BIGINT`, `FLOAT`, `DOUBLE`, `DECIMAL` (up to precision
38), `STRING`, `DATE`,
+`TIMESTAMP`, `TIMESTAMP_LTZ`, `BYTES`, or a nested array or object.
`TIMESTAMP` and `TIMESTAMP_LTZ`
+are stored with microsecond precision and `DATE` as a day count. There is no
`TIME` kind.
+
+The `PARSE_JSON` function produces only the kinds that JSON syntax can express:
+
+| JSON input | Stored `VARIANT` kind
|
+|-----------------------------------------------------|-------------------------------------------------|
+| `null` | `NULL`
|
+| `true` / `false` | `BOOLEAN`
|
+| Integer within the 64-bit signed range | smallest of
`TINYINT`/`SMALLINT`/`INT`/`BIGINT` |
+| Integer beyond 64-bit, up to 38 significant digits | `DECIMAL`
|
+| Decimal in plain notation, up to precision/scale 38 | `DECIMAL`
|
+| Number in scientific notation, e.g. `1.5e3` | `DOUBLE`
|
+| Number exceeding 38 digits of precision or scale | `DOUBLE`
|
+| String | `STRING`
|
+| Array | array (elements
encoded by the same rules) |
+| Object | object (values encoded
by the same rules) |
+
+Because JSON has no literal for float, date, timestamp, or binary values,
`PARSE_JSON` never
+produces `FLOAT`, `DATE`, `TIMESTAMP`, `TIMESTAMP_LTZ`, or `BYTES`. A
`VARIANT` can still hold
+those kinds when built by other producers, such as the programmatic
`VariantBuilder` API or format
+conversions like Avro.
+
+The JSON specification has no `NaN` or infinity literals, so
`PARSE_JSON('NaN')`,
+`PARSE_JSON('Infinity')`, and `PARSE_JSON('-Infinity')` fail, and
`TRY_PARSE_JSON` returns `NULL`.
+A `VARIANT` has no dedicated kind for these values. To keep one, store it as a
JSON string and cast
+it back out, for example `CAST(CAST(PARSE_JSON('"Infinity"') AS STRING) AS
FLOAT)`.
+
A primitive-valued `VARIANT` can be converted to a scalar type with `CAST` or
`TRY_CAST`. Numeric
-targets are lenient: a variant holding any numeric value casts to any numeric
type, so a JSON integer
-such as `PARSE_JSON('42')` casts to `INT` or `BIGINT`. Other targets require
the stored value to be of
-the matching kind. When the value cannot be converted, `CAST` fails the job
and `TRY_CAST` returns
-`NULL`. Use the `JSON_STRING` function to obtain the JSON string
representation of a `VARIANT`.
+targets accept any numeric value, so `PARSE_JSON('42')` casts to `INT`,
`BIGINT`, or `DOUBLE`. A
+value outside an integer or `DECIMAL` target's range fails `CAST` and returns
`NULL` for
+`TRY_CAST`. Other targets require the stored value to be of the matching kind,
and a mismatch is
+handled the same way. Casting a `VARIANT` to `CHAR`/`VARCHAR` extracts the
scalar value, so a
+stored string is returned unquoted (for example `foo`), while objects and
arrays return their JSON
+representation. Use `JSON_STRING` for the JSON representation, where a string
stays quoted (for
+example `"foo"`). Casting to `TIME` is not supported.
+
+{{< hint info >}}
+Unlike a regular numeric cast, casting a numeric `VARIANT` never wraps on
overflow. To narrow a
+`VARIANT` with wrap-around semantics, first cast it to a type that fits the
stored value, then apply a
+regular narrowing cast:
+
+```sql
+CAST(CAST(PARSE_JSON('1000') AS INT) AS TINYINT) -- returns -24 (wrap-around)
Review Comment:
```suggestion
CAST(CAST(PARSE_JSON('1000') AS INT) AS TINYINT) -- returns -24 (after
overflow)
```
##########
flink-table/flink-table-common/src/test/java/org/apache/flink/table/types/LogicalTypeCastsTest.java:
##########
@@ -284,11 +285,12 @@ private static Stream<Arguments> testData() {
new VarBinaryType(VarBinaryType.MAX_LENGTH),
false,
true),
+ Arguments.of(new VariantType(), new CharType(), false, true),
+ Arguments.of(new VariantType(), VarCharType.STRING_TYPE,
false, true),
// variant identity cast is implicit
Arguments.of(new VariantType(), new VariantType(), true, true),
// TIME, character strings and constructed targets are not
castable from variant
Review Comment:
```suggestion
// TIME and constructed targets are not castable from variant
```
##########
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/functions/casting/VariantToPrimitiveCastRule.java:
##########
@@ -166,38 +164,35 @@ protected String generateCodeBlockInternal(
}
/**
- * Converts a numeric variant to the numeric {@code target} via the
matching {@link Number}
- * accessor, mirroring regular numeric cast semantics. A non-numeric
variant raises {@link
- * ClassCastException}, failing {@code CAST} and yielding {@code null} for
{@code TRY_CAST}.
+ * Converts a numeric variant to the numeric {@code target}. Integer and
decimal targets are
+ * range-checked so that an out-of-range value fails {@code CAST} and
yields {@code null} for
+ * {@code TRY_CAST}, following Spark's variant cast semantics; {@code
FLOAT} and {@code DOUBLE}
Review Comment:
not sure if we follow 100%, better remove
```suggestion
* {@code TRY_CAST}; {@code FLOAT} and {@code DOUBLE}
```
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/VariantCastUtils.java:
##########
@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.functions;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.data.DecimalData;
+import org.apache.flink.types.variant.Variant;
+
+import java.math.BigDecimal;
+import java.math.BigInteger;
+
+/**
+ * Runtime helpers for casting a {@code VARIANT} value to a SQL type.
+ *
+ * <p>Numeric targets are range-checked and raise {@link ArithmeticException}
on overflow instead of
+ * wrapping. Casting to a string extracts the scalar value (a string stays
unquoted), while objects
+ * and arrays use their JSON representation.
+ */
+@Internal
+public final class VariantCastUtils {
+
+ private VariantCastUtils() {}
+
+ public static byte toByteExact(Number value) {
+ return (byte) toIntegralExact(value, Byte.MIN_VALUE, Byte.MAX_VALUE,
"TINYINT");
+ }
+
+ public static short toShortExact(Number value) {
+ return (short) toIntegralExact(value, Short.MIN_VALUE,
Short.MAX_VALUE, "SMALLINT");
+ }
+
+ public static int toIntExact(Number value) {
+ return (int) toIntegralExact(value, Integer.MIN_VALUE,
Integer.MAX_VALUE, "INTEGER");
+ }
+
+ public static long toLongExact(Number value) {
+ return toIntegralExact(value, Long.MIN_VALUE, Long.MAX_VALUE,
"BIGINT");
+ }
+
+ public static DecimalData toDecimalExact(Number value, int precision, int
scale) {
+ final DecimalData decimal =
+ DecimalData.fromBigDecimal(toBigDecimal(value), precision,
scale);
+ if (decimal == null) {
+ throw new ArithmeticException(
Review Comment:
is it guaranteed that this becomes a `TableRuntimeException` eventually? or
how do we treat other cast cases currently?
##########
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/functions/casting/VariantToStringCastRule.java:
##########
@@ -45,6 +51,6 @@ public String generateStringExpression(
String inputTerm,
LogicalType inputLogicalType,
LogicalType targetLogicalType) {
- return methodCall(inputTerm, "toString");
+ return staticCall(VariantCastUtils.class, "toStringValue", inputTerm);
Review Comment:
according to LogicalTypeCasts, we support VARCHAR and CHAR both with various
length. this cast rule should consider the target length as well. I guess it
should throw if length exceeds?
##########
flink-core/src/main/java/org/apache/flink/types/variant/Variant.java:
##########
@@ -127,18 +127,19 @@ public interface Variant {
/**
* Get the scalar value of variant as LocalDateTime, if the variant type
is {@link
- * Type#TIMESTAMP}.
+ * Type#TIMESTAMP}. The returned value has microsecond precision.
*
* @throws VariantTypeException If this variant is not a scalar value or
is not {@link
* Type#TIMESTAMP}.
*/
LocalDateTime getDateTime() throws VariantTypeException;
/**
- * Get the scalar value of variant as Instant, if the variant type is
{@link Type#TIMESTAMP}.
+ * Get the scalar value of variant as Instant, if the variant type is
{@link
Review Comment:
can you apply to all JavaDocs in this class?
```suggestion
* Get the scalar value of variant as {@link Instant}, if the variant
type is {@link
```
##########
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/functions/CastFunctionITCase.java:
##########
@@ -151,35 +151,47 @@ private static List<TestSetSpec> variantCasts() {
"CAST(PARSE_JSON('42') AS INT)",
42,
INT().notNull())
- // Integer overflow wraps around (Java narrowing),
like a regular numeric
- // cast.
+ // Spark-style overflow: an integer value outside the
target range fails
+ // CAST and returns NULL for TRY_CAST, instead of
wrapping around.
+ .testTableApiRuntimeError(
+ call("PARSE_JSON", "40000").cast(SMALLINT()),
"overflowed")
+ .testSqlRuntimeError("CAST(PARSE_JSON('40000') AS
SMALLINT)", "overflowed")
.testResult(
- call("PARSE_JSON", "40000").cast(SMALLINT()),
- "CAST(PARSE_JSON('40000') AS SMALLINT)",
- (short) -25536,
- SMALLINT().notNull())
- .testResult(
- call("PARSE_JSON", "128").cast(TINYINT()),
- "CAST(PARSE_JSON('128') AS TINYINT)",
- (byte) -128,
- TINYINT().notNull())
+ call("PARSE_JSON",
"40000").tryCast(SMALLINT()),
+ "TRY_CAST(PARSE_JSON('40000') AS SMALLINT)",
+ null,
+ SMALLINT())
+ .testTableApiRuntimeError(
+ call("PARSE_JSON", "128").cast(TINYINT()),
"overflowed")
Review Comment:
to conclude on this: whereas for regular casts people opt-in on the
overflow, for variant casts they should first extract the correct value before
opt-in to overflow.
##########
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/functions/CastFunctionITCase.java:
##########
@@ -151,35 +151,47 @@ private static List<TestSetSpec> variantCasts() {
"CAST(PARSE_JSON('42') AS INT)",
42,
INT().notNull())
- // Integer overflow wraps around (Java narrowing),
like a regular numeric
- // cast.
+ // Spark-style overflow: an integer value outside the
target range fails
Review Comment:
```suggestion
// An integer value outside the target range fails
```
##########
docs/content.zh/docs/sql/reference/data-types.md:
##########
@@ -1507,19 +1507,62 @@ close to the semantics of JSON. Compared to `ROW` and
`STRUCTURED` type, `VARIAN
flexibility to support highly nested and evolving schema.
`VARIANT` allows for deeply nested data structures, such as arrays within
arrays, maps within maps,
-or combinations of both.This capability makes `VARIANT` ideal for scenarios
where data complexity
+or combinations of both. This capability makes `VARIANT` ideal for scenarios
where data complexity
and nesting are significant.
`VARIANT` allows schema evolution, enabling the storage of data with changing
or unknown schemas
without requiring upfront schema definition. For example, if a new field is
added to the data, it
can be directly incorporated into the `VARIANT` data without modifying the
table schema. This is
particularly useful in dynamic environments where schemas may evolve over time.
+A `VARIANT` stores a single value of one of the following kinds: `NULL`,
`BOOLEAN`, `TINYINT`,
+`SMALLINT`, `INT`, `BIGINT`, `FLOAT`, `DOUBLE`, `DECIMAL` (up to precision
38), `STRING`, `DATE`,
+`TIMESTAMP`, `TIMESTAMP_LTZ`, `BYTES`, or a nested array or object.
`TIMESTAMP` and `TIMESTAMP_LTZ`
+are stored with microsecond precision and `DATE` as a day count. There is no
`TIME` kind.
+
+The `PARSE_JSON` function produces only the kinds that JSON syntax can express:
+
+| JSON input | Stored `VARIANT` kind
|
+|-----------------------------------------------------|-------------------------------------------------|
+| `null` | `NULL`
|
+| `true` / `false` | `BOOLEAN`
|
+| Integer within the 64-bit signed range | smallest of
`TINYINT`/`SMALLINT`/`INT`/`BIGINT` |
+| Integer beyond 64-bit, up to 38 significant digits | `DECIMAL`
|
+| Decimal in plain notation, up to precision/scale 38 | `DECIMAL`
|
+| Number in scientific notation, e.g. `1.5e3` | `DOUBLE`
|
+| Number exceeding 38 digits of precision or scale | `DOUBLE`
|
+| String | `STRING`
|
+| Array | array (elements
encoded by the same rules) |
+| Object | object (values encoded
by the same rules) |
+
+Because JSON has no literal for float, date, timestamp, or binary values,
`PARSE_JSON` never
+produces `FLOAT`, `DATE`, `TIMESTAMP`, `TIMESTAMP_LTZ`, or `BYTES`. A
`VARIANT` can still hold
+those kinds when built by other producers, such as the programmatic
`VariantBuilder` API or format
+conversions like Avro.
+
+The JSON specification has no `NaN` or infinity literals, so
`PARSE_JSON('NaN')`,
+`PARSE_JSON('Infinity')`, and `PARSE_JSON('-Infinity')` fail, and
`TRY_PARSE_JSON` returns `NULL`.
+A `VARIANT` has no dedicated kind for these values. To keep one, store it as a
JSON string and cast
+it back out, for example `CAST(CAST(PARSE_JSON('"Infinity"') AS STRING) AS
FLOAT)`.
+
A primitive-valued `VARIANT` can be converted to a scalar type with `CAST` or
`TRY_CAST`. Numeric
-targets are lenient: a variant holding any numeric value casts to any numeric
type, so a JSON integer
-such as `PARSE_JSON('42')` casts to `INT` or `BIGINT`. Other targets require
the stored value to be of
-the matching kind. When the value cannot be converted, `CAST` fails the job
and `TRY_CAST` returns
-`NULL`. Use the `JSON_STRING` function to obtain the JSON string
representation of a `VARIANT`.
+targets accept any numeric value, so `PARSE_JSON('42')` casts to `INT`,
`BIGINT`, or `DOUBLE`. A
+value outside an integer or `DECIMAL` target's range fails `CAST` and returns
`NULL` for
+`TRY_CAST`. Other targets require the stored value to be of the matching kind,
and a mismatch is
+handled the same way. Casting a `VARIANT` to `CHAR`/`VARCHAR` extracts the
scalar value, so a
+stored string is returned unquoted (for example `foo`), while objects and
arrays return their JSON
Review Comment:
this is what JSON_STRING should be good for.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
