This is an automated email from the ASF dual-hosted git repository.
snuyanzin pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/flink.git
The following commit(s) were added to refs/heads/master by this push:
new 2c629d77610 [FLINK-37925][table] Define `CAST` and `TRY_CAST` from
`VARIANT` to scalar types
2c629d77610 is described below
commit 2c629d776107f07d76968d2518645f93714a600b
Author: Ramin Gharib <[email protected]>
AuthorDate: Mon Aug 24 15:32:32 2026 +0200
[FLINK-37925][table] Define `CAST` and `TRY_CAST` from `VARIANT` to scalar
types
---
docs/content.zh/docs/sql/reference/data-types.md | 81 ++++-
docs/content/docs/sql/reference/data-types.md | 81 ++++-
docs/data/sql_functions.yml | 4 +
docs/data/sql_functions_zh.yml | 4 +
.../org/apache/flink/types/variant/Variant.java | 15 +-
.../strategies/CastInputTypeStrategy.java | 10 -
.../types/logical/utils/LogicalTypeCasts.java | 97 +++---
.../flink/table/types/LogicalTypeCastsTest.java | 6 +-
.../casting/VariantToPrimitiveCastRule.java | 150 ++++-----
.../functions/casting/VariantToStringCastRule.java | 75 ++++-
.../planner/functions/CastFunctionITCase.java | 243 +++++++++++---
.../planner/functions/casting/CastRulesTest.java | 255 +++++++++++++-
.../table/runtime/functions/VariantCastUtils.java | 369 +++++++++++++++++++++
13 files changed, 1154 insertions(+), 236 deletions(-)
diff --git a/docs/content.zh/docs/sql/reference/data-types.md
b/docs/content.zh/docs/sql/reference/data-types.md
index 6ad9cd02c7a..62266a004eb 100644
--- a/docs/content.zh/docs/sql/reference/data-types.md
+++ b/docs/content.zh/docs/sql/reference/data-types.md
@@ -1086,7 +1086,7 @@ The type can be declared using the above combinations
where `p1` is the number o
and `9` (both inclusive). If no `p1` is specified, it is equal to `2` by
default. If no `p2` is
specified, it is equal to `6` by default.
-### Constructured Data Types
+### Constructed Data Types
#### `ARRAY`
@@ -1507,7 +1507,7 @@ close to the semantics of JSON. Compared to `ROW` and
`STRUCTURED` type, `VARIAN
flexibility to support highly nested and evolving schema.
`VARIANT` allows for deeply nested data structures, such as arrays within
arrays, maps within maps,
-or combinations of both.This capability makes `VARIANT` ideal for scenarios
where data complexity
+or combinations of both. This capability makes `VARIANT` ideal for scenarios
where data complexity
and nesting are significant.
`VARIANT` allows schema evolution, enabling the storage of data with changing
or unknown schemas
@@ -1515,11 +1515,78 @@ without requiring upfront schema definition. For
example, if a new field is adde
can be directly incorporated into the `VARIANT` data without modifying the
table schema. This is
particularly useful in dynamic environments where schemas may evolve over time.
-A primitive-valued `VARIANT` can be converted to a scalar type with `CAST` or
`TRY_CAST`. Numeric
-targets are lenient: a variant holding any numeric value casts to any numeric
type, so a JSON integer
-such as `PARSE_JSON('42')` casts to `INT` or `BIGINT`. Other targets require
the stored value to be of
-the matching kind. When the value cannot be converted, `CAST` fails the job
and `TRY_CAST` returns
-`NULL`. Use the `JSON_STRING` function to obtain the JSON string
representation of a `VARIANT`.
+A `VARIANT` stores a single value of one of the following kinds: `NULL`,
`BOOLEAN`, `TINYINT`,
+`SMALLINT`, `INT`, `BIGINT`, `FLOAT`, `DOUBLE`, `DECIMAL` (up to precision
38), `STRING`, `DATE`,
+`TIMESTAMP`, `TIMESTAMP_LTZ`, `BYTES`, or a nested array or object.
`TIMESTAMP` and `TIMESTAMP_LTZ`
+are stored with microsecond precision.
+
+The `PARSE_JSON` function produces only the kinds that JSON syntax can express:
+
+| JSON input | Stored `VARIANT` kind
|
+|-----------------------------------------------------|-------------------------------------------------|
+| `null` | `NULL`
|
+| `true` / `false` | `BOOLEAN`
|
+| Integer within the 64-bit signed range | smallest of
`TINYINT`/`SMALLINT`/`INT`/`BIGINT` |
+| Integer beyond 64-bit, up to 38 significant digits | `DECIMAL`
|
+| Decimal in plain notation, up to precision/scale 38 | `DECIMAL`
|
+| Number in scientific notation, e.g. `1.5e3` | `DOUBLE`
|
+| Number exceeding 38 digits of precision or scale | `DOUBLE`
|
+| String | `STRING`
|
+| Array | array (elements
encoded by the same rules) |
+| Object | object (values encoded
by the same rules) |
+
+Because JSON has no literal for them, `PARSE_JSON` never produces `FLOAT`,
`DATE`, `TIMESTAMP`,
+`TIMESTAMP_LTZ`, or `BYTES`.
+
+The JSON specification has no `NaN` or infinity literals, so
`PARSE_JSON('NaN')`,
+`PARSE_JSON('Infinity')`, and `PARSE_JSON('-Infinity')` fail.
`PARSE_JSON('1e400')` fails as well
+because a value outside the `DOUBLE` range cannot be stored as a finite
number. In all of these
+cases `TRY_PARSE_JSON` returns `NULL`.
+A `VARIANT` has no dedicated kind for these values. To keep one, store it as a
JSON string and cast
+it back out, for example `CAST(CAST(PARSE_JSON('"Infinity"') AS STRING) AS
FLOAT)`.
+
+A `VARIANT` can be converted to a scalar type with `CAST` or `TRY_CAST`. A
cast succeeds only when
+the target holds the stored value without reinterpreting it, so a value is
never wrapped or rounded
+to make it fit. Otherwise `CAST` fails and `TRY_CAST` returns `NULL`.
+
+| Stored kind | Succeeds for |
+|-----------------|-----------------------------------------------------|
+| numeric kinds | any numeric target that holds the value |
+| `BOOLEAN` | `BOOLEAN` |
+| `DATE` | `DATE` |
+| `TIMESTAMP` | `TIMESTAMP(p)` |
+| `TIMESTAMP_LTZ` | `TIMESTAMP_LTZ(p)` |
+| `BYTES` | `BINARY(n)`, `VARBINARY(n)`, and a character string |
+| any scalar | `STRING`, `CHAR(n)`, `VARCHAR(n)` |
+| `NULL` | SQL `NULL` for any nullable target |
+
+The conditions above mean:
+
+- An **integer target** needs the value in range and without a fractional
part, so
+ `PARSE_JSON('7.0')` reaches `INT` as `7` while `PARSE_JSON('7.2')` does not,
and
+ `CAST(PARSE_JSON('1000') AS TINYINT)` fails instead of wrapping.
+- A **`DECIMAL`** target has to fit the precision and the scale. Trailing
zeros may be appended, so
+ `42` reaches `DECIMAL(5, 2)` as `42.00`, but a scale that would have to
round is rejected.
+- **`FLOAT`** and **`DOUBLE`** are approximate by definition, so they take any
numeric kind and drop
+ decimal digits, rejecting only a magnitude out of range such as `1e40` to a
`FLOAT`.
+- A **length or precision** is adjusted the same way a regular cast into that
type would: a value
+ longer than the target is trimmed, fractional seconds beyond the target
precision are truncated,
+ and the fixed width types `CHAR(n)` and `BINARY(n)` pad a shorter value.
+
+To reach a type the table does not list, wrap the cast in a regular cast. Only
the inner cast is a
+`VARIANT` cast, so the outer one applies the usual rules and may round,
truncate, or overflow:
+
+```sql
+CAST(CAST(PARSE_JSON('1000') AS SMALLINT) AS TINYINT) -- returns -24
(after overflow)
+CAST(CAST(PARSE_JSON('3.9') AS DECIMAL(2, 1)) AS INT) -- returns 3
(truncated)
+```
+
+A cast to a character string renders the value exactly as a regular SQL cast
of the stored kind
+would, so a boolean becomes `TRUE`, a timestamp uses the SQL format, a
`TIMESTAMP_LTZ` is shifted into
+the session time zone, and a binary value is read as UTF-8. Only an object or
an array has no such
+rendering. Use `JSON_STRING` for the JSON representation instead, where a
string stays quoted as
+`"foo"` and an object or array is serialized. A variant that stores a JSON
`null` casts to SQL
+`NULL`.
**Declaration**
diff --git a/docs/content/docs/sql/reference/data-types.md
b/docs/content/docs/sql/reference/data-types.md
index 45ace31e365..207c83d2792 100644
--- a/docs/content/docs/sql/reference/data-types.md
+++ b/docs/content/docs/sql/reference/data-types.md
@@ -1094,7 +1094,7 @@ The type can be declared using the above combinations
where `p1` is the number o
and `9` (both inclusive). If no `p1` is specified, it is equal to `2` by
default. If no `p2` is
specified, it is equal to `6` by default.
-### Constructured Data Types
+### Constructed Data Types
#### `ARRAY`
@@ -1515,7 +1515,7 @@ close to the semantics of JSON. Compared to `ROW` and
`STRUCTURED` type, `VARIAN
flexibility to support highly nested and evolving schema.
`VARIANT` allows for deeply nested data structures, such as arrays within
arrays, maps within maps,
-or combinations of both.This capability makes `VARIANT` ideal for scenarios
where data complexity
+or combinations of both. This capability makes `VARIANT` ideal for scenarios
where data complexity
and nesting are significant.
`VARIANT` allows schema evolution, enabling the storage of data with changing
or unknown schemas
@@ -1523,11 +1523,78 @@ without requiring upfront schema definition. For
example, if a new field is adde
can be directly incorporated into the `VARIANT` data without modifying the
table schema. This is
particularly useful in dynamic environments where schemas may evolve over time.
-A primitive-valued `VARIANT` can be converted to a scalar type with `CAST` or
`TRY_CAST`. Numeric
-targets are lenient: a variant holding any numeric value casts to any numeric
type, so a JSON integer
-such as `PARSE_JSON('42')` casts to `INT` or `BIGINT`. Other targets require
the stored value to be of
-the matching kind. When the value cannot be converted, `CAST` fails the job
and `TRY_CAST` returns
-`NULL`. Use the `JSON_STRING` function to obtain the JSON string
representation of a `VARIANT`.
+A `VARIANT` stores a single value of one of the following kinds: `NULL`,
`BOOLEAN`, `TINYINT`,
+`SMALLINT`, `INT`, `BIGINT`, `FLOAT`, `DOUBLE`, `DECIMAL` (up to precision
38), `STRING`, `DATE`,
+`TIMESTAMP`, `TIMESTAMP_LTZ`, `BYTES`, or a nested array or object.
`TIMESTAMP` and `TIMESTAMP_LTZ`
+are stored with microsecond precision.
+
+The `PARSE_JSON` function produces only the kinds that JSON syntax can express:
+
+| JSON input | Stored `VARIANT` kind
|
+|-----------------------------------------------------|-------------------------------------------------|
+| `null` | `NULL`
|
+| `true` / `false` | `BOOLEAN`
|
+| Integer within the 64-bit signed range | smallest of
`TINYINT`/`SMALLINT`/`INT`/`BIGINT` |
+| Integer beyond 64-bit, up to 38 significant digits | `DECIMAL`
|
+| Decimal in plain notation, up to precision/scale 38 | `DECIMAL`
|
+| Number in scientific notation, e.g. `1.5e3` | `DOUBLE`
|
+| Number exceeding 38 digits of precision or scale | `DOUBLE`
|
+| String | `STRING`
|
+| Array | array (elements
encoded by the same rules) |
+| Object | object (values encoded
by the same rules) |
+
+Because JSON has no literal for them, `PARSE_JSON` never produces `FLOAT`,
`DATE`, `TIMESTAMP`,
+`TIMESTAMP_LTZ`, or `BYTES`.
+
+The JSON specification has no `NaN` or infinity literals, so
`PARSE_JSON('NaN')`,
+`PARSE_JSON('Infinity')`, and `PARSE_JSON('-Infinity')` fail.
`PARSE_JSON('1e400')` fails as well
+because a value outside the `DOUBLE` range cannot be stored as a finite
number. In all of these
+cases `TRY_PARSE_JSON` returns `NULL`.
+A `VARIANT` has no dedicated kind for these values. To keep one, store it as a
JSON string and cast
+it back out, for example `CAST(CAST(PARSE_JSON('"Infinity"') AS STRING) AS
FLOAT)`.
+
+A `VARIANT` can be converted to a scalar type with `CAST` or `TRY_CAST`. A
cast succeeds only when
+the target holds the stored value without reinterpreting it, so a value is
never wrapped or rounded
+to make it fit. Otherwise `CAST` fails and `TRY_CAST` returns `NULL`.
+
+| Stored kind | Succeeds for |
+|-----------------|-----------------------------------------------------|
+| numeric kinds | any numeric target that holds the value |
+| `BOOLEAN` | `BOOLEAN` |
+| `DATE` | `DATE` |
+| `TIMESTAMP` | `TIMESTAMP(p)` |
+| `TIMESTAMP_LTZ` | `TIMESTAMP_LTZ(p)` |
+| `BYTES` | `BINARY(n)`, `VARBINARY(n)`, and a character string |
+| any scalar | `STRING`, `CHAR(n)`, `VARCHAR(n)` |
+| `NULL` | SQL `NULL` for any nullable target |
+
+The conditions above mean:
+
+- An **integer target** needs the value in range and without a fractional
part, so
+ `PARSE_JSON('7.0')` reaches `INT` as `7` while `PARSE_JSON('7.2')` does not,
and
+ `CAST(PARSE_JSON('1000') AS TINYINT)` fails instead of wrapping.
+- A **`DECIMAL`** target has to fit the precision and the scale. Trailing
zeros may be appended, so
+ `42` reaches `DECIMAL(5, 2)` as `42.00`, but a scale that would have to
round is rejected.
+- **`FLOAT`** and **`DOUBLE`** are approximate by definition, so they take any
numeric kind and drop
+ decimal digits, rejecting only a magnitude out of range such as `1e40` to a
`FLOAT`.
+- A **length or precision** is adjusted the same way a regular cast into that
type would: a value
+ longer than the target is trimmed, fractional seconds beyond the target
precision are truncated,
+ and the fixed width types `CHAR(n)` and `BINARY(n)` pad a shorter value.
+
+To reach a type the table does not list, wrap the cast in a regular cast. Only
the inner cast is a
+`VARIANT` cast, so the outer one applies the usual rules and may round,
truncate, or overflow:
+
+```sql
+CAST(CAST(PARSE_JSON('1000') AS SMALLINT) AS TINYINT) -- returns -24
(after overflow)
+CAST(CAST(PARSE_JSON('3.9') AS DECIMAL(2, 1)) AS INT) -- returns 3
(truncated)
+```
+
+A cast to a character string renders the value exactly as a regular SQL cast
of the stored kind
+would, so a boolean becomes `TRUE`, a timestamp uses the SQL format, a
`TIMESTAMP_LTZ` is shifted into
+the session time zone, and a binary value is read as UTF-8. Only an object or
an array has no such
+rendering. Use `JSON_STRING` for the JSON representation instead, where a
string stays quoted as
+`"foo"` and an object or array is serialized. A variant that stores a JSON
`null` casts to SQL
+`NULL`.
**Declaration**
diff --git a/docs/data/sql_functions.yml b/docs/data/sql_functions.yml
index 19a50dad2f0..992f77f6234 100644
--- a/docs/data/sql_functions.yml
+++ b/docs/data/sql_functions.yml
@@ -1322,6 +1322,8 @@ variant:
parser will keep the last occurrence of all fields with the same key,
otherwise when
`allowDuplicateKeys` is false it will throw an error. The default value
of
`allowDuplicateKeys` is false.
+
+ See the `VARIANT` data type documentation for how JSON values are mapped
to variant kinds.
- sql: TRY_PARSE_JSON(json_string[, allow_duplicate_keys])
description: |
Try to parse a JSON string into a Variant if possible. If the JSON
string is invalid, return
@@ -1331,6 +1333,8 @@ variant:
parser will keep the last occurrence of all fields with the same key,
otherwise when
`allowDuplicateKeys` is false it will throw an error. The default value
of
`allowDuplicateKeys` is false.
+
+ See the `VARIANT` data type documentation for how JSON values are mapped
to variant kinds.
- sql: JSON_STRING(variant)
description: |
Generate a json string from a Variant object.
diff --git a/docs/data/sql_functions_zh.yml b/docs/data/sql_functions_zh.yml
index 613c561d838..cec2237d5af 100644
--- a/docs/data/sql_functions_zh.yml
+++ b/docs/data/sql_functions_zh.yml
@@ -1408,6 +1408,8 @@ variant:
同键的字段，否则当 allowDuplicateKeys 为 false 时，它会抛出一个错误。默认情况下，
allowDuplicateKeys 的值为 false。
+ See the `VARIANT` data type documentation for how JSON values are mapped
to variant kinds.
+
- sql: TRY_PARSE_JSON(json_string[, allow_duplicate_keys])
description: |
尽可能将 JSON 字符串解析为 Variant。如果 JSON 字符串无效，则返回 NULL。如果希望抛出错误而不是返回 NULL，
@@ -1417,6 +1419,8 @@ variant:
同键的字段，否则当 allowDuplicateKeys 为 false 时，它会抛出一个错误。默认情况下，
allowDuplicateKeys 的值为 false。
+ See the `VARIANT` data type documentation for how JSON values are mapped
to variant kinds.
+
- sql: JSON_STRING(variant)
description: |
Generate a json string from a Variant object.
diff --git
a/flink-core/src/main/java/org/apache/flink/types/variant/Variant.java
b/flink-core/src/main/java/org/apache/flink/types/variant/Variant.java
index c0f15788f3f..dd7ac96b6fc 100644
--- a/flink-core/src/main/java/org/apache/flink/types/variant/Variant.java
+++ b/flink-core/src/main/java/org/apache/flink/types/variant/Variant.java
@@ -94,7 +94,8 @@ public interface Variant {
float getFloat() throws VariantTypeException;
/**
- * Get the scalar value of variant as BigDecimal, if the variant type is
{@link Type#DECIMAL}.
+ * Get the scalar value of variant as {@link BigDecimal}, if the variant
type is {@link
+ * Type#DECIMAL}.
*
* @throws VariantTypeException If this variant is not a scalar value or
is not {@link
* Type#DECIMAL}.
@@ -118,7 +119,8 @@ public interface Variant {
String getString() throws VariantTypeException;
/**
- * Get the scalar value of variant as LocalDate, if the variant type is
{@link Type#DATE}.
+ * Get the scalar value of variant as {@link LocalDate}, if the variant
type is {@link
+ * Type#DATE}.
*
* @throws VariantTypeException If this variant is not a scalar value or
is not {@link
* Type#DATE}.
@@ -126,8 +128,8 @@ public interface Variant {
LocalDate getDate() throws VariantTypeException;
/**
- * Get the scalar value of variant as LocalDateTime, if the variant type
is {@link
- * Type#TIMESTAMP}.
+ * Get the scalar value of variant as {@link LocalDateTime}, if the
variant type is {@link
+ * Type#TIMESTAMP}. The returned value has microsecond precision.
*
* @throws VariantTypeException If this variant is not a scalar value or
is not {@link
* Type#TIMESTAMP}.
@@ -135,10 +137,11 @@ public interface Variant {
LocalDateTime getDateTime() throws VariantTypeException;
/**
- * Get the scalar value of variant as Instant, if the variant type is
{@link Type#TIMESTAMP}.
+ * Get the scalar value of variant as {@link Instant}, if the variant type
is {@link
+ * Type#TIMESTAMP_LTZ}. The returned value has microsecond precision.
*
* @throws VariantTypeException If this variant is not a scalar value or
is not {@link
- * Type#TIMESTAMP}.
+ * Type#TIMESTAMP_LTZ}.
*/
Instant getInstant() throws VariantTypeException;
diff --git
a/flink-table/flink-table-common/src/main/java/org/apache/flink/table/types/inference/strategies/CastInputTypeStrategy.java
b/flink-table/flink-table-common/src/main/java/org/apache/flink/table/types/inference/strategies/CastInputTypeStrategy.java
index 3bd66e01cd5..fbd4bf188d7 100644
---
a/flink-table/flink-table-common/src/main/java/org/apache/flink/table/types/inference/strategies/CastInputTypeStrategy.java
+++
b/flink-table/flink-table-common/src/main/java/org/apache/flink/table/types/inference/strategies/CastInputTypeStrategy.java
@@ -35,7 +35,6 @@ import java.util.Collections;
import java.util.List;
import java.util.Optional;
-import static
org.apache.flink.table.types.logical.utils.LogicalTypeCasts.getUnsupportedCastHint;
import static
org.apache.flink.table.types.logical.utils.LogicalTypeCasts.supportsExplicitCast;
/**
@@ -70,15 +69,6 @@ class CastInputTypeStrategy implements InputTypeStrategy {
return Optional.of(argumentDataTypes);
}
if (!supportsExplicitCast(fromType, toType)) {
- final Optional<String> hint = getUnsupportedCastHint(fromType,
toType);
- if (hint.isPresent()) {
- return callContext.fail(
- throwOnFailure,
- "Unsupported cast from '%s' to '%s'. %s",
- fromType,
- toType,
- hint.get());
- }
return callContext.fail(
throwOnFailure, "Unsupported cast from '%s' to '%s'.",
fromType, toType);
}
diff --git
a/flink-table/flink-table-common/src/main/java/org/apache/flink/table/types/logical/utils/LogicalTypeCasts.java
b/flink-table/flink-table-common/src/main/java/org/apache/flink/table/types/logical/utils/LogicalTypeCasts.java
index 58f474ec0eb..d22b16399c3 100644
---
a/flink-table/flink-table-common/src/main/java/org/apache/flink/table/types/logical/utils/LogicalTypeCasts.java
+++
b/flink-table/flink-table-common/src/main/java/org/apache/flink/table/types/logical/utils/LogicalTypeCasts.java
@@ -37,7 +37,6 @@ import java.util.HashMap;
import java.util.HashSet;
import java.util.List;
import java.util.Map;
-import java.util.Optional;
import java.util.Set;
import java.util.function.BiFunction;
import java.util.function.BiPredicate;
@@ -210,7 +209,7 @@ public final class LogicalTypeCasts {
castTo(CHAR)
.implicitFrom(CHAR)
.explicitFromFamily(PREDEFINED, CONSTRUCTED)
- .explicitFrom(RAW, NULL, STRUCTURED_TYPE, BITMAP)
+ .explicitFrom(RAW, NULL, STRUCTURED_TYPE, BITMAP, VARIANT)
.injectiveFrom(WHEN_LENGTH_FITS, CHAR)
.injectiveFrom(WHEN_MAX_CHAR_LENGTH_FITS,
STRING_INJECTIVE_SOURCES)
.injectiveFrom(WHEN_CHAR_LENGTH_FITS_UTF8, BINARY, VARBINARY)
@@ -219,7 +218,7 @@ public final class LogicalTypeCasts {
castTo(VARCHAR)
.implicitFromFamily(CHARACTER_STRING)
.explicitFromFamily(PREDEFINED, CONSTRUCTED)
- .explicitFrom(RAW, NULL, STRUCTURED_TYPE, BITMAP)
+ .explicitFrom(RAW, NULL, STRUCTURED_TYPE, BITMAP, VARIANT)
.injectiveFrom(WHEN_LENGTH_FITS, CHAR, VARCHAR)
.injectiveFrom(WHEN_MAX_CHAR_LENGTH_FITS,
STRING_INJECTIVE_SOURCES)
.injectiveFrom(WHEN_CHAR_LENGTH_FITS_UTF8, BINARY, VARBINARY)
@@ -232,7 +231,7 @@ public final class LogicalTypeCasts {
castTo(BINARY)
.implicitFrom(BINARY)
.explicitFromFamily(CHARACTER_STRING)
- .explicitFrom(VARBINARY, RAW)
+ .explicitFrom(VARBINARY, RAW, VARIANT)
.injectiveFrom(WHEN_LENGTH_FITS, BINARY)
.injectiveFrom(WHEN_BINARY_LENGTH_FITS_UTF8, CHAR, VARCHAR)
.build();
@@ -240,7 +239,7 @@ public final class LogicalTypeCasts {
castTo(VARBINARY)
.implicitFromFamily(BINARY_STRING)
.explicitFromFamily(CHARACTER_STRING)
- .explicitFrom(BINARY, RAW)
+ .explicitFrom(BINARY, RAW, VARIANT)
.injectiveFrom(WHEN_LENGTH_FITS, BINARY, VARBINARY)
.injectiveFrom(WHEN_BINARY_LENGTH_FITS_UTF8, CHAR, VARCHAR)
.build();
@@ -252,35 +251,55 @@ public final class LogicalTypeCasts {
castTo(TINYINT)
.implicitFrom(TINYINT)
.explicitFromFamily(NUMERIC, CHARACTER_STRING, INTERVAL)
- .explicitFrom(BOOLEAN, TIMESTAMP_WITHOUT_TIME_ZONE,
TIMESTAMP_WITH_LOCAL_TIME_ZONE)
+ .explicitFrom(
+ BOOLEAN,
+ TIMESTAMP_WITHOUT_TIME_ZONE,
+ TIMESTAMP_WITH_LOCAL_TIME_ZONE,
+ VARIANT)
.injectiveFrom(TINYINT)
.build();
castTo(SMALLINT)
.implicitFrom(TINYINT, SMALLINT)
.explicitFromFamily(NUMERIC, CHARACTER_STRING, INTERVAL)
- .explicitFrom(BOOLEAN, TIMESTAMP_WITHOUT_TIME_ZONE,
TIMESTAMP_WITH_LOCAL_TIME_ZONE)
+ .explicitFrom(
+ BOOLEAN,
+ TIMESTAMP_WITHOUT_TIME_ZONE,
+ TIMESTAMP_WITH_LOCAL_TIME_ZONE,
+ VARIANT)
.injectiveFrom(TINYINT, SMALLINT)
.build();
castTo(INTEGER)
.implicitFrom(TINYINT, SMALLINT, INTEGER)
.explicitFromFamily(NUMERIC, CHARACTER_STRING, INTERVAL)
- .explicitFrom(BOOLEAN, TIMESTAMP_WITHOUT_TIME_ZONE,
TIMESTAMP_WITH_LOCAL_TIME_ZONE)
+ .explicitFrom(
+ BOOLEAN,
+ TIMESTAMP_WITHOUT_TIME_ZONE,
+ TIMESTAMP_WITH_LOCAL_TIME_ZONE,
+ VARIANT)
.injectiveFrom(TINYINT, SMALLINT, INTEGER)
.build();
castTo(BIGINT)
.implicitFrom(TINYINT, SMALLINT, INTEGER, BIGINT)
.explicitFromFamily(NUMERIC, CHARACTER_STRING, INTERVAL)
- .explicitFrom(BOOLEAN, TIMESTAMP_WITHOUT_TIME_ZONE,
TIMESTAMP_WITH_LOCAL_TIME_ZONE)
+ .explicitFrom(
+ BOOLEAN,
+ TIMESTAMP_WITHOUT_TIME_ZONE,
+ TIMESTAMP_WITH_LOCAL_TIME_ZONE,
+ VARIANT)
.injectiveFrom(TINYINT, SMALLINT, INTEGER, BIGINT)
.build();
castTo(DECIMAL)
.implicitFromFamily(NUMERIC)
.explicitFromFamily(CHARACTER_STRING, INTERVAL)
- .explicitFrom(BOOLEAN, TIMESTAMP_WITHOUT_TIME_ZONE,
TIMESTAMP_WITH_LOCAL_TIME_ZONE)
+ .explicitFrom(
+ BOOLEAN,
+ TIMESTAMP_WITHOUT_TIME_ZONE,
+ TIMESTAMP_WITH_LOCAL_TIME_ZONE,
+ VARIANT)
.injectiveFrom(WHEN_PRECISION_AND_SCALE_MATCH, DECIMAL)
.build();
@@ -291,14 +310,22 @@ public final class LogicalTypeCasts {
castTo(FLOAT)
.implicitFrom(TINYINT, SMALLINT, INTEGER, BIGINT, FLOAT,
DECIMAL)
.explicitFromFamily(NUMERIC, CHARACTER_STRING)
- .explicitFrom(BOOLEAN, TIMESTAMP_WITHOUT_TIME_ZONE,
TIMESTAMP_WITH_LOCAL_TIME_ZONE)
+ .explicitFrom(
+ BOOLEAN,
+ TIMESTAMP_WITHOUT_TIME_ZONE,
+ TIMESTAMP_WITH_LOCAL_TIME_ZONE,
+ VARIANT)
.injectiveFrom(FLOAT)
.build();
castTo(DOUBLE)
.implicitFromFamily(NUMERIC)
.explicitFromFamily(CHARACTER_STRING)
- .explicitFrom(BOOLEAN, TIMESTAMP_WITHOUT_TIME_ZONE,
TIMESTAMP_WITH_LOCAL_TIME_ZONE)
+ .explicitFrom(
+ BOOLEAN,
+ TIMESTAMP_WITHOUT_TIME_ZONE,
+ TIMESTAMP_WITH_LOCAL_TIME_ZONE,
+ VARIANT)
.injectiveFrom(DOUBLE)
.build();
@@ -309,6 +336,7 @@ public final class LogicalTypeCasts {
castTo(BOOLEAN)
.implicitFrom(BOOLEAN)
.explicitFromFamily(CHARACTER_STRING, INTEGER_NUMERIC)
+ .explicitFrom(VARIANT)
.injectiveFrom(BOOLEAN)
.build();
@@ -319,6 +347,7 @@ public final class LogicalTypeCasts {
castTo(DATE)
.implicitFrom(DATE, TIMESTAMP_WITHOUT_TIME_ZONE)
.explicitFromFamily(TIMESTAMP, CHARACTER_STRING)
+ .explicitFrom(VARIANT)
.injectiveFrom(DATE)
.build();
@@ -331,6 +360,7 @@ public final class LogicalTypeCasts {
castTo(TIMESTAMP_WITHOUT_TIME_ZONE)
.implicitFrom(TIMESTAMP_WITHOUT_TIME_ZONE,
TIMESTAMP_WITH_LOCAL_TIME_ZONE)
.explicitFromFamily(DATETIME, CHARACTER_STRING, NUMERIC)
+ .explicitFrom(VARIANT)
.injectiveFrom(
WHEN_PRECISION_MATCHES,
TIMESTAMP_WITHOUT_TIME_ZONE,
@@ -346,6 +376,7 @@ public final class LogicalTypeCasts {
castTo(TIMESTAMP_WITH_LOCAL_TIME_ZONE)
.implicitFrom(TIMESTAMP_WITH_LOCAL_TIME_ZONE,
TIMESTAMP_WITHOUT_TIME_ZONE)
.explicitFromFamily(DATETIME, CHARACTER_STRING, NUMERIC)
+ .explicitFrom(VARIANT)
.injectiveFrom(
WHEN_PRECISION_MATCHES,
TIMESTAMP_WITH_LOCAL_TIME_ZONE,
@@ -648,9 +679,6 @@ public final class LogicalTypeCasts {
// BITMAP can only be cast to BYTES (unbounded VARBINARY), because
trimming or padding
// would corrupt the serialized bitmap data.
return allowExplicit && getLength(targetType) ==
VarBinaryType.MAX_LENGTH;
- } else if (sourceRoot == VARIANT) {
- // a VARIANT can only be explicitly cast to a supported scalar type
- return allowExplicit && supportsVariantToScalarCast(targetType);
}
if (implicitCastingRules.get(targetRoot).contains(sourceRoot)) {
@@ -731,45 +759,6 @@ public final class LogicalTypeCasts {
return false;
}
- private static boolean supportsVariantToScalarCast(LogicalType targetType)
{
- switch (targetType.getTypeRoot()) {
- case BOOLEAN:
- case TINYINT:
- case SMALLINT:
- case INTEGER:
- case BIGINT:
- case FLOAT:
- case DOUBLE:
- case DECIMAL:
- case BINARY:
- case VARBINARY:
- case DATE:
- case TIMESTAMP_WITHOUT_TIME_ZONE:
- case TIMESTAMP_WITH_LOCAL_TIME_ZONE:
- return true;
- default:
- // TIME has no counterpart in the Variant type model.
CHARACTER_STRING is handled by
- // the display-oriented VariantToStringCastRule and is
intentionally not offered as
- // a
- // user-facing cast here.
- return false;
- }
- }
-
- /**
- * Returns a hint pointing to the function that performs a conceptually
related operation when
- * an explicit cast is unsupported, or empty when no specific hint applies.
- */
- public static Optional<String> getUnsupportedCastHint(
- LogicalType sourceType, LogicalType targetType) {
- if (sourceType.is(VARIANT) && targetType.is(CHARACTER_STRING)) {
- return Optional.of(
- "Use the JSON_STRING function to convert a VARIANT to its
JSON string "
- + "representation.");
- }
- return Optional.empty();
- }
-
private static CastingRuleBuilder castTo(LogicalTypeRoot targetType) {
return new CastingRuleBuilder(targetType);
}
diff --git
a/flink-table/flink-table-common/src/test/java/org/apache/flink/table/types/LogicalTypeCastsTest.java
b/flink-table/flink-table-common/src/test/java/org/apache/flink/table/types/LogicalTypeCastsTest.java
index 6fb3574b1f5..c51408edd81 100644
---
a/flink-table/flink-table-common/src/test/java/org/apache/flink/table/types/LogicalTypeCastsTest.java
+++
b/flink-table/flink-table-common/src/test/java/org/apache/flink/table/types/LogicalTypeCastsTest.java
@@ -270,6 +270,7 @@ class LogicalTypeCastsTest {
// variant to scalar is explicit only
Arguments.of(new VariantType(), new BooleanType(), false,
true),
Arguments.of(new VariantType(), new TinyIntType(), false,
true),
+ Arguments.of(new VariantType(), new SmallIntType(), false,
true),
Arguments.of(new VariantType(), new IntType(), false, true),
Arguments.of(new VariantType(), new BigIntType(), false, true),
Arguments.of(new VariantType(), new DoubleType(), false, true),
@@ -284,11 +285,12 @@ class LogicalTypeCastsTest {
new VarBinaryType(VarBinaryType.MAX_LENGTH),
false,
true),
+ Arguments.of(new VariantType(), new CharType(), false, true),
+ Arguments.of(new VariantType(), VarCharType.STRING_TYPE,
false, true),
// variant identity cast is implicit
Arguments.of(new VariantType(), new VariantType(), true, true),
- // TIME, character strings and constructed targets are not
castable from variant
+ // TIME and constructed targets are not castable from variant
Arguments.of(new VariantType(), new TimeType(), false, false),
- Arguments.of(new VariantType(), VarCharType.STRING_TYPE,
false, false),
Arguments.of(new VariantType(), new ArrayType(new IntType()),
false, false),
Arguments.of(new VariantType(), new RowType(List.of()), false,
false),
Arguments.of(
diff --git
a/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/functions/casting/VariantToPrimitiveCastRule.java
b/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/functions/casting/VariantToPrimitiveCastRule.java
index 8a278f41498..9b42628498f 100644
---
a/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/functions/casting/VariantToPrimitiveCastRule.java
+++
b/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/functions/casting/VariantToPrimitiveCastRule.java
@@ -18,33 +18,30 @@
package org.apache.flink.table.planner.functions.casting;
-import org.apache.flink.table.data.DecimalData;
-import org.apache.flink.table.data.TimestampData;
import
org.apache.flink.table.planner.functions.casting.CastRuleUtils.CodeWriter;
+import org.apache.flink.table.runtime.functions.VariantCastUtils;
import org.apache.flink.table.types.logical.DecimalType;
import org.apache.flink.table.types.logical.LogicalType;
import org.apache.flink.table.types.logical.LogicalTypeRoot;
import org.apache.flink.table.types.logical.utils.LogicalTypeChecks;
import org.apache.flink.types.variant.Variant;
-import java.math.BigDecimal;
-import java.util.Arrays;
-
-import static org.apache.flink.table.planner.codegen.CodeGenUtils.className;
-import static org.apache.flink.table.planner.codegen.CodeGenUtils.newName;
-import static
org.apache.flink.table.planner.functions.casting.CastRuleUtils.arrayLength;
+import static
org.apache.flink.table.planner.codegen.CodeGenUtils.primitiveTypeTermForType;
import static
org.apache.flink.table.planner.functions.casting.CastRuleUtils.cast;
-import static
org.apache.flink.table.planner.functions.casting.CastRuleUtils.constructorCall;
import static
org.apache.flink.table.planner.functions.casting.CastRuleUtils.methodCall;
import static
org.apache.flink.table.planner.functions.casting.CastRuleUtils.staticCall;
-import static
org.apache.flink.table.planner.functions.casting.CastRuleUtils.ternaryOperator;
+import static
org.apache.flink.table.planner.functions.casting.CastRuleUtils.strLiteral;
/**
* {@link LogicalTypeRoot#VARIANT} to primitive type cast rule.
*
- * <p>Numeric targets are lenient and follow regular numeric cast semantics;
other targets require
- * the stored value to match the target kind. On a mismatch {@code CAST} fails
and {@code TRY_CAST}
- * returns {@code null}.
+ * <p>A cast succeeds only when the target holds the stored value without
altering it, otherwise it
+ * fails {@code CAST} and yields {@code null} for {@code TRY_CAST}. An integer
widens or narrows as
+ * long as it stays in range, a {@code DECIMAL} has to fit the precision and
scale without rounding,
+ * and a timestamp has to fit the target precision. {@code FLOAT} and {@code
DOUBLE} are approximate
+ * by definition, so they take any numeric kind and reject only a magnitude
out of range. Changing
+ * the kind itself is not implicit, so a decimal is not read as an integer and
a {@code TIMESTAMP}
+ * is not read as a {@code TIMESTAMP_LTZ}.
*
* <p>{@code CHARACTER_STRING} is handled by {@link VariantToStringCastRule};
{@code TIME} has no
* variant counterpart and is unsupported.
@@ -128,14 +125,49 @@ class VariantToPrimitiveCastRule extends
AbstractNullAwareCodeGeneratorCastRule<
case SMALLINT:
case INTEGER:
case BIGINT:
+ writer.assignStmt(
+ returnVariable,
+ cast(
+ primitiveTypeTermForType(targetLogicalType),
+ staticCall(
+ VariantCastUtils.class,
+ "toIntegral",
+ inputTerm,
+ // A long literal needs the suffix to
compile, since
+ // Long.MIN_VALUE does not fit an int
literal.
+ integralMin(targetLogicalType) + "L",
+ integralMax(targetLogicalType) + "L",
+
strLiteral(targetLogicalType.getTypeRoot().name()))));
+ break;
case FLOAT:
+ writer.assignStmt(
+ returnVariable, staticCall(VariantCastUtils.class,
"toFloat", inputTerm));
+ break;
case DOUBLE:
+ writer.assignStmt(
+ returnVariable, staticCall(VariantCastUtils.class,
"toDouble", inputTerm));
+ break;
case DECIMAL:
- writer.assignStmt(returnVariable, numericExpression(inputTerm,
targetLogicalType));
+ final DecimalType decimalType = (DecimalType)
targetLogicalType;
+ writer.assignStmt(
+ returnVariable,
+ staticCall(
+ VariantCastUtils.class,
+ "toDecimal",
+ inputTerm,
+ decimalType.getPrecision(),
+ decimalType.getScale()));
break;
case BINARY:
case VARBINARY:
- generateToBytes(context, inputTerm, returnVariable,
targetLogicalType, writer);
+ writer.assignStmt(
+ returnVariable,
+ staticCall(
+ VariantCastUtils.class,
+ "toBytes",
+ inputTerm,
+ LogicalTypeChecks.getLength(targetLogicalType),
+ targetLogicalType.is(LogicalTypeRoot.BINARY)));
break;
case DATE:
writer.assignStmt(
@@ -146,17 +178,19 @@ class VariantToPrimitiveCastRule extends
AbstractNullAwareCodeGeneratorCastRule<
writer.assignStmt(
returnVariable,
staticCall(
- TimestampData.class,
- "fromLocalDateTime",
- methodCall(inputTerm, "getDateTime")));
+ VariantCastUtils.class,
+ "toTimestamp",
+ inputTerm,
+
LogicalTypeChecks.getPrecision(targetLogicalType)));
break;
case TIMESTAMP_WITH_LOCAL_TIME_ZONE:
writer.assignStmt(
returnVariable,
staticCall(
- TimestampData.class,
- "fromInstant",
- methodCall(inputTerm, "getInstant")));
+ VariantCastUtils.class,
+ "toTimestampLtz",
+ inputTerm,
+
LogicalTypeChecks.getPrecision(targetLogicalType)));
break;
default:
throw new IllegalArgumentException(
@@ -165,73 +199,29 @@ class VariantToPrimitiveCastRule extends
AbstractNullAwareCodeGeneratorCastRule<
return writer.toString();
}
- /**
- * Converts a numeric variant to the numeric {@code target} via the
matching {@link Number}
- * accessor, mirroring regular numeric cast semantics. A non-numeric
variant raises {@link
- * ClassCastException}, failing {@code CAST} and yielding {@code null} for
{@code TRY_CAST}.
- */
- private static String numericExpression(String inputTerm, LogicalType
target) {
- final String number = cast(className(Number.class),
methodCall(inputTerm, "get"));
- if (!target.is(LogicalTypeRoot.DECIMAL)) {
- return methodCall(number, numberAccessor(target));
- }
- final DecimalType decimalType = (DecimalType) target;
- return staticCall(
- DecimalData.class,
- "fromBigDecimal",
- constructorCall(BigDecimal.class, methodCall(number,
"toString")),
- decimalType.getPrecision(),
- decimalType.getScale());
- }
-
- private static String numberAccessor(LogicalType target) {
+ private static long integralMin(LogicalType target) {
switch (target.getTypeRoot()) {
case TINYINT:
- return "byteValue";
+ return Byte.MIN_VALUE;
case SMALLINT:
- return "shortValue";
+ return Short.MIN_VALUE;
case INTEGER:
- return "intValue";
- case BIGINT:
- return "longValue";
- case FLOAT:
- return "floatValue";
- case DOUBLE:
- return "doubleValue";
+ return Integer.MIN_VALUE;
default:
- throw new IllegalArgumentException(
- "Unsupported numeric target for casting from VARIANT:
" + target);
+ return Long.MIN_VALUE;
}
}
- private static void generateToBytes(
- CodeGeneratorCastRule.Context context,
- String inputTerm,
- String returnVariable,
- LogicalType targetLogicalType,
- CodeWriter writer) {
- final int targetLength =
LogicalTypeChecks.getLength(targetLogicalType);
- // Read the bytes once to avoid decoding the variant twice.
- final String bytesTerm = newName(context.getCodeGeneratorContext(),
"variantBytes");
- writer.declStmt("byte[]", bytesTerm, methodCall(inputTerm,
"getBytes"));
- if (BinaryToBinaryCastRule.couldPad(targetLogicalType, targetLength)) {
- // BINARY(n): pad or trim to the exact target length.
- writer.assignStmt(
- returnVariable,
- ternaryOperator(
- arrayLength(bytesTerm) + " == " + targetLength,
- bytesTerm,
- staticCall(Arrays.class, "copyOf", bytesTerm,
targetLength)));
- } else if (BinaryToBinaryCastRule.couldTrim(targetLength)) {
- // VARBINARY(n): trim only when longer than the target length.
- writer.assignStmt(
- returnVariable,
- ternaryOperator(
- arrayLength(bytesTerm) + " <= " + targetLength,
- bytesTerm,
- staticCall(Arrays.class, "copyOf", bytesTerm,
targetLength)));
- } else {
- writer.assignStmt(returnVariable, bytesTerm);
+ private static long integralMax(LogicalType target) {
+ switch (target.getTypeRoot()) {
+ case TINYINT:
+ return Byte.MAX_VALUE;
+ case SMALLINT:
+ return Short.MAX_VALUE;
+ case INTEGER:
+ return Integer.MAX_VALUE;
+ default:
+ return Long.MAX_VALUE;
}
}
}
diff --git
a/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/functions/casting/VariantToStringCastRule.java
b/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/functions/casting/VariantToStringCastRule.java
index 29e7821bc3a..eeb9fcec9f6 100644
---
a/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/functions/casting/VariantToStringCastRule.java
+++
b/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/functions/casting/VariantToStringCastRule.java
@@ -18,16 +18,39 @@
package org.apache.flink.table.planner.functions.casting;
+import org.apache.flink.table.data.StringData;
+import org.apache.flink.table.runtime.functions.VariantCastUtils;
import org.apache.flink.table.types.logical.LogicalType;
import org.apache.flink.table.types.logical.LogicalTypeFamily;
import org.apache.flink.table.types.logical.LogicalTypeRoot;
+import org.apache.flink.table.types.logical.utils.LogicalTypeChecks;
import org.apache.flink.types.variant.Variant;
+import static
org.apache.flink.table.planner.codegen.calls.BuiltInMethods.BINARY_STRING_DATA_FROM_STRING;
import static
org.apache.flink.table.planner.functions.casting.CastRuleUtils.methodCall;
-import static org.apache.flink.table.types.logical.VarCharType.STRING_TYPE;
+import static
org.apache.flink.table.planner.functions.casting.CastRuleUtils.staticCall;
-/** {@link LogicalTypeRoot#VARIANT} to {@link
LogicalTypeFamily#CHARACTER_STRING} cast rule. */
-class VariantToStringCastRule extends
AbstractCharacterFamilyTargetRule<Variant> {
+/**
+ * {@link LogicalTypeRoot#VARIANT} to {@link
LogicalTypeFamily#CHARACTER_STRING} cast rule.
+ *
+ * <p>Renders the scalar value the way a regular SQL cast of the stored kind
would, so a boolean
+ * becomes {@code TRUE}, a timestamp uses the SQL format, and a {@code
TIMESTAMP_LTZ} is shifted
+ * into the session time zone and a binary value is read as UTF-8. A variant
holding an object or
+ * array has no scalar rendering and fails; use {@code JSON_STRING} for its
JSON representation.
+ *
+ * <p>A binary value that is not well-formed UTF-8 fails instead of decoding
to {@code U+FFFD}. Cast
+ * it to {@code BYTES} to inspect the raw value, or wrap that in {@code
MAKE_VALID_UTF8} to accept
+ * the lenient decode explicitly.
+ *
+ * <p>A value that does not fit a {@code CHAR(n)} or {@code VARCHAR(n)} target
is trimmed, and a
+ * {@code CHAR} target pads a shorter value to its fixed width. The length is
enforced here rather
+ * than by {@link CharVarCharTrimPadCastRule}, so that a variant storing a
JSON {@code null} still
+ * reaches SQL {@code NULL} and so that trimming counts code points rather
than UTF-16 units.
+ *
+ * <p>Printing a result is not a cast and cannot fail, so it renders every
variant as JSON rather
+ * than extracting the scalar value. A stored string therefore prints quoted.
+ */
+class VariantToStringCastRule extends
AbstractExpressionCodeGeneratorCastRule<Variant, StringData> {
static final VariantToStringCastRule INSTANCE = new
VariantToStringCastRule();
@@ -35,16 +58,56 @@ class VariantToStringCastRule extends
AbstractCharacterFamilyTargetRule<Variant>
super(
CastRulePredicate.builder()
.input(LogicalTypeRoot.VARIANT)
- .target(STRING_TYPE)
+ .target(LogicalTypeRoot.CHAR)
+ .target(LogicalTypeRoot.VARCHAR)
.build());
}
@Override
- public String generateStringExpression(
+ public boolean canFail(LogicalType inputLogicalType, LogicalType
targetLogicalType) {
+ return true;
+ }
+
+ /**
+ * Treats a variant that stores a JSON {@code null} as a {@code NULL}
input, so it casts to SQL
+ * {@code NULL} instead of the text {@code null}. Only applied for a
nullable target: a {@code
+ * NOT NULL} result cannot carry {@code NULL}, so a null-valued variant
then fails.
+ */
+ @Override
+ public CastCodeBlock generateCodeBlock(
+ CodeGeneratorCastRule.Context context,
+ String inputTerm,
+ String inputIsNullTerm,
+ LogicalType inputLogicalType,
+ LogicalType targetLogicalType) {
+ if (!targetLogicalType.isNullable()) {
+ return super.generateCodeBlock(
+ context, inputTerm, inputIsNullTerm, inputLogicalType,
targetLogicalType);
+ }
+ final String isNullTerm =
+ "(" + inputIsNullTerm + " || " + methodCall(inputTerm,
"isNull") + ")";
+ return super.generateCodeBlock(
+ context, inputTerm, isNullTerm, inputLogicalType,
targetLogicalType);
+ }
+
+ @Override
+ public String generateExpression(
CodeGeneratorCastRule.Context context,
String inputTerm,
LogicalType inputLogicalType,
LogicalType targetLogicalType) {
- return methodCall(inputTerm, "toString");
+ if (context.isPrinting()) {
+ // Every result has to be displayable, including an object or an
array, which have no
+ // scalar rendering and would fail the cast. toJson returns a
String, so it needs the
+ // wrap that toStringValue applies itself.
+ return staticCall(BINARY_STRING_DATA_FROM_STRING(),
methodCall(inputTerm, "toJson"));
+ }
+ return staticCall(
+ VariantCastUtils.class,
+ "toStringValue",
+ inputTerm,
+ context.getSessionTimeZoneTerm(),
+ LogicalTypeChecks.getLength(targetLogicalType),
+ targetLogicalType.is(LogicalTypeRoot.CHAR));
}
}
diff --git
a/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/functions/CastFunctionITCase.java
b/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/functions/CastFunctionITCase.java
index 6ad3ab7cb11..aee2d70c1cf 100644
---
a/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/functions/CastFunctionITCase.java
+++
b/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/functions/CastFunctionITCase.java
@@ -138,75 +138,221 @@ public class CastFunctionITCase extends
BuiltInFunctionTestBase {
}
private static List<TestSetSpec> variantCasts() {
- // A variant is produced with PARSE_JSON so that the source column
stays a STRING literal;
- // there is no VARIANT literal to feed a source column directly. A
JSON integer is stored in
- // the smallest integer type that fits (42 -> TINYINT), and numeric
casts are lenient, so it
- // still widens to INT.
+ // A variant is produced with PARSE_JSON since there is no VARIANT
literal. Numeric casts
+ // succeed only when the value is preserved exactly, otherwise CAST
fails and TRY_CAST
+ // returns NULL.
return List.of(
TestSetSpec.forExpression("Cast a VARIANT produced by
PARSE_JSON to a primitive")
.onFieldsWithData("unused")
.andDataTypes(STRING())
+ // An integer converts to any integer target while the
value stays in
+ // range, and to FLOAT or DOUBLE which are approximate
by definition.
+ .testResult(
+ call("PARSE_JSON", "42").cast(TINYINT()),
+ "CAST(PARSE_JSON('42') AS TINYINT)",
+ (byte) 42,
+ TINYINT().notNull())
+ .testResult(
+ call("PARSE_JSON", "42").cast(SMALLINT()),
+ "CAST(PARSE_JSON('42') AS SMALLINT)",
+ (short) 42,
+ SMALLINT().notNull())
.testResult(
call("PARSE_JSON", "42").cast(INT()),
"CAST(PARSE_JSON('42') AS INT)",
42,
INT().notNull())
- // Integer overflow wraps around (Java narrowing),
like a regular numeric
- // cast.
.testResult(
- call("PARSE_JSON", "40000").cast(SMALLINT()),
- "CAST(PARSE_JSON('40000') AS SMALLINT)",
- (short) -25536,
+ call("PARSE_JSON", "42").cast(BIGINT()),
+ "CAST(PARSE_JSON('42') AS BIGINT)",
+ 42L,
+ BIGINT().notNull())
+ .testResult(
+ call("PARSE_JSON", "42").cast(FLOAT()),
+ "CAST(PARSE_JSON('42') AS FLOAT)",
+ 42.0f,
+ FLOAT().notNull())
+ .testResult(
+ call("PARSE_JSON", "42").cast(DOUBLE()),
+ "CAST(PARSE_JSON('42') AS DOUBLE)",
+ 42.0d,
+ DOUBLE().notNull())
+ // An out-of-range value is rejected rather than
wrapped.
+ .testResult(
+ call("PARSE_JSON", "1000").cast(SMALLINT()),
+ "CAST(PARSE_JSON('1000') AS SMALLINT)",
+ (short) 1000,
SMALLINT().notNull())
+ .testTableApiRuntimeError(
+ call("PARSE_JSON", "1000").cast(TINYINT()),
"overflowed")
+ .testSqlRuntimeError("CAST(PARSE_JSON('1000') AS
TINYINT)", "overflowed")
.testResult(
- call("PARSE_JSON", "128").cast(TINYINT()),
- "CAST(PARSE_JSON('128') AS TINYINT)",
- (byte) -128,
- TINYINT().notNull())
+ call("PARSE_JSON", "1000").tryCast(TINYINT()),
+ "TRY_CAST(PARSE_JSON('1000') AS TINYINT)",
+ null,
+ TINYINT())
+ // A decimal reaches an integer target when it is
already integral.
.testResult(
- call("PARSE_JSON", "2147483648").cast(INT()),
- "CAST(PARSE_JSON('2147483648') AS INT)",
- -2147483648,
+ call("PARSE_JSON", "7.0").cast(INT()),
+ "CAST(PARSE_JSON('7.0') AS INT)",
+ 7,
INT().notNull())
+ // A fractional value is rejected, since converting
would drop digits.
+ .testTableApiRuntimeError(
+ call("PARSE_JSON", "123.456").cast(INT()),
"lose precision")
.testResult(
- call("PARSE_JSON",
"9223372036854775808").cast(BIGINT()),
- "CAST(PARSE_JSON('9223372036854775808') AS
BIGINT)",
- -9223372036854775808L,
- BIGINT().notNull())
- // An out-of-range floating point value saturates when
cast to an integer,
- // and a fractional value is truncated toward zero.
+ call("PARSE_JSON", "123.456").tryCast(INT()),
+ "TRY_CAST(PARSE_JSON('123.456') AS INT)",
+ null,
+ INT())
+ // A DECIMAL target has to hold the value exactly.
.testResult(
- call("PARSE_JSON", "1e20").cast(INT()),
- "CAST(PARSE_JSON('1e20') AS INT)",
- 2147483647,
- INT().notNull())
+ call("PARSE_JSON", "123.456").cast(DECIMAL(6,
3)),
+ "CAST(PARSE_JSON('123.456') AS DECIMAL(6, 3))",
+ new BigDecimal("123.456"),
+ DECIMAL(6, 3).notNull())
+ .testTableApiRuntimeError(
+ call("PARSE_JSON", "123.456").cast(DECIMAL(6,
2)), "lose precision")
.testResult(
- call("PARSE_JSON", "3.9").cast(INT()),
- "CAST(PARSE_JSON('3.9') AS INT)",
- 3,
- INT().notNull())
- // A value beyond the FLOAT range becomes infinity.
+ call("PARSE_JSON",
"123.456").tryCast(DECIMAL(6, 2)),
+ "TRY_CAST(PARSE_JSON('123.456') AS DECIMAL(6,
2))",
+ null,
+ DECIMAL(6, 2))
+ .testTableApiRuntimeError(
+ call("PARSE_JSON", "123.456").cast(DECIMAL(5,
3)), "overflowed")
.testResult(
- call("PARSE_JSON", "1e40").cast(FLOAT()),
- "CAST(PARSE_JSON('1e40') AS FLOAT)",
- Float.POSITIVE_INFINITY,
+ call("PARSE_JSON",
"123.456").tryCast(DECIMAL(5, 3)),
+ "TRY_CAST(PARSE_JSON('123.456') AS DECIMAL(5,
3))",
+ null,
+ DECIMAL(5, 3))
+ // An integer is exact, so it reaches a DECIMAL that
has room for it.
+ .testResult(
+ call("PARSE_JSON", "42").cast(DECIMAL(5, 2)),
+ "CAST(PARSE_JSON('42') AS DECIMAL(5, 2))",
+ new BigDecimal("42.00"),
+ DECIMAL(5, 2).notNull())
+ // A decimal reaches an approximate target, where
losing digits is expected.
+ .testResult(
+ call("PARSE_JSON", "123.456").cast(FLOAT()),
+ "CAST(PARSE_JSON('123.456') AS FLOAT)",
+ 123.456f,
FLOAT().notNull())
- // DECIMAL overflow yields NULL instead of wrapping;
TRY_CAST surfaces it.
.testResult(
- call("PARSE_JSON",
"123.456").tryCast(DECIMAL(4, 2)),
- "TRY_CAST(PARSE_JSON('123.456') AS DECIMAL(4,
2))",
+ call("PARSE_JSON", "123.456").cast(DOUBLE()),
+ "CAST(PARSE_JSON('123.456') AS DOUBLE)",
+ 123.456d,
+ DOUBLE().notNull())
+ // A magnitude the target cannot represent is still
rejected.
+ .testTableApiRuntimeError(
+ call("PARSE_JSON", "1e40").cast(FLOAT()),
"overflowed")
+ .testResult(
+ call("PARSE_JSON", "1e40").tryCast(FLOAT()),
+ "TRY_CAST(PARSE_JSON('1e40') AS FLOAT)",
null,
- DECIMAL(4, 2))
+ FLOAT())
.testResult(
- call("PARSE_JSON", "42").cast(BIGINT()),
- "CAST(PARSE_JSON('42') AS BIGINT)",
- 42L,
- BIGINT().notNull())
+ call("PARSE_JSON", "1e20").cast(DOUBLE()),
+ "CAST(PARSE_JSON('1e20') AS DOUBLE)",
+ 1e20,
+ DOUBLE().notNull())
.testResult(
call("PARSE_JSON", "true").cast(BOOLEAN()),
"CAST(PARSE_JSON('true') AS BOOLEAN)",
true,
BOOLEAN().notNull())
+ // CAST returns the raw scalar value (string unquoted)
+ .testResult(
+ call("PARSE_JSON", "\"foo\"").cast(STRING()),
+ "CAST(PARSE_JSON('\"foo\"') AS STRING)",
+ "foo",
+ STRING().notNull())
+ .testResult(
+ call("PARSE_JSON", "123.456").cast(STRING()),
+ "CAST(PARSE_JSON('123.456') AS STRING)",
+ "123.456",
+ STRING().notNull())
+ // The rendering matches a regular cast of the stored
kind, so a boolean
+ // becomes TRUE rather than the JSON true.
+ .testResult(
+ call("PARSE_JSON", "true").cast(STRING()),
+ "CAST(PARSE_JSON('true') AS STRING)",
+ "TRUE",
+ STRING().notNull())
+ // An object or array has no scalar value, so the
error points to
+ // JSON_STRING.
+ .testTableApiRuntimeError(
+ call("PARSE_JSON", "[\"a\",
\"b\"]").cast(STRING()), "JSON_STRING")
+ .testResult(
+ call("PARSE_JSON", "[\"a\",
\"b\"]").tryCast(STRING()),
+ "TRY_CAST(PARSE_JSON('[\"a\", \"b\"]') AS
STRING)",
+ null,
+ STRING())
+ .testTableApiRuntimeError(
+ call("PARSE_JSON", "{\"a\":
1}").cast(STRING()), "JSON_STRING")
+ .testResult(
+ call("PARSE_JSON", "{\"a\":
1}").tryCast(STRING()),
+ "TRY_CAST(PARSE_JSON('{\"a\": 1}') AS STRING)",
+ null,
+ STRING())
+ // A bounded CHAR/VARCHAR target trims a longer value,
and CHAR pads a
+ // shorter one to its fixed width, the same as a
regular cast into it.
+ .testResult(
+ call("PARSE_JSON", "\"ab\"").cast(VARCHAR(3)),
+ "CAST(PARSE_JSON('\"ab\"') AS VARCHAR(3))",
+ "ab",
+ VARCHAR(3).notNull())
+ .testResult(
+ call("PARSE_JSON",
"\"foobar\"").cast(VARCHAR(3)),
+ "CAST(PARSE_JSON('\"foobar\"') AS VARCHAR(3))",
+ "foo",
+ VARCHAR(3).notNull())
+ .testResult(
+ call("PARSE_JSON",
"\"foobar\"").tryCast(VARCHAR(3)),
+ "TRY_CAST(PARSE_JSON('\"foobar\"') AS
VARCHAR(3))",
+ "foo",
+ VARCHAR(3))
+ .testResult(
+ call("PARSE_JSON", "\"abc\"").cast(CHAR(3)),
+ "CAST(PARSE_JSON('\"abc\"') AS CHAR(3))",
+ "abc",
+ CHAR(3).notNull())
+ .testResult(
+ call("PARSE_JSON", "\"abcdef\"").cast(CHAR(3)),
+ "CAST(PARSE_JSON('\"abcdef\"') AS CHAR(3))",
+ "abc",
+ CHAR(3).notNull())
+ .testResult(
+ call("PARSE_JSON", "\"ab\"").cast(CHAR(5)),
+ "CAST(PARSE_JSON('\"ab\"') AS CHAR(5))",
+ "ab ",
+ CHAR(5).notNull())
+ .testResult(
+ call("PARSE_JSON", "\"ab\"").tryCast(CHAR(5)),
+ "TRY_CAST(PARSE_JSON('\"ab\"') AS CHAR(5))",
+ "ab ",
+ CHAR(5))
+ // A variant holding a JSON null casts to SQL NULL,
not to the text 'null'.
+ // The length of that text must not be checked against
the target either.
+ .testResult(
+ call("TRY_PARSE_JSON", "null").cast(STRING()),
+ "CAST(TRY_PARSE_JSON('null') AS STRING)",
+ null,
+ STRING())
+ .testResult(
+ call("PARSE_JSON", "null").tryCast(STRING()),
+ "TRY_CAST(PARSE_JSON('null') AS STRING)",
+ null,
+ STRING())
+ .testResult(
+ call("TRY_PARSE_JSON", "null").cast(CHAR(2)),
+ "CAST(TRY_PARSE_JSON('null') AS CHAR(2))",
+ null,
+ CHAR(2))
+ .testResult(
+ call("PARSE_JSON", "null").tryCast(VARCHAR(2)),
+ "TRY_CAST(PARSE_JSON('null') AS VARCHAR(2))",
+ null,
+ VARCHAR(2))
// TRY_CAST of a value whose kind does not match the
target returns NULL
.testResult(
call("PARSE_JSON", "\"foo\"").tryCast(INT()),
@@ -227,15 +373,10 @@ public class CastFunctionITCase extends
BuiltInFunctionTestBase {
BOOLEAN())
// A nullable variant with a concrete value still
casts normally.
.testResult(
- call("TRY_PARSE_JSON", "42").cast(INT()),
- "CAST(TRY_PARSE_JSON('42') AS INT)",
- 42,
- INT())
- // Casting a VARIANT to a string points the user to
JSON_STRING.
- .testTableApiValidationError(
- call("PARSE_JSON", "42").cast(STRING()),
- "Use the JSON_STRING function to convert a
VARIANT to its JSON "
- + "string representation."));
+ call("TRY_PARSE_JSON", "42").cast(TINYINT()),
+ "CAST(TRY_PARSE_JSON('42') AS TINYINT)",
+ (byte) 42,
+ TINYINT()));
}
private static List<TestSetSpec> allTypesBasic() {
diff --git
a/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/functions/casting/CastRulesTest.java
b/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/functions/casting/CastRulesTest.java
index aca973fa3ca..e00c99c55ce 100644
---
a/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/functions/casting/CastRulesTest.java
+++
b/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/functions/casting/CastRulesTest.java
@@ -161,6 +161,33 @@ class CastRulesTest {
private static final Bitmap DEFAULT_BITMAP = Bitmap.fromArray(new int[]
{0, 1, 2});
+ /** A two-byte lead followed by a byte that is not a continuation byte. */
+ private static final byte[] INVALID_UTF8 = new byte[] {(byte) 0xC3, (byte)
0x28};
+
+ /** U+1D54F, one code point but two UTF-16 units and four UTF-8 bytes. */
+ private static final String NON_BMP = "𝕏";
+
+ private static final Variant VARIANT_ARRAY =
+ Variant.newBuilder()
+ .array()
+ .add(Variant.newBuilder().of(1))
+ .add(Variant.newBuilder().of("two"))
+ .add(Variant.newBuilder().of(false))
+ .add(Variant.newBuilder().ofNull())
+ .build();
+
+ private static final Variant VARIANT_OBJECT =
+ Variant.newBuilder()
+ .object()
+ .add(
+ "k",
+ Variant.newBuilder()
+ .array()
+ .add(Variant.newBuilder().of(1))
+ .add(Variant.newBuilder().of(2))
+ .build())
+ .build();
+
private static final DataType MY_STRUCTURED_TYPE =
STRUCTURED(
MyStructuredType.class,
@@ -1544,35 +1571,138 @@ class CastRulesTest {
.fromCase(BITMAP(), DEFAULT_BITMAP,
DEFAULT_BITMAP.toBytes())
.fromCase(BITMAP(), Bitmap.empty(),
Bitmap.empty().toBytes())
.fromCase(BITMAP(), null, null),
- // From VARIANT to primitive types. Numeric targets are
lenient: a variant holding
- // any numeric kind converts to the requested numeric type
(widening and narrowing).
- // Non-numeric targets are strict: the stored kind must match,
otherwise the cast
- // fails and TRY_CAST returns null.
+ // From VARIANT to primitive types. A cast succeeds only when
the target holds the
+ // stored value unaltered, except for the approximate FLOAT
and DOUBLE.
+ // A character string renders like a regular cast of the
stored kind, so these
+ // expectations reuse the constants of the native cases above.
+ CastTestSpecBuilder.testCastTo(STRING())
+ .fromCase(VARIANT(), Variant.newBuilder().of(true),
fromString("TRUE"))
+ .fromCase(VARIANT(), Variant.newBuilder().of(false),
fromString("FALSE"))
+ .fromCase(VARIANT(), Variant.newBuilder().of("foo"),
fromString("foo"))
+ .fromCase(VARIANT(), Variant.newBuilder().of(42),
fromString("42"))
+ .fromCase(
+ VARIANT(),
+ Variant.newBuilder().of(new
BigDecimal("123.456")),
+ fromString("123.456"))
+ // a small scale stays plain instead of turning into
scientific notation
+ .fromCase(
+ VARIANT(),
+ Variant.newBuilder().of(new
BigDecimal("0.0000000001")),
+ fromString("0.0000000001"))
+ .fromCase(
+ VARIANT(),
+
Variant.newBuilder().of(LocalDate.parse("2021-09-24")),
+ DATE_STRING)
+ .fromCase(
+ VARIANT(),
+
Variant.newBuilder().of(TIMESTAMP.toLocalDateTime()),
+ TIMESTAMP_STRING)
+ .fromCase(
+ VARIANT(),
+ CET_CONTEXT,
+ Variant.newBuilder().of(TIMESTAMP.toInstant()),
+ TIMESTAMP_STRING_CET)
+ // a binary value is read as UTF-8, like a regular
BINARY to string cast
+ .fromCase(
+ VARIANT(),
+
Variant.newBuilder().of("hello".getBytes(StandardCharsets.UTF_8)),
+ fromString("hello"))
+ // a multi-byte sequence passes the well-formedness
check unchanged
+ .fromCase(
+ VARIANT(),
+
Variant.newBuilder().of("héllo".getBytes(StandardCharsets.UTF_8)),
+ fromString("héllo"))
+ // bytes that are not valid UTF-8 are rejected rather
than decoded into
+ // the U+FFFD replacement character
+ .fail(
+ VARIANT(),
+ Variant.newBuilder().of(INVALID_UTF8),
+ TableRuntimeException.class)
+ // an object or an array has no scalar rendering and
fails the cast
+ .fail(VARIANT(), VARIANT_ARRAY,
TableRuntimeException.class)
+ .fail(VARIANT(), VARIANT_OBJECT,
TableRuntimeException.class)
+ // printing is not a cast and cannot fail, so it
renders JSON instead,
+ // which leaves a stored string quoted
+ .fromCasePrinting(
+ VARIANT(), VARIANT_ARRAY,
fromString("[1,\"two\",false,null]"))
+ .fromCasePrinting(VARIANT(), VARIANT_OBJECT,
fromString("{\"k\":[1,2]}"))
+ .fromCasePrinting(
+ VARIANT(), Variant.newBuilder().of("foo"),
fromString("\"foo\""))
+ .fromCasePrinting(VARIANT(),
Variant.newBuilder().of(42), fromString("42")),
+ // A bounded character target pads and trims like any other
cast into it, and its
+ // length counts code points, so a character outside the BMP
fills one position
+ // even though it occupies two UTF-16 units.
+ CastTestSpecBuilder.testCastTo(CHAR(1))
+ .fromCase(VARIANT(), Variant.newBuilder().of("x"),
fromString("x"))
+ .fromCase(VARIANT(), Variant.newBuilder().of(NON_BMP),
fromString(NON_BMP))
+ .fromCase(
+ VARIANT(),
+ Variant.newBuilder().of(NON_BMP + NON_BMP),
+ fromString(NON_BMP))
+ .fromCase(
+ VARIANT(),
Variant.newBuilder().of("abcdefghij"), fromString("a")),
+ CastTestSpecBuilder.testCastTo(CHAR(5))
+ // shorter than the target, so it is padded to the
fixed width
+ .fromCase(VARIANT(), Variant.newBuilder().of("ab"),
fromString("ab "))
+ .fromCase(
+ VARIANT(),
+ Variant.newBuilder().of("abcdefghij"),
+ fromString("abcde")),
+ CastTestSpecBuilder.testCastTo(VARCHAR(2))
+ .fromCase(VARIANT(), Variant.newBuilder().of(NON_BMP),
fromString(NON_BMP))
+ .fromCase(
+ VARIANT(),
+ Variant.newBuilder().of(NON_BMP + NON_BMP),
+ fromString(NON_BMP + NON_BMP))
+ // longer than the target, so it is trimmed rather
than rejected, and a
+ // variable width target is not padded
+ .fromCase(
+ VARIANT(),
+ Variant.newBuilder().of(NON_BMP + NON_BMP +
NON_BMP),
+ fromString(NON_BMP + NON_BMP))
+ .fromCase(VARIANT(), Variant.newBuilder().of("a"),
fromString("a")),
CastTestSpecBuilder.testCastTo(BOOLEAN())
.fromCase(VARIANT(), Variant.newBuilder().of(true),
true)
.fromCase(VARIANT(), Variant.newBuilder().of(false),
false)
.fail(VARIANT(), Variant.newBuilder().of(1),
TableRuntimeException.class),
CastTestSpecBuilder.testCastTo(TINYINT())
.fromCase(VARIANT(), Variant.newBuilder().of((byte)
42), (byte) 42)
+ // a wider integer kind narrows while the value is in
range
.fromCase(VARIANT(), Variant.newBuilder().of(42),
(byte) 42)
+ // out of range is rejected instead of wrapping
+ .fail(VARIANT(), Variant.newBuilder().of(1000),
TableRuntimeException.class)
.fail(VARIANT(), Variant.newBuilder().of("x"),
TableRuntimeException.class),
CastTestSpecBuilder.testCastTo(SMALLINT())
.fromCase(VARIANT(), Variant.newBuilder().of((short)
42), (short) 42)
.fromCase(VARIANT(), Variant.newBuilder().of((byte)
42), (short) 42)
+ .fromCase(VARIANT(), Variant.newBuilder().of(1000),
(short) 1000)
+ .fail(
+ VARIANT(),
+ Variant.newBuilder().of(40000),
+ TableRuntimeException.class)
.fail(
VARIANT(),
Variant.newBuilder().of(true),
TableRuntimeException.class),
CastTestSpecBuilder.testCastTo(INT())
- // widening: a JSON integer is stored in the smallest
type but still casts
- // up
+ .fromCase(VARIANT(), Variant.newBuilder().of(42), 42)
+ // every integer kind converts as long as the value
fits
.fromCase(VARIANT(), Variant.newBuilder().of((byte)
42), 42)
.fromCase(VARIANT(), Variant.newBuilder().of((short)
42), 42)
- .fromCase(VARIANT(), Variant.newBuilder().of(42), 42)
.fromCase(VARIANT(), Variant.newBuilder().of(42L), 42)
- // narrowing from a floating point or decimal value
truncates
- .fromCase(VARIANT(), Variant.newBuilder().of(3.9d), 3)
- .fromCase(VARIANT(), Variant.newBuilder().of(new
BigDecimal("7.2")), 7)
+ .fail(
+ VARIANT(),
+ Variant.newBuilder().of(2147483648L),
+ TableRuntimeException.class)
+ // an approximate or decimal kind converts when the
value is integral
+ .fromCase(VARIANT(), Variant.newBuilder().of(7.0d), 7)
+ .fromCase(VARIANT(), Variant.newBuilder().of(new
BigDecimal("7.0")), 7)
+ // a fractional value would have to be rounded away,
so it is rejected
+ .fail(VARIANT(), Variant.newBuilder().of(7.2d),
TableRuntimeException.class)
+ .fail(
+ VARIANT(),
+ Variant.newBuilder().of(new BigDecimal("7.2")),
+ TableRuntimeException.class)
// a non-numeric variant cannot be cast to a number
.fail(
VARIANT(),
@@ -1587,14 +1717,28 @@ class CastRulesTest {
.fromCase(VARIANT(), Variant.newBuilder().of(42), 42L)
.fail(VARIANT(), Variant.newBuilder().of("x"),
TableRuntimeException.class),
CastTestSpecBuilder.testCastTo(FLOAT())
+ // every numeric kind reaches an approximate target
.fromCase(VARIANT(), Variant.newBuilder().of(1.5f),
1.5f)
.fromCase(VARIANT(), Variant.newBuilder().of(1.5d),
1.5f)
.fromCase(VARIANT(), Variant.newBuilder().of(3), 3.0f)
+ .fromCase(
+ VARIANT(),
+ Variant.newBuilder().of(new
BigDecimal("123.456")),
+ 123.456f)
+ // a magnitude a FLOAT cannot represent is still
rejected
+ .fail(
+ VARIANT(),
+ Variant.newBuilder().of(1e40d),
+ TableRuntimeException.class)
.fail(VARIANT(), Variant.newBuilder().of("x"),
TableRuntimeException.class),
CastTestSpecBuilder.testCastTo(DOUBLE())
.fromCase(VARIANT(), Variant.newBuilder().of(1.5d),
1.5d)
.fromCase(VARIANT(), Variant.newBuilder().of(1.5f),
1.5d)
.fromCase(VARIANT(), Variant.newBuilder().of(3), 3.0d)
+ .fromCase(
+ VARIANT(),
+ Variant.newBuilder().of(new
BigDecimal("123.456")),
+ 123.456d)
.fail(VARIANT(), Variant.newBuilder().of("x"),
TableRuntimeException.class),
CastTestSpecBuilder.testCastTo(DECIMAL(5, 2))
.fromCase(VARIANT(), null, null)
@@ -1602,10 +1746,23 @@ class CastRulesTest {
VARIANT(),
Variant.newBuilder().of(new
BigDecimal("123.45")),
DecimalData.fromBigDecimal(new
BigDecimal("123.45"), 5, 2))
+ // trailing zeros may be appended to reach the target
scale
+ .fromCase(
+ VARIANT(),
+ Variant.newBuilder().of(new
BigDecimal("123.4")),
+ DecimalData.fromBigDecimal(new
BigDecimal("123.40"), 5, 2))
+ // an integer is exact, so it converts when it fits
.fromCase(
VARIANT(),
Variant.newBuilder().of(42),
- DecimalData.fromBigDecimal(new
BigDecimal("42"), 5, 2))
+ DecimalData.fromBigDecimal(new
BigDecimal("42.00"), 5, 2))
+ // a scale that would have to round is rejected
+ .fail(
+ VARIANT(),
+ Variant.newBuilder().of(new
BigDecimal("123.456")),
+ TableRuntimeException.class)
+ // an approximate kind is not read as a decimal
+ .fail(VARIANT(), Variant.newBuilder().of(1.5d),
TableRuntimeException.class)
.fail(VARIANT(), Variant.newBuilder().of("x"),
TableRuntimeException.class),
CastTestSpecBuilder.testCastTo(BYTES())
.fromCase(VARIANT(), null, null)
@@ -1613,6 +1770,9 @@ class CastRulesTest {
VARIANT(),
Variant.newBuilder().of(new byte[] {1, 2, 3}),
new byte[] {1, 2, 3})
+ // the raw bytes stay reachable when the character
string cast rejects
+ // them, which is what makes this the way to inspect
such a value
+ .fromCase(VARIANT(),
Variant.newBuilder().of(INVALID_UTF8), INVALID_UTF8)
.fail(
VARIANT(),
Variant.newBuilder().of("foo"),
@@ -1630,13 +1790,82 @@ class CastRulesTest {
Variant.newBuilder().of(LocalDateTime.of(2020,
1, 1, 12, 0, 0)),
TimestampData.fromLocalDateTime(
LocalDateTime.of(2020, 1, 1, 12, 0,
0)))
- .fail(VARIANT(), Variant.newBuilder().of(1),
TableRuntimeException.class),
+ .fail(VARIANT(), Variant.newBuilder().of(1),
TableRuntimeException.class)
+ // a TIMESTAMP_LTZ is a different kind and is not read
as a TIMESTAMP
+ .fail(
+ VARIANT(),
+
Variant.newBuilder().of(Instant.ofEpochSecond(1_600_000_000L)),
+ TableRuntimeException.class),
+ // A variant keeps microseconds, so fractional seconds beyond
the target precision
+ // are truncated, matching a regular cast into a narrower
TIMESTAMP.
+ CastTestSpecBuilder.testCastTo(TIMESTAMP(3))
+ .fromCase(
+ VARIANT(),
+ Variant.newBuilder().of(LocalDateTime.of(2020,
1, 1, 12, 0, 0)),
+ TimestampData.fromLocalDateTime(
+ LocalDateTime.of(2020, 1, 1, 12, 0,
0)))
+ .fromCase(
+ VARIANT(),
+ Variant.newBuilder()
+ .of(LocalDateTime.of(2020, 1, 1, 12,
0, 0, 123000000)),
+ TimestampData.fromLocalDateTime(
+ LocalDateTime.of(2020, 1, 1, 12, 0, 0,
123000000)))
+ .fromCase(
+ VARIANT(),
+ Variant.newBuilder()
+ .of(LocalDateTime.of(2020, 1, 1, 12,
0, 0, 123456000)),
+ TimestampData.fromLocalDateTime(
+ LocalDateTime.of(2020, 1, 1, 12, 0, 0,
123000000))),
+ CastTestSpecBuilder.testCastTo(TIMESTAMP(0))
+ .fromCase(
+ VARIANT(),
+ Variant.newBuilder()
+ .of(LocalDateTime.of(2020, 1, 1, 12,
0, 0, 123000000)),
+ TimestampData.fromLocalDateTime(
+ LocalDateTime.of(2020, 1, 1, 12, 0,
0))),
CastTestSpecBuilder.testCastTo(TIMESTAMP_LTZ())
.fromCase(
VARIANT(),
Variant.newBuilder().of(Instant.ofEpochSecond(1_600_000_000L)),
TimestampData.fromInstant(Instant.ofEpochSecond(1_600_000_000L)))
- .fail(VARIANT(), Variant.newBuilder().of(1),
TableRuntimeException.class));
+ .fail(VARIANT(), Variant.newBuilder().of(1),
TableRuntimeException.class)
+ // a TIMESTAMP is not read as a TIMESTAMP_LTZ either
+ .fail(
+ VARIANT(),
+ Variant.newBuilder().of(LocalDateTime.of(2020,
1, 1, 12, 0, 0)),
+ TableRuntimeException.class),
+ CastTestSpecBuilder.testCastTo(TIMESTAMP_LTZ(3))
+ .fromCase(
+ VARIANT(),
+ Variant.newBuilder()
+
.of(Instant.ofEpochSecond(1_600_000_000L, 123456000)),
+ TimestampData.fromInstant(
+ Instant.ofEpochSecond(1_600_000_000L,
123000000))),
+ // A binary target pads a shorter value and truncates a longer
one, matching a
+ // regular cast into the same type.
+ CastTestSpecBuilder.testCastTo(BINARY(4))
+ .fromCase(
+ VARIANT(),
+ Variant.newBuilder().of(new byte[] {1, 2, 3,
4}),
+ new byte[] {1, 2, 3, 4})
+ .fromCase(
+ VARIANT(),
+ Variant.newBuilder().of(new byte[] {1, 2}),
+ new byte[] {1, 2, 0, 0})
+ .fromCase(
+ VARIANT(),
+ Variant.newBuilder().of(new byte[] {1, 2, 3,
4, 5, 6}),
+ new byte[] {1, 2, 3, 4}),
+ CastTestSpecBuilder.testCastTo(VARBINARY(4))
+ // a variable width target is trimmed but never padded
+ .fromCase(
+ VARIANT(),
+ Variant.newBuilder().of(new byte[] {1, 2}),
+ new byte[] {1, 2})
+ .fromCase(
+ VARIANT(),
+ Variant.newBuilder().of(new byte[] {1, 2, 3,
4, 5, 6}),
+ new byte[] {1, 2, 3, 4}));
}
@TestFactory
diff --git
a/flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/VariantCastUtils.java
b/flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/VariantCastUtils.java
new file mode 100644
index 00000000000..89f112348fe
--- /dev/null
+++
b/flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/VariantCastUtils.java
@@ -0,0 +1,369 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.functions;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.api.TableRuntimeException;
+import org.apache.flink.table.data.DecimalData;
+import org.apache.flink.table.data.TimestampData;
+import org.apache.flink.table.data.binary.BinaryStringData;
+import org.apache.flink.table.data.binary.BinaryStringDataUtil;
+import org.apache.flink.table.data.binary.StringUtf8Utils;
+import org.apache.flink.table.utils.DateTimeUtils;
+import org.apache.flink.types.variant.Variant;
+
+import java.math.BigDecimal;
+import java.math.RoundingMode;
+import java.nio.charset.StandardCharsets;
+import java.util.Arrays;
+import java.util.TimeZone;
+
+/**
+ * Runtime helpers for casting a {@code VARIANT} value to a SQL type.
+ *
+ * <p>A cast never reinterprets the stored value: one kind is not read as
another, and a numeric
+ * value is never wrapped or rounded to make it fit. Any numeric kind
therefore reaches an integer
+ * target as long as the value is integral and in range. {@code FLOAT} and
{@code DOUBLE} are the
+ * exception to exactness: they are approximate by definition, so they accept
any numeric kind and
+ * reject only a magnitude they cannot represent at all.
+ *
+ * <p>A length or a precision is not part of the value in that sense, so it
follows the rules of a
+ * regular cast into the same type. A value longer than the target is trimmed,
fractional seconds
+ * beyond the target precision are truncated, and the fixed width targets
{@code CHAR(n)} and {@code
+ * BINARY(n)} pad a shorter value.
+ */
+@Internal
+public final class VariantCastUtils {
+
+ /**
+ * The magnitude 2^63, the exclusive bound for a {@code double} that still
fits a {@code long}.
+ * Taken from {@link Long#MIN_VALUE} because that is exactly -2^63,
whereas widening {@link
+ * Long#MAX_VALUE} would reach the same number only by rounding up.
+ */
+ private static final double LONG_MAGNITUDE_LIMIT = -(double)
Long.MIN_VALUE;
+
+ /** A variant stores a timestamp with microsecond precision. */
+ private static final int TIMESTAMP_PRECISION = 6;
+
+ private VariantCastUtils() {}
+
+ /**
+ * Reads a numeric variant as a {@code long} and checks it against the
target range. An
+ * approximate or decimal value is accepted only when it is already
integral, so nothing is
+ * rounded away.
+ */
+ public static long toIntegral(Variant variant, long min, long max, String
targetType) {
+ final long value;
+ switch (variant.getType()) {
+ case TINYINT:
+ case SMALLINT:
+ case INT:
+ case BIGINT:
+ value = ((Number) variant.get()).longValue();
+ break;
+ case FLOAT:
+ case DOUBLE:
+ final double approximate = ((Number)
variant.get()).doubleValue();
+ // Below 2^63 the narrowing conversion stays exact. The
comparison is negated so
+ // that NaN fails it too.
+ if (!(Math.abs(approximate) < LONG_MAGNITUDE_LIMIT)) {
+ throw overflow(approximate, targetType);
+ }
+ value = (long) approximate;
+ if (value != approximate) {
+ throw lossyCast(approximate, targetType);
+ }
+ break;
+ case DECIMAL:
+ final BigDecimal decimal = variant.getDecimal();
+ final BigDecimal integral;
+ try {
+ // UNNECESSARY throws unless the value is already integral.
+ integral = decimal.setScale(0, RoundingMode.UNNECESSARY);
+ } catch (ArithmeticException e) {
+ throw lossyCast(decimal, targetType);
+ }
+ try {
+ // longValueExact rejects a value that does not fit a long
instead of returning
+ // its low-order bits.
+ value = integral.longValueExact();
+ } catch (ArithmeticException e) {
+ throw overflow(decimal, targetType);
+ }
+ break;
+ default:
+ throw unsupportedKind(variant, targetType);
+ }
+ if (value < min || value > max) {
+ throw overflow(value, targetType);
+ }
+ return value;
+ }
+
+ /**
+ * Reads any numeric variant as a {@code float}. Dropping decimal digits
is expected of an
+ * approximate type, but a magnitude outside the {@code FLOAT} range is
rejected.
+ */
+ public static float toFloat(Variant variant) {
+ final float value = numeric(variant, "FLOAT").floatValue();
+ if (!Float.isFinite(value)) {
+ throw overflow(variant.get(), "FLOAT");
+ }
+ return value;
+ }
+
+ /** Reads any numeric variant as a {@code double}. See {@link
#toFloat(Variant)}. */
+ public static double toDouble(Variant variant) {
+ final double value = numeric(variant, "DOUBLE").doubleValue();
+ if (!Double.isFinite(value)) {
+ throw overflow(variant.get(), "DOUBLE");
+ }
+ return value;
+ }
+
+ /**
+ * Reads an integer or decimal variant as the target {@code DECIMAL}. The
value has to fit the
+ * precision and scale without rounding, although trailing zeros may be
appended to reach the
+ * scale.
+ */
+ public static DecimalData toDecimal(Variant variant, int precision, int
scale) {
+ final BigDecimal value;
+ switch (variant.getType()) {
+ case TINYINT:
+ case SMALLINT:
+ case INT:
+ case BIGINT:
+ value = BigDecimal.valueOf(((Number)
variant.get()).longValue());
+ break;
+ case DECIMAL:
+ value = variant.getDecimal();
+ break;
+ default:
+ throw unsupportedKind(variant, decimalTarget(precision,
scale));
+ }
+ // The integral part must fit the digits the target reserves for it.
+ if (value.precision() - value.scale() > precision - scale) {
+ throw overflow(value, decimalTarget(precision, scale));
+ }
+ final BigDecimal rescaled;
+ try {
+ // UNNECESSARY throws unless the value fits the target scale
exactly.
+ rescaled = value.setScale(scale, RoundingMode.UNNECESSARY);
+ } catch (ArithmeticException e) {
+ throw lossyCast(value, decimalTarget(precision, scale));
+ }
+ final DecimalData decimal = DecimalData.fromBigDecimal(rescaled,
precision, scale);
+ if (decimal == null) {
+ throw overflow(value, decimalTarget(precision, scale));
+ }
+ return decimal;
+ }
+
+ private static String decimalTarget(int precision, int scale) {
+ return String.format("DECIMAL(%d, %d)", precision, scale);
+ }
+
+ /**
+ * Reads a timestamp variant as the target {@code TIMESTAMP}. A variant
keeps microseconds, so
+ * fractional seconds beyond the target precision are truncated, the same
as a regular {@code
+ * TIMESTAMP} to {@code TIMESTAMP(p)} cast.
+ */
+ public static TimestampData toTimestamp(Variant variant, int precision) {
+ if (variant.getType() != Variant.Type.TIMESTAMP) {
+ throw unsupportedKind(variant, String.format("TIMESTAMP(%d)",
precision));
+ }
+ return DateTimeUtils.truncate(
+ TimestampData.fromLocalDateTime(variant.getDateTime()),
precision);
+ }
+
+ /** Reads a timestamp with local time zone variant. See {@link
#toTimestamp(Variant, int)}. */
+ public static TimestampData toTimestampLtz(Variant variant, int precision)
{
+ if (variant.getType() != Variant.Type.TIMESTAMP_LTZ) {
+ throw unsupportedKind(variant, String.format("TIMESTAMP_LTZ(%d)",
precision));
+ }
+ return
DateTimeUtils.truncate(TimestampData.fromInstant(variant.getInstant()),
precision);
+ }
+
+ /**
+ * Reads a binary variant as the target binary type. {@code BINARY} is
fixed width, so a shorter
+ * value is padded with zero bytes, and either target truncates a value
longer than {@code
+ * targetLength}. This matches a regular cast into the same type.
+ */
+ public static byte[] toBytes(Variant variant, int targetLength, boolean
fixedLength) {
+ final byte[] value = variant.getBytes();
+ if (fixedLength) {
+ return value.length == targetLength ? value : Arrays.copyOf(value,
targetLength);
+ }
+ return value.length <= targetLength ? value : Arrays.copyOf(value,
targetLength);
+ }
+
+ /**
+ * Casts a scalar {@code VARIANT} to a character string, rendering the
value the way a regular
+ * SQL cast of the stored kind would. A value longer than {@code
targetLength} is trimmed, and a
+ * {@code CHAR} target pads a shorter value to its fixed width. Both are
measured in code
+ * points, so a character outside the BMP fills a single position even
though it occupies two
+ * UTF-16 units.
+ *
+ * <p>A stored binary value has to be well-formed UTF-8, since a character
string cannot carry
+ * bytes that no character maps to. Invalid input is rejected rather than
decoded into {@code
+ * U+FFFD}, which would silently substitute a character the value never
held.
+ *
+ * @param sessionZone the session time zone, applied to a {@code
TIMESTAMP_LTZ} value
+ */
+ public static BinaryStringData toStringValue(
+ Variant variant, TimeZone sessionZone, int targetLength, boolean
charTarget) {
+ final String value = getVariantTypeAsString(variant, sessionZone,
targetLength, charTarget);
+ // numChars and substring both count code points, so a character
outside the BMP fills one
+ // position rather than the two UTF-16 units it occupies.
+ final BinaryStringData result = BinaryStringData.fromString(value);
+ final int length = result.numChars();
+ if (length > targetLength) {
+ return result.substring(0, targetLength);
+ }
+ if (charTarget && length < targetLength) {
+ return BinaryStringDataUtil.concat(
+ result, BinaryStringData.blankString(targetLength -
length));
+ }
+ return result;
+ }
+
+ private static String getVariantTypeAsString(
+ final Variant variant,
+ final TimeZone sessionZone,
+ final int targetLength,
+ final boolean charTarget) {
+ final String value;
+ switch (variant.getType()) {
+ case BOOLEAN:
+ value = variant.getBoolean() ? "TRUE" : "FALSE";
+ break;
+ case TINYINT:
+ case SMALLINT:
+ case INT:
+ case BIGINT:
+ case FLOAT:
+ case DOUBLE:
+ value = variant.get().toString();
+ break;
+ case DECIMAL:
+ // toPlainString rather than toString, so that a small scale
is not rendered in
+ // scientific notation, matching a regular DECIMAL to string
cast.
+ value = variant.getDecimal().toPlainString();
+ break;
+ case STRING:
+ value = variant.getString();
+ break;
+ case BYTES:
+ // SQL reads a binary value as UTF-8, the same as a regular
BINARY to string cast.
+ final byte[] utf8 = variant.getBytes();
+ final int invalidAt =
+ StringUtf8Utils.firstInvalidUtf8ByteIndex(utf8, 0,
utf8.length);
+ if (invalidAt >= 0) {
+ throw new TableRuntimeException(
+ String.format(
+ "Cannot cast the VARIANT binary value to
%s because it is not "
+ + "valid UTF-8; the first invalid
byte is at index %d "
+ + "of %d. Cast to BYTES to inspect
the raw value, or "
+ + "wrap that in MAKE_VALID_UTF8 to
replace every "
+ + "invalid byte with the U+FFFD
replacement character.",
+ characterTarget(targetLength, charTarget),
+ invalidAt,
+ utf8.length));
+ }
+ value = new String(utf8, StandardCharsets.UTF_8);
+ break;
+ case DATE:
+ value = DateTimeUtils.formatDate((int)
variant.getDate().toEpochDay());
+ break;
+ case TIMESTAMP:
+ // A wall-clock value needs no zone shift, which is what
UTC_ZONE achieves here. A
+ // variant keeps microseconds, so the precision is always 6.
+ value =
+ DateTimeUtils.formatTimestamp(
+
TimestampData.fromLocalDateTime(variant.getDateTime()),
+ DateTimeUtils.UTC_ZONE,
+ TIMESTAMP_PRECISION);
+ break;
+ case TIMESTAMP_LTZ:
+ value =
+ DateTimeUtils.formatTimestamp(
+
TimestampData.fromInstant(variant.getInstant()),
+ sessionZone,
+ TIMESTAMP_PRECISION);
+ break;
+ case NULL:
+ // Only reachable for a NOT NULL target. A nullable target
maps a null-valued
+ // variant to SQL NULL before this method is called.
+ throw new TableRuntimeException(
+ String.format(
+ "Cannot cast a VARIANT null value to %s
because the target does not "
+ + "accept NULL.",
+ characterTarget(targetLength, charTarget)));
+ default:
+ // An object or array has no scalar rendering.
+ throw new TableRuntimeException(
+ String.format(
+ "Cannot cast a VARIANT %s value to a character
string. Use the "
+ + "JSON_STRING function to obtain its
JSON representation.",
+ variant.getType()));
+ }
+ return value;
+ }
+
+ private static String characterTarget(int targetLength, boolean
charTarget) {
+ return String.format("%s(%d)", charTarget ? "CHAR" : "VARCHAR",
targetLength);
+ }
+
+ private static Number numeric(Variant variant, String targetType) {
+ switch (variant.getType()) {
+ case TINYINT:
+ case SMALLINT:
+ case INT:
+ case BIGINT:
+ case FLOAT:
+ case DOUBLE:
+ case DECIMAL:
+ return (Number) variant.get();
+ default:
+ throw unsupportedKind(variant, targetType);
+ }
+ }
+
+ private static TableRuntimeException unsupportedKind(Variant variant,
String targetType) {
+ return new TableRuntimeException(
+ String.format(
+ "Cannot cast a VARIANT %s value to %s. A VARIANT cast
does not change the "
+ + "type of the stored value, so cast it to its
own type first and "
+ + "then convert with a regular cast.",
+ variant.getType(), targetType));
+ }
+
+ private static TableRuntimeException overflow(Object value, String
targetType) {
+ return new TableRuntimeException(
+ String.format("Casting the VARIANT value %s to %s
overflowed.", value, targetType));
+ }
+
+ private static TableRuntimeException lossyCast(Object value, String
targetType) {
+ return new TableRuntimeException(
+ String.format(
+ "Casting the VARIANT value %s to %s would lose
precision. Cast it to a type "
+ + "that holds the value exactly first, then
narrow if needed.",
+ value, targetType));
+ }
+}
