This is an automated email from the ASF dual-hosted git repository.
twalthr pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/flink.git
The following commit(s) were added to refs/heads/master by this push:
new 44edc3c9694 [FLINK-40580][table] Render VARIANT object/array as SQL
when casting to character string
44edc3c9694 is described below
commit 44edc3c9694c46a45a62d53d20f98fa595e9559a
Author: Ramin Gharib <[email protected]>
AuthorDate: Wed Sep 9 11:25:40 2026 +0200
[FLINK-40580][table] Render VARIANT object/array as SQL when casting to
character string
This closes #29125.
---
docs/content.zh/docs/sql/reference/data-types.md | 11 +--
docs/content/docs/sql/reference/data-types.md | 11 +--
.../functions/casting/VariantToStringCastRule.java | 10 ++-
.../planner/functions/CastFunctionITCase.java | 41 ++++++++--
.../planner/functions/casting/CastRulesTest.java | 31 +++++--
.../table/runtime/functions/VariantCastUtils.java | 94 ++++++++++++++++------
6 files changed, 146 insertions(+), 52 deletions(-)
diff --git a/docs/content.zh/docs/sql/reference/data-types.md
b/docs/content.zh/docs/sql/reference/data-types.md
index 9bc67dd733e..fa20d4c273f 100644
--- a/docs/content.zh/docs/sql/reference/data-types.md
+++ b/docs/content.zh/docs/sql/reference/data-types.md
@@ -1583,10 +1583,11 @@ CAST(CAST(PARSE_JSON('3.9') AS DECIMAL(2, 1)) AS INT)
-- returns 3 (truncate
A cast to a character string renders the value exactly as a regular SQL cast
of the stored kind
would, so a boolean becomes `TRUE`, a timestamp uses the SQL format, a
`TIMESTAMP_LTZ` is shifted into
-the session time zone, and a binary value is read as UTF-8. Only an object or
an array has no such
-rendering. Use `JSON_STRING` for the JSON representation instead, where a
string stays quoted as
-`"foo"` and an object or array is serialized. A variant that stores a JSON
`null` casts to SQL
-`NULL`.
+the session time zone, and a binary value is read as UTF-8. An object or an
array has no scalar
+form, so it renders like a regular `ARRAY` or `MAP` cast to a string: an array
as `[e1, e2]` and an
+object as `{k1=v1, k2=v2}`, with each value rendered by these same rules and a
nested variant null
+shown as `NULL`. A string is never quoted, at any depth. Use `JSON_STRING` for
the JSON form with
+quoted strings. A variant that stores a JSON `null` casts to SQL `NULL`.
A `VARIANT` can also be cast to a constructed target, which imposes a schema
on it. A variant array
casts to `ARRAY<T>`. The variant must be an array, otherwise the cast fails.
Each element is itself a
@@ -1873,7 +1874,7 @@ COALESCE(TRY_CAST('non-number' AS INT), 0) --- 结果返回数字 0 的
INT 格
| `ROW` | Y
| N | N | N | N
| N | N | N | N | N | N | N |
N | N | N | N | N | N | !³
| N | N | N | N |
| `STRUCTURED` | Y
| N | N | N | N
| N | N | N | N | N | N | N |
N | N | N | N | N | N | N
| !³ | N | N | N |
| `RAW` | Y
| ! | N | N | N
| N | N | N | N | N | N | N |
N | N | N | N | N | N | N
| N | Y⁴ | N | N |
-| `VARIANT` | N
| ! | ! | ! | !
| ! | ! | ! | ! | ! | ! | N |
! | ! | N | !³ | N | !³ | !³
| !³ | N | Y | N |
+| `VARIANT` | !
| ! | ! | ! | !
| ! | ! | ! | ! | ! | ! | N |
! | ! | N | !³ | N | !³ | !³
| !³ | N | Y | N |
| `BITMAP` | Y
| Y⁷ | N | N | N
| N | N | N | N | N | N | N |
N | N | N | N | N | N | N
| N | N | N | N |
备注：
diff --git a/docs/content/docs/sql/reference/data-types.md
b/docs/content/docs/sql/reference/data-types.md
index 8e43c2ec12e..e62f1997d29 100644
--- a/docs/content/docs/sql/reference/data-types.md
+++ b/docs/content/docs/sql/reference/data-types.md
@@ -1591,10 +1591,11 @@ CAST(CAST(PARSE_JSON('3.9') AS DECIMAL(2, 1)) AS INT)
-- returns 3 (truncate
A cast to a character string renders the value exactly as a regular SQL cast
of the stored kind
would, so a boolean becomes `TRUE`, a timestamp uses the SQL format, a
`TIMESTAMP_LTZ` is shifted into
-the session time zone, and a binary value is read as UTF-8. Only an object or
an array has no such
-rendering. Use `JSON_STRING` for the JSON representation instead, where a
string stays quoted as
-`"foo"` and an object or array is serialized. A variant that stores a JSON
`null` casts to SQL
-`NULL`.
+the session time zone, and a binary value is read as UTF-8. An object or an
array has no scalar
+form, so it renders like a regular `ARRAY` or `MAP` cast to a string: an array
as `[e1, e2]` and an
+object as `{k1=v1, k2=v2}`, with each value rendered by these same rules and a
nested variant null
+shown as `NULL`. A string is never quoted, at any depth. Use `JSON_STRING` for
the JSON form with
+quoted strings. A variant that stores a JSON `null` casts to SQL `NULL`.
A `VARIANT` can also be cast to a constructed target, which imposes a schema
on it. A variant array
casts to `ARRAY<T>`. The variant must be an array, otherwise the cast fails.
Each element is itself a
@@ -1882,7 +1883,7 @@ The matrix below describes the supported cast pairs,
where "Y" means supported,
| `ROW` | Y
| N | N | N | N
| N | N | N | N | N | N | N |
N | N | N | N | N | N | !³
| N | N | N | N |
| `STRUCTURED` | Y
| N | N | N | N
| N | N | N | N | N | N | N |
N | N | N | N | N | N | N
| !³ | N | N | N |
| `RAW` | Y
| ! | N | N | N
| N | N | N | N | N | N | N |
N | N | N | N | N | N | N
| N | Y⁴ | N | N |
-| `VARIANT` | N
| ! | ! | ! | !
| ! | ! | ! | ! | ! | ! | N |
! | ! | N | !³ | N | !³ | !³
| !³ | N | Y | N |
+| `VARIANT` | !
| ! | ! | ! | !
| ! | ! | ! | ! | ! | ! | N |
! | ! | N | !³ | N | !³ | !³
| !³ | N | Y | N |
| `BITMAP` | Y
| Y⁷ | N | N | N
| N | N | N | N | N | N | N |
N | N | N | N | N | N | N
| N | N | N | N |
Notes:
diff --git
a/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/functions/casting/VariantToStringCastRule.java
b/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/functions/casting/VariantToStringCastRule.java
index eeb9fcec9f6..08cef8e0b3f 100644
---
a/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/functions/casting/VariantToStringCastRule.java
+++
b/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/functions/casting/VariantToStringCastRule.java
@@ -36,7 +36,9 @@ import static
org.apache.flink.table.planner.functions.casting.CastRuleUtils.sta
* <p>Renders the scalar value the way a regular SQL cast of the stored kind
would, so a boolean
* becomes {@code TRUE}, a timestamp uses the SQL format, and a {@code
TIMESTAMP_LTZ} is shifted
* into the session time zone and a binary value is read as UTF-8. A variant
holding an object or
- * array has no scalar rendering and fails; use {@code JSON_STRING} for its
JSON representation.
+ * array has no scalar form, so it renders like a regular {@code MAP} or
{@code ARRAY} to string
+ * cast, an array as {@code [e1, e2]} and an object as {@code {k1=v1, k2=v2}},
with strings unquoted
+ * at every depth. This is a SQL rendering, not JSON; use {@code JSON_STRING}
for the JSON form.
*
* <p>A binary value that is not well-formed UTF-8 fails instead of decoding
to {@code U+FFFD}. Cast
* it to {@code BYTES} to inspect the raw value, or wrap that in {@code
MAKE_VALID_UTF8} to accept
@@ -97,9 +99,9 @@ class VariantToStringCastRule extends
AbstractExpressionCodeGeneratorCastRule<Va
LogicalType inputLogicalType,
LogicalType targetLogicalType) {
if (context.isPrinting()) {
- // Every result has to be displayable, including an object or an
array, which have no
- // scalar rendering and would fail the cast. toJson returns a
String, so it needs the
- // wrap that toStringValue applies itself.
+ // Printing renders every variant as JSON, so a scalar string
shows quoted rather than
+ // extracted as the cast below would. toJson returns a String, so
it needs the wrap that
+ // toStringValue applies itself.
return staticCall(BINARY_STRING_DATA_FROM_STRING(),
methodCall(inputTerm, "toJson"));
}
return staticCall(
diff --git
a/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/functions/CastFunctionITCase.java
b/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/functions/CastFunctionITCase.java
index b73054fbbf1..7dbbc379c67 100644
---
a/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/functions/CastFunctionITCase.java
+++
b/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/functions/CastFunctionITCase.java
@@ -301,21 +301,27 @@ public class CastFunctionITCase extends
BuiltInFunctionTestBase {
"CAST(PARSE_JSON('true') AS STRING)",
"TRUE",
STRING().notNull())
- // An object or array has no scalar value, so the
error points to
- // JSON_STRING.
- .testTableApiRuntimeError(
- lit("[\"a\",
\"b\"]").parseJson().cast(STRING()), "JSON_STRING")
+ // An object or array has no scalar form, so it
renders like a regular ARRAY
+ // or MAP to string cast, with strings unquoted at
every depth.
+ .testResult(
+ lit("[\"a\",
\"b\"]").parseJson().cast(STRING()),
+ "CAST(PARSE_JSON('[\"a\", \"b\"]') AS STRING)",
+ "[a, b]",
+ STRING().notNull())
.testResult(
lit("[\"a\",
\"b\"]").parseJson().tryCast(STRING()),
"TRY_CAST(PARSE_JSON('[\"a\", \"b\"]') AS
STRING)",
- null,
+ "[a, b]",
STRING())
- .testTableApiRuntimeError(
- lit("{\"a\": 1}").parseJson().cast(STRING()),
"JSON_STRING")
+ .testResult(
+ lit("{\"a\": 1}").parseJson().cast(STRING()),
+ "CAST(PARSE_JSON('{\"a\": 1}') AS STRING)",
+ "{a=1}",
+ STRING().notNull())
.testResult(
lit("{\"a\":
1}").parseJson().tryCast(STRING()),
"TRY_CAST(PARSE_JSON('{\"a\": 1}') AS STRING)",
- null,
+ "{a=1}",
STRING())
// A bounded CHAR/VARCHAR target trims a longer value,
and CHAR pads a
// shorter one to its fixed width, the same as a
regular cast into it.
@@ -354,6 +360,25 @@ public class CastFunctionITCase extends
BuiltInFunctionTestBase {
"TRY_CAST(PARSE_JSON('\"ab\"') AS CHAR(5))",
"ab ",
CHAR(5))
+ // A nested variant null renders as NULL and a nested
container renders in
+ // full, with strings unquoted, like a regular ARRAY
or MAP to string cast.
+ .testResult(
+ lit("[\"a\", null,
1]").parseJson().cast(STRING()),
+ "CAST(PARSE_JSON('[\"a\", null, 1]') AS
STRING)",
+ "[a, NULL, 1]",
+ STRING().notNull())
+ .testResult(
+ lit("{\"k\": [\"a\",
\"b\"]}").parseJson().cast(STRING()),
+ "CAST(PARSE_JSON('{\"k\": [\"a\", \"b\"]}') AS
STRING)",
+ "{k=[a, b]}",
+ STRING().notNull())
+ // A container renders in full and is then trimmed to
a bounded target, the
+ // same as any other value longer than the target.
+ .testResult(
+ lit("[1, 2, 3]").parseJson().cast(VARCHAR(5)),
+ "CAST(PARSE_JSON('[1, 2, 3]') AS VARCHAR(5))",
+ "[1, 2",
+ VARCHAR(5).notNull())
// A variant holding a JSON null casts to SQL NULL,
not to the text 'null'.
// The length of that text must not be checked against
the target either.
.testResult(
diff --git
a/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/functions/casting/CastRulesTest.java
b/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/functions/casting/CastRulesTest.java
index 274626557d1..440ff8a5826 100644
---
a/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/functions/casting/CastRulesTest.java
+++
b/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/functions/casting/CastRulesTest.java
@@ -1847,11 +1847,32 @@ class CastRulesTest {
VARIANT(),
VARIANT_BUILDER.of(INVALID_UTF8),
TableRuntimeException.class)
- // an object or an array has no scalar rendering and
fails the cast
- .fail(VARIANT(), VARIANT_ARRAY,
TableRuntimeException.class)
- .fail(VARIANT(), VARIANT_OBJECT,
TableRuntimeException.class)
- // printing is not a cast and cannot fail, so it
renders JSON instead,
- // which leaves a stored string quoted
+ // a byte value nested in a container is rendered
unbounded, so its error
+ // names STRING rather than the container's target;
the same holds for an
+ // object field value
+ .fail(
+ VARIANT(),
+ VARIANT_BUILDER
+ .array()
+ .add(VARIANT_BUILDER.of(INVALID_UTF8))
+ .build(),
+ TableRuntimeException.class,
+ "binary value to STRING")
+ .fail(
+ VARIANT(),
+ VARIANT_BUILDER
+ .object()
+ .add("k",
VARIANT_BUILDER.of(INVALID_UTF8))
+ .build(),
+ TableRuntimeException.class,
+ "binary value to STRING")
+ // an object or an array has no scalar form, so it
renders like a regular
+ // ARRAY or MAP to string cast, with strings unquoted
and a nested null
+ // shown as NULL
+ .fromCase(VARIANT(), VARIANT_ARRAY, fromString("[1,
two, FALSE, NULL]"))
+ .fromCase(VARIANT(), VARIANT_OBJECT,
fromString("{k=[1, 2]}"))
+ // printing renders every variant as JSON instead, so
a nested string is
+ // quoted and a null is the JSON null
.fromCasePrinting(
VARIANT(), VARIANT_ARRAY,
fromString("[1,\"two\",false,null]"))
.fromCasePrinting(VARIANT(), VARIANT_OBJECT,
fromString("{\"k\":[1,2]}"))
diff --git
a/flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/VariantCastUtils.java
b/flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/VariantCastUtils.java
index f4920a4d353..ddd86cdda64 100644
---
a/flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/VariantCastUtils.java
+++
b/flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/VariantCastUtils.java
@@ -263,36 +263,74 @@ public final class VariantCastUtils {
}
/**
- * Casts a scalar {@code VARIANT} to a character string, rendering the
value the way a regular
- * SQL cast of the stored kind would. A value longer than {@code
targetLength} is trimmed, and a
- * {@code CHAR} target pads a shorter value to its fixed width. Both are
measured in code
- * points, so a character outside the BMP fills a single position even
though it occupies two
- * UTF-16 units.
- *
- * <p>A stored binary value has to be well-formed UTF-8, since a character
string cannot carry
- * bytes that no character maps to. Invalid input is rejected rather than
decoded into {@code
- * U+FFFD}, which would silently substitute a character the value never
held.
+ * Casts a {@code VARIANT} to a character string, rendering the stored
kind the way a regular
+ * SQL cast would. An object or array renders like a {@code MAP} or {@code
ARRAY} cast, an array
+ * as {@code [e1, e2]} and an object as {@code {k1=v1, k2=v2}}, with a
nested null shown as
+ * {@code NULL}. Strings are never quoted, at any depth; this is a SQL
rendering, not JSON, so
+ * use {@code JSON_STRING} for the JSON form. A value longer than {@code
targetLength} is
+ * trimmed and a {@code CHAR} target pads a shorter one, both counted in
code points rather than
+ * UTF-16 units. A binary value must be well-formed UTF-8, and is rejected
rather than decoded
+ * into {@code U+FFFD}.
*
* @param sessionZone the session time zone, applied to a {@code
TIMESTAMP_LTZ} value
*/
public static BinaryStringData toStringValue(
Variant variant, TimeZone sessionZone, int targetLength, boolean
charTarget) {
- final String value = getVariantTypeAsString(variant, sessionZone,
targetLength, charTarget);
+ final String value = renderValue(variant, sessionZone, targetLength,
charTarget);
// numChars and substring both count code points, so a character
outside the BMP fills one
// position rather than the two UTF-16 units it occupies.
- final BinaryStringData result = BinaryStringData.fromString(value);
- final int length = result.numChars();
- if (length > targetLength) {
- return result.substring(0, targetLength);
+ return variantKey(value, targetLength, charTarget);
+ }
+
+ /**
+ * Renders a variant as a character string. An array becomes {@code [e1,
e2]} and an object
+ * becomes {@code {k1=v1, k2=v2}}, matching how a regular {@code ARRAY} or
{@code MAP} casts to
+ * a string. Elements and field values recurse through the same rendering,
so a string stays
+ * unquoted at every depth. A scalar renders like a regular cast of its
stored kind.
+ */
+ private static String renderValue(
+ final Variant variant,
+ final TimeZone sessionZone,
+ final int targetLength,
+ final boolean charTarget) {
+ if (variant.isArray()) {
+ final int size = variant.getArraySize();
+ final StringBuilder sb = new StringBuilder();
+ sb.append('[');
+ for (int i = 0; i < size; i++) {
+ if (i > 0) {
+ sb.append(", ");
+ }
+ sb.append(renderElement(variant.getElement(i), sessionZone));
+ }
+ return sb.append(']').toString();
}
- if (charTarget && length < targetLength) {
- return BinaryStringDataUtil.concat(
- result, BinaryStringData.blankString(targetLength -
length));
+ if (variant.isObject()) {
+ final StringBuilder sb = new StringBuilder();
+ sb.append('{');
+ boolean first = true;
+ for (final String fieldName : variant.getFieldNames()) {
+ if (!first) {
+ sb.append(", ");
+ }
+ first = false;
+ sb.append(fieldName)
+ .append('=')
+ .append(renderElement(variant.getField(fieldName),
sessionZone));
+ }
+ return sb.append('}').toString();
}
- return result;
+ return renderScalar(variant, sessionZone, targetLength, charTarget);
}
- private static String getVariantTypeAsString(
+ /** Renders one array element or object field value; a nested null shows
as {@code NULL}. */
+ private static String renderElement(final Variant element, final TimeZone
sessionZone) {
+ return element.isNull()
+ ? "NULL"
+ : renderValue(element, sessionZone, Integer.MAX_VALUE, false);
+ }
+
+ private static String renderScalar(
final Variant variant,
final TimeZone sessionZone,
final int targetLength,
@@ -324,6 +362,7 @@ public final class VariantCastUtils {
final int invalidAt =
StringUtf8Utils.firstInvalidUtf8ByteIndex(utf8, 0,
utf8.length);
if (invalidAt >= 0) {
+ final String targetDescription =
characterTarget(targetLength, charTarget);
throw new TableRuntimeException(
String.format(
"Cannot cast the VARIANT binary value to
%s because it is not "
@@ -331,9 +370,7 @@ public final class VariantCastUtils {
+ "of %d. Cast to BYTES to inspect
the raw value, or "
+ "wrap that in MAKE_VALID_UTF8 to
replace every "
+ "invalid byte with the U+FFFD
replacement character.",
- characterTarget(targetLength, charTarget),
- invalidAt,
- utf8.length));
+ targetDescription, invalidAt,
utf8.length));
}
value = new String(utf8, StandardCharsets.UTF_8);
break;
@@ -359,13 +396,14 @@ public final class VariantCastUtils {
case NULL:
// Only reachable for a NOT NULL target. A nullable target
maps a null-valued
// variant to SQL NULL before this method is called.
+ final String targetDescription = characterTarget(targetLength,
charTarget);
throw new TableRuntimeException(
String.format(
"Cannot cast a VARIANT null value to %s
because the target does not "
+ "accept NULL.",
- characterTarget(targetLength, charTarget)));
+ targetDescription));
default:
- // An object or array has no scalar rendering.
+ // Any remaining kind has no scalar string rendering;
JSON_STRING serializes it.
throw new TableRuntimeException(
String.format(
"Cannot cast a VARIANT %s value to a character
string. Use the "
@@ -376,7 +414,13 @@ public final class VariantCastUtils {
}
private static String characterTarget(int targetLength, boolean
charTarget) {
- return String.format("%s(%d)", charTarget ? "CHAR" : "VARCHAR",
targetLength);
+ if (charTarget) {
+ return String.format("%s(%d)", "CHAR", targetLength);
+ }
+ if (targetLength == Integer.MAX_VALUE) {
+ return "STRING";
+ }
+ return String.format("%s(%d)", "VARCHAR", targetLength);
}
private static Number numeric(Variant variant, String targetType) {
