morningman opened a new pull request, #67476:
URL: https://github.com/apache/doris/pull/67476
### What problem does this PR solve?
Issue Number: close #67367
Related PR: #65615 (Arrow Flight SQL tracking issue)
Problem Summary:
A `VARIANT` row that holds an empty JSON object carries no payload at all,
so what makes it read back as `{}` lives in the column's shape rather than in
its data. Two read paths lost that and returned a different, valid-looking
value for a persisted `{}`.
**1. Over Arrow Flight a `{}` came back as an empty string.**
```sql
CREATE TABLE t (k INT, vn VARIANT NOT NULL) DUPLICATE KEY(k)
DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES("replication_num" = "1");
INSERT INTO t VALUES (1, '{}'), (2, '{}'), (3, '{}');
INSERT INTO t VALUES (4, '{}'), (5, '{"a" : 1}');
SELECT k, vn FROM t ORDER BY k;
-- MySQL : 1 {} 2 {} 3 {} 4 {} 5 {"a":1}
-- Arrow : 1 '' 2 '' 3 '' 4 {} 5 {"a":1} <-- rows of the first
merged run
```
The Arrow Flight result writer re-materializes every block through
`MutableBlock` (`varrow_flight_result_writer.cpp`), one copy more than the
MySQL writer does, and unlike `insert_indices_from` that copy is not finalized.
An unfinalized `Subcolumn` keeps one part per source range, and
`Subcolumn::insert_range_from` appends a new part instead of rewriting the
earlier ones, so a later typed part promotes only the **column-level** least
common type:
```
lct=Nullable(JSONB) ndip=0 parts=2 num_rows=5
serialize part i=0 ind=0 part_size=3 ptype=Nothing <-- the empty-object
rows
```
The "untyped root serializes as an empty object" rule in
`Subcolumn::serialize_text_json` only consulted that column-level type, so the
rows still sitting in an untyped part fell through to the `Nothing` serde and
rendered as an empty string. The fix applies the same rule per part.
Because the trigger is the block shape (a sort merging rows from more than
one source block), not the connection, the original report looked like a
per-connection or first-read problem. It is neither: the same connection
returns `{}` for a query shape that does not merge runs, and a fresh connection
returns `""` for one that does.
**2. `CAST(VARIANT AS STRING/JSON)` returned SQL `NULL` for the same value**
whenever the column held no path at all — on both protocols:
```sql
-- every row is an empty object
SELECT v, CAST(v AS STRING), CAST(v AS JSON) FROM only_empty; --> {}
NULL NULL
-- the same value in a column that also holds paths
SELECT v, CAST(v AS STRING) FROM mixed WHERE k = 1; --> {} {}
```
Such a column is still a scalar variant — its root simply never got a type —
and `is_scalar_variant()` short-circuited `is_root_valuable`, so the cast took
the scalar-root fast path, found nothing to convert and produced `NULL` for
every row. A `NOT NULL` column returned `NULL` this way too. The fix requires
the root to carry a value, falls back to serializing the tree when no row's
root does, and lets a STRING/JSONB target serialize the tree before the
all-defaults branch turns it into `NULL`. Rows whose root does hold a value
keep the root conversion, which is what unwraps a JSON string into its text.
Both fixes are read-path only; nothing about how a `VARIANT` is stored
changes.
Known remaining gap, deliberately out of scope: in a column that mixes `{}`
with scalar roots (e.g. `123`), `CAST` of the `{}` rows still returns `NULL`.
Closing that needs a row-level substitution in the root fast path, which would
change the existing JSON-string unwrapping contract that
`variant_p0/column_name` depends on.
### Release note
Fix `VARIANT` empty JSON objects being returned as an empty string over
Arrow Flight SQL, and as SQL `NULL` by `CAST(VARIANT AS STRING/JSON)` when the
column holds no path.
### Check List (For Author)
- Test
- [x] Regression test
- [x] Unit Test
- [x] Manual test (add detailed scripts or steps below)
New coverage, both verified to fail without this change and pass with it:
- `be/test/core/column/column_variant_test.cpp` — 5 cases under
`ColumnVariantEmptyObjectTest`.
`empty_object_survives_copy_of_mixed_type_parts` reproduces the exact shape
from the log above and fails on rows 0-2 without the fix. Note for anyone
extending these: the source parts must disagree on type (one `READ_MODE`, one
`WRITE_MODE` finalize) **and** the copy must be left unfinalized, otherwise the
test passes either way.
- `regression-test/suites/variant_p0/test_variant_empty_object_cast.groovy`
— the cast, across the all-empty / with-NULL / mixed shapes, plus an assertion
that both shapes agree.
- `regression-test/suites/arrow_flight_sql_p0/test_select_variant.groovy` —
the sorted multi-block read over Arrow Flight, with a JDBC-vs-Arrow
`assertEquals` so the two protocols must return the same rows.
Manual: reproduced and re-verified end to end against a local single FE +
single BE cluster through both the MySQL protocol and the Python
`adbc_driver_flightsql` client from the original report.
Regression sweep of `variant_p0` + `arrow_flight_sql_p0` (174 suites). The
remaining failures were each confirmed to reproduce on a pristine build of the
same commit without this change: `test_all_prdefine_type_to_sparse` (decimal256
precision), `test_outfile_csv_variant_type` (S3 `curlCode: 43`),
`test_sql_cache_over_arrow_flight` (FE-side `IllegalStateException`),
`test_variant_compaction_with_sparse_limit` (array rendering spacing).
- Behavior changed:
- [x] Yes. A persisted empty JSON object now reads back as `{}` instead
of an empty string over Arrow Flight, and `CAST(VARIANT AS STRING/JSON)` on a
column that holds no path returns `{}` instead of `NULL`. Both make the value
agree with what `SELECT <variant>` and every other shape of the same column
already returned.
- Does this need documentation?
- [x] No.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_01Aju9eFEiRNAJUozsDjE9aj
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
