raghav-reglobe opened a new issue, #69220:
URL: https://github.com/apache/doris/issues/69220
**Search before asking:** yes — the FAQ entry (website PR 3669) and docs PR
4170 describe today's behavior; PR 63322
(V1 parse path) tried to fix it and was closed when the V2 parser landed (PR
66709, 68232). No open issue exists.
**Description**
With a schema template that declares a DECIMAL path, a JSON *number* on that
path still goes through `double`
before the template is applied, so digits beyond binary64 are lost; a JSON
*string* on the same path is exact.
```sql
CREATE TABLE t (id INT, v VARIANT<MATCH_NAME 'money.amount':
DECIMAL(38,18)>) ...;
-- Stream Load, format json:
{"id":1,"v":{"money":{"amount":0.123456789012345678}}} -- stored
0.123456789012345680
{"id":2,"v":{"money":{"amount":"0.123456789012345678"}}} -- stored
0.123456789012345678
```
The regression suite pins the lossy result (variant_p0/v2/predefine: input
`188118222.011121933`, expected output
`188118222.011121920`). The FAQ documents the workaround ("write it as a
string"). That works when you control the
producer; it does not when the JSON comes from a system that emits numbers
(MySQL `json`, most APIs).
**Why it happens (master, V2 ingestion):** `JsonStringToVariantEncoder`
parses with the simdjson DOM collector, which
stores `getDouble()` on the tape; the token text is gone before anything
knows the destination type. The template only
reaches the BE as `TColumn.children_column` on the *table* column and is
applied at segment flush
(`variant_shredder` → `cast_column` Float64 → Decimal). `TScalarType` /
`DataTypeVariantV2` carry no fields, so neither
the Stream Load serde nor `parse_to_variant` can know a path is DECIMAL.
**Proposal (what we run on a fork of master, FE + BE):**
1. thrift: `TScalarType` gets an optional `list<TVariantDecimalPath
{pattern, is_glob, precision, scale}>`; an old BE
ignores it, a new BE without it keeps today's behavior.
2. FE: `VariantType.toThrift` fills it from the predefined DECIMAL /
ARRAY<DECIMAL> fields. For `INSERT … SELECT
parse_to_variant(text)` the query's project emits the call as a slot and
the sink casts that slot; the two projects
merge in the rewrite phase, where `SimplifyCastRule` rewrites
`CAST(parse_to_variant(x) AS variant<template>)` into
the call typed with the template (both parse functions). Stream Load
already types the call with the column type.
3. BE: `DataTypeVariantV2` carries the path set; the serde and
`parse_to_variant` hand it to the encoder. Before the
(unchanged) DOM parse, a single text pass quotes every `-?digits.digits`
token (no exponent) that sits on a declared
DECIMAL path, so the value takes the string route the parser already has
and the shredder's cast is exact. Integers,
exponent forms and undeclared paths are untouched; overflow still yields
NULL; malformed text falls through to the
parser's own verdict. Untyped paths keep DOUBLE, so the least-supertype
behavior (DOUBLE + DECIMAL → JSONB) is not
disturbed.
Cost: one extra scan of the row text only for rows that contain a fractional
number on a declared DECIMAL path;
rows without one are parsed from the original buffer.
**Visible change:** the pinned expectation above becomes the exact value;
the FAQ paragraph would say the number form
is exact on a declared DECIMAL path.
**Question for the maintainers:** is this direction acceptable for V2 (an
ondemand-based collector that keeps the raw
token would be the alternative to the text pass)? If so I will open the PR
with the regression and docs changes.
**Verified:** FE unit tests (thrift paths, cast rule), BE unit tests (path
matching + quoting, serde and parse-function
round trips), and a live battery on a two-FE / seven-BE cluster: Stream Load
JSON and CSV exact, INSERT … SELECT exact
(plain, UNION ALL, CTE and LIMIT-subquery shapes), `CAST(CAST(text AS JSON)
AS VARIANT)` unchanged (JSONB route, out of
scope).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
