xiangfu0 opened a new pull request, #18756: URL: https://github.com/apache/pinot/pull/18756
## What A `dataType: JSON` column with a JSON index pays a **serialize → re-parse round-trip** per row: `DataTypeTransformer` serializes the parsed `Map` to a string for the forward index, and the mutable JSON index then **re-parses that same string** (`stringToJsonNode` → flatten). For large documents (log messages with stack traces) the re-tokenization dominates ingestion CPU. This caches the already-parsed `Map` on the `GenericRow` and feeds it to the JSON index, which flattens it directly: - `JsonUtils.flattenParsed(Object)` — flatten a parsed `Map`/`List`/`JsonNode` via `valueToTree`, skipping string tokenization. - `MutableJsonIndex.addParsed(Object)` + `MutableJsonIndexImpl` override — index the parsed value directly. - `GenericRow` — transient per-row parsed-value cache (cleared per row; not part of value/equality/copy/serialized state). - `DataTypeTransformer` — caches the `Map` only for JSON columns that have a JSON-family index (computed once at construction). - `MutableSegmentImpl` — feeds the cached `Map` to the index, gated on `supportsParsedValue()`. ## Behavior-preserving - `flattenParsed` produces records identical to the old serialize+reparse path. For non-JSON-native leaf types (e.g. a `BigDecimal`/`Float` placed on a JSON column by a non-JSON RecordReader) `valueToTree` would **not** round-trip identically, so it **falls back to serialize+reparse** — byte-identical to today. Verified by `JsonUtilsTest#testFlattenParsedValueMatchesString` (diverse leaf types) and `JsonIndexTest#testMutableJsonIndexParsedMatchesString` (identical `getMatchingDocIds` via both paths). - `supportsParsedValue()` (default `false`) gates feeding the parsed value: an index that doesn't override `addParsed` keeps getting the already-serialized string, so there's **no extra serialize** / regression. `MutableJsonIndexImpl` opts in. - Detection covers any **JSON-family index** (the `json` index, or a plugin index whose id contains "json"); those benefit once their mutable index overrides `addParsed` + `supportsParsedValue`. ## Performance (`BenchmarkJsonFlatten` / real `MutableJsonIndexImpl`) | message | re-parse path (today) | parsed path (this PR) | gain | |---|---|---|---| | ~330 B | 460 docs/ms | 508 docs/ms | ~1.0x | | ~2.9 KB | 152 docs/ms | 515 docs/ms | 3.4x | | ~8 KB | 56 docs/ms | 400 docs/ms | 6-7x | `addParsed(Map)` is ~constant; `add(String)` degrades with size because it re-tokenizes the document. So big-document (log) tables win 3-7x; tiny JSON is unchanged (which is why the cache is gated on having a JSON index). ## SPI surface New default methods on `MutableJsonIndex` (`addParsed`, `supportsParsedValue`), new public `GenericRow` accessors, new `JsonUtils.flattenParsed` overload — all additive (source/binary compatible). Implementers of `MutableJsonIndex` should note the `add(Object,…)` default dispatch now routes `Map`/`List` to `addParsed` (other types still fail fast). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
