xiangfu0 opened a new pull request, #19169: URL: https://github.com/apache/pinot/pull/19169
## Description Projecting a whole `MAP` column as a string — `SELECT attributes`, `LASTWITHTIME(attributes, ...)` — parses every entry into a `HashMap` and then serializes that map back to JSON, once per row. Both steps are avoidable. The frame already stores each value as the JSON bytes `serializeMap` produced, so values can be copied through verbatim and only keys need quoting. ## Changes - Add `MapUtils#frameToJsonString`, rendering a frame to JSON without Jackson and without materializing the map. - Add a `ForwardIndexReader#getMapAsJsonString` hook and route the `case MAP` branch of `readValuesSV` through it. - Override it in the three readers that hold the map as a frame: `VarByteSVMutableForwardIndex`, `VarByteChunkSVForwardIndexReader`, `VarByteChunkForwardIndexReaderV4`. The default still materializes the map, so a reader holding the map columnar-decomposed is unaffected. ## Output is unchanged Both forward-index write paths — `ForwardIndexCreator#putValue` at segment build and `MutableSegmentImpl` while consuming — frame maps through the key-sorting `serializeMap(Map)`, and nested values are sorted by that same writer. So emitting entries in frame order reproduces exactly what `toString(deserializeMap(frame))` produced. `testFrameToJsonStringMatchesToString` pins that equivalence across scalars, nesting, unicode, and keys that need escaping. ## Performance Isolated JMH (`BenchmarkMapProjection`), JDK 25, 2 forks x 5x1s, `-prof gc`: | entries | shape | before us/op | after us/op | speedup | before B/op | after B/op | |--------:|:------|-------------:|------------:|--------:|------------:|-----------:| | 4 | flat | 0.866 | 0.258 | 3.4x | 4,768 | 768 | | 4 | nested | 1.312 | 0.274 | 4.8x | 7,320 | 848 | | 16 | flat | 3.261 | 1.018 | 3.2x | 16,936 | 2,688 | | 16 | nested | 5.352 | 1.049 | 5.1x | 26,968 | 3,008 | | 64 | flat | 14.111 | 3.885 | 3.6x | 65,656 | 10,464 | | 64 | nested | 24.071 | 4.227 | 5.7x | 104,312 | 11,744 | Nested values cost the old path 71% more than flat ones at 64 entries, because Jackson materializes a container per value. The new path stays within 9% of flat since it never looks inside a value. The allocation that remains is the output buffer and the returned `String`. This is an isolated forward-index measurement, not an end-to-end query latency result. ## Validation - `MapUtilsTest` 25/25 (4 new) - `TableIndexingTest` 506, `DataBlockBuilderTest` 97, `GenericRowSerDeTest` 7, `DataTableSerDeTest` 5, `OpenStructDataTypeTest` 5 — all passing - `spotless:apply`, `license:check`, `checkstyle:check` clean on `pinot-spi`, `pinot-segment-spi`, `pinot-segment-local`, `pinot-perf` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
