xiangfu0 opened a new pull request, #18953:
URL: https://github.com/apache/pinot/pull/18953
## Summary
`JSONMessageDecoder` only understood UTF-8 text JSON. This adds a pluggable
payload-format layer so a stream can also carry **PostgreSQL jsonb**, **SQLite
JSONB**, **Smile**, or **CBOR**, selected via a new `jsonFormat` decoder
property.
Every format decodes to the same `Map<String, Object>` value contract
Jackson produces for text JSON, so the existing `JSONRecordExtractor` handles
all of them unchanged.
## Configuration
| `jsonFormat` | Payload |
|---|---|
| *(unset)* / `AUTO` | Detected per message from leading magic/version
bytes; falls back to text |
| `TEXT` | UTF-8 text JSON (historical behavior) |
| `POSTGRES_JSONB` | PostgreSQL `jsonb` wire format |
| `SQLITE_JSONB` | SQLite 3.45+ JSONB |
| `SMILE` | Jackson Smile |
| `CBOR` | CBOR (RFC 8949) |
```
"stream.kafka.decoder.prop.jsonFormat": "SQLITE_JSONB"
```
## Backward compatibility
The default changes from "always text" to `AUTO`. This is
**behavior-preserving for text JSON**: detection is allocation-free and cannot
mis-route a well-formed text document, because a top-level `{` / `[`
(optionally after whitespace) collides with none of the binary signatures —
Smile `3A 29 0A`, CBOR `D9 D9 F7`, Postgres `0x01`+JSON, SQLite object nibble
`0x?C`. Anything unrecognized falls back to text. Operators who want to skip
detection entirely can pin `jsonFormat: TEXT`.
Detection order is Smile → CBOR → Postgres → SQLite → text. No existing
config keys, SPI signatures, or serialization formats change.
## Notes on the two hand-rolled decoders
**PostgreSQL jsonb's wire format is not its on-disk `JsonbContainer`
layout.** `jsonb_send` renders the value with `JsonbToCString` and emits a
version byte followed by *text* JSON; `jsonb_recv` reverses it. Every standard
binary producer — the v3 extended-query protocol, `COPY ... WITH (FORMAT
binary)`, and logical replication via `pgoutput` — routes through that same
send function. So this parser strips the version byte and parses the text body,
which also makes its type contract identical to `TEXT`.
**SQLite JSONB** is a genuine binary format, decoded per
[spec](https://sqlite.org/jsonb.html) including the JSON5 element types
(`INT5`, `FLOAT5`, `TEXTJ`, `TEXT5`, `TEXTRAW`) and all four size descriptors.
Nested elements are bounded by their enclosing container rather than merely by
the payload, so a crafted element cannot overrun its parent and silently
swallow later sibling key/value pairs. `TEXTJ` rejects invalid escapes rather
than passing them through; string unescaping is single-pass with a
no-allocation fast path.
## Dependencies
Adds `jackson-dataformat-smile` and `jackson-dataformat-cbor`. Versions are
managed by the existing `jackson-bom` import (no version overrides). Both are
recorded in `LICENSE-binary`.
## Testing
59 tests pass in the module (34 new). Coverage includes:
- Per-format decoding with **type-strict** assertions (`Integer` / `Long` /
`BigInteger` narrowing tiers, `Double`, and `Float` / `byte[]` passthrough for
the binary formats).
- **Cross-format equivalence** — one logical document decoded through all
five formats yields an equal `Map`.
- **AUTO detection**, including the fallback path and adversarial input
(empty payload, `0xFF`, truncated Smile header, lone `0x01`), and that tag-less
CBOR is *not* auto-detected but still decodes when pinned.
- **Negative / malformed input** — truncated payloads, reserved element
types, non-object top level, unsupported version byte, nested-element container
overrun, invalid `TEXTJ` escape, and end-to-end `decode()` wrapping parse
failures in `RuntimeException`.
- End-to-end `decode()` → `GenericRow` for every format, both pinned and
auto-detected.
## Release notes
Adds the `jsonFormat` decoder property to `JSONMessageDecoder`; unset
behaves as `AUTO` (per-message format detection, text-JSON fallback).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
