Jackie-Jiang opened a new pull request, #20154: URL: https://github.com/apache/pinot/pull/20154
## Summary `HyperLogLogPlusUtils.byteSize(p, sp)` returned `sizeof() + 12` as the serialized size of a stream-lib `HyperLogLogPlus`, which matches neither serialized format: - **Normal format** (`sp == 0`): the header is a 4-byte version plus varints of `p`, `sp`, the format flag and the register set size, so the real size is 3 bytes smaller (10933 vs 10936 for `p = 14`). Ingestion aggregation stores the sketch in a `FixedByteSVMutableForwardIndex` whose `setBytes` requires the value length to equal the fixed width, so HLL++ ingestion aggregation failed on the first row. - **Sparse format** (`sp > 0`): the size grows with the sparse set until it converts to the normal format at `0.75 * 2^p` entries, and is unrelated to `sizeof()`. For `p = 14, sp = 25`, 5000 values already serialize to 11412 bytes, above the reported 10936. The star-tree passes this size as the max entry length of the raw forward index, which the V2/V3 writers use to allocate their chunk buffer. Changes: - Rename `byteSize(int, int)` to `maxByteSize(int, int)` and make it a true upper bound: the exact normal-format size for `sp == 0`, and the largest sparse sketch (one varint of up to 5 bytes per entry, up to the conversion threshold) for `sp > 0`. - Remove the unused `byteSize(HyperLogLogPlus)`. stream-lib does not expose `sp` or the format, so an accurate per-value size would require serializing the sketch. - Reject `sp > 25` in `DISTINCTCOUNTHLLPLUS` (and its raw / MV variants), `DISTINCTCOUNTSMARTHLLPLUS` and the HLL++ value aggregator. stream-lib accepts `sp` up to 32, but its sparse encoding packs `sp + 7` bits into an `int`, so a larger `sp` corrupts the registers once the sketch converts to the normal format (e.g. `p = 14`, 5000 values: `sp = 26` estimates 4235 and `sp = 32` estimates 129, vs 4974 for `sp <= 25`). - Validate star-tree `aggregationConfigs` function parameters at table config validation by creating the value aggregator the same way the star-tree builder does, so invalid parameters fail config validation instead of segment creation. ## Backward incompatibility Queries and table configs with an HLL++ `sp` above 25 are now rejected instead of silently returning wrong distinct counts. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
