xiangfu0 opened a new pull request, #19247:
URL: https://github.com/apache/pinot/pull/19247
## Summary
- Parse SQL binary literals such as `X'0102'` as `BYTES` and construct
`BYTES_ARRAY` values in both query engines.
- Support bare literal projection, literal composition, and predicates that
compare an ingested multi-value BYTES column with a BYTES array literal.
- Normalize the multi-stage runtime representation from `byte[][]` to
`ByteArray[]` before block serialization.
- Add focused parser, transform, broker, planner-serde, and runtime tests
plus an ingestion/query E2E test covering dictionary and raw forward indexes.
## Usage
Declare a stored multi-value BYTES column with the existing Pinot schema
representation:
```json
{
"name": "byte_values",
"dataType": "BYTES",
"singleValueField": false
}
```
Construct a BYTES array with standard SQL hex binary literals:
```sql
SELECT ARRAY[X'00', X'0102', X'FF'] AS byte_values;
```
Use the literal with an ingested multi-value BYTES column:
```sql
SELECT id, byte_values
FROM events
WHERE ARRAYS_OVERLAP(byte_values, ARRAY[X'0102', X'CAFE']);
```
Use `X'CAFE'` for bytes rather than the string `'CAFE'`. Query response
values are hex strings, and result metadata reports `BYTES_ARRAY`.
## Compatibility
This change adds no Thrift or protobuf fields. The single-stage engine
retains the existing `arrayValueConstructor` expression with scalar
`binaryValue` operands on the broker/server wire, while the multi-stage engine
uses its existing `BYTES_ARRAY` protobuf representation. This avoids
introducing a new literal union arm that older servers cannot decode.
## Validation
- `BytesMvTypeTest`: 14/14 passed, covering Avro `array<bytes>` ingestion,
dictionary and raw storage, projection, literal-only queries, and
`ARRAYS_OVERLAP` under both query engines.
- `LiteralOnlyBrokerRequestTest`, `RequestUtilsTest`, and
`ArrayLiteralTransformFunctionTest`: passed in a clean focused reactor build.
- `RexExpressionSerDeTest`: 20/20 passed.
- `TransformOperatorTest`: 8/8 passed.
- `spotless:apply`, `checkstyle:check`, `license:format`, and
`license:check`: passed for all six affected modules.
The full `test-compile -Xlint:all` reactor check stops in unchanged
`pinot-segment-local` because `ZstandardDecompressor` cannot resolve
`org.jetbrains.annotations.NotNull` under JDK 25. All affected sources were
compiled by the focused green test runs above, and no warning was emitted for
an added line before that baseline failure.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
