gianm opened a new issue, #18340: URL: https://github.com/apache/druid/issues/18340
When `ScanQueryFrameProcessor` delegates work to a realtime server, it sends a `scan` query and writes the response into a frame. Their serialization when selected in a `scan` query is done by Jackson, and it is common for such objects to have "one-way" serialization (they can be serialized using Jackson, but cannot be deserialized). For aggregating queries, typically these objects are deserialized by the associated aggregator's `AggregatorFactory#deserialize` method rather than by Jackson. This is problematic for `scan` queries, because they don't use `AggregatorFactory`. This leads to errors when attempting to write the resulting objects to a frame, because they aren't the expected type. To fix it, we need to serialize and deserialize these complex objects in a coherent way. There are a couple options that come to mind, - We could add a new parameter to `scan` that causes it to serialize complex objects using `TypeStrategy` rather than allowing Jackson to handle it. MSQ realtime delegation would use this parameter. Then, when reading, we use the same `TypeStrategy` to deserialize the object. - We could rewrite `scan` queries in MSQ's realtime delegation code to wrap all complex types in `complex_encode_base64`, which would accomplish more or less the same thing- it would use the `TypeStrategy` to serialize the object. This bug affects all MSQ queries that use realtime data and delegate `scan` involving complex types. An example query would be like: ```sql SELECT "hllSketch" FROM tbl WHERE "foo" = 'bar' ``` Queries that involve aggregations on complex types wouldn't be affected. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
