gianm commented on issue #12262: URL: https://github.com/apache/druid/issues/12262#issuecomment-1175423688
Next patch (frame format): https://github.com/apache/druid/pull/12745 @nishantmonu51 this is responsive to your question earlier! I thought about using Arrow, but didn't end up using it for a couple reasons: - We need both row-based and columnar-based formats: row-based is faster for sorting operations, which is an important part of the processing we need to do. (We can sort using direct memory comparison, no deserialization.) And I think it's important that the formats be linked. - I want to hook the columnar format pretty closely into Druid's query stack, and this is more straightforward to do with our own format. Example: propagating dictionary codes from segments to frames, avoiding copies whenever possible, etc. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
