xiangfu0 commented on PR #17269: URL: https://github.com/apache/pinot/pull/17269#issuecomment-4310054450
@Jackie-Jiang To have both a dictionary-encoded forward index and a RAW forward index on the same column simultaneously would require storing two separate forward indexes on disk, which is not something the current segment format supports and not a use case this PR targets. The use case this PR addresses: a user configures a column as RAW (for storage efficiency) but also wants an inverted or FST index (which require a dictionary). In that case we create a standalone shared dictionary alongside the RAW forward index — but the forward index itself stays raw-encoded. If a user wants a dictionary-encoded forward index, they simply remove the noDictionaryColumns / EncodingType.RAW config and the forward index will be dictionary-encoded as usual, with the dictionary serving both the forward index and secondary indexes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
