siddharthteotia commented on issue #8755: URL: https://github.com/apache/pinot/issues/8755#issuecomment-1134952819
hey @Jackie-Jiang - there is another related issue here https://github.com/apache/pinot/issues/7870 I had discussed the point (4) in that issue with @somandal and we are planning to do it in one of the two ways: - Support for general purpose compression on dictionary encoded columns. So, for a dictionary encoded column it is not absolutely necessary to dictionary encode the forward index as well. The raw data in forward index can be compressed using LZ4 / SNAPPY etc just like we would have done for a noDictionary column - The other way we are thinking is to also add support for no forward index. I think both approaches can be implemented and will benefit the case where forward index can be huge (like the MV column example in my issue) and the column is only used in filtering so ok to have the forward index compressed or not have it at all. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
