[GitHub] [hudi] alexeykudinkin commented on issue #6758: [SUPPORT] Will metatable support partitions inside col_stat & files?
alexeykudinkin commented on issue #6758: URL: https://github.com/apache/hudi/issues/6758#issuecomment-1262814782 @Zhangshunyu we're able to do this filtering even w/o physical partitioning (thanks to relying on HFile and elaborate key encoding scheme) -- we only read the records corresponding to files (in case of Column Stats) pertaining to particular partition. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] alexeykudinkin commented on issue #6758: [SUPPORT] Will metatable support partitions inside col_stat & files?
alexeykudinkin commented on issue #6758: URL: https://github.com/apache/hudi/issues/6758#issuecomment-1258624704 @Zhangshunyu as @yihua already called out we approach partition-based filtering differently in MT leveraging HFile, which is (sorted) key-value based format allowing to do faster lookups based on the key's prefix, which allows us to only scan a portion of the Column Stats Index when we look it up for particular partition(s) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
