[ https://issues.apache.org/jira/browse/KYLIN-3905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
XiaoXiang Yu updated KYLIN-3905: -------------------------------- Description: In dev mail list's discussion, I suggest to enable shrunken dictionary by default, and received some dev's aggrement. When using bitmap measure on a large cardinality column(require global dictionaty), build base cuboid step need frequent cache swap so it cannot finished within a reasonable period. When shrunken dictionary enabled, a new step will be added to build separated dictionary for each `InputSplit`, Mapper of **BuildBaseCuboid** step only has to fetch a smaller dictionary for itself, instead of a larger global dictionary. It will reduce cache swap and make **BuildBaseCuboid** step run as quicker as possible. http://mail-archives.apache.org/mod_mbox/kylin-dev//201903.mbox/%3c62efcb72-b235-4fc3-9add-0fc510d97...@kyligence.io%3e was: In dev mail list's discussion, I suggest to enable http://mail-archives.apache.org/mod_mbox/kylin-dev//201903.mbox/%3c62efcb72-b235-4fc3-9add-0fc510d97...@kyligence.io%3e When using bitmap measure on a large cardinality column(require global dictionaty), build base cuboid step need frequent cache swap so it cannot finished within a reasonable period. When shrunken dictionary enabled, a new step will be added to build separated dictionary for each `InputSplit`, Mapper of **BuildBaseCuboid** step only has to fetch a smaller dictionary for itself, instead of a larger global dictionary. It will reduce cache swap and make **BuildBaseCuboid** step run as quicker as possible. > Enable shrunken dictionary default > ---------------------------------- > > Key: KYLIN-3905 > URL: https://issues.apache.org/jira/browse/KYLIN-3905 > Project: Kylin > Issue Type: Improvement > Components: Measure - Count Distinct > Reporter: XiaoXiang Yu > Assignee: XiaoXiang Yu > Priority: Minor > Fix For: v2.6.2 > > Attachments: image-2019-03-25-11-26-59-198.png, > image-2019-03-25-11-27-26-149.png, image-2019-03-25-11-27-46-175.png, > image-2019-03-25-11-28-14-256.png, image-2019-03-25-11-29-19-383.png > > > In dev mail list's discussion, I suggest to enable shrunken dictionary by > default, and received some dev's aggrement. > When using bitmap measure on a large cardinality column(require global > dictionaty), build base cuboid step need frequent cache swap so it cannot > finished within a reasonable period. > When shrunken dictionary enabled, a new step will be added to build > separated dictionary for each `InputSplit`, Mapper of **BuildBaseCuboid** > step only has to fetch a smaller dictionary for itself, instead of a larger > global dictionary. It will reduce cache swap and make **BuildBaseCuboid** > step run as quicker as possible. > > http://mail-archives.apache.org/mod_mbox/kylin-dev//201903.mbox/%3c62efcb72-b235-4fc3-9add-0fc510d97...@kyligence.io%3e -- This message was sent by Atlassian JIRA (v7.6.3#76005)