[jira] [Updated] (KYLIN-2269) Reduce MR memory usage for global dict
[ https://issues.apache.org/jira/browse/KYLIN-2269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kangkaisen updated KYLIN-2269: -- Attachment: KYLIN-2269.patch Update the patch to add the check for shardBy column. > Reduce MR memory usage for global dict > -- > > Key: KYLIN-2269 > URL: https://issues.apache.org/jira/browse/KYLIN-2269 > Project: Kylin > Issue Type: Improvement >Affects Versions: v1.6.0 >Reporter: kangkaisen >Assignee: kangkaisen > Attachments: KYLIN-2269.patch > > > currently, in {{Build Base Cuboid Data}}, if user use the global dict and the > global dict size significantly larger the mapper memory size, the > {{CachedTreeMap}} will load all values as much as possible and the soft > references object will stick around for a while when GC, So which will make > the {{Build Base Cuboid Data}} mapper pause for a long time even could not > finish. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KYLIN-2269) Reduce MR memory usage for global dict
[ https://issues.apache.org/jira/browse/KYLIN-2269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kangkaisen updated KYLIN-2269: -- Attachment: (was: KYLIN-2269.patch) > Reduce MR memory usage for global dict > -- > > Key: KYLIN-2269 > URL: https://issues.apache.org/jira/browse/KYLIN-2269 > Project: Kylin > Issue Type: Improvement >Affects Versions: v1.6.0 >Reporter: kangkaisen >Assignee: kangkaisen > Attachments: KYLIN-2269.patch > > > currently, in {{Build Base Cuboid Data}}, if user use the global dict and the > global dict size significantly larger the mapper memory size, the > {{CachedTreeMap}} will load all values as much as possible and the soft > references object will stick around for a while when GC, So which will make > the {{Build Base Cuboid Data}} mapper pause for a long time even could not > finish. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KYLIN-2269) Reduce MR memory usage for global dict
[ https://issues.apache.org/jira/browse/KYLIN-2269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kangkaisen updated KYLIN-2269: -- Attachment: KYLIN-2269.patch This is the patch. This patch has run a some time in our production env. which could reduce the MR memory usage and mapper running time of Base Cuboid for global dict significantly. > Reduce MR memory usage for global dict > -- > > Key: KYLIN-2269 > URL: https://issues.apache.org/jira/browse/KYLIN-2269 > Project: Kylin > Issue Type: Improvement >Affects Versions: v1.6.0 >Reporter: kangkaisen >Assignee: kangkaisen > Attachments: KYLIN-2269.patch > > > currently, in {{Build Base Cuboid Data}}, if user use the global dict and the > global dict size significantly larger the mapper memory size, the > {{CachedTreeMap}} will load all values as much as possible and the soft > references object will stick around for a while when GC, So which will make > the {{Build Base Cuboid Data}} mapper pause for a long time even could not > finish. -- This message was sent by Atlassian JIRA (v6.3.4#6332)