wangxiaojing created KYLIN-4342: ----------------------------------- Summary: Build Global Dict by MR/Hive New Version Key: KYLIN-4342 URL: https://issues.apache.org/jira/browse/KYLIN-4342 Project: Kylin Issue Type: Improvement Affects Versions: Future Reporter: wangxiaojing Assignee: wangxiaojing
At present, there are two limitations in the implementation of global dictionary through MR/Hive: 1. Limited by Hive order by global sorting on the shuffle stage, the memory and build time becomes uncontrollable with data volume reaching billion level. We have tested the base of 800 million level to configure 15g memory, and the build time of build dictionary needs more than 10 hours; 2. Multi global dictionary columns is calculated serially. -- This message was sent by Atlassian Jira (v8.3.4#803005)