[ https://issues.apache.org/jira/browse/KYLIN-1323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15149804#comment-15149804 ]
Yerui Sun commented on KYLIN-1323: ---------------------------------- [~Shaofengshi], [~mahongbin], thanks for your comment. It's indeed better to expose user with the config like 'how large a hfile is', instead of 'how many hfiles in one region', according to shaofeng's advise. However, the config 'kylin.job.mapreduce.default.reduce.input.mb' is not proper for hfile, we could introduce a config 'kylin.hbase.hfile.size' instead of 'kylin.hbase.hfile.per.region'. How do you think about this? If we agreed on this, I'll update the patch later. > Improve performance of converting data to hfile > ----------------------------------------------- > > Key: KYLIN-1323 > URL: https://issues.apache.org/jira/browse/KYLIN-1323 > Project: Kylin > Issue Type: Improvement > Components: Job Engine > Affects Versions: v1.2 > Reporter: Yerui Sun > Assignee: Yerui Sun > Fix For: v2.0, v1.3 > > Attachments: KYLIN-1323-1.x-staging.patch > > > Supposed that we got 100GB data after cuboid building, and with setting that > 10GB per region. For now, 10 split keys was calculated, and 10 region > created, 10 reducer used in ‘convert to hfile’ step. > With optimization, we could calculate 100 (or more) split keys, and use all > them in ‘covert to file’ step, but sampled 10 keys in them to create regions. > The result is still 10 region created, but 100 reducer used in ‘convert to > file’ step. Of course, the hfile created is also 100, and load 10 files per > region. That’s should be fine, doesn’t affect the query performance > dramatically. -- This message was sent by Atlassian JIRA (v6.3.4#6332)