[ https://issues.apache.org/jira/browse/KYLIN-1323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15171053#comment-15171053 ]
Shaofeng SHI commented on KYLIN-1323: ------------------------------------- I checked the commit, only one minor issue: KylinConfigBase. getHBaseHFileSizeGB() is using 0 as the default value; as we discussed previously, it's better to set it to a non-zero value like 5. Besides, this patch is for v1 engine right? for v2 engine it need a different implementation, that would need another patch. For 2.0-rc, which only accept critical issues, you can skip it as doing change on multiple branches is laborious. > Improve performance of converting data to hfile > ----------------------------------------------- > > Key: KYLIN-1323 > URL: https://issues.apache.org/jira/browse/KYLIN-1323 > Project: Kylin > Issue Type: Improvement > Components: Job Engine > Affects Versions: v1.2 > Reporter: Yerui Sun > Assignee: Yerui Sun > Fix For: v2.0, v1.3 > > Attachments: KYLIN-1323-1.x-staging.2.patch, > KYLIN-1323-1.x-staging.patch, KYLIN-1323-2.x-staging.2.patch > > > Supposed that we got 100GB data after cuboid building, and with setting that > 10GB per region. For now, 10 split keys was calculated, and 10 region > created, 10 reducer used in ‘convert to hfile’ step. > With optimization, we could calculate 100 (or more) split keys, and use all > them in ‘covert to file’ step, but sampled 10 keys in them to create regions. > The result is still 10 region created, but 100 reducer used in ‘convert to > file’ step. Of course, the hfile created is also 100, and load 10 files per > region. That’s should be fine, doesn’t affect the query performance > dramatically. -- This message was sent by Atlassian JIRA (v6.3.4#6332)