[ https://issues.apache.org/jira/browse/KYLIN-1323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15290443#comment-15290443 ]
kangkaisen commented on KYLIN-1323: ----------------------------------- Hi,shaofeng. it may be have a little bug in "saveHFileSplits", when the HDFS cluster of HBase is different from the HDFS cluster for MR job,the Configuration use in "saveHFileSplits" should be "HBaseConnection.getCurrentHBaseConfiguration()", and not "HadoopUtil.getCurrentConfiguration()"。 I have merged the KYLIN-1323 patch in our branch, after I changed the Configuration use in "saveHFileSplits", the cube builded successfully。 > Improve performance of converting data to hfile > ----------------------------------------------- > > Key: KYLIN-1323 > URL: https://issues.apache.org/jira/browse/KYLIN-1323 > Project: Kylin > Issue Type: Improvement > Components: Job Engine > Affects Versions: v1.2 > Reporter: Yerui Sun > Assignee: Yerui Sun > Fix For: v1.4.0, v1.3.0, v1.5.2 > > Attachments: KYLIN-1323-1.x-staging.2.patch, > KYLIN-1323-1.x-staging.patch, KYLIN-1323-2.x-staging.2.patch > > > Supposed that we got 100GB data after cuboid building, and with setting that > 10GB per region. For now, 10 split keys was calculated, and 10 region > created, 10 reducer used in ‘convert to hfile’ step. > With optimization, we could calculate 100 (or more) split keys, and use all > them in ‘covert to file’ step, but sampled 10 keys in them to create regions. > The result is still 10 region created, but 100 reducer used in ‘convert to > file’ step. Of course, the hfile created is also 100, and load 10 files per > region. That’s should be fine, doesn’t affect the query performance > dramatically. -- This message was sent by Atlassian JIRA (v6.3.4#6332)