[
https://issues.apache.org/jira/browse/KYLIN-1323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15292640#comment-15292640
]
kangkaisen commented on KYLIN-1323:
-----------------------------------
That's OK. I looked your commit of "a bug in hbase read-write separation mode"
for github-master branch.
You may be missing another configuration in
" Configuration conf = HadoopUtil.getCurrentConfiguration();
SequenceFile.Writer hfilePartitionWriter =
SequenceFile.createWriter(conf, SequenceFile.Writer.file(hfilePartitionFile),
SequenceFile.Writer.keyClass(ImmutableBytesWritable.class),
SequenceFile.Writer.valueClass(NullWritable.class));".
> Improve performance of converting data to hfile
> -----------------------------------------------
>
> Key: KYLIN-1323
> URL: https://issues.apache.org/jira/browse/KYLIN-1323
> Project: Kylin
> Issue Type: Improvement
> Components: Job Engine
> Affects Versions: v1.2
> Reporter: Yerui Sun
> Assignee: Yerui Sun
> Fix For: v1.4.0, v1.3.0, v1.5.2
>
> Attachments: KYLIN-1323-1.x-staging.2.patch,
> KYLIN-1323-1.x-staging.patch, KYLIN-1323-2.x-staging.2.patch
>
>
> Supposed that we got 100GB data after cuboid building, and with setting that
> 10GB per region. For now, 10 split keys was calculated, and 10 region
> created, 10 reducer used in ‘convert to hfile’ step.
> With optimization, we could calculate 100 (or more) split keys, and use all
> them in ‘covert to file’ step, but sampled 10 keys in them to create regions.
> The result is still 10 region created, but 100 reducer used in ‘convert to
> file’ step. Of course, the hfile created is also 100, and load 10 files per
> region. That’s should be fine, doesn’t affect the query performance
> dramatically.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
