[ https://issues.apache.org/jira/browse/KYLIN-2866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16297838#comment-16297838 ]
Zhong Yanghong commented on KYLIN-2866: --------------------------------------- Hi [~lidong], a patch refined for reading statistics files has been uploaded. org.apache.kylin.engine.mr.common.MapReduceUtil#getHLLShardBase is used for getting the reducer number for calculating hll as commented. > Enlarge the reducer number for hyperloglog statistics calculation at step > FactDistinctColumnsJob > ------------------------------------------------------------------------------------------------ > > Key: KYLIN-2866 > URL: https://issues.apache.org/jira/browse/KYLIN-2866 > Project: Kylin > Issue Type: Improvement > Components: Job Engine > Reporter: Zhong Yanghong > Assignee: Zhong Yanghong > Fix For: v2.3.0 > > Attachments: APACHE-KYLIN-2866-refined.patch, APACHE-KYLIN-2866.patch > > > Currently only one reducer is assigned for hll stats calculation, which may > become the bottleneck for slow down this step. Since the stats for different > cuboids will not influence each other, it's better to divide the cuboid set > into several and assign a reduce for each subset. > The strategy of this patch is to assign 100 cuboids into a subset. And > there's a upper limit of reducers for hll stats calculation. Currently it's > 50. -- This message was sent by Atlassian JIRA (v6.4.14#64029)