[
https://issues.apache.org/jira/browse/KYLIN-2736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16285535#comment-16285535
]
Zhong Yanghong edited comment on KYLIN-2736 at 12/11/17 10:05 AM:
------------------------------------------------------------------
Yes, this will cause CPU over-subscription. However, I think there should be a
way to enable this feature to speed up the process.
The thread number mainly depends on the how many cuboids are there whose
statistics needs to be calculated. The detailed code is as follows:
{code}
private int getStatsThreadNum(int cuboidNum) {
int unitNum =
cubeDesc.getConfig().getCuboidNumberPerStatisticsCalculator();
if (unitNum <= 0) {
logger.warn("config from getCuboidNumberPerStatisticsCalculator() "
+ unitNum + " is should larger than 0");
logger.info("Will use single thread for cuboid statistics
calculation");
return 1;
}
int maxCalculatorNum =
cubeDesc.getConfig().getCuboidStatisticsCalculatorMaxNumber();
int calculatorNum = (cuboidNum - 1) / unitNum + 1;
if (calculatorNum > maxCalculatorNum) {
calculatorNum = maxCalculatorNum;
}
return calculatorNum;
}
{code}
was (Author: yaho):
Yes, this will cause CPU over-subscription. However, I think there should be a
way to enable this feature to speed up the process.
The thread number mainly depends on the how many cuboids are there whose
statistics needs to be calculated. The detailed code is as follows:
{code}
private int getStatsThreadNum(int cuboidNum) {
int unitNum =
cubeDesc.getConfig().getCuboidNumberPerStatisticsCalculator();
if (unitNum <= 0) {
logger.warn("config from getCuboidNumberPerStatisticsCalculator() "
+ unitNum + " is should larger than 0");
logger.info("Will use single thread for cuboid statistics
calculation");
return 1;
}
int calculatorNum = (cuboidNum - 1) / unitNum + 1;
if (calculatorNum >
cubeDesc.getConfig().getCuboidStatisticsCalculatorMaxNumber()) {
calculatorNum =
cubeDesc.getConfig().getCuboidStatisticsCalculatorMaxNumber();
}
return calculatorNum;
}
{code}
> Use multiple threads to calculate HyperLogLogPlusCounter in
> FactDistinctColumnsMapper
> -------------------------------------------------------------------------------------
>
> Key: KYLIN-2736
> URL: https://issues.apache.org/jira/browse/KYLIN-2736
> Project: Kylin
> Issue Type: Improvement
> Reporter: Zhong Yanghong
> Assignee: Zhong Yanghong
> Fix For: v2.3.0
>
> Attachments: APACHE-KYLIN-2736.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
