[
https://issues.apache.org/jira/browse/CALCITE-3594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16996969#comment-16996969
]
Shuo Cheng edited comment on CALCITE-3594 at 12/16/19 5:05 AM:
---------------------------------------------------------------
Hi, [~amaliujia] , I'm working on CALCITE-3590; Actually, groupBy hints is a
kind of Aggregate hints and just as what Danny says, hints statement are put
just after SELECT keyword or table ref, and what we need is to support
propagating hints to Aggregate properly.
[~danny0405] , for your cases about hot keys, what if hot keys themself contain
"," , we can not extract hot keys from the string literal correctly. Maybe we
need some structure like MultiValueMap, and define hot keys hint like:
*select /*+ AGG_HOT_KEY(k1='a', k1='b', k1='c') */ ...*
what do you think?
was (Author: icshuo):
Hi, [~amaliujia] , I'm working on CALCITE-3590; Actually, groupBy hints is a
kind of Aggregate hints and just as what Danny says, hints statement are put
just after SELECT keyword or table ref, and what we need is to support
propagating hints to Aggregate properly.
[~danny0405] , for your cases about hot keys, what if hot keys themself contain
"," , we can not extract hot keys from the string literal correctly. May be we
need some structure like MultiValueMap, and define hot keys hint like:
select /*+ AGG_HOT_KEY(k1='a', k1='b', k1='c') */ ...
> Support hot Groupby keys hint
> -----------------------------
>
> Key: CALCITE-3594
> URL: https://issues.apache.org/jira/browse/CALCITE-3594
> Project: Calcite
> Issue Type: Sub-task
> Reporter: Rui Wang
> Assignee: Rui Wang
> Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> It will be useful for Apache Beam if we support the following SqlHint:
> SELECT * FROM t
> GROUP BY t.key_column /* + hot_key(key1=fanout_factor, ...) */)
> The hot key strategy works on aggregation and it provides a list of hot keys
> with fanout factor for a column. The fanout factor says how many partition
> should be created for that specific key, such that we can have a per
> partition aggregate and then have a final aggregate. One example to explain
> it:
> SELECT * FROM t
> GROUP BY t.key_column /* + hot_key("value1"=2) */)
> // for the key_column, there is a "value1" which appear so many times (so
> it's hot), please consider split it into two partition and process separately.
> Such problem is common for big data processing, where hot key creates slowest
> machine which either slow down the whole pipeline or make retries. In such
> case, one common resolution is to split data to multiple partition and
> aggregate per partition, and then have a final combine.
> Usually execution engine won't know what is the hot key(s). SqlHint provides
> a good way to tell the engine which key is useful to deal with it.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
