GitHub user hbdeshmukh opened a pull request:
https://github.com/apache/incubator-quickstep/pull/121
Support for performing partitioned aggregation.
- Used for creating a pool of hash tables such that each hash table
belongs to a unique partition.
- The partitioning is done on the group-by keys.
- Wrote a utility function to compute composite hash of a group of
TypedValues.
- The Tuple class now supports a method to compute the hash of the entire
tuple (i.e. hash key is the composite key made up of all the
attributes in the tuple).
- Added a check for whether the aggregation is partitioned or not.
- The conditions for whether the aggregation can be partitioned
are as follows:
1. The query has a GROUP BY clause.
2. There are no aggrgeations with a DISTINCT clause.
3. The estimated number of groups are greater than a pre-defined
threshold.
4. The query has at least one aggregation function.
- Method for partitioned aggregation with GROUP BY
- StorageBlock now provides a method for performing GROUP BY aggregation
in a partitioned way.
- AggregationOperationState calls appropriate method (i.e.
aggregateGroupBy or aggregateGroupByPartitioned) based on the way in
which aggregation is being performed.
Performance comparison: TPC-H SF100.
**Query**|**master**|**partitioned aggregation**|**Speedup over master**
:-----:|:-----:|:-----:|:-----:
1|16,046|14,625|1.10
2|5,625|3,316|1.70
3|6,861|7,987|0.86
4|2,662|5,149|0.52
5|4,364|4,586|0.95
6|398|397|1.00
7|23,367|24,600|0.95
8|3,274|3,701|0.88
9|10,050|13,029|0.77
10|15,296|9,502|1.61
11|2,110|931|2.27
12|1,805|3,206|0.56
13|34,220|17,551|1.95
14|771|815|0.95
15|4,435|915|4.84
16|8,661|8,489|1.02
17|160,707|18,885|8.51
18|66,309|27,744|2.39
19|1,475|1,566|0.94
20|55,381|11,429|4.85
21|121,310|127,343|0.95
22|6,792|6,683|1.02
Total|551,921|312,448|1.77
Some queries e.g. 13, 15, 17, 18, 20 see drastic improvement, primarily
because the FinalizeAggregation operation is multi-threaded, compared to the
current single threaded implementation. Overall, we see a 1.77x improvement on
the entire workload.
For some queries we see significant degradation e.g. 4, 9 and 12. I looked
at 9 and 12's individual runs and it seemed mostly an issue of large variance
between the runs. For Q4, I saw a consistent degradation. So far, I couldn't
figure out its reason.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/apache/incubator-quickstep
partitioned-aggregate-new
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-quickstep/pull/121.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #121
----
commit 4045f192e19bdba3fa2faa6f49a47ce2bd7f48a6
Author: Harshad Deshmukh <[email protected]>
Date: 2016-09-21T16:43:39Z
Support for performing partitioned aggregation.
- Used for creating a pool of hash tables such that each hash table
belongs to a unique partition.
- The partitioning is done on the group-by keys.
- Wrote a utility function to compute composite hash of a group of
TypedValues.
- Added a check for whether the aggregation is partitioned or not.
- The conditions for whether the aggregation can be partitioned
are as follows:
1. The query has a GROUP BY clause.
2. There are no aggrgeations with a DISTINCT clause.
3. The estimated number of groups are greater than a pre-defined
threshold.
4. The query has at least one aggregation function.
- Method for partitioned aggregation with GROUP BY
- StorageBlock now provides a method for performing GROUP BY aggregation
in a partitioned way.
- The Tuple class now supports a method to compute the hash of the entire
tuple (i.e. hash key is the composite key made up of all the
attributes in the tuple).
- AggregationOperationState calls appropriate method (i.e.
aggregateGroupBy or aggregateGroupByPartitioned) based on the way in
which aggregation is being performed.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
