GitHub user jianqiao opened a pull request:
https://github.com/apache/incubator-quickstep/pull/179
QUICKSTEP-70-71 Improve aggregation performance
This PR implements two features that improve aggregation performance:
1. Adds `CollisionFreeVectorTable` to support specialized high-performance
aggregation.
2. Adds support for aggregation copy elision that we only materialize
intermediate results for non-trivial expressions.
For feature 1, when the group-by attribute is a range-bounded single
attribute of `INT` or `LONG` type. We can use a vector of type
`std::vector<std::atomic<StateT>>` to store the aggregation states, where
`StateT` is the aggregation state type (currently restricted to `LONG` and
`DOUBLE`). Then during aggregation, for each tuple, we locate the aggregation
state with the group-by key's value as index to the state vector, and
concurrently update the state with C++'s atomic primitives.
For feature 2, note that the current implementation of aggregation always
creates a `ColumnVectorsValueAccessor` to store the results of ALL the input
expressions. However, we can avoid the creation of a column vector (thus
avoiding copying values into the column vector) if the aggregation is on a
simple attribute, e.g. `SUM(x)`. Thus by PR, when performing aggregation we
prepare two input `ValueAccessor`s: one BASE accessor that is created directly
from the input relation's storage block, and one DERIVED accessor that is the
temporary result `ColumnVectorsValueAccessor`. Each aggregation argument may be
from the base accessor (meaning that it is a simple attribute) or from the
derived accessor (meaning that it is a non-trivial expression that gets
evaluated). The two accessors are then properly handled in aggregation handles
and aggregation hash tables.
**Main changes:**
`expressions/aggregation`: Updated the aggregation handles to support copy
elision. Also did some cleanups.
`relational_operators`: Added `InitializeAggregationOperator` to support
parallel initialization of the aggregation state (just `memset` the memory to
0) -- because it takes a relatively long time to do the initialization with
single thread if the aggregation hash table is large.
`storage`: Added `CollisionFreeVectorTable`. Renamed `FastHashTable` to
`PackedPayloadHashTable`, made it support copy elision, and cleaned up the
class to remove unused methods. Refactored `AggregationOperationState` to
support copy elision and support the new aggregation. Moved aggregation code
out of `StorageBlock`.
This PR significantly improves some TPC-H queries' performance. For
example, it improves TPC-H Q18 from ~27.5s to ~3.5s, with scale factor 100 on a
cloudlab machine.
Below shows the TPC-H performance (scale factor 100 on a cloudlab machine)
with recently committed optimizations up to this point:
| **TPCH SF100** | **master (ms)** | **w/ optimizations (ms)** |
| ------ | ------ | ------ |
| Q01 | 13629 | 11221 |
| Q02 | 537 | 460 |
| Q03 | 4824 | 4124 |
| Q04 | 2185 | 2203 |
| Q05 | 5517 | 5282 |
| Q06 | 399 | 401 |
| Q07 | 18563 | 3456 |
| Q08 | 1746 | 899 |
| Q09 | 7247 | 5586 |
| Q10 | 6745 | 5665 |
| Q11 | 1053 | 247 |
| Q12 | 1713 | 1698 |
| Q13 | 22896 | 15582 |
| Q14 | 805 | 745 |
| Q15 | 897 | 431 |
| Q16 | 9942 | 9158 |
| Q17 | 1588 | 1117 |
| Q18 | 27459 | 3507 |
| Q19 | 1711 | 1609 |
| Q20 | 1204 | 1014 |
| Q21 | 8671 | 7886 |
| Q22 | 6178 | 724 |
| **Total** | **145509** | **83016** |
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/apache/incubator-quickstep collision-free-agg
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-quickstep/pull/179.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #179
----
commit 68be4a614f55412632f13051295327fecba1fada
Author: Jianqiao Zhu <[email protected]>
Date: 2017-01-30T20:46:39Z
- Adds CollisionFreeVectorTable to support specialized fast path
aggregation for range-bounded single integer group-by key.
- Supports copy elision for aggregation.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
