clintropolis opened a new pull request, #20469:
URL: https://github.com/apache/druid/pull/20469
### Description
Optimizes SQL planning so that multi-value string operators that would plan
to a specialized `ListFilteredVirtualColumn` were `sqlUseExtractionFn` set to
true, but currently plan to a pure expression, to instead still plan to the
specialized column pointed at an expression virtual column. If the expression
virtual column is actually a transform on top of a dictionary encoded column,
the performance can be approximately the same as the extractionFn version since
those selectors can do deferred evaluation and provide access to the underlying
dictionary ids.
From the added benchmark:
#### `basic` dataset: 1.5M rows, `dimMultivalEnumerated` has 5 distinct
values
| # | query | expression (ms/op) | specialized (ms/op) | extractionFn
(ms/op) | speedup |
|---|---|---:|---:|---:|---:|
| 0 | `GROUP BY MV_FILTER_ONLY(LOOKUP(dimMultivalEnumerated, ...),
ARRAY['greeting'])` | 1072 ± 185 | 161 ± 52 | 154 ± 18 | 6.6x |
| 1 | `GROUP BY MV_FILTER_NONE(LOOKUP(dimMultivalEnumerated, ...),
ARRAY['greeting'])` | 1098 ± 145 | 168 ± 25 | 176 ± 16 | 6.5x |
| 2 | `GROUP BY MV_FILTER_REGEX(LOOKUP(dimMultivalEnumerated, ...), '^g.*')`
| 1179 ± 255 | 156 ± 8 | 154 ± 18 | 7.6x |
| 3 | `GROUP BY MV_FILTER_PREFIX(LOOKUP(dimMultivalEnumerated, ...), 'g')` |
1141 ± 490 | 156 ± 20 | 154 ± 19 | 7.3x |
| 4 | `WHERE MV_FILTER_ONLY(LOOKUP(dimMultivalEnumerated, ...),
ARRAY['greeting']) = 'greeting'` | 1060 ± 185 | 85 ± 7 | 82 ± 12 | 12.5x |
| 9 | `GROUP BY MV_FILTER_ONLY(SUBSTRING(dimMultivalEnumerated, 1, 1),
ARRAY['B'])` | 1069 ± 158 | 152 ± 42 | 147 ± 16 | 7.1x |
| 10 | `GROUP BY MV_FILTER_ONLY(REGEXP_EXTRACT(dimMultivalEnumerated,
'^[A-Z][a-z]'), ARRAY['Ba'])` | 1086 ± 186 | 152 ± 29 | 157 ± 16 | 7.2x |
#### `grouper` dataset: 2 segments × 1.5M rows, multi-value string columns
with 1M distinct values; the lookup maps 1M keys to 1000 values
| # | query | expression (ms/op) | specialized (ms/op) | extractionFn
(ms/op) | speedup |
|---|---|---:|---:|---:|---:|
| 5 | `GROUP BY MV_FILTER_ONLY(LOOKUP("multi-string-Uniform-1_000_000",
...), ARRAY['bucket-1', 'bucket-2'])` | 7119 ± 2195 | 1025 ± 335 | 981 ± 517 |
6.9x |
| 6 | `GROUP BY MV_FILTER_ONLY(LOOKUP("multi-string-ZipF-1_000_000", ...),
ARRAY['bucket-1', 'bucket-2'])` | 5859 ± 606 | 1192 ± 125 | 1180 ± 52 | 4.9x |
| 7 | `GROUP BY MV_FILTER_PREFIX(LOOKUP("multi-string-Uniform-1_000_000",
...), 'bucket-1')` | 7893 ± 4515 | 1351 ± 21 | 1316 ± 384 | 5.8x |
| 8 | `WHERE MV_FILTER_ONLY(LOOKUP("multi-string-Uniform-1_000_000", ...),
ARRAY['bucket-1']) = 'bucket-1'` | 7557 ± 367 | 887 ± 359 | 911 ± 214 | 8.5x |
| 11 | `GROUP BY MV_FILTER_ONLY(SUBSTRING("multi-string-Uniform-1_000_000",
1, 2), ARRAY['12', '34'])` | 3858 ± 1815 | 568 ± 103 | 597 ± 214 | 6.8x |
| 12 | `GROUP BY
MV_FILTER_ONLY(REGEXP_EXTRACT("multi-string-Uniform-1_000_000", '[0-9]{2}$'),
ARRAY['12', '34'])` | 5059 ± 5050 | 796 ± 83 | 747 ± 22 | 6.4x |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
