nooneuse opened a new pull request, #69257:
URL: https://github.com/apache/doris/pull/69257
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary:
Apache DataSketches HLL sketches may occupy significant storage space when a
large `lgK` is used. For historical or cold partitions, retaining the original
precision may no longer be necessary, but Doris previously had no scalar
function that could read a serialized DataSketches HLL sketch and output
another serialized sketch with a lower `lgK`.
This PR introduces the following function:
```sql
datasketches_hll_downsample(sketch, target_lg_k [, target_type])
```
The function accepts a serialized Apache DataSketches HLL sketch and
produces another serialized sketch whose `lgK` is:
```text
min(source_lg_k, target_lg_k)
```
It can be used with `INSERT OVERWRITE` to rewrite sketch columns in
historical partitions, reducing their serialized size without rebuilding the
sketches from raw upstream data.
#### Arguments
- `sketch`
- A serialized Apache DataSketches HLL sketch.
- Supported Doris types: `STRING`, `VARCHAR`, and `VARBINARY`.
- The returned value uses the same type family as the input.
- `target_lg_k`
- A constant integer between `4` and `21`.
- The function never increases the source sketch's `lgK`.
- `target_type`
- An optional constant string.
- Supported values: `HLL_4`, `HLL_6`, and `HLL_8`.
- The default value is `HLL_4`.
When the source sketch already has an `lgK` less than or equal to
`target_lg_k` and already uses the requested target type, the function returns
the original serialized bytes unchanged.
If the source `lgK` is already lower than the requested value but the target
type is different, only the HLL representation type is converted; the source
`lgK` is preserved.
`NULL` input produces `NULL`. Invalid parameters or malformed serialized
sketches return an explicit error.
> Lowering `lgK` is irreversible and reduces the accuracy available to
subsequent HLL unions and estimates. The target value should therefore be
selected according to the acceptable accuracy for each data retention period.
Dense sketches generally receive the most significant storage reduction, while
sparse LIST or SET mode sketches may see little or no reduction.
#### Usage examples
Downsample a sketch to at most `lgK = 10` using the default `HLL_4`
representation:
```sql
SELECT datasketches_hll_downsample(sketch_column, 10)
FROM user_sketches;
```
Use `HLL_6` as the target representation:
```sql
SELECT datasketches_hll_downsample(sketch_column, 10, 'HLL_6')
FROM user_sketches;
```
Rewrite the sketch column of a historical partition while preserving the
other columns:
```sql
INSERT OVERWRITE TABLE user_sketches PARTITION(p202401)
SELECT
event_date,
dimension_id,
datasketches_hll_downsample(sketch_column, 10, 'HLL_4'),
other_column
FROM user_sketches PARTITION(p202401);
```
Check whether a partition contains at least one sketch that still requires
conversion:
```sql
SELECT 1
FROM user_sketches PARTITION(p202401)
WHERE datasketches_hll_downsample(
sketch_column,
10,
'HLL_4'
) <> sketch_column
LIMIT 1;
```
An empty result means that every non-NULL sketch in the partition already
satisfies the requested `lgK` and target type.
The converted sketches remain compatible with the existing DataSketches HLL
aggregate function:
```sql
SELECT datasketches_hll_union_agg(
datasketches_hll_downsample(sketch_column, 10)
)
FROM user_sketches;
```
### Release note
Add the `DATASKETCHES_HLL_DOWNSAMPLE` scalar function for converting
serialized Apache DataSketches HLL sketches to a configurable lower `lgK` and
optional `HLL_4`, `HLL_6`, or `HLL_8` representation.
The function supports `STRING`, `VARCHAR`, and `VARBINARY` inputs,
propagates `NULL`, never increases the source `lgK`, and returns the original
bytes when no conversion is required.
Lowering `lgK` reduces serialized storage for dense sketches at the cost of
lower estimation accuracy. The operation is irreversible. All BE nodes should
be upgraded before the function is used.
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should
merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
