nooneuse commented on code in PR #67469:
URL: https://github.com/apache/doris/pull/67469#discussion_r3922924949
##########
be/src/exprs/aggregate/aggregate_function_datasketches_hll_union_agg.h:
##########
@@ -91,12 +84,10 @@ struct AggregateFunctionHllSketchData {
"Internal error happened when update HLL sketch:
unknown exception.");
}
}
- void reset() {
- if (hll_union_data.has_value()) {
- hll_union_data->reset();
- }
- hll_union_data.reset();
+ void merge(const Sketch& sketch_data) {
+ merge(sketch_data, std::max<uint8_t>(sketch_data.get_lg_config_k(),
MIN_UNION_LOG_K));
Review Comment:
We revised the compatibility strategy instead of adding a version field to
the intermediate state.
The one-argument form has no user-specified lg_max_k, so enforcing a new cap
of 12 would be incompatible with legacy partial states whose effective lgK can
be as high as 21. DEFAULT_UNION_LOG_K is now set to MAX_LOG_K (21). read() and
partial-state merge() continue restoring the effective lgK carried by the
serialized sketch, so an old lgK=21 state is not unexpectedly downsampled by a
new BE.
This means the one-argument form can use more memory when its effective
state becomes dense at a high lgK; this is intentional and documented. A
lower-precision dense input can still reduce the effective lgK, and sparse
states do not immediately allocate a full dense array.
The new two-argument form is the mechanism for setting an explicit precision
and memory cap. Since older BEs do not recognize that signature, it is
documented as supported only after all BEs in the cluster have been upgraded.
We also added a test that serializes and restores a dense lgK=21 partial
state to cover this compatibility case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
