wgtmac commented on code in PR #55449:
URL: https://github.com/apache/arrow/pull/55449#discussion_r4536462279
##########
cpp/src/arrow/util/alp/alp_sampler_internal.h:
##########
@@ -34,7 +34,7 @@ namespace arrow::util::alp {
template <AlpFloatingType T>
class ARROW_EXPORT AlpSampler {
public:
- AlpSampler() = default;
+ explicit AlpSampler(int64_t sampling_window_size);
Review Comment:
There are two levels of sampling:
1. **First-level sampling builds a shortlist of encoding parameters.** For
example, with 122,880 input values (`sampling_window_size`), we have 30 chunks
of 4,096 values (`kChunkSize`). We target 8 sampled chunks
(`kSampleChunksPerWindow`), so we select every fourth chunk: 0, 4, …, 28. From
each selected chunk, we sample only its first 1,024 values
(`kChunkSampleWindowSize`), taking every fourth value to collect 256 samples
(`kSamplesPerChunk`). We search all legal exponent/factor pairs on these
samples and keep up to 5 candidates (`kMaxCombinations`).
2. **Second-level sampling chooses one candidate for each encoded vector.**
With `vector_size = 1024`, we sample 256 values across that vector, reusing
`kSamplesPerChunk` as the limit, and compare the shortlisted candidates. If
there is only one candidate, this search is skipped. `kEarlyExitThreshold`
controls how many consecutive non-improving candidates we tolerate; its current
value of 4 saves no work with at most 5 candidates.
`sampling_window_size` counts **input values, not collected samples**.
Before reaching it, each page builds its own preset. At the page boundary where
the accumulated count reaches it, we build and cache a preset from the
accumulated samples, then reuse it for the remaining pages in that column chunk.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
