GitHub user x-at-01 created a discussion: Sharing benchmark observations on decimal floating-point compression: comparing integer mapping with Gorilla/Chimp on time-series datasets
Hello DataFusion community, Apache DataFusion is widely used across modern analytical and time-series query engines. Since floating-point columns represent a major workload in columnar analytics, I wanted to share benchmark observations and implementation findings on lossless floating-point compression from [fastalp](https://github.com/webc-site/wedb_embed/tree/main/fastalp) (available on [crates.io](https://crates.io/crates/fastalp)), an implementation of the Adaptive Lossless Floating-Point Compression (ALP) algorithm. ### Benchmark results on real-world time-series datasets I evaluated fastalp against classic floating-point codecs (Gorilla, Chimp, Chimp128, Patas) on public time-series datasets including city_temperature.csv, Stocks-Germany-sample.txt, and SSD_HDD_benchmarks.csv. Running 25 iterations per dataset on Apple Silicon (M2 Max) produced the following unified averages: | Codec | Compression Ratio | Encoding Latency (µs / 1000 vals) | Decoding Latency (µs / 1000 vals) | | :--- | :--- | :--- | :--- | | fastalp | 16.34 bits/val | 2.255 µs | 0.423 µs | | Gorilla | 52.70 bits/val | 6.042 µs | 5.920 µs | | Patas | 21.51 bits/val | 6.818 µs | 6.502 µs | | Chimp128 | 17.29 bits/val | 7.468 µs | 7.637 µs | | Chimp | 41.08 bits/val | 8.631 µs | 9.270 µs | Overall throughput across standard IEEE 754 float benchmarks:  ### Implementation observations for columnar analytics 1. Decimal integer mapping: In real-world telemetry and business metrics, floating-point numbers often have a fixed or small number of decimal places (e.g. 19.99, 0.05). Mapping values to integers via adaptive decimal scaling (10^e) shifts the data domain to Frame-of-Reference (FOR) integer packing, enabling SIMD-accelerated execution rather than serial bitwise XOR streams. 2. Exact decimal division: Precomputed floating-point multiplication (value * 10^e) occasionally introduces 1-ULP precision noise due to IEEE 754 rounding (for example, 0.35 * 100.0 evaluates to 34.99999999999999, casting to 34 instead of 35). Adding an exact division path during sampling and reconstruction eliminates these pseudo-exceptions and preserves bit-packing efficiency. 3. First-order difference (Delta-ALP): For monotonic timestamps, cumulative metrics, and smooth waveforms, cascading first-order delta encoding onto the scaled integers significantly narrows residual values, improving compression ratios while preserving O(1) random block access. 4. Pure register SIMD decoding: Vectorized integer unpacking achieves 0.423 µs per 1000 values (~2.36 billion values/sec), reaching 55 to 77 GB/s linear memory bandwidth on modern CPUs. 5. Zero-heap streaming API: The compress_into and decompress_into interfaces allow encoding and decoding directly into caller-provided buffers, eliminating allocation overhead during query scans. I hope these benchmark results and implementation findings provide useful reference points for columnar storage and query execution discussions. GitHub link: https://github.com/apache/datafusion/discussions/24935 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
