alamb commented on code in PR #119:
URL: https://github.com/apache/parquet-testing/pull/119#discussion_r3785039184
##########
data/README.md:
##########
@@ -592,6 +593,69 @@ java -jar
parquet-cli/target/parquet-cli-1.16.0-SNAPSHOT-runtime.jar cat /home/r
{"utf8_full_truncation": "Kevin Bacon", "binary_full_truncation": "Kevin
Bacon", "utf8_partial_truncation": "🚀Kevin Bacon", "binary_partial_truncation":
"ÿÿ\u0001\u0002", "utf8_no_truncation": "Ke", "binary_no_truncation": "Ke"}
```
+## ALP encoding
+
+`alp_extended.zstd.parquet` contains FLOAT and DOUBLE columns encoded with
+[Adaptive Lossless floating-Point
(ALP)](https://github.com/apache/parquet-format/blob/master/Encodings.md#adaptive-lossless-floating-point-alp--10)
+(`ALP = 10`).
+It was created with the code in this
[PR](https://github.com/apache/arrow/pull/49154).
+
+All columns contain the same 9032 values. The same values appear all columns
so
+decoders can bit-compare the `ALP` columns against known results stored with
+`PLAIN` encoding.
+
+
+| Column | Encoding | Rationale
/ coverage |
+|-------------------------------------|---------------------------|-----------------------------------------------------------------------------------|
+| `float_plain`, `double_plain` | `PLAIN` + zstd | In-file
reference: readers can bit-compare the ALP columns against these |
+| `float_alp_1024`, `double_alp_1024` | `ALP`, 1024-value vectors | The
default vector size of 1024 values |
+| `float_alp_4096`, `double_alp_4096` | `ALP`, 4096-value vectors | Readers
must honor `log_vector_size` from the page header rather than assume 1024 |
+| `float_alp_32`, `double_alp_32` | `ALP`, 32-value vectors | Many
vectors per page, stresses the per-vector metadata loop |
+
+
+### Data distribution (9032 rows)
+
+The "base distribution" means random values in `[-10.00, 10.00]` with exactly 2
+decimal digits (e.g. `9.43`), which are losslessly encodable by ALP (no
exceptions).
+
+The contents of the 9032 rows are as follows:
+
+| Rows | Contents
| Rationale / coverage
|
+|-----------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| 0–1023 | base
| Happy path: full vector, small
frame-of-reference bit width, no exceptions
|
+| 1024–2047 | base, plus: NaN at 1024, 1500 and 2047 (three distinct bit
patterns, see below), +Inf at 2000, −Inf at 2001, −0.0 at 2002, subnormal
(`5e-324` double / `1e-45` float) at 2003 | NaN / Inf and sign/precision edge
values via the exception mechanism; exceptions at exact vector boundaries; NaN
payload preservation; statistics conventions (NaN excluded from min/max, ±Inf
included, `nan_count`, −0.0/+0.0 normalization) |
+| 2048–3071 | base, plus: `3.141592653589793` at 2500
| Exactly one exception: the full-mantissa value
cannot round-trip as a decimal, and nothing else in the vector is exceptional
|
+| 3072–4095 | base, plus: `44974934523.343` at 3100 and `-1243432432.3432` at
3711
| Large-magnitude values
|
+| 4096–5119 | every 2nd value full-mantissa random, rest base
| Many (but not all) exceptions
|
+| 5120–6143 | all values full-mantissa random
| All exceptions
|
+| 6144–7167 | base but with 4 decimal digits (e.g. `3.1416`)
| Different exponent/factor than the other
vectors
|
+| 7168–8191 | constant (all `7.77`)
| `bit_width = 0` vectors
|
+| 8192–8999 | base, with nulls at every 100th row (8 nulls)
| Partial vector + null handling
|
Review Comment:
Indeed you are right, fixed. in 7019116.
I confirmed that the null rows are in fact 8200, etc using DataFusion cli:
```sql
> select file_row_index() as fir, * from 'data/alp_extended.zstd.parquet'
where float_plain IS NULL;
+------+-------------+--------------+----------------+-----------------+----------------+-----------------+--------------+---------------+
| fir | float_plain | double_plain | float_alp_1024 | double_alp_1024 |
float_alp_4096 | double_alp_4096 | float_alp_32 | double_alp_32 |
+------+-------------+--------------+----------------+-----------------+----------------+-----------------+--------------+---------------+
| 8200 | NULL | NULL | NULL | NULL |
NULL | NULL | NULL | NULL |
| 8300 | NULL | NULL | NULL | NULL |
NULL | NULL | NULL | NULL |
| 8400 | NULL | NULL | NULL | NULL |
NULL | NULL | NULL | NULL |
| 8500 | NULL | NULL | NULL | NULL |
NULL | NULL | NULL | NULL |
| 8600 | NULL | NULL | NULL | NULL |
NULL | NULL | NULL | NULL |
| 8700 | NULL | NULL | NULL | NULL |
NULL | NULL | NULL | NULL |
| 8800 | NULL | NULL | NULL | NULL |
NULL | NULL | NULL | NULL |
| 8900 | NULL | NULL | NULL | NULL |
NULL | NULL | NULL | NULL |
+------+-------------+--------------+----------------+-----------------+----------------+-----------------+--------------+---------------+
8 row(s) fetched.
Elapsed 0.007 seconds.
```
```sql
> select file_row_index() as fir, * from 'data/alp_extended.zstd.parquet'
where file_row_index() BETWEEN 8180 AND 8250;
+------+-------------+--------------+----------------+-----------------+----------------+-----------------+--------------+---------------+
| fir | float_plain | double_plain | float_alp_1024 | double_alp_1024 |
float_alp_4096 | double_alp_4096 | float_alp_32 | double_alp_32 |
+------+-------------+--------------+----------------+-----------------+----------------+-----------------+--------------+---------------+
| 8180 | 7.77 | 7.77 | 7.77 | 7.77 |
7.77 | 7.77 | 7.77 | 7.77 |
| 8181 | 7.77 | 7.77 | 7.77 | 7.77 |
7.77 | 7.77 | 7.77 | 7.77 |
| 8182 | 7.77 | 7.77 | 7.77 | 7.77 |
7.77 | 7.77 | 7.77 | 7.77 |
| 8183 | 7.77 | 7.77 | 7.77 | 7.77 |
7.77 | 7.77 | 7.77 | 7.77 |
| 8184 | 7.77 | 7.77 | 7.77 | 7.77 |
7.77 | 7.77 | 7.77 | 7.77 |
| 8185 | 7.77 | 7.77 | 7.77 | 7.77 |
7.77 | 7.77 | 7.77 | 7.77 |
| 8186 | 7.77 | 7.77 | 7.77 | 7.77 |
7.77 | 7.77 | 7.77 | 7.77 |
| 8187 | 7.77 | 7.77 | 7.77 | 7.77 |
7.77 | 7.77 | 7.77 | 7.77 |
| 8188 | 7.77 | 7.77 | 7.77 | 7.77 |
7.77 | 7.77 | 7.77 | 7.77 |
| 8189 | 7.77 | 7.77 | 7.77 | 7.77 |
7.77 | 7.77 | 7.77 | 7.77 |
| 8190 | 7.77 | 7.77 | 7.77 | 7.77 |
7.77 | 7.77 | 7.77 | 7.77 |
| 8191 | 7.77 | 7.77 | 7.77 | 7.77 |
7.77 | 7.77 | 7.77 | 7.77 |
| 8192 | -1.04 | -1.04 | -1.04 | -1.04 |
-1.04 | -1.04 | -1.04 | -1.04 |
| 8193 | -7.82 | -7.82 | -7.82 | -7.82 |
-7.82 | -7.82 | -7.82 | -7.82 |
| 8194 | -2.12 | -2.12 | -2.12 | -2.12 |
-2.12 | -2.12 | -2.12 | -2.12 |
| 8195 | 9.72 | 9.72 | 9.72 | 9.72 |
9.72 | 9.72 | 9.72 | 9.72 |
| 8196 | -5.09 | -5.09 | -5.09 | -5.09 |
-5.09 | -5.09 | -5.09 | -5.09 |
| 8197 | 9.14 | 9.14 | 9.14 | 9.14 |
9.14 | 9.14 | 9.14 | 9.14 |
| 8198 | -9.4 | -9.4 | -9.4 | -9.4 |
-9.4 | -9.4 | -9.4 | -9.4 |
| 8199 | 0.77 | 0.77 | 0.77 | 0.77 |
0.77 | 0.77 | 0.77 | 0.77 |
| 8200 | NULL | NULL | NULL | NULL |
NULL | NULL | NULL | NULL |
| 8201 | -4.63 | -4.63 | -4.63 | -4.63 |
-4.63 | -4.63 | -4.63 | -4.63 |
| 8202 | -1.53 | -1.53 | -1.53 | -1.53 |
-1.53 | -1.53 | -1.53 | -1.53 |
| 8203 | -0.13 | -0.13 | -0.13 | -0.13 |
-0.13 | -0.13 | -0.13 | -0.13 |
| 8204 | -9.36 | -9.36 | -9.36 | -9.36 |
-9.36 | -9.36 | -9.36 | -9.36 |
| 8205 | 6.35 | 6.35 | 6.35 | 6.35 |
6.35 | 6.35 | 6.35 | 6.35 |
| 8206 | -4.15 | -4.15 | -4.15 | -4.15 |
-4.15 | -4.15 | -4.15 | -4.15 |
| 8207 | 7.27 | 7.27 | 7.27 | 7.27 |
7.27 | 7.27 | 7.27 | 7.27 |
| 8208 | -8.3 | -8.3 | -8.3 | -8.3 |
-8.3 | -8.3 | -8.3 | -8.3 |
| 8209 | 2.96 | 2.96 | 2.96 | 2.96 |
2.96 | 2.96 | 2.96 | 2.96 |
| 8210 | 9.12 | 9.12 | 9.12 | 9.12 |
9.12 | 9.12 | 9.12 | 9.12 |
| 8211 | 7.22 | 7.22 | 7.22 | 7.22 |
7.22 | 7.22 | 7.22 | 7.22 |
| 8212 | -4.66 | -4.66 | -4.66 | -4.66 |
-4.66 | -4.66 | -4.66 | -4.66 |
| 8213 | 4.14 | 4.14 | 4.14 | 4.14 |
4.14 | 4.14 | 4.14 | 4.14 |
| 8214 | 9.63 | 9.63 | 9.63 | 9.63 |
9.63 | 9.63 | 9.63 | 9.63 |
| 8215 | -3.08 | -3.08 | -3.08 | -3.08 |
-3.08 | -3.08 | -3.08 | -3.08 |
| 8216 | 3.5 | 3.5 | 3.5 | 3.5 | 3.5
| 3.5 | 3.5 | 3.5 |
| 8217 | -8.93 | -8.93 | -8.93 | -8.93 |
-8.93 | -8.93 | -8.93 | -8.93 |
| 8218 | 8.7 | 8.7 | 8.7 | 8.7 | 8.7
| 8.7 | 8.7 | 8.7 |
| 8219 | 1.85 | 1.85 | 1.85 | 1.85 |
1.85 | 1.85 | 1.85 | 1.85 |
| .
|
| .
|
| .
|
+------+-------------+--------------+----------------+-----------------+----------------+-----------------+--------------+---------------+
71 row(s) fetched. (First 40 displayed. Use --maxrows to adjust)
Elapsed 0.012 seconds.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
