alamb commented on issue #105: URL: https://github.com/apache/parquet-testing/issues/105#issuecomment-5172111570
I spent some time designing what I think would be an example file that covers all the relevant corner cases. 1. A single parquet file with multiple columns (to keep the size down) 2. Columns with the same values encoded with multiple encodings (so downstream decoders can compare with known results, following the model of [`byte_stream_split_extended.gzip.parquet`](https://github.com/apache/parquet-testing/blob/master/data/byte_stream_split_extended.gzip.parquet)) ### Columns | Column | Encoding | Rationale / coverage | |---|---|---| | `float_plain`, `double_plain` | PLAIN + zstd | In-file reference: readers can bit-compare the ALP columns against these | | `float_alp_1024`, `double_alp_1024` | ALP, 1024-value vectors | The default vector size | | `float_alp_4096`, `double_alp_4096` | ALP, 4096-value vectors | Readers must honor `log_vector_size` from the page header rather than assume 1024 | | `float_alp_32`, `double_alp_32` | ALP, 32-value vectors | many vectors per page, stress per-vector metadata loop | ### Data distribution (9000 rows) Base distribution: random values in [-10.00, 10.00] with **exactly 2 decimal digits** (e.g. 9.43) — losslessly encodable by ALP, so base values produce zero exceptions. Total row count is deliberately **not** a multiple of any vector size to ensure coverage of incomplete vectors | Rows | Contents | Rationale / coverage | |---|---|---| | 0–1023 | base | Happy path: full vector, small FOR bit width, **no exceptions** | | 1024–2047 | base, plus: NaN at 1024 and 2047 (first/last element of the vector), NaN at 1500, +inf at 2000, −inf at 2001, −0.0 at 2002, subnormal at 2003 (`5e-324` double / `1e-45` float) | **NaN / Inf** and sign/precision edge values via the exception mechanism; exceptions at exact vector boundaries; statistics conventions (NaN excluded from min/max, ±inf included, `nan_count`, −0.0/+0.0 normalization) | | 2048–3071 | base, plus `3.141592653589793` at 2500 | **Exactly one exception**: the full-mantissa value cannot round-trip as a decimal, and nothing else in the vector is exceptional | | 3072–4095 | base, plus `44974934523.343` at 3100 and `-1243432432.3432` at 3711 | Large-magnitude values | | 4096–5119 | every 2nd value full-mantissa random (e.g. from `random() * 20 - 10` unrounded), rest base | **Many (but not all) exceptions** | | 5120–6143 | all values full-mantissa random | **All exceptions** (the "random float data" case) | | 6144–7167 | base but with 4 decimal digits (e.g. 3.1416) | **different exponent/factor** than the other vectors | | 7168–8191 | constant (all `7.77`) | **bit_width = 0** vectors | | 8192–8999 | base, with **nulls** at every 100th row | Trailing **partial vector** + null handling | ### Documentation Following the model of [Additional types](https://github.com/apache/parquet-testing/blob/master/data/README.md#additional-types) and the [variant README](https://github.com/apache/parquet-testing/tree/master/variant#descriptions), the PR should add a `data/README.md` section with: the schema, the row-range table above, and verification instructions. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
