This is an automated email from the ASF dual-hosted git repository.
alamb pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/parquet-testing.git
The following commit(s) were added to refs/heads/master by this push:
new e25b7b6 GH-112 Harden variant decoding: example data (#113)
e25b7b6 is described below
commit e25b7b6970143ea20bb2f65b8171e2d11ba846f2
Author: Steve Loughran <[email protected]>
AuthorDate: Wed Aug 12 17:47:14 2026 +0100
GH-112 Harden variant decoding: example data (#113)
* GH-3561 Harden variant decoding
Test files are added to parquet-format project with commentary.
* add a file where the variant is tagged as v2 rather than v1;
All parquet readers must reject this.
* Add invalid files from tests of Iceberg variant hardening PR
* Add unknown primitive type test file
* Verify two dictionary entries can point to same offset
...from discussion in the iceberg PR.
* updated int overflow in bounds check test file
---
bad_data/README.md | 28 +++++++++++++++++++++
bad_data/variants/duplicate_field_offsets.parquet | Bin 0 -> 552 bytes
bad_data/variants/field_id_out_of_range.parquet | Bin 0 -> 524 bytes
.../variants/int_overflow_in_bounds_check.parquet | Bin 0 -> 501 bytes
...lformed_child_inside_well_formed_parent.parquet | Bin 0 -> 538 bytes
bad_data/variants/negative_dictionary_size.parquet | Bin 0 -> 517 bytes
.../variants/out_of_range_child_offset.parquet | Bin 0 -> 501 bytes
.../variants/out_of_range_dictionary_size.parquet | Bin 0 -> 501 bytes
.../variants/out_of_range_element_count.parquet | Bin 0 -> 508 bytes
.../variants/over_deep_nested_children.parquet | Bin 0 -> 3655 bytes
bad_data/variants/oversized_primitive_size.parquet | Bin 0 -> 501 bytes
.../short_string_length_exceeds_buffer.parquet | Bin 0 -> 472 bytes
bad_data/variants/truncated_primitive_size.parquet | Bin 0 -> 472 bytes
bad_data/variants/unknown_primitive_type.parquet | Bin 0 -> 544 bytes
bad_data/variants/variant_version_2_header.parquet | Bin 0 -> 501 bytes
15 files changed, 28 insertions(+)
diff --git a/bad_data/README.md b/bad_data/README.md
index 4fbc0c4..8e9c808 100644
--- a/bad_data/README.md
+++ b/bad_data/README.md
@@ -35,3 +35,31 @@ These are files used for reproducing various bugs that have
been reported.
where repetition levels start with a 1 instead of 0.
* ARROW-GH-47662.parquet: test case identified in
https://github.com/apache/arrow/issues/47662
where a required column contains null values (an incorrect version of
data/fixed_length_byte_array.parquet).
+
+
+## Directory `variants`
+
+This subdirectory contains files with malformed variant structures.
+
+Robust implementations of variant decoders SHOULD reject these.
+
+| File | Malformed
Structure |
+|---------------------------------------------------------------|----------------------------------------------------------------------------|
+| `variant/int_overflow_in_bounds_check.parquet` | Triggers an
overflow if 32 bit multiplication is used to calculate ranges. |
+| `variant/out_of_range_dictionary_size.parquet` | The
dictionary is declared as larger than the data |
+| `variant/malformed_child_inside_well_formed_parent.parquet` | Parent is
well formed; child is malformed |
+| `variant/out_of_range_child_offset.parquet` | The offset
of an child element is out of range |
+| `variant/out_of_range_element_count.parquet` | The number
of declared array elements is larger than the data |
+| `variant/bad_data/variants/over_deep_nested_children.parquet` | The
hierarchy is excessively deep |
+
+The first of these is the most critical, as this can trigger a memory
allocation of many GiB, which may affect the operations of other worker threads
in a shared process; an oversized dictionary may also trigger excessive memory
allocation.
+
+The out of range child and element files contain metadata referring to content
past the end of the actual data field.
+On languages with strict range check, this will fail on read; extra
verification simply changes when the failure is detected.
+For languages where range checks are not automatically, there is a risk of
variant data referencing other data on the stack/in the heap.
+As this data is read only, there's no _direct_ threat to the integrity of the
process, but it is still highly dangerous.
+
+One notable file is `bad_data/variants/over_deep_nested_children.parquet`,
which verifies that nested variant children over 500 levels deep is rejected.
This number is subjective; it was chosen to be consistent with the JSON parser
`org.apache.parquet.variant.VariantJsonParser`.
+
+Currently excluded from these tests is any with an explicit limit on the size
of a variant.
+Apache Spark places a limit on 128 MiB on each of the metadata and value
fields here.
\ No newline at end of file
diff --git a/bad_data/variants/duplicate_field_offsets.parquet
b/bad_data/variants/duplicate_field_offsets.parquet
new file mode 100644
index 0000000..1299d7d
Binary files /dev/null and b/bad_data/variants/duplicate_field_offsets.parquet
differ
diff --git a/bad_data/variants/field_id_out_of_range.parquet
b/bad_data/variants/field_id_out_of_range.parquet
new file mode 100644
index 0000000..abb2b31
Binary files /dev/null and b/bad_data/variants/field_id_out_of_range.parquet
differ
diff --git a/bad_data/variants/int_overflow_in_bounds_check.parquet
b/bad_data/variants/int_overflow_in_bounds_check.parquet
new file mode 100644
index 0000000..c004d85
Binary files /dev/null and
b/bad_data/variants/int_overflow_in_bounds_check.parquet differ
diff --git
a/bad_data/variants/malformed_child_inside_well_formed_parent.parquet
b/bad_data/variants/malformed_child_inside_well_formed_parent.parquet
new file mode 100644
index 0000000..7f9cd68
Binary files /dev/null and
b/bad_data/variants/malformed_child_inside_well_formed_parent.parquet differ
diff --git a/bad_data/variants/negative_dictionary_size.parquet
b/bad_data/variants/negative_dictionary_size.parquet
new file mode 100644
index 0000000..2186b4c
Binary files /dev/null and b/bad_data/variants/negative_dictionary_size.parquet
differ
diff --git a/bad_data/variants/out_of_range_child_offset.parquet
b/bad_data/variants/out_of_range_child_offset.parquet
new file mode 100644
index 0000000..19e391e
Binary files /dev/null and
b/bad_data/variants/out_of_range_child_offset.parquet differ
diff --git a/bad_data/variants/out_of_range_dictionary_size.parquet
b/bad_data/variants/out_of_range_dictionary_size.parquet
new file mode 100644
index 0000000..f3e520d
Binary files /dev/null and
b/bad_data/variants/out_of_range_dictionary_size.parquet differ
diff --git a/bad_data/variants/out_of_range_element_count.parquet
b/bad_data/variants/out_of_range_element_count.parquet
new file mode 100644
index 0000000..836c67c
Binary files /dev/null and
b/bad_data/variants/out_of_range_element_count.parquet differ
diff --git a/bad_data/variants/over_deep_nested_children.parquet
b/bad_data/variants/over_deep_nested_children.parquet
new file mode 100644
index 0000000..9c2c389
Binary files /dev/null and
b/bad_data/variants/over_deep_nested_children.parquet differ
diff --git a/bad_data/variants/oversized_primitive_size.parquet
b/bad_data/variants/oversized_primitive_size.parquet
new file mode 100644
index 0000000..2df8168
Binary files /dev/null and b/bad_data/variants/oversized_primitive_size.parquet
differ
diff --git a/bad_data/variants/short_string_length_exceeds_buffer.parquet
b/bad_data/variants/short_string_length_exceeds_buffer.parquet
new file mode 100644
index 0000000..61bfbd6
Binary files /dev/null and
b/bad_data/variants/short_string_length_exceeds_buffer.parquet differ
diff --git a/bad_data/variants/truncated_primitive_size.parquet
b/bad_data/variants/truncated_primitive_size.parquet
new file mode 100644
index 0000000..92d3725
Binary files /dev/null and b/bad_data/variants/truncated_primitive_size.parquet
differ
diff --git a/bad_data/variants/unknown_primitive_type.parquet
b/bad_data/variants/unknown_primitive_type.parquet
new file mode 100644
index 0000000..16834b0
Binary files /dev/null and b/bad_data/variants/unknown_primitive_type.parquet
differ
diff --git a/bad_data/variants/variant_version_2_header.parquet
b/bad_data/variants/variant_version_2_header.parquet
new file mode 100644
index 0000000..b66044e
Binary files /dev/null and b/bad_data/variants/variant_version_2_header.parquet
differ
