Doris-Breakwater commented on issue #66430: URL: https://github.com/apache/doris/issues/66430#issuecomment-5176679799
Breakwater-GitHub-Analysis-Slot: slot_7876f7977bf5 ### Initial triage This is a BE crash in the legacy Parquet reader used by a load, not a failure in the cloud meta-service startup path. The cloud configuration is useful context, but no cloud metadata component appears in the failing stack. **Judgment:** there is a verified process-fatal handling problem in the reader, and a compressed Parquet DataPageV2 containing definition levels but no physical BOOLEAN values is the most likely trigger. That page can be valid when every logical value is NULL; a malformed page can reach a similar state, so file corruption is not established from this stack alone. ### Verified facts - At the reported commit, [`ColumnChunkReader::load_page_data()`](https://github.com/apache/doris/blob/d5cfef083307be7f2dd5c3601b770e88fba40501/be/src/vec/exec/format/parquet/vparquet_column_chunk_reader.cpp#L163-L250) subtracts the DataPageV2 repetition/definition-level lengths from `uncompressed_page_size`. It then creates `_page_data` from the decompression buffer and unconditionally calls the page decoder's `set_data()`. - When the remaining value-section size is zero, `_reserve_decompress_buf(0)` does not allocate a buffer, so `_page_data` can be `Slice(nullptr, 0)`. - [`BoolPlainDecoder::set_data()`](https://github.com/apache/doris/blob/d5cfef083307be7f2dd5c3601b770e88fba40501/be/src/vec/exec/format/parquet/bool_plain_decoder.h#L48-L54) forwards that pointer to `BatchedBitReader::Reset()`, whose [`DCHECK(buffer != nullptr)`](https://github.com/apache/doris/blob/d5cfef083307be7f2dd5c3601b770e88fba40501/be/src/util/bit_stream_utils.h#L184-L190) aborts the BE. This matches every relevant frame and source line in the supplied stack. - The [Parquet format specification](https://github.com/apache/parquet-format/blob/master/README.md#nulls) explicitly permits an all-NULL nullable page to have definition levels and no encoded values. - The newer native Parquet reader already models this case with an [`EmptyValueSectionDecoder`](https://github.com/apache/doris/blob/a8b1fd95e2a0e08f1b1f88c8c34b244b85ea96a7/be/src/format_v2/parquet/reader/native/column_chunk_reader.cpp#L264-L275): it accepts a level-only all-NULL page and returns `Corruption` if definition levels require values that are absent. Its [tests cover both outcomes, including BOOLEAN](https://github.com/apache/doris/blob/a8b1fd95e2a0e08f1b1f88c8c34b244b85ea96a7/be/test/format_v2/parquet/native_decoder_test.cpp#L3667-L3699). This is useful implementation precedent, but it does **not** prove that this issue is fixed: loads currently remain on the legacy scanner path, and the legacy reader still has the unconditional `set_data()` behavior. - The reported build identity needs clarification. Full SHA `d5cfef083307be7f2dd5c3601b770e88fba40501` is based directly on the public `4.0.5` tag commit plus one OSS SDK commit; it is not the public `4.1.2` tag (`aec169d20256a788de448f46737b5fb53ee3b4e3`). The same risky legacy-reader pattern is present in the public 4.1.2 source, but reproduction against an official 4.1.2 build has not been provided. ### Missing information needed to confirm the trigger 1. The exact load mechanism and minimal SQL: `CREATE TABLE` plus `LOAD`, `COPY INTO`, or `INSERT ... SELECT`, including relevant properties with credentials removed. 2. The failing Parquet file, or a sanitized/minimized reproducer preserving its schema and page boundaries. If the job reads many files, please first isolate the file that crashes query `c22121b9ab894ef4-acd8857bdff4d757`. 3. Page-level metadata for the suspected BOOLEAN column: writer library/version, compression codec, DataPage version, encoding, `num_values`, `num_nulls`, compressed/uncompressed page sizes, and repetition/definition-level byte lengths. Please also indicate whether the column/page is all NULL. A `parquet-tools`/PyArrow metadata dump and whether another conforming reader can read the file would help. 4. The complete BE INFO/WARNING log around the query ID, the corresponding FE audit/load log, and the query/load profile if one was produced. These should identify the load statement, scan range, and source file. 5. The exact package/build provenance (`doris_be --version` or equivalent full build information) and why it reports 4.1.2 while embedding the SHA above. Please also retry with an official 4.1.2 binary if possible. ### Recommended next steps - Add a focused legacy-reader reproducer for compressed DataPageV2, nullable BOOLEAN, PLAIN encoding, and an all-NULL/level-only page. Also cover DataPageV1/V2, compressed/uncompressed pages, and an invalid level-only page whose definition levels require a non-NULL value. - Handle an empty physical value section in `ColumnChunkReader`, before invoking an encoding-specific decoder. Accept it only when decoded definition levels require zero physical values; otherwise return a bounded `Status::Corruption`. Do not merely relax `BatchedBitReader::Reset()` and do not leave a reused decoder with stale state. - Ensure malformed input fails the load/query without terminating the BE. Suggested triage labels: `kind/fix`, `area/load`, and `investigation needed` until the file/reproducer confirms the trigger. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
