This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-4.2
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-4.2 by this push:
new 388e30e9249 [fix](parquet) Include file and Variant field context in
footer errors (#69119)
388e30e9249 is described below
commit 388e30e9249ee552db1e56e9c67230916abd1fbd
Author: Gabriel <[email protected]>
AuthorDate: Sun Oct 18 19:10:01 2026 +0800
[fix](parquet) Include file and Variant field context in footer errors
(#69119)
### What problem does this PR solve?
Related PRs: #69092, #69094
Port the Parquet V2 diagnostic improvement to branch-4.2. Footer/schema
failures currently omit the input file path, and Variant fallback errors
identify only a wrapper name such as `element`.
Attach the file path while preserving the original status code and
message. Invalid Variant fallback errors also include the
Variant-relative field path, declared physical type, repetition, and
child count. Diagnostic paths are constructed only on failure; accepted
layouts and read semantics remain unchanged.
### Validation
- Rebased onto `branch-4.2` at `9ffd818bec3`. Resolved the footer parser
call conflict by preserving the new `preserve_binary_uuid` argument and
the diagnostic error context. `git range-diff` confirms this argument is
the only adaptation to the original patch; the follow-up Variant fixture
correction is unchanged.
- clang-format 16.0.6 verification on all affected files and `git diff
--check` passed.
- The source PR passed all 45 `ParquetSchemaTest.*` tests in a focused
ASAN binary on branch-4.1. No new local build or unit-test run was
performed for this branch-4.2 rebase; branch validation is pending CI.
- Includes three diagnostic tests covering required/repeated fallback
leaves, incorrect physical types, group fallback values, a valid
control, unsupported Variant versions, and truncated footers.
### Release note
Parquet V2 footer errors now identify the input file. Invalid Variant
fallback errors additionally describe the field path and actual schema
declaration.
### Check List (For Author)
- Test
- [x] Unit Test (ported; branch-4.2 execution pending CI)
- Behavior changed:
- [x] Yes. Diagnostic context is added; status codes and validation
rules are preserved.
- Does this need documentation?
- [x] No.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
---
be/src/format_v2/parquet/native_schema_desc.cpp | 40 +++++++-
be/src/format_v2/parquet/parquet_file_context.cpp | 10 +-
be/test/format_v2/parquet/parquet_schema_test.cpp | 119 ++++++++++++++++++++++
3 files changed, 165 insertions(+), 4 deletions(-)
diff --git a/be/src/format_v2/parquet/native_schema_desc.cpp
b/be/src/format_v2/parquet/native_schema_desc.cpp
index 6a21ce165cd..ef6ec867c6c 100644
--- a/be/src/format_v2/parquet/native_schema_desc.cpp
+++ b/be/src/format_v2/parquet/native_schema_desc.cpp
@@ -419,9 +419,45 @@ Status validate_variant_layout(const NativeFieldSchema&
group_field,
if (fallback != nullptr &&
(!fallback->children.empty() || fallback->physical_type !=
tparquet::Type::BYTE_ARRAY ||
!valid_fallback_repetition)) {
+ // Resolve the Variant-relative path only on failure; valid
schemas need no extra
+ // per-node path allocation just to distinguish repeated wrapper
names such as element.
+ std::string path;
+ std::function<bool(const NativeFieldSchema&)> find_path =
+ [&](const NativeFieldSchema& field) {
+ const auto prefix_size = path.size();
+ if (!path.empty()) {
+ path += '.';
+ }
+ path += field.name;
+ if (&field == fallback) {
+ return true;
+ }
+ for (const auto& child : field.children) {
+ if (find_path(child)) {
+ return true;
+ }
+ }
+ path.resize(prefix_size);
+ return false;
+ };
+ find_path(group_field);
+ const auto* physical = &fallback->parquet_schema;
+ // A repeated primitive is normalized into an array whose child
retains the original
+ // SchemaElement. Report that declaration rather than the
synthetic array wrapper.
+ if (!physical->__isset.repetition_type &&
fallback->children.size() == 1 &&
+ fallback->children[0].parquet_schema.repetition_type ==
+ tparquet::FieldRepetitionType::REPEATED) {
+ physical = &fallback->children[0].parquet_schema;
+ }
return Status::Corruption(
- "Parquet Variant wrapper {} value must be an {}
BYTE_ARRAY", wrapper.name,
- allow_required_fallback ? "optional or required" :
"optional");
+ "Parquet Variant wrapper {} value must be an {}
BYTE_ARRAY; "
+ "variant_path={}, physical_type={}, repetition={},
children={}",
+ wrapper.name, allow_required_fallback ? "optional or
required" : "optional",
+ path, physical->__isset.type ?
tparquet::to_string(physical->type) : "GROUP",
+ physical->__isset.repetition_type
+ ? tparquet::to_string(physical->repetition_type)
+ : "UNSET",
+ physical->__isset.num_children ? physical->num_children :
0);
}
if (typed != nullptr) {
if (!typed->parquet_schema.__isset.repetition_type ||
diff --git a/be/src/format_v2/parquet/parquet_file_context.cpp
b/be/src/format_v2/parquet/parquet_file_context.cpp
index 72151b9aac0..6a89b363efd 100644
--- a/be/src/format_v2/parquet/parquet_file_context.cpp
+++ b/be/src/format_v2/parquet/parquet_file_context.cpp
@@ -344,9 +344,15 @@ Status ParquetFileContext::open(io::FileReaderSPtr
input_file_reader, io::IOCont
++native_footer_cache_hits;
} else {
size_t native_footer_size = 0;
- RETURN_IF_ERROR(parse_native_parquet_footer(
+ auto footer_status = parse_native_parquet_footer(
native_file, &loaded->metadata_owner, &native_footer_size,
io_ctx,
- enable_mapping_varbinary, enable_mapping_timestamp_tz,
preserve_binary_uuid));
+ enable_mapping_varbinary, enable_mapping_timestamp_tz,
preserve_binary_uuid);
+ if (!footer_status.ok()) {
+ // Schema validators have no file identity. Attach it before
leaving the file
+ // layer, preserving the original code used by callers to
classify the failure.
+ return footer_status.prepend(fmt::format("Failed to read
Parquet footer for '{}': ",
+
native_file->path().native()));
+ }
++native_footer_read_calls;
if (has_stable_meta_cache_identity && meta_cache != nullptr &&
meta_cache->enabled()) {
meta_cache->insert(meta_cache_key,
loaded->metadata_owner.release(),
diff --git a/be/test/format_v2/parquet/parquet_schema_test.cpp
b/be/test/format_v2/parquet/parquet_schema_test.cpp
index 97a10c3bd69..f51462de177 100644
--- a/be/test/format_v2/parquet/parquet_schema_test.cpp
+++ b/be/test/format_v2/parquet/parquet_schema_test.cpp
@@ -17,6 +17,9 @@
#include <gtest/gtest.h>
+#include <algorithm>
+#include <array>
+#include <cstring>
#include <limits>
#include <optional>
#include <string>
@@ -37,6 +40,10 @@
#include "format_v2/parquet/parquet_column_schema.h"
#include "format_v2/parquet/parquet_file_context.h"
#include "format_v2/parquet/reader/native_column_reader.h"
+#include "io/fs/file_reader.h"
+#include "io/io_common.h"
+#include "util/coding.h"
+#include "util/thrift_util.h"
namespace doris::format::parquet {
@@ -221,6 +228,118 @@ std::vector<tparquet::SchemaElement>
shredded_array_variant_schema(bool include_
} // namespace
+namespace {
+
+class FooterDiagnosticFileReader final : public io::FileReader {
+public:
+ explicit FooterDiagnosticFileReader(std::string bytes) :
_bytes(std::move(bytes)) {}
+ Status close() override { return Status::OK(); }
+ const io::Path& path() const override { return _path; }
+ size_t size() const override { return _bytes.size(); }
+ bool closed() const override { return false; }
+ int64_t mtime() const override { return 0; }
+
+protected:
+ Status read_at_impl(size_t offset, Slice result, size_t* bytes_read,
+ const io::IOContext*) override {
+ *bytes_read = offset < _bytes.size() ? std::min(result.size,
_bytes.size() - offset) : 0;
+ if (*bytes_read != 0) {
+ memcpy(result.data, _bytes.data() + offset, *bytes_read);
+ }
+ return Status::OK();
+ }
+
+private:
+ io::Path _path {"s3://example-bucket/nested.parquet"};
+ std::string _bytes;
+};
+
+Status open_diagnostic_footer(const std::vector<tparquet::SchemaElement>&
schema) {
+ tparquet::FileMetaData metadata;
+ metadata.__set_version(1);
+ metadata.__set_schema(schema);
+ metadata.__set_num_rows(0);
+ metadata.__set_row_groups({});
+ std::vector<uint8_t> footer;
+ ThriftSerializer serializer(true, 1024);
+ RETURN_IF_ERROR(serializer.serialize(&metadata, &footer));
+ std::string bytes = "PAR1";
+ bytes.append(reinterpret_cast<const char*>(footer.data()), footer.size());
+ std::array<uint8_t, 4> length;
+ encode_fixed32_le(length.data(), static_cast<uint32_t>(footer.size()));
+ bytes.append(reinterpret_cast<const char*>(length.data()), length.size());
+ bytes.append("PAR1");
+ auto file = std::make_shared<FooterDiagnosticFileReader>(std::move(bytes));
+ ParquetFileContext context;
+ return context.open(file, nullptr, false, io::FileDescription(), false,
false, nullptr);
+}
+
+} // namespace
+
+TEST(ParquetSchemaTest, VariantFallbackErrorReportsPathAndActualSchema) {
+ // A typed carrier still needs an optional fallback; only fallback-only
elements may require it.
+ auto schema = shredded_array_variant_schema(true, true);
+ schema[7].__set_repetition_type(tparquet::FieldRepetitionType::REQUIRED);
+ auto status = open_diagnostic_footer(schema);
+ ASSERT_TRUE(status.is<ErrorCode::CORRUPTION>()) << status;
+ EXPECT_NE(status.to_string().find("s3://example-bucket/nested.parquet"),
std::string::npos);
+
EXPECT_NE(status.to_string().find("variant_path=payload.typed_value.element.value"),
+ std::string::npos);
+ EXPECT_NE(status.to_string().find("physical_type=BYTE_ARRAY"),
std::string::npos);
+ EXPECT_NE(status.to_string().find("repetition=REQUIRED"),
std::string::npos);
+ EXPECT_NE(status.to_string().find("children=0"), std::string::npos);
+
+ schema[7].__set_repetition_type(tparquet::FieldRepetitionType::OPTIONAL);
+ ASSERT_TRUE(open_diagnostic_footer(schema).ok());
+ schema[7].__set_type(tparquet::Type::INT64);
+ status = open_diagnostic_footer(schema);
+ ASSERT_TRUE(status.is<ErrorCode::CORRUPTION>()) << status;
+ EXPECT_NE(status.to_string().find("physical_type=INT64"),
std::string::npos);
+ EXPECT_NE(status.to_string().find("repetition=OPTIONAL"),
std::string::npos);
+
+ schema[7].__set_type(tparquet::Type::BYTE_ARRAY);
+ schema[7].__set_repetition_type(tparquet::FieldRepetitionType::REPEATED);
+ status = open_diagnostic_footer(schema);
+ ASSERT_TRUE(status.is<ErrorCode::CORRUPTION>()) << status;
+ EXPECT_NE(status.to_string().find("physical_type=BYTE_ARRAY"),
std::string::npos);
+ EXPECT_NE(status.to_string().find("repetition=REPEATED"),
std::string::npos);
+}
+
+TEST(ParquetSchemaTest, VariantFallbackGroupErrorReportsActualChildren) {
+ auto schema = shredded_object_variant_schema();
+ auto child = schema[6];
+ child.__set_name("child");
+ schema[6].__isset.type = false;
+ schema[6].__set_num_children(1);
+ schema.insert(schema.begin() + 7, child);
+ const auto status = open_diagnostic_footer(schema);
+ ASSERT_TRUE(status.is<ErrorCode::CORRUPTION>()) << status;
+
EXPECT_NE(status.to_string().find("variant_path=payload.typed_value.field.value"),
+ std::string::npos);
+ EXPECT_NE(status.to_string().find("physical_type=GROUP"),
std::string::npos);
+ EXPECT_NE(status.to_string().find("children=1"), std::string::npos);
+}
+
+TEST(ParquetSchemaTest, FooterContextPreservesUnsupportedAndCorruptionCodes) {
+ auto schema = unshredded_variant_schema(2);
+ NativeFieldDescriptor descriptor;
+ const auto expected = descriptor.parse_from_thrift(schema);
+ ASSERT_FALSE(expected.ok());
+ const auto unsupported = open_diagnostic_footer(schema);
+ EXPECT_EQ(unsupported.code(), expected.code());
+ EXPECT_NE(unsupported.to_string().find(expected.msg()), std::string::npos);
+
EXPECT_NE(unsupported.to_string().find("s3://example-bucket/nested.parquet"),
+ std::string::npos);
+
+ auto file = std::make_shared<FooterDiagnosticFileReader>("PAR1");
+ ParquetFileContext context;
+ const auto corrupt =
+ context.open(file, nullptr, false, io::FileDescription(), false,
false, nullptr);
+ ASSERT_TRUE(corrupt.is<ErrorCode::CORRUPTION>()) << corrupt;
+ EXPECT_NE(corrupt.to_string().find("s3://example-bucket/nested.parquet"),
std::string::npos);
+ EXPECT_NE(corrupt.to_string().find("too small for a footer"),
std::string::npos);
+}
+
TEST(ParquetSchemaTest, NativeSchemaRecognizesVariantLogicalGroup) {
for (const auto version : {std::optional<int8_t> {}, std::optional<int8_t>
{1}}) {
NativeFieldDescriptor descriptor;
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
