This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 6b69bc4c715 [fix](be) Fail the scan when a deletion vector file is
missing (#69578)
6b69bc4c715 is described below
commit 6b69bc4c715ece6d4cfa664e4034b025c247adc6
Author: Mingyu Chen (Rayner) <[email protected]>
AuthorDate: Mon Oct 26 07:09:41 2026 +0000
[fix](be) Fail the scan when a deletion vector file is missing (#69578)
### What problem does this PR solve?
Issue Number: None
Related PR: #69571, #68142
Problem Summary:
**In short.** When a Paimon split's deletion vector file is missing, the
native readers report `NOT_FOUND`, and both file scanners take that for
a missing data file and skip the whole range. The query succeeds and
silently drops every surviving row of the data file. This PR reports a
missing deletion vector as an error, so the query fails instead. A
missing data file is still skipped as before.
**Background.** A Paimon split for a table with deletion vectors names a
data file and the deletion vector that marks its deleted rows. The
native readers open the deletion vector before they read a row: V1 in
`PaimonParquetReader` / `PaimonOrcReader::_init_deletion_vector`, V2 in
`TableReader::_parse_delete_predicates`, both through
`DeletionVectorReader`. `FileScanner` and `FileScannerV2` skip a range
whose reader reports `NOT_FOUND` while
`ignore_not_found_file_in_external_table` is on (the default). That
setting exists for file listings that are cached by the FE and may name
a file that has since been removed. #68142 already exempted Iceberg
ranges, whose files all belong to a committed snapshot.
**The problem, and what it cost.** A deletion vector is never a stale
listing: it belongs to the snapshot that planned its data file. It can
still be missing, for example when Paimon expires the snapshot that
wrote it while the data file stays live under a newer one, or when it is
looked up under a partition name that is never found (#69571). The data
file is still there, but the scanners skip the whole range, so its
surviving rows are dropped together with the deleted ones. The BE logs
`Skip missing file` with the path of the data file, which exists.
Paimon's own reader, the JNI reader and the Rust reader fail on the same
split instead.
**How this PR fixes it.** `DeletionVectorReader`, which opens every
deletion vector for V1 Paimon, V1 Iceberg and the V2 table reader, now
reports a missing file as `INTERNAL_ERROR` with the path of the deletion
vector, both when it opens the file and when it reads it. The scanners
only skip `NOT_FOUND`, so the query fails. A missing data file still
reports `NOT_FOUND` and is skipped as before, with or without a deletion
vector. For Iceberg, which already failed on any missing file (#68142),
only the error code of a missing deletion vector changes.
One boundary: if the data file is gone as well, the result depends on
which file the reader reaches first. FileScannerV2 parses the deletion
vector while it prepares the split, before it opens the data file, so
such a split now fails; in the local run below V1 Parquet failed too,
while V1 ORC, which opens the data file first, still skipped it.
**Results.** A local cluster with a `file://` Paimon catalog, two append
tables (Parquet and ORC, `deletion-vectors.enabled = true`), five rows
each, the first row deleted through a deletion vector the way a Spark
`DELETE` persists it:
| Files on storage | Reader | Before | After |
|---|---|---|---|
| data file and deletion vector | V1 / V2, Parquet / ORC | `2,3,4,5` |
`2,3,4,5` |
| data file only (deletion vector removed) | V1 / V2, Parquet / ORC |
empty result, query succeeds; `Skip missing file: path=<data file>` in
be.INFO | `[INTERNAL_ERROR]Deletion vector file is missing: path
.../index/index-...-0, failed to get file size ...: No such file or
directory` |
| deletion vector only (data file removed) | V1 / V2, Parquet / ORC |
empty result | empty result (unchanged) |
| neither | V2, V1 Parquet | empty result | the deletion vector error
above |
| neither | V1 ORC | empty result | empty result (unchanged) |
---
be/src/format/table/deletion_vector_reader.cpp | 19 ++++-
be/test/format/table/paimon_reader_test.cpp | 111 +++++++++++++++++++++++++
be/test/format_v2/table/paimon_reader_test.cpp | 97 +++++++++++++++++++++
3 files changed, 225 insertions(+), 2 deletions(-)
diff --git a/be/src/format/table/deletion_vector_reader.cpp
b/be/src/format/table/deletion_vector_reader.cpp
index 8e8404839be..853dcf549b5 100644
--- a/be/src/format/table/deletion_vector_reader.cpp
+++ b/be/src/format/table/deletion_vector_reader.cpp
@@ -63,6 +63,20 @@ Status validate_deletion_vector_read_range(const char*
description, int64_t offs
return Status::OK();
}
+// A deletion vector belongs to the snapshot that planned its data file, so a
missing one is never a
+// stale file listing. The file scanners skip a data file whose reader reports
NOT_FOUND when
+// ignore_not_found_file_in_external_table is on. Example: a Paimon split
reads data file F with
+// deletion vector D1, and D1 is gone (expired with an older snapshot, or
written under a partition
+// name that is never found) while F is still live. NOT_FOUND would skip F and
drop its surviving rows
+// together with the deleted ones, so the query must fail instead.
+Status missing_file_as_error(Status status, const std::string& path) {
+ if (status.is<ErrorCode::NOT_FOUND>()) {
+ return Status::InternalError("Deletion vector file is missing: path
{}, {}", path,
+ status.msg());
+ }
+ return status;
+}
+
} // namespace
Status validate_iceberg_deletion_vector_read_range(int64_t offset, int64_t
size,
@@ -134,7 +148,7 @@ Status DeletionVectorReader::open() {
_init_system_properties();
_init_file_description();
- RETURN_IF_ERROR(_create_file_reader());
+ RETURN_IF_ERROR(missing_file_as_error(_create_file_reader(), _desc.path));
const size_t file_size = _file_reader->size();
const size_t start_offset = static_cast<size_t>(_desc.start_offset);
@@ -157,7 +171,8 @@ Status DeletionVectorReader::read_at(size_t offset, Slice
result) {
_io_ctx->should_stop = _parent_io_ctx->should_stop;
}
size_t bytes_read = 0;
- RETURN_IF_ERROR(_file_reader->read_at(offset, result, &bytes_read,
_io_ctx));
+ RETURN_IF_ERROR(missing_file_as_error(
+ _file_reader->read_at(offset, result, &bytes_read, _io_ctx),
_desc.path));
if (bytes_read != result.size) [[unlikely]] {
return Status::IOError("Failed to read fully at offset {}, expected
{}, got {}", offset,
result.size, bytes_read);
diff --git a/be/test/format/table/paimon_reader_test.cpp
b/be/test/format/table/paimon_reader_test.cpp
new file mode 100644
index 00000000000..35d1c1932d1
--- /dev/null
+++ b/be/test/format/table/paimon_reader_test.cpp
@@ -0,0 +1,111 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "format/table/paimon_reader.h"
+
+#include <cctz/time_zone.h>
+#include <gtest/gtest.h>
+
+#include <string>
+
+#include "common/status.h"
+#include "format/format_common.h"
+#include "io/fs/file_meta_cache.h"
+#include "io/io_common.h"
+#include "runtime/runtime_profile.h"
+#include "runtime/runtime_state.h"
+#include "util/timezone_utils.h"
+
+namespace doris {
+
+namespace {
+
+TFileScanRangeParams make_local_paimon_scan_params(TFileFormatType::type
format_type) {
+ TFileScanRangeParams scan_params;
+ scan_params.__set_file_type(TFileType::FILE_LOCAL);
+ scan_params.__set_format_type(format_type);
+ return scan_params;
+}
+
+TFileRangeDesc make_paimon_range_with_deletion_file(const std::string&
deletion_file_path) {
+ TFileRangeDesc range;
+ range.__set_path("data.parquet");
+ range.__set_start_offset(0);
+ range.__set_size(0);
+ range.__set_fs_name("");
+ range.__isset.table_format_params = true;
+ range.table_format_params.__set_table_format_type("paimon");
+
+ TPaimonFileDesc paimon_params;
+ paimon_params.__set_file_format("parquet");
+ TPaimonDeletionFileDesc deletion_file;
+ deletion_file.__set_path(deletion_file_path);
+ deletion_file.__set_offset(0);
+ deletion_file.__set_length(16);
+ paimon_params.__set_deletion_file(deletion_file);
+ range.table_format_params.__set_paimon_params(paimon_params);
+ return range;
+}
+
+} // namespace
+
+// A missing deletion vector must not report NOT_FOUND: FileScanner would take
it for a missing data
+// file and skip the whole range, surviving rows included.
+TEST(PaimonDeletionVectorTest,
V1ParquetReaderFailsOnMissingDeletionVectorFile) {
+ RuntimeState runtime_state {TQueryOptions(), TQueryGlobals()};
+ RuntimeProfile profile("paimon_v1_parquet_dv_test");
+ auto scan_params =
make_local_paimon_scan_params(TFileFormatType::FORMAT_PARQUET);
+ const auto range = make_paimon_range_with_deletion_file(
+ "./be/test/exec/test_data/missing_paimon_v1_delete_vector.bin");
+ cctz::time_zone ctz;
+ TimezoneUtils::find_cctz_time_zone(TimezoneUtils::default_time_zone, ctz);
+ io::IOContext io_ctx;
+ FileMetaCache meta_cache(1024);
+ ShardedKVCache kv_cache(8);
+
+ PaimonParquetReader reader(&profile, scan_params, range, 1024, &ctz,
&kv_cache, &io_ctx,
+ &runtime_state, &meta_cache);
+ const auto status = reader.TEST_init_deletion_vector();
+
+ EXPECT_TRUE(status.is<ErrorCode::INTERNAL_ERROR>()) << status;
+
EXPECT_NE(status.to_string().find(range.table_format_params.paimon_params.deletion_file.path),
+ std::string::npos)
+ << status;
+}
+
+TEST(PaimonDeletionVectorTest, V1OrcReaderFailsOnMissingDeletionVectorFile) {
+ RuntimeState runtime_state {TQueryOptions(), TQueryGlobals()};
+ RuntimeProfile profile("paimon_v1_orc_dv_test");
+ auto scan_params =
make_local_paimon_scan_params(TFileFormatType::FORMAT_ORC);
+ auto range = make_paimon_range_with_deletion_file(
+
"./be/test/exec/test_data/missing_paimon_v1_orc_delete_vector.bin");
+ range.table_format_params.paimon_params.__set_file_format("orc");
+ io::IOContext io_ctx;
+ FileMetaCache meta_cache(1024);
+ ShardedKVCache kv_cache(8);
+
+ PaimonOrcReader reader(&profile, &runtime_state, scan_params, range, 1024,
+ TimezoneUtils::default_time_zone, &kv_cache,
&io_ctx, &meta_cache);
+ const auto status = reader.TEST_init_deletion_vector();
+
+ EXPECT_TRUE(status.is<ErrorCode::INTERNAL_ERROR>()) << status;
+
EXPECT_NE(status.to_string().find(range.table_format_params.paimon_params.deletion_file.path),
+ std::string::npos)
+ << status;
+}
+
+} // namespace doris
diff --git a/be/test/format_v2/table/paimon_reader_test.cpp
b/be/test/format_v2/table/paimon_reader_test.cpp
index bebacbfe140..466737c0810 100644
--- a/be/test/format_v2/table/paimon_reader_test.cpp
+++ b/be/test/format_v2/table/paimon_reader_test.cpp
@@ -997,6 +997,103 @@ TEST(PaimonReaderTest, AppliesBitmapDeletionVectorFile) {
std::filesystem::remove_all(test_dir);
}
+// A missing deletion vector must not report NOT_FOUND: FileScannerV2 would
take it for a missing data
+// file and skip the whole split, surviving rows included.
+TEST(PaimonReaderTest, MissingDeletionVectorFileFailsSplit) {
+ const auto test_dir =
+ std::filesystem::temp_directory_path() /
"doris_paimon_missing_deletion_vector_test";
+ std::filesystem::remove_all(test_dir);
+ std::filesystem::create_directories(test_dir);
+
+ const auto file_path = (test_dir / "split.parquet").string();
+ const auto dv_path = (test_dir / "missing-delete-vector.bin").string();
+ write_int_pair_parquet_file(file_path, {1, 2, 3}, {10, 20, 30}, {"one",
"two", "three"});
+
+ std::vector<ColumnDefinition> projected_columns;
+ projected_columns.push_back(make_table_column(0, "id",
std::make_shared<DataTypeInt32>()));
+
+ RuntimeProfile profile("test_profile");
+ RuntimeState state {TQueryOptions(), TQueryGlobals()};
+ auto scan_params = make_local_parquet_scan_params();
+ io::FileReaderStats file_reader_stats;
+ io::FileCacheStatistics file_cache_stats;
+ auto io_ctx = make_io_context(&file_reader_stats, &file_cache_stats);
+ ShardedKVCache cache(1);
+ paimon::PaimonReader reader;
+ ASSERT_TRUE(reader.init({
+ .projected_columns = projected_columns,
+ .conjuncts = {},
+ .format = FileFormat::PARQUET,
+ .scan_params = &scan_params,
+ .io_ctx = io_ctx,
+ .runtime_state = &state,
+ .scanner_profile = &profile,
+ })
+ .ok());
+
+ auto split_options = build_split_options(file_path);
+ split_options.cache = &cache;
+ split_options.current_range.__set_table_format_params(
+ make_paimon_table_format_desc(dv_path, 0, 16));
+ const auto status = reader.prepare_split(split_options);
+
+ EXPECT_TRUE(status.is<ErrorCode::INTERNAL_ERROR>()) << status;
+ EXPECT_NE(status.to_string().find(dv_path), std::string::npos) << status;
+ ASSERT_TRUE(reader.close().ok());
+ std::filesystem::remove_all(test_dir);
+}
+
+// Only the deletion vector is snapshot metadata here: a missing data file
still reports NOT_FOUND,
+// which FileScannerV2 skips under ignore_not_found_file_in_external_table.
+TEST(PaimonReaderTest, MissingDataFileWithDeletionVectorStillReportsNotFound) {
+ const auto test_dir =
+ std::filesystem::temp_directory_path() /
"doris_paimon_missing_data_file_test";
+ std::filesystem::remove_all(test_dir);
+ std::filesystem::create_directories(test_dir);
+
+ const auto file_path = (test_dir / "missing-split.parquet").string();
+ const auto dv_path = (test_dir / "delete-vector.bin").string();
+ const auto dv_length = write_paimon_deletion_vector_file(dv_path, {0});
+
+ std::vector<ColumnDefinition> projected_columns;
+ projected_columns.push_back(make_table_column(0, "id",
std::make_shared<DataTypeInt32>()));
+
+ RuntimeProfile profile("test_profile");
+ RuntimeState state {TQueryOptions(), TQueryGlobals()};
+ auto scan_params = make_local_parquet_scan_params();
+ io::FileReaderStats file_reader_stats;
+ io::FileCacheStatistics file_cache_stats;
+ auto io_ctx = make_io_context(&file_reader_stats, &file_cache_stats);
+ ShardedKVCache cache(1);
+ paimon::PaimonReader reader;
+ ASSERT_TRUE(reader.init({
+ .projected_columns = projected_columns,
+ .conjuncts = {},
+ .format = FileFormat::PARQUET,
+ .scan_params = &scan_params,
+ .io_ctx = io_ctx,
+ .runtime_state = &state,
+ .scanner_profile = &profile,
+ })
+ .ok());
+
+ SplitReadOptions split_options;
+ split_options.current_range.__set_path(file_path);
+ split_options.current_range.__set_file_size(1024);
+ split_options.cache = &cache;
+ split_options.current_range.__set_table_format_params(
+ make_paimon_table_format_desc(dv_path, 0, dv_length));
+ ASSERT_TRUE(reader.prepare_split(split_options).ok());
+
+ Block block = build_table_block(projected_columns);
+ bool eos = false;
+ const auto status = reader.get_block(&block, &eos);
+
+ EXPECT_TRUE(status.is<ErrorCode::NOT_FOUND>()) << status;
+ ASSERT_TRUE(reader.close().ok());
+ std::filesystem::remove_all(test_dir);
+}
+
TEST(PaimonHybridReaderTest, ClassifiesJniSplitByReaderType) {
EXPECT_FALSE(paimon::PaimonHybridReader::TEST_is_jni_split(
make_paimon_native_range(TFileFormatType::FORMAT_PARQUET)));
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
