This is an automated email from the ASF dual-hosted git repository.
Gabriel39 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 61906ce9a1c [fix](be) Preserve signed decimal bounds in Iceberg ORC
writes (#69888)
61906ce9a1c is described below
commit 61906ce9a1c89d76bb450f1d6b4a10acbb712efe
Author: daidai <[email protected]>
AuthorDate: Tue Nov 3 14:39:46 2026 +0800
[fix](be) Preserve signed decimal bounds in Iceberg ORC writes (#69888)
### What problem does this PR solve?
Issue Number: N/A
Related PR: N/A
Problem Summary:
ORC `Int128` statistics already contain two's-complement bits, but the
Iceberg ORC writer negates negative values again before encoding them. A
file containing `[-100, -1]` consequently reports bounds `[100, 1]`, so
a negative predicate can prune a file that contains matching rows.
ORC footer statistics also trim trailing fractional zeros. For example,
`-1.00` is returned as `value=-1, scale=0`; a `DECIMAL(...,2)` manifest
needs the unscaled value `-100`. Preserving the sign alone therefore
leaves incorrect bounds.
Restore each bound to the declared column scale with the existing ORC
scaling utility, then serialize its original signed bits using minimal
big-endian encoding. Preserve necessary sign prefixes such as `128 -> 00
80` and `-129 -> FF 7F`.
The fix applies to newly collected bounds. Existing incorrect metrics
require recomputation or data-file rewriting.
### Release note
Correct decimal bounds emitted by Iceberg ORC writes to prevent
incorrect predicate pruning for negative values and values with trailing
fractional zeros.
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [x] Regression test
- [x] Unit Test
- [x] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason
Added BE tests cover 29 encoding cases, real ORC statistics at
precisions 1/9/18/38, scale restoration through scale 38, nullable
columns, and all-null columns. The external regression checks manifest
pruning for negative and mixed decimal data across decimal32/64/128.
Local validation:
- An isolated codec harness built from the production function passed
all 29 encoding cases.
- Five real ORC files passed independent Iceberg 1.12.0 manifest
write/read and predicate-pruning probes.
- clang-format 16, build hygiene, changed-line clang-tidy, and Groovy
4.0.19 syntax checks passed. Static analysis used real ASAN UT
compilation commands with PCH expanded as headers.
- The full local BE-UT build was stopped during the monolithic
dependency build before tests executed.
- Regenerated `test_iceberg_write_stats2.out` with the standard
regression runner after restoring decimal scales. Only `sql_4/5/7/9`
changed, and the decimal bounds match both stored rows and the Parquet
metrics.
- Re-ran `test_iceberg_write_stats2` and
`test_iceberg_orc_decimal_bounds` locally using this PR's CI FE/BE
artifacts: 2 suites passed, 0 failed, 0 skipped.
- Behavior changed:
- [ ] No.
- [x] Yes. Newly written ORC decimal bounds preserve the sign and the
column scale.
- Does this need documentation?
- [x] No.
- [ ] Yes.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
---
be/src/format/transformer/vorc_transformer.cpp | 38 +++---
be/src/format/transformer/vorc_transformer.h | 2 +-
.../format/transformer/vorc_transformer_test.cpp | 150 +++++++++++++++++++++
.../iceberg/write/test_iceberg_write_stats2.out | 8 +-
.../iceberg/test_iceberg_orc_decimal_bounds.groovy | 103 ++++++++++++++
5 files changed, 275 insertions(+), 26 deletions(-)
diff --git a/be/src/format/transformer/vorc_transformer.cpp
b/be/src/format/transformer/vorc_transformer.cpp
index 842f3bfa5cf..28799eb8a63 100644
--- a/be/src/format/transformer/vorc_transformer.cpp
+++ b/be/src/format/transformer/vorc_transformer.cpp
@@ -677,32 +677,27 @@ bool VOrcTransformer::_collect_column_bounds(const
orc::ColumnStatistics* col_st
dynamic_cast<const
orc::DecimalColumnStatistics*>(col_stats)) {
if (decimal_stats->hasMinimum() && decimal_stats->hasMaximum()) {
has_bounds = true;
- (*lower_bounds)[field_id] =
_decimal_to_bytes(decimal_stats->getMinimum());
- (*upper_bounds)[field_id] =
_decimal_to_bytes(decimal_stats->getMaximum());
+ const auto scale =
static_cast<int32_t>(remove_nullable(data_type)->get_scale());
+ (*lower_bounds)[field_id] =
_decimal_to_bytes(decimal_stats->getMinimum(), scale);
+ (*upper_bounds)[field_id] =
_decimal_to_bytes(decimal_stats->getMaximum(), scale);
}
}
return has_bounds;
}
-std::string VOrcTransformer::_decimal_to_bytes(const orc::Decimal& decimal) {
- orc::Int128 val = decimal.value;
- if (val == 0) {
- char zero = 0;
- return std::string(&zero, 1);
- }
-
- // Convert Int128 -> signed big-endian minimal bytes
- bool negative = val < 0;
- auto high = static_cast<uint64_t>(val.getHighBits());
- auto low = val.getLowBits();
+std::string VOrcTransformer::_decimal_to_bytes(const orc::Decimal& decimal,
int32_t scale) {
+ // ORC footer statistics trim trailing zeros: -1.00 becomes value=-1,
scale=0.
+ // Iceberg requires the unscaled value at the column's declared scale,
which is -100 here.
+ DORIS_CHECK_GE(scale, decimal.scale);
+ bool overflow = false;
+ auto val = orc::scaleUpInt128ByPowerOfTen(decimal.value, scale -
decimal.scale, overflow);
+ DORIS_CHECK(!overflow);
- // If negative, convert to two's complement explicitly
- if (negative) {
- // two's complement for 128-bit
- low = ~low + 1;
- high = ~high + (low == 0 ? 1 : 0);
- }
+ // Int128 already exposes two's-complement bits; negating -1 again would
encode +1.
+ const bool negative = val < 0;
+ const auto high = static_cast<uint64_t>(val.getHighBits());
+ const auto low = val.getLowBits();
// Serialize to big-endian bytes
uint8_t buf[16];
@@ -711,9 +706,10 @@ std::string VOrcTransformer::_decimal_to_bytes(const
orc::Decimal& decimal) {
buf[i + 8] = static_cast<uint8_t>(low >> (56 - i * 8));
}
- // Strip leading sign-extension bytes (Iceberg minimal encoding)
+ // Strip redundant sign extension, retaining one byte for zero and the
sign prefix
+ // when the next byte has the opposite sign: 128 -> 00 80, -129 -> FF 7F.
int start = 0;
- uint8_t sign_byte = negative ? 0xFF : 0x00;
+ const uint8_t sign_byte = negative ? 0xFF : 0x00;
while (start < 15 && buf[start] == sign_byte &&
((buf[start + 1] & 0x80) == (sign_byte & 0x80))) {
++start;
diff --git a/be/src/format/transformer/vorc_transformer.h
b/be/src/format/transformer/vorc_transformer.h
index 6515439cc2c..3afd5810fb6 100644
--- a/be/src/format/transformer/vorc_transformer.h
+++ b/be/src/format/transformer/vorc_transformer.h
@@ -117,7 +117,7 @@ private:
const DataTypePtr& data_type,
std::map<int32_t, std::string>* lower_bounds,
std::map<int32_t, std::string>* upper_bounds);
- std::string _decimal_to_bytes(const orc::Decimal& decimal);
+ static std::string _decimal_to_bytes(const orc::Decimal& decimal, int32_t
scale);
std::shared_ptr<io::FileSystem> _fs = nullptr;
doris::io::FileWriter* _file_writer = nullptr;
std::vector<std::string> _column_names;
diff --git a/be/test/format/transformer/vorc_transformer_test.cpp
b/be/test/format/transformer/vorc_transformer_test.cpp
index fdf81b03161..3da58120862 100644
--- a/be/test/format/transformer/vorc_transformer_test.cpp
+++ b/be/test/format/transformer/vorc_transformer_test.cpp
@@ -17,6 +17,7 @@
#include "format/transformer/vorc_transformer.h"
+#include <fmt/format.h>
#include <gtest/gtest.h>
#include <orc/OrcFile.hh>
@@ -31,6 +32,7 @@
#include "core/column/column_vector.h"
#include "core/data_type/data_type_array.h"
#include "core/data_type/data_type_date_or_datetime_v2.h"
+#include "core/data_type/data_type_decimal.h"
#include "core/data_type/data_type_map.h"
#include "core/data_type/data_type_nullable.h"
#include "core/data_type/data_type_number.h"
@@ -48,6 +50,7 @@
#include "io/fs/local_file_system.h"
#include "runtime/runtime_state.h"
#include "testutil/mock/mock_slot_ref.h"
+#include "util/slice.h"
#include "util/timezone_utils.h"
#include "util/uid_util.h"
@@ -243,6 +246,153 @@ TEST_F(VOrcTransformerTest,
CollectsBoundsForTopLevelFieldAfterStruct) {
EXPECT_EQ("hello", stats.upper_bounds.at(3));
}
+TEST_F(VOrcTransformerTest, DecimalBoundsUseMinimalTwosComplement) {
+ struct TestCase {
+ const char* value;
+ int32_t footer_scale;
+ int32_t column_scale;
+ std::string expected;
+ };
+ const std::vector<TestCase> cases = {
+ {"-99999999999999999999999999999999999999", 0, 0,
+
std::string("\xb4\xc4\xb3\x57\xa5\x79\x3b\x85\xf6\x75\xdd\xc0\x00\x00\x00\x01",
16)},
+ {"-18446744073709551616", 0, 0,
std::string("\xff\x00\x00\x00\x00\x00\x00\x00\x00", 9)},
+ {"-9223372036854775808", 0, 0,
std::string("\x80\x00\x00\x00\x00\x00\x00\x00", 8)},
+ {"-32769", 0, 0, std::string("\xff\x7f\xff", 3)},
+ {"-32768", 0, 0, std::string("\x80\x00", 2)},
+ {"-256", 0, 0, std::string("\xff\x00", 2)},
+ {"-255", 0, 0, std::string("\xff\x01", 2)},
+ {"-129", 0, 0, std::string("\xff\x7f", 2)},
+ {"-128", 0, 0, std::string("\x80", 1)},
+ {"-100", 0, 0, std::string("\x9c", 1)},
+ {"-1", 0, 0, std::string("\xff", 1)},
+ {"0", 0, 0, std::string("\x00", 1)},
+ {"1", 0, 0, std::string("\x01", 1)},
+ {"127", 0, 0, std::string("\x7f", 1)},
+ {"128", 0, 0, std::string("\x00\x80", 2)},
+ {"255", 0, 0, std::string("\x00\xff", 2)},
+ {"256", 0, 0, std::string("\x01\x00", 2)},
+ {"9223372036854775807", 0, 0,
std::string("\x7f\xff\xff\xff\xff\xff\xff\xff", 8)},
+ {"18446744073709551616", 0, 0,
std::string("\x01\x00\x00\x00\x00\x00\x00\x00\x00", 9)},
+ {"99999999999999999999999999999999999999", 0, 0,
+
std::string("\x4b\x3b\x4c\xa8\x5a\x86\xc4\x7a\x09\x8a\x22\x3f\xff\xff\xff\xff",
16)},
+ {"-1", 0, 1, std::string("\xf6", 1)},
+ {"-1", 0, 2, std::string("\x9c", 1)},
+ {"-10", 1, 2, std::string("\x9c", 1)},
+ {"1", 0, 2, std::string("\x64", 1)},
+ {"128", 0, 2, std::string("\x32\x00", 2)},
+ {"0", 0, 38, std::string("\x00", 1)},
+ {"-1", 1, 38,
+
std::string("\xf8\x7a\x11\xef\x2a\x25\xb9\x26\xff\x0b\xc9\x60\x00\x00\x00\x00",
16)},
+ {"1", 38, 38, std::string("\x01", 1)},
+ {"-1", 38, 38, std::string("\xff", 1)},
+ };
+ for (const auto& test : cases) {
+ SCOPED_TRACE(test.value);
+ SCOPED_TRACE(test.footer_scale);
+ SCOPED_TRACE(test.column_scale);
+ const orc::Decimal decimal(orc::Int128(test.value), test.footer_scale);
+ EXPECT_EQ(test.expected, VOrcTransformer::_decimal_to_bytes(decimal,
test.column_scale));
+ }
+}
+
+TEST_F(VOrcTransformerTest, CollectsDecimalBoundsAtColumnScale) {
+ struct TestCase {
+ int precision;
+ int scale;
+ std::vector<std::string> values;
+ std::string lower;
+ std::string upper;
+ };
+ const std::vector<TestCase> cases = {
+ {1, 0, {"-9", "9"}, std::string("\xf7", 1), std::string("\x09",
1)},
+ {9, 0, {"-100", "-1"}, std::string("\x9c", 1), std::string("\xff",
1)},
+ {9, 0, {"-100", "128"}, std::string("\x9c", 1),
std::string("\x00\x80", 2)},
+ {9, 2, {"-100.00", "-1.00"}, std::string("\xd8\xf0", 2),
std::string("\x9c", 1)},
+ {18, 2, {"-100.00", "1.28"}, std::string("\xd8\xf0", 2),
std::string("\x00\x80", 2)},
+ {18, 2, {"1.00", "128.00"}, std::string("\x64", 1),
std::string("\x32\x00", 2)},
+ {38,
+ 0,
+ {"-99999999999999999999999999999999999999",
"99999999999999999999999999999999999999"},
+
std::string("\xb4\xc4\xb3\x57\xa5\x79\x3b\x85\xf6\x75\xdd\xc0\x00\x00\x00\x01",
16),
+
std::string("\x4b\x3b\x4c\xa8\x5a\x86\xc4\x7a\x09\x8a\x22\x3f\xff\xff\xff\xff",
16)},
+ {38,
+ 38,
+ {"-0.10000000000000000000000000000000000000",
+ "0.00000000000000000000000000000000000001"},
+
std::string("\xf8\x7a\x11\xef\x2a\x25\xb9\x26\xff\x0b\xc9\x60\x00\x00\x00\x00",
16),
+ std::string("\x01", 1)},
+ {38,
+ 38,
+ {"0.00000000000000000000000000000000000000"},
+ std::string("\x00", 1),
+ std::string("\x00", 1)},
+ };
+ for (const auto& test : cases) {
+ for (bool nullable : {false, true}) {
+ SCOPED_TRACE(test.precision);
+ SCOPED_TRACE(test.scale);
+ SCOPED_TRACE(nullable);
+ auto decimal_type = create_decimal(test.precision, test.scale,
false);
+ auto type = nullable ? make_nullable(decimal_type) : decimal_type;
+ auto column = type->create_column();
+ auto serde = type->get_serde();
+ DataTypeSerDe::FormatOptions options;
+ for (const auto& value : test.values) {
+ Slice slice(value);
+ ASSERT_TRUE(serde->deserialize_one_cell_from_json(*column,
slice, options).ok());
+ }
+ // A null beside the extrema must not change either decimal bound.
+ if (nullable) {
+ column->insert_default();
+ }
+ auto exprs = MockSlotRef::create_mock_contexts(DataTypes {type});
+ const auto json = fmt::format(
+
R"json({{"type":"struct","fields":[{{"id":1,"name":"d","required":false,"type":"decimal({},{})"}}]}})json",
+ test.precision, test.scale);
+ auto schema = iceberg::SchemaParser::from_json(json);
+ io::FileWriterPtr file_writer;
+ ASSERT_TRUE(_fs->create_file(_file_path, &file_writer).ok());
+ RuntimeState state;
+ VOrcTransformer transformer(&state, file_writer.get(), exprs, "",
{"d"}, false,
+ TFileCompressType::PLAIN,
schema.get(), _fs);
+ ASSERT_TRUE(transformer.open().ok());
+ ASSERT_TRUE(transformer.write(Block({{std::move(column), type,
"d"}})).ok());
+ ASSERT_TRUE(transformer.close().ok());
+ TIcebergColumnStats stats;
+
ASSERT_TRUE(transformer.collect_file_statistics_after_close(&stats).ok());
+ ASSERT_TRUE(stats.lower_bounds.contains(1));
+ ASSERT_TRUE(stats.upper_bounds.contains(1));
+ EXPECT_EQ(test.lower, stats.lower_bounds.at(1));
+ EXPECT_EQ(test.upper, stats.upper_bounds.at(1));
+ }
+ }
+}
+
+TEST_F(VOrcTransformerTest, AllNullDecimalsHaveNoBounds) {
+ auto type = make_nullable(create_decimal(38, 2, false));
+ auto column = type->create_column();
+ column->insert_default();
+ column->insert_default();
+ auto exprs = MockSlotRef::create_mock_contexts(DataTypes {type});
+ const std::string json =
+
R"json({"type":"struct","fields":[{"id":1,"name":"d","required":false,"type":"decimal(38,2)"}]})json";
+ auto schema = iceberg::SchemaParser::from_json(json);
+ io::FileWriterPtr file_writer;
+ ASSERT_TRUE(_fs->create_file(_file_path, &file_writer).ok());
+ RuntimeState state;
+ VOrcTransformer transformer(&state, file_writer.get(), exprs, "", {"d"},
false,
+ TFileCompressType::PLAIN, schema.get(), _fs);
+ ASSERT_TRUE(transformer.open().ok());
+ ASSERT_TRUE(transformer.write(Block({{std::move(column), type,
"d"}})).ok());
+ ASSERT_TRUE(transformer.close().ok());
+ TIcebergColumnStats stats;
+ ASSERT_TRUE(transformer.collect_file_statistics_after_close(&stats).ok());
+ EXPECT_FALSE(stats.lower_bounds.contains(1));
+ EXPECT_FALSE(stats.upper_bounds.contains(1));
+ EXPECT_EQ(2, stats.null_value_counts.at(1));
+}
+
TEST_F(VOrcTransformerTest, IcebergBinaryTypesOverrideLegacyStringCarrier) {
const std::string schema_json = R"({
"type": "struct",
diff --git
a/regression-test/data/external_table_p0/iceberg/write/test_iceberg_write_stats2.out
b/regression-test/data/external_table_p0/iceberg/write/test_iceberg_write_stats2.out
index df313184df9..68295992854 100644
---
a/regression-test/data/external_table_p0/iceberg/write/test_iceberg_write_stats2.out
+++
b/regression-test/data/external_table_p0/iceberg/write/test_iceberg_write_stats2.out
@@ -14,20 +14,20 @@ false 22 222 2.2 2.2 2222
8765.4321 8765.432100 987654321098765432.987654321099
true 11 111 1.1 1.1 1111 1234.5678 1234.567890
123456789012345678.123456789012 aaa 2023-01-01 2023-01-01T12:34:56
-- !sql_4 --
-0 ORC 2 {1:2, 2:2, 3:2, 4:2, 5:2, 6:2, 7:2, 8:2, 9:2, 10:2,
11:2, 12:2} {} {1:0x00, 2:0x0B00000000000000, 3:0x6F00000000000000,
4:0xCDCC8C3F, 5:0x9A9999999999F13F, 6:0x0457, 7:0x00BC614E, 8:0x075BCD15,
9:0x018EE90FF6C373E0393713FA14, 10:0x616161, 11:0x9E4B0000,
12:0x005CE70F33F10500} {1:0x01, 2:0x1600000000000000,
3:0xDE00000000000000, 4:0xCDCC0C40, 5:0x9A99999999990140, 6:0x08AE,
7:0x05397FB1, 8:0x05397FB1, 9:0x0C7748819DFFB62505316873CB, 10:0x626262,
11:0x434C0000, 12:0x40D91FA833FE0500}
+0 ORC 2 {1:2, 2:2, 3:2, 4:2, 5:2, 6:2, 7:2, 8:2, 9:2, 10:2,
11:2, 12:2} {} {1:0x00, 2:0x0B00000000000000, 3:0x6F00000000000000,
4:0xCDCC8C3F, 5:0x9A9999999999F13F, 6:0x0457, 7:0x00BC614E, 8:0x499602D2,
9:0x018EE90FF6C373E0393713FA14, 10:0x616161, 11:0x9E4B0000,
12:0x005CE70F33F10500} {1:0x01, 2:0x1600000000000000,
3:0xDE00000000000000, 4:0xCDCC0C40, 5:0x9A99999999990140, 6:0x08AE,
7:0x05397FB1, 8:0x020A75E124, 9:0x0C7748819DFFB62505316873CB, 10:0x626262,
11:0x434C0000, 12:0x40D91FA833FE0500}
-- !sql_5 --
-{"bigint_col":{"column_size":null, "value_count":2, "null_value_count":null,
"nan_value_count":null, "lower_bound":111, "upper_bound":222},
"boolean_col":{"column_size":null, "value_count":2, "null_value_count":null,
"nan_value_count":null, "lower_bound":0, "upper_bound":1},
"date_col":{"column_size":null, "value_count":2, "null_value_count":null,
"nan_value_count":null, "lower_bound":"2023-01-01",
"upper_bound":"2023-06-15"}, "datetime_col1":{"column_size":null,
"value_count":2, "null_v [...]
+{"bigint_col":{"column_size":null, "value_count":2, "null_value_count":null,
"nan_value_count":null, "lower_bound":111, "upper_bound":222},
"boolean_col":{"column_size":null, "value_count":2, "null_value_count":null,
"nan_value_count":null, "lower_bound":0, "upper_bound":1},
"date_col":{"column_size":null, "value_count":2, "null_value_count":null,
"nan_value_count":null, "lower_bound":"2023-01-01",
"upper_bound":"2023-06-15"}, "datetime_col1":{"column_size":null,
"value_count":2, "null_v [...]
-- !sql_6 --
{1:0x00, 2:0x0B000000, 3:0x6F00000000000000, 4:0xCDCC8C3F,
5:0x9A9999999999F13F, 6:0x0457, 7:0x00BC614E, 8:0x499602D2,
9:0x018EE90FF6C373E0393713FA14, 10:0x616161, 11:0x9E4B0000,
12:0x005CE70F33F10500}
-- !sql_7 --
-{1:0x00, 2:0x0B00000000000000, 3:0x6F00000000000000, 4:0xCDCC8C3F,
5:0x9A9999999999F13F, 6:0x0457, 7:0x00BC614E, 8:0x075BCD15,
9:0x018EE90FF6C373E0393713FA14, 10:0x616161, 11:0x9E4B0000,
12:0x005CE70F33F10500}
+{1:0x00, 2:0x0B00000000000000, 3:0x6F00000000000000, 4:0xCDCC8C3F,
5:0x9A9999999999F13F, 6:0x0457, 7:0x00BC614E, 8:0x499602D2,
9:0x018EE90FF6C373E0393713FA14, 10:0x616161, 11:0x9E4B0000,
12:0x005CE70F33F10500}
-- !sql_8 --
{1:0x01, 2:0x16000000, 3:0xDE00000000000000, 4:0xCDCC0C40,
5:0x9A99999999990140, 6:0x08AE, 7:0x05397FB1, 8:0x020A75E124,
9:0x0C7748819DFFB62505316873CB, 10:0x626262, 11:0x434C0000,
12:0x40D91FA833FE0500}
-- !sql_9 --
-{1:0x01, 2:0x1600000000000000, 3:0xDE00000000000000, 4:0xCDCC0C40,
5:0x9A99999999990140, 6:0x08AE, 7:0x05397FB1, 8:0x05397FB1,
9:0x0C7748819DFFB62505316873CB, 10:0x626262, 11:0x434C0000,
12:0x40D91FA833FE0500}
+{1:0x01, 2:0x1600000000000000, 3:0xDE00000000000000, 4:0xCDCC0C40,
5:0x9A99999999990140, 6:0x08AE, 7:0x05397FB1, 8:0x020A75E124,
9:0x0C7748819DFFB62505316873CB, 10:0x626262, 11:0x434C0000,
12:0x40D91FA833FE0500}
diff --git
a/regression-test/suites/external_table_p0/iceberg/test_iceberg_orc_decimal_bounds.groovy
b/regression-test/suites/external_table_p0/iceberg/test_iceberg_orc_decimal_bounds.groovy
new file mode 100644
index 00000000000..0a64c8121c6
--- /dev/null
+++
b/regression-test/suites/external_table_p0/iceberg/test_iceberg_orc_decimal_bounds.groovy
@@ -0,0 +1,103 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("test_iceberg_orc_decimal_bounds", "p0,external") {
+ String enabled = context.config.otherConfigs.get("enableIcebergTest")
+ if (enabled == null || !enabled.equalsIgnoreCase("true")) {
+ logger.info("disable iceberg test.")
+ return
+ }
+
+ String rest_port = context.config.otherConfigs.get("iceberg_rest_uri_port")
+ String minio_port = context.config.otherConfigs.get("iceberg_minio_port")
+ String externalEnvIp = context.config.otherConfigs.get("externalEnvIp")
+ String catalog_name = "test_iceberg_orc_decimal_bounds"
+ String db_name = "test_iceberg_orc_decimal_bounds_db"
+
+ sql """drop catalog if exists ${catalog_name}"""
+ sql """create catalog ${catalog_name} properties (
+ 'type' = 'iceberg',
+ 'iceberg.catalog.type' = 'rest',
+ 'uri' = 'http://${externalEnvIp}:${rest_port}',
+ 's3.access_key' = 'admin',
+ 's3.secret_key' = 'password',
+ 's3.endpoint' = 'http://${externalEnvIp}:${minio_port}',
+ 's3.region' = 'us-east-1'
+ )"""
+ sql """create database if not exists ${catalog_name}.${db_name}"""
+ sql """use ${catalog_name}.${db_name}"""
+ // EXPLAIN must finish planning splits so it exposes manifest pruning.
+ sql "set enable_external_table_batch_mode = false"
+
+ sql "drop table if exists orc_decimal_bounds_negative"
+ sql """create table orc_decimal_bounds_negative (
+ id int,
+ d32 decimal(9, 0),
+ d64 decimal(18, 2),
+ d128 decimal(38, 2)
+ ) properties ('write-format' = 'orc')"""
+ sql """insert into orc_decimal_bounds_negative values
+ (1, -100, -100.00, -999999999999999999999999999999999999.99),
+ (2, -1, -1.00, -1.00)"""
+
+ ["d32", "d64", "d128"].each { column ->
+ // Negating the footer's two's-complement value makes both bounds
positive
+ // and incorrectly excludes this file from every negative range.
+ explain {
+ sql("select id from orc_decimal_bounds_negative where ${column} <
0")
+ contains "inputSplitNum=1"
+ }
+ explain {
+ sql("select id from orc_decimal_bounds_negative where ${column} >
0")
+ contains "inputSplitNum=0"
+ }
+ }
+ // ORC serializes -100.00 as value=-100, scale=0. Fixing only its sign
+ // still leaves -1.00 in the manifest unless the column scale is restored.
+ explain {
+ sql("select id from orc_decimal_bounds_negative where d64 = -100.00")
+ contains "inputSplitNum=1"
+ }
+
+ sql "drop table if exists orc_decimal_bounds_mixed"
+ sql """create table orc_decimal_bounds_mixed (
+ id int,
+ d32 decimal(9, 0),
+ d64 decimal(18, 2),
+ d128 decimal(38, 2)
+ ) properties ('write-format' = 'orc')"""
+ sql """insert into orc_decimal_bounds_mixed values
+ (1, -100, -100.00, -100.00),
+ (2, 128, 128.00, 128.00),
+ (3, null, null, null)"""
+
+ ["d32", "d64", "d128"].each { column ->
+ explain {
+ sql("select id from orc_decimal_bounds_mixed where ${column} =
-100")
+ contains "inputSplitNum=1"
+ }
+ // Positive values with trailing zeros must retain their scale too.
+ explain {
+ sql("select id from orc_decimal_bounds_mixed where ${column} =
128")
+ contains "inputSplitNum=1"
+ }
+ explain {
+ sql("select id from orc_decimal_bounds_mixed where ${column} >
128")
+ contains "inputSplitNum=0"
+ }
+ }
+}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
