furquan39 commented on issue #66298:
URL: https://github.com/apache/doris/issues/66298#issuecomment-5130413629
Answers to the requested items below.
## Corrections to our report
We withdraw the claim in "How to Reproduce" that long plain STRING columns
reproduce the same segment-wide failure. Our reproduction is VARIANT-only, and
we have no E-3113 stack trace from a plain STRING column. We accept the
analysis that `ScalarColumnWriter` writes strings incrementally into pages, so
the segment-wide accumulation mechanism we described does not apply to that
path. If we ever hit a STRING-path overflow we will report it separately with
its own DDL and stack.
## New finding since filing: failures persist after lowering the segment cap
8x
The triage analysis states that fixing the estimator is a sizing guard, not
a proof of safety. Our data now supports that directly. After the original
failures at the default `vertical_compaction_max_segment_size = 268435456`, we
lowered it to `33554432` (32 MiB, an 8x reduction). E-3113 failures continue
with the same signature: every failure quoted below is from 2026-07-30 under
the 32 MiB cap, still tripping just past 2^32 uncompressed bytes (total_length
approximately 4.295 to 4.306 GB) at roughly 860k to 911k rows.
Our interpretation, offered for what it is worth: recompacted data
compresses better than freshly loaded data, so a row estimator denominated in
compressed bytes admits more rows per segment as compaction proceeds. Any
static compressed-size cap is therefore chasing a moving target. This matches
your conclusion that the writer needs an independent enforcement of the
uncompressed in-memory invariant (streaming/chunked writing) rather than a
sizing-side fix alone.
## 1. Full symbolized stack trace
A fully symbolized stack for the E-3113 throw is logged into be.WARNING by
the exception machinery (the release build logs the numeric code E-3113, not
the enum name; `grep STRING_OVERFLOW` returns 0 matches on all three BEs). The
BE does not crash on E-3113, so be.out contains only startup output and no
stack.
Identical stacks (same 26 frames) accompany every E-3113 occurrence on all
three BEs; only the total_length / element_number / rows values differ.
<details>
<summary>26-frame symbolized stack, verbatim from be.WARNING (BE-0)</summary>
```
RuntimeLogger W20260730 06:26:57.844934 1309 tablet.cpp:1889] failed to do
cumulative compaction, tablet=1785247989947 : [E-3113][E-3113] string column
length is too large: total_length=4295027788, element_number=870034, rows=869477
0# doris::Exception::Exception(int, std::basic_string_view<char,
std::char_traits<char> > const&, bool) at
/home/zcp/repo_center/doris_release/doris/be/src/common/exception.cpp:0
1# doris::Exception::Exception(int, std::basic_string_view<char,
std::char_traits<char> > const&) at
/usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/basic_string.h:239
2# doris::Exception::Exception<unsigned long&, unsigned long&,
unsigned long&>(int, std::basic_string_view<char, std::char_traits<char> >
const&, unsigned long&, unsigned long&, unsigned long&) at
/usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/basic_string.h:239
3# auto doris::ColumnStr<unsigned
int>::insert_range_from(doris::IColumn const&, unsigned long, unsigned
long)::{lambda(auto:1 const&)#1}::operator()<doris::ColumnStr<unsigned int>
>(doris::ColumnStr<unsigned int> const&) const at
/home/zcp/repo_center/doris_release/doris/be/src/core/column/column_string.cpp:0
4# doris::ColumnStr<unsigned int>::insert_range_from(doris::IColumn
const&, unsigned long, unsigned long) at
/home/zcp/repo_center/doris_release/doris/be/src/core/column/column_string.cpp:213
5#
doris::ColumnVariant::Subcolumn::insert_range_from(doris::ColumnVariant::Subcolumn
const&, unsigned long, unsigned long) at
/home/zcp/repo_center/doris_release/doris/be/src/core/column/column_variant.cpp:457
6# doris::ColumnVariant::insert_range_from(doris::IColumn const&,
unsigned long, unsigned long) at
/home/zcp/repo_center/doris_release/doris/be/src/core/column/column_variant.cpp:1193
7# doris::segment_v2::VariantColumnWriterImpl::append_data(unsigned
char const**, unsigned long) at
/home/zcp/repo_center/doris_release/doris/be/src/common/status.h:370
8#
doris::segment_v2::VariantColumnWriterImpl::append_nullable(unsigned char
const*, unsigned char const**, unsigned long) at
/home/zcp/repo_center/doris_release/doris/be/src/common/status.h:526
9# doris::segment_v2::VariantColumnWriter::append_nullable(unsigned
char const*, unsigned char const**, unsigned long) at
/home/zcp/repo_center/doris_release/doris/be/src/storage/segment/column_writer.cpp:1396
10# doris::segment_v2::ColumnWriter::append(unsigned char const*, void
const*, unsigned long) at
/home/zcp/repo_center/doris_release/doris/be/src/storage/segment/column_writer.cpp:466
11# doris::segment_v2::SegmentWriter::append_block(doris::Block const*,
unsigned long, unsigned long) at
/home/zcp/repo_center/doris_release/doris/be/src/storage/segment/segment_writer.cpp:0
12#
doris::VerticalBetaRowsetWriter<doris::BetaRowsetWriter>::add_columns(doris::Block
const*, std::vector<unsigned int, std::allocator<unsigned int> > const&, bool,
unsigned int, bool) at
/home/zcp/repo_center/doris_release/doris/be/src/common/status.h:526
13#
doris::Merger::vertical_compact_one_group(std::shared_ptr<doris::BaseTablet>,
doris::ReaderType, doris::TabletSchema const&, bool, std::vector<unsigned int,
std::allocator<unsigned int> > const&, doris::RowSourcesBuffer*,
std::vector<std::shared_ptr<doris::RowsetReader>,
std::allocator<std::shared_ptr<doris::RowsetReader> > > const&,
doris::RowsetWriter*, unsigned int, doris::Merger::Statistics*,
std::vector<unsigned int, std::allocator<unsigned int> >, long,
doris::CompactionSampleInfo*, bool) at
/home/zcp/repo_center/doris_release/doris/be/src/storage/merger.cpp:0
14#
doris::Merger::vertical_merge_rowsets(std::shared_ptr<doris::BaseTablet>,
doris::ReaderType, doris::TabletSchema const&,
std::vector<std::shared_ptr<doris::RowsetReader>,
std::allocator<std::shared_ptr<doris::RowsetReader> > > const&,
doris::RowsetWriter*, unsigned int, long, doris::Merger::Statistics*) at
/home/zcp/repo_center/doris_release/doris/be/src/storage/merger.cpp:557
15# doris::Compaction::merge_input_rowsets() at
/home/zcp/repo_center/doris_release/doris/be/src/storage/compaction/compaction.cpp:0
16# doris::CompactionMixin::execute_compact_impl(long) at
/home/zcp/repo_center/doris_release/doris/be/src/common/status.h:526
17# doris::CompactionMixin::execute_compact() at
/home/zcp/repo_center/doris_release/doris/be/src/common/status.h:526
18# doris::CumulativeCompaction::execute_compact() at
/home/zcp/repo_center/doris_release/doris/be/src/storage/compaction/cumulative_compaction.cpp:0
19# doris::Tablet::execute_compaction(doris::CompactionMixin&) at
/home/zcp/repo_center/doris_release/doris/be/src/storage/tablet/tablet.cpp:1885
20#
doris::StorageEngine::_handle_compaction(std::shared_ptr<doris::Tablet>,
std::shared_ptr<doris::CompactionMixin>, doris::CompactionType, long, bool) at
/home/zcp/repo_center/doris_release/doris/be/src/storage/olap_server.cpp:0
21# std::_Function_handler<void (),
doris::StorageEngine::_submit_compaction_task(std::shared_ptr<doris::Tablet>,
doris::CompactionType, bool)::$_0>::_M_invoke(std::_Any_data const&) at
/home/zcp/repo_center/doris_release/doris/be/src/storage/olap_server.cpp:0
22# doris::ThreadPool::dispatch_thread() at
/usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/shared_ptr_base.h:1097
23# doris::Thread::supervise_thread(void*) at
/usr/local/ldb-toolchain-v0.26/bin/../usr/include/pthread.h:562
24# ?
25# clone
```
</details>
The frames between `ColumnStr::insert_range_from` and the compaction entry
point are therefore:
```
Merger::vertical_compact_one_group -> VerticalBetaRowsetWriter::add_columns
-> SegmentWriter::append_block -> ColumnWriter::append
-> VariantColumnWriter::append_nullable ->
VariantColumnWriterImpl::append_nullable
-> VariantColumnWriterImpl::append_data -> ColumnVariant::insert_range_from
-> ColumnVariant::Subcolumn::insert_range_from ->
ColumnStr<uint32>::insert_range_from (column_string.cpp:213)
```
This is the non-streaming `VariantColumnWriterImpl::append_data`
accumulation path you identified as the trigger candidate.
## 2. SHOW CREATE TABLE (VARIANT properties)
Three VARIANT columns, each with `variant_max_subcolumns_count = 2048`,
`variant_enable_typed_paths_to_sparse = false`,
`variant_max_sparse_column_statistics_size = 10000`,
`variant_sparse_hash_shard_count = 0`. No doc-mode or nested-group properties
beyond what is shown. DUPLICATE KEY, replication 1.
<details>
<summary>SHOW CREATE TABLE, verbatim</summary>
```sql
CREATE TABLE `evaluated_docs_scale` (
`created_at_short` date NOT NULL,
`document_id` varchar(64) NULL,
`customer` varchar(128) NULL,
`product` varchar(128) NULL,
`market` varchar(32) NULL,
`created_at` datetime(3) NULL,
`last_updated_at` datetime(3) NULL,
`category` array<varchar(128)> NULL,
`doc_lang` varchar(16) NULL,
`document` variant<PROPERTIES ("variant_max_subcolumns_count" =
"2048","variant_enable_typed_paths_to_sparse" =
"false","variant_max_sparse_column_statistics_size" =
"10000","variant_sparse_hash_shard_count" = "0")> NULL,
`evaluation_summary` variant<PROPERTIES ("variant_max_subcolumns_count" =
"2048","variant_enable_typed_paths_to_sparse" =
"false","variant_max_sparse_column_statistics_size" =
"10000","variant_sparse_hash_shard_count" = "0")> NULL,
`label_details` variant<PROPERTIES ("variant_max_subcolumns_count" =
"2048","variant_enable_typed_paths_to_sparse" =
"false","variant_max_sparse_column_statistics_size" =
"10000","variant_sparse_hash_shard_count" = "0")> NULL,
`kafka_topic` varchar(255) NULL,
`kafka_partition` int NULL,
`kafka_offset` bigint NULL,
`ingested_at` datetime(3) NULL DEFAULT CURRENT_TIMESTAMP,
`usage_status` array<varchar(64)> NULL,
`amounts_are_consistent` boolean NULL
) ENGINE=OLAP
DUPLICATE KEY(`created_at_short`, `document_id`)
AUTO PARTITION BY RANGE (date_trunc(`created_at_short`, 'month'))
(PARTITION p20260501000000 VALUES [('2026-05-01'), ('2026-06-01')),
PARTITION p20260601000000 VALUES [('2026-06-01'), ('2026-07-01')),
PARTITION p20260701000000 VALUES [('2026-07-01'), ('2026-08-01')))
DISTRIBUTED BY HASH(`document_id`) BUCKETS 16
PROPERTIES (
"replication_allocation" = "tag.location.default: 1",
"min_load_replica_num" = "-1",
"is_being_synced" = "false",
"storage_medium" = "hdd",
"storage_format" = "V2",
"inverted_index_storage_format" = "V3",
"light_schema_change" = "true",
"disable_auto_compaction" = "false",
"enable_single_replica_compaction" = "false",
"group_commit_interval_ms" = "10000",
"group_commit_data_bytes" = "134217728"
);
```
</details>
## 3. BE settings
Via `/api/show_config`, identical on all three BEs:
```
[["enable_vertical_compaction","bool","true","true"]]
[["enable_vertical_compact_variant_subcolumns","bool","true","true"]]
[["vertical_compaction_num_columns_per_group","int32_t","5","true"]]
[["compaction_batch_size","int64_t","-1","true"]]
[["vertical_compaction_max_segment_size","int64_t","33554432","true"]]
```
Timeline for `vertical_compaction_max_segment_size`: the first E-3113
failures occurred at the default `268435456`; the current `33554432` is a
mitigation we applied ourselves, and the failures documented below occurred
under this lowered value (see the "New finding" section above).
## 4. Failing compaction input statistics and max_rows_per_segment
Traced end to end for tablet 1785247989988 (BE-2). The failing run merged 99
input rowsets (`permits: 99` matching `way cnt: 99`), started 08:26:04, threw
E-3113 35 seconds later, and two retries reproduced byte-identical numbers, so
the failure is deterministic per tablet:
```
RuntimeLogger I20260730 08:26:04.372887 1311 compaction.cpp:589] start
cumulative compaction. tablet=1785247989988, output_version=[149-699], permits:
99
RuntimeLogger I20260730 08:26:04.373230 1311 merger.cpp:478] Start to do
vertical compaction, tablet_id: 1785247989988
RuntimeLogger W20260730 08:26:39.689736 1311 tablet.cpp:1889] failed to do
cumulative compaction, tablet=1785247989988 : [E-3113][E-3113] string column
length is too large: total_length=4305822753, element_number=881085, rows=878820
RuntimeLogger I20260730 08:30:34.625535 1314 compaction.cpp:589] start
cumulative compaction. tablet=1785247989988, output_version=[149-699], permits:
99
RuntimeLogger I20260730 08:35:32.389564 1311 compaction.cpp:589] start
cumulative compaction. tablet=1785247989988, output_version=[149-699], permits:
99
RuntimeLogger W20260730 08:36:01.855674 1311 tablet.cpp:1889] failed to do
cumulative compaction, tablet=1785247989988 : [E-3113][E-3113] string column
length is too large: total_length=4305822753, element_number=881085, rows=878820
```
Immediately preceding successful compactions of the same tablet, for input
scale context (per-input-segment row counts are not individually logged by this
build; these are the logged aggregates):
```
RuntimeLogger I20260730 08:15:43.729545 1314 compaction.cpp:601] succeed to
do cumulative compaction is_vertical=1. tablet=1785247989988,
output_version=[2-148], current_max_version=1690,
disk=/opt/apache-doris/be/storage, input_segments=105,
input_rowsets_data_size=180.13 MB, input_rowsets_index_size=0,
input_rowsets_total_size=180.13 MB, output_rowset_data_size=139.83 MB,
output_rowset_index_size=0, output_rowset_total_size=139.83 MB,
input_row_num=3176496, output_row_num=3176496, filtered_row_num=0,
merged_row_num=0. elapsed time=33.6225s.
cumulative_compaction_policy=size_based, compact_row_per_second=94475.2
RuntimeLogger I20260730 08:22:25.033587 1314 compaction.cpp:601] succeed to
do cumulative compaction is_vertical=1. tablet=1785247989988,
output_version=[149-650], current_max_version=1690,
disk=/opt/apache-doris/be/storage, input_segments=999,
input_rowsets_data_size=910.46 MB, input_rowsets_index_size=0,
input_rowsets_total_size=910.46 MB, output_rowset_data_size=18.85 MB,
output_rowset_index_size=0, output_rowset_total_size=18.85 MB,
input_row_num=10335472, output_row_num=10335472, filtered_row_num=0,
merged_row_num=0. elapsed time=100.446s.
cumulative_compaction_policy=size_based, compact_row_per_second=102896
```
`max_rows_per_segment`: `grep max_rows_per_segment be.INFO be.WARNING`
returns 0 matches on all three BEs; nothing is logged under that name in this
build. The closest observable sizing signal is the `merger.cpp:460` "estimate
batch size" lines, quoted in full for the failing run:
<details>
<summary>Complete estimate-batch-size sequence for the failing run (08:26:04
to 08:26:39)</summary>
```
RuntimeLogger I20260730 08:26:04.373250 1311 merger.cpp:460] estimate batch
size for vertical compaction, tablet id: 1785247989988 group data size: 0 row
num: 3265343 consume bytes: 146940435 way cnt: 99 batch size: 4064
RuntimeLogger I20260730 08:26:07.448575 1311 merger.cpp:460] estimate batch
size for vertical compaction, tablet id: 1785247989988 group data size: 0 row
num: 3265343 consume bytes: 107756319 way cnt: 99 batch size: 4064
RuntimeLogger I20260730 08:26:07.717113 1311 merger.cpp:460] estimate batch
size for vertical compaction, tablet id: 1785247989988 group data size: 0 row
num: 3265343 consume bytes: 218153954 way cnt: 99 batch size: 4064
RuntimeLogger I20260730 08:26:08.933276 1311 merger.cpp:460] estimate batch
size for vertical compaction, tablet id: 1785247989988 group data size: 0 row
num: 3265343 consume bytes: 190677387 way cnt: 99 batch size: 4064
RuntimeLogger I20260730 08:26:09.545127 1311 merger.cpp:460] estimate batch
size for vertical compaction, tablet id: 1785247989988 group data size: 0 row
num: 3265343 consume bytes: 106451362 way cnt: 99 batch size: 4064
RuntimeLogger I20260730 08:26:10.001435 1311 merger.cpp:460] estimate batch
size for vertical compaction, tablet id: 1785247989988 group data size: 0 row
num: 3265343 consume bytes: 94694947 way cnt: 99 batch size: 4064
RuntimeLogger I20260730 08:26:10.490581 1311 merger.cpp:460] estimate batch
size for vertical compaction, tablet id: 1785247989988 group data size: 0 row
num: 3265343 consume bytes: 101117135 way cnt: 99 batch size: 4064
RuntimeLogger I20260730 08:26:10.976086 1311 merger.cpp:460] estimate batch
size for vertical compaction, tablet id: 1785247989988 group data size: 0 row
num: 3265343 consume bytes: 220407766 way cnt: 99 batch size: 4064
RuntimeLogger I20260730 08:26:11.629590 1311 merger.cpp:460] estimate batch
size for vertical compaction, tablet id: 1785247989988 group data size: 0 row
num: 3265343 consume bytes: 207379839 way cnt: 99 batch size: 4064
RuntimeLogger I20260730 08:26:12.194929 1311 merger.cpp:460] estimate batch
size for vertical compaction, tablet id: 1785247989988 group data size: 0 row
num: 3265343 consume bytes: 146940435 way cnt: 99 batch size: 4064
RuntimeLogger I20260730 08:26:12.430644 1311 merger.cpp:460] estimate batch
size for vertical compaction, tablet id: 1785247989988 group data size: 0 row
num: 3265343 consume bytes: 107756319 way cnt: 99 batch size: 4064
RuntimeLogger I20260730 08:26:12.806689 1311 merger.cpp:460] estimate batch
size for vertical compaction, tablet id: 1785247989988 group data size: 0 row
num: 1055943 consume bytes: 1071192143 way cnt: 99 batch size: 4064
RuntimeLogger I20260730 08:26:21.952812 1311 merger.cpp:460] estimate batch
size for vertical compaction, tablet id: 1785247989988 group data size: 0 row
num: 3265343 consume bytes: 83400983 way cnt: 99 batch size: 4064
RuntimeLogger I20260730 08:26:22.098752 1311 merger.cpp:460] estimate batch
size for vertical compaction, tablet id: 1785247989988 group data size: 0 row
num: 3265343 consume bytes: 199163780 way cnt: 99 batch size: 4064
RuntimeLogger I20260730 08:26:22.817622 1311 merger.cpp:460] estimate batch
size for vertical compaction, tablet id: 1785247989988 group data size: 0 row
num: 3265343 consume bytes: 102918201 way cnt: 99 batch size: 4064
RuntimeLogger I20260730 08:26:23.037478 1311 merger.cpp:460] estimate batch
size for vertical compaction, tablet id: 1785247989988 group data size: 0 row
num: 3265343 consume bytes: 134173631 way cnt: 99 batch size: 4064
RuntimeLogger I20260730 08:26:23.270869 1311 merger.cpp:460] estimate batch
size for vertical compaction, tablet id: 1785247989988 group data size: 0 row
num: 3265343 consume bytes: 103887809 way cnt: 99 batch size: 4064
RuntimeLogger I20260730 08:26:23.795578 1311 merger.cpp:460] estimate batch
size for vertical compaction, tablet id: 1785247989988 group data size: 0 row
num: 3265343 consume bytes: 118738460 way cnt: 99 batch size: 4064
RuntimeLogger I20260730 08:26:24.537241 1311 merger.cpp:460] estimate batch
size for vertical compaction, tablet id: 1785247989988 group data size: 0 row
num: 3265343 consume bytes: 212871507 way cnt: 99 batch size: 4064
RuntimeLogger I20260730 08:26:25.422828 1311 merger.cpp:460] estimate batch
size for vertical compaction, tablet id: 1785247989988 group data size: 0 row
num: 3265343 consume bytes: 296487543 way cnt: 99 batch size: 4064
RuntimeLogger I20260730 08:26:26.256064 1311 merger.cpp:460] estimate batch
size for vertical compaction, tablet id: 1785247989988 group data size: 0 row
num: 3265343 consume bytes: 146940435 way cnt: 99 batch size: 4064
RuntimeLogger I20260730 08:26:26.573335 1311 merger.cpp:460] estimate batch
size for vertical compaction, tablet id: 1785247989988 group data size: 0 row
num: 3265343 consume bytes: 133879063 way cnt: 99 batch size: 4064
RuntimeLogger I20260730 08:26:26.927311 1311 merger.cpp:460] estimate batch
size for vertical compaction, tablet id: 1785247989988 group data size: 0 row
num: 3265343 consume bytes: 185078226 way cnt: 99 batch size: 4064
RuntimeLogger I20260730 08:26:27.805542 1311 merger.cpp:460] estimate batch
size for vertical compaction, tablet id: 1785247989988 group data size: 0 row
num: 2662557 consume bytes: 955077800 way cnt: 99 batch size: 4064
RuntimeLogger I20260730 08:26:33.426851 1311 merger.cpp:460] estimate batch
size for vertical compaction, tablet id: 1785247989988 group data size: 0 row
num: 220779 consume bytes: 1057232779 way cnt: 99 batch size: 2265
```
</details>
Reading of the sequence: most column groups are cheap (roughly 100 to 300 MB
consumed for 3.27M rows, batch size capped at 4064). The VARIANT groups are the
heavy ones (`consume bytes: 955077800` and `1057232779`, about 1 GB per batch),
and on the last group the estimator cuts the batch to 2265 rows. About six
seconds after that final line the writer throws E-3113 at
`total_length=4305822753` (just past 2^32) after 878,820 rows: the per-segment
row cap derived from the estimator/segment-size logic still allows a single
VARIANT subcolumn's `ColumnStr<uint32>` offsets to exceed 4 GiB inside one
output segment build.
Corpus statistics behind the compression ratio (from `SHOW DATA` and `SHOW
TABLETS`):
```
Size: 135.037 GB RowCount: 2150260716 (48 tablets, replication 1)
```
That is about 67 bytes/row compressed on disk. A previously sampled
`AVG(LENGTH(CAST(document AS STRING)))` measured about 7280 bytes/row for the
`document` VARIANT column alone, an effective compression ratio on the order of
100x. That ratio is why compressed-byte sizing admits roughly 4.7 to 5.0 KB of
decompressed string data per row times about 900k rows into one segment.
<details>
<summary>All 23 E-3113 failure lines from 2026-07-30 across the three BEs
(verbatim)</summary>
```
BE-0:
RuntimeLogger W20260730 06:24:04.065166 1303 tablet.cpp:1889] failed to do
cumulative compaction, tablet=1785247989955 : [E-3113][E-3113] string column
length is too large: total_length=4298331471, element_number=875139, rows=874540
RuntimeLogger W20260730 06:24:29.481729 1306 tablet.cpp:1889] failed to do
cumulative compaction, tablet=1785247989982 : [E-3113][E-3113] string column
length is too large: total_length=4295454002, element_number=910938, rows=910364
RuntimeLogger W20260730 06:24:32.919359 1303 tablet.cpp:1889] failed to do
cumulative compaction, tablet=1785247989980 : [E-3113][E-3113] string column
length is too large: total_length=4296959620, element_number=893018, rows=892456
RuntimeLogger W20260730 06:25:08.129467 1306 tablet.cpp:1889] failed to do
cumulative compaction, tablet=1785247989992 : [E-3113][E-3113] string column
length is too large: total_length=4295980692, element_number=908820, rows=908259
RuntimeLogger W20260730 06:25:30.369625 1309 tablet.cpp:1889] failed to do
cumulative compaction, tablet=1785247989943 : [E-3113][E-3113] string column
length is too large: total_length=4295803300, element_number=868118, rows=867564
RuntimeLogger W20260730 06:26:10.158970 1305 tablet.cpp:1889] failed to do
cumulative compaction, tablet=1785247989941 : [E-3113][E-3113] string column
length is too large: total_length=4295476064, element_number=859360, rows=858836
RuntimeLogger W20260730 06:26:57.844934 1309 tablet.cpp:1889] failed to do
cumulative compaction, tablet=1785247989947 : [E-3113][E-3113] string column
length is too large: total_length=4295027788, element_number=870034, rows=869477
BE-1:
RuntimeLogger W20260730 06:23:02.893852 1312 tablet.cpp:1889] failed to do
cumulative compaction, tablet=1785247989984 : [E-3113][E-3113] string column
length is too large: total_length=4297246108, element_number=895980, rows=895435
RuntimeLogger W20260730 06:23:29.057267 1314 tablet.cpp:1889] failed to do
cumulative compaction, tablet=1785247989971 : [E-3113][E-3113] string column
length is too large: total_length=4295895212, element_number=863993, rows=863460
RuntimeLogger W20260730 06:23:50.456548 1312 tablet.cpp:1889] failed to do
cumulative compaction, tablet=1785247989957 : [E-3113][E-3113] string column
length is too large: total_length=4295421282, element_number=869375, rows=868840
RuntimeLogger W20260730 06:24:03.853626 1314 tablet.cpp:1889] failed to do
cumulative compaction, tablet=1785247989994 : [E-3113][E-3113] string column
length is too large: total_length=4295666480, element_number=911200, rows=910656
RuntimeLogger W20260730 06:24:47.159302 1313 tablet.cpp:1889] failed to do
cumulative compaction, tablet=1785247990011 : [E-3113][E-3113] string column
length is too large: total_length=4296424418, element_number=902266, rows=899999
BE-2:
RuntimeLogger W20260730 06:25:29.299242 1313 tablet.cpp:1889] failed to do
cumulative compaction, tablet=1785247989998 : [E-3113][E-3113] string column
length is too large: total_length=4296799270, element_number=906399, rows=905850
RuntimeLogger W20260730 06:25:38.449867 1311 tablet.cpp:1889] failed to do
cumulative compaction, tablet=1785247989949 : [E-3113][E-3113] string column
length is too large: total_length=4297775108, element_number=884906, rows=884355
RuntimeLogger W20260730 06:26:03.206351 1317 tablet.cpp:1889] failed to do
cumulative compaction, tablet=1785247990009 : [E-3113][E-3113] string column
length is too large: total_length=4297841904, element_number=898064, rows=897472
RuntimeLogger W20260730 06:26:03.209679 1313 tablet.cpp:1889] failed to do
cumulative compaction, tablet=1785247990015 : [E-3113][E-3113] string column
length is too large: total_length=4295142356, element_number=887932, rows=887376
RuntimeLogger W20260730 06:26:21.955569 1309 tablet.cpp:1889] failed to do
cumulative compaction, tablet=1785247990004 : [E-3113][E-3113] string column
length is too large: total_length=4296834998, element_number=895950, rows=895400
RuntimeLogger W20260730 06:26:30.077806 1317 tablet.cpp:1889] failed to do
cumulative compaction, tablet=1785247989951 : [E-3113][E-3113] string column
length is too large: total_length=4297725673, element_number=858663, rows=858126
RuntimeLogger W20260730 06:26:48.198082 1308 tablet.cpp:1889] failed to do
cumulative compaction, tablet=1785247990021 : [E-3113][E-3113] string column
length is too large: total_length=4296363367, element_number=897195, rows=896660
RuntimeLogger W20260730 06:26:57.292194 1313 tablet.cpp:1889] failed to do
cumulative compaction, tablet=1785247990023 : [E-3113][E-3113] string column
length is too large: total_length=4295638636, element_number=901460, rows=900912
RuntimeLogger W20260730 06:27:24.741868 1313 tablet.cpp:1889] failed to do
cumulative compaction, tablet=1785247990013 : [E-3113][E-3113] string column
length is too large: total_length=4297936225, element_number=905261, rows=904708
RuntimeLogger W20260730 08:26:39.689736 1311 tablet.cpp:1889] failed to do
cumulative compaction, tablet=1785247989988 : [E-3113][E-3113] string column
length is too large: total_length=4305822753, element_number=881085, rows=878820
RuntimeLogger W20260730 08:36:01.855674 1311 tablet.cpp:1889] failed to do
cumulative compaction, tablet=1785247989988 : [E-3113][E-3113] string column
length is too large: total_length=4305822753, element_number=881085, rows=878820
```
</details>
## 5. Plain STRING reproduction
Withdrawn; see the corrections section. We have no plain-STRING DDL, error
line, or stack trace to offer, and we will not claim that scope again without
one.
## Gaps and follow-ups
- The failure lines from the original 256 MiB era rotated out of be.WARNING
when the BEs were restarted on 2026-07-28. Every line quoted above is from
2026-07-30 under the lowered 32 MiB cap, which itself demonstrates that
lowering the compressed-size cap does not close the defect.
- We are preparing a hardened, controlled rerun environment. From it we can
provide anything the retained logs cannot, on request: failure lines at the
default cap, per-input rowset breakdowns, or additional traces at other cap
values.
- Patch validation offer: the reproduction corpus is 145 GB / 2.15 billion
rows of high-duplication VARIANT data on which the failure is deterministic per
tablet. We are happy to run candidate patches (streaming/chunked VARIANT writer
paths, estimator changes, or both) against it and report success/failure, row
counts, and checksums for the affected tablets.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
