ShummGen commented on issue #63609:
URL: https://github.com/apache/doris/issues/63609#issuecomment-5086682897
# Additional diagnostic information: corrupted segment pages and BE crashes
Thank you for the detailed review and for outlining the information required
to investigate this issue.
I can provide most of the requested current metadata, query information,
configuration, and diagnostic evidence from the recent incident. Raw core dumps
and segment files may contain sensitive business data, so I would prefer to
provide those through a private upload channel.
## Important clarification about the age of the original incident
The original incident was reported too long ago, and the corresponding BE
log files have already expired under our log-retention policy. I am therefore
unable to provide the exact one-to-two-minute log interval surrounding the
original crash, and I do not want to reconstruct or guess any missing log lines.
The diagnostic information below comes from a **new, recent incident in the
same environment**, observed on `2026-07-27`. It uses the same Doris build and
again involves corrupted segment pages, ZSTD decompression failures, query
failures, compaction failures, and a BE crash.
Please treat the information below as new reproducible evidence that may
help investigate the same class of robustness problem. It is not the original
historical crash log, and I cannot currently prove that the recent crash and
the original crash have exactly the same root cause.
One important clarification is that the latest crash observed in this
environment does not have exactly the same top stack frames as the previously
reported `ColumnStr::serialize_impl` / `DistinctStreamingAgg` crash. The latest
crash and the ongoing compaction failures may share the same underlying
corrupted segment pages, but they may also represent separate error-propagation
paths. I have kept them separate below rather than assuming they have the same
root cause.
## 1. Exact Doris build information
All FE and BE nodes report the same build:
```text
doris-4.1.1-rc01-b10073ad9ca
```
`SELECT VERSION()` returns:
```text
5.7.99
```
Cluster topology:
```text
FE: 1 node, alive, elected master
BE: 3 nodes, all alive
Deployment mode: local storage
Replication: 3 replicas
```
The FE and all three BEs report the same `doris-4.1.1-rc01-b10073ad9ca`
build string in `SHOW FRONTENDS` and `SHOW BACKENDS`.
The binaries were obtained from the Doris binary distribution rather than
compiled locally. I do not currently have a separate source-build commit beyond
the build identifier embedded in the version string.
## 2. Incident timeline
All timestamps below are China Standard Time (`Asia/Shanghai`, UTC+8).
```text
2026-07-27 09:31:41 One BE terminated with SIGSEGV.
2026-07-27 09:31:53 systemd restarted the BE; it rejoined the cluster and
is currently alive.
2026-07-27 09:46:28 A query reading the affected log table failed with:
[INTERNAL_ERROR] ZSTD_decompressDCtx error: Data
corruption detected
2026-07-27 10:22:00 Compaction status was checked on all three replicas of
two affected tablets.
All six replicas were still reporting the same ZSTD
decompression error.
```
The latest BE crash stack included frames around:
```text
doris::vectorized::ColumnDictI32::filter_by_selector
doris::segment_v2::SegmentIterator
doris::OlapScanner
```
Earlier crashes also included an exception similar to:
```text
std::out_of_range: map::at
```
The stack above differs from the earlier `ColumnStr<unsigned
int>::serialize_impl` / `DistinctStreamingAgg` stack. I would appreciate
guidance on whether these should remain in the same issue or be tracked as two
related issues.
## 3. Timestamp clarification
The previously reported value:
```text
Aborted at 1748134471
```
corresponds to `2025-05-25`, not `2026-05-25`.
Because the original logs are no longer retained, I cannot reliably
reconstruct whether the original event occurred on `2025-05-25` or
`2026-05-25`. The timestamp/date combination in the original report should
therefore be treated as inaccurate and should not be used for source-version
matching.
The exact recent incident described in this update occurred on `2026-07-27`,
and its timestamps were collected directly from the current environment.
Any timestamps or source line numbers previously presented as examples or
reconstructed excerpts should be treated as illustrative. The timestamps in the
timeline above and the errors in the following sections were collected from the
running environment.
## 4. Affected table schema
The affected table is a local-storage OLAP table. The table and database
names can be anonymized if required; the relevant schema is:
```sql
CREATE TABLE `alb_access_log` (
`log_date` DATE NULL,
`time` DATETIME NULL,
`status` VARCHAR(16) NULL,
`request_method` VARCHAR(16) NULL,
`host` VARCHAR(256) NULL,
`client_ip` VARCHAR(64) NULL,
`request_time` DECIMAL(10,3) NULL,
`request_uri` VARCHAR(65533) NULL,
`http_user_agent` VARCHAR(65533) NULL,
`cqid` VARCHAR(128) NULL,
`h1` VARCHAR(256) NULL,
`click_id` VARCHAR(255) NULL,
`event_type` VARCHAR(32) NULL,
`traffic` VARCHAR(64) NULL,
`import_time` DATETIME NULL,
`source_file` VARCHAR(64) NULL,
`request_id` VARCHAR(255) NULL,
`city_region` VARCHAR(128) NULL
)
ENGINE=OLAP
DUPLICATE KEY(`log_date`, `time`)
PARTITION BY RANGE(`log_date`)
DISTRIBUTED BY HASH(`client_ip`) BUCKETS 32
PROPERTIES (
"replication_allocation" = "tag.location.default: 3",
"storage_medium" = "hdd",
"storage_format" = "V2",
"inverted_index_storage_format" = "V3",
"light_schema_change" = "true",
"disable_auto_compaction" = "false",
"enable_single_replica_compaction" = "false"
);
```
The real table also contains inverted indexes on `log_date`, `request_uri`,
`click_id`, `h1`, `source_file`, `traffic`, and `event_type`.
The other table participating in the failing query contains:
```text
id BIGINT NOT NULL
tk_create_time DATETIME NOT NULL
click_id VARCHAR(675) NULL
```
## 5. Affected partition and tablets
Affected partition:
```text
Partition: p20260726
Range: [2026-07-26, 2026-07-27)
State: NORMAL according to FE metadata
Buckets: 32
Replicas: 3
Rows: 75,370,092 according to partition metadata
Size: approximately 46.177 GB
```
Two tablets are currently known to contain unreadable pages:
| Tablet ID | Rows per replica | Replica count | FE metadata state | Actual
read/compaction result |
|---|---:|---:|---|---|
| `1784714655425` | `2,628,463` | 3 | `NORMAL`, `IsBad=false` | ZSTD
decompression corruption on all replicas |
| `1784714655445` | `2,695,687` | 3 | `NORMAL`, `IsBad=false` | ZSTD
decompression corruption on all replicas |
The replica versions were aligned at version `140`. FE tablet diagnosis
therefore reports healthy metadata, although actual page reads fail.
At approximately `2026-07-27 10:22 CST`, the cumulative compaction status of
both tablets on every BE reported:
```text
[INTERNAL_ERROR]ZSTD_decompressDCtx error: Data corruption detected
```
This produced six matching failures:
```text
BE-A / tablet 1784714655425: ZSTD decompression corruption
BE-B / tablet 1784714655425: ZSTD decompression corruption
BE-C / tablet 1784714655425: ZSTD decompression corruption
BE-A / tablet 1784714655445: ZSTD decompression corruption
BE-B / tablet 1784714655445: ZSTD decompression corruption
BE-C / tablet 1784714655445: ZSTD decompression corruption
```
The most recent successful cumulative compaction on BE-A and BE-B was on
`2026-07-26`. BE-C had no successful cumulative or base compaction recorded for
these two replicas after its current data copy was created.
Because all three replicas fail actual reads, marking a single replica bad
and cloning it from another replica does not provide a safe recovery path.
## 6. Query that exposes the failure
The production operation is an `INSERT INTO ... WITH ... SELECT` statement.
The relevant query structure is shown below. Business-specific output columns
and unrelated joins have been omitted, but the scans of the affected table are
unchanged:
```sql
SET enable_spill = true;
SET exec_mem_limit = 8589934592;
SET parallel_pipeline_task_num = 4;
SET @order_start = '2026-07-26 00:00:00';
SET @order_end = '2026-07-27 00:00:00';
INSERT INTO target_table (...)
WITH order_window AS (
SELECT id, click_id, tk_create_time, ...
FROM order_detail_table
WHERE click_id IS NOT NULL
AND click_id != ''
AND tk_create_time >= CAST(@order_start AS DATETIME)
AND tk_create_time < CAST(@order_end AS DATETIME)
),
click_ids AS (
SELECT DISTINCT click_id
FROM order_window
),
click_candidates AS (
SELECT
alb.h1 AS click_id,
alb.time AS click_time,
alb.cqid,
alb.traffic,
alb.request_uri
FROM alb_access_log alb
INNER JOIN click_ids ids ON alb.h1 = ids.click_id
WHERE alb.h1 IS NOT NULL
AND alb.h1 != ''
),
click_candidates_huawei AS (
SELECT
ids.click_id,
alb.time AS click_time,
alb.cqid,
alb.traffic,
alb.request_uri
FROM alb_access_log alb
INNER JOIN huawei_click_ids ids
ON alb.h1 = ids.log_id
AND CAST(alb.cqid AS STRING) = ids.cqid
WHERE alb.traffic = 'HUAWEI'
AND alb.h1 IS NOT NULL
AND alb.h1 != ''
AND alb.cqid IS NOT NULL
),
event_candidates AS (
SELECT
alb.click_id,
alb.time,
alb.event_type,
alb.request_id,
extract_url_parameter(alb.request_uri, 'project_id'),
extract_url_parameter(alb.request_uri, 'promotion_id')
FROM alb_access_log alb
INNER JOIN click_ids ids ON alb.click_id = ids.click_id
WHERE alb.click_id IS NOT NULL
AND alb.click_id != ''
AND alb.time IS NOT NULL
AND alb.source_file >= 'tracer'
AND alb.source_file < 'traces'
AND alb.event_type IN ('3', '16', '17', '18', 'bigIncome')
)
SELECT ...;
```
The query contains:
- `SELECT DISTINCT click_id`, implemented by the plan as a streaming
vectorized aggregate grouped by `click_id`.
- Multiple `ROW_NUMBER()` window functions.
- Sorting by `request_uri`, whose type is `VARCHAR(65533)`.
- URL extraction and URL decoding from `request_uri`.
- Three independent scans of `alb_access_log`.
The `DISTINCT`/grouping column from the order table is `VARCHAR(675)`.
Related join columns in the log table are `h1 VARCHAR(256)` and `click_id
VARCHAR(255)`.
## 7. Relevant EXPLAIN output
The full plan contains 33 plan fragments. The important result is that the
order table is correctly pruned to one partition, but every scan of
`alb_access_log` reads all partitions because no `log_date` predicate is
present.
Order table scan:
```text
TABLE: order_detail_table
PREDICATES:
tk_create_time >= '2026-07-26 00:00:00'
tk_create_time < '2026-07-27 00:00:00'
partitions=1/364 (p20260726)
tablets=32/32
```
Log table scans:
```text
TABLE: alb_access_log
partitions=83/83
tablets=2656/2656
numNodes=3
```
This occurs for the normal click lookup, the Huawei click lookup, and the
event lookup.
The `SELECT DISTINCT click_id` portion is planned as:
```text
VAGGREGATE (update serialize)
STREAMING
group by: click_id
```
Therefore the failing query can exercise both the segment-scanning path and
a downstream streaming aggregation path.
I can attach the complete non-anonymized `EXPLAIN` output separately if
required.
## 8. Session and BE configuration
Query-specific session settings:
```text
enable_spill = true
exec_mem_limit = 8589934592 # 8 GiB
parallel_pipeline_task_num = 4
```
Current session defaults observed independently of the query:
```text
skip_bad_tablet = false
fallback_other_replica_when_fixed_corrupt = false
```
`SHOW VARIABLES LIKE 'enable_query_memory_overcommit'` returned no row in
this build, so I cannot currently provide a runtime value for that variable.
Relevant configuration from the BE that most recently crashed:
```text
deploy_mode = "" # local-storage deployment
mem_limit = 90%
memory_mode = moderate
storage_page_cache_limit = 20%
disable_storage_page_cache = false
disable_segment_cache = false
check_segment_when_build_rowset_meta = false
storage_strict_check_incompatible_old_format = true
sync_file_on_close = true
disable_auto_compaction = false
storage_root_path = <two local storage paths>
```
The BE has one original Doris storage path and one additional local disk
path added later for capacity expansion. The expansion was performed through
Doris `storage_root_path` configuration and Doris-managed balancing. To the
best of my knowledge, the affected segment files were not manually edited,
overwritten, restored, or copied into the tablet directory outside Doris.
The FE was migrated to another host before this incident, but the BE data
directories remained managed by their existing BE processes. No BE tablet
directory was manually copied as part of the FE migration.
## 9. Relationship between the core dumps
I cannot confirm that all previously observed historical core dumps had the
same trigger. The original BE logs needed to correlate those dumps have already
expired.
The relationship analysis below therefore applies primarily to the recent
`2026-07-27` incident, not as a reconstruction of the original historical
incident.
Current evidence suggests at least two paths must be investigated separately:
1. Query execution reading corrupted segment pages and forwarding data
toward vectorized operators.
2. Cumulative/base compaction reading the same corrupted pages and
repeatedly returning ZSTD errors.
At least one recent SIGSEGV occurred during an OLAP scanner/segment iterator
path. The compaction status APIs show repeated corruption failures, but I have
not yet proven that every BE core dump was created by a compaction thread.
I will correlate the following fields for each retained core and log
interval:
```text
crash timestamp
BE node
thread ID
query ID
fragment instance ID
tablet ID
rowset ID
segment path
top stack frames
whether a compaction task was active
```
I cannot reliably claim that the historical dumps were identical. If a core
from the recent crash is available, I will correlate it with the recent
retained logs instead.
## 10. GDB and core-dump availability
The historical BE logs are no longer available. I am checking whether any
matching historical core file is still retained, but I cannot currently promise
a complete GDB analysis of the original crash.
For the recent `2026-07-27` crash, I will check whether the new core file
and the exact matching BE binary are retained on the crashed node.
If available, I will collect at least:
```gdb
set pagination off
set print pretty on
thread apply all bt full
```
For a core containing the `ColumnStr<unsigned int>::serialize_impl` frame, I
will also attempt:
```gdb
frame <ColumnStr::serialize_impl frame>
info args
info locals
p row
p this
p offsets.size()
p chars.size()
p offsets[row]
p offsets[row - 1]
```
For the serialized-key caller, if symbols permit:
```gdb
p keys[row]
p keys[row].data
p keys[row].size
```
If the distributed binary is stripped or optimized and GDB cannot evaluate
these expressions, I will include the exact GDB error messages.
No string contents or raw user data will be included. Pointer values, sizes,
offsets, symbols, and stack frames can be provided.
## 11. Complete BE log interval
The log interval surrounding the original historical crash is no longer
available because it has expired under the retention policy.
I am checking retention for the exact log interval around the recent
`2026-07-27 09:31:41 CST` crash.
The requested export will cover approximately:
```text
2026-07-27 09:29:30 CST
through
2026-07-27 09:33:30 CST
```
It will preserve, when present:
```text
thread IDs
query IDs
fragment instance IDs
tablet IDs
rowset IDs
segment paths
the first checksum/decompression error
scanner and iterator errors
query cancellation messages
compaction messages
the final crash stack
```
Business values inside URLs, request payloads, hostnames, and user
identifiers will be redacted without changing technical identifiers such as
query IDs, tablet IDs, rowset IDs, or stack frames.
## 12. Corrupted segment sample and metadata
The affected replicas and storage paths are still online, so it should be
possible to collect:
```text
one corrupted segment file
tablet metadata
rowset metadata
tablet schema
segment and rowset identifiers
file size
SHA-256 checksum
```
However, all three replicas of the two known tablets currently fail
decompression. I do not currently have a confirmed healthy copy of the same
segment for comparison.
The segment may contain request URLs and other business data. I will not
attach it to the public GitHub issue. Please provide a private upload channel
if the raw segment or core dump would be useful.
If preferred, I can first provide only:
- File size and SHA-256 for each replica.
- Tablet/rowset/segment metadata.
- A byte-level comparison between replicas.
- A reduced or intentionally corrupted synthetic sample that reproduces the
same error without business data.
## 13. Current conclusions
The current evidence supports the following limited conclusions:
1. The FE metadata alone does not detect this corruption because replica
state, replica count, and versions are normal.
2. Actual page reads and compaction consistently detect ZSTD data corruption.
3. The same two tablets fail on all three replicas, so this is not currently
recoverable by cloning from another in-cluster replica.
4. A production query scans all log-table partitions and reliably reaches
the affected partition.
5. Doris should return a safe query or compaction error when the page cannot
be decompressed; the BE should not terminate with SIGSEGV while handling that
error.
6. The different observed top stack frames may indicate multiple unsafe
error-propagation paths, so query execution and compaction should be
investigated separately.
## 14. Requested next steps
Could you please confirm:
1. Whether the current `ColumnDictI32::filter_by_selector` /
`SegmentIterator` crash should remain in this issue together with the earlier
`ColumnStr::serialize_impl` / `DistinctStreamingAgg` crash.
2. Whether there is a private upload channel for one representative core
dump, a complete BE log interval, and one corrupted segment sample.
3. Whether the build identifier `doris-4.1.1-rc01-b10073ad9ca` is sufficient
to map the distributed binary to the exact source revision, or whether another
command/file should be used to extract the build commit.
I apologize that the original historical log interval is no longer
available. I hope the recent incident, for which the affected tablets are still
online and the corruption remains reproducible, can provide a more useful
investigation target.
I can provide the full `SHOW FRONTENDS`, `SHOW BACKENDS`, `SHOW CREATE
TABLE`, `EXPLAIN`, BE configuration, and GDB output as separate sanitized
attachments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
