[
https://issues.apache.org/jira/browse/IGNITE-29102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Egor Baranov updated IGNITE-29102:
----------------------------------
Description:
*Problem:*
When a B+Tree cursor encounters a broken invariant (typically an AssertionError
in AbstractForwardCursor.find() or AbstractForwardCursor.fillFromBuffer()), the
exception is wrapped into a CorruptedTreeException and propagated up the call
chain to GridMapQueryExecutor. There, it is caught in a generic catch
(Throwable e) block and logged as "Failed to execute local query" at ERROR/WARN
level.
There is currently *no metric* that tracks how often such corruption-related
query failures occur. Operators can only detect this problem by parsing log
files, which is impractical for:
- Large clusters with high query throughput
- Intermittent corruption events that are hard to spot in log noise
- Automated alerting on infrastructure degradation
Example stack trace:
{code:text}
java.lang.AssertionError: null
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$AbstractForwardCursor.find(BPlusTree.java:5559)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$AbstractForwardCursor.reinitialize(BPlusTree.java:5572)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$AbstractForwardCursor.nextPage(BPlusTree.java:5627)
...
at
org.apache.ignite.internal.processors.query.h2.twostep.GridMapQueryExecutor.onQueryRequest0(GridMapQueryExecutor.java:538)
{code}
{*}Solution{*}{*}{{*}}
Add a dedicated long counter metric in the map query execution path that is
incremented whenever a query fails due to a CorruptedTreeException.
{*}Resulting Metric{*}{*}{{*}}
||*Registry*||*Name*||*Type*||*Description*||
│ sql.queries.treeCorruption │ count │ long │ Number of queries that failed due
to B+Tree cursor corruption (AssertionError during B+Tree page iteration). │
*This metric enables:*
- Automated alerting on tree corruption frequency
- Correlation of corruption events with specific operations (rebalance,
compaction, etc.)
- Quantification of data reliability without log scraping
was:
{*}Problem{*}{*}{*}
When a B+Tree cursor encounters a broken invariant (typically an
AssertionError in AbstractForwardCursor.find() or
AbstractForwardCursor.fillFromBuffer()), the exception is wrapped
into a CorruptedTreeException and propagated up the call chain to
GridMapQueryExecutor. There, it is caught in a generic catch (Throwable e)
block and logged as "Failed to execute
local query" at ERROR/WARN level.
There is currently *no metric* that tracks how often such
corruption-related query failures occur. Operators can only detect this problem
by parsing log files, which is impractical for:
- Large clusters with high query throughput
- Intermittent corruption events that are hard to spot in log noise
- Automated alerting on infrastructure degradation
Example stack trace:
1 java.lang.AssertionError: null
2 at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$AbstractForwardCursor.find(BPlusTree.java:5559)
3 at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$AbstractForwardCursor.reinitialize(BPlusTree.java:5572)
4 at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$AbstractForwardCursor.nextPage(BPlusTree.java:5627)
5 ...
6 at
org.apache.ignite.internal.processors.query.h2.twostep.GridMapQueryExecutor.onQueryRequest0(GridMapQueryExecutor.java:538)
{*}Solution{*}{*}{*}
Add a dedicated long counter metric in the map query execution path that is
incremented whenever a query fails due to a CorruptedTreeException. This
follows the same pattern as
FailureProcessor metrics (IGNITE-29047) and the existing sql.queries.user
metrics.
{*}Changes{*}{*}{*}
1. *`GridMapQueryExecutor`* (modules/indexing):
- Register a new metric registry sql.queries.treeCorruption with a
single long metric count.
- Increment the counter in both error-handling paths (onQueryRequest0
and onNextPageRequest) when the caught throwable's cause chain contains a
CorruptedTreeException.
2. *Documentation* (docs/_docs/monitoring-metrics/new-metrics.adoc):
- Document the new registry and metric.
3. *Test* (TreeCorruptionMetricTest):
- Verify the metric is registered and initially zero.
- Simulate B+Tree corruption via BPlusTree.testHndWrapper, execute a
query, and assert the counter is incremented.
{*}Resulting Metric{*}{*}{*}
┌────────────────────────────┬───────┬──────┬──────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ *Registry* │ *Name* │ *Type* │ *Description*
│
├────────────────────────────┼───────┼──────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ sql.queries.treeCorruption │ count │ long │ Number of queries that failed
due to B+Tree cursor corruption (AssertionError during B+Tree page iteration). │
└────────────────────────────┴───────┴──────┴──────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
This metric enables:
- Automated alerting on tree corruption frequency
- Correlation of corruption events with specific operations (rebalance,
compaction, etc.)
- Quantification of data reliability without log scraping
> Add a metric to track B+Tree cursor corruption failures during SQL query
> execution
> ----------------------------------------------------------------------------------
>
> Key: IGNITE-29102
> URL: https://issues.apache.org/jira/browse/IGNITE-29102
> Project: Ignite
> Issue Type: Task
> Reporter: Egor Baranov
> Assignee: Egor Baranov
> Priority: Major
> Labels: from-log-to-metric
>
> *Problem:*
> When a B+Tree cursor encounters a broken invariant (typically an
> AssertionError in AbstractForwardCursor.find() or
> AbstractForwardCursor.fillFromBuffer()), the exception is wrapped into a
> CorruptedTreeException and propagated up the call chain to
> GridMapQueryExecutor. There, it is caught in a generic catch (Throwable e)
> block and logged as "Failed to execute local query" at ERROR/WARN level.
> There is currently *no metric* that tracks how often such corruption-related
> query failures occur. Operators can only detect this problem by parsing log
> files, which is impractical for:
> - Large clusters with high query throughput
> - Intermittent corruption events that are hard to spot in log noise
> - Automated alerting on infrastructure degradation
> Example stack trace:
> {code:text}
> java.lang.AssertionError: null
> at
> org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$AbstractForwardCursor.find(BPlusTree.java:5559)
> at
> org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$AbstractForwardCursor.reinitialize(BPlusTree.java:5572)
> at
> org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$AbstractForwardCursor.nextPage(BPlusTree.java:5627)
> ...
> at
> org.apache.ignite.internal.processors.query.h2.twostep.GridMapQueryExecutor.onQueryRequest0(GridMapQueryExecutor.java:538)
> {code}
> {*}Solution{*}{*}{{*}}
> Add a dedicated long counter metric in the map query execution path that
> is incremented whenever a query fails due to a CorruptedTreeException.
> {*}Resulting Metric{*}{*}{{*}}
> ||*Registry*||*Name*||*Type*||*Description*||
> │ sql.queries.treeCorruption │ count │ long │ Number of queries that failed
> due to B+Tree cursor corruption (AssertionError during B+Tree page
> iteration). │
> *This metric enables:*
> - Automated alerting on tree corruption frequency
> - Correlation of corruption events with specific operations (rebalance,
> compaction, etc.)
> - Quantification of data reliability without log scraping
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
