[
https://issues.apache.org/jira/browse/IGNITE-28832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dmitry Werner updated IGNITE-28832:
-----------------------------------
Summary: Fix flaky
StatisticsConfigurationTest.updateStatisticsOnChangeTopology[persist=true]
(was: Fix flaky StatisticsConfigurationTest.updateStatisticsOnChangeTopology)
> Fix flaky
> StatisticsConfigurationTest.updateStatisticsOnChangeTopology[persist=true]
> ------------------------------------------------------------------------------------
>
> Key: IGNITE-28832
> URL: https://issues.apache.org/jira/browse/IGNITE-28832
> Project: Ignite
> Issue Type: Test
> Reporter: Dmitry Werner
> Assignee: Dmitry Werner
> Priority: Minor
> Labels: MakeTeamcityGreenAgain, ise
> Time Spent: 10m
> Remaining Estimate: 0h
>
> StatisticsConfigurationTest.updateStatisticsOnChangeTopology[persist=true]
> Success Rate: 21.7% (Last 106 Runs) 83 failed, 23 successful
> [https://ci2.ignite.apache.org/test/2355900617476832260?currentProjectId=IgniteTests24Java8&branch=%3Cdefault%3E]
>
> *1. Failure reason (stacktrace from master log
> _Apache_Ignite_2.x_Tests_Queries_3_17496.log)*
> The test fails in IgniteBinaryCacheQueryTestSuite3 with an exchange timeout
> (default 30 s). Stacktrace:
> {code:java}
> class org.apache.ignite.IgniteException: Timeout of waiting for
> topology map update [igniteInstanceName=stat.StatisticsConfigurationTest1,
> cache=SMALLnull, cacheId=1165137646,
> topVer=AffinityTopologyVersion [topVer=7, minorTopVer=2], p=0,
> readVer=AffinityTopologyVersion [topVer=7, minorTopVer=2],
> locNode=TcpDiscoveryNode [id=311d5634-29a9-41fe-83e1-744048a00001,
> ...]]
> at
> org.apache.ignite.testframework.junits.common.GridCommonAbstractTest.awaitPartitionMapExchange(GridCommonAbstractTest.java:806)
> at
> org.apache.ignite.testframework.junits.common.GridCommonAbstractTest.awaitPartitionMapExchange(GridCommonAbstractTest.java:599)
> at
> org.apache.ignite.testframework.junits.common.GridCommonAbstractTest.awaitPartitionMapExchange(GridCommonAbstractTest.java:584)
> at
> org.apache.ignite.internal.processors.query.stat.StatisticsConfigurationTest.stopGridAndChangeBaseline(StatisticsConfigurationTest.java:160)
> at
> org.apache.ignite.internal.processors.query.stat.StatisticsConfigurationTest.updateStatisticsOnChangeTopology(StatisticsConfigurationTest.java:300)
> ...{code}
> The test passes locally on fast machines and only flakes on slow CI nodes.
> {*}2. Root cause{*}{*}{{*}}
> updateStatisticsOnChangeTopology() repeatedly stops/restarts server nodes and
> (in persist=true mode) manually adjusts the baseline via
> cluster().setBaselineTopology(...).
> After a server node leaves the baseline, the partitions it owned become
> *LOST* (log confirms Detected lost partitions [grp=SMALLnull, parts=[...],
> topVer=7.0]). The table is created with BACKUPS=1 and default
> PartitionLossPolicy = READ_ONLY_SAFE. Because partition-loss detection runs
> during the partition exchange _before_ a surviving backup is promoted to
> primary, partitions whose owner left are marked LOST instead of being
> rebalanced.
> LOST partitions have {*}no owners{*}, so awaitPartitionMapExchange() — which
> waits until the topology is fully rebalanced (rebalanced=true) — can never
> converge and times out after 30 s. Hence the intermittent failure regardless
> of actual machine slowness.
> *3. The fix*
> Instead of manually re-setting the baseline after each node start/stop (which
> is what makes partitions become LOST), the baseline is set once to
> *auto-adjust with zero timeout* at cluster creation. This makes the baseline
> update automatically whenever nodes join or leave, so backups are promoted to
> primaries _before_ detectLostPartitions runs during the exchange — partitions
> no longer become LOST and the rebalance completes.
> The exchange timeout is also increased (×3 of the default) to tolerate slow
> CI machines, since rebalancing a persistent cache can take longer than the
> default 30 s.
> *Validation note:* after this fix the test ran 30 times on CI and failed only
> once, and that single failure was a pure 30 s timeout on rebalanced=false (no
> LOST partitions) caused by the default exchange timeout — addressed by
> raising getPartitionMapExchangeTimeout().
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
