[
https://issues.apache.org/jira/browse/IGNITE-29088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Maksim Davydov updated IGNITE-29088:
------------------------------------
Description:
Under any partition loss policy, the coordinator's partition map can keep a
stale state of a lost partition until the next exchange.
When a server node leaves with the only copy of a partition, the new primary
creates the partition as MOVING on the coordinator's full message. Then, in
detectLostPartitions, it owns the partition (IGNORE) or marks it LOST (SAFE
policies). The coordinator sets the same state in its own detectLostPartitions.
If the new primary sends its partition map between these two steps, for example
a scheduled resend after an eviction, the map carries MOVING with a newer
update sequence, and the coordinator takes it. Nothing sends the map again
after the partition changes.
Under IGNORE, the coordinator and every node that takes its full map see no
owner of the partition until the next exchange. SQL queries from those nodes
fail with `Failed to map SQL query to topology during timeout: 30000ms`, while
key operations and scans work. `awaitPartitionMapExchange()` times out in
tests:` IgniteTopologyValidatorGridSplitCacheTest` (32 nodes, 50 caches without
backups) hits this in about half of local runs. Under SAFE, the coordinator's
map shows MOVING instead of LOST.
Fix: when detectLostPartitions changes a local partition, a non-coordinator
node resends its single map. GridDhtPartitionTopologyImpl#detectLostPartitions
reports a change of any local partition, not only of the last one it checks.
was:
*Problem*
In an in-memory cluster with the IGNORE partition loss policy (baseline
auto-adjust on, timeout 0), a node leaves and takes the only copy of some
partitions with it (backups = 0). Each lost partition is recreated empty on its
new primary and owned there. Sometimes the coordinator never learns that: its
partition map keeps the partition MOVING until the next exchange.
*Impact*
Until the next topology change, which in a stable cluster can be hours away,
the coordinator and every node that takes its full map see no owner of the
partition:
- SQL queries from those nodes fail after the retry timeout:
"Failed to map SQL query to topology during timeout: 30000ms". With a MOVING
partition in the map, ReducePartitionMapper maps partitions by their owners and
finds none. On the new primary, whose own map is right, the same query succeeds.
- awaitPartitionMapExchange() in tests times out. Key operations and scan
queries still work.
This is not what IGNORE is meant to do: the policy resets a lost partition
silently, no
partition is LOST, and there is nothing to reset by hand.
*Cause*
With the exchange merge protocol, the new primary creates the lost partition as
MOVING when it gets the coordinator's full message, and owns it a moment later
in detectLostPartitions. The coordinator, in its own detectLostPartitions,
already marks the partition OWNING for that node.
If the new primary sends its partition map between these two steps, the map
carries MOVING with a newer update sequence, and the coordinator takes it. A
typical sender is the scheduled resend (scheduleResendPartitions(), 1.5 s after
an eviction or a map change). Nothing sends the node's map again after it owns
the partition: the result of detectLostPartitions is dropped in
GridDhtPartitionsExchangeFuture#detectLostPartitions.
Also, inside GridDhtPartitionTopologyImpl#detectLostPartitions the result
reflects only the last lost partition.
*How it shows*
IgniteTopologyValidatorGridSplitCacheTest (32 nodes, 50 caches without backups)
times out in awaitPartitionMapExchange() after stopping its configless node in
9 of 15 local runs. Small clusters rarely send a map inside that window, so
IgniteCachePartitionLossPolicySelfTest (3 nodes) doesn't hit it.
*Fix*
When detectLostPartitions changes a local partition, a non-coordinator node
sends its single map again. A new test, CachePartitionLossIgnorePolicyMapTest,
makes the new primary send its map inside the window every time and checks that
the coordinator's map matches the nodes.
The resend is not limited to IGNORE: it fires whenever detectLostPartitions
changes a local partition. Under the SAFE policies the node marks the partition
LOST instead of owning it, and the same order of messages can leave the
coordinator's map at MOVING for that node instead of LOST until the next
exchange; the resend corrects that as well. It also fires on cluster activation
with lost partitions. Loss events, lostPartitions() and resetLostPartitions()
work as before: a late map with LOST can't override the reset, since the reset
is a new exchange with a newer topology version. The cost is one extra single
map per affected non-coordinator node per such exchange; the coordinator
answers with one delayed full-map resend that covers them all.
> Fix stale lost partition states in the coordinator's partition map
> ------------------------------------------------------------------
>
> Key: IGNITE-29088
> URL: https://issues.apache.org/jira/browse/IGNITE-29088
> Project: Ignite
> Issue Type: Bug
> Reporter: Maksim Davydov
> Assignee: Maksim Davydov
> Priority: Major
> Labels: ise
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Under any partition loss policy, the coordinator's partition map can keep a
> stale state of a lost partition until the next exchange.
> When a server node leaves with the only copy of a partition, the new primary
> creates the partition as MOVING on the coordinator's full message. Then, in
> detectLostPartitions, it owns the partition (IGNORE) or marks it LOST (SAFE
> policies). The coordinator sets the same state in its own
> detectLostPartitions. If the new primary sends its partition map between
> these two steps, for example a scheduled resend after an eviction, the map
> carries MOVING with a newer update sequence, and the coordinator takes it.
> Nothing sends the map again after the partition changes.
> Under IGNORE, the coordinator and every node that takes its full map see no
> owner of the partition until the next exchange. SQL queries from those nodes
> fail with `Failed to map SQL query to topology during timeout: 30000ms`,
> while key operations and scans work. `awaitPartitionMapExchange()` times out
> in tests:` IgniteTopologyValidatorGridSplitCacheTest` (32 nodes, 50 caches
> without backups) hits this in about half of local runs. Under SAFE, the
> coordinator's map shows MOVING instead of LOST.
> Fix: when detectLostPartitions changes a local partition, a non-coordinator
> node resends its single map.
> GridDhtPartitionTopologyImpl#detectLostPartitions reports a change of any
> local partition, not only of the last one it checks.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
