[ 
https://issues.apache.org/jira/browse/IGNITE-29088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Maksim Davydov updated IGNITE-29088:
------------------------------------
    Description: 
When a node leaves the cluster with the only copy of a partition, another node 
takes the partition over. Under the IGNORE policy it becomes the owner of an 
empty partition, and under the safe policies it marks the partition as lost. 
The other nodes don't wait for it to report this and write the new state into 
their partition maps themselves. If the new owner sent its partition map 
shortly before, the coordinator treats that map as newer and sends it to the 
other nodes, so until the next partition map exchange they see the partition 
without an owner or not marked as lost. Under IGNORE, SQL queries from these 
nodes fail with "Failed to map SQL query to topology", while key operations 
work.

With the fix, only the node that holds a partition reports its state: it sends 
its partition map at the end of the exchange, and the other nodes take the 
state from it. Until the map arrives, about 1.5 seconds by default, they see no 
owner of the partition, and SQL queries wait instead of failing. The same 
applies when a node takes over partitions that no other node holds, or marks 
its partitions as lost when it joins. In a rolling upgrade, the nodes switch to 
the new behavior only after the whole cluster runs the new version.

  was:
Under any partition loss policy, the coordinator's partition map can keep a 
stale state of a lost partition until the next exchange.

When a server node leaves with the only copy of a partition, the new primary 
creates the partition as MOVING on the coordinator's full message. Then, in 
detectLostPartitions, it owns the partition (IGNORE) or marks it LOST (SAFE 
policies). The coordinator sets the same state in its own detectLostPartitions. 
If the new primary sends its partition map between these two steps, for example 
a scheduled resend after an eviction, the map carries MOVING with a newer 
update sequence, and the coordinator takes it. Nothing sends the map again 
after the partition changes.

Under IGNORE, the coordinator and every node that takes its full map see no 
owner of the partition until the next exchange. SQL queries from those nodes 
fail with `Failed to map SQL query to topology during timeout: 30000ms`, while 
key operations and scans work. `awaitPartitionMapExchange()` times out in 
tests:` IgniteTopologyValidatorGridSplitCacheTest` (32 nodes, 50 caches without 
backups) hits this in about half of local runs. Under SAFE, the coordinator's 
map shows MOVING instead of LOST.

Fix: when detectLostPartitions changes a local partition, a non-coordinator 
node resends its single map. GridDhtPartitionTopologyImpl#detectLostPartitions 
reports a change of any local partition, not only of the last one it checks.


> Fix stale lost partition states in the coordinator's partition map
> ------------------------------------------------------------------
>
>                 Key: IGNITE-29088
>                 URL: https://issues.apache.org/jira/browse/IGNITE-29088
>             Project: Ignite
>          Issue Type: Bug
>            Reporter: Maksim Davydov
>            Assignee: Maksim Davydov
>            Priority: Major
>              Labels: ise
>          Time Spent: 3h 10m
>  Remaining Estimate: 0h
>
> When a node leaves the cluster with the only copy of a partition, another 
> node takes the partition over. Under the IGNORE policy it becomes the owner 
> of an empty partition, and under the safe policies it marks the partition as 
> lost. The other nodes don't wait for it to report this and write the new 
> state into their partition maps themselves. If the new owner sent its 
> partition map shortly before, the coordinator treats that map as newer and 
> sends it to the other nodes, so until the next partition map exchange they 
> see the partition without an owner or not marked as lost. Under IGNORE, SQL 
> queries from these nodes fail with "Failed to map SQL query to topology", 
> while key operations work.
> With the fix, only the node that holds a partition reports its state: it 
> sends its partition map at the end of the exchange, and the other nodes take 
> the state from it. Until the map arrives, about 1.5 seconds by default, they 
> see no owner of the partition, and SQL queries wait instead of failing. The 
> same applies when a node takes over partitions that no other node holds, or 
> marks its partitions as lost when it joins. In a rolling upgrade, the nodes 
> switch to the new behavior only after the whole cluster runs the new version.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to