Seems I got the vision, thanks. There should be only 2 ways to reset lost partition: to gain an owner from resurrected first or to remove ex-owner from baseline (partition will be rearranged). And we should make a decision for every lost partition before calling the reset.
On Wed, May 6, 2020 at 8:02 PM Alexei Scherbakov < alexey.scherbak...@gmail.com> wrote: > ср, 6 мая 2020 г. в 12:54, Anton Vinogradov <a...@apache.org>: > > > Alexei, > > > > 1,2,4,5 - looks good to me, no objections here. > > > > >> 3. Lost state is impossible to reset if a topology doesn't have at > least > > >> one owner for each lost partition. > > > > Do you mean that, according to your example, where > > >> a node2 has left, soon a node3 has left. If the node2 is returned to > > >> the topology first, it would have stale data for some keys. > > we have to have node2 at cluster to be able to reset "lost" to node2's > > data? > > > > Not sure if I understand a question, but try to answer using an example: > Assume 3 nodes n1, n2, n3, 1 backup, persistence enabled, partition p is > owned by n2 and n3. > 1. Topology is activated. > 2. cache.put(p, 0) // n2 and n3 have p->0, updateCounter=1 > 3. n2 has failed. > 4. cache.put(p, 1) // n3 has p->1, updateCounter=2 > 5. n3 has failed, partition loss is happened. > 6. n2 joins a topology, it has stale data (p->0) > > We actually have 2 issues: > 7. cache.put(p, 2) will success, n2 has p->2, n3 has p->0, data is diverged > and will not be adjusted by counters rebalancing if n3 is later joins a > topology. > or > 8. n3 joins a topology, it has actual data (p->1) but rebalancing will not > work because joining node has highest counter (it can only be a demander in > this scenario). > > In both cases rebalancing by counters will not work causing data divergence > in copies. > > > > > > >> at least one owner for each lost partition. > > What the reason to have owners for all lost partitions when we want to > > reset only some (available)? > > > > It's never were possible to reset only subset of lost partitions. The > reason is to make guarantee of resetLostPartitions method - all cache > operations are resumed, data is correct. > > > > Will it be possible to perform operations on non-lost partitions when the > > cluster has at least one lost partition? > > > > Yes it will be. > > > > > > On Wed, May 6, 2020 at 11:45 AM Alexei Scherbakov < > > alexey.scherbak...@gmail.com> wrote: > > > > > Folks, > > > > > > I've almost finished a patch bringing some improvements to the data > loss > > > handling code, and I wish to discuss proposed changes with the > community > > > before submitting. > > > > > > *The issue* > > > > > > During the grid's lifetime, it's possible to get into a situation when > > some > > > data nodes have failed or mistakenly stopped. If a number of stopped > > nodes > > > exceeds a certain threshold depending on configured backups, count a > data > > > loss will occur. For example, a grid having one backup (meaning at > least > > > two copies of each data partition exist at the same time) can tolerate > > only > > > one node loss at the time. Generally, data loss is guaranteed to occur > if > > > backups + 1 or more nodes have failed simultaneously using default > > affinity > > > function. > > > > > > For in-memory caches, data is lost forever. For persistent caches, data > > is > > > not physically lost and accessible again after failed nodes are > returned > > to > > > the topology. > > > > > > Possible data loss should be taken into consideration while designing > an > > > application. > > > > > > > > > > > > *Consider an example: money is transferred from one deposit to another, > > and > > > all nodes holding data for one of the deposits are gone.In such a > case, a > > > transaction temporary cannot be completed until a cluster is recovered > > from > > > the data loss state. Ignoring this can cause data inconsistency.* > > > It is necessary to have an API telling us if an operation is safe to > > > complete from the perspective of data loss. > > > > > > Such an API exists for some time [1] [2] [3]. In short, a grid can be > > > configured to switch caches to the partial availability mode if data > loss > > > is detected. > > > > > > Let's give two definitions according to the Javadoc for > > > *PartitionLossPolicy*: > > > > > > · *Safe* (data loss handling) *policy* - cache operations are only > > > available for non-lost partitions (PartitionLossPolicy != IGNORE). > > > > > > · *Unsafe policy* - cache operations are always possible > > > (PartitionLossPolicy = IGNORE). If the unsafe policy is configured, > lost > > > partitions automatically re-created on the remaining nodes if needed or > > > immediately owned if a last supplier has left during rebalancing. > > > > > > *That needs to be fixed* > > > > > > 1. The default loss policy is unsafe, even for persistent caches in the > > > current implementation. It can result in unintentional data loss and > > > business invariants' failure. > > > > > > 2. Node restarts in the persistent grid with detected data loss will > > cause > > > automatic resetting of LOST state after the restart, even if the safe > > > policy is configured. It can result in data loss or partition desync if > > not > > > all nodes are returned to the topology or returned in the wrong order. > > > > > > > > > *An example: a grid has three nodes, one backup. The grid is under > load. > > > First, a node2 has left, soon a node3 has left. If the node2 is > returned > > to > > > the topology first, it would have stale data for some keys. Most recent > > > data are on node3, which is not in the topology yet. Because a lost > state > > > was reset, all caches are fully available, and most probably will > become > > > inconsistent even in safe mode.* > > > 3. Configured loss policy doesn't provide guarantees described in the > > > Javadoc depending on the cluster configuration[4]. In particular, > unsafe > > > policy (IGNORE) cannot be guaranteed if a baseline is fixed (not > > > automatically readjusted on node left), because partitions are not > > > automatically get reassigned on topology change, and no nodes are > > existing > > > to fulfill a read/write request. Same for READ_ONLY_ALL and > > READ_WRITE_ALL. > > > > > > 4. Calling resetLostPartitions doesn't provide a guarantee for full > cache > > > operations availability if a topology doesn't have at least one owner > for > > > each lost partition. > > > > > > The ultimate goal of the patch is to fix API inconsistencies and fix > the > > > most crucial bugs related to data loss handling. > > > > > > *The planned changes are:* > > > > > > 1. The safe policy is used by default, except for in-memory grids with > > > enabled baseline auto-adjust [5] with zero timeout [6]. In the latter > > case, > > > the unsafe policy is used by default. It protects from unintentional > data > > > loss. > > > > > > 2. Lost state is never reset in the case of grid nodes restart (despite > > > full restart). It makes real data loss impossible in persistent grids > if > > > following the recovery instruction. > > > > > > 3. Lost state is impossible to reset if a topology doesn't have at > least > > > one owner for each lost partition. If nodes are physically dead, they > > > should be removed from a baseline first before calling > > resetLostPartitions. > > > > > > 4. READ_WRITE_ALL, READ_ONLY_ALL is a subject for deprecation because > > their > > > guarantees are impossible to fulfill, not on the full baseline. > > > > > > 5. Any operation failed due to data loss contains > > > CacheInvalidStateException as a root cause. > > > > > > In addition to code fixes, I plan to write a tutorial for safe data > loss > > > recovery in the persistent mode in the Ignite wiki. > > > > > > Any comments for the proposed changes are welcome. > > > > > > [1] > > > > > > > > > org.apache.ignite.configuration.CacheConfiguration#setPartitionLossPolicy(PartitionLossPolicy > > > partLossPlc) > > > [2] org.apache.ignite.Ignite#resetLostPartitions(caches) > > > [3] org.apache.ignite.IgniteCache#lostPartitions > > > [4] https://issues.apache.org/jira/browse/IGNITE-10041 > > > [5] org.apache.ignite.IgniteCluster#baselineAutoAdjustEnabled(boolean) > > > [6] org.apache.ignite.IgniteCluster#baselineAutoAdjustTimeout(long) > > > > > > > > -- > > Best regards, > Alexei Scherbakov >
