Igniters, Let me explain problem in detail. Read Repair at pessimistic tx (locks acquired on primary, full sync, 2pc) able to see consistency violation because backups are not updated yet. This seems to be not a good idea to "fix" code to unlock primary only when backups updated, this definitely will cause a performance drop. Currently, there is no explicit sync feature allows waiting for backups updated during the previous tx. Previous tx just sends GridNearTxFinishResponse to the originating node.
Bad ideas how to handle this: - retry some times (still possible to gain false positive) - lock tx entry on backups (will definitely break failover logic) - wait for same entry version on backups during some timeout (will require huge changes at "get" logic and false positive still possible) Is there any simple fix for this issue? Thanks for tips in advance. Ivan, thanks for your interest >> 4. Very fast and lucky txB writes a value 2 for the key on primary and backup. AFAIK, reordering not possible since backups "prepared" before primary releases lock. So, consistency guaranteed by failover and by "prepare" feature of 2PC. Seems, the problem is NOT with consistency at AI, but with consistency detection implementation (RR) and possible "false positive" results. BTW, checked 1PC case (only one data node at test) and gained no issues. On Fri, Jul 12, 2019 at 9:26 AM Павлухин Иван <vololo...@gmail.com> wrote: > Anton, > > Is such behavior observed for 2PC or for 1PC optimization? Does not it > mean that the things can be even worse and an inconsistent write is > possible on a backup? E.g. in scenario: > 1. txA writes a value 1 for the key on primary. > 2. txA unlocks the key on primary. > 3. txA freezes before updating backup. > 4. Very fast and lucky txB writes a value 2 for the key on primary and > backup. > 5. txB wakes up and writes 1 for the key. > 6. As result there is 2 on primary and 1 on backup. > > Naively it seems that locks should be released after all replicas are > updated. > > ср, 10 июл. 2019 г. в 16:36, Anton Vinogradov <a...@apache.org>: > > > > Folks, > > > > Investigating now unexpected repairs [1] in case of ReadRepair usage at > > testAccountTxNodeRestart. > > Updated [2] the test to check is there any repairs happen. > > Test's name now is "testAccountTxNodeRestartWithReadRepair". > > > > Each get method now checks the consistency. > > Check means: > > 1) tx lock acquired on primary > > 2) gained data from each owner (primary and backups) > > 3) data compared > > > > Sometime, backup may have obsolete value during such check. > > > > Seems, this happen because tx commit on primary going in the following > way > > (check code [2] for details): > > 1) performing localFinish (releases tx lock) > > 2) performing dhtFinish (commits on backups) > > 3) transferring control back to the caller > > > > So, seems, the problem here is that "tx lock released on primary" does > not > > mean that backups updated, but "commit() method finished at caller's > > thread" does. > > This means that, currently, there is no happens-before between > > 1) thread 1 committed data on primary and tx lock can be reobtained > > 2) thread 2 reads from backup > > but still strong HB between "commit() finished" and "backup updated" > > > > So, it seems to be possible, for example, to gain notification by a > > continuous query, then read from backup and gain obsolete value. > > > > Is this "partial happens before" behavior expected? > > > > [1] https://issues.apache.org/jira/browse/IGNITE-11973 > > [2] https://github.com/apache/ignite/pull/6679/files > > [3] > > > org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxLocal#finishTx > > > > -- > Best regards, > Ivan Pavlukhin >
