I've linked to IGNITE-10078 additional related/dependent tickets requiring investigation as soon as main contribution will be accepted.
вт, 7 мая 2019 г. в 18:10, Alexei Scherbakov <alexey.scherbak...@gmail.com>: > Anton, > > It's ready for review, look for Patch Available status. > Yes, atomic caches are not fixed by this contribution. See [1] > > [1] https://issues.apache.org/jira/browse/IGNITE-11797 > > вт, 7 мая 2019 г. в 17:30, Anton Vinogradov <a...@apache.org>: > >> Alexei, >> >> Got it. >> Could you please let me know once PR will be ready for review? >> Currently, have some questions, but, possible, they caused by non-final PR >> (eg. why atomic counter still ignores misses?). >> >> On Tue, May 7, 2019 at 4:43 PM Alexei Scherbakov < >> alexey.scherbak...@gmail.com> wrote: >> >> > Anton, >> > >> > 1) Extended counters indeed will answer the question if partition could >> be >> > safely restored to synchronized state on all owners. >> > The only condition - one of owners has no missed updates. >> > If not, partition must be moved to LOST state, see [1], >> > TxPartitionCounterStateOnePrimaryTwoBackupsFailAll*Test, >> > >> > >> IgniteSystemProperties#IGNITE_FAIL_NODE_ON_UNRECOVERABLE_PARTITION_INCONSISTENCY >> > This is known issue and could happen if all partition owners were >> > unavailable at some point. >> > In such case we could try to recover consistency using some complex >> > recovery protocol as you described. Related ticket [2] >> > >> > 2) Bitset implementation is considered as an option in GG Community >> > Edition. No specific implementation dates at the moment. >> > >> > 3) As for "online" partition verification, I think the best option >> right >> > now is to do verification partition by partition using read only mode >> per >> > group partition under load. >> > While verification is in progress, all write ops are waiting, not >> rejected. >> > This is only 100% reliable way to compare partitions - by touching >> actual >> > data, all other ways like pre-computed hash are error prone. >> > There is already ticket [3] for simplifing grid consistency verification >> > which could be used as basis for such functionality. >> > As for avoiding cache pollution, we could try read pages sequentially >> from >> > disk without lifting them to pagemem and computing some kind of >> commutative >> > hash. It's safe under partition write lock. >> > >> > [1] https://issues.apache.org/jira/browse/IGNITE-11611 >> > [2] https://issues.apache.org/jira/browse/IGNITE-6324 >> > [3] https://issues.apache.org/jira/browse/IGNITE-11256 >> > >> > пн, 6 мая 2019 г. в 16:12, Anton Vinogradov <a...@apache.org>: >> > >> > > Ivan, >> > > >> > > 1) I've checked the PR [1] and it looks like it does not solve the >> issue >> > > too. >> > > AFAICS, the main goal here (at PR) is to produce >> > > PartitionUpdateCounter#sequential which can be false for all backups, >> > what >> > > backup should win in that case? >> > > >> > > Is there any IEP or some another design page for this fix? >> > > >> > > Looks like extended counters should be able to recover the whole >> cluster >> > > even in case all copies of the same partition are broken. >> > > So, seems, the counter should provide detailed info: >> > > - biggest applied updateCounter >> > > - list of all missed counters before biggest applied >> > > - optional hash >> > > >> > > In that case, we'll be able to perform some exchange between broken >> > copies. >> > > For example, we'll found that copy1 missed key1, and copy2 missed >> key2. >> > > It's pretty simple to fix both copies in that case. >> > > In case all misses can be solved this way, we'll continue cluster >> > > activation like it was not broken before. >> > > >> > > 2) Seems I see the simpler solution to handle misses (than at PR). >> > > Once you have newUpdateCounter > curUpdateCounter + 1, you should add >> > byte >> > > (or int or long (smaplest possible)) value to special structure. >> > > This value will represent delta between newUpdateCounter and >> > > curUpdateCounter in bitmask way. >> > > In case you'll handle updateCounter less that curUpdateCounter, you >> > should >> > > update the value at structure responsible to this delta. >> > > For example, when you have delta "2 to 6", you will have 00000000 >> > initially >> > > and 00011111 finally. >> > > Each delta update should be finished with check it completed (value >> == 31 >> > > in this case). Once it finished, it should be removed from the >> structure. >> > > Deltas can and should be reused to solve GC issue. >> > > >> > > What do you think about the proposed solution? >> > > >> > > 3) Hash computation can be an additional extension for extended >> counters, >> > > just one more dimension to be extremely sure everything is ok. >> > > Any objections? >> > > >> > > [1] https://github.com/apache/ignite/pull/5765 >> > > >> > > On Mon, May 6, 2019 at 12:48 PM Ivan Rakov <ivan.glu...@gmail.com> >> > wrote: >> > > >> > > > Anton, >> > > > >> > > > Automatic quorum-based partition drop may work as a partial >> workaround >> > > > for IGNITE-10078, but discussed approach surely doesn't replace >> > > > IGNITE-10078 activity. We still don't know what do to when quorum >> can't >> > > > be reached (2 partitions have hash X, 2 have hash Y) and keeping >> > > > extended update counters is the only way to resolve such case. >> > > > On the other hand, precalculated partition hashes validation on PME >> can >> > > > be a good addition to IGNITE-10078 logic: we'll be able to detect >> > > > situations when extended update counters are equal, but for some >> reason >> > > > (bug or whatsoever) partition contents are different. >> > > > >> > > > Best Regards, >> > > > Ivan Rakov >> > > > >> > > > On 06.05.2019 12:27, Anton Vinogradov wrote: >> > > > > Ivan, just to make sure ... >> > > > > The discussed case will fully solve the issue [1] in case we'll >> also >> > > add >> > > > > some strategy to reject partitions with missed updates >> > (updateCnt==Ok, >> > > > > Hash!=Ok). >> > > > > For example, we may use the Quorum strategy, when the majority >> wins. >> > > > > Sounds correct? >> > > > > >> > > > > [1] https://issues.apache.org/jira/browse/IGNITE-10078 >> > > > > >> > > > > On Tue, Apr 30, 2019 at 3:14 PM Anton Vinogradov <a...@apache.org> >> > > wrote: >> > > > > >> > > > >> Ivan, >> > > > >> >> > > > >> Thanks for the detailed explanation. >> > > > >> I'll try to implement the PoC to check the idea. >> > > > >> >> > > > >> On Mon, Apr 29, 2019 at 8:22 PM Ivan Rakov < >> ivan.glu...@gmail.com> >> > > > wrote: >> > > > >> >> > > > >>>> But how to keep this hash? >> > > > >>> I think, we can just adopt way of storing partition update >> > counters. >> > > > >>> Update counters are: >> > > > >>> 1) Kept and updated in heap, see >> > > > >>> IgniteCacheOffheapManagerImpl.CacheDataStoreImpl#pCntr (accessed >> > > during >> > > > >>> regular cache operations, no page replacement latency issues) >> > > > >>> 2) Synchronized with page memory (and with disk) on every >> > checkpoint, >> > > > >>> see GridCacheOffheapManager#saveStoreMetadata >> > > > >>> 3) Stored in partition meta page, see >> > > > PagePartitionMetaIO#setUpdateCounter >> > > > >>> 4) On node restart, we init onheap counter with value from disk >> > (for >> > > > the >> > > > >>> moment of last checkpoint) and update it to latest value during >> WAL >> > > > >>> logical records replay >> > > > >>> >> > > > >>>> 2) PME is a rare operation on production cluster, but, seems, >> we >> > > have >> > > > >>>> to check consistency in a regular way. >> > > > >>>> Since we have to finish all operations before the check, >> should we >> > > > >>>> have fake PME for maintenance check in this case? >> > > > >>> From my experience, PME happens on prod clusters from time to >> > time >> > > > >>> (several times per week), which can be enough. In case it's >> needed >> > to >> > > > >>> check consistency more often than regular PMEs occur, we can >> > > implement >> > > > >>> command that will trigger fake PME for consistency checking. >> > > > >>> >> > > > >>> Best Regards, >> > > > >>> Ivan Rakov >> > > > >>> >> > > > >>> On 29.04.2019 18:53, Anton Vinogradov wrote: >> > > > >>>> Ivan, thanks for the analysis! >> > > > >>>> >> > > > >>>>>> With having pre-calculated partition hash value, we can >> > > > >>>> automatically detect inconsistent partitions on every PME. >> > > > >>>> Great idea, seems this covers all broken synс cases. >> > > > >>>> >> > > > >>>> It will check alive nodes in case the primary failed >> immediately >> > > > >>>> and will check rejoining node once it finished a rebalance >> (PME on >> > > > >>>> becoming an owner). >> > > > >>>> Recovered cluster will be checked on activation PME (or even >> > before >> > > > >>>> that?). >> > > > >>>> Also, warmed cluster will be still warmed after check. >> > > > >>>> >> > > > >>>> Have I missed some cases leads to broken sync except bugs? >> > > > >>>> >> > > > >>>> 1) But how to keep this hash? >> > > > >>>> - It should be automatically persisted on each checkpoint (it >> > should >> > > > >>>> not require recalculation on restore, snapshots should be >> covered >> > > too) >> > > > >>>> (and covered by WAL?). >> > > > >>>> - It should be always available at RAM for every partition >> (even >> > for >> > > > >>>> cold partitions never updated/readed on this node) to be >> > immediately >> > > > >>>> used once all operations done on PME. >> > > > >>>> >> > > > >>>> Can we have special pages to keep such hashes and never allow >> > their >> > > > >>>> eviction? >> > > > >>>> >> > > > >>>> 2) PME is a rare operation on production cluster, but, seems, >> we >> > > have >> > > > >>>> to check consistency in a regular way. >> > > > >>>> Since we have to finish all operations before the check, >> should we >> > > > >>>> have fake PME for maintenance check in this case? >> > > > >>>> >> > > > >>>> On Mon, Apr 29, 2019 at 4:59 PM Ivan Rakov < >> ivan.glu...@gmail.com >> > > > >>>> <mailto:ivan.glu...@gmail.com>> wrote: >> > > > >>>> >> > > > >>>> Hi Anton, >> > > > >>>> >> > > > >>>> Thanks for sharing your ideas. >> > > > >>>> I think your approach should work in general. I'll just >> share >> > > my >> > > > >>>> concerns about possible issues that may come up. >> > > > >>>> >> > > > >>>> 1) Equality of update counters doesn't imply equality of >> > > > >>>> partitions content under load. >> > > > >>>> For every update, primary node generates update counter >> and >> > > then >> > > > >>>> update is delivered to backup node and gets applied with >> the >> > > > >>>> corresponding update counter. For example, there are two >> > > > >>>> transactions (A and B) that update partition X by the >> > following >> > > > >>>> scenario: >> > > > >>>> - A updates key1 in partition X on primary node and >> > increments >> > > > >>>> counter to 10 >> > > > >>>> - B updates key2 in partition X on primary node and >> > increments >> > > > >>>> counter to 11 >> > > > >>>> - While A is still updating another keys, B is finally >> > > committed >> > > > >>>> - Update of key2 arrives to backup node and sets update >> > counter >> > > > to >> > > > >>> 11 >> > > > >>>> Observer will see equal update counters (11), but update >> of >> > > key 1 >> > > > >>>> is still missing in the backup partition. >> > > > >>>> This is a fundamental problem which is being solved here: >> > > > >>>> https://issues.apache.org/jira/browse/IGNITE-10078 >> > > > >>>> "Online verify" should operate with new complex update >> > counters >> > > > >>>> which take such "update holes" into account. Otherwise, >> > online >> > > > >>>> verify may provide false-positive inconsistency reports. >> > > > >>>> >> > > > >>>> 2) Acquisition and comparison of update counters is fast, >> but >> > > > >>>> partition hash calculation is long. We should check that >> > update >> > > > >>>> counter remains unchanged after every K keys handled. >> > > > >>>> >> > > > >>>> 3) >> > > > >>>> >> > > > >>>>> Another hope is that we'll be able to pause/continue >> scan, >> > for >> > > > >>>>> example, we'll check 1/3 partitions today, 1/3 tomorrow, >> and >> > > in >> > > > >>>>> three days we'll check the whole cluster. >> > > > >>>> Totally makes sense. >> > > > >>>> We may find ourselves into a situation where some "hot" >> > > > partitions >> > > > >>>> are still unprocessed, and every next attempt to calculate >> > > > >>>> partition hash fails due to another concurrent update. We >> > > should >> > > > >>>> be able to track progress of validation (% of calculation >> > time >> > > > >>>> wasted due to concurrent operations may be a good metric, >> > 100% >> > > is >> > > > >>>> the worst case) and provide option to stop/pause activity. >> > > > >>>> I think, pause should return an "intermediate results >> report" >> > > > with >> > > > >>>> information about which partitions have been successfully >> > > > checked. >> > > > >>>> With such report, we can resume activity later: partitions >> > from >> > > > >>>> report will be just skipped. >> > > > >>>> >> > > > >>>> 4) >> > > > >>>> >> > > > >>>>> Since "Idle verify" uses regular pagmem, I assume it >> > replaces >> > > > hot >> > > > >>>>> data with persisted. >> > > > >>>>> So, we have to warm up the cluster after each check. >> > > > >>>>> Are there any chances to check without cooling the >> cluster? >> > > > >>>> I don't see an easy way to achieve it with our page memory >> > > > >>>> architecture. We definitely can't just read pages from >> disk >> > > > >>>> directly: we need to synchronize page access with >> concurrent >> > > > >>>> update operations and checkpoints. >> > > > >>>> From my point of view, the correct way to solve this >> issue is >> > > > >>>> improving our page replacement [1] mechanics by making it >> > truly >> > > > >>>> scan-resistant. >> > > > >>>> >> > > > >>>> P. S. There's another possible way of achieving online >> > verify: >> > > > >>>> instead of on-demand hash calculation, we can always keep >> > > > >>>> up-to-date hash value for every partition. We'll need to >> > update >> > > > >>>> hash on every insert/update/remove operation, but there >> will >> > be >> > > > no >> > > > >>>> reordering issues as per function that we use for >> aggregating >> > > > hash >> > > > >>>> results (+) is commutative. With having pre-calculated >> > > partition >> > > > >>>> hash value, we can automatically detect inconsistent >> > partitions >> > > > on >> > > > >>>> every PME. What do you think? >> > > > >>>> >> > > > >>>> [1] - >> > > > >>>> >> > > > >>> >> > > > >> > > >> > >> https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Durable+Memory+-+under+the+hood#IgniteDurableMemory-underthehood-Pagereplacement(rotationwithdisk) >> > > > >>>> Best Regards, >> > > > >>>> Ivan Rakov >> > > > >>>> >> > > > >>>> On 29.04.2019 12:20, Anton Vinogradov wrote: >> > > > >>>>> Igniters and especially Ivan Rakov, >> > > > >>>>> >> > > > >>>>> "Idle verify" [1] is a really cool tool, to make sure >> that >> > > > >>>>> cluster is consistent. >> > > > >>>>> >> > > > >>>>> 1) But it required to have operations paused during >> cluster >> > > > check. >> > > > >>>>> At some clusters, this check requires hours (3-4 hours at >> > > cases >> > > > I >> > > > >>>>> saw). >> > > > >>>>> I've checked the code of "idle verify" and it seems it >> > > possible >> > > > >>>>> to make it "online" with some assumptions. >> > > > >>>>> >> > > > >>>>> Idea: >> > > > >>>>> Currently "Idle verify" checks that partitions hashes, >> > > generated >> > > > >>>>> this way >> > > > >>>>> while (it.hasNextX()) { >> > > > >>>>> CacheDataRow row = it.nextX(); >> > > > >>>>> partHash += row.key().hashCode(); >> > > > >>>>> partHash += >> > > > >>>>> >> > > > >>> >> > > Arrays.hashCode(row.value().valueBytes(grpCtx.cacheObjectContext())); >> > > > >>>>> } >> > > > >>>>> , are the same. >> > > > >>>>> >> > > > >>>>> What if we'll generate same pairs >> > updateCounter-partitionHash >> > > > but >> > > > >>>>> will compare hashes only in case counters are the same? >> > > > >>>>> So, for example, will ask cluster to generate pairs for >> 64 >> > > > >>>>> partitions, then will find that 55 have the same counters >> > (was >> > > > >>>>> not updated during check) and check them. >> > > > >>>>> The rest (64-55 = 9) partitions will be re-requested and >> > > > >>>>> rechecked with an additional 55. >> > > > >>>>> This way we'll be able to check cluster is consistent >> even >> > in >> > > > >>>>> сase operations are in progress (just retrying modified). >> > > > >>>>> >> > > > >>>>> Risks and assumptions: >> > > > >>>>> Using this strategy we'll check the cluster's consistency >> > ... >> > > > >>>>> eventually, and the check will take more time even on an >> > idle >> > > > >>>>> cluster. >> > > > >>>>> In case operationsPerTimeToGeneratePartitionHashes > >> > > > >>>>> partitionsCount we'll definitely gain no progress. >> > > > >>>>> But, in case of the load is not high, we'll be able to >> check >> > > all >> > > > >>>>> cluster. >> > > > >>>>> >> > > > >>>>> Another hope is that we'll be able to pause/continue >> scan, >> > for >> > > > >>>>> example, we'll check 1/3 partitions today, 1/3 tomorrow, >> and >> > > in >> > > > >>>>> three days we'll check the whole cluster. >> > > > >>>>> >> > > > >>>>> Have I missed something? >> > > > >>>>> >> > > > >>>>> 2) Since "Idle verify" uses regular pagmem, I assume it >> > > replaces >> > > > >>>>> hot data with persisted. >> > > > >>>>> So, we have to warm up the cluster after each check. >> > > > >>>>> Are there any chances to check without cooling the >> cluster? >> > > > >>>>> >> > > > >>>>> [1] >> > > > >>>>> >> > > > >>> >> > > > >> > > >> > >> https://apacheignite-tools.readme.io/docs/control-script#section-verification-of-partition-checksums >> > > > >> > > >> > >> > >> > -- >> > >> > Best regards, >> > Alexei Scherbakov >> > >> > > > -- > > Best regards, > Alexei Scherbakov > -- Best regards, Alexei Scherbakov
