Hi Sachin, There is a command that can help you to find which backup is inconsistent: https://ignite.apache.org/docs/latest/tools/control-script#verifying-partition-checksums
Then you can clean only a single node which has a problem, and then Ignite will automatically rebalance data for the new node. Since Apache Ignite 2.11 there is the ReadRepair feature [1]. It actually does what you want to do. But it is experimental for 2.11 and 2.12, and will be more solid in 2.13. [1] https://ignite.apache.org/docs/2.11.1/read-repair On Mon, Feb 14, 2022 at 7:44 AM <sachin1.g...@ril.com> wrote: > Thanks for responding. > > > > Yes that is correct. So we are looking for some command from which we can > repair the data. This cache is configured with one backup. > > Possible solution we can think of > > > > 1) Export the entire cache and import it back. But it has 200million > rows. So we can have extended downtime. > > 2) Remove all the three node one at a time, clean them, add them > back to cluster and let ignite rebuild them as new nodes. > > 3) Change the cache config with readFromBackup = false. So we get > consistent data. > > > > > > Thanks, > > Sachin. > > > > > > *From:* Maksim Timonin <timoninma...@apache.org> > *Sent:* 12 February 2022 00:35 > *To:* user@ignite.apache.org > *Subject:* [External]Re: Ignite data not in sync > > > > The e-mail below is from an external source. Please do not open > attachments or click links from an unknown or suspicious origin. > > Hi! > > > > Am I correctly understanding your case: > > 1. There is a 3-node cluster > > 2. You restart one node > > 3. After restart you query a cache with the same query, and sometimes it > returns a value for the field "CLERK_ID" and sometimes not. > > > > Maksim > > > > On Fri, Feb 11, 2022 at 11:27 AM <sachin1.g...@ril.com> wrote: > > We are running Apache Ignite version 2.7, It is three node cluster. > > After restart of one of the Ignite node, we are getting inconsistent > select result on one of our cache. > > > > Is there any way we can remove this inconsistency ? Below is the couple > sql which clerk_id is intermittently visible. > > > > > > > > > > > > 0: jdbc:ignite:thin://10.142.49.43:10800 > <https://protect2.fireeye.com/v1/url?k=ae202bb7-f1bb6940-ae21a148-0cc47adc5e70-d32f15a40df0f7bf&q=1&e=7d299f0b-8ec3-41cd-987b-f98a402bbf31&u=http%3A%2F%2F10.142.49.43%3A10800%2F>> > select clerk_id,tstamp_trans from crawler_fin_l_r where > UNIQUENESS_KEY='893689191'; > > +--------------------------------+--------------------------------+ > > | CLERK_ID | TSTAMP_TRANS | > > +--------------------------------+--------------------------------+ > > | 1212652456 | 2022011018152249 | > > +--------------------------------+--------------------------------+ > > 1 row selected (0.017 seconds) > > 0: jdbc:ignite:thin://10.142.49.43:10800 > <https://protect2.fireeye.com/v1/url?k=8d6ee83f-d2f5aac8-8d6f62c0-0cc47adc5e70-0f4ca0bf14700892&q=1&e=7d299f0b-8ec3-41cd-987b-f98a402bbf31&u=http%3A%2F%2F10.142.49.43%3A10800%2F>> > select clerk_id,tstamp_trans from crawler_fin_l_r where > UNIQUENESS_KEY='893689191'; > > +--------------------------------+--------------------------------+ > > | CLERK_ID | TSTAMP_TRANS | > > +--------------------------------+--------------------------------+ > > | 1212652456 | 2022011018152249 | > > +--------------------------------+--------------------------------+ > > 1 row selected (0.002 seconds) > > 0: jdbc:ignite:thin://10.142.49.43:10800 > <https://protect2.fireeye.com/v1/url?k=e708ebd4-b893a923-e709612b-0cc47adc5e70-5643327425bdbf64&q=1&e=7d299f0b-8ec3-41cd-987b-f98a402bbf31&u=http%3A%2F%2F10.142.49.43%3A10800%2F>> > select clerk_id,tstamp_trans from crawler_fin_l_r where > UNIQUENESS_KEY='893689191'; > > +--------------------------------+--------------------------------+ > > | CLERK_ID | TSTAMP_TRANS | > > +--------------------------------+--------------------------------+ > > | | 2022011018152249 | > > +--------------------------------+--------------------------------+ > > 1 row selected (0.003 seconds) > > 0: jdbc:ignite:thin://10.142.49.43:10800 > <https://protect2.fireeye.com/v1/url?k=dba7e7df-843ca528-dba66d20-0cc47adc5e70-07f1ace70ea8e997&q=1&e=7d299f0b-8ec3-41cd-987b-f98a402bbf31&u=http%3A%2F%2F10.142.49.43%3A10800%2F>> > select clerk_id,tstamp_trans from crawler_fin_l_r where > UNIQUENESS_KEY='893689191'; > > +--------------------------------+--------------------------------+ > > | CLERK_ID | TSTAMP_TRANS | > > +--------------------------------+--------------------------------+ > > | | 2022011018152249 | > > +--------------------------------+--------------------------------+ > > 1 row selected (0.002 seconds) > > 0: jdbc:ignite:thin://10.142.49.43:10800 > <https://protect2.fireeye.com/v1/url?k=b93007af-e6ab4558-b9318d50-0cc47adc5e70-e3a72731027b5faa&q=1&e=7d299f0b-8ec3-41cd-987b-f98a402bbf31&u=http%3A%2F%2F10.142.49.43%3A10800%2F> > > > > > "*Confidentiality Warning*: This message and any attachments are intended > only for the use of the intended recipient(s), are confidential and may be > privileged. If you are not the intended recipient, you are hereby notified > that any review, re-transmission, conversion to hard copy, copying, > circulation or other use of this message and any attachments is strictly > prohibited. If you are not the intended recipient, please notify the sender > immediately by return email and delete this message and any attachments > from your system. > > *Virus Warning:* Although the company has taken reasonable precautions to > ensure no viruses are present in this email. The company cannot accept > responsibility for any loss or damage arising from the use of this email or > attachment." > > > "*Confidentiality Warning*: This message and any attachments are intended > only for the use of the intended recipient(s), are confidential and may be > privileged. If you are not the intended recipient, you are hereby notified > that any review, re-transmission, conversion to hard copy, copying, > circulation or other use of this message and any attachments is strictly > prohibited. If you are not the intended recipient, please notify the sender > immediately by return email and delete this message and any attachments > from your system. > > *Virus Warning:* Although the company has taken reasonable precautions to > ensure no viruses are present in this email. The company cannot accept > responsibility for any loss or damage arising from the use of this email or > attachment." >
