Re: About Cassandra Inconsistencies
Hello Raphael, On 20/02/2020 14:35, Raphaël Ouazana-Sustowski wrote: > Hello, > [...] > > > I don't see how you expect waiting for some time on one node of a > distributed system can help. Distributed concensus cannot be resolved > just by waiting some time on one node. > > >> - Confirm the inconsistency >> - Then fix it. > > > You still have the exact same window of error: between "confirm the > inconsistency" and "fix it" a concurrent operation can happen, either on > the same server, or on an other one. We agree on that, However, given the hypothesis that "a mailbox creation is time bounded", then we will be certain at the moment we "confirm" that the concurrent "mailbox creation" did complete. The "confirm" phase will then be able to distinguish inconsistencies from concurrent valid operations. I "forgot" to mention the hypothesis of validity in which the statement of previous email is valid. I'm sorry about it because I believe it ease the understanding of the above proposal. To discuss its limitation, a kill-the-world GC can very well invalidate it, causing a write to take longer that the aformentioned wait delay. That's why I agree with you that "no 'triking' can protect us from all possible errors". (I admit there are errors, I'm not pretending to come up with an approach that systematically solves any error.) > > Telling that the solution is pretty simple, if you can "fix it" in a > lightweight transaction assuring your assumption is correct. > > >> (night is generally of good advice) >> >> >> On 19/02/2020 23:27, Raphaël Ouazana-Sustowski wrote: >>> Hello, >>> >>> My main concern is that trying to solve some inconsistencies, we risk to >>> introduce some incorrectness. >>> >>> Your proposal is probably reducing the window of error, but it >>> potentially adds errors where we had only inconsistencies. Do we accept >>> that trade-off? >> An inconsistency is an error in itself, so I am. > > > IMHO not the same kind of error. An inconsistency is an error that has > been reported to the user. Here we can potentially blindly change some > valid data without warning the user. > > > Regards, > > Raphaël. > > > - > To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org > For additional commands, e-mail: server-dev-h...@james.apache.org > - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
Re: About Cassandra Inconsistencies
Hello, Le 20/02/2020 à 03:22, Tellier Benoit a écrit : Hello Raphael. You mention risk of cancelling a legitimate action that could have been taken as an inconsistency. To prevent that, a strategy based on delays could be effective as well: - Detect all the inconsistencies - Wait long enough to be sure ongoing operations are finished. 2 times the Cassandra driver time out should be enough. I don't see how you expect waiting for some time on one node of a distributed system can help. Distributed concensus cannot be resolved just by waiting some time on one node. - Confirm the inconsistency - Then fix it. You still have the exact same window of error: between "confirm the inconsistency" and "fix it" a concurrent operation can happen, either on the same server, or on an other one. Telling that the solution is pretty simple, if you can "fix it" in a lightweight transaction assuring your assumption is correct. (night is generally of good advice) On 19/02/2020 23:27, Raphaël Ouazana-Sustowski wrote: Hello, My main concern is that trying to solve some inconsistencies, we risk to introduce some incorrectness. Your proposal is probably reducing the window of error, but it potentially adds errors where we had only inconsistencies. Do we accept that trade-off? An inconsistency is an error in itself, so I am. IMHO not the same kind of error. An inconsistency is an error that has been reported to the user. Here we can potentially blindly change some valid data without warning the user. Regards, Raphaël. - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
Re: About Cassandra Inconsistencies
Hello Raphael. You mention risk of cancelling a legitimate action that could have been taken as an inconsistency. To prevent that, a strategy based on delays could be effective as well: - Detect all the inconsistencies - Wait long enough to be sure ongoing operations are finished. 2 times the Cassandra driver time out should be enough. - Confirm the inconsistency - Then fix it. (night is generally of good advice) On 19/02/2020 23:27, Raphaël Ouazana-Sustowski wrote: > Hello, > > My main concern is that trying to solve some inconsistencies, we risk to > introduce some incorrectness. > > Your proposal is probably reducing the window of error, but it > potentially adds errors where we had only inconsistencies. Do we accept > that trade-off? An inconsistency is an error in itself, so I am. Regards, Benoit > > Regards, > > Raphaël. > > Le 19/02/2020 à 09:10, Tellier Benoit a écrit : >> Hello Raphael, >> >> The simple fix could be : >> >> - 1. Iterating the in memory list of record >> - 2. For each record, refresh it (re read it from DB to be sure to have >> the latest version of it) >> - 3. Then detect inconsistencies (reading the associated >> denormalization table) >> >> Step 2., currently not implemented in [1], could be a way to mitigate >> these risks. >> >> [1] Solving mailbox inconsistencies: >> https://github.com/linagora/james-project/pull/3110 >> >> That way you decrease the concurrency window to the maximum. >> >> Cheers, >> >> Benoit >> >> On 18/02/2020 23:50, Raphaël Ouazana-Sustowski wrote: >>> Hello, >>> >>> Is there a way to solve inconsistencies while not adding new ones? >>> >>> When you solve inconsistencies, you have a view of data in memory and >>> try to fix it. If the data change between the moment you take this view >>> and the moment you try to fix the inconsistency, you risk to add an >>> other inconsistency, or worst to cancel a legitimate action. >>> >>> Do you see a way to mitigate this risk? >>> >>> Regards, >>> >>> Raphaël. >>> >>> Le 14/02/2020 à 11:06, Đức Trần Tiến a écrit : Hello people, In recent days, James had been recieved a report about the strage *MailboxNotFoundException*. There were some investigation to find the cause of the issue. We found that there are some inconsistent between mailbox and mailboxPathV2 tables. The issue is considered critical and it and bother/annoying users to creates, rename, delete mailboxes. I had open a PR for demonstrate this problem in unit test https://github.com/linagora/james-project/pull/3100. There are some taken actions to be resolving the mailbox inconsistencies. But look back to the bigger picture, there're many tables have close relationships, mailbox and mailboxPathV2 is just an example. There are several proposals to prevent inconsistent but some have not been discussed further than an idea: - Introducing a *transaction* bounded with cassandra (it can systematically resolve the issue) - Re write the MailboxManager by *EventSourcing* - Having tools to fix the already existing inconsistent *Having tools to fix the already existing inconsistent* We have: - Webadmin Task for solving inconsistent cassandra mappings - A self healing MessageFastViewProjections In developement: - Webadmin Task for solving inconsistent mailbox & mailboxPath - Retrying on failed mailbox creation/deletion I want to bring this conversation to the mailling list and hope that we can contribute solutions, enrich the proposals in order to reduce/fix the cassandra inconsistencies. Thank you! Tran Tien Duc >>> - >>> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org >>> For additional commands, e-mail: server-dev-h...@james.apache.org >>> >> - >> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org >> For additional commands, e-mail: server-dev-h...@james.apache.org >> >> > > - > To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org > For additional commands, e-mail: server-dev-h...@james.apache.org > - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
Re: About Cassandra Inconsistencies
Hello, My main concern is that trying to solve some inconsistencies, we risk to introduce some incorrectness. Your proposal is probably reducing the window of error, but it potentially adds errors where we had only inconsistencies. Do we accept that trade-off? Regards, Raphaël. Le 19/02/2020 à 09:10, Tellier Benoit a écrit : Hello Raphael, The simple fix could be : - 1. Iterating the in memory list of record - 2. For each record, refresh it (re read it from DB to be sure to have the latest version of it) - 3. Then detect inconsistencies (reading the associated denormalization table) Step 2., currently not implemented in [1], could be a way to mitigate these risks. [1] Solving mailbox inconsistencies: https://github.com/linagora/james-project/pull/3110 That way you decrease the concurrency window to the maximum. Cheers, Benoit On 18/02/2020 23:50, Raphaël Ouazana-Sustowski wrote: Hello, Is there a way to solve inconsistencies while not adding new ones? When you solve inconsistencies, you have a view of data in memory and try to fix it. If the data change between the moment you take this view and the moment you try to fix the inconsistency, you risk to add an other inconsistency, or worst to cancel a legitimate action. Do you see a way to mitigate this risk? Regards, Raphaël. Le 14/02/2020 à 11:06, Đức Trần Tiến a écrit : Hello people, In recent days, James had been recieved a report about the strage *MailboxNotFoundException*. There were some investigation to find the cause of the issue. We found that there are some inconsistent between mailbox and mailboxPathV2 tables. The issue is considered critical and it and bother/annoying users to creates, rename, delete mailboxes. I had open a PR for demonstrate this problem in unit test https://github.com/linagora/james-project/pull/3100. There are some taken actions to be resolving the mailbox inconsistencies. But look back to the bigger picture, there're many tables have close relationships, mailbox and mailboxPathV2 is just an example. There are several proposals to prevent inconsistent but some have not been discussed further than an idea: - Introducing a *transaction* bounded with cassandra (it can systematically resolve the issue) - Re write the MailboxManager by *EventSourcing* - Having tools to fix the already existing inconsistent *Having tools to fix the already existing inconsistent* We have: - Webadmin Task for solving inconsistent cassandra mappings - A self healing MessageFastViewProjections In developement: - Webadmin Task for solving inconsistent mailbox & mailboxPath - Retrying on failed mailbox creation/deletion I want to bring this conversation to the mailling list and hope that we can contribute solutions, enrich the proposals in order to reduce/fix the cassandra inconsistencies. Thank you! Tran Tien Duc - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
Re: About Cassandra Inconsistencies
I was not knowing this pattern. I would not use it as it prevents concurrent execution. (You can very well refresh and check entries in parrallel to go faster) On 19/02/2020 16:38, Đức Trần Tiến wrote: > Hi, > What do you think about `.setFetchSize()` to 1 or a number small enough? > This helps to avoid re-read and by the same time, the whole process > requires only one connection to read from a table. > > The cassandra documentation dis-encourage this practice because of its poor > performance. But we're expecting a task to be taking really long. > > Vào Th 4, 19 thg 2, 2020 vào lúc 15:10 Tellier Benoit > đã viết: > >> Hello Raphael, >> >> The simple fix could be : >> >> - 1. Iterating the in memory list of record >> - 2. For each record, refresh it (re read it from DB to be sure to have >> the latest version of it) >> - 3. Then detect inconsistencies (reading the associated >> denormalization table) >> >> Step 2., currently not implemented in [1], could be a way to mitigate >> these risks. >> >> [1] Solving mailbox inconsistencies: >>https://github.com/linagora/james-project/pull/3110 >> >> That way you decrease the concurrency window to the maximum. >> >> Cheers, >> >> Benoit >> >> On 18/02/2020 23:50, Raphaël Ouazana-Sustowski wrote: >>> Hello, >>> >>> Is there a way to solve inconsistencies while not adding new ones? >>> >>> When you solve inconsistencies, you have a view of data in memory and >>> try to fix it. If the data change between the moment you take this view >>> and the moment you try to fix the inconsistency, you risk to add an >>> other inconsistency, or worst to cancel a legitimate action. >>> >>> Do you see a way to mitigate this risk? >>> >>> Regards, >>> >>> Raphaël. >>> >>> Le 14/02/2020 à 11:06, Đức Trần Tiến a écrit : Hello people, In recent days, James had been recieved a report about the strage *MailboxNotFoundException*. There were some investigation to find the cause of the issue. We found that there are some inconsistent between mailbox and mailboxPathV2 tables. The issue is considered critical and it and bother/annoying >> users to creates, rename, delete mailboxes. I had open a PR for demonstrate this problem in unit test https://github.com/linagora/james-project/pull/3100. There are some >> taken actions to be resolving the mailbox inconsistencies. But look back to the bigger picture, there're many tables have close relationships, mailbox and mailboxPathV2 is just an example. There are several proposals to prevent inconsistent but some have not been discussed further than an idea: - Introducing a *transaction* bounded with cassandra (it can systematically resolve the issue) - Re write the MailboxManager by *EventSourcing* - Having tools to fix the already existing inconsistent *Having tools to fix the already existing inconsistent* We have: - Webadmin Task for solving inconsistent cassandra mappings - A self healing MessageFastViewProjections In developement: - Webadmin Task for solving inconsistent mailbox & mailboxPath - Retrying on failed mailbox creation/deletion I want to bring this conversation to the mailling list and hope that we can contribute solutions, enrich the proposals in order to reduce/fix the cassandra inconsistencies. Thank you! Tran Tien Duc >>> >>> - >>> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org >>> For additional commands, e-mail: server-dev-h...@james.apache.org >>> >> >> - >> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org >> For additional commands, e-mail: server-dev-h...@james.apache.org >> >> > - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
Re: About Cassandra Inconsistencies
Hi, What do you think about `.setFetchSize()` to 1 or a number small enough? This helps to avoid re-read and by the same time, the whole process requires only one connection to read from a table. The cassandra documentation dis-encourage this practice because of its poor performance. But we're expecting a task to be taking really long. Vào Th 4, 19 thg 2, 2020 vào lúc 15:10 Tellier Benoit đã viết: > Hello Raphael, > > The simple fix could be : > > - 1. Iterating the in memory list of record > - 2. For each record, refresh it (re read it from DB to be sure to have > the latest version of it) > - 3. Then detect inconsistencies (reading the associated > denormalization table) > > Step 2., currently not implemented in [1], could be a way to mitigate > these risks. > > [1] Solving mailbox inconsistencies: >https://github.com/linagora/james-project/pull/3110 > > That way you decrease the concurrency window to the maximum. > > Cheers, > > Benoit > > On 18/02/2020 23:50, Raphaël Ouazana-Sustowski wrote: > > Hello, > > > > Is there a way to solve inconsistencies while not adding new ones? > > > > When you solve inconsistencies, you have a view of data in memory and > > try to fix it. If the data change between the moment you take this view > > and the moment you try to fix the inconsistency, you risk to add an > > other inconsistency, or worst to cancel a legitimate action. > > > > Do you see a way to mitigate this risk? > > > > Regards, > > > > Raphaël. > > > > Le 14/02/2020 à 11:06, Đức Trần Tiến a écrit : > >> Hello people, > >> > >> In recent days, James had been recieved a report about the strage > >> *MailboxNotFoundException*. There were some investigation to find the > >> cause > >> of the issue. > >> We found that there are some inconsistent between mailbox and > >> mailboxPathV2 > >> tables. The issue is considered critical and it and bother/annoying > users > >> to creates, rename, delete mailboxes. > >> I had open a PR for demonstrate this problem in unit test > >> https://github.com/linagora/james-project/pull/3100. There are some > taken > >> actions to be resolving the mailbox inconsistencies. > >> > >> But look back to the bigger picture, there're many tables have close > >> relationships, mailbox and mailboxPathV2 is just an example. There are > >> several proposals to prevent inconsistent but some have not been > >> discussed > >> further than an idea: > >> > >> - Introducing a *transaction* bounded with cassandra (it can > >> systematically resolve the issue) > >> - Re write the MailboxManager by *EventSourcing* > >> - Having tools to fix the already existing inconsistent > >> > >> *Having tools to fix the already existing inconsistent* > >> > >> We have: > >> - Webadmin Task for solving inconsistent cassandra mappings > >> - A self healing MessageFastViewProjections > >> > >> In developement: > >> - Webadmin Task for solving inconsistent mailbox & mailboxPath > >> - Retrying on failed mailbox creation/deletion > >> > >> I want to bring this conversation to the mailling list and hope that > >> we can > >> contribute solutions, enrich the proposals in order to reduce/fix the > >> cassandra inconsistencies. > >> > >> Thank you! > >> > >> Tran Tien Duc > >> > > > > - > > To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org > > For additional commands, e-mail: server-dev-h...@james.apache.org > > > > - > To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org > For additional commands, e-mail: server-dev-h...@james.apache.org > >
Re: About Cassandra Inconsistencies
Hello Raphael, The simple fix could be : - 1. Iterating the in memory list of record - 2. For each record, refresh it (re read it from DB to be sure to have the latest version of it) - 3. Then detect inconsistencies (reading the associated denormalization table) Step 2., currently not implemented in [1], could be a way to mitigate these risks. [1] Solving mailbox inconsistencies: https://github.com/linagora/james-project/pull/3110 That way you decrease the concurrency window to the maximum. Cheers, Benoit On 18/02/2020 23:50, Raphaël Ouazana-Sustowski wrote: > Hello, > > Is there a way to solve inconsistencies while not adding new ones? > > When you solve inconsistencies, you have a view of data in memory and > try to fix it. If the data change between the moment you take this view > and the moment you try to fix the inconsistency, you risk to add an > other inconsistency, or worst to cancel a legitimate action. > > Do you see a way to mitigate this risk? > > Regards, > > Raphaël. > > Le 14/02/2020 à 11:06, Đức Trần Tiến a écrit : >> Hello people, >> >> In recent days, James had been recieved a report about the strage >> *MailboxNotFoundException*. There were some investigation to find the >> cause >> of the issue. >> We found that there are some inconsistent between mailbox and >> mailboxPathV2 >> tables. The issue is considered critical and it and bother/annoying users >> to creates, rename, delete mailboxes. >> I had open a PR for demonstrate this problem in unit test >> https://github.com/linagora/james-project/pull/3100. There are some taken >> actions to be resolving the mailbox inconsistencies. >> >> But look back to the bigger picture, there're many tables have close >> relationships, mailbox and mailboxPathV2 is just an example. There are >> several proposals to prevent inconsistent but some have not been >> discussed >> further than an idea: >> >> - Introducing a *transaction* bounded with cassandra (it can >> systematically resolve the issue) >> - Re write the MailboxManager by *EventSourcing* >> - Having tools to fix the already existing inconsistent >> >> *Having tools to fix the already existing inconsistent* >> >> We have: >> - Webadmin Task for solving inconsistent cassandra mappings >> - A self healing MessageFastViewProjections >> >> In developement: >> - Webadmin Task for solving inconsistent mailbox & mailboxPath >> - Retrying on failed mailbox creation/deletion >> >> I want to bring this conversation to the mailling list and hope that >> we can >> contribute solutions, enrich the proposals in order to reduce/fix the >> cassandra inconsistencies. >> >> Thank you! >> >> Tran Tien Duc >> > > - > To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org > For additional commands, e-mail: server-dev-h...@james.apache.org > - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
Re: About Cassandra Inconsistencies
Hello, Is there a way to solve inconsistencies while not adding new ones? When you solve inconsistencies, you have a view of data in memory and try to fix it. If the data change between the moment you take this view and the moment you try to fix the inconsistency, you risk to add an other inconsistency, or worst to cancel a legitimate action. Do you see a way to mitigate this risk? Regards, Raphaël. Le 14/02/2020 à 11:06, Đức Trần Tiến a écrit : Hello people, In recent days, James had been recieved a report about the strage *MailboxNotFoundException*. There were some investigation to find the cause of the issue. We found that there are some inconsistent between mailbox and mailboxPathV2 tables. The issue is considered critical and it and bother/annoying users to creates, rename, delete mailboxes. I had open a PR for demonstrate this problem in unit test https://github.com/linagora/james-project/pull/3100. There are some taken actions to be resolving the mailbox inconsistencies. But look back to the bigger picture, there're many tables have close relationships, mailbox and mailboxPathV2 is just an example. There are several proposals to prevent inconsistent but some have not been discussed further than an idea: - Introducing a *transaction* bounded with cassandra (it can systematically resolve the issue) - Re write the MailboxManager by *EventSourcing* - Having tools to fix the already existing inconsistent *Having tools to fix the already existing inconsistent* We have: - Webadmin Task for solving inconsistent cassandra mappings - A self healing MessageFastViewProjections In developement: - Webadmin Task for solving inconsistent mailbox & mailboxPath - Retrying on failed mailbox creation/deletion I want to bring this conversation to the mailling list and hope that we can contribute solutions, enrich the proposals in order to reduce/fix the cassandra inconsistencies. Thank you! Tran Tien Duc - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
Re: About Cassandra Inconsistencies
Hello Duc, answers inlined. On 14/02/2020 17:06, Đức Trần Tiến wrote: > Hello people, > > In recent days, James had been recieved a report about the strage > *MailboxNotFoundException*. There were some investigation to find the cause > of the issue. > We found that there are some inconsistent between mailbox and mailboxPathV2 > tables. The issue is considered critical and it and bother/annoying users > to creates, rename, delete mailboxes. > I had open a PR for demonstrate this problem in unit test > https://github.com/linagora/james-project/pull/3100. There are some taken > actions to be resolving the mailbox inconsistencies. > > But look back to the bigger picture, there're many tables have close > relationships, mailbox and mailboxPathV2 is just an example. I just want to add that mailboxPath/mailbox projection is specific as it is a rare place we use a projection for holding an invariant (not too mailboxes shall have the same name). Our other projections are based on a "source_of_truth + projection" model. WHich is easier to handle. > There are > several proposals to prevent inconsistent but some have not been discussed > further than an idea: > > - Introducing a *transaction* bounded with cassandra (it can > systematically resolve the issue) I don't understand this line as Cassandra don't allow transactions further than the Row. Can you describe what you are referring to? > - Re write the MailboxManager by *EventSourcing* I propose myself to investigate "mailbox/cassandra" projections requiring a task to fix them. > - Having tools to fix the already existing inconsistent > > *Having tools to fix the already existing inconsistent* > > We have: > - Webadmin Task for solving inconsistent cassandra mappings > - A self healing MessageFastViewProjections > > In developement: > - Webadmin Task for solving inconsistent mailbox & mailboxPath > - Retrying on failed mailbox creation/deletion > > I want to bring this conversation to the mailling list and hope that we can > contribute solutions, enrich the proposals in order to reduce/fix the > cassandra inconsistencies. > > Thank you! > > Tran Tien Duc > - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
