I might start tomorrow relaxing LWT transaction for users & domains. Let's see where it gets us!
Cheers, Benoit Le 20/10/2020 à 09:08, Tellier Benoit a écrit : > Hello there, > > Cassandra is an eventually consistent datastore, that can be used in a > consistant fashion. To do so, we rely on a mechanism called "LightWeight > Transactions (LWT)". Lightweight transactions relies on the PAXOS > distributed consensus algorithm to enforce a condition upon data > mutation. A table, system.paxos, is used to track the state of > transactions. Furthermore, upon writes, several round-trips (two) are > needed to ensure data integrity accross replica(minimum round trips to > achieve consensus) and the system.paxos table is read / written to in > addition to the applicative table. > > All of this causes LWT to be significantly slower than their lower > consistency counterparts. On some Linagora owned production instances, > regular reads takes 2ms while reads on tables relying on LWT takes 6ms. > Similar figures are found for writes. We also noticed some high > compaction throughtput on the paxos table, leading to many back-ground > writes. > > Given the massive impact of LWT usage on performance, and given the lack > of debate upon LWT adoption, I would like to re-challenge their usage... > > Here are the places we rely on LWT for the Distributed Server: > > - IMAP UID generation (monotic integer) - strong consistency is > strictly required to not loose data as overwriting a uid means > overwriting a message. > > - IMAP ModSeq generation (monotic integer) - strong consistency is > required, as modseq overwrites can lead to some data not being well > synchronised. > > - Domain and users - we rely on LWT to return an error when deleting a > user that do not exist, or creating an already existing user. It sounds > unecessary. > > - Message flags relies on LWT to ensure updates are not overwritten. As > an often read metadata, the impact is high, for limited criticity for > the end user. After all, no data is lost, only a user action like > marking a message as Seen, an action that he can very well perform again > > - Mailbox path registration, ACL - required to prevent data races > > My proposal would be: > > - Keep using LWT for UID and modseq generation, as well as Mailbox path > registration. > - Make the use of LWT for message flags update configurable - as an > admin I can choose to disable it. > - I am also fine with completly removing LWT usage for message flags > update. > - No longer use LWT on domain or users. Instead use idempotent create / > delete. The contract test will thus need to be relaxed. > - On the long term, relying on a CRDT to represent ACLs at the > Cassandra level, instead of serialized JSON, would enable to get rid of > LWT usage on the ACL table. > > Any feedback? > > Benoit > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org > For additional commands, e-mail: server-dev-h...@james.apache.org > --------------------------------------------------------------------- To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
