Hi folks,Thanks for discussion on this proposal, and also to Benedict who’s
been fielding questions on the list!I’d like to restate the goals and problem
statement captured by this proposal and frame context.Today, lightweight
transactions limit users to transacting over a single partition. This unit of
atomicity has a very low upper limit in terms of the amount of data that can be
CAS’d over; and doing so leads many to design contorted data models to cram
different types of data into one partition for the purposes of being able to
CAS over it. We propose that Cassandra can and should be extended to remove
this limit, enabling users to issue one-shot transactions that CAS over
multiple keys – including CAS batches, which may modify multiple keys.To enable
this, the CEP authors have designed a novel, leaderless paxos-based protocol
unique to Cassandra, offered a proof of its correctness, a whitepaper outlining
it in detail, along with a prototype implementation to incubate development,
and integrated it with Maelstrom from jepsen.io to validate linearizability as
more specific test infrastructure is developed. This rigor is remarkable, and
I’m thrilled to see such a degree of investment in the area.Even users who do
not require the capability to transact across partition boundaries will
benefit. The protocol reduces message/WAN round-trips by 4x on writes (4 → 1)
and 2x on reads (2 → 1) in the common case against today’s baseline. These
latency improvements coupled with the enhanced flexibility of what can be
transacted over in Cassandra enable new classes of applications to use the
database.In particular, 1xRTT read/write transactions across partitions enable
Cassandra to be thought of not just as a strongly consistent database, but even
a transactional database - a mode many may even prefer to use by default. Given
this capability, Apache Cassandra has an opportunity to become one of – or
perhaps the only – database in the industry that can store multiple petabytes
of data in a single database; replicate it across many regions; and allow users
to transact over any subset of it. These are capabilities that can be met by no
other system I’m aware of on the market. Dynamo’s transactions are single-DC.
Google Cloud BigTable does not support transactions. Spanner, Aurora, CloudSQL,
and RDS have far lower scalability limits or require specialized hardware,
etc.This is an incredible opportunity for Apache Cassandra - to surpass the
scalability and transactional capability of some of the most advanced systems
in our industry - and to do so in open source, where anyone can download and
deploy the software to achieve this without cost; and for students and
researchers to learn from and build upon as well (a team from UT-Austin has
already reached out to this effect).As Benedict and Blake noted, the scope of
what’s captured in this proposal is also not terminal. While the first
implementation may extend today’s CAS semantics to multiple partitions with
lower latency, the foundation is suitable to build interactive transactions as
well — which would be remarkable and is something that I hadn’t considered
myself at the onset of this project.To that end, the CEP proposes the protocol,
offers a validated implementation, and the initial capability of extending
today’s single-partition transactions to multi-partition; while providing the
flexibility to build upon this work further.A simple example of what becomes
possible when this work lands and is integrated might be:–––
BEGIN BATCHUPDATE tbl1 SET value1 = newValue1 WHERE partitionKey = k1UPDATE
tbl2 SET value2 = newValue2 WHERE partitionKey = k2 AND conditionValue =
someConditionAPPLY BATCH
–––I understand that this query is present in the CEP and my intent isn’t to recommend that folks reread it if they’ve given a careful reading already.
But I do think it’s important to elaborate upon what becomes possible when this query can be issued.Users of Cassandra who have designed data models that
cram many types of data into a single partition for the purposes of atomicity no longer need to. They can design their applications with appropriate
schemas that wouldn’t leave Codd holding his nose. They’re no longer pushed into antipatterns that result in these partitions becoming huge and
potentially unreadable. Cassandra doesn’t become fully relational in this CEP - but it becomes possible and even easy to design applications that transact
across tables that mimic a large amount of relational functionality. And for users who are content to transact over a single table, they’ll find those
transactions become up to 4x faster today due to the protocol’s reduction in round-trips. The library’s loose coupling to Apache Cassandra and ability to
be incubated out-of-tree also enables other applications to take advantage of the protocol and is a nice step toward bringing modularity to the project.
There are a lot of good things happening here.I know I’m listed as an author - but figured I should go on record to say “I support this CEP.” :)Thanks,–
ScottOn Oct 6, 2021, at 8:05 AM, Jonathan Ellis <[email protected]> wrote:The problem that I keep pointing out is that you've created this CEP
forAccord without first getting consensus that the goals and the tradeoffs itmakes to achieve those goals (and that it will impose on future work
aroundtransactions) are the right ones for Cassandra long term.At this point I'm done repeating myself. For the convenience of anyonefollowing this
thread intermittently, I'll quote my first reply on thisthread to illustrate the kind of discussion I'd like to have.-----The whitepaper here is a good
description of the consensus algorithm itselfas well as its robustness and stability characteristics, and its comparisonwith other state-of-the-art
consensus algorithms is very useful. In thecontext of Cassandra, where a consensus algorithm is only part of what willbe implemented, I'd like to see a
more complete evaluation of thetransactional side of things as well, including performance characteristicsas well as the types of transactions that can be
supported and at least ageneral idea of what it would look like applied to Cassandra. This willallow the PMC to make a more informed decision about what
tradeoffs arebest for the entire long-term project of first supplementing and ultimatelyreplacing LWT.(Allowing users to mix LWT and AP Cassandra
operations against the samerows was probably a mistake, so in contrast with LWT we’re not looking forsomething fast enough for occasional use but rather
something within areasonable factor of AP operations, appropriate to being the only way tointeract with tables declared as such.)Besides Accord, this
should cover- Calvin and FaunaDB- A Spanner derivative (no opinion on whether that should be Cockroach orYugabyte, I don’t think it’s necessary to cover
both)- A 2PC implementation (the Accord paper mentions DynamoDB but I suspectthere is more public information about MongoDB)- RAMPHere’s an example of
what I mean:=Calvin=Approach: global consensus (Paxos in Calvin, Raft in FaunaDB) to ordertransactions, then replicas execute the transactions
independently with nofurther coordination. No SPOF. Transactions are batched by each sequencerto keep this from becoming a bottleneck.Performance:
Calvin paper (published 2012) reports linear scaling of TPC-CNew Order up to 500,000 transactions/s on 100 machines (EC2 XL machineswith 7GB ram and 8
virtual cores). Note that TPC-C New Order is composedof four reads and four writes, so this is effectively 2M reads and 2Mwrites as we normally measure
them in C*.Calvin supports mixed read/write transactions, but because the transactionexecution logic requires knowing all partition keys in advance to
ensurethat all replicas can reproduce the same results with no coordination,reads against non-PK predicates must be done ahead of time (transparently,by
the server) to determine the set of keys, and this must be retried ifthe set of rows affected is updated before the actual transaction executes.Batching
and global consensus adds latency -- 100ms in the Calvin paper andapparently about 50ms in FaunaDB. Glass half full: all transactions(including
multi-partition updates) are equally performant in Calvin sincethe coordination is handled up front in the sequencing step. Glass halfempty: even
single-row reads and writes have to pay the full coordinationcost. Fauna has optimized this away for reads but I am not aware of adescription of how they
changed the design to allow this.Functionality and limitations: since the entire transaction must be knownin advance to allow coordination-less execution
at the replicas, Calvincannot support interactive transactions at all. FaunaDB mitigates this byallowing server-side logic to be included, but a Calvin
approach will neverbe able to offer SQL compatibility.Guarantees: Calvin transactions are strictly serializable. There is noadditional complexity or
performance hit to generalizing to multipleregions, apart from the speed of light. And since Calvin is already payinga batching latency penalty, this is
less painful than for other systems.Application to Cassandra: B-. Distributed transactions are handled by thesequencing and scheduling layers, which are
leaderless, and Calvin’srequirements for the storage layer are easily met by C*. But Calvin alsorequires a global consensus protocol and LWT is almost
certainly notsufficiently performant, so this would require ZK or etcd (reasonable for alibrary approach but not for replacing LWT in C* itself), or
animplementation of Accord. I don’t believe Calvin would require additionaltable-level metadata in Cassandra.On Wed, Oct 6, 2021 at 9:53 AM
[email protected] <[email protected]>wrote:The problem with dropping a patch on Jira is that there is no opportunityto point out problems,
either with the fundamental approach or with thespecific implementation. So please point out some problems I can engagewith!From: Jonathan Ellis
<[email protected]>Date: Wednesday, 6 October 2021 at 15:48To: dev <[email protected]>Subject: Re: [DISCUSS] CEP-15: General Purpose
TransactionsOn Wed, Oct 6, 2021 at 9:21 AM [email protected] <[email protected]>wrote:> The goals of the CEP are stated clearly, and these
were the goals we had> going into the (multi-month) research project we undertook beforeproposing> this CEP. These goals are necessarily value
judgements, so we cannotexpect> that everyone will agree that they are optimal.>Right, so I'm saying that this is exactly the most important thing
to getconsensus on, and creating a CEP for a protocol to achieve goals that youhave not discussed with the community is the CEP equivalent of dropping
apatch on Jira without discussing its goals either.That's why our conversations haven't gone anywhere, because I keep saying"we need discuss the
goals and tradeoffs", and I'll give an example of whatI mean, and you keep addressing the examples (sometimes very shallowly, "itwould be
possible to X" or "Y could be done as an optimization") whileignoring the request to open a discussion around the big picture.-- Jonathan
Ellisco-founder, http://www.datastax.com@spyced
