Excuse me this part was non-sense: if the latest update to a document in es always win in Cassandra's LWW, they will "eventually" "converge". ________________________________ From: AmirHossein Roozbahany<mailto:band...@outlook.com> Sent: 11/8/2016 8:16 AM To: users@kafka.apache.org<mailto:users@kafka.apache.org> Subject: RE: is there a way to make sure two consumers receive the same message from the broker?
Generally Cassandra itself is not consistent enough, even with quorum read-writes, say one of the writes fail, the nodes who received the data won't roll back and it might lead to dirty reads which in turn makes roll back logic tricky (if not impossible). You can use linearizable writes but if you want to use them all the time, why bother using Cassandra!? The important thing about Cassandra is that all of the nodes will eventually have the same data(after anti-entropy or read-repair). they are "convergent", they will "eventually" converge, and practically they converge pretty fast. I think what you might actually need is to make two databases convergent(not necessarily fully consistent at any given time) if the latest update to a document in es leads always win when Cassandra is doing es they will "eventually" "converge". Doing so is easy, as fast as I know es assigns a _version number to a document and increases it on every update, now if your use of in Cassandra insert statement as the "writetime". Now when Cassandra is doing read repair the record with higher writetime will win. Using es's document _version field is just one option, you can use something from you domain or kafka's offset or machine timestamp (not recommended at all). I hope it could help ________________________________ From: kant kodali<mailto:kanth...@gmail.com> Sent: 11/7/2016 8:18 PM To: users@kafka.apache.org<mailto:users@kafka.apache.org> Subject: Re: is there a way to make sure two consumers receive the same message from the broker? Hi AmitHossein, I still don't see how that guarantees consistency at any given time. other words how do I know at time X the data in Cassandra and ES are the same. Thanks On Mon, Nov 7, 2016 at 3:26 AM, AmirHossein Roozbahany <diver...@outlook.com > wrote: > Hi > > Can you use elasticsearch _version field as cassandra's > writetime?(_version is strictly increasing, cassandra uses writetime for > applying LWW, so last write in elasticsearch will always win) > > It needs no transaction and makes databases convergent. > > > ________________________________ > From: kant kodali <kanth...@gmail.com> > Sent: Monday, November 7, 2016 3:08 AM > To: users@kafka.apache.org > Subject: Re: is there a way to make sure two consumers receive the same > message from the broker? > > Hi Hans, > > The two storages we use are Cassandra and Elastic search and they are on > the same datacenter for now. > The Programming Language we use is Java and OS would be Ubuntu or CentOS. > We get messages in JSON format so we insert into Elastic Search directly > and for Cassandra we transform JSON message into appropriate model so we > could insert into a Cassandra table. > The rate we currently get is about 100K/sec which is awesome but I am > pretty sure this will go down once when we implement 2PC or transactional > writes. > > Thanks, > kant >
