:) On Tue, Nov 8, 2016 at 1:37 AM, AmirHossein Roozbahany <band...@outlook.com> wrote:
> Excuse me this part was non-sense: if the latest update to a document in > es always win in Cassandra's LWW, they will "eventually" "converge". > ________________________________ > From: AmirHossein Roozbahany<mailto:band...@outlook.com> > Sent: 11/8/2016 8:16 AM > To: users@kafka.apache.org<mailto:users@kafka.apache.org> > Subject: RE: is there a way to make sure two consumers receive the same > message from the broker? > > Generally Cassandra itself is not consistent enough, even with quorum > read-writes, say one of the writes fail, the nodes who received the data > won't roll back and it might lead to dirty reads which in turn makes roll > back logic tricky (if not impossible). You can use linearizable writes but > if you want to use them all the time, why bother using Cassandra!? > > The important thing about Cassandra is that all of the nodes will > eventually have the same data(after anti-entropy or read-repair). they are > "convergent", they will "eventually" converge, and practically they > converge pretty fast. > > I think what you might actually need is to make two databases > convergent(not necessarily fully consistent at any given time) if the > latest update to a document in es leads always win when Cassandra is doing > es they will "eventually" "converge". > > Doing so is easy, as fast as I know es assigns a _version number to a > document and increases it on every update, now if your use of in Cassandra > insert statement as the "writetime". Now when Cassandra is doing read > repair the record with higher writetime will win. > > Using es's document _version field is just one option, you can use > something from you domain or kafka's offset or machine timestamp (not > recommended at all). > > I hope it could help > ________________________________ > From: kant kodali<mailto:kanth...@gmail.com> > Sent: 11/7/2016 8:18 PM > To: users@kafka.apache.org<mailto:users@kafka.apache.org> > Subject: Re: is there a way to make sure two consumers receive the same > message from the broker? > > Hi AmitHossein, > > I still don't see how that guarantees consistency at any given time. other > words how do I know at time X the data in Cassandra and ES are the same. > > Thanks > > > On Mon, Nov 7, 2016 at 3:26 AM, AmirHossein Roozbahany < > diver...@outlook.com > > wrote: > > > Hi > > > > Can you use elasticsearch _version field as cassandra's > > writetime?(_version is strictly increasing, cassandra uses writetime for > > applying LWW, so last write in elasticsearch will always win) > > > > It needs no transaction and makes databases convergent. > > > > > > ________________________________ > > From: kant kodali <kanth...@gmail.com> > > Sent: Monday, November 7, 2016 3:08 AM > > To: users@kafka.apache.org > > Subject: Re: is there a way to make sure two consumers receive the same > > message from the broker? > > > > Hi Hans, > > > > The two storages we use are Cassandra and Elastic search and they are on > > the same datacenter for now. > > The Programming Language we use is Java and OS would be Ubuntu or CentOS. > > We get messages in JSON format so we insert into Elastic Search directly > > and for Cassandra we transform JSON message into appropriate model so we > > could insert into a Cassandra table. > > The rate we currently get is about 100K/sec which is awesome but I am > > pretty sure this will go down once when we implement 2PC or transactional > > writes. > > > > Thanks, > > kant > > >
