Hi! Thanks. any pointers on how to do that? On Sun, Nov 6, 2016 at 2:32 PM, Tauzell, Dave <dave.tauz...@surescripts.com> wrote:
> You should have one consumer pull the message and submit the data to each > storage using an XA transaction. > > > On Nov 5, 2016, at 19:49, kant kodali <kanth...@gmail.com> wrote: > > > > yes this problem can definetly be approached in many ways but given the > > hard constraints by our clients we don't seem to have many options. so > the > > problem is we have to keep two storages systems in sync all the time. so > > whatever the data that is storage 1 should also be in storage 2 at any > > given time. so we explored the following options > > > > 1) we thought about ETL from storage 1 to storage 2 but that approach > has > > bunch of drawbacks given the time constraints. > > 2) Add a common service on top of two storages and do some sort of 2PC > but > > that would degrade the write performance. Morever we dont really have a > > control over how fast each write/store can happen at each storage layer > > (because these two storages are completely different). > > > > so I started exploring if there is any tricks I could do with Kafka? > > > > > > > >> On Sat, Nov 5, 2016 at 5:01 PM, Hans Jespersen <h...@confluent.io> > wrote: > >> > >> Yes exactly. If consumer 1 gets message with offset 17 then it can write > >> that offset into an external storage that consumer 2 can also check to > >> ensure that it keeps in sync with consumer 1. > >> > >> Just curious though why you would need to do this? What is the use case > >> because there may be a better way to get you the functionality you want? > >> > >> -hans > >> > >> > >> > >> > >>> On Nov 5, 2016, at 4:31 PM, kant kodali <kanth...@gmail.com> wrote: > >>> > >>> I am new to Kafka and reading this statement "write consumer 1 and > >> consumer > >>> 2 to share a common external offset storage" I can interpret it many > ways > >>> but my best guess is as follows. > >>> > >>> Are you saying write the current offset of each consumer to a common > >>> external storage? > >>> > >>> > >>>> On Sat, Nov 5, 2016 at 4:15 PM, kant kodali <kanth...@gmail.com> > wrote: > >>>> > >>>> Hi Hans, > >>>> > >>>> What do you mean by "write consumer 1 and consumer 2 to share a common > >>>> external offset storage" ? can you please elaborate a bit more. > >>>> > >>>> Thanks! > >>>> > >>>> On Sat, Nov 5, 2016 at 4:00 PM, Hans Jespersen <h...@confluent.io> > >> wrote: > >>>> > >>>>> There is no built in mechanism to do this in Apache Kafka but if you > >> can > >>>>> write consumer 1 and consumer 2 to share a common external offset > >> storage > >>>>> then you may be able to build the functionality you seek. > >>>>> > >>>>> -hans > >>>>> > >>>>> > >>>>> > >>>>>> On Nov 5, 2016, at 3:55 PM, kant kodali <kanth...@gmail.com> wrote: > >>>>>> > >>>>>> Sorry there is a typo. here is a restatement. > >>>>>> > >>>>>> Is there a way to make sure two consumers receive the same message > >> from > >>>>> the > >>>>>> kafka broker in a atomic way? such that if consumer 1 gets a message > >>>>>> consumer 2 should also get that message and if consumer 1 fails for > >>>>>> whatever reason consumer 2 should also rollback to previous offset > (to > >>>>> the > >>>>>> same offset as consumer 1) or invalidate or something like that. is > >> that > >>>>>> possible? > >>>>> > >>>>> > >>>> > >> > >> > This e-mail and any files transmitted with it are confidential, may > contain sensitive information, and are intended solely for the use of the > individual or entity to whom they are addressed. If you have received this > e-mail in error, please notify the sender by reply e-mail immediately and > destroy all copies of the e-mail and any attachments. >
