Re: [infinispan-dev] Infinispan and change data capture

2016-12-15 Thread Tristan Tarrant
On 09/12/16 18:25, Emmanuel Bernard wrote: > The total order would not be global but per key. > Each node has a Debezium connector instance embedded that listens to the > operations happening (primary and replicas alike). > All of this process is happening async compared to the operation. > Per

Re: [infinispan-dev] Infinispan and change data capture

2016-12-15 Thread Gustavo Fernandes
On Thu, Dec 15, 2016 at 5:47 PM, Emmanuel Bernard wrote: > > > On 15 Dec 2016, at 16:09, Sanne Grinovero wrote: > > > > Thanks Randall, > > those clarifications have been great. > > > > Emmanuel: some of your statements conflict with Randall's > >

Re: [infinispan-dev] Infinispan and change data capture

2016-12-15 Thread Emmanuel Bernard
> On 15 Dec 2016, at 16:09, Sanne Grinovero wrote: > > Thanks Randall, > those clarifications have been great. > > Emmanuel: some of your statements conflict with Randall's > clarifications and with the feasibility points I've been pointing at. > You say "collect *all*

Re: [infinispan-dev] Infinispan and change data capture

2016-12-15 Thread Emmanuel Bernard
> On 15 Dec 2016, at 15:59, Gustavo Fernandes wrote: > > > > On Thu, Dec 15, 2016 at 2:53 PM, Emmanuel Bernard > wrote: > >> On 15 Dec 2016, at 11:18, Gustavo Fernandes >

Re: [infinispan-dev] Infinispan and change data capture

2016-12-15 Thread Randall Hauch
> On Dec 15, 2016, at 9:09 AM, Sanne Grinovero wrote: > > Thanks Randall, > those clarifications have been great. > > Emmanuel: some of your statements conflict with Randall's > clarifications and with the feasibility points I've been pointing at. > You say "collect *all*

Re: [infinispan-dev] Infinispan and change data capture

2016-12-15 Thread Sanne Grinovero
Thanks Randall, those clarifications have been great. Emmanuel: some of your statements conflict with Randall's clarifications and with the feasibility points I've been pointing at. You say "collect *all* changes". I've been questioning that Infinispan can not keep *all* changes around for a

Re: [infinispan-dev] Infinispan and change data capture

2016-12-15 Thread Tristan Tarrant
On 15/12/16 15:59, Gustavo Fernandes wrote: > You mentioned that "The ability to read back history would then be > handled by the Debezium / Kafka tail, not infinispan itself", my question > was how the other consumers would get access to that history. At that point it is out of the source's

Re: [infinispan-dev] Infinispan and change data capture

2016-12-15 Thread Emmanuel Bernard
> On 15 Dec 2016, at 11:18, Gustavo Fernandes wrote: > > On Thu, Dec 15, 2016 at 9:54 AM, Emmanuel Bernard > wrote: > The goal is as followed: allow to collect all changes to push them to > Debezium and thus

Re: [infinispan-dev] Infinispan and change data capture

2016-12-15 Thread Gustavo Fernandes
On Thu, Dec 15, 2016 at 2:53 PM, Emmanuel Bernard wrote: > > On 15 Dec 2016, at 11:18, Gustavo Fernandes > wrote: > > On Thu, Dec 15, 2016 at 9:54 AM, Emmanuel Bernard > wrote: > >> The goal is as followed: allow to

Re: [infinispan-dev] Infinispan and change data capture

2016-12-15 Thread Emmanuel Bernard
Read my email from Dec 9th and the one from today and let me know which question is not sufficiently answered. I can try and clarify or refine. > On 15 Dec 2016, at 14:36, Tristan Tarrant wrote: > > On 12/12/16 16:13, Sanne Grinovero wrote: >> In short, what's the ultimate

Re: [infinispan-dev] Infinispan and change data capture

2016-12-15 Thread Tristan Tarrant
On 12/12/16 16:13, Sanne Grinovero wrote: > In short, what's the ultimate goal? I think that is the most important question, and we really need to make sure that the requirements are such that they fit with Infinispan's model, or they can achieved without fundamentally turning Infinispan into

Re: [infinispan-dev] Infinispan and change data capture

2016-12-15 Thread Gustavo Fernandes
On Thu, Dec 15, 2016 at 9:54 AM, Emmanuel Bernard wrote: > The goal is as followed: allow to collect all changes to push them to > Debezium and thus Kafka. > > This need does not require to remember all changes since the beginning of > time in Infinispan. Just enough to:

Re: [infinispan-dev] Infinispan and change data capture

2016-12-15 Thread Emmanuel Bernard
The goal is as followed: allow to collect all changes to push them to Debezium and thus Kafka. This need does not require to remember all changes since the beginning of time in Infinispan. Just enough to: - let Kafka catchup assuming it is the bottleneck - let us not lose a change in Kafka when