Re: [infinispan-dev] Infinispan and change data capture

Ok, So proposal "Emmanuel Dec 9th" is hereby rechristened Deelog. I’ve captured it in the wiki https://github.com/infinispan/infinispan/wiki/Deelog:-direct-integration-with-Debezium Randall and I discussed

Re: [infinispan-dev] Infinispan and change data capture

On 16/12/16 16:30, Emmanuel Bernard wrote: > The Emmanuel Dec 9th proposal handles I think the case of topology changes > and nodes going down. We need a better name :) > For a cluster-wide shutdown with no time to flush queues, I think both > Artemis and the local debezium talking to

Re: [infinispan-dev] Infinispan and change data capture

> On 16 Dec 2016, at 16:25, Emmanuel Bernard wrote: > > >> On 16 Dec 2016, at 13:38, Tristan Tarrant wrote: >> >> On 16/12/16 13:12, Emmanuel Bernard wrote: >>> On 16 Dec 2016, at 09:48, Tristan Tarrant wrote:

Re: [infinispan-dev] Infinispan and change data capture

> On 16 Dec 2016, at 13:38, Tristan Tarrant wrote: > > On 16/12/16 13:12, Emmanuel Bernard wrote: >> >>> On 16 Dec 2016, at 09:48, Tristan Tarrant wrote: >>> >>> On 16/12/16 09:34, Emmanuel Bernard wrote: > Yes, the above design is what sprung to

Re: [infinispan-dev] Infinispan and change data capture

On 16 Dec 2016 12:39, "Tristan Tarrant" wrote: On 16/12/16 13:12, Emmanuel Bernard wrote: > >> On 16 Dec 2016, at 09:48, Tristan Tarrant wrote: >> >> On 16/12/16 09:34, Emmanuel Bernard wrote: Yes, the above design is what sprung to mind initially.

Re: [infinispan-dev] Infinispan and change data capture

On 16/12/16 13:12, Emmanuel Bernard wrote: > >> On 16 Dec 2016, at 09:48, Tristan Tarrant wrote: >> >> On 16/12/16 09:34, Emmanuel Bernard wrote: Yes, the above design is what sprung to mind initially. Not sure about the need of keeping the log in memory, as we

Re: [infinispan-dev] Infinispan and change data capture

> On 16 Dec 2016, at 09:48, Tristan Tarrant wrote: > > On 16/12/16 09:34, Emmanuel Bernard wrote: >>> Yes, the above design is what sprung to mind initially. Not sure about >>> the need of keeping the log in memory, as we would probably need some >>> form of persistent log

Re: [infinispan-dev] Infinispan and change data capture

On 16/12/16 09:34, Emmanuel Bernard wrote: >> Yes, the above design is what sprung to mind initially. Not sure about >> the need of keeping the log in memory, as we would probably need some >> form of persistent log for cache shutdown. Since this looks a lot like >> the append-log of the Artemis

Re: [infinispan-dev] Infinispan and change data capture

> On 16 Dec 2016, at 08:46, Tristan Tarrant wrote: > > On 09/12/16 18:25, Emmanuel Bernard wrote: >> The total order would not be global but per key. >> Each node has a Debezium connector instance embedded that listens to the >> operations happening (primary and replicas

Re: [infinispan-dev] Infinispan and change data capture

On 09/12/16 18:25, Emmanuel Bernard wrote: > The total order would not be global but per key. > Each node has a Debezium connector instance embedded that listens to the > operations happening (primary and replicas alike). > All of this process is happening async compared to the operation. > Per

Re: [infinispan-dev] Infinispan and change data capture

On Thu, Dec 15, 2016 at 5:47 PM, Emmanuel Bernard wrote: > > > On 15 Dec 2016, at 16:09, Sanne Grinovero wrote: > > > > Thanks Randall, > > those clarifications have been great. > > > > Emmanuel: some of your statements conflict with Randall's > >

Re: [infinispan-dev] Infinispan and change data capture

> On 15 Dec 2016, at 16:09, Sanne Grinovero wrote: > > Thanks Randall, > those clarifications have been great. > > Emmanuel: some of your statements conflict with Randall's > clarifications and with the feasibility points I've been pointing at. > You say "collect *all*

Re: [infinispan-dev] Infinispan and change data capture

> On 15 Dec 2016, at 15:59, Gustavo Fernandes wrote: > > > > On Thu, Dec 15, 2016 at 2:53 PM, Emmanuel Bernard > wrote: > >> On 15 Dec 2016, at 11:18, Gustavo Fernandes >

Re: [infinispan-dev] Infinispan and change data capture

> On Dec 15, 2016, at 9:09 AM, Sanne Grinovero wrote: > > Thanks Randall, > those clarifications have been great. > > Emmanuel: some of your statements conflict with Randall's > clarifications and with the feasibility points I've been pointing at. > You say "collect *all*

Re: [infinispan-dev] Infinispan and change data capture

Thanks Randall, those clarifications have been great. Emmanuel: some of your statements conflict with Randall's clarifications and with the feasibility points I've been pointing at. You say "collect *all* changes". I've been questioning that Infinispan can not keep *all* changes around for a

Re: [infinispan-dev] Infinispan and change data capture

On 15/12/16 15:59, Gustavo Fernandes wrote: > You mentioned that "The ability to read back history would then be > handled by the Debezium / Kafka tail, not infinispan itself", my question > was how the other consumers would get access to that history. At that point it is out of the source's

Re: [infinispan-dev] Infinispan and change data capture

> On 15 Dec 2016, at 11:18, Gustavo Fernandes wrote: > > On Thu, Dec 15, 2016 at 9:54 AM, Emmanuel Bernard > wrote: > The goal is as followed: allow to collect all changes to push them to > Debezium and thus

Re: [infinispan-dev] Infinispan and change data capture

On Thu, Dec 15, 2016 at 2:53 PM, Emmanuel Bernard wrote: > > On 15 Dec 2016, at 11:18, Gustavo Fernandes > wrote: > > On Thu, Dec 15, 2016 at 9:54 AM, Emmanuel Bernard > wrote: > >> The goal is as followed: allow to

Re: [infinispan-dev] Infinispan and change data capture

Read my email from Dec 9th and the one from today and let me know which question is not sufficiently answered. I can try and clarify or refine. > On 15 Dec 2016, at 14:36, Tristan Tarrant wrote: > > On 12/12/16 16:13, Sanne Grinovero wrote: >> In short, what's the ultimate

Re: [infinispan-dev] Infinispan and change data capture

On 12/12/16 16:13, Sanne Grinovero wrote: > In short, what's the ultimate goal? I think that is the most important question, and we really need to make sure that the requirements are such that they fit with Infinispan's model, or they can achieved without fundamentally turning Infinispan into

Re: [infinispan-dev] Infinispan and change data capture

On Thu, Dec 15, 2016 at 9:54 AM, Emmanuel Bernard wrote: > The goal is as followed: allow to collect all changes to push them to > Debezium and thus Kafka. > > This need does not require to remember all changes since the beginning of > time in Infinispan. Just enough to:

Re: [infinispan-dev] Infinispan and change data capture

The goal is as followed: allow to collect all changes to push them to Debezium and thus Kafka. This need does not require to remember all changes since the beginning of time in Infinispan. Just enough to: - let Kafka catchup assuming it is the bottleneck - let us not lose a change in Kafka when

Re: [infinispan-dev] Infinispan and change data capture

> On Dec 14, 2016, at 7:58 AM, Sanne Grinovero wrote: > > On 12 December 2016 at 17:56, Gustavo Fernandes > wrote: >> On Mon, Dec 12, 2016 at 3:13 PM, Sanne Grinovero >> wrote: >> >>> >>> In

Re: [infinispan-dev] Infinispan and change data capture

On 12 December 2016 at 17:56, Gustavo Fernandes wrote: > On Mon, Dec 12, 2016 at 3:13 PM, Sanne Grinovero > wrote: > >> >> In short, what's the ultimate goal? I see two main but different >> options intertwined: >> - allow to synchronize the *final

Re: [infinispan-dev] Infinispan and change data capture

-- Galder Zamarreño Infinispan, Red Hat > On 12 Dec 2016, at 14:58, Gustavo Fernandes wrote: > > > > On Fri, Dec 9, 2016 at 9:13 AM, Radim Vansa wrote: > But introducing globally monotonous counter means that > there will be a single contention

Re: [infinispan-dev] Infinispan and change data capture

On 12/12/2016 06:56 PM, Gustavo Fernandes wrote: > On Mon, Dec 12, 2016 at 3:13 PM, Sanne Grinovero > wrote: > > In short, what's the ultimate goal? I see two main but different > options intertwined: > - allow to synchronize the

Re: [infinispan-dev] Infinispan and change data capture

On Mon, Dec 12, 2016 at 3:13 PM, Sanne Grinovero wrote: > In short, what's the ultimate goal? I see two main but different > options intertwined: > - allow to synchronize the *final state* of a replica > I'm assuming this case is already in place when using remote

Re: [infinispan-dev] Infinispan and change data capture

On Fri, Dec 9, 2016 at 9:13 AM, Radim Vansa wrote: > But introducing globally monotonous counter means that > there will be a single contention point. > I wonder if the trade-off of Flake Ids [1] could be acceptable for this use case. [1]

Re: [infinispan-dev] Infinispan and change data capture

On Fri, Dec 9, 2016 at 9:13 AM, Radim Vansa wrote: > 1) 'cache that would persist the events with a monotonically increasing id' > > I assume that you mean globally (for all entries) monotonous. How will > you obtain such ID? Currently, commands have unique IDs that are >

Re: [infinispan-dev] Infinispan and change data capture

Could BiCache help here? It does establish total order per key (essentially established by the primary) and since multiple node (or all) have ordering information, modifications can always be rolled forward... On 09/12/16 22:43, Radim Vansa wrote: > On 12/09/2016 06:25 PM, Emmanuel Bernard

Re: [infinispan-dev] Infinispan and change data capture

On 12/09/2016 06:25 PM, Emmanuel Bernard wrote: > Randall and I had a chat on $subject. Here is a proposal worth > exploring as it is very lightweight on Infinispan's code. > > Does an operation has a unique id sahred by the master and replicas? > If not could we add that? Yes, each modification

Re: [infinispan-dev] Infinispan and change data capture

On 12/09/2016 05:08 PM, Randall Hauch wrote: > >> On Dec 9, 2016, at 3:13 AM, Radim Vansa > > wrote: >> >> On 12/08/2016 10:13 AM, Gustavo Fernandes wrote: >>> >>> I recently updated a proposal [1] based on several discussions we had >>> in the past

Re: [infinispan-dev] Infinispan and change data capture

> On Dec 9, 2016, at 10:08 AM, Randall Hauch wrote: > >> >> On Dec 9, 2016, at 3:13 AM, Radim Vansa > > wrote: >> >> On 12/08/2016 10:13 AM, Gustavo Fernandes wrote: >>> >>> I recently updated a proposal [1] based on several

Re: [infinispan-dev] Infinispan and change data capture

Randall and I had a chat on $subject. Here is a proposal worth exploring as it is very lightweight on Infinispan's code. Does an operation has a unique id sahred by the master and replicas? If not could we add that? The proposal itself: The total order would not be global but per key. Each node

Re: [infinispan-dev] Infinispan and change data capture

> On Dec 9, 2016, at 3:13 AM, Radim Vansa wrote: > > On 12/08/2016 10:13 AM, Gustavo Fernandes wrote: >> >> I recently updated a proposal [1] based on several discussions we had >> in the past that is essentially about introducing an event storage >> mechanism (write ahead

Re: [infinispan-dev] Infinispan and change data capture

On 12/08/2016 10:13 AM, Gustavo Fernandes wrote: > > I recently updated a proposal [1] based on several discussions we had > in the past that is essentially about introducing an event storage > mechanism (write ahead log) in order to improve reliability, failover > and "replayability" for the

Re: [infinispan-dev] Infinispan and change data capture

> On Dec 8, 2016, at 3:13 AM, Gustavo Fernandes wrote: > > On Wed, Dec 7, 2016 at 9:20 PM, Randall Hauch > wrote: > Reviving this old thread, and as before I appreciate any help the Infinispan > community might provide.

Re: [infinispan-dev] Infinispan and change data capture

On Wed, Dec 7, 2016 at 9:20 PM, Randall Hauch wrote: > Reviving this old thread, and as before I appreciate any help the > Infinispan community might provide. There definitely is interest in > Debezium capturing the changes being made to an Infinispan cluster. This > isn’t as

Re: [infinispan-dev] Infinispan and change data capture

Reviving this old thread, and as before I appreciate any help the Infinispan community might provide. There definitely is interest in Debezium capturing the changes being made to an Infinispan cluster. This isn’t as important when Infinispan is used as a cache, but when Infinispan is used as a

Re: [infinispan-dev] Infinispan and change data capture

-- Galder Zamarreño Infinispan, Red Hat > On 11 Jul 2016, at 16:41, Randall Hauch wrote: > >> >> On Jul 11, 2016, at 3:42 AM, Adrian Nistor wrote: >> >> Hi Randall, >> >> Infinispan supports both push and pull access models. The push model is >>

Re: [infinispan-dev] Infinispan and change data capture

> On Jul 11, 2016, at 3:42 AM, Adrian Nistor wrote: > > Hi Randall, > > Infinispan supports both push and pull access models. The push model is > supported by events (and listeners), which are cluster wide and are available > in both library and remote mode (hotrod). The

Re: [infinispan-dev] Infinispan and change data capture

Hi Randall, Infinispan supports both push and pull access models. The push model is supported by events (and listeners), which are cluster wide and are available in both library and remote mode (hotrod). The notification system is pretty advanced as there is a filtering mechanism available

[infinispan-dev] Infinispan and change data capture

The Debezium project [1] is working on building change data capture connectors for a variety of databases. MySQL is available now, MongoDB will be soon, and PostgreSQL and Oracle are next on our roadmap. One way in which Debezium and Infinispan can be used together is when Infinispan is being