Re: Kafka for event sourcing architecture

I have created the JIRA: https://issues.apache.org/jira/browse/KAFKA-3726 – Best regards,
 Radek Gruchalski 
ra...@gruchalski.com de.linkedin.com/in/radgruchalski Confidentiality: This communication is intended for the above-named person and may be confidential and/or legally privileged. If it

Re: Kafka for event sourcing architecture

Hi Tom, There is, indeed, the problem with replication in case of the leader change for the partition. Hence, I think the best, best approach would to have Kafka emit events in case of: - partition leader change - offset file to be cleaned up This still leaves a lot of work for the ops people

Re: Kafka for event sourcing architecture

The issue I have with that log.cleanup.policy approach is how it impacts with replication. Different replicas often have different sets of segment files. There are effectively two options then: 1. All replicas upload all their segments. This has issues with storage space, but is easy to implement.

Re: Kafka for event sourcing architecture

So Kafka is a fine solution as part of an event-sourced story. It's not a simple solution, but it fits. Kakfa can store data for a long time and you shouldn't discount this, however, using it as the primary long-term data store might not be a good fit if we're talking storing raw events for years

Re: Kafka for event sourcing architecture

Dave: Thanks for the suggestion, Bookkeeper looks interesting. I assume you meant "more high-level interface"? Chris: Thanks Chris, I actually came across a few of your talks and articles when researching this style of architecture (along with stuff by Greg Young and Martin Kleppmann) and

Re: Kafka for event sourcing architecture

A curious question: How does database + Kafka broker solution work? Would the consumer(s) load the database and then overlay the events? Or is it that the events are overlaid on the database and the database is queried? I am curious on how a complete solution would look like with "database +

Re: Kafka for event sourcing architecture

Oli, Kafka is only a partial solution. As I describe here ( http://www.slideshare.net/chris.e.richardson/hacksummit-2016-eventdriven-microservices-events-on-the-outside-on-the-inside-and-at-the-core/57) an event store is a hybrid of a database and a message broker. It is a database because it

Re: Kafka for event sourcing architecture

com> wrote: >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> +1 to your solution of log.cleanup.policy. Other brokers (ie, ActiveMQ) >>> have a feature like that. >>> Is there a JIRA

Re: Kafka for event sourcing architecture

, 2016 at 4:48 PM, Radoslaw Gruchalski > > wrote: > > > > > I have described a cold storage solution for Kafka: > > > > > > https://medium.com/@rad_g/the-case-for-kafka-cold-storage-32929d0a57b2#.kf0jf8cwv > > . > > > Also described it here a

Re: Kafka for event sourcing architecture

; > https://medium.com/@rad_g/the-case-for-kafka-cold-storage-32929d0a57b2#.kf0jf8cwv > . > > Also described it here a couple of times. Thd potential solution seems > > rather straightforward. > > Get Outlook for iOS > > > > _ > &g

Re: Kafka for event sourcing architecture

> rather straightforward. > > Get Outlook for iOS > > > > _ > > From: Luke Steensen > > Sent: Tuesday, May 17, 2016 11:22 pm > > Subject: Re: Kafka for event sourcing architecture > > To: > > > > > >

Re: Kafka for event sourcing architecture

rward. > Get Outlook for iOS > > _ > From: Luke Steensen > Sent: Tuesday, May 17, 2016 11:22 pm > Subject: Re: Kafka for event sourcing architecture > To: > > > It's harder in Kafka because the unit of replication is an entire > partition, not a single

Re: Kafka for event sourcing architecture

Steensen <luke.steen...@braintreepayments.com> Sent: Tuesday, May 17, 2016 11:22 pm Subject: Re: Kafka for event sourcing architecture To: <users@kafka.apache.org> It's harder in Kafka because the unit of replication is an entire partition, not a single key/value pair. Partitio

Re: Kafka for event sourcing architecture

om> > Sent: Tuesday, May 17, 2016 11:22 pm > Subject: Re: Kafka for event sourcing architecture > To: <users@kafka.apache.org> > > > It's harder in Kafka because the unit of replication is an entire > partition, not a single key/value pair. Partitions are large and

Re: Kafka for event sourcing architecture

It's harder in Kafka because the unit of replication is an entire partition, not a single key/value pair. Partitions are large and constantly growing, where key/value pairs are typically much smaller and don't change in size. There would theoretically be no difference if you had one partition per

Re: Kafka for event sourcing architecture

Hi Oli, Inline. On Tuesday, 17 May 2016, Olivier Lalonde wrote: > Hi all, > > I am considering adopting an "event sourcing" architecture for a system I > am developing and Kafka seems like a good choice of store for events. > > For those who aren't aware, this architecture

Kafka for event sourcing architecture

Hi all, I am considering adopting an "event sourcing" architecture for a system I am developing and Kafka seems like a good choice of store for events. For those who aren't aware, this architecture style consists in storing all state changes of the system as an ordered log of events and building