Re: Event sourcing with Kafka and Kafka Streams. How to deal with atomicity

approach we have taken is to implement event sourcing using a database and Kafka. For instance: see https://blog.eventuate.io/2016/10/06/eventuate-local-event-sourcing-and-cqrs-with-spring-boot-apache-kafka-and-mysql/ Chris -- Signature Michal Borowiecki <http://www.openbet.com/> *Sen

Re: Kafka compatibility matrix needed

data from kafka via druid/tranquility and they have clients of version 0.8.x implemented but my broker is running 0.10.x. Also if such a table can be posted on kafka documentation page or github page that would be great. Thanks Sachin -- Signature <http://www.openbet.com/>

Re: Kafka Streams Usage Patterns

://stackoverflow.com/a/44345374/7897191 Please let me know if you have a better design for this? Cheers, Michal On 27/05/17 21:16, Jay Kreps wrote: This is great! -Jay On Sat, May 27, 2017 at 12:47 PM, Michal Borowiecki <michal.borowie...@openbet.com <mailto:michal.borowie...@openb

Re: ticketing system Design

gned. I am not sure whether this can be solved using Kafka.Any help is appreciated Regards, Abhimanyu -- Signature <http://www.openbet.com/> Michal Borowiecki Senior Software Engineer L4 T: +44 208 742 1600 +44 203 249 8448

Re: Kafka Streams: Problems implementing rate limiting processing step

something? -Matthias On 6/20/17 1:02 AM, Michal Borowiecki wrote: I didn't know you could write to state stores from outside a processor/transformer. Interesting to hear that it is working although I'd be careful as KIP-67 warns it can introduce undefined behaviour: https://cwiki.apache.org

Re: [DISCUSS] KIP-163: Lower the Minimum Required ACL Permission of OffsetFetch

IP accordingly. --Vahid From: Michal Borowiecki <michal.borowie...@openbet.com> To:users@kafka.apache.org, Vahid S Hashemian < vahidhashem...@us.ibm.com>, d...@kafka.apache.org Date:06/13/2017 01:35 AM Subject:Re: [DISCUSS] KIP-163: Lower the Minimum Re

Re: Kafka Streams: Problems implementing rate limiting processing step

l else fails you could always consider producing to a topic rather than trying to forward directly, then you don't have to touch the relatively delicate Processor semantics. -- Signature <http://www.openbet.com/> Michal Borowiecki Senior Software Engineer L4 T:

Re: Kafka 0.11 transactions API question

s support it with only minor changes to the API (do not fail automatically dangling transactions on Producer restart)? Flushing is already there so that _should_ handle the pre-commit. Again, maybe I'm missing something and for sure I am not familiar with Kafka's internals.

Re: Kafka Streams vs Spark Streaming : reduce by window

. Perhaps this is in fact what you are proposing? -Jay On Fri, Jun 16, 2017 at 2:38 AM, Michal Borowiecki <michal.borowie...@openbet.com <mailto:michal.borowie...@openbet.com>> wrote: I wonder if it's a frequent enough use case that Kafka Streams should consider provid

Re: Kafka 0.11 transactions API question

data for such transactions. 2. making sure that committing already committed transactions doesn't brake anything Or maybe there is some other way to integrate Kafka into such two phase commit system that I'm missing? Thanks, Piotrek -- Signature <http://www.openbet.com/>

Re: Kafka Streams vs Spark Streaming : reduce by window

evel ? Thanks, Paolo Paolo Patierno Senior Software Engineer (IoT) @ Red Hat Microsoft MVP on Windows Embedded & IoT Microsoft Azure Advisor Twitter : @ppatierno<http://twitter.com/ppatierno> Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno> Blog : Dev

Re: [DISCUSS] KIP-163: Lower the Minimum Required ACL Permission of OffsetFetch

ACL+Permission+of+OffsetFetch The KIP is to address KAFKA-4585. Feedback and suggestions are welcome! Thanks. --Vahid -- Signature <http://www.openbet.com/> Michal Borowiecki Senior Software Engineer L4 T: +44 208 742 1600 +44 203 249 8448

Re: Re: synchronous request response using kafka

: Can someone please share some thoughts whether we can do synchronous call (request response) using kafka similar to JMS Thanks Sanjay 913-221-9164 -- Signature <http://www.openbet.com/> Michal Borowiecki Senior Software Engineer L4 T: +44 208 742 1600

Re: Zookeeper on same server as Kafka

clusters, I'd recommend to always have 5 ZK nodes, but for Kafka, you can scale well past that, or keep it small while you are starting out. Thanks, Tom Crayford Heroku Kafka On Sat, Jun 3, 2017 at 8:20 AM, Michal Borowiecki <michal.borowie...@openbet.com <mailto:michal.borowie...@openb

Re: Zookeeper on same server as Kafka

eper on the same server as Kafka vs having a separate zookeeper cluster ? Thanks, Meghana -- Signature <http://www.openbet.com/> Michal Borowiecki Senior Software Engineer L4 T: +44 208 742 1600 +44 203 249 8448

Re: Kafka Streams Usage Patterns

verage". Hope we can collect more example like this! https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Stream+Usage+Patterns -Matthias -- Signature <http://www.openbet.com/> Michal Borowiecki Senior Software Engineer L4 T: +44 208 742 1600

Re: Partition assignment with multiple topics

for each topic - and we're worried about how far this will scale. Thank you Mike G -- Signature <http://www.openbet.com/> Michal Borowiecki Senior Software Engineer L4 T: +44 208 742 1600 +44 203 249 8448 E: michal.b

Re: Order of punctuate() and process() in a stream processor

Hi Sini, This is beyond the score of KIP-138 but https://issues.apache.org/jira/browse/KAFKA-3514 exists to track such improvements Thanks, Michal On 17 May 2017 5:10 p.m., Peter Sinoros Szabo wrote: Hi, I see, now its clear why the repeated punctuations

Re: How to chain increasing window operations one after another

downsides to that? Cheers, Michał On 09/05/17 09:17, Michal Borowiecki wrote: Hi Matthias, Yes, the ever growing stores were my concern too. That was the intention behind my TODO note in the first reply just didn't want to touch on this until I've dug deeper into it. I understand compaction

Re: How to chain increasing window operations one after another

there yet. @Garret: did you try this out? This seems to be a question that might affect many users, and it might we worth to document it somewhere as a recommended pattern. -Matthias On 5/8/17 1:43 AM, Michal Borowiecki wrote: Apologies, In the code snippet of course only oneMinuteWindowed

Re: How to chain increasing window operations one after another

: did you try this out? This seems to be a question that might affect many users, and it might we worth to document it somewhere as a recommended pattern. -Matthias On 5/8/17 1:43 AM, Michal Borowiecki wrote: Apologies, In the code snippet of course only oneMinuteWindowed KTable will have

Re: How to chain increasing window operations one after another

many users, and it might we worth to document it somewhere as a recommended pattern. -Matthias On 5/8/17 1:43 AM, Michal Borowiecki wrote: Apologies, In the code snippet of course only oneMinuteWindowed KTable will have a Windowed key (KTable<Windowed, Value>), all others would be just

Re: How to chain increasing window operations one after another

Apologies, In the code snippet of course only oneMinuteWindowed KTable will have a Windowed key (KTable<Windowed, Value>), all others would be just KTable<Tuple2<Key, Long>, Value>. Michał On 07/05/17 16:09, Michal Borowiecki wrote: Hi Garrett, I've encountered a

Re: How to chain increasing window operations one after another

e docs which are awesome, I cannot determine if the KTable.groupby() would work over a window, and would reduce or aggregate thus do what I need? Any ideas? -- Signature <http://www.openbet.com/> Michal Borowiecki Senior Software Engineer L4 T: +44 208 7

Re: Does Kafka producer waits till previous batch returns responce before sending next one?

ay is introduced only by producer internal send loop time and linger? If a timeout would be localized only to a single batch send request for some reason, does it affect the next batch (assuming this batch can go through successfully)? Many thanks, Petr -- Signature <http://www.open

Re: Caching in Kafka Streams to ignore garbage message

will not return any result. -Matthias On 4/30/17 7:23 AM, Michal Borowiecki wrote: I have something working on the same principle (except not using connect), that is, I put ids to filter on into a ktable and then (inner) join a kstream with that ktable. I don't believe the value can be null though

Re: Issue in Kafka running for few days

2017-04-30 15:41 GMT+02:00 Michal Borowiecki <michal.borowie...@openbet.com <mailto:michal.borowie...@openbet.com>>: Hi Jan, Correct. As I said before it's not common or recommended practice to run an even number, and I wouldn't recommend it myself. I hope it di

Re: Controller connection failures

figure out the cause of the connect failures? 2. What's the controller anyway? 3. Are there some command-line diagnostic tools for inspecting the health of the system? Thanks for any help, Chuck -- Signature <http://www.openbet.com/> Michal Borowiecki Senior Software Engin

Re: Caching in Kafka Streams to ignore garbage message

rocess the messages. Should I just cache all ids on all nodes (which may be a large amount), or is there a way to only cache the id on the same kafka streams node which will receive data for that id? -- Signature <http://www.openbet.com/> Michal Borowiecki Senior Software Engineer

Re: Issue in Kafka running for few days

On 30/04/2017, Michal Borowiecki <michal.borowie...@openbet.com> wrote: Svante, I don't share your opinion. Having an even number of zookeepers is not a problem in itself, it simply means you don't get any better resilience than if you had one fewer instance. Yes, it's not common or recomme

Re: Issue in Kafka running for few days

e for [mytopicOLD,18]. Select 2 from ISR 2 to be the leader. (kafka.controller.OfflinePartitionLeaderSelector)* Typical broker config attached.. Please do share some valid inputs... Abhi !wq *-- * If you can't succeed, call it version 1.0 -- If you can't succeed, call it version 1.0 -- Signatur

Re: topics stuck in "Leader: -1" after crash while migrating topics

er seen a cluster fail this way...​ -- James Brown Engineer -- James Brown Engineer -- Signature <http://www.openbet.com/> Michal Borowiecki Senior Software Engineer L4 T: +44 208 742 1600 +44 203 249 8448 E: m

Re: [VOTE] 0.10.2.1 RC3

s://twitter.com/ConfluentInc> | blog <http://www.confluent.io/blog> -- Signature <http://www.openbet.com/> Michal Borowiecki Senior Software Engineer L4 T: +44 208 742 1600 +44 203 249 8448 E: michal.bo

Re: [VOTE] 0.10.2.1 RC0

wen Shapira -- Gwen Shapira Product Manager | Confluent 650.450.2760 | @gwenshap Follow us: Twitter | blog -- Signature <http://www.openbet.com/> Michal Borowiecki Senior Software Engineer L4 T: +44 208 742 1600 +44 203 249 8448