[akka-user] Akka Kafka supervision hell

Apologies for the alarmist subject, but we're having a very difficult time getting supervision solid for a committablePartitionedSource. Latest issue is that someone used the Kafka REST Proxy to post an invalid message (we *are* using the schema registry). When running our consumer, that resulted

Re: [akka-user] Akka Kafka Stream - only once delivery

Thanks Michał. On Wednesday, June 28, 2017 at 4:15:30 PM UTC+5:30, Michal Borowiecki wrote: > > If you need exactly once semantics against your target database, the > common pattern is to store your last processed offset in that database > transactionally together with your output records,

Re: [akka-user] Akka Kafka Stream - only once delivery

If you need exactly once semantics against your target database, the common pattern is to store your last processed offset in that database transactionally together with your output records, instead of committing back to kafka. On startup you'd read the last offset from your database and seek

[akka-user] Akka Kafka Stream - only once delivery

Hi, I am using Akka Kafka Consumer.committablePartitionedSource to stream messages from kafka and group them based on group key with groupedWithin . Grouped records should be sink into database and then it should able to commit offset. The code skeleton is as following: val

[akka-user] akka kafka stream parallel processing

Hi, How does akka kafka stream (or how can i do) handle parallel processing based on topic partitions, if i run multiple threads within an instance or run multiple instances, how does it rebalance and/or failover if one thread dies or if i add/remove instances. Thanks Shannon --

Re: [akka-user] akka-kafka-stream "Offset Storage in Kafka" question/clarification

On Thu, Nov 24, 2016 at 4:51 AM, Arun wrote: > Thanks Patrik. > > So am I safe to use mapAsync greater than 1 (>1) , still my offset will be > stored correctly? > yes > > what will happen if my stream fail/crash which offset will be processed.. > it will start from

Re: [akka-user] akka-kafka-stream "Offset Storage in Kafka" question/clarification

Thanks Patrik. So am I safe to use mapAsync greater than 1 (>1) , still my offset will be stored correctly? what will happen if my stream fail/crash which offset will be processed.. Thanks, Arun On Saturday, August 27, 2016 at 12:26:30 PM UTC+5:30, Patrik Nordwall wrote: > > The commit

Re: [akka-user] akka-kafka-stream "Offset Storage in Kafka" question/clarification

The commit commands are emitted to the underlying KafkaConsumer in right order. It is waiting for the results of the commits that are done in parallel, and will possibly increase total throughput. We observed better throughput in performance tests. It would be great to clarify this in docs. You

[akka-user] akka-kafka-stream "Offset Storage in Kafka" question/clarification

Hi, In the "Offset Storage in Kafka” on http://doc.akka.io/docs/akka-stream-kafka/current/consumer.html#offset-storage-in-kafka there is the following example: .groupedWithin(10, 5.seconds) .map(group

Re: [akka-user] Akka/Kafka

Hi Richard, Kafka so far makes a conscious choice to base its operations on consistency, favoring it over availability where that tradeoff is necessary. Due to this design decision Akka Cluster or Distributed Data would not be a good fit—the result would be a different model than what Kafka

Re: [akka-user] Akka/Kafka

Thanks for the reply, Roland. I have a lot to learn. Exciting times. On Tue, Nov 17, 2015 at 7:43 AM, Roland Kuhn wrote: > Hi Richard, > > Kafka so far makes a conscious choice to base its operations on > consistency, favoring it over availability where that tradeoff is >

[akka-user] Akka/Kafka

Apologies if this is a naïve question. Kafka seems like great technology and a great companion to Akka. It has a dependency on Zookeeper, and my understanding is that's just for the watermarks. I'm not sure if this dependency on ZK is a bad thing, but I've certainly seen criticism of ZK and

[akka-user] akka-kafka consumer

Since there were a few kafka related questions here lately, figured I would factor this code out of some stuff I'm doing in case it's of use to others. https://github.com/sclasen/akka-kafka The consumer does explicit management of committing offsets, and allows actor based async processing of