RE: Building API to make Kafka reactive

Hi Shekar, Alternatively, you could make each stage of your pipeline to write to a Cassandra (or other DB) and your API will read from it. With Cassandra TTL, the row will be deleted after TTL is passed. No manual cleanup is required. Best regards / Mit freundlichen Grüßen / Sincères sal

Building API to make Kafka reactive

I am looking at building a reactive api on top of Kafka. This API produces event to Kafka topic. I want to add a unique session id into the payload. The data gets transformed as it goes through different stages of a pipeline. I want to specify a final topic where I want the api to know that the pro

Re: Question about bootstrap processing in KafkaStreams.

Upgrade :) On Tue, Jun 28, 2016 at 6:49 PM, Rohit Valsakumar wrote: > Hi Jay, > > Thanks for the reply. > > Unfortunately in our case due to legacy reasons we are using > WallclockTimestampExtractor in the application for all the streams and the > existing messages in the stream probably won¹t ha

Re: Question about bootstrap processing in KafkaStreams.

Hi Jay, Thanks for the reply. Unfortunately in our case due to legacy reasons we are using WallclockTimestampExtractor in the application for all the streams and the existing messages in the stream probably won¹t have timestamps as they are being produced by legacy clients. So the events are bein

Re: kafka + autoscaling groups fuckery

Charity, 1. Nothing you do seems crazy to me. Kafka should be able to work with auto-scaling and we should be able to fix the issues you are running into. There are few things you should be careful about when using the method you described though: 1.1 Your life may be a bit simpler if you have a

Re: Question about bootstrap processing in KafkaStreams.

I think you may get this for free as Kafka Streams attempts to align consumption across different topics/partitions by the timestamp in the messages. So in a case where you are starting a job fresh and it has a database changelog to consume and a event stream to consume, it will attempt to keep the

Re: kafka + autoscaling groups fuckery

Reasons. Investigated it thoroughly, believe me. Some of the limitations that Kinesis uses to protect itself are non starters for us. Forgot to mention, we are using 0.9.0.1-0. On Tue, Jun 28, 2016 at 3:56 PM, Pradeep Gollakota wrote: > Just out of curiosity, if you guys are in AWS for ever

Re: kafka + autoscaling groups fuckery

Just out of curiosity, if you guys are in AWS for everything, why not use Kinesis? On Tue, Jun 28, 2016 at 3:49 PM, Charity Majors wrote: > Hi there, > > I just finished implementing kafka + autoscaling groups in a way that made > sense to me. I have a _lot_ of experience with ASGs and various

Question about bootstrap processing in KafkaStreams.

Hi all, Is there a way to consume all the contents of a kafka topic into a KTable before doing a left join with another Kstream? I am looking at something that simulates a bootstrap topic in a Samza job. Thanks, Rohit Valsakumar This email and any attachments

Re: Mirror maker setup - multi node

Then it's much simpler, just use replication-factor=3 for your topics, either by setting it when creating topics manually, or by setting default.replication.factor=3 in the broker settings. Kafka will then take care of the leadership on the partitions so both the load is balanced, and in case some

Re: Producer Properties

Thank you, Dan! Chris From: "Dan Bahir (BLOOMBERG/ 120 PARK)" To: users@kafka.apache.org Date: 06/28/2016 09:49 AM Subject:Re: Producer Properties Hi Chris, The new producer returns a future so it works in an async manner hence no need for the producer.type property. 0

Re: Producer Properties

Hi Chris, The new producer returns a future so it works in an async manner hence no need for the producer.type property. 0.8 -> 0.10 batch.num.messages -> batch.size queue.buffering.max.ms -> linger.ms Your assumptions for the serializers look correct. Take a look at the documentation for bot

Re: Mirror maker setup - multi node

Hi there, I mean 1 cluster with 3 nodes. So I will need to run the mirror maker cluster on each of the 3 nodes in the cluster, in case of the loss of a node, the other 2 will continue to pull messages off the consumer cluster. It does seem to work correctly when I tested it. It just warns about to

Re: Kafka Streams reducebykey and tumbling window - need final windowed KTable values

Hi Clive, As promised, here is the link to the KIP that just went out today. Feedback welcome: https://cwiki.apache.org/confluence/display/KAFKA/KIP-67%3A+Queryable+state+for+Kafka+Streams Thanks Eno

Re: Mirror maker setup - multi node

With 3 nodes, I assume you mean 3 clusters? If I understand correctly, say you have 3 clusters, A, B, and C, you simultaneously: - want to copy from A and B to C, to get an aggregation in C - want to copy fram A and C to B, to get a fail-back aggregation in B. Now what will happen when a message is

Mirror maker setup - multi node

Hi All, So I understand I can run the following to aggregate topics from two different clusters into a mirror: bin/kafka-run-class.sh kafka.tools.MirrorMaker --consumer.config sourceCluster1Consumer.config --consumer.config sourceCluster2Consumer.config --num.streams 2 --producer.config targetClu