Re: Question about bootstrap processing in KafkaStreams.

Upgrade :) On Tue, Jun 28, 2016 at 6:49 PM, Rohit Valsakumar wrote: > Hi Jay, > > Thanks for the reply. > > Unfortunately in our case due to legacy reasons we are using > WallclockTimestampExtractor in the application for all the streams and the > existing messages in the

Re: Question about bootstrap processing in KafkaStreams.

Hi Jay, Thanks for the reply. Unfortunately in our case due to legacy reasons we are using WallclockTimestampExtractor in the application for all the streams and the existing messages in the stream probably won¹t have timestamps as they are being produced by legacy clients. So the events are

Re: kafka + autoscaling groups fuckery

Charity, 1. Nothing you do seems crazy to me. Kafka should be able to work with auto-scaling and we should be able to fix the issues you are running into. There are few things you should be careful about when using the method you described though: 1.1 Your life may be a bit simpler if you have a

Re: Question about bootstrap processing in KafkaStreams.

I think you may get this for free as Kafka Streams attempts to align consumption across different topics/partitions by the timestamp in the messages. So in a case where you are starting a job fresh and it has a database changelog to consume and a event stream to consume, it will attempt to keep

Re: kafka + autoscaling groups fuckery

Reasons. Investigated it thoroughly, believe me. Some of the limitations that Kinesis uses to protect itself are non starters for us. Forgot to mention, we are using 0.9.0.1-0. On Tue, Jun 28, 2016 at 3:56 PM, Pradeep Gollakota wrote: > Just out of curiosity, if you

Re: kafka + autoscaling groups fuckery

Just out of curiosity, if you guys are in AWS for everything, why not use Kinesis? On Tue, Jun 28, 2016 at 3:49 PM, Charity Majors wrote: > Hi there, > > I just finished implementing kafka + autoscaling groups in a way that made > sense to me. I have a _lot_ of experience

Question about bootstrap processing in KafkaStreams.

Hi all, Is there a way to consume all the contents of a kafka topic into a KTable before doing a left join with another Kstream? I am looking at something that simulates a bootstrap topic in a Samza job. Thanks, Rohit Valsakumar This email and any attachments

Re: Producer Properties

Thank you, Dan! Chris From: "Dan Bahir (BLOOMBERG/ 120 PARK)" To: users@kafka.apache.org Date: 06/28/2016 09:49 AM Subject:Re: Producer Properties Hi Chris, The new producer returns a future so it works in an async manner hence no need for the

Re: Producer Properties

Hi Chris, The new producer returns a future so it works in an async manner hence no need for the producer.type property. 0.8 -> 0.10 batch.num.messages -> batch.size queue.buffering.max.ms -> linger.ms Your assumptions for the serializers look correct. Take a look at the documentation for

Re: Mirror maker setup - multi node

Hi there, I mean 1 cluster with 3 nodes. So I will need to run the mirror maker cluster on each of the 3 nodes in the cluster, in case of the loss of a node, the other 2 will continue to pull messages off the consumer cluster. It does seem to work correctly when I tested it. It just warns about

Re: Kafka Streams reducebykey and tumbling window - need final windowed KTable values

Hi Clive, As promised, here is the link to the KIP that just went out today. Feedback welcome: https://cwiki.apache.org/confluence/display/KAFKA/KIP-67%3A+Queryable+state+for+Kafka+Streams Thanks Eno

Re: Mirror maker setup - multi node

With 3 nodes, I assume you mean 3 clusters? If I understand correctly, say you have 3 clusters, A, B, and C, you simultaneously: - want to copy from A and B to C, to get an aggregation in C - want to copy fram A and C to B, to get a fail-back aggregation in B. Now what will happen when a message

Mirror maker setup - multi node

Hi All, So I understand I can run the following to aggregate topics from two different clusters into a mirror: bin/kafka-run-class.sh kafka.tools.MirrorMaker --consumer.config sourceCluster1Consumer.config --consumer.config sourceCluster2Consumer.config --num.streams 2 --producer.config