Re: Large Kafka Streams deployment takes a long time to bootstrap

That's exactly what I did and it worked ok... Still slow but bootstraps within an acceptable amount of time. On Tue, May 9, 2017 at 9:05 PM Sachin Mittal wrote: > What we do is start and instance and wait till it get all the partitions, > then start second and so on. > Hope this works as a workar

Kafka - Move topics to different Partition in same hardware

Hello All, We are running Brokers in AWS. The brokers have drives of 15TB attached, We have outgrown the single partitions and are running out of space. I need some help on - How to add a new log dir - How to move topics to a new log dir - what will work well is moving partitions to

Re: Large Kafka Streams deployment takes a long time to bootstrap

What we do is start and instance and wait till it get all the partitions, then start second and so on. Hope this works as a workaround till they fix this in next release. On Wed, May 10, 2017 at 12:05 AM, João Peixoto wrote: > Guozhang thanks a lot for that info, that is exactly what I'm observ

Re: [VOTE] KIP-156 Add option "dry run" to Streams application reset tool

+1 On Tue, May 9, 2017 at 3:41 PM BigData dev wrote: > Hi, Everyone, > > Since this is a relatively simple change, I would like to start the voting > process for KIP-156: Add option "dry run" to Streams application reset tool > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=69

Re: Issue with kafka-server-stop on RedHat7

Hi Vedant, Just try to run kill -s TERM $KafkaProcessPID . On Wed, May 10, 2017 at 12:31 AM, Vedant Nighojkar wrote: > Hi Team, > > We are using Apache Kafka in one of our products. We support Windows, AIX > and Linux RedHat6 and above. > > I am seeing an issue with the kafka-server-stop.sh scr

Issue with kafka-server-stop on RedHat7

Hi Team, We are using Apache Kafka in one of our products. We support Windows, AIX and Linux RedHat6 and above. I am seeing an issue with the kafka-server-stop.sh script on RedHat7 machines. This used to work with RedHat6. ps ax | grep -i 'kafka.Kafka' - this is not able to find any running

Re: [VOTE] KIP-156 Add option "dry run" to Streams application reset tool

Eno, Got info from the JIRA all tools and their parameters are public API. So, I have started voting for this KIP. Thanks, Bharat On Tue, May 9, 2017 at 1:09 PM, Eno Thereska wrote: > +1 for me. I’m not sure we even need a KIP for this but it’s better to be > safe I guess. > > Eno > > > On May

Re: [VOTE] KIP-156 Add option "dry run" to Streams application reset tool

Eno, Got info from the JIRA all tools and their parameters are public API. So, I have started voting for this KIP. Thanks, Bharat On Tue, May 9, 2017 at 1:09 PM, Eno Thereska wrote: > +1 for me. I’m not sure we even need a KIP for this but it’s better to be > safe I guess. > > Eno > > > On May

Re: [VOTE] KIP-156 Add option "dry run" to Streams application reset tool

+1 for me. I’m not sure we even need a KIP for this but it’s better to be safe I guess. Eno > On May 9, 2017, at 8:41 PM, BigData dev wrote: > > Hi, Everyone, > > Since this is a relatively simple change, I would like to start the voting > process for KIP-156: Add option "dry run" to Streams

[VOTE] KIP-156 Add option "dry run" to Streams application reset tool

Hi, Everyone, Since this is a relatively simple change, I would like to start the voting process for KIP-156: Add option "dry run" to Streams application reset tool https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=69410150 The vote will run for a minimum of 72 hours. Thanks,

Re: Large Kafka Streams deployment takes a long time to bootstrap

Guozhang thanks a lot for that info, that is exactly what I'm observing it seems. I'll keep an eye out. JP On Mon, May 8, 2017 at 3:17 PM Guozhang Wang wrote: > Hello, > > Just to adds a few more pointers that there is a few improvements we have > added in trunk and are considering to also pig

Kafka Streams Usage Patterns

Hi, I started a new Wiki page to collect some common usage patterns for Kafka Streams. Right now, it contains a quick example on "how to compute average". Hope we can collect more example like this! https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Stream+Usage+Patterns -Matthias sign

Kafka Socket error

Hi, We got this below error in our production set-up. We then tried to read from kafka consumer shell, we got this error again 2-3 times, but after that it started working and now, we arent able to reproduce. Any indicator on why this error occurs and how it auto corrected.. Googling got me to som

Re: Kafka Stream stops polling new messages

I would be great if we could help to simplify debugging. Any ideas? -Matthias On 5/9/17 7:11 AM, João Peixoto wrote: > I'll leave it to your discretion, after realizing the problem it was an > easy work around, the bad experience was debugging and figuring out what > was going on. > > Thanks f

Re: Kafka Stream stops polling new messages

I'll leave it to your discretion, after realizing the problem it was an easy work around, the bad experience was debugging and figuring out what was going on. Thanks for the help once again JP On Tue, May 9, 2017 at 4:36 AM Eno Thereska wrote: > Yeah that's a good point, I'm not taking action t

Re: How to chain increasing window operations one after another

Just had a thought: If you implement the Windowed/Tuple serde to store the timestamp(s) before the actual record key then you can simply periodically do a ranged query on each of the state stores to find and delete all data older than ... (using punctuate() inside a Processor). Any downsides

Re: Kafka Stream stops polling new messages

Yeah that's a good point, I'm not taking action then. Eno On Mon, May 8, 2017 at 10:38 PM, Matthias J. Sax wrote: > Hey, > > I am not against opening a JIRA, but I am wondering what we should > describe/report there. If I understand the scenario correctly, João uses > a custom RocksDB store and

RE: producer and consumer sample code

NICE! Thanks! Adaryl "Bob" Wakefield, MBA Principal Mass Street Analytics, LLC 913.938.6685 www.massstreet.net www.linkedin.com/in/bobwakefieldmba Twitter: @BobLovesData -Original Message- From: BigData dev [mailto:bigdatadev...@gmail.com] Sent: Monday, May 8, 2017 7:40 PM To: users@kafk

Re: Deduplicating KStream-KStream join

Thanks Matthias for the detailed explanation. I'll go for the transform() workaround, let's see how it goes. Ofir. *Ofir Sharony* BackEnd Tech Lead Mobile: +972-54-7560277 | ofir.shar...@myheritage.com | www.myheritage.com MyHeritage Ltd., 3 Ariel Sharon St., Or Yehuda 60250, Israel

Re: How to chain increasing window operations one after another

Hi Matthias, Yes, the ever growing stores were my concern too. That was the intention behind my TODO note in the first reply just didn't want to touch on this until I've dug deeper into it. I understand compaction+retention policy on the backing changelog topics takes care of cleaning up on

Re: How to chain increasing window operations one after another

Hi Garrett, I'm glad this helped. You're absolutely right, only the "oneMinuteWindowed" KTable has a Windowed key - apologies again for getting it wrong the first time. I admit I used window().end() arbitrarily. If window().start() matches your semantics better, use that. Further on that not

Re: How to chain increasing window operations one after another

This seems to be a question that might affect many users, and it might we worth to document it somewhere as a recommended pattern. I was thinking the same thing :) How about a page on the wiki listing useful patterns with subpages for each patten in detail? (like for KIPs) Thanks, Michał O