Re: Idle cluster high CPU usage

2017-09-25 Thread Elliot Crosby-McCullough
ng the running threads; If that doesn't elucidate the cause, > you could move onto sampling or profiling via JMX to see what's taking up > all that CPU. > > - Jordan Pilat > > On 2017-09-21 07:58, Elliot Crosby-McCullough <elliot.crosby-mccullough@ > freeagent.com> wro

Re: Idle cluster high CPU usage

2017-09-21 Thread Elliot Crosby-McCullough
Nothing, that value (that group of values) was at default when we started the debugging. On 21 September 2017 at 15:08, Ismael Juma <ism...@juma.me.uk> wrote: > Thanks. What happens if you reduce num.replica.fetchers? > > On Thu, Sep 21, 2017 at 3:02 PM, Elliot Cr

Re: Idle cluster high CPU usage

2017-09-21 Thread Elliot Crosby-McCullough
50 partition topic in an otherwise empty cluster. On 21 September 2017 at 14:20, Ismael Juma <ism...@juma.me.uk> wrote: > A couple of questions: how many partitions in the cluster and what are your > broker configs? > > On Thu, Sep 21, 2017 at 1:58 PM, Elliot Crosby-McCullough

Idle cluster high CPU usage

2017-09-21 Thread Elliot Crosby-McCullough
Hello, We've been trying to debug an issue with our kafka cluster for several days now and we're close to out of options. We have 3 kafka brokers associated with 3 zookeeper nodes and 3 registry nodes, plus a few streams clients and a ruby producer. Two of the three brokers are pinning a core

Re: Custom stream processor not triggering #punctuate()

2017-03-28 Thread Elliot Crosby-McCullough
If you need to schedule a call every N minutes of wall-clock time you'd > need to use your own scheduler. > > Does that help? > Michael > > > > On Tue, Mar 28, 2017 at 10:58 AM, Elliot Crosby-McCullough < > elliot.crosby-mccullo...@freeagent.com> wrote: > >

Custom stream processor not triggering #punctuate()

2017-03-28 Thread Elliot Crosby-McCullough
Hi there, I've written a simple processor which expects to have #process called on it for each message and configures regular punctuate calls via `context.schedule`. Regardless of what configuration I try for timestamp extraction I cannot get #punctuate to be called, despite #process being

Re: Table a KStream

2017-02-12 Thread Elliot Crosby-McCullough
> > Thanks > Eno > > > On 11 Feb 2017, at 17:56, Elliot Crosby-McCullough <elliot...@gmail.com> > wrote: > > > > For my own clarity, is there any actual distinction between > > `stream.to('topic')` > > where `topic` is set to compact and the

Re: Table a KStream

2017-02-11 Thread Elliot Crosby-McCullough
For my own clarity, is there any actual distinction between `stream.to('topic')` where `topic` is set to compact and the upcoming `stream.toTable('topic')` if you're not going to immediately use the table in this topology, i.e. if you want to use it as a table in some other processor application?

Re: Kafka Streams punctuate with slow-changing KTables

2017-02-02 Thread Elliot Crosby-McCullough
ted before you receive another message that is referring > to the same dimension. So you'd need to handle this somehow yourself. > > On Thu, 2 Feb 2017 at 08:26 Elliot Crosby-McCullough <elliot...@gmail.com> > wrote: > > > Sorry I left out too much context there. > > >

Re: Kafka Streams punctuate with slow-changing KTables

2017-02-02 Thread Elliot Crosby-McCullough
m into DBs like redshift. On 1 February 2017 at 22:40, Matthias J. Sax <matth...@confluent.io> wrote: > I am not sure if I can follow... what do you mean by "find or create" > semantics? > > What do you mean by "my first pass processor"? > > > -Matthi

Re: Kafka Streams punctuate with slow-changing KTables

2017-02-01 Thread Elliot Crosby-McCullough
es (but only only the "main topology") and thus the current > punctuate() issues should be resolved for this case. > > cf. > https://cwiki.apache.org/confluence/display/KAFKA/KIP- > 99%3A+Add+Global+Tables+to+Kafka+Streams > > > > -Matthias > > On

Re: Kafka Streams punctuate with slow-changing KTables

2017-02-01 Thread Elliot Crosby-McCullough
a single call after the > > KTable update. (Of course, only if all partitions advance time > accordingly.) > > > > > > Does this make sense? > > > > > > -Matthias > > > > On 2/1/17 7:37 AM, Elliot Crosby-McCullough wrote: > >> Hi the

Kafka Streams punctuate with slow-changing KTables

2017-02-01 Thread Elliot Crosby-McCullough
Hi there, I've been reading through the Kafka Streams documentation and there seems to be a tricky limitation that I'd like to make sure I've understood correctly. The docs[1] talk about the `punctuate` callback being based on stream time and that all incoming partitions of all incoming topics