Re: kafka streams rocksdb tuning (or memory leak?)

This indeed was the issue and cp2 did fix it! Thank you! Ara. > On Nov 16, 2016, at 4:11 PM, Damian Guy wrote: > > Hi Ara, > > Are you running Kafka Streams v0.10.1? If so, there is a resource leak in > the window store. See: https://github.com/apache/kafka/pull/2122 > You

Regarding Kafka server integration with Kerberos

Hi, I am using Kafka Server kafka_2.10-0.10.1.0. I am trying to integrate it with Kerberos server. I followed steps mentioned at http://kafka.apache.org/documentation.html#security_sasl However I am not able to start the server and getting the following error. [2016-11-16 09:47:46,578] INFO

Re: kafka streams rocksdb tuning (or memory leak?)

Hi Ara, Are you running Kafka Streams v0.10.1? If so, there is a resource leak in the window store. See: https://github.com/apache/kafka/pull/2122 You can either try checking out the Apache Kafka 0.10.1 branch (which has the fix) and building it manually, or you can try using the latest confluent

Re: --group flag for console consumer

Thank you, Vahid! On Wed, Nov 16, 2016 at 1:53 PM, Vahid S Hashemian wrote: > I'll open a JIRA. > > Andrew, let me know if you want to take over the implementation. > Otherwise, I'd be happy to work on it. > > Thanks. > --Vahid > > > > > From: Gwen Shapira

kafka streams rocksdb tuning (or memory leak?)

Hi, I have a few KTables in my application. Some of them have unlimited windows. If I leave the application to run for a few hours, I see the java process consume more and more memory, way above the -Xmx limit. I understand this is due to the rocksdb native lib used by kafka streams. What I

RE: Kafka 0.10 Monitoring tool

But most of the time I get this error: Error while executing consumer group command null org.apache.kafka.common.errors.DisconnectException Any help is appreciable? Thanks Achintya -Original Message- From: Vincent Dautremont [mailto:vincent.dautrem...@olamobile.com] Sent: Wednesday,

Re: --group flag for console consumer

I'll open a JIRA. Andrew, let me know if you want to take over the implementation. Otherwise, I'd be happy to work on it. Thanks. --Vahid From: Gwen Shapira To: Users Date: 11/16/2016 01:23 PM Subject:Re: --group flag for console

Re: --group flag for console consumer

Makes sense to me. Do you want to contribute a pull request? On Wed, Nov 16, 2016 at 11:33 AM, Andrew Pennebaker wrote: > Could the kafka-console-consumer shell script please get a --group > flag? > > Loading configs from properties files is helpful, but a direct

Re: Topic retension.ms

I would say "definitely" :) I fact I have seen people setting it to a couple of years. Guozhang On Wed, Nov 16, 2016 at 6:20 AM, Keshava Bharadwaj wrote: > Hello, > > One question. The maximum retention.ms per topic is a long which is 2^63-1 > => 9.223372e+18 =>

Re: Producer send message fail after upgrade broker from 0.8.0 to 0.10.0.0

Johnny, Seems like you are already following the suggested procedure for upgrade: https://kafka.apache.org/documentation#upgrade What versions of the producer are you using that had such issues? And are there any exception or error logs on the producer client? Guozhang On Tue, Nov 15, 2016

Re: Topics vs Partitions

Messages in partitions are logically the same. Topic is a logical group of partitions for messages that are of same type and/or from same source e.g. if you have two web-apps producing data, you will like to tell data from one source that from another. These are candidates for a topic each. If

--group flag for console consumer

Could the kafka-console-consumer shell script please get a --group flag? Loading configs from properties files is helpful, but a direct --group flag would be a simpler user interface for this common use case. -- Cheers, Andrew

Re: Kafka Streams internal topic naming

Srikanth, no, there's isn't any API to control the naming of internal topics. Is the reason you're asking for such functionality only/mostly about multi-tenancy issues (as you mentioned in your first message)? -Michael On Wed, Nov 16, 2016 at 8:20 PM, Srikanth wrote:

Re: What is causing Broker Unclean shutdown

Which version of Kafka are you using? I vaguely remember seeing similar issues some long time ago, due to a race condition between broker and controller, which I believe is fixed in the newer versions. Guozhang On Mon, Nov 14, 2016 at 11:43 AM, Zakee wrote: > Are these

Re: Pooling Kafka Java Consumers

This seems workable to me. But how about just using a single consumer and re-assigning topic-partitions on-the-fly then? Guozhang On Mon, Nov 14, 2016 at 12:17 AM, Enrico Olivelli wrote: > Hi, > I'm using Java Consumer API (latest version, 0.10.1.0), I store > consumed

Re: kafka 0.10 offset storage on zookeeper

I think Onur has once mentioned similar issues that currently we do not have a fully automated smooth upgrade path from old consumer (offsets stored in ZK) to new consumer (offsets stored in Kafka), maybe he can provide more context here and how it can be resolved. Currently the following

Kafka Streams internal topic naming

Hello, Does kafka stream provide an API to control how internal topics are named? Right now it uses appId, operator name, etc. In a shared kafka cluster its common to have naming convention that may require some prefix/suffix. Srikanth

RE: Topics vs Partitions

Partitions are used to distribute the messages in a topic between several different broker instances. This provides higher throughput. Partitions can also be replicate which allows for high availability. -Dave From: Doyle, Keith [mailto:keith.do...@greenwayhealth.com] Sent: Wednesday,

Topics vs Partitions

Trying to wrap my head around some of the concepts... For semantic partitioning, is there any rationale for choosing to divide a = single topic into multiple partitions as opposed to just creating multiple = topics each with a single partition? -- [Greenway_Health_PNG_large_NO_tag]

Topic retension.ms

Hello, One question. The maximum retention.ms per topic is a long which is 2^63-1 => 9.223372e+18 => 292471207 years. So essentially can we use this for infinite retention or in other words "never delete" ?

Re: getting an error about: java.lang.OutOfMemoryError: Direct buffer memory

java.lang.OutOfMemoryError is not really necessary directly related with memory usage. In your config, it requests only 1G. If your system is not stressed, I would suggest you to check ulimit for kafka runtime user, particularly check max number of open file descriptor and max number of

Kafka high cpu usage

Hi, Our production env uses Kafka 0.9.0.1 cluster of 12 m3.large nodes. Partitions count per broker is ~450, percent of leaders per broker is 30-40%. The average messages load is ~3K/s, bytes flow in is ~10MB/s and bytes flow out is ~60 MB/s. We observed strange behaviour while putting one

Re: Kafka 0.10 Monitoring tool

Just a note on that matter Sam : http://mail-archives.apache.org/mod_mbox/kafka-users/201611.mbox/%3CCAD2WViSAgwc9i4-9xEw1oz1xzpsbveFt1%3DSZ0qkHRiFEc3fXbw%40mail.gmail.com%3E On Tue, Nov 15, 2016 at 5:26 PM, Sam Pegler wrote: > If the consumer group is