G1 tuning

[Cory Kolbeck]

Hi folks,

I'm a bit new to the operational side of G1, but pretty familiar with its
basic concept. We recently set up a Kafka cluster to support a new product,
and are seeing some suboptimal GC performance. We're using the parameters
suggested in the docs, except for having switched to java 1.8_40 in order
to get better memory debugging. Even though the cluster is handling only
2-3k messages per second per node, we see periodic 11-18 second
stop-the-world pauses on a roughly hourly cadence. I've turned on
additional GC logging, and see no humongous allocations, it all seems to be
buffers making it into the tenured gen. They appear to be collectable, as
the collection triggered by dumping the heap collects them all. Ideas for
additional diagnosis or tuning very welcome.

--Cory

Re: G1 tuning

[Todd Palino]

We've had no problems with G1 in all of our clusters with varying load
levels. I think we've seen an occasional long GC here and there, but
nothing recurring at this point.

What's the full command line that you're using with all the options?

-Todd


On Wed, Oct 14, 2015 at 2:18 PM, Scott Clasen  wrote:

> You can also use -Xmn with that gc to size the new gen such that those
> buffers don't get tenured
>
> I don't think that's an option with G1
>
> On Wednesday, October 14, 2015, Cory Kolbeck  wrote:
>
> > I'm not sure that will help here, you'll likely have the same
> > medium-lifetime buffers getting into the tenured generation and forcing
> > large collections.
> >
> > On Wed, Oct 14, 2015 at 10:00 AM, Gerrit Jansen van Vuuren <
> > gerrit...@gmail.com > wrote:
> >
> > > Hi,
> > >
> > > I've seen pauses using G1 in other applications and have found that
> > > -XX:+UseParallelGC
> > > -XX:+UseParallelOldGC  works best if you're having GC issues in general
> > on
> > > the JVM.
> > >
> > >
> > > Regards,
> > >  Gerrit
> > >
> > > On Wed, Oct 14, 2015 at 4:28 PM, Cory Kolbeck  > > wrote:
> > >
> > > > Hi folks,
> > > >
> > > > I'm a bit new to the operational side of G1, but pretty familiar with
> > its
> > > > basic concept. We recently set up a Kafka cluster to support a new
> > > product,
> > > > and are seeing some suboptimal GC performance. We're using the
> > parameters
> > > > suggested in the docs, except for having switched to java 1.8_40 in
> > order
> > > > to get better memory debugging. Even though the cluster is handling
> > only
> > > > 2-3k messages per second per node, we see periodic 11-18 second
> > > > stop-the-world pauses on a roughly hourly cadence. I've turned on
> > > > additional GC logging, and see no humongous allocations, it all seems
> > to
> > > be
> > > > buffers making it into the tenured gen. They appear to be
> collectable,
> > as
> > > > the collection triggered by dumping the heap collects them all. Ideas
> > for
> > > > additional diagnosis or tuning very welcome.
> > > >
> > > > --Cory
> > > >
> > >
> >
>

Facing Issue to create asyn producer in kafka 0.8.2

[prateek arora]

Hi

I want to create async producer so i can buffer messages in queue and send
after every 5 sec .

my kafka version is 0.8.2.0.

and i am using  kafka-clients 0.8.2.0 to create kafka producer in java.


below is my sample code :

package com.intel.labs.ive.cloud.testKafkaProducerJ;

import java.nio.charset.Charset;
import java.util.HashMap;

import java.util.Map;

import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.Producer;
import org.apache.kafka.clients.producer.ProducerConfig;
import org.apache.kafka.clients.producer.ProducerRecord;
import org.apache.kafka.common.Metric;
import org.apache.kafka.common.MetricName;
import org.apache.kafka.common.serialization.Serializer;
import org.apache.kafka.common.serialization.StringSerializer;
import org.apache.kafka.common.serialization.ByteArraySerializer;

import java.nio.file.DirectoryStream;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;

public class TestKafkaProducer {

Map props = new HashMap();
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, metadataBroker);
props.put("producer.type", "async");
props.put("queue.buffering.max.ms", "5000");

Serializer keySerializer = new StringSerializer();
Serializer valueSerializer = new ByteArraySerializer();

producer = new KafkaProducer(props, keySerializer,
valueSerializer);

ProducerRecord imageRecord;

while ( true ) {
imageRecord = new ProducerRecord(topicName,
recordKey,imageBytes);

producer.send(imageRecord);
}
}

size of my message is around 77K

but its work like a synchronous producer , send every message to broker  .
not buffering a message in to queue and send after 5 sec


please help to find out a solution.


Regards
Prateek

facing issue in async producer application

[prateek arora]

Hi

I want to create producer in async mode so i can send message in 5 sec
interval.

below is my code :


Map props = new HashMap();
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG,
metadataBroker);
//  props.put(ProducerConfig.ACKS_CONFIG, "0"); // don't wait
for ACKs
props.put("producer.type", "async"); // don't wait for ACKs
props.put("queue.buffering.max.ms", "5000"); // don't wait
for ACKs

Serializer keySerializer = new StringSerializer();
Serializer valueSerializer = new
ByteArraySerializer();
producer = new KafkaProducer(props,
keySerializer, valueSerializer);
 ProducerRecord imageRecord;
imageRecord = new ProducerRecord(topicName,
recordKey,imageBytes);
producer.send(imageRecord);

Re: G1 tuning

[Cory Kolbeck]

My current theory, which I haven't dug into the source to confirm, is that
said buffers are being pre-allocated. Because the kafka instance is
relatively bored, they end up living long enough to see a few collections
and be promoted. I could be way off base though.

Command line, broken out for a little better readability:
18918 /opt/java/1.8.0_40/bin/java -cp
:/mnt/services/kafka08/etc::/mnt/services/kafka08/current/lib/java:/mnt/services/kafka08/current/lib/java/zookeeper-3.4.6.jar:/mnt/services/kafka08/current/lib/java/scala-library-2.11.5.jar:/mnt/services/kafka08/current/lib/java/lz4-1.2.0.jar:/mnt/services/kafka08/current/lib/java/scala-parser-combinators_2.11-1.0.2.jar:/mnt/services/kafka08/current/lib/java/metrics-core-2.2.0.jar:/mnt/services/kafka08/current/lib/java/kafka_2.11-0.8.2.0-scaladoc.jar:/mnt/services/kafka08/current/lib/java/kafka_2.11-0.8.2.0-javadoc.jar:/mnt/services/kafka08/current/lib/java/scala-xml_2.11-1.0.2.jar:/mnt/services/kafka08/current/lib/java/kafka_2.11-0.8.2.0.jar:/mnt/services/kafka08/current/lib/java/kafka-clients-0.8.2.0.jar:/mnt/services/kafka08/current/lib/java/snappy-java-1.1.1.6.jar:/mnt/services/kafka08/current/lib/java/kafka_2.11-0.8.2.0-test.jar:/mnt/services/kafka08/current/lib/java/log4j-1.2.16.jar:/mnt/services/kafka08/current/lib/java/kafka_2.11-0.8.2.0-sources.jar:/mnt/services/kafka08/current/lib/java/slf4j-api-1.7.6.jar:/mnt/services/kafka08/current/lib/java/slf4j-log4j12-1.6.1.jar:/mnt/services/kafka08/current/lib/java/zkclient-0.3.jar:/mnt/services/kafka08/current/lib/java/jopt-simple-3.2.jar:
 -Xms4096M -Xmx4096M
-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=/mnt/services/kafka08/var/dump/2015-10-13-17_34_17.hprof
-XX:+PrintAdaptiveSizePolicy -XX:+UseG1GC -XX:MaxGCPauseMillis=20
-XX:InitiatingHeapOccupancyPercent=35
-Xloggc:/mnt/services/kafka08/var/log/kafka08-gc-2015-10-13-17_34_17.log
-XX:+PrintGCDetails -XX:+PrintGCDateStamps
-Dlog4j.configuration=file:///mnt/services/kafka08/etc/log4j.properties
kafka.Kafka /mnt/services/kafka08/etc/kafka08.properties


On Wed, Oct 14, 2015 at 11:37 AM, Todd Palino  wrote:

> We've had no problems with G1 in all of our clusters with varying load
> levels. I think we've seen an occasional long GC here and there, but
> nothing recurring at this point.
>
> What's the full command line that you're using with all the options?
>
> -Todd
>
>
> On Wed, Oct 14, 2015 at 2:18 PM, Scott Clasen  wrote:
>
> > You can also use -Xmn with that gc to size the new gen such that those
> > buffers don't get tenured
> >
> > I don't think that's an option with G1
> >
> > On Wednesday, October 14, 2015, Cory Kolbeck  wrote:
> >
> > > I'm not sure that will help here, you'll likely have the same
> > > medium-lifetime buffers getting into the tenured generation and forcing
> > > large collections.
> > >
> > > On Wed, Oct 14, 2015 at 10:00 AM, Gerrit Jansen van Vuuren <
> > > gerrit...@gmail.com > wrote:
> > >
> > > > Hi,
> > > >
> > > > I've seen pauses using G1 in other applications and have found that
> > > > -XX:+UseParallelGC
> > > > -XX:+UseParallelOldGC  works best if you're having GC issues in
> general
> > > on
> > > > the JVM.
> > > >
> > > >
> > > > Regards,
> > > >  Gerrit
> > > >
> > > > On Wed, Oct 14, 2015 at 4:28 PM, Cory Kolbeck  > > > wrote:
> > > >
> > > > > Hi folks,
> > > > >
> > > > > I'm a bit new to the operational side of G1, but pretty familiar
> with
> > > its
> > > > > basic concept. We recently set up a Kafka cluster to support a new
> > > > product,
> > > > > and are seeing some suboptimal GC performance. We're using the
> > > parameters
> > > > > suggested in the docs, except for having switched to java 1.8_40 in
> > > order
> > > > > to get better memory debugging. Even though the cluster is handling
> > > only
> > > > > 2-3k messages per second per node, we see periodic 11-18 second
> > > > > stop-the-world pauses on a roughly hourly cadence. I've turned on
> > > > > additional GC logging, and see no humongous allocations, it all
> seems
> > > to
> > > > be
> > > > > buffers making it into the tenured gen. They appear to be
> > collectable,
> > > as
> > > > > the collection triggered by dumping the heap collects them all.
> Ideas
> > > for
> > > > > additional diagnosis or tuning very welcome.
> > > > >
> > > > > --Cory
> > > > >
> > > >
> > >
> >
>

Re: Facing Issue to create asyn producer in kafka 0.8.2

[Grant Henke]

Looks like you may be mixing the new producer with old producer configs.
See the new config documentation here:
http://kafka.apache.org/documentation.html#newproducerconfigs. You will
likely want to set the "batch.size" and "linger.ms" to achieve your goal.

Thanks,
Grant

On Wed, Oct 14, 2015 at 1:29 PM, prateek arora 
wrote:

> Hi
>
> Thanks for help .
>
> but same behavior even after changing batch.size
>
> I have changes  batch.size value to 33554432.
>  props.put("batch.size","33554432");
>
>
>
> On Wed, Oct 14, 2015 at 11:09 AM, Zakee  wrote:
>
> > Hi Prateek,
> >
> > Looks like you are using default batch.size which is ~16K and it forces
> > the send of messages immediately as your single message is larger than
> > that. Try using larger batch.size.
> >
> > Thanks
> > Zakee
> >
> >
> >
> > > On Oct 14, 2015, at 10:29 AM, prateek arora <
> prateek.arora...@gmail.com>
> > wrote:
> > >
> > > Hi
> > >
> > > I want to create async producer so i can buffer messages in queue and
> > send
> > > after every 5 sec .
> > >
> > > my kafka version is 0.8.2.0.
> > >
> > > and i am using  kafka-clients 0.8.2.0 to create kafka producer in java.
> > >
> > >
> > > below is my sample code :
> > >
> > > package com.intel.labs.ive.cloud.testKafkaProducerJ;
> > >
> > > import java.nio.charset.Charset;
> > > import java.util.HashMap;
> > >
> > > import java.util.Map;
> > >
> > > import org.apache.kafka.clients.producer.KafkaProducer;
> > > import org.apache.kafka.clients.producer.Producer;
> > > import org.apache.kafka.clients.producer.ProducerConfig;
> > > import org.apache.kafka.clients.producer.ProducerRecord;
> > > import org.apache.kafka.common.Metric;
> > > import org.apache.kafka.common.MetricName;
> > > import org.apache.kafka.common.serialization.Serializer;
> > > import org.apache.kafka.common.serialization.StringSerializer;
> > > import org.apache.kafka.common.serialization.ByteArraySerializer;
> > >
> > > import java.nio.file.DirectoryStream;
> > > import java.nio.file.Files;
> > > import java.nio.file.Path;
> > > import java.nio.file.Paths;
> > >
> > > public class TestKafkaProducer {
> > >
> > > Map props = new HashMap();
> > >props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG,
> > metadataBroker);
> > >props.put("producer.type", "async");
> > >props.put("queue.buffering.max.ms", "5000");
> > >
> > > Serializer keySerializer = new StringSerializer();
> > >Serializer valueSerializer = new ByteArraySerializer();
> > >
> > >producer = new KafkaProducer(props,
> keySerializer,
> > > valueSerializer);
> > >
> > > ProducerRecord imageRecord;
> > >
> > > while ( true ) {
> > > imageRecord = new ProducerRecord(topicName,
> > > recordKey,imageBytes);
> > >
> > >producer.send(imageRecord);
> > > }
> > > }
> > >
> > > size of my message is around 77K
> > >
> > > but its work like a synchronous producer , send every message to broker
> > .
> > > not buffering a message in to queue and send after 5 sec
> > >
> > >
> > > please help to find out a solution.
> > >
> > >
> > > Regards
> > > Prateek
> >
> > 
> > A Balance Transfer Card With An Outrageously Long Intro Rate And No
> > Balance Transfer Fees That Can Save You Thousands
> > http://thirdpartyoffers.netzero.net/TGL3231/561e9a75a77071a74763fst04vuc
>



-- 
Grant Henke
Software Engineer | Cloudera
gr...@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke

Re: Facing Issue to create asyn producer in kafka 0.8.2

[prateek arora]

Hi

Thanks for help .

but same behavior even after changing batch.size

I have changes  batch.size value to 33554432.
 props.put("batch.size","33554432");



On Wed, Oct 14, 2015 at 11:09 AM, Zakee  wrote:

> Hi Prateek,
>
> Looks like you are using default batch.size which is ~16K and it forces
> the send of messages immediately as your single message is larger than
> that. Try using larger batch.size.
>
> Thanks
> Zakee
>
>
>
> > On Oct 14, 2015, at 10:29 AM, prateek arora 
> wrote:
> >
> > Hi
> >
> > I want to create async producer so i can buffer messages in queue and
> send
> > after every 5 sec .
> >
> > my kafka version is 0.8.2.0.
> >
> > and i am using  kafka-clients 0.8.2.0 to create kafka producer in java.
> >
> >
> > below is my sample code :
> >
> > package com.intel.labs.ive.cloud.testKafkaProducerJ;
> >
> > import java.nio.charset.Charset;
> > import java.util.HashMap;
> >
> > import java.util.Map;
> >
> > import org.apache.kafka.clients.producer.KafkaProducer;
> > import org.apache.kafka.clients.producer.Producer;
> > import org.apache.kafka.clients.producer.ProducerConfig;
> > import org.apache.kafka.clients.producer.ProducerRecord;
> > import org.apache.kafka.common.Metric;
> > import org.apache.kafka.common.MetricName;
> > import org.apache.kafka.common.serialization.Serializer;
> > import org.apache.kafka.common.serialization.StringSerializer;
> > import org.apache.kafka.common.serialization.ByteArraySerializer;
> >
> > import java.nio.file.DirectoryStream;
> > import java.nio.file.Files;
> > import java.nio.file.Path;
> > import java.nio.file.Paths;
> >
> > public class TestKafkaProducer {
> >
> > Map props = new HashMap();
> >props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG,
> metadataBroker);
> >props.put("producer.type", "async");
> >props.put("queue.buffering.max.ms", "5000");
> >
> > Serializer keySerializer = new StringSerializer();
> >Serializer valueSerializer = new ByteArraySerializer();
> >
> >producer = new KafkaProducer(props, keySerializer,
> > valueSerializer);
> >
> > ProducerRecord imageRecord;
> >
> > while ( true ) {
> > imageRecord = new ProducerRecord(topicName,
> > recordKey,imageBytes);
> >
> >producer.send(imageRecord);
> > }
> > }
> >
> > size of my message is around 77K
> >
> > but its work like a synchronous producer , send every message to broker
> .
> > not buffering a message in to queue and send after 5 sec
> >
> >
> > please help to find out a solution.
> >
> >
> > Regards
> > Prateek
>
> 
> A Balance Transfer Card With An Outrageously Long Intro Rate And No
> Balance Transfer Fees That Can Save You Thousands
> http://thirdpartyoffers.netzero.net/TGL3231/561e9a75a77071a74763fst04vuc

Re: G1 tuning

[Gerrit Jansen van Vuuren]

Hi,

I've seen pauses using G1 in other applications and have found that
-XX:+UseParallelGC
-XX:+UseParallelOldGC  works best if you're having GC issues in general on
the JVM.


Regards,
 Gerrit

On Wed, Oct 14, 2015 at 4:28 PM, Cory Kolbeck  wrote:

> Hi folks,
>
> I'm a bit new to the operational side of G1, but pretty familiar with its
> basic concept. We recently set up a Kafka cluster to support a new product,
> and are seeing some suboptimal GC performance. We're using the parameters
> suggested in the docs, except for having switched to java 1.8_40 in order
> to get better memory debugging. Even though the cluster is handling only
> 2-3k messages per second per node, we see periodic 11-18 second
> stop-the-world pauses on a roughly hourly cadence. I've turned on
> additional GC logging, and see no humongous allocations, it all seems to be
> buffers making it into the tenured gen. They appear to be collectable, as
> the collection triggered by dumping the heap collects them all. Ideas for
> additional diagnosis or tuning very welcome.
>
> --Cory
>

RE: Questions on the new Kafka consumer

[prajod.vettiyattil]

Hi,

According to the wiki: 
https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Client+Re-Design, 
allowing manual partition and topic access is a design goal.

Also the new API has functions to seek and subscribe at the partition level: 
http://people.apache.org/~nehanarkhede/kafka-0.9-consumer-javadoc/doc/

This for the KafkaConsumer from kafka version 0.9.

Prajod

-Original Message-
From: Rajiv Kurian [mailto:ra...@signalfx.com]
Sent: 14 October 2015 01:44
To: users@kafka.apache.org
Subject: Questions on the new Kafka consumer

I was reading the documentation for the new Kafka consumer API at 
https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java
and came across this:

"Each Kafka consumer must specify a consumer group that it belongs to."

Currently we use Kafka but WITHOUT the consumer group functionality since I do 
the partition distribution based on my own app logic. I also persist my own 
offsets in an external store.
I also sometimes need to consume from the beginning of a partition or just some 
arbitrary offset. Currently I use the SimpleConsumer to do this. I just give it 
the topics and partitions I want to consume from and I maintain the partition 
-> broker mapping myself. I also maintain the offset myself and when I need to 
consume from the beginning of a partition I ask the broker for the earliest 
offset and start consuming from there.

I am glad that the new consumer does the broker -> partition mapping itself and 
I can get rid of a lot of code if I transitioned to it. But given that the 
documentation suggests that I have to use the Consumer group, it doesn't seem 
that it will support the lower level features that I need. I do see the 
commitSync method which suggests that I am in control of the offset management 
and for my use case I could just never call it. Is there also a way for me to 
specify which partition to consume from exactly (in addition to the topic) and 
also from which offset?

Thanks,
Rajiv
The information contained in this electronic message and any attachments to 
this message are intended for the exclusive use of the addressee(s) and may 
contain proprietary, confidential or privileged information. If you are not the 
intended recipient, you should not disseminate, distribute or copy this e-mail. 
Please notify the sender immediately and destroy all copies of this message and 
any attachments. WARNING: Computer viruses can be transmitted via email. The 
recipient should check this email and any attachments for the presence of 
viruses. The company accepts no liability for any damage caused by any virus 
transmitted by this email. www.wipro.com

Some help to understand the Kafka exposed MBeans

[pubudu gunawardena]

Hi All,

I am in the process of performance testing an application that
produces messages to Kafka. I am testing the application with
increasing loads.
I would like to get some statistics from Kafka so that I know that
more Kafka nodes should be added to handle a specific load. I have
connected to Kafka JMX console, but having some trouble interpreting
the values shown there.
For example, should I look at the
kafka.network.RequestChannel.RequestQueueSize/ResponseQueueSize or
ReqeuestMetrics.LocalTimeMs.Produce etc..
Can someone please let me know which statistics to look at in order to
achieve the above and how to interpret them? Or if there is some
document that I should refer to in order to understand those values?

-- 
Thanks,
Pubudu

Strange ZK Error precedes frequent rebalances

[noah]

A number of our developers are seeing errors like the one below in their
console when running a consumer on their laptop. The error is always
followed by logging indicating that the local consumer is rebalancing, and
in the meantime we are not making much progress.

I'm reading this as the consumer trying to read a ZK node for another
consumer in the same group (running on a different machine,) but the node
is no longer there. I can't tell if that is triggering a rebalance, or if
it's just coincident.

In our dev environment, we have a lot (hundreds) of consumers coming and
going from the same consumer group, but they are mostly subscribed to
different topics. Is this setup (sharing a consumer group across topics)
potentially causing more rebalances than we would otherwise need? Or is
something else entirely going on?

LOG:

INFO  [2015-10-14 20:32:49,138] kafka.consumer.ZookeeperConsumerConnector:
[real-time-updates_Noahs-MacBook-Pro.local-1444853969114-7b52ecb5],
exception during rebalance
! org.I0Itec.zkclient.exception.ZkNoNodeException:
org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode =
NoNode for
/consumers/real-time-updates/ids/real-time-updates_infra-buildagent-06-1444854764478-4dd4d6af
! at org.I0Itec.zkclient.exception.ZkException.create(ZkException.java:47)
~[zkclient-0.3.jar:0.3]
! at org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:685)
~[zkclient-0.3.jar:0.3]
! at org.I0Itec.zkclient.ZkClient.readData(ZkClient.java:766)
~[zkclient-0.3.jar:0.3]
! at org.I0Itec.zkclient.ZkClient.readData(ZkClient.java:761)
~[zkclient-0.3.jar:0.3]
! at kafka.utils.ZkUtils$.readData(ZkUtils.scala:443)
~[kafka_2.10-0.8.2.1.jar:na]
! at kafka.consumer.TopicCount$.constructTopicCount(TopicCount.scala:61)
~[kafka_2.10-0.8.2.1.jar:na]
! at
kafka.utils.ZkUtils$$anonfun$getConsumersPerTopic$1.apply(ZkUtils.scala:665)
~[kafka_2.10-0.8.2.1.jar:na]
! at
kafka.utils.ZkUtils$$anonfun$getConsumersPerTopic$1.apply(ZkUtils.scala:664)
~[kafka_2.10-0.8.2.1.jar:na]
! at scala.collection.Iterator$class.foreach(Iterator.scala:727)
~[scala-library-2.10.4.jar:na]
! at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
~[scala-library-2.10.4.jar:na]
! at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
~[scala-library-2.10.4.jar:na]
! at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
~[scala-library-2.10.4.jar:na]
! at kafka.utils.ZkUtils$.getConsumersPerTopic(ZkUtils.scala:664)
~[kafka_2.10-0.8.2.1.jar:na]
! at kafka.consumer.AssignmentContext.(PartitionAssignor.scala:52)
~[kafka_2.10-0.8.2.1.jar:na]
! at
kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.kafka$consumer$ZookeeperConsumerConnector$ZKRebalancerListener$$rebalance(ZookeeperConsumerConnector.scala:659)
[kafka_2.10-0.8.2.1.jar:na]
! at
kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener$$anonfun$syncedRebalance$1$$anonfun$apply$mcV$sp$1.apply$mcVI$sp(ZookeeperConsumerConnector.scala:608)
~[kafka_2.10-0.8.2.1.jar:na]
! at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:141)
[scala-library-2.10.4.jar:na]
! at
kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener$$anonfun$syncedRebalance$1.apply$mcV$sp(ZookeeperConsumerConnector.scala:602)
[kafka_2.10-0.8.2.1.jar:na]
! at
kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener$$anonfun$syncedRebalance$1.apply(ZookeeperConsumerConnector.scala:599)
[kafka_2.10-0.8.2.1.jar:na]
! at
kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener$$anonfun$syncedRebalance$1.apply(ZookeeperConsumerConnector.scala:599)
[kafka_2.10-0.8.2.1.jar:na]
! at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
[kafka_2.10-0.8.2.1.jar:na]
! at
kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.syncedRebalance(ZookeeperConsumerConnector.scala:598)
[kafka_2.10-0.8.2.1.jar:na]
! at
kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener$$anon$1.run(ZookeeperConsumerConnector.scala:551)
[kafka_2.10-0.8.2.1.jar:na]
Caused by: ! org.apache.zookeeper.KeeperException$NoNodeException:
KeeperErrorCode = NoNode for
/consumers/real-time-updates/ids/real-time-updates_infra-buildagent-06-1444854764478-4dd4d6af
! at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
~[zookeeper-3.4.6.jar:3.4.6-1569965]
! at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
~[zookeeper-3.4.6.jar:3.4.6-1569965]
! at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
~[zookeeper-3.4.6.jar:3.4.6-1569965]
! at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1184)
~[zookeeper-3.4.6.jar:3.4.6-1569965]
! at org.I0Itec.zkclient.ZkConnection.readData(ZkConnection.java:103)
~[zkclient-0.3.jar:0.3]
! at org.I0Itec.zkclient.ZkClient$9.call(ZkClient.java:770)
~[zkclient-0.3.jar:0.3]
! at org.I0Itec.zkclient.ZkClient$9.call(ZkClient.java:766)
~[zkclient-0.3.jar:0.3]
! at org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:675)
~[zkclient-0.3.jar:0.3]
!... 21 common frames omitted

Re: kafka metrics emitting to graphite

[Alexander Pakulov]

You can try this plugin: https://github.com/apakulov/kafka-graphite

On Sun, Oct 11, 2015 at 3:19 AM, sunil kalva  wrote:

> How to configure, to emit kafka broker metrics to graphite.
>
> t
> SunilKalva
>



-- 
_
Best Regards, Alexander

Re: Facing Issue to create asyn producer in kafka 0.8.2

[prateek arora]

Thanks
 its work ...

Regards
Prateek

On Wed, Oct 14, 2015 at 11:46 AM, Grant Henke  wrote:

> Looks like you may be mixing the new producer with old producer configs.
> See the new config documentation here:
> http://kafka.apache.org/documentation.html#newproducerconfigs. You will
> likely want to set the "batch.size" and "linger.ms" to achieve your goal.
>
> Thanks,
> Grant
>
> On Wed, Oct 14, 2015 at 1:29 PM, prateek arora  >
> wrote:
>
> > Hi
> >
> > Thanks for help .
> >
> > but same behavior even after changing batch.size
> >
> > I have changes  batch.size value to 33554432.
> >  props.put("batch.size","33554432");
> >
> >
> >
> > On Wed, Oct 14, 2015 at 11:09 AM, Zakee  wrote:
> >
> > > Hi Prateek,
> > >
> > > Looks like you are using default batch.size which is ~16K and it forces
> > > the send of messages immediately as your single message is larger than
> > > that. Try using larger batch.size.
> > >
> > > Thanks
> > > Zakee
> > >
> > >
> > >
> > > > On Oct 14, 2015, at 10:29 AM, prateek arora <
> > prateek.arora...@gmail.com>
> > > wrote:
> > > >
> > > > Hi
> > > >
> > > > I want to create async producer so i can buffer messages in queue and
> > > send
> > > > after every 5 sec .
> > > >
> > > > my kafka version is 0.8.2.0.
> > > >
> > > > and i am using  kafka-clients 0.8.2.0 to create kafka producer in
> java.
> > > >
> > > >
> > > > below is my sample code :
> > > >
> > > > package com.intel.labs.ive.cloud.testKafkaProducerJ;
> > > >
> > > > import java.nio.charset.Charset;
> > > > import java.util.HashMap;
> > > >
> > > > import java.util.Map;
> > > >
> > > > import org.apache.kafka.clients.producer.KafkaProducer;
> > > > import org.apache.kafka.clients.producer.Producer;
> > > > import org.apache.kafka.clients.producer.ProducerConfig;
> > > > import org.apache.kafka.clients.producer.ProducerRecord;
> > > > import org.apache.kafka.common.Metric;
> > > > import org.apache.kafka.common.MetricName;
> > > > import org.apache.kafka.common.serialization.Serializer;
> > > > import org.apache.kafka.common.serialization.StringSerializer;
> > > > import org.apache.kafka.common.serialization.ByteArraySerializer;
> > > >
> > > > import java.nio.file.DirectoryStream;
> > > > import java.nio.file.Files;
> > > > import java.nio.file.Path;
> > > > import java.nio.file.Paths;
> > > >
> > > > public class TestKafkaProducer {
> > > >
> > > > Map props = new HashMap();
> > > >props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG,
> > > metadataBroker);
> > > >props.put("producer.type", "async");
> > > >props.put("queue.buffering.max.ms", "5000");
> > > >
> > > > Serializer keySerializer = new StringSerializer();
> > > >Serializer valueSerializer = new
> ByteArraySerializer();
> > > >
> > > >producer = new KafkaProducer(props,
> > keySerializer,
> > > > valueSerializer);
> > > >
> > > > ProducerRecord imageRecord;
> > > >
> > > > while ( true ) {
> > > > imageRecord = new ProducerRecord(topicName,
> > > > recordKey,imageBytes);
> > > >
> > > >producer.send(imageRecord);
> > > > }
> > > > }
> > > >
> > > > size of my message is around 77K
> > > >
> > > > but its work like a synchronous producer , send every message to
> broker
> > > .
> > > > not buffering a message in to queue and send after 5 sec
> > > >
> > > >
> > > > please help to find out a solution.
> > > >
> > > >
> > > > Regards
> > > > Prateek
> > >
> > > 
> > > A Balance Transfer Card With An Outrageously Long Intro Rate And No
> > > Balance Transfer Fees That Can Save You Thousands
> > >
> http://thirdpartyoffers.netzero.net/TGL3231/561e9a75a77071a74763fst04vuc
> >
>
>
>
> --
> Grant Henke
> Software Engineer | Cloudera
> gr...@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke
>

Re: Partitions - Brokers - Servers

[Julio Camara]

Gwen,

that helps. Thank you,

Julio

On 10/13/15 13:57, Gwen Shapira wrote:

Hi,

We normally run 1 broker per 1 physical server, and up to around 1000
partitions per broker (although that depends on the specific machine the
broker is on and specific configuration).

In order to enjoy replication, we recommend a minimum of 3 brokers in the
cluster, to support 3 replicas per partition.

Hope this helps?

Gwen

On Mon, Oct 12, 2015 at 12:30 PM, Julio Camara 
wrote:


I think I understand the concept of partitioning a topic and it's
benefits. But I can't find any clear explanation of relationship between
partitions, brokers and physical servers. Is there any document or article
that discusses the number of partitions for each broker and brokers for
each server, for example?

In some articles there were examples of a topic with 4 partitions in 4
brokers. Is this a typical implementation of partitions and brokers, i.e. n
partitions require n brokers?

I would like to be able to use best practices to size the number of
servers, brokers and partitions.

Thank you,
--
Julio Camara

Re: Strange ZK Error precedes frequent rebalances

[Gwen Shapira]

It is not strange, it means that one of the consumers lost connectivity to
Zookeeper, its session timed-out and this caused ephemeral ZK nodes (like
/consumers/real-time-updates/ids/real-time-updates_infra-
buildagent-06-1444854764478-4dd4d6af) to be removed and ultimately cause
the rebalance.

What you need is to make sure your consumers don't lose connectivity to
Zookeeper or that sessions don't time out. You do this by:
1. Tuning garbage collection on the consumer apps (G1 is recommended) to
avoid long GC pauses - leading cause for timeouts
2. Increasing Zookeeper session timeout on the consumer

Gwen

On Wed, Oct 14, 2015 at 1:47 PM, noah  wrote:

> A number of our developers are seeing errors like the one below in their
> console when running a consumer on their laptop. The error is always
> followed by logging indicating that the local consumer is rebalancing, and
> in the meantime we are not making much progress.
>
> I'm reading this as the consumer trying to read a ZK node for another
> consumer in the same group (running on a different machine,) but the node
> is no longer there. I can't tell if that is triggering a rebalance, or if
> it's just coincident.
>
> In our dev environment, we have a lot (hundreds) of consumers coming and
> going from the same consumer group, but they are mostly subscribed to
> different topics. Is this setup (sharing a consumer group across topics)
> potentially causing more rebalances than we would otherwise need? Or is
> something else entirely going on?
>
> LOG:
>
> INFO  [2015-10-14 20:32:49,138] kafka.consumer.ZookeeperConsumerConnector:
> [real-time-updates_Noahs-MacBook-Pro.local-1444853969114-7b52ecb5],
> exception during rebalance
> ! org.I0Itec.zkclient.exception.ZkNoNodeException:
> org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode =
> NoNode for
>
> /consumers/real-time-updates/ids/real-time-updates_infra-buildagent-06-1444854764478-4dd4d6af
> ! at org.I0Itec.zkclient.exception.ZkException.create(ZkException.java:47)
> ~[zkclient-0.3.jar:0.3]
> ! at org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:685)
> ~[zkclient-0.3.jar:0.3]
> ! at org.I0Itec.zkclient.ZkClient.readData(ZkClient.java:766)
> ~[zkclient-0.3.jar:0.3]
> ! at org.I0Itec.zkclient.ZkClient.readData(ZkClient.java:761)
> ~[zkclient-0.3.jar:0.3]
> ! at kafka.utils.ZkUtils$.readData(ZkUtils.scala:443)
> ~[kafka_2.10-0.8.2.1.jar:na]
> ! at kafka.consumer.TopicCount$.constructTopicCount(TopicCount.scala:61)
> ~[kafka_2.10-0.8.2.1.jar:na]
> ! at
>
> kafka.utils.ZkUtils$$anonfun$getConsumersPerTopic$1.apply(ZkUtils.scala:665)
> ~[kafka_2.10-0.8.2.1.jar:na]
> ! at
>
> kafka.utils.ZkUtils$$anonfun$getConsumersPerTopic$1.apply(ZkUtils.scala:664)
> ~[kafka_2.10-0.8.2.1.jar:na]
> ! at scala.collection.Iterator$class.foreach(Iterator.scala:727)
> ~[scala-library-2.10.4.jar:na]
> ! at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
> ~[scala-library-2.10.4.jar:na]
> ! at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
> ~[scala-library-2.10.4.jar:na]
> ! at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
> ~[scala-library-2.10.4.jar:na]
> ! at kafka.utils.ZkUtils$.getConsumersPerTopic(ZkUtils.scala:664)
> ~[kafka_2.10-0.8.2.1.jar:na]
> ! at kafka.consumer.AssignmentContext.(PartitionAssignor.scala:52)
> ~[kafka_2.10-0.8.2.1.jar:na]
> ! at
>
> kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.kafka$consumer$ZookeeperConsumerConnector$ZKRebalancerListener$$rebalance(ZookeeperConsumerConnector.scala:659)
> [kafka_2.10-0.8.2.1.jar:na]
> ! at
>
> kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener$$anonfun$syncedRebalance$1$$anonfun$apply$mcV$sp$1.apply$mcVI$sp(ZookeeperConsumerConnector.scala:608)
> ~[kafka_2.10-0.8.2.1.jar:na]
> ! at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:141)
> [scala-library-2.10.4.jar:na]
> ! at
>
> kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener$$anonfun$syncedRebalance$1.apply$mcV$sp(ZookeeperConsumerConnector.scala:602)
> [kafka_2.10-0.8.2.1.jar:na]
> ! at
>
> kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener$$anonfun$syncedRebalance$1.apply(ZookeeperConsumerConnector.scala:599)
> [kafka_2.10-0.8.2.1.jar:na]
> ! at
>
> kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener$$anonfun$syncedRebalance$1.apply(ZookeeperConsumerConnector.scala:599)
> [kafka_2.10-0.8.2.1.jar:na]
> ! at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> [kafka_2.10-0.8.2.1.jar:na]
> ! at
>
> kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.syncedRebalance(ZookeeperConsumerConnector.scala:598)
> [kafka_2.10-0.8.2.1.jar:na]
> ! at
>
> kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener$$anon$1.run(ZookeeperConsumerConnector.scala:551)
> [kafka_2.10-0.8.2.1.jar:na]
> Caused by: ! org.apache.zookeeper.KeeperException$NoNodeException:
> KeeperErrorCode = NoNode for
>
> 

Re: Strange ZK Error precedes frequent rebalances

[noah]

Thanks Gwen.

So am I right in deducing that any consumer in the same group dropping will
cause a rebalance, regardless of which topics they are subscribed to?

On Wed, Oct 14, 2015 at 3:52 PM Gwen Shapira  wrote:

> It is not strange, it means that one of the consumers lost connectivity to
> Zookeeper, its session timed-out and this caused ephemeral ZK nodes (like
> /consumers/real-time-updates/ids/real-time-updates_infra-
> buildagent-06-1444854764478-4dd4d6af) to be removed and ultimately cause
> the rebalance.
>
> What you need is to make sure your consumers don't lose connectivity to
> Zookeeper or that sessions don't time out. You do this by:
> 1. Tuning garbage collection on the consumer apps (G1 is recommended) to
> avoid long GC pauses - leading cause for timeouts
> 2. Increasing Zookeeper session timeout on the consumer
>
> Gwen
>
> On Wed, Oct 14, 2015 at 1:47 PM, noah  wrote:
>
> > A number of our developers are seeing errors like the one below in their
> > console when running a consumer on their laptop. The error is always
> > followed by logging indicating that the local consumer is rebalancing,
> and
> > in the meantime we are not making much progress.
> >
> > I'm reading this as the consumer trying to read a ZK node for another
> > consumer in the same group (running on a different machine,) but the node
> > is no longer there. I can't tell if that is triggering a rebalance, or if
> > it's just coincident.
> >
> > In our dev environment, we have a lot (hundreds) of consumers coming and
> > going from the same consumer group, but they are mostly subscribed to
> > different topics. Is this setup (sharing a consumer group across topics)
> > potentially causing more rebalances than we would otherwise need? Or is
> > something else entirely going on?
> >
> > LOG:
> >
> > INFO  [2015-10-14 20:32:49,138]
> kafka.consumer.ZookeeperConsumerConnector:
> > [real-time-updates_Noahs-MacBook-Pro.local-1444853969114-7b52ecb5],
> > exception during rebalance
> > ! org.I0Itec.zkclient.exception.ZkNoNodeException:
> > org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode =
> > NoNode for
> >
> >
> /consumers/real-time-updates/ids/real-time-updates_infra-buildagent-06-1444854764478-4dd4d6af
> > ! at
> org.I0Itec.zkclient.exception.ZkException.create(ZkException.java:47)
> > ~[zkclient-0.3.jar:0.3]
> > ! at org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:685)
> > ~[zkclient-0.3.jar:0.3]
> > ! at org.I0Itec.zkclient.ZkClient.readData(ZkClient.java:766)
> > ~[zkclient-0.3.jar:0.3]
> > ! at org.I0Itec.zkclient.ZkClient.readData(ZkClient.java:761)
> > ~[zkclient-0.3.jar:0.3]
> > ! at kafka.utils.ZkUtils$.readData(ZkUtils.scala:443)
> > ~[kafka_2.10-0.8.2.1.jar:na]
> > ! at kafka.consumer.TopicCount$.constructTopicCount(TopicCount.scala:61)
> > ~[kafka_2.10-0.8.2.1.jar:na]
> > ! at
> >
> >
> kafka.utils.ZkUtils$$anonfun$getConsumersPerTopic$1.apply(ZkUtils.scala:665)
> > ~[kafka_2.10-0.8.2.1.jar:na]
> > ! at
> >
> >
> kafka.utils.ZkUtils$$anonfun$getConsumersPerTopic$1.apply(ZkUtils.scala:664)
> > ~[kafka_2.10-0.8.2.1.jar:na]
> > ! at scala.collection.Iterator$class.foreach(Iterator.scala:727)
> > ~[scala-library-2.10.4.jar:na]
> > ! at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
> > ~[scala-library-2.10.4.jar:na]
> > ! at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
> > ~[scala-library-2.10.4.jar:na]
> > ! at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
> > ~[scala-library-2.10.4.jar:na]
> > ! at kafka.utils.ZkUtils$.getConsumersPerTopic(ZkUtils.scala:664)
> > ~[kafka_2.10-0.8.2.1.jar:na]
> > ! at kafka.consumer.AssignmentContext.(PartitionAssignor.scala:52)
> > ~[kafka_2.10-0.8.2.1.jar:na]
> > ! at
> >
> >
> kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.kafka$consumer$ZookeeperConsumerConnector$ZKRebalancerListener$$rebalance(ZookeeperConsumerConnector.scala:659)
> > [kafka_2.10-0.8.2.1.jar:na]
> > ! at
> >
> >
> kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener$$anonfun$syncedRebalance$1$$anonfun$apply$mcV$sp$1.apply$mcVI$sp(ZookeeperConsumerConnector.scala:608)
> > ~[kafka_2.10-0.8.2.1.jar:na]
> > ! at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:141)
> > [scala-library-2.10.4.jar:na]
> > ! at
> >
> >
> kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener$$anonfun$syncedRebalance$1.apply$mcV$sp(ZookeeperConsumerConnector.scala:602)
> > [kafka_2.10-0.8.2.1.jar:na]
> > ! at
> >
> >
> kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener$$anonfun$syncedRebalance$1.apply(ZookeeperConsumerConnector.scala:599)
> > [kafka_2.10-0.8.2.1.jar:na]
> > ! at
> >
> >
> kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener$$anonfun$syncedRebalance$1.apply(ZookeeperConsumerConnector.scala:599)
> > [kafka_2.10-0.8.2.1.jar:na]
> > ! at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> > [kafka_2.10-0.8.2.1.jar:na]
> > ! at
> >
> 

Re: Strange ZK Error precedes frequent rebalances

[Gwen Shapira]

Yes. The rebalance is on consumers in the group and does not take topics
into account.

On Wed, Oct 14, 2015 at 1:59 PM, noah  wrote:

> Thanks Gwen.
>
> So am I right in deducing that any consumer in the same group dropping will
> cause a rebalance, regardless of which topics they are subscribed to?
>
> On Wed, Oct 14, 2015 at 3:52 PM Gwen Shapira  wrote:
>
> > It is not strange, it means that one of the consumers lost connectivity
> to
> > Zookeeper, its session timed-out and this caused ephemeral ZK nodes (like
> > /consumers/real-time-updates/ids/real-time-updates_infra-
> > buildagent-06-1444854764478-4dd4d6af) to be removed and ultimately cause
> > the rebalance.
> >
> > What you need is to make sure your consumers don't lose connectivity to
> > Zookeeper or that sessions don't time out. You do this by:
> > 1. Tuning garbage collection on the consumer apps (G1 is recommended) to
> > avoid long GC pauses - leading cause for timeouts
> > 2. Increasing Zookeeper session timeout on the consumer
> >
> > Gwen
> >
> > On Wed, Oct 14, 2015 at 1:47 PM, noah  wrote:
> >
> > > A number of our developers are seeing errors like the one below in
> their
> > > console when running a consumer on their laptop. The error is always
> > > followed by logging indicating that the local consumer is rebalancing,
> > and
> > > in the meantime we are not making much progress.
> > >
> > > I'm reading this as the consumer trying to read a ZK node for another
> > > consumer in the same group (running on a different machine,) but the
> node
> > > is no longer there. I can't tell if that is triggering a rebalance, or
> if
> > > it's just coincident.
> > >
> > > In our dev environment, we have a lot (hundreds) of consumers coming
> and
> > > going from the same consumer group, but they are mostly subscribed to
> > > different topics. Is this setup (sharing a consumer group across
> topics)
> > > potentially causing more rebalances than we would otherwise need? Or is
> > > something else entirely going on?
> > >
> > > LOG:
> > >
> > > INFO  [2015-10-14 20:32:49,138]
> > kafka.consumer.ZookeeperConsumerConnector:
> > > [real-time-updates_Noahs-MacBook-Pro.local-1444853969114-7b52ecb5],
> > > exception during rebalance
> > > ! org.I0Itec.zkclient.exception.ZkNoNodeException:
> > > org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode =
> > > NoNode for
> > >
> > >
> >
> /consumers/real-time-updates/ids/real-time-updates_infra-buildagent-06-1444854764478-4dd4d6af
> > > ! at
> > org.I0Itec.zkclient.exception.ZkException.create(ZkException.java:47)
> > > ~[zkclient-0.3.jar:0.3]
> > > ! at
> org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:685)
> > > ~[zkclient-0.3.jar:0.3]
> > > ! at org.I0Itec.zkclient.ZkClient.readData(ZkClient.java:766)
> > > ~[zkclient-0.3.jar:0.3]
> > > ! at org.I0Itec.zkclient.ZkClient.readData(ZkClient.java:761)
> > > ~[zkclient-0.3.jar:0.3]
> > > ! at kafka.utils.ZkUtils$.readData(ZkUtils.scala:443)
> > > ~[kafka_2.10-0.8.2.1.jar:na]
> > > ! at
> kafka.consumer.TopicCount$.constructTopicCount(TopicCount.scala:61)
> > > ~[kafka_2.10-0.8.2.1.jar:na]
> > > ! at
> > >
> > >
> >
> kafka.utils.ZkUtils$$anonfun$getConsumersPerTopic$1.apply(ZkUtils.scala:665)
> > > ~[kafka_2.10-0.8.2.1.jar:na]
> > > ! at
> > >
> > >
> >
> kafka.utils.ZkUtils$$anonfun$getConsumersPerTopic$1.apply(ZkUtils.scala:664)
> > > ~[kafka_2.10-0.8.2.1.jar:na]
> > > ! at scala.collection.Iterator$class.foreach(Iterator.scala:727)
> > > ~[scala-library-2.10.4.jar:na]
> > > ! at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
> > > ~[scala-library-2.10.4.jar:na]
> > > ! at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
> > > ~[scala-library-2.10.4.jar:na]
> > > ! at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
> > > ~[scala-library-2.10.4.jar:na]
> > > ! at kafka.utils.ZkUtils$.getConsumersPerTopic(ZkUtils.scala:664)
> > > ~[kafka_2.10-0.8.2.1.jar:na]
> > > ! at
> kafka.consumer.AssignmentContext.(PartitionAssignor.scala:52)
> > > ~[kafka_2.10-0.8.2.1.jar:na]
> > > ! at
> > >
> > >
> >
> kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.kafka$consumer$ZookeeperConsumerConnector$ZKRebalancerListener$$rebalance(ZookeeperConsumerConnector.scala:659)
> > > [kafka_2.10-0.8.2.1.jar:na]
> > > ! at
> > >
> > >
> >
> kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener$$anonfun$syncedRebalance$1$$anonfun$apply$mcV$sp$1.apply$mcVI$sp(ZookeeperConsumerConnector.scala:608)
> > > ~[kafka_2.10-0.8.2.1.jar:na]
> > > ! at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:141)
> > > [scala-library-2.10.4.jar:na]
> > > ! at
> > >
> > >
> >
> kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener$$anonfun$syncedRebalance$1.apply$mcV$sp(ZookeeperConsumerConnector.scala:602)
> > > [kafka_2.10-0.8.2.1.jar:na]
> > > ! at
> > >
> > >
> >
> 

Re: Kafka 0.9.0 release branch

[Guozhang Wang]

Just assigned reviewers for all the blocker issues. Please always feel free
to reassign, the purpose is to drive the review process towards the release.

On Tue, Oct 13, 2015 at 2:29 PM, Guozhang Wang  wrote:

> If time permits, it would be great to have both KAFKA-2397 and KAFKA-2017
> in 0.9.0 as both of them are trying to fix some regressions of the new
> consumer compared with the old consumer.
>
> Priority-wise, I think 2017 is higher than 2397 since upon a broker
> shutdown / failure, all connected consumers to this broker will go through
> a full rebalance whereas for 2017, upon a consumer shutdown / failure it
> will only affect the consumers within the same group.
>
> Effort-wise, 2397 is simpler than 2017 since we need to consider what ZK
> data format to achieve good trade-off between storage / efficiency.
>
> For now we can work on both of them in parallel: Onur already has a patch
> for 2397 that Jason is reviewing, I will re-start working on 2017.
>
> Guozhang
>
> On Tue, Oct 13, 2015 at 12:50 PM, Rajiv Kurian  wrote:
>
>> A bit off topic but does this release contain the new single threaded
>> consumer that supports the poll interface?
>>
>> Thanks!
>>
>> On Mon, Oct 12, 2015 at 1:31 PM, Jun Rao  wrote:
>>
>> > Hi, Everyone,
>> >
>> > As we are getting closer to the 0.9.0 release, we plan to cut an 0.9.0
>> > release branch in about two weeks from now. In the meantime, we will
>> try to
>> > resolve most if not all 0.9.0 blockers listed below.
>> >
>> >
>> >
>> https://issues.apache.org/jira/issues/?jql=project%20%3D%20KAFKA%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened%2C%20%22Patch%20Available%22)%20AND%20priority%20%3D%20Blocker%20AND%20fixVersion%20%3D%200.9.0.0%20ORDER%20BY%20updated%20DESC
>> >
>> > Are there any concerns? We will also discuss the 0.9.0 release in
>> > tomorrow's KIP meeting.
>> >
>> > Thanks,
>> >
>> > Jun
>> >
>>
>
>
>
> --
> -- Guozhang
>



-- 
-- Guozhang

Re: G1 tuning

[Scott Clasen]

You can also use -Xmn with that gc to size the new gen such that those
buffers don't get tenured

I don't think that's an option with G1

On Wednesday, October 14, 2015, Cory Kolbeck  wrote:

> I'm not sure that will help here, you'll likely have the same
> medium-lifetime buffers getting into the tenured generation and forcing
> large collections.
>
> On Wed, Oct 14, 2015 at 10:00 AM, Gerrit Jansen van Vuuren <
> gerrit...@gmail.com > wrote:
>
> > Hi,
> >
> > I've seen pauses using G1 in other applications and have found that
> > -XX:+UseParallelGC
> > -XX:+UseParallelOldGC  works best if you're having GC issues in general
> on
> > the JVM.
> >
> >
> > Regards,
> >  Gerrit
> >
> > On Wed, Oct 14, 2015 at 4:28 PM, Cory Kolbeck  > wrote:
> >
> > > Hi folks,
> > >
> > > I'm a bit new to the operational side of G1, but pretty familiar with
> its
> > > basic concept. We recently set up a Kafka cluster to support a new
> > product,
> > > and are seeing some suboptimal GC performance. We're using the
> parameters
> > > suggested in the docs, except for having switched to java 1.8_40 in
> order
> > > to get better memory debugging. Even though the cluster is handling
> only
> > > 2-3k messages per second per node, we see periodic 11-18 second
> > > stop-the-world pauses on a roughly hourly cadence. I've turned on
> > > additional GC logging, and see no humongous allocations, it all seems
> to
> > be
> > > buffers making it into the tenured gen. They appear to be collectable,
> as
> > > the collection triggered by dumping the heap collects them all. Ideas
> for
> > > additional diagnosis or tuning very welcome.
> > >
> > > --Cory
> > >
> >
>

How Kafka work to send message to consumer?

[Kiran Singh]

Hello

I have one query related to Kafka data flow towards consumer.

Means whether kafka used push or pull technic to send data to a consumer
using high level API?

Re: How Kafka work to send message to consumer?

[Mayuresh Gharat]

Kafka is a pull mechanism.

Thanks,

Mayuresh

On Wed, Oct 14, 2015 at 9:47 PM, Kiran Singh  wrote:

> Hello
>
> I have one query related to Kafka data flow towards consumer.
>
> Means whether kafka used push or pull technic to send data to a consumer
> using high level API?
>



-- 
-Regards,
Mayuresh R. Gharat
(862) 250-7125

Difference between storing offset on Kafka and on Zookeeper server?

[Kiran Singh]

What are the major advantage to store Offset on kafka server instead of
zookeeper?

Please share any link for the same.

Re: How Kafka work to send message to consumer?

[Kiran Singh]

So it's consumer responsibility to keep on checking whether a new data
available on Kafka server or not.
Am I right?

On Thu, Oct 15, 2015 at 10:21 AM, Mayuresh Gharat <
gharatmayures...@gmail.com> wrote:

> Kafka is a pull mechanism.
>
> Thanks,
>
> Mayuresh
>
> On Wed, Oct 14, 2015 at 9:47 PM, Kiran Singh 
> wrote:
>
> > Hello
> >
> > I have one query related to Kafka data flow towards consumer.
> >
> > Means whether kafka used push or pull technic to send data to a consumer
> > using high level API?
> >
>
>
>
> --
> -Regards,
> Mayuresh R. Gharat
> (862) 250-7125
>