Re: 0.8 best practices for migrating / electing leaders in failure situations?

You can't create the topic with the same name before deleting it first. Thanks, Jun On Mon, Mar 25, 2013 at 9:58 AM, Scott Clasen wrote: > Jun Thanks. To clarify, do you mean that clients will have cached broker > lists or some other data that will make them ignore the new brokers? > > Like so

Re: Connection reset by peer

Thanks Neha, btw have you seen this exception. We didn't restart any service it happens in deep night. java.lang.RuntimeException: A broker is already registered on the path /brokers/ids/0. This probably indicates that you either have configured a brokerid that is already in use, or else you have

Re: Connection reset by peer

That really depends on your consumer application's memory allocation patterns. If it is a thin wrapper over a Kafka consumer, I would imagine you can get away with using CMS for the tenured generation and parallel collector for the new generation with a small heap like 1gb or so. Thanks, Neha On

Re: Connection reset by peer

Any suggestion on consumer side? 在 2013-3-25 下午9:49，"Neha Narkhede" 写道： > For Kafka 0.7 in production at Linkedin, we use a heap of size 3G, new gen > 256 MB, CMS collector with occupancy of 70%. > > Thanks, > Neha > > On Sunday, March 24, 2013, Yonghui Zhao wrote: > > > Hi Jun, > > > > I used kaf

question about virtual ip

Hi， We use kafka 0.7.2 and use virtual IP for producer end, the VIP tools we are using is LVS. Sadly, it seems not working on LVS, when the broker changes, the producer seems not reconnect to the new broker. So has anyone been successfully using the same VIP mode? What VIP tools are you using? TH

Re: 0.8 best practices for migrating / electing leaders in failure situations?

The assignment of partitions to replicas (brokers) happens at the create topic time. After that, it can only be changed through the partition reassignment tool. The replicas are identified using the broker id, so if you keep the broker ids intact, Kafka cluster and clients cannot tell that the prev

Re: log.file.size limit?

We limit individual segment files to 2GB for slightly esoteric reasons. There is no limit for the log as a whole so limiting the segments to 2GB does not constrain how much data you can retain. -Jay On Mon, Mar 25, 2013 at 11:42 AM, S Ahmed wrote: > Is there any limit to how large a log file can

Re: log.file.size limit?

If you look at the description of the "map" method, it states: size - The size of the region to be mapped; must be non-negative and no greater than Integer.MAX_VALUE -David On 3/25/13 4:37 PM, S Ahmed wrote: but it show's long not int? Isn't it then Long.MAX_VALUE ? On Mon, Mar 25, 2013 a

Re: log.file.size limit?

but it show's long not int? Isn't it then Long.MAX_VALUE ? On Mon, Mar 25, 2013 at 3:14 PM, David Arthur wrote: > FileChannel#map docs indicate the max size is Integer.MAX_VALUE, so yea 2gb > > http://docs.oracle.com/javase/**6/docs/api/java/nio/channels/** > FileChannel.html#map(java.nio.**ch

Re: log.file.size limit?

FileChannel#map docs indicate the max size is Integer.MAX_VALUE, so yea 2gb http://docs.oracle.com/javase/6/docs/api/java/nio/channels/FileChannel.html#map(java.nio.channels.FileChannel.MapMode, long, long) On 3/25/13 2:42 PM, S Ahmed wrote: Is there any limit to how large a log file can be?

log.file.size limit?

Is there any limit to how large a log file can be? I swear I read somewhere that java's memory mapped implementation is limited to 2GB but I'm not sure.

Re: Kafka mirroring fault tolerance

Thanks Jun. We have a cluster as well. However, in our setup each Kafka node has both the mirror maker and broker process running, with the mirror maker writing only to the local broker. The main rationale was to limit further use of network. But, I guess that wasn't a good idea given the failure s

Re: 0.8 best practices for migrating / electing leaders in failure situations?

Jun Thanks. To clarify, do you mean that clients will have cached broker lists or some other data that will make them ignore the new brokers? Like so topic-1 replication factor 3, on broker-ids 1,2,3 all brokers 1,2,3 die, and are never coming back. delete all kafka data in zookeeper. boot 4,5,6,

Re: Anyone working on a Kafka book?

Thanks Jun, I've updated the example with this information. I've also removed some of the unnecessary newlines. Thanks, Chris On Mon, Mar 25, 2013 at 12:04 PM, Jun Rao wrote: > Chris, > > This looks good. One thing about partitioning. Currently, if a message > doesn't have a key, we always

Re: Anyone working on a Kafka book?

Chris, This looks good. One thing about partitioning. Currently, if a message doesn't have a key, we always use the random partitioner (regardless of what "partitioner.class" is set to). Thanks, Jun On Thu, Mar 21, 2013 at 11:42 AM, Chris Curtin wrote: > I published my first Wiki example: > >

Re: Connection reset by peer

For Kafka 0.7 in production at Linkedin, we use a heap of size 3G, new gen 256 MB, CMS collector with occupancy of 70%. Thanks, Neha On Sunday, March 24, 2013, Yonghui Zhao wrote: > Hi Jun, > > I used kafka-server-start.sh to start kafka, there is only one jvm setting > "-Xmx512M“ > > Do you hav

Kafka wiki Documentation conventions - looking for feedback

Hi David, Thanks for the feedback. I've seen the example before and after in different books/articles and it doesn't matter to me. Anyone else want to help define a style guide or is there one I didn't see already? Thanks, Chris On Thu, Mar 21, 2013 at 7:46 PM, David Arthur wrote: > This lo

Re: Anyone working on a Kafka book?

Thanks for finding those. Looks like a copy and paste issue. I've updated the document. Thanks, Chris On Sat, Mar 23, 2013 at 11:27 AM, Jonathan Hodges wrote: > Many thanks for contributing! The docs are very helpful. I found a couple > small possible typos. The partitioning code example l