Re: G1 tuning

You can also use -Xmn with that gc to size the new gen such that those buffers don't get tenured I don't think that's an option with G1 On Wednesday, October 14, 2015, Cory Kolbeck wrote: > I'm not sure that will help here, you'll likely have the same > medium-lifetime buffers getting into the

Re: Inconsistency with Zookeeper

each zk needs a myid file in the data dir, with a different number 1,2,3 http://zookeeper.apache.org/doc/r3.3.3/zookeeperAdmin.html You can find the meanings of these and other configuration settings in the section Configuration Parameters

Re: ping kafka server

I have used nagios in this manner with kafaka before and worked fine. On Mon, Feb 9, 2015 at 2:48 PM, Koert Kuipers wrote: > i would like to be able to ping kafka servers from nagios to confirm they > are alive. since kafka servers dont run a http server (web ui) i am not > sure how to do this.

Re: kafka consumer to write into DB

if you are using scala/akka this will handle the batching and acks for you. https://github.com/sclasen/akka-kafka#akkabatchconsumer On Fri, Dec 5, 2014 at 9:21 AM, Sa Li wrote: > Thank you very much for the reply, Neha, I have a question about consumer, > I consume the data from kafka and write

Re: Announcing Confluent

Awesome. Congrats to all of you! On Thu, Nov 6, 2014 at 10:28 AM, Jay Kreps wrote: > Hey all, > > I’m happy to announce that Jun Rao, Neha Narkhede and I are creating a > company around Kafka called Confluent. We are planning on productizing the > kind of Kafka-based real-time data platform we b

Re: how to ensure strong consistency with reasonable availability

have been acked at > least > > >> by the the local broker. > > >> > > >> I guess I am hoping for some intuition around why 'replaying' the > > messages > > >> in question would cause bad things. > > >> > > >> Than

Re: how to ensure strong consistency with reasonable availability

Ahh, yes that message loss case. I've wondered about that myself. I guess I dont really understand why truncating messages is ever the right thing to do. As kafka is an 'at least once' system. (send a message, get no ack, it still might be on the topic) consumers that care will have to de-dupe an

Re: how to ensure strong consistency with reasonable availability

You will probably need 0.8.2 which gives https://issues.apache.org/jira/browse/KAFKA-1028 On Mon, Jul 21, 2014 at 6:37 PM, Jiang Wu (Pricehistory) (BLOOMBERG/ 731 LEX -) wrote: > Hi everyone, > > With a cluster of 3 brokers and a topic of 3 replicas, we want to achieve > the following two prop

event-shuttle

Thought Id post this here, as Ive seen questions here around "What do I do if kafka is down?" here is one possibility. https://github.com/sclasen/event-shuttle this is a go service meant to run as a unix daemon, and apps running on the same box can just post messages over http to a service on lo

Re: Reaction to Reactive Streams?

Would be great to get a reactive consumer api (in 0.9 ??) On Thu, Apr 17, 2014 at 11:58 AM, Richard Rodseth wrote: > Announcement from Typesafe et al: > > http://www.reactive-streams.org/ > https://typesafe.com/blog/typesafe-announces-akka-streams >

Re: New Producer Public API

+1 to zk bootstrap + close as an option at least On Tue, Jan 28, 2014 at 10:09 AM, Neha Narkhede wrote: > >> The producer since 0.8 is actually zookeeper free, so this is not new to > this client it is true for the current client as well. Our experience was > that direct zookeeper connections f

Re: Securing kafka

Please contribute that back!, Would potentially be huge for mirroring clusters across Amazon Regions, for instance. On Thu, Aug 29, 2013 at 8:22 PM, Rajasekar Elango wrote: > We have made changes to kafka code to support certificate based mutual SSL > authentication. So the clients and broker wi

Re: message loss

nk if we included this knob, the tests that aphyr (jepsen) ran, would > make more sense. > > Thanks, > Neha > > > On Thu, Aug 22, 2013 at 12:50 PM, Scott Clasen wrote: > >> So looks like there is a jespen post coming on kafka 0.8 replication, based >> on this

message loss

So looks like there is a jespen post coming on kafka 0.8 replication, based on this thats circulating on twitter. https://www.refheap.com/17932/raw Understanding that kafka isnt designed particularly to be partition tolerant, the result is not completely surprising. But my question is, is there s

Re: Kafka 08 clients

bpot/poseidon on github is a ruby 0.8 client, works fine for me Sent from my iPhone On Aug 10, 2013, at 3:08 PM, Timothy Chen wrote: > That's definitely means it's not up to date to the protocol, I'm tried the > java client and it was working with latest 0.8 api. > > Not sure about any other l

Re: Metrics: via Broker vs. Producer vs. Consumer

Valid assumptions IMO Sent from my iPhone On Jul 24, 2013, at 8:22 AM, Jay Kreps wrote: > Yeah all our monitoring is just of the local process (basically just a > counter exposed through yammer metrics which support jmx and other > outputs). If I understand what you want instead of having a co

Re: Replacing brokers in a cluster (0.8)

Here's a ruby cli that you can use to replace brokers...it shells out to the kafka-reassign-partitions.sh tool after figuring out broker lists from zk. Hope its useful. #!/usr/bin/env ruby require 'excon' require 'json' require 'zookeeper' def replace(arr, o, n) arr.map{|v| v == o ? n : v } e

0.8 backup strategy anyone?

So despite 0.8 being a release that will give much higher availability do people do anything at all to back up the data? For instance if any of you are running on EC2 and using ephemeral disk for perf reasons, what do you do about messages that you absolutely cant afford to lose. Basically look

Re: Monitor Kafka Stats

Hi Jun Have you looked at Kafka-898 in jira? It has a patch for reporting metrics to librato, which would make it very easy for people to get started with Kafka metrics. Sent from my iPhone On Jun 6, 2013, at 8:55 AM, Jun Rao wrote: > Those stats on the broker are available through JMX. In 0

Re: Apache Kafka in AWS

t; > Jason > > > > From: Scott Clasen [sc...@heroku.com] > Sent: Wednesday, May 22, 2013 19:27 > To: users > Subject: Re: Apache Kafka in AWS > > Hey Jason, > > question what openjdk version did you have issues with? Im

Re: Apache Kafka in AWS

Hey Jason, question what openjdk version did you have issues with? Im running kafka on it now and has been ok. Was it a crash only at load? Thanks SC On Wed, May 22, 2013 at 1:42 PM, Jason Weiss wrote: > All, > > I asked a number of questions of the group over the last week, and I'm > happy

Re: Relationship between Zookeeper and Kafka

attending the AWS event in NYC a > few weeks ago was presented as the "fastest storage option" for EC2. A > number of partners presented success stories in terms of throughput with > provisioned IOPS. I've tried to follow that model. > > > Jason > > On 5/

Re: Relationship between Zookeeper and Kafka

My guess, EBS is likely your bottleneck. Try running on instance local disks, and compare your results. Is this 0.8? What replication factor are you using? On Mon, May 20, 2013 at 8:11 AM, Jason Weiss wrote: > I'm trying to maximize my throughput and seem to have hit a ceiling. > Everything d

Re: Update: RE: are commitOffsets botched to zookeeper?

since partitions will be highly available, offsets could be > stored in Kafka topics. However, we haven't ironed out the design for this > yet. > > Thanks, > Neha > > > On Fri, May 17, 2013 at 2:19 PM, Scott Clasen wrote: > > > afaik you dont 'have' to st

Re: Update: RE: are commitOffsets botched to zookeeper?

afaik you dont 'have' to store the consumed offsets in zk right, this is only automatic with some of the clients? why not store them in a data store that can write at the rate that you require? On Fri, May 17, 2013 at 2:15 PM, Withers, Robert wrote: > Update from our OPS team, regarding zookeep

Re: Using Stunnel to encrypt/authenticate Kafka producers and consumers...

I think you are right, even if you did put an ELB in front of kafka, it would only be used for getting the initial broker list afaik. Producers and consumers need to be able to talk to each broker directly, and also consumers need to be able to talk to zookeeper as well to store offsets. Probably

Re: How to create the initial zookeeper chroot path for zk.connect?

the problem. > > But the connection string as "Zk1:2181,zk2:2181,zk3;2181/Kafka", seems > unintuitive? > > > On Sun, Apr 21, 2013 at 2:29 AM, Scott Clasen wrote: > >> Afaik you only put the chroot on the end of the zk conn str... >> >> Zk1:2181,zk2:

Re: How to create the initial zookeeper chroot path for zk.connect?

Afaik you only put the chroot on the end of the zk conn str... Zk1:2181,zk2:2181,zk3;2181/Kafka Not Zk1:2181/kafka,zk2:2181/Kafka,zk3:2181/Kafka Sent from my iPhone On Apr 20, 2013, at 9:03 AM, Neha Narkhede wrote: > Please file a bug and mention the Kafka and zookeeper versions used for th

Re: kafka 0.8 release schedule?

+1 On Wed, Apr 3, 2013 at 4:03 PM, Ryan LeCompte wrote: > A 2.10 version would be awesome! Please please :-) > > > On Wed, Apr 3, 2013 at 4:02 PM, Soby Chacko wrote: > > > Hello, > > > > I am sure this question has been asked before. But, can someone tell me > the > > tentative release schedul

Re: 0.8 best practices for migrating / electing leaders in failure situations?

l after you have run the partition > reassignment tool. > > Thanks, > > Jun > > On Fri, Mar 22, 2013 at 9:23 PM, Scott Clasen wrote: > > > Thanks! > > > > Would there be any difference if I instead deleted all the Kafka data > > from zookeepe

Re: 0.8 best practices for migrating / electing leaders in failure situations?

can bring up 3 new brokers with the same broker id. You > won't get the data back. However, new data can be published to and consumed > from the new brokers. > > Thanks, > > Jun > > On Fri, Mar 22, 2013 at 2:17 PM, Scott Clasen wrote: > >> Thanks Neha- >

Re: 0.8 best practices for migrating / electing leaders in failure situations?

use reassign partitions tool and for all partitions that > had a replica on broker 2, move it to broker 4 > > * Scenario 2: BrokerID 1,2,3 Catastrophic failure 1,2,3 die but ZK still > there. > > There is no way to recover any data here since there is nothing > available to consume data

0.8 best practices for migrating / electing leaders in failure situations?

What would the recommended practice be for the following scenarios? Running on EC2, ephemperal disks only for kafka. There are 3 kafka servers. The broker ids are always increasing. If a broker dies its never coming back. All topics have a replication factor of 3. * Scenario 1: BrokerID 1,2,3

Re: 0.8 reassignment issue (built from 7b14ebae3382427b44a928d0b186001735c15efb)

KAFKA-821 Cheers! On Thu, Mar 21, 2013 at 1:58 PM, Neha Narkhede wrote: > Please can you file a bug, this needs to be fixed. > > Thanks > Neha > > > On Thu, Mar 21, 2013 at 1:50 PM, Scott Clasen wrote: > > > So, deleting /admin/reassign_partitions from Z

Re: 0.8 reassignment issue (built from 7b14ebae3382427b44a928d0b186001735c15efb)

So, deleting /admin/reassign_partitions from Zookeeper, which had an empty array of partitions seemed to fix things. Probably isnt the preferred way though :) On Thu, Mar 21, 2013 at 12:40 PM, Scott Clasen wrote: > Hi all, Im currently testing out 0.8 failure/recovery stuff. Having an >

Re: Not seeing any replies

testing 1 2 3 for Bob On Thu, Mar 21, 2013 at 12:48 PM, Bob Jervis wrote: > How do I get to see replies from your e-mails??? > > > I have seen nothing weither through my gmail account or through my company > account. > > THis mailing list is not wokring for me. > > How do I communicate with

0.8 reassignment issue (built from 7b14ebae3382427b44a928d0b186001735c15efb)

Hi all, Im currently testing out 0.8 failure/recovery stuff. Having an issue with reassigning a partition, after making a mistake in my first attempt to reassign, seems like reassignment is hosed. Is this known or should I open a ticket? Have 3 brokers running. Ids 25,26,27 ./bin/kafka-create-t