Kafka network metrics with Ambari Metrics System

2016-08-31 Thread Jahn Roux
since moving to AMS it does not seem like there are any kafka.network.* metrics being reported. I have checked that these metrics are not being excluded in the Kafka configs. Anyone have any idea how I could track down these metrics with AMS? Thanks! Kind regards, Jahn Roux

RE: Large kafka deployment on virtual hardware

2016-05-24 Thread Jahn Roux
d a lot of leader change activity - could this be a symptom of the offline partitions? Kind regards, Jahn Roux -Original Message- From: Tom Crayford [mailto:tcrayf...@heroku.com] Sent: Tuesday, May 24, 2016 3:07 PM To: Users Subject: Re: Large kafka deployment on virtual hardware Jahn, Are

RE: Large kafka deployment on virtual hardware

2016-05-24 Thread Jahn Roux
just contention for resources? Network clogged or some other simpler explanation like that? On Mon, May 23, 2016 at 9:42 PM, Jahn Roux wrote: > I have a large Kafka deployment on virtual hardware: 120 brokers on > 32gb memory 8 core virtual machines. Gigabit network, RHEL 6.7. 4 > Topics, 1

RE: Large kafka deployment on virtual hardware

2016-05-24 Thread Jahn Roux
Thanks for the response Jens. That is one of the first things we looked at, was the ZK performance. ZK seems nominal, low load and no exhaustion of any resources. ZK offset commit is not set by us explicitly, so default. Kind regards, Jahn Roux -Original Message- From: Jens Rantil

Large kafka deployment on virtual hardware

2016-05-23 Thread Jahn Roux
a look at? Is there known issues with Kafka on virtualized hardware or things to watch out for when deploying to VMs? Are there use cases where Kafka is being used in a similar way - +4 million messages a second of discrete 150 byte messages? Kind regards, Jahn Roux --- This emai