How do I turn off INFO log for embedded kafka server

Hey guys, I have some unit test that has an embedded kafka server running. I want to skip all debug and info logs from kafka server. But having this set in log4j.properties does work. Some INFO log still keep showing up like this: 2016-07-08 18:01:14,288 [kafka-request-handler-4] INFO cluster.Par

log4j setting for embedded kafka server

Hi guys, I start server grammatically in my application using KafkaStatableServer.startup() method. And in the log4j.properties setting. I add this log4j.logger.org.apacke.kafka=WARN log4j.logger.kafka=WARN But I always get INFO log, Do you guys know how to enforce the log level here? Thanks!

Log level for consumer properties

Hi, Right now, when we initialize kafka consumer, it always log the consumer properties at INFO level, can we put it into DEBUG level? I have to periodically create consumer instance to just pull some metadata of some topic and I don't want to get this noisy log. Regards, Siyuan

Exceptions when programmatically start multiple kafka brokers

I'm trying to start 2 brokers in my kafka ingestion unit test and I got exception javax.management.InstanceAlreadyExistsException: kafka.server:type=app-info,id=0 at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:437) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.re

Where can I find the document for consumer metrics

I can find some broker/producer metrics here http://kafka.apache.org/documentation.html#monitoring but where can I find consumer metrics docs Everytime I have to log this to find out what metrics I want MetricName [name=join-rate, group=consumer-coordinator-metrics, description=The number of gro

Re: how to programatically monitor Kafka availability

Hey Hohl, I use *partitionsFor * method to monitor the partition info for particular topics On Tue, Dec 15, 2015 at 11:27 AM, Hohl, Ken wrote: > We want to be able to

Re: Kafka 0.9 consumer API question

Hi Rajiv, I think it makes sense to return a read-only assignments. What we can improve here is we can have addPartition&removePartition method for consumer. Then we don't have to do any operations on the assignments returned by assignment method BTW, I think you can implement PartitionAssignor i

Is 0.9 new consumer API compatible with 0.8.x.x broker

Is 0.9 new consumer API compatible with 0.8.x.x broker

partitionsFor method doesn't return latest partition metadata

Hi guys, I want to use partitionsFor method of new consumer API periodically to get the monitor partition metadata change, It seems it only issue remote call to the server for the first time. If I add partitions after that, partitionsFor will return stale value. Is there a way to reuse consumer ob

Re: 0.9.0.0 RC4

In http://kafka.apache.org/090/documentation.html#newconsumerconfigs partition.assignment.strategy should string, not a list of string? On Fri, Nov 20, 2015 at 5:21 PM, Jun Rao wrote: > This is the fourth candidate for release of Apache Kafka 0.9.0.0. This a > major release that includes (1) aut

Re: Commit offsets only work for subscribe(), not assign()

iyuan, > > > > The commit API should work the same regardless whether subscribe() or > > assign() was used. Does this not appear to be working? > > > > Thanks, > > Jason > > > > On Wed, Nov 18, 2015 at 4:40 PM, hsy...@gmail.com > > wrote: >

Commit offsets only work for subscribe(), not assign()

In the new API, the explicit commit offset method call only works for subscribe consumer, not the assign consumer, correct? Best, Siyuan

Re: Q about PartitionAssignor

over other new leaders can still access the past assignment history. > > Guozhang > > > > On Wed, Nov 18, 2015 at 9:02 AM, hsy...@gmail.com > wrote: > > > Hey guys, > > > > I saw the PartitionAssignor is not in public doc API and the package name >

Q about PartitionAssignor

Hey guys, I saw the PartitionAssignor is not in public doc API and the package name is internals. Does it mean this API is not stable and could be changed even in minor release? And in the assign method signature, the key for the "subscription" map is memberId, what is memberId, can I manually s

Re: Questions about new consumer API

he > leaders of each partition. Then you could just use subscribe() and let > Kafka manage the group for you. This is similar to how we were thinking of > implementing consumer rack-awareness. > > -Jason > > On Tue, Nov 17, 2015 at 4:04 PM, hsy...@gmail.com > wrote: > >

Re: Questions about new consumer API

the current partition leaders and assign() to set > the assignment in each thread. > > -Jason > > On Tue, Nov 17, 2015 at 10:25 AM, hsy...@gmail.com > wrote: > > > Thanks Guozhang, > > > > Maybe I should give a few words about what I'm going to achieve wi

Re: Questions about new consumer API

elector will close idle ones after some > timeout. So in worst case it will only maintain N+1 sockets in total for N > Kafka brokers at one time. > > Guozhang > > On Mon, Nov 16, 2015 at 4:22 PM, hsy...@gmail.com > wrote: > > > The new consumer API looks good. If

Re: build error

And I couldn't find wrapper jar files under the gradle folder https://github.com/apache/kafka/tree/0.9.0/gradle On Mon, Nov 16, 2015 at 10:17 PM, Guozhang Wang wrote: > Did you just use "./gradlew build" in root directory? > > Guozhang > > On Mon, Nov 16, 201

Re: build error

I got main class not found error. So I installed gradle 2.5 and run gradle build (not the wrapper) On Mon, Nov 16, 2015 at 10:17 PM, Guozhang Wang wrote: > Did you just use "./gradlew build" in root directory? > > Guozhang > > On Mon, Nov 16, 2015 at 6:41 PM,

Re: build error

cute a > command like the following: > gradle clients:build > > Thanks, > Grant > > On Mon, Nov 16, 2015 at 6:33 PM, Guozhang Wang > wrote: > > > Siyuan, > > > > Which command did you use to build? > > > > Guozhang > > > > On Mon

Questions about new consumer API

The new consumer API looks good. If I understand it correctly you can use it like simple consumer or high-level consumer. But I have couple questions about it's internal implementation First of all does the consumer have any internal fetcher threads like high-level consumer? When you assign multi

build error

I got a build error on both trunk and 0.9.0 branch > docs/producer_config.html (No such file or directory) Do I miss anything before build Thanks, Siyuan

Given brokers, is it able to know all the zookeepers that brokers connect to

Hi Guys, If I know the brokers. Is there a way to know the zookeeper host from broker list? Thanks! Siyuan

Re: Questions about new consumer API

> Yes, it will. > > 4. Will simple consumer api and high-level consumer api still be supported? > > Over time, we will phase out the current high-level and simple consumer > since the > 0.9 API supports both. > > Thanks, > Neha > > On Tue, Dec 2, 2014 at 12:07 PM,

Questions about new consumer API

Hi guys, I'm interested in the new Consumer API. http://people.apache.org/~nehanarkhede/kafka-0.9-consumer-javadoc/doc/ I have couple of question. 1. In this doc it says kafka consumer will automatically do load balance. Is it based on throughtput or same as what we have now balance the cardinali

Is there a plan to build a ubiquitous web service API to manage the kafka cluster

Hi guys, Nowadays, all kafka administration work (add, tear down node, topic management, throughput monitor) are done by various different tool talk to brokers, zookeeper etc. Is there a plan for core team to build a central universal server providing webservice API to do all the admin work? Best

Re: log4j dir?

g insists > on using 'kafka/logs', so I create a soft link from there to the desired > destination directory: > > # kafka scripts hard-code the logs dir, so point that path to where we want > the logs to be. > ln -s $STREAM_BUFFER_LOGS_DIR kafka/logs > > -Ben > >

Re: log4j dir?

Anyone has any idea how do I config the log4j file dir? On Thu, Nov 13, 2014 at 4:58 PM, hsy...@gmail.com wrote: > Hi guys, > > Just notice kafka.logs.dir in log4j.properties doesn't take effect > > It's always set to *$base_dir/logs* in kafka-run-class.sh &

log4j dir?

Hi guys, Just notice kafka.logs.dir in log4j.properties doesn't take effect It's always set to *$base_dir/logs* in kafka-run-class.sh LOG_DIR=$base_dir/logs KAFKA_LOG4J_OPTS="-Dkafka.logs.dir=$LOG_DIR $KAFKA_LOG4J_OPTS" Best, Siyuan

Create topic programmatically

Hi guys, Besides TopicCommand, which I believe is not provided to create topic programmatically, is there any other way to automate creating topic in code? Thanks! Best, Siyuan

Re: Kafka on yarn

gt; solutions. If there is interest I could certainly list the relevant JIRA's > along with some additional JIRA's > > required IMO. > > > > Thanks > > Kam > > > > > > On Wednesday, July 23, 2014 2:37 PM, "hsy...@gmail.com"

Kafka on yarn

Hi guys, Kafka is getting more and more popular and in most cases people run kafka as long-term service in the cluster. Is there a discussion of running kafka on yarn cluster which we can utilize the convenient configuration/resource management and HA. I think there is a big potential and require

Re: Interested in contributing to Kafka?

Is there a scala API doc for the entire kafka library? On Wed, Jul 16, 2014 at 5:34 PM, hsy...@gmail.com wrote: > Hi Jay, > > I would like to take a look at the code base and maybe start working on > some jiras. > > Best, > Siyuan > > > On Wed, Jul 16, 201

Re: Interested in contributing to Kafka?

Hi Jay, I would like to take a look at the code base and maybe start working on some jiras. Best, Siyuan On Wed, Jul 16, 2014 at 3:09 PM, Jay Kreps wrote: > Hey All, > > A number of people have been submitting really nice patches recently. > > If you are interested in contributing and are loo

Re: Too Many Open Files Broker Error

I have the same problem. I didn't dig deeper but I saw this happen when I launch kafka in daemon mode. I found the daemon mode is just launch kafka with nohup. Not quite clear why this happen. On Wed, Jul 9, 2014 at 9:59 AM, Lung, Paul wrote: > Yup. In fact, I just ran the test program again wh

Re: Help is processing huge data through Kafka-storm cluster

ue, Jun 17, 2014 at 10:55 AM, Robert Rodgers >> wrote: >> >> > we have been experimenting with Samza which is also worth a look. It's >> > basically a topic-to-topic node on Yarn. >> > >> > >> > >> > On Jun 17, 201

Re: delete topic ?

I'm using 0.8.1.1 I use DeleteTopicCommand to delete topic args[0] = "--topic"; args[1] = the topic you want to delete args[2] = "--zookeeper"; args[3] = kafkaZookeepers; DeleteTopicCommand.main(args); You can write your own script to delete the topic, I guess. And I think it o

Re: Help is processing huge data through Kafka-storm cluster

Hi Shaikh, I heard some throughput bottleneck of storm. It cannot really scale up with kafka. I recommend you to try DataTorrent platform(https://www.datatorrent.com/) The platform itself is not open-source but it has a open-source library ( https://github.com/DataTorrent/Malhar) which contains a

How to get last message

Is there a way to get the last message of a partition for a given topic?

Async producer callback?

Hi guys, So far, is there a way to track the asyn producer callback. My requirement is basically if all nodes of the topic goes down, can I pause the producer and after the broker comes back online, continue to produce from the failure point? Best, Siyuan

Is there a way to delete partition at runtime?

Hi guys, I found there is a tool to add partition on the fly. My question is, is there a way to delete a partition at runtime? Thanks! Best, Siyuan

Re: kafka_2.8.0/0.8.0 pom seems invalid

Open Source Security LLC > http://www.stealth.ly > Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop> > / > > > On Wed, Dec 4, 2013 at 4:48 PM, hsy...@gmail.com wrote: > > > Hi All, I was trying

kafka_2.8.0/0.8.0 pom seems invalid

Hi All, I was trying to upgrade the kafka to 0.8 but I get an empty jar file for org.apache.kafka kafka_2.8.0 0.8.0 However org.apache.kafka kafka_2.8.2 0.8.0 is good for me. BTW from the download page I can only see kafka_2.8.0_0.8.0. Where can I download the scala

Re: Consuming from a replica

What I did for my project is I have a thread send metadata request to a random broker and monitor the metadata change periodically. The good thing is, to my knowledge, any broker in the cluster know the metadata for all the topics served in this cluster. Another options is you can always query zook

Re: Producer reaches a max of 7Mbps

I think the max 50Mbps is almost the disk bottleneck My guess is IO is the bottle neck for kafka if you set to same type(async without ack) I got throughput at about 30Mb Try to increase if you don't care about latency very much log.flush.interval.messages=1 log.flush.interval.ms=3000 On Tue

High-level consumer load-balancing problem

Hi, I have questions about the load balancing of kafka high-level consumer Suppose I have 4 partition And the producer throughput to these 4 partitions are like this 01 23 10MB/s 10MB/s 1MB/s1MB/s 1kMsg/s,10kMs

Re: will this cause message loss?

Also if you use HEAD, you can create more partitions at runtime, you just need dynamic partitioner class I think On Thu, Nov 14, 2013 at 7:23 AM, Neha Narkhede wrote: > There is no way to delete topics in Kafka yet. You can add partitions to > existing topics, but you may have to use 0.8 HEAD si

A problem of fault-tolerant high-level consumer group

I'm working on some fault-tolerant consumer group. The idea is this, to maximize the throughput of kafka. I request the metadata from broker and create #{num of partition} consumers for each topic and distribute them on different nodes. Moreover, there is mechanism to detect fail of any node and re

Re: pom warning

Consultant > Big Data Open Source Security LLC > http://www.stealth.ly > Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop> > ****/ > > > On Tue, Nov 12, 2013 at 2:56 PM, hsy...@gmail.com > wrote: > > &g

Re: High level consumer Blocked when there is still message in topic

Since you have a cluster, why not distribute the consumers in different nodes instead of threads. I think that's the only way to scale up with kafka. Question here: if there are more and more high-level consumers, is there a bottleneck on the zookeeper? On Tue, Nov 12, 2013 at 9:27 PM, Jun Rao w

Re: Kafka cluster with lots of topics

I didn't see any auto leader election for adding nodes. The data are still skewed on the old nodes. You have to force it by running script? On Wed, Nov 13, 2013 at 6:41 AM, Neha Narkhede wrote: > At those many topics, zookeeper will be the main bottleneck. Leader > election process will take ver

pom warning

Hi guys, When I built my project using maven I got WARNING [WARNING] The POM for org.apache.kafka:kafka_2.8.0:jar:0.8.0-beta1 is invalid, transitive dependencies (if any) will not be available: 1 problem was encountered while building the effective model And I looked at the kafka_2.8.0-0.8.0-bet

Detail description of metrcs value?

Hi guys, Is there a detail document about the attributes and objectnames about the mbeans? For example, what does attribute "MeanRate" of Object "MessagesPerSec" mean? It's the mean value of last 1 sec/1min ? http://kafka.apache.org/documentation.html#monitoring only have a little information ab

Re: Is there a way to add partition to a particular topic

I mean I assume the messages not yet consumed before delete-topic will be delivered before you create same topic, correct? On Fri, Nov 8, 2013 at 6:30 PM, hsy...@gmail.com wrote: > It's in the branch, cool, I'll wait for it's release. actually I find I > can use ./kafk

Re: Is there a way to add partition to a particular topic

lease check the add-partition tool: > > > https://cwiki.apache.org/confluence/display/KAFKA/Replication+tools#Replicationtools-5.AddPartitionTool > > Guozhang > > > On Fri, Nov 8, 2013 at 5:32 PM, hsy...@gmail.com wrote: > > > Hi guys, since kafka is able to add new broker

Is there a way to add partition to a particular topic

Hi guys, since kafka is able to add new broker into the cluster at runtime, I'm wondering is there a way to add new partition for a specific topic at run time? If not what will you do if you want to add more partition to a topic? Thanks!

Throughput Questions

Hi guys, I have some throughput questions. I try to test the throughput using both the High Level Consumer and Simple Consumer example from the document. But I get much lower throughput of simple consumer than the high level consumer. I run the test in the cluster and I'm sure I distribute the le

Re: partition reassignment

There is a ticket for auto-rebalancing, hopefully they'll do auto redistribution soon https://issues.apache.org/jira/browse/KAFKA-930 On Wed, Oct 16, 2013 at 12:29 AM, Kane Kane wrote: > Yes, thanks, looks like that's what i need, do you know why it tends to > choose the leader for all partitio

Re: Is there a programmatic way to create topic

his is probably something we can improve > in the forthcoming releases. > > Thanks, > Neha > > > On Mon, Oct 14, 2013 at 3:02 PM, hsy...@gmail.com > wrote: > > > Hi kafka, > > > > Is there a programmatic way to create topic. > > > > > http:/

Is there a programmatic way to create topic

Hi kafka, Is there a programmatic way to create topic. http://stackoverflow.com/questions/16946778/how-can-we-create-a-topic-in-kafka-from-the-ide-using-api/18480684#18480684 is too hacky, plus it's not a sync function. I'm asking this because I'm writing a test case which will start kafka ser

KafkaStream bug?

I found some weird behavior, I follow the exact code example for HighlevelConsumer https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Group+Example# but add one debug line here " public void run() { ConsumerIterator it = m_stream.iterator(); while (it.hasNext()){ * //

Re: Question about auto-rebalancing

lancing logic, you can find it at > > > > > https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-CanIpredictthere > >sultsoftheconsumerrebabalance%3F > > > >Guozhang > > > > > >On Fri, Oct 11, 2013 at 11:06 AM, hsy...@gmail.com > >wrote: >

Re: Is there a way to pull out kafka metadata from zookeeper?

ce > the > > metadata request also returns the list of brokers in a cluster. The > reason > > this is better than reading from zookeeper is because the same operation > > would require multiple zookeeper roundtrips, instead of a single > > TopicMetadataRequest ro

Re: Is there a way to pull out kafka metadata from zookeeper?

okeeper roundtrips, instead of a single > TopicMetadataRequest roundtrip to some kafka broker. > > Thanks, > Neha > > > On Fri, Oct 11, 2013 at 11:30 AM, hsy...@gmail.com > wrote: > > > Thanks guys! > > But I feel weird. Assume I have 20 brokers for 10 different top

Re: Is there a way to pull out kafka metadata from zookeeper?

al IP or load > balancer, you need to round robin once through all the brokers before > failing a TopicMetadataRequest. So unless all the brokers in your cluster > are down, this should not be a problem. > > Thanks, > Neha > > > On Thu, Oct 10, 2013 at 10:50 PM,

Re: Question about auto-rebalancing

Hi Jun, Thanks for your reply, but in a real cluster, one broker could serve different topics and different partitions, the simple consumer only has knowledge of brokers that are available but it has no knowledge to decide which broker is best to pick up to consume messages. If you don't choose c

Question about auto-rebalancing

Hi guys, Here is a case I observed, I have a single-node 3 broker instance cluster. I created 1 topic with 2 partitions and 2 replica for each partition. The initial distribution is like this topic1/partition0 ->(broker0, broker2) topic1/partition1 ->(broker1,broker2). So broker0 is leader broker

Is there a way to pull out kafka metadata from zookeeper?

Hi guys, I'm trying to maintain a bunch of simple kafka consumer to consume messages from brokers. I know there is a way to send TopicMetadataRequest to broker and get the response from the broker. But you have to specify the broker list to query the information. But broker might not be available