Fwd: Messages from producer are immediately going to /tmp/logs in kafka

Hi, In kafka-0.8 there are three important properties given for log.flush.interval.messages=1 log.flush.interval.ms=90 log.flush.scheduler.interval.ms=90 I have set the above properties as I have mentioned above.Then I started Kafka Console Producer given with kafka bundle-0.8 and

Question about auto-rebalancing

Hi guys, Here is a case I observed, I have a single-node 3 broker instance cluster. I created 1 topic with 2 partitions and 2 replica for each partition. The initial distribution is like this topic1/partition0 ->(broker0, broker2) topic1/partition1 ->(broker1,broker2). So broker0 is leader broker

Re: Kafka server error

Ok, finally, I find the problem, in the /etc/hosts, I set the hostname as 127.0.0.1 ip-10-0-0-108 It seems cannot use its internal hostname, and caused the connection to close. Thanks guys, if I encounter any further issue, I will let you know. 2013/10/11 Jiang Jacky > Hi, Since I changed the p

Is there a way to pull out kafka metadata from zookeeper?

Hi guys, I'm trying to maintain a bunch of simple kafka consumer to consume messages from brokers. I know there is a way to send TopicMetadataRequest to broker and get the response from the broker. But you have to specify the broker list to query the information. But broker might not be available

Re: kafka.common.UnknownTopicOrPartitionException

Yes, but I have commented the the option in log4J. properties Thanks, Snehalata -- From: "Monika Garg" Sent: Friday, October 11, 2013 11:01 AM To: Subject: Re: kafka.common.UnknownTopicOrPartitionException Hi Snehalata state-change log are auto

Re: Kafka server error

Hi, Since I changed the port to 9093, it seems works, I can produce message, and another node can consume that. But there is still exception [2013-10-11 05:32:42,706] ERROR [KafkaApi-2] Error while fetching metadata for partition [my-replicated-topic,0] (kafka.server.KafkaApis) kafka.common.Replic

Re: kafka.common.UnknownTopicOrPartitionException

Hi Snehalata state-change log are automatically created if you are using kafka-0.8 version. On Wed, Oct 9, 2013 at 10:42 AM, Snehalata Nagaje < snehalata.nag...@harbingergroup.com> wrote: > Actually state-change log setting is not on. So could not find the > state-change log. > > ---

Re: Kafka server error

no, it is not reproducible. Thank you. 2013/10/11 Jun Rao > I meant the broker crash. Is that reproducible? > > Thanks, > > Jun > > > On Thu, Oct 10, 2013 at 10:10 PM, Jiang Jacky wrote: > > > Yes, everytime, I send message, it throws that error. > > > > > > 2013/10/11 Jun Rao > > > > > Is th

Re: Kafka server error

I meant the broker crash. Is that reproducible? Thanks, Jun On Thu, Oct 10, 2013 at 10:10 PM, Jiang Jacky wrote: > Yes, everytime, I send message, it throws that error. > > > 2013/10/11 Jun Rao > > > Is that reproducible? > > > > Thanks, > > > > Jun > > > > > > On Thu, Oct 10, 2013 at 9:54 P

Re: Kafka server error

Sorry, even I don't send message, it shows up that message about every 1 minute 2013/10/11 Jiang Jacky > Yes, everytime, I send message, it throws that error. > > > 2013/10/11 Jun Rao > >> Is that reproducible? >> >> Thanks, >> >> Jun >> >> >> On Thu, Oct 10, 2013 at 9:54 PM, Jiang Jacky wrot

Re: Kafka server error

Yes, everytime, I send message, it throws that error. 2013/10/11 Jun Rao > Is that reproducible? > > Thanks, > > Jun > > > On Thu, Oct 10, 2013 at 9:54 PM, Jiang Jacky wrote: > > > Yes, it just says "INFO Reconnect due to socket error" > > But why and how come it comes? my zookeeper and storm

Re: Kafka cannot produce message

under logs/. Thanks, Jun On Thu, Oct 10, 2013 at 9:56 PM, Jiang Jacky wrote: > Thanks Jun Rao > Where can I find those logs? > > > 2013/10/11 Jun Rao > > > Any error in the controller and state-change log? > > > > Thanks, > > > > Jun > > > > > > On Thu, Oct 10, 2013 at 9:38 PM, Jiang Jacky

Re: Kafka server error

Is that reproducible? Thanks, Jun On Thu, Oct 10, 2013 at 9:54 PM, Jiang Jacky wrote: > Yes, it just says "INFO Reconnect due to socket error" > But why and how come it comes? my zookeeper and storm have no any problem > to collaborate each other. > > > 2013/10/11 Jun Rao > > > The log you p

Re: Kafka cannot produce message

Thanks Jun Rao Where can I find those logs? 2013/10/11 Jun Rao > Any error in the controller and state-change log? > > Thanks, > > Jun > > > On Thu, Oct 10, 2013 at 9:38 PM, Jiang Jacky wrote: > > > Hi, Aniket > > Thanks for your reply > > I checked the log from /tmp/kafka-logs/ > > There are

Re: Hi, Can anyone tell me why I cannot produce any message to remote kafka server from my local machine

Hi, Thanks Joe I have disabled the firewall, but I did not tried that again, I changed another way by using Netty sending the socket package to kafka server, then it will produce in the local. But recently I suddenly ran into below issue, can you have a look? https://mail.google.com/mail/ca/u/0/#la

Re: Kafka server error

Yes, it just says "INFO Reconnect due to socket error" But why and how come it comes? my zookeeper and storm have no any problem to collaborate each other. 2013/10/11 Jun Rao > The log you posted for the second broker didn't say why it crashed. Is that > all you got? > > Thanks, > > Jun > > > O

Re: Kafka cannot produce message

Any error in the controller and state-change log? Thanks, Jun On Thu, Oct 10, 2013 at 9:38 PM, Jiang Jacky wrote: > Hi, Aniket > Thanks for your reply > I checked the log from /tmp/kafka-logs/ > There are some empty files, likes .index, > .log. I cannot

Re: Kafka server error

The log you posted for the second broker didn't say why it crashed. Is that all you got? Thanks, Jun On Thu, Oct 10, 2013 at 9:22 PM, Jiang Jacky wrote: > *Hi, Guys,* > *I am currently running into the kafka server issue. * > *I have a 5 nodes cluster and zookeeper running without any problem

Re: Kafka cannot produce message

Hi, Aniket Thanks for your reply I checked the log from /tmp/kafka-logs/ There are some empty files, likes .index, .log. I cannot find any useful information there. However, i found some other interesting things, I have posted here https://mail.google.com/mai

Kafka server error

*Hi, Guys,* *I am currently running into the kafka server issue. * *I have a 5 nodes cluster and zookeeper running without any problem. when I manually boot each node by using* "*JMX_PORT=9997 bin/kafka-server-start.sh config/server-x.properties &*" command. *The scenario is:* *Then, first node, i

Re: Kafka cannot produce message

Hi Jiang Just thinking out loud, it could be because Kafka can't connect to Zookeeper. Can you check Kafka logs to validate the same? On 11 October 2013 08:22, Jiang Jacky wrote: > Hi, Everyone, I use the larger memory instance to replace that small one, > for now, I can create the topic, but

Re: Hi, Can anyone tell me why I cannot produce any message to remote kafka server from my local machine

another thing to note is that your brokers may be running on port 6667 instead of 9092. it appears that the default recently changed from 9092 to 6667 so make sure if you have firewall rules for 9092 that you specifically set the port setting in your configuration to what you expect. -Joe Sent

Re: Hi, Can anyone tell me why I cannot produce any message to remote kafka server from my local machine

it may be that the broker is reporting its internal ip address to zookeeper which then may interfere with your ability to submit the the broker with a producer that isn't on AWS's network. you can make the broker report the public dns name instead of the internal host name which i believe it re

Re: Kafka cannot produce message

Hi, Everyone, I use the larger memory instance to replace that small one, for now, I can create the topic, but when I try to produce message, it still gives the same error. Its so weird. can any one help me? Thanks 2013/10/10 Jiang Jacky > Another thing need is my instance comes with very small

Re: Hi, Can anyone tell me why I cannot produce any message to remote kafka server from my local machine

did anyone experience same issue? can you share with me? Thanks a lot. 2013/10/7 Jun Rao > Kafka port is documented in > http://kafka.apache.org/documentation.html#brokerconfigs > > Thanks, > > Jun > > > On Sat, Oct 5, 2013 at 12:05 PM, Jiang Jacky wrote: > > > Hi, I tried to setup the host.na

Re: Kafka cannot produce message

Another thing need is my instance comes with very small memory, just 683M, I am not sure if it is related. I will try to use 8G memory instance. 2013/10/10 Jiang Jacky > yes, even I cannot create new topic, same error, leadernotavailable. > My zookeeper and storm all are working very well, exce

Re: Kafka cannot produce message

yes, even I cannot create new topic, same error, leadernotavailable. My zookeeper and storm all are working very well, except of Kafka. 2013/10/10 Neha Narkhede > Are you trying to produce these messages to a newly created topic? > > > On Thu, Oct 10, 2013 at 6:15 PM, Jiang Jacky wrote: > > >

Re: Kafka cannot produce message

Are you trying to produce these messages to a newly created topic? On Thu, Oct 10, 2013 at 6:15 PM, Jiang Jacky wrote: > can anyone help me? Thanks a lot > > > 2013/10/10 Jiang Jacky > > > Hi, I run into kafka issue recently, when I try to produce a message, it > > gives me follow error > > >

Re: Kafka cannot produce message

can anyone help me? Thanks a lot 2013/10/10 Jiang Jacky > Hi, I run into kafka issue recently, when I try to produce a message, it > gives me follow error > > [2013-10-11 00:30:26,740] INFO Fetching metadata from broker > id:0,host:localhost,port:9092 with correlation id 47 for 1 topic(s) > Set

Kafka cannot produce message

Hi, I run into kafka issue recently, when I try to produce a message, it gives me follow error [2013-10-11 00:30:26,740] INFO Fetching metadata from broker id:0,host:localhost,port:9092 with correlation id 47 for 1 topic(s) Set(my-replicated-topic) (kafka.client.ClientUtils$) [2013-10-11 00:30:26,

Re: Problem when follows the "chapter 1.3 quick start" in kafka documentation

Hello, Could you try to check your zookeeper log to see if the topic has been successfully created and the broker has been selected by the controller to be the leader of this topic? The log entry should be something like Broker %d handling LeaderAndIsr request correlation id ... BTW from your lo

Problem when follows the "chapter 1.3 quick start" in kafka documentation

Hi, Following the _kafka documentation_ , I succeeded in Step 1: Download the code, Step 2: Start the server and Step 3: Create a topic in three different terminals. While I tried Step 4: Send some messages, the problem happened, the prod

Re: Anyone running kafka with a single broker in production? what about only 8GB ram?

A 10/10/2013, às 23:14, S Ahmed escreveu: > Is anyone out there running a single broker kafka setup? > > How about with only 8 GB RAM? > > I'm looking at one of the better dedicated server prodivers, and a 8GB > server is pretty much what I want to spend at the moment, would it make > sense goi

Anyone running kafka with a single broker in production? what about only 8GB ram?

Is anyone out there running a single broker kafka setup? How about with only 8 GB RAM? I'm looking at one of the better dedicated server prodivers, and a 8GB server is pretty much what I want to spend at the moment, would it make sense going this route? This same server would also potentially be

Re: Topic Fetch metadata in producer

Do you see the following warn messages on the producer ? Fetching topic metadata with correlation id %d for topics [%s] from broker [%s] failed fetching topic metadata for topics [%s] from broker [%s] failed On the server side - 1. could you enable DEBUG on kafka.server.KafkaApis and see if y

Re: PHP client for Kafka .8

Thanks guys. Since we have some time constraints we are going with a Java based REST endpoint within our already existing API to act as a proxy for the message. Since it's a low volume of messages coming from our PHP side to kafka we are willing to accept the additional overhead of doubling the ne

Re: Best way to start / stop / restart a Consumer?

Each time we create a new consumer connector, we assign a random consumer id by default. You can try setting "consumer.id" to use a fixed consumer id. In any case, we probably should deregister those beans when shutting down the connector. Could you file a jira? Thanks, Jun On Thu, Oct 10, 2013

Re: Best way to start / stop / restart a Consumer?

It's in 0.8. The JMX names are not exactly the same, but I see 2 beans with similar names, something like: "my_consumer_group-my_consumer-mytopic-my_consumer_hostname.thing.com-1381416134138-3573c8bf-7-FetchQueueSize" -- Tanguy 2013/10/10 Jun Rao > Is that in 0.7 or 0.8? JMX won't allow a

Re: Kafka startup, no broker partitions consumed by consumer thread

Could you try the latest code in the 0.8 branch? We recently fixed a consumer side issue (KAFKA-1030) that seems related. Thanks, Jun On Thu, Oct 10, 2013 at 6:36 AM, Viktor Kolodrevskiy < viktor.kolodrevs...@gmail.com> wrote: > On our current project we want to have a single shell script that:

Re: Best way to start / stop / restart a Consumer?

Is that in 0.7 or 0.8? JMX won't allow a bean with the same name to be registered twice. Do you see 2 beans with similar names? What are the exact bean names? Thanks, Jun On Thu, Oct 10, 2013 at 2:42 AM, Tanguy tlrx wrote: > Hi all, > > In our application, we are currently starting a Kafka Co

Kafka startup, no broker partitions consumed by consumer thread

On our current project we want to have a single shell script that: first starts kafka with zookeeper and second strarts tomcat server with our webapp. We run kafka and tomcat on single machine, use kafka 0.8 We set num.partitions=4 in server.properties. In webapp we start 4 threads that do kafka.co

Best way to start / stop / restart a Consumer?

Hi all, In our application, we are currently starting a Kafka Consumer with the following lines of code: connector = Consumer.createJavaConsumerConnector(consumerConfig); streams = connector .createMessageStreams(map); Then, each KafkaStream is processed in a dedicated thread per topic and parti

Re: Kafka in public maven repositories

Great ! I didn't check, recently. :-) Thanks I hope also to see 0.7.2. On Thu, Oct 10, 2013 at 5:05 PM, Bruno D. Rodrigues < bruno.rodrig...@litux.org> wrote: > A 10/10/2013, às 08:58, Cheolwoo Choi escreveu: > > > Why we can't find Kafka in public maven repositories ? > > > > What's obstacle

Re: Kafka in public maven repositories

A 10/10/2013, às 08:58, Cheolwoo Choi escreveu: > Why we can't find Kafka in public maven repositories ? > > What's obstacle to publish into public maven repositories ? > > > Thanks, > Choi what do you mean? http://search.maven.org/#search%7Cga%7C1%7Ckafka signature.asc Description: Mess

Kafka in public maven repositories

Why we can't find Kafka in public maven repositories ? What's obstacle to publish into public maven repositories ? Thanks, Choi