Re: PID files for monitoring two different instances of Kafka on same machine

Hi Jun, Thanks a lot for your reply. You can figure out the instance from the > config dir in the command line. > What does this mean? Can you explain a bit how can it be figured out? Regards, Hanish > On Thu, Jul 4, 2013 at 7:20 AM, Hanish Bansal < > hanish.bansal.agar...@gmail.com> wrote:

Re: Kafka rolling upgrade

Kafka 0.8 is not backward compatible with 0.7. So you can't do inplace upgrade. We do have a migration tool. See https://cwiki.apache.org/confluence/display/KAFKA/Migrating+from+0.7+to+0.8 Thanks, Jun On Thu, Jul 4, 2013 at 7:53 PM, Hanish Bansal < hanish.bansal.agar...@gmail.com> wrote: > Hi

Re: Changing the number of partitions after a topic is created

Currently, once a topic is created, the number of partitions can't be changed. We are working on a tool to allow that. For now, you will have to either use a new topic with more partitions or if you don't care about messages being consumed in order, you can feed the consumed messages to a separate

Re: Handling error

We don't have that documented right now. The easiest thing is to catch all exceptions. Thanks, Jun On Thu, Jul 4, 2013 at 8:34 AM, Calvin Lei wrote: > Hi, is there any documents for exceptions thrown by Kafka? I am trying to > capture errors coming from Kafka. For example, when a producer fai

Re: PID files for monitoring two different instances of Kafka on same machine

We don't have PID files now. You can figure out the instance from the config dir in the command line. Thanks, Jun On Thu, Jul 4, 2013 at 7:20 AM, Hanish Bansal < hanish.bansal.agar...@gmail.com> wrote: > Hi all, > > > I was just trying to create a monitoring service of my own for Kafka. I > wa

Kafka rolling upgrade

Hi All, I am using *Kafka-0.7.2 *in distributed mode. Now i want to upgrade Kafka version to recent stable release-*0.8 Beta *with rolling upgrade without * *any downtime. I tried to upgrade one kafka node but after starting that node i am getting Null Pointer Exception. This may be due *data f

Re: Changing the number of partitions after a topic is created

On Thu, Jul 4, 2013 at 12:06 PM, Calvin Lei wrote: > Thanks Phillip. I used the kafka-topic.sh to create the topic to have 1 > partition. Would the changing the server properties still work in this case? I'm not familiar with that script, but server.properties is what matters. Change that and it

Re: Kafka/Hadoop consumers and producers

Vadim, The advantages of Camus compared to the contrib consumer are the following (but perhaps I'm forgetting some) : - The ability to fetch all/many topics in one job (Map Reduce can otherwise introduce a lot of overhead for small topics). - Smarter load balancing of topic partitions ac

Re: Changing the number of partitions after a topic is created

Thanks Phillip. I used the kafka-topic.sh to create the topic to have 1 partition. Would the changing the server properties still work in this case? Also this setting sounds like a global overwrite, correct? regards, Cal On Jul 4, 2013, at 11:49 AM, Philip O'Toole wrote: > If you can pause y

Re: Changing the number of partitions after a topic is created

If you can pause your Producers, simply change the partition count to 10 in the Kafka server properties file, and restart it. If you use the high-level consumer, it will automatically start draining all partitions that exist for a given topic. And our Kafka producers always write to partition -1 f

Handling error

Hi, is there any documents for exceptions thrown by Kafka? I am trying to capture errors coming from Kafka. For example, when a producer fails to connects to zookeeper because it is down. Regards, Cal

Changing the number of partitions after a topic is created

Hi I have a few topics created with 1 partition. After running the cluster for a few days, I want to increase the partition to 10 to improve the consumer throughput. I learnt that it is not supported in 0.8. What is the recommendation of improving consumer throughput after a topic is created and t

PID files for monitoring two different instances of Kafka on same machine

Hi all, I was just trying to create a monitoring service of my own for Kafka. I wanted to know if Kafka provides any sort of functionality for pid files which I can see to check if Kafka is up or not. My scenario is: I will have 2 Kafka instances running on same machine with different configur

Re: HBase upgrade

Apologise for putting query in wrong mailing-list by mistake. On Thu, Jul 4, 2013 at 2:07 PM, Hanish Bansal < hanish.bansal.agar...@gmail.com> wrote: > Hi All, > > I am using HBase in distributed mode. A number of master and regionserver > nodes are running on different machines. > > Now i want

HBase upgrade

Hi All, I am using HBase in distributed mode. A number of master and regionserver nodes are running on different machines. Now i want to upgrade HBase version to recent stable release. My main purpose is to upgrade HBase without shutting down whole cluster. I want to know if on one machine i up

RE: [ANNOUNCEMENT] Apache Kafka 0.8.0-beta1 Released

Thanks, Jun. I also checked out code from git repo. Regards, Libo -Original Message- From: Jun Rao [mailto:jun...@gmail.com] Sent: Thursday, July 04, 2013 12:43 AM To: users@kafka.apache.org Subject: Re: [ANNOUNCEMENT] Apache Kafka 0.8.0-beta1 Released git pull --tags origin git tag -

RE: partition numbers and mirrormaker

num.partition will set a number of partition for all topics. If the topics have different partition numbers, then this will not work. Regards, Libo -Original Message- From: Joel Koshy [mailto:jjkosh...@gmail.com] Sent: Wednesday, July 03, 2013 1:27 PM To: users@kafka.apache.org Subje

Re: CallbackHandler in Kafka 0.8

Thanks Joel for the response. Sure, I can achieve same by using some data structure before we push messages to Kafka. I was hoping if I can utilize any kafka capabilities specially as we are using async producer. thanks On Wed, Jul 3, 2013 at 11:18 PM, Joel Koshy wrote: > Hi Nitin, > > > > We

Re: Kafka 0.7 Quickstart Errors

Hey Jun Thanks for your prompt response. I don't really get "wipe out both Zookeeper and Kafka 0.8 data". I just changed the log data directories in both kafka and zookeeper configs and still I am getting the same error. Isn't that sufficient? What else do I need to do to wipe out the data? What d