Re: MirrorMaker doesn't exit on exception?

This seems to be a problem. MirrorMaker waits for all threads to finish before shutting down. In your case, only one thread died. Are you configuring the producer in sync mode? One potential solution is to run the producer in async mode. In this mode, the mirror maker thread won't be killed when th

Re: Subscribe me

You can follow the instructions in http://kafka.apache.org/contact.html Thanks, Jun On Tue, Jul 2, 2013 at 3:01 PM, Nandigam, Sujitha wrote: > > > > "This message (including any attachments) is intended only for the use of > the individual or entity to which it is addressed, and may contain >

Re: how to keep the same partition when using mirror maker

One of the issues is that in 0.7, the partitioning key is not stored on the broker. So, mirror maker won't know what key to use for partitioning. In 0.8, the partitioning key will be stored on the broker. Once kafka-957 is resolved, mirror maker will be able to partition messages based on the same

Subscribe me

"This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed, and may contain information that is non-public, proprietary, privileged, confidential and exempt from disclosure under applicable law or may be constituted as attorney

MirrorMaker doesn't exit on exception?

Hello, I've noticed that when a broker in the target cluster restarts (i.e. immediately comes back online), the MirrorMaker runs into a FATAL exception but the process hangs indefinitely instead of exiting. This seems to me like undesirable behavior but I'm not sure that it's a bug. I'd like for t

how to keep the same partition when using mirror maker

HI guys, We are using kafka0.7.2, in our cluster, we use customized partition functions in producer. Say, we compute partition id with user id in our log. But when we use mirror maker to pull log, we find that mirror maker uses random partition to push log into destination brokers. In my mind, w

Kafka/Hadoop consumers and producers

We currently have a contrib package for consuming and producing messages from mapreduce ( https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tree;f=contrib;h=e53e1fb34893e733b10ff27e79e6a1dcbb8d7ab0;hb=HEAD ). We keep running into problems (e.g. KAFKA-946) that are basically due to the fact tha

Re: CallbackHandler in Kafka 0.8

Callback handlers are no longer supported in 0.8. Can you go into why the filtering needs to be done at this stage as opposed to before actually sending to the producer? Thanks, Joel On Tue, Jul 2, 2013 at 10:41 AM, Nitin Supekar wrote: > Hello- > >Is CallbackHandler supported in Kafka 0.8

Re: partition numbers and mirrormaker

You can override the default number of partitions on the source and target cluster to N. (num.partitions in the broker config). If you need a higher (or lower) number of partitions for specific topics you can override that on a per-topic basis (again, the config can be kept identical between the so

partition numbers and mirrormaker

Hi I asked this question previously and didn't get an answer. If a topic has N partitions on source cluster, after being mirrored, on the destination cluster it has only one partition. To avoid this issue, before mirroring, I create the same topic with N partitions on the Destination cluster. B

Re: Partitions and highlevel consumers

This is exactly right. Partitions is configurable so set it to some reasonable upper bound on the concurrency you desire. To give further examples, let's say you have 5 threads: - if you have 2 partitions only two threads will get data - if you have 100 partitions each thread will get 20 partitions

Re: Kafka User Group Meeting Jun. 27

Erp, actually I don't see a way to get the recording there, nm. -Jay On Tue, Jul 2, 2013 at 12:47 PM, Jay Kreps wrote: > The recording for the user group talks is available here: > http://www.ustream.tv/linkedin-events > > -Jay > > > On Wed, Jun 26, 2013 at 8:22 AM, Jun Rao wrote: > >> Hi, Ev

Re: Kafka User Group Meeting Jun. 27

The recording for the user group talks is available here: http://www.ustream.tv/linkedin-events -Jay On Wed, Jun 26, 2013 at 8:22 AM, Jun Rao wrote: > Hi, Everyone, > > We have finalized our agenda of the meetup Thursday evening, with Speakers > from LinkedIn, RichRelevance, Netflix and Square

Re: Partitions and highlevel consumers

Hi, am a also new to Kafka but let me explain my understanding which someone with more knowledge can confirm.  There are actually 2 scenarios: 1.  If all 5 of your consumers are in different "consumer groups" then this will behave like a JMS topic where all 5 of your consumers will each get a co

Re: check out 0.8 beta from git repo

The current 0.8.0-beta1 release is tagged so you can git clone and then git checkout 0.8.0-beta1 for the current release On Tue, Jul 2, 2013 at 2:56 PM, Yu, Libo wrote: > Hi, > > I want to check out beta version from git repo. > Which branch should I use? Thank.s > origin/0.8 > origin/0.8.

check out 0.8 beta from git repo

Hi, I want to check out beta version from git repo. Which branch should I use? Thank.s origin/0.8 origin/0.8.0-beta1-candidate1 Regards, Libo

Partitions and highlevel consumers

Hi guys, we are starting with kafka in our project. We are using version 0.8. I come from a traditional JMS MoM architecture, and some things are new to me. One thing that I'm not getting is the mapping between partitions and number of threads. On a single consumer I can see the relationship but w

CallbackHandler in Kafka 0.8

Hello- Is CallbackHandler supported in Kafka 0.8 for async producers? If yes, can I use it to alter the batched messages before they are pushed to broker? For example, I may want to delete some of the messages in the batch based on some business logic in my application? If no, is there any al

Re: Kafka User Group Meeting Jun. 27

Was this recorded by any chance? On Wed, Jun 26, 2013 at 11:22 AM, Jun Rao wrote: > Hi, Everyone, > > We have finalized our agenda of the meetup Thursday evening, with Speakers > from LinkedIn, RichRelevance, Netflix and Square. Please RSVP to the meetup > link below. For remote people, we will

Re: auto.offset.reset in kafka 0.8

Martin, "other" is not a valid value for auto.offset.reset. This is explained in the exception itself "kafka.common.InvalidConfigException: Wrong value other of auto.offset.reset in ConsumerConfig; Valid values are smallest and largest". Thanks, Jun On Mon, Jul 1, 2013 at 9:28 PM, Martin Eigen

Re: Kafka Metrics in 0.8

In 0.8, there are more mbeans than 0.7. Thanks, Jun On Mon, Jul 1, 2013 at 9:06 PM, Hanish Bansal < hanish.bansal.agar...@gmail.com> wrote: > Okay i'll try for same. > > Also want to know that in kafka-0.7 there is single MBean - > SocketServerStats which provides all kafka Metrics. But in Kaf

Re: Loading kafka in Eclipse IDE

Thanks for sharing. Could you add this to the wiki? Jun On Mon, Jul 1, 2013 at 9:01 PM, Tejas Patil wrote: > Hi, > > [0] is an old wiki entry for getting Scala setup for development. After all > huffs and puffs, I gave up getting it loaded in Intellij IDEA. However, I > could get it setup with