MetadataResponse error code handling

Hi folks, I was wondering in which cases the PartitionMetadata section of a MetadataResponse [1] can contain useful information? I had been working under the assumption that when the PartitionErrorCode was 0 (NoError) then the rest of the data would be legitimate, and that when the error was non-

Re: MetadataResponse error code handling

ng a metadata request yourself and then you need to handle the > error code accordingly. > > I agree that such documentation is kind of lacking for now, and we would > better add that in the response class Java doc. > > Guozhang > > > On Tue, Nov 25, 2014 at 12:29 PM,

Re: MetadataResponse error code handling

1609 patch, it handles this error code by setting the > leader field to null. > > Guozhang > > On Wed, Nov 26, 2014 at 11:08 AM, Evan Huus wrote: > > > Hi Guozhang, thanks for the information. > > > > I am implementing/maintaining the Golang producer [1] so I do ne

Re: [kafka-clients] is RequestTimedOut (error code: 7) retryable in producer?

Hi there, As far as I know, RequestTimedOut is returned exclusively when the `Timeout` value provided in a Produce Request is exceeded. The message will likely still be committed on the local broker, but it means that some replicas may not have received it yet. I'm honestly not sure what the use c

Does RequestTimedOut Error Commit Locally?

If a producer produces a request with RequiredAcks of -1 (wait for all ISRs), and the broker returns a RequestTimedOut error, are the messages still committed locally on the leader broker? The protocol states "we will not terminate a local write" which implies to me that even when RequestTimedOut i

Re: Simple Consumer and offsets

On Thu, Feb 19, 2015 at 8:43 PM, Joel Koshy wrote: > If you are using v0 of OffsetCommit/FetchRequest then you can issue > that to any broker. For version > 0 you will need to issue it to the > coordinator. You can discover the coordinator by sending a > ConsumerMetadataRequest to any broker. >

Re: Topicmetadata response miss some partitions information sometimes

On Sun, Mar 1, 2015 at 1:46 AM, Guozhang Wang wrote: > Hi Honghai, > > 1. If a partition has no leader (i.e. all of its replicas are down) it will > become offline, and hence the metadata response will not have this > partition's info. > If I am understanding this correctly, then this is a probl

Re: Topicmetadata response miss some partitions information sometimes

data gets refreshed, and message send retry successfully. > Hence, if some of your partitions becomes offline for too long some data > will be lost on the producer side (unless you set infinite retry, of > course). > > Guozhang > > On Sun, Mar 1, 2015 at 5:25 AM, Evan Huus wrote: &g

Re: Topicmetadata response miss some partitions information sometimes

e, you can use the total number of partitions for the topic, not > the number of available partitions to compute partition id. > > Guozhang > > On Sun, Mar 1, 2015 at 6:42 PM, Evan Huus wrote: > > > My concern is more with the partitioner that determines the partition of >

Re: Topicmetadata response miss some partitions information sometimes

cer class. Could you file a JIRA? > > Guozhang > > On Sun, Mar 1, 2015 at 7:11 PM, Evan Huus wrote: > > > Which I think is my point - based on my current understanding, there is > > *no* way to find out the total number of partitions for a topic besides > > hard-codin

New Errors in 0.8.2 Protocol

Hey all, it seems that 0.8.2 has added a handful more errors to the protocol which are not yet reflected on the wiki page [1]. Specifically, [2] seems to indicate that codes 17-20 now have associated meanings. My questions are: - Which of these are exposed "publicly"? (for example, the existing er

Re: New Errors in 0.8.2 Protocol

't provide the information everyone needs fluidly enough. > > ~ Joe Stein > - - - - - - - - - - - - - - - - - > > http://www.stealth.ly > - - - - - - - - - - - - - - - - - > > On Wed, Mar 4, 2015 at 12:44 PM, Evan Huus wrote: > > > Hey all, it seems that 0.8.2 has added a handful more errors to

Fetch Request Purgatory and Mirrormaker

Hey Folks, we're running into an odd issue with mirrormaker and the fetch request purgatory on the brokers. Our setup consists of two six-node clusters (all running 0.8.2.1 on identical hw with the same config). All "normal" producing and consuming happens on cluster A. Mirrormaker has been set up

Re: Fetch Request Purgatory and Mirrormaker

Any ideas on this? It's still occurring... Is there a separate mailing list or project for mirrormaker that I could ask? Thanks, Evan On Thu, Apr 9, 2015 at 4:36 PM, Evan Huus wrote: > Hey Folks, we're running into an odd issue with mirrormaker and the fetch > request purgator

Re: Fetch Request Purgatory and Mirrormaker

e.interval.requests` and `producer.purgatory.purge.interval.requests` configuration values on cluster B from 1000 to 200, but it had no effect, which I find really weird. Thanks, Evan > Jiangjie (Becket) Qin > > On 4/14/15, 6:55 AM, "Evan Huus" wrote: > > >Any ideas on this

Re: cpu over-consumption in Kafka producer -?

Hi Victor, two points: - Based on the backtrace, you are using a very old version of Sarama. You might have better luck using a more recent stable version. - Are you setting `MaxBufferTime` in the configuration to 0 or a very small value? If so the loop will spin on that timer. Try making this val

Re: cpu over-consumption in Kafka producer -?

= 1024 > > How's it versioned? Should i just download zip file from project github? > By default go will use whatever version you originally checked out with `go get` or the like. You can use a properly versioned copy via http://gopkg.in/Shopify/sarama.v1. > On Wed, Apr 15,

Re: cpu over-consumption in Kafka producer -?

for efficiency, whatever your use case dictates. Evan On Wed, Apr 15, 2015 at 1:47 PM, Victor L wrote: > I inherited this code, it just set to 1000, what would be reasonable time: > 1000 * time.Millisecond? > > On Wed, Apr 15, 2015 at 1:41 PM, Evan Huus wrote: > > > On Wed,

Re: cpu over-consumption in Kafka producer -?

t is outside the > range of offsets maintained by the server for the given topic/partition > > On Wed, Apr 15, 2015 at 2:09 PM, Victor L wrote: > > > I set it to 10ms and it did fix cpu consumption problem but i don't see > > any messages coming up in consumer queue >

Re: Fetch Request Purgatory and Mirrormaker

understand that purgatory has been rewritten (again) in 0.8.3, so might it be worth trying a trunk build? Is there an ETA for a beta release of 0.8.3? Thanks, Evan On Tue, Apr 14, 2015 at 8:40 PM, Evan Huus wrote: > On Tue, Apr 14, 2015 at 8:31 PM, Jiangjie Qin > wrote: > >> Hey

Re: New and old producers partition messages differently

Related to this topic: why the choice of murmur2 over murmur3? I'm not super-familiar with the differences between the two, but I'd assume murmur3 would be faster or have a more even distribution or something. Evan P.S. Also, there appear to be many murmur3 implementations for other languages, wh

Re: Horizontally Scaling Kafka Consumers

On Thu, Apr 30, 2015 at 2:15 AM, Nimi Wariboko Jr wrote: > My mistake, it seems the Java drivers are a lot more advanced than the > Shopify's Kafka driver (or I am missing something) - and I haven't used > Kafka before. > > With the Go driver - it seems you have to manage offsets and partitions >