Re: failed with LeaderNotAvailableError -

Hi David, Are you running in docker? Are you trying to connect from to a remote box? We found we could connect locally but couldn't connect from another remote host. (I've just started using kafka also) We had the same issue and found out: host.name=<%=@ipaddress%> needed to be the FQDN of the

Re: failed with LeaderNotAvailableError -

I probably should of mentioned that this was using Amazon ECS. On Thu, Dec 17, 2015 at 12:18 PM, Marko Bonaći wrote: > It doesn't have to be FQDN. > > Here's how I run Kafka in a container: > docker run --name st-kafka -p 2181:2181 -p 9092:9092 -e >

Kafka User Group meeting Link

Hi, Can anyone provide me the link for the KAFKA USER Group meetings which happened on Jun. 14, 2012 and June 3, 2014?? Link provided in the below wiki page is not a valid one.. https://cwiki.apache.org/confluence/display/KAFKA/Kafka+papers+and+presentations -- Regards, Prabhu.V --

Re: reassign __consumer_offsets partitions

Hi Damian The reassignment should treat the offsets topic as any other topic. I did a quick test and it seemed to work for me. Do you see anything suspicious in the controller log? B > On 16 Dec 2015, at 14:51, Damian Guy wrote: > > Hi, > > > We have had some

Re: Low-latency, high message size variance

Hi Jens, Gut feeling is that it's not a trivial patch, but you're more than welcome to take a shot. We should probably do a KIP also since it's a change to a public API (even if we only just released it). That's also a good way to get feedback and make sure we're not missing a better approach.

Re: Fallout from upgrading to kafka 0.9 from 0.8.2.3

Hi Jun, Answers inline: On Thu, Dec 17, 2015 at 9:41 AM, Jun Rao wrote: > Rajiv, > > Thanks for reporting this. > > 1. How did you verify that 3 of the topics are corrupted? Did you use > DumpLogSegments tool? Also, is there a simple way to reproduce the > corruption? > No I

Re: Fallout from upgrading to kafka 0.9 from 0.8.2.3

Rajiv, Thanks for reporting this. 1. How did you verify that 3 of the topics are corrupted? Did you use DumpLogSegments tool? Also, is there a simple way to reproduce the corruption? 2. As Lance mentioned, if you are using snappy, make sure that you include the right snappy jar (1.1.1.7). 3. For

Re: failed with LeaderNotAvailableError -

Hi Ben and Marko -- great suggestions re: connection failures and docker. The specific error here is: LeaderNotAvailableError: TopicMetadata(topic='topic-test-production', error=5, partitions=[]) That is an error code (5) returned from a MetadataRequest. In this context it means that the topic

Re: Kafka User Group meeting Link

Hi, In which part of the world? Cheers, Jens – Skickat från Mailbox On Thu, Dec 17, 2015 at 8:23 AM, prabhu v wrote: > Hi, > Can anyone provide me the link for the KAFKA USER Group meetings which > happened on Jun. 14, 2012 and June 3, 2014?? > Link

Re: Fallout from upgrading to kafka 0.9 from 0.8.2.3

The topic which stopped working had clients that were only using the old Java producer that is a wrapper over the Scala producer. Again it seemed to work perfectly in another of our realms where we have the same topics, same producers/consumers etc but with less traffic. On Thu, Dec 17, 2015 at

Re: how to programatically monitor Kafka availability

Hey Hohl, I use *partitionsFor * method to monitor the partition info for particular topics On Tue, Dec 15, 2015 at 11:27 AM, Hohl, Ken wrote: > We

Where can I find the document for consumer metrics

I can find some broker/producer metrics here http://kafka.apache.org/documentation.html#monitoring but where can I find consumer metrics docs Everytime I have to log this to find out what metrics I want MetricName [name=join-rate, group=consumer-coordinator-metrics, description=The number of

Re: Fallout from upgrading to kafka 0.9 from 0.8.2.3

Are you using the new java producer? Thanks, Jun On Thu, Dec 17, 2015 at 9:58 AM, Rajiv Kurian wrote: > Hi Jun, > Answers inline: > > On Thu, Dec 17, 2015 at 9:41 AM, Jun Rao wrote: > > > Rajiv, > > > > Thanks for reporting this. > > > > 1. How did you

Re: Kafka 0.9 consumer API question

Hi Rajiv, I think it makes sense to return a read-only assignments. What we can improve here is we can have addPartition method for consumer. Then we don't have to do any operations on the assignments returned by assignment method BTW, I think you can implement PartitionAssignor interface to

Re: Fallout from upgrading to kafka 0.9 from 0.8.2.3

Hmm, anything special with those 3 topics? Also, the broker log shows that the producer uses ack=0, which means the producer shouldn't get errors like leader not found. Could you clarify on the ack used by the producer? Thanks, Jun On Thu, Dec 17, 2015 at 12:41 PM, Rajiv Kurian

Re: Fallout from upgrading to kafka 0.9 from 0.8.2.3

I can't think of anything special about the topics besides the clients being very old (Java wrappers over Scala). I do think it was using ack=0. But my guess is that the logging was done by the Kafka producer thread. My application itself was not getting exceptions from Kafka. On Thu, Dec 17,

Re: Kafka User Group meeting Link

I am put up in India. Looking for the below user group meetings. I am able to access 2nd Kafka User group meeting, but not 1st & 3rd. User group meetings: - 1st Kafka user group meeting at LinkedIn, Jun. 14, 2012. video (part 1) , video (part 2)

Re: Fallout from upgrading to kafka 0.9 from 0.8.2.3

I was mistaken about the version. We were actually using 0.8.2.1 before upgrading to 0.9. On Thu, Dec 17, 2015 at 6:13 PM, Dana Powers wrote: > I don't have much to add on this, but q: what is version 0.8.2.3? I thought > the latest in 0.8 series was 0.8.2.2? > > -Dana >

Re: Where can I find the document for consumer metrics

We should add a section for that. Siyuan can you file a JIRA? Guozhang On Thu, Dec 17, 2015 at 11:08 AM, hsy...@gmail.com wrote: > I can find some broker/producer metrics here > http://kafka.apache.org/documentation.html#monitoring > > but where can I find consumer metrics

Local Storage

Maybe someone can answer this question for, because I cannot seem to find it. What is the data store that Kafka uses when it writes the logs to disk? I thought I saw a reference to KahaDB, but I am not sure if that is correct. Heath Ivie Solutions Architect Warning: This e-mail may contain

Re: Local Storage

Hi, Kafka *is* a data store. It writes data to files on the OS file system. One directory per partition, and a new file every specific amount of time (you can control this with log.roll.ms). The data format is specific to Kafka. Hope this helps, Gwen On Thu, Dec 17, 2015 at 3:32 PM, Heath Ivie

Re: failed with LeaderNotAvailableError -

It doesn't have to be FQDN. Here's how I run Kafka in a container: docker run --name st-kafka -p 2181:2181 -p 9092:9092 -e ADVERTISED_HOST=`docker-machine ip dev-st` -e ADVERTISED_PORT=9092 -d spotify/kafka And then you have access to Kafka on the docker host VM from any other machine. BTW I use

Re: Fallout from upgrading to kafka 0.9 from 0.8.2.3

Yes, the new java producer is available in 0.8.2.x and we recommend people use that. Also, when those producers had the issue, were there any other things weird in the broker (e.g., broker's ZK session expires)? Thanks, Jun On Thu, Dec 17, 2015 at 2:37 PM, Rajiv Kurian

Re: Fallout from upgrading to kafka 0.9 from 0.8.2.3

Yes we are in the process of upgrading to the new producers. But the problem seems deeper than a compatibility issue. We have one environment where the old producers work with the new 0.9 broker. Further when we reverted our messed up 0.9 environment to 0.8.2.3 the problem with those topics didn't

Re: failed with LeaderNotAvailableError -

FYI I am using digitialocean. I do not use docker. On Thu, Dec 17, 2015 at 6:33 PM, Ben Davison wrote: > Hi David, > > Are you running in docker? Are you trying to connect from to a remote box? > We found we could connect locally but couldn't connect from another

Re: Fallout from upgrading to kafka 0.9 from 0.8.2.3

I don't have much to add on this, but q: what is version 0.8.2.3? I thought the latest in 0.8 series was 0.8.2.2? -Dana On Dec 17, 2015 5:56 PM, "Rajiv Kurian" wrote: > Yes we are in the process of upgrading to the new producers. But the > problem seems deeper than a

Re: failed with LeaderNotAvailableError -

So what do I do? Kill my production servers and rebuild? Restarting all services does nit work. This seems kinda extreme. At this point I feel I have to kill all servers and rebuild. Thanks On Fri, Dec 18, 2015 at 2:28 AM, Dana Powers wrote: > Hi Ben and Marko --

Re: failed with LeaderNotAvailableError -

No - sorry if I wasn't clear. The "error" is not related to your servers. Just make sure that you create the topic before sending messages to it. If you do not create the topic beforehand, the server will auto-create, but it does take a few seconds to initialize. If you plan to rely on