>> So my question is if we go with a hardware load balancer, do all the
broker nodes have to treated equally? Ie: All broker nodes will have the
same topics and number of partitions for each topic?
All the brokers behind the same hardware load balancer or virtual IP will
be treated equally.
>> Or
Are you using Kafka 07 or 08 ?
On Thu, Sep 12, 2013 at 1:56 PM, prashant amar wrote:
> I created a topic with 4 partitions and for some reason the producer is
> pushing only to one partition.
>
> This is consistently happening across all topics that I created ...
>
> Is there a specific config
Which revision of 0.8 are you using? In a recent change, a producer will
stick to a partition for topic.metadata.refresh.interval.ms (defaults to 10
mins) time before picking another partition at random.
Thanks,
Jun
On Thu, Sep 12, 2013 at 1:56 PM, prashant amar wrote:
> I created a topic with
Hi Richard,
we are currently writing a C# driver as all c# drivers are not very active. It
has started as a simple translation of the scala driver (producer only) and
targets 0.8 protocol.
--
Grégoire
This indicates the controller (see the end of
http://kafka.apache.org/documentation.html#replication) of the cluster.
Thanks,
Jun
On Thu, Sep 12, 2013 at 5:29 PM, Lu Xuechao wrote:
> Thanks Rao. I found both log.dir and log.dirs worked.
>
> When I start up all my brokers, I see below log mess
Any error in the controller and the state-change log? Are brokers 2,3,4
alive?
Thanks,
Jun
On Thu, Sep 12, 2013 at 4:56 PM, Rajasekar Elango wrote:
> We are seeing a problem that we we try to send messages to new topic it
> fails kafka.common.LeaderNotAvailableException. But usually this probl
Hi, Everyone,
We have been stabilizing the 0.8 branch since the beta1 release. I think we
are getting close to an 0.8 final release. I made an initial list of the
remaining jiras that should be fixed in 0.8.
https://issues.apache.org/jira/issues/?jql=project%20%3D%20KAFKA%20AND%20fixVersion%20%3D
As Jun suggested, one reason could be that the
topic.metadata.refresh.interval.ms is too high. Did you observe if the
distribution improves after topic.metadata.refresh.interval.ms has passed ?
Thanks
Neha
On Fri, Sep 13, 2013 at 4:47 AM, prashant amar wrote:
> I am using kafka 08 version ...
I am using kafka 0.8.0-beta1 ..
Seems like messages are being delivered only to one partition (since
installation)
Should I upgrade or apply a patch to mitigate this issue.
Please advice
On Thu, Sep 12, 2013 at 8:44 PM, Jun Rao wrote:
> Which revision of 0.8 are you using? In a recent change
Yes, I was trying to find out how we can scale out with our Kafka cluster later
if we wanted to add more topics. But as you say, it might be simpler just to
use another Kafka node at some point.
Thanks for your response, it was very helpful!
Xuyen
-Original Message-
From: Neha Narkhede
This means broker 1 is the controller. It uses a generic zookeeper based
leader election module which is where this log4j message is coming from.
Thanks,
Neha
On Sep 12, 2013 10:52 PM, "Lu Xuechao" wrote:
> Thanks Rao. I found both log.dir and log.dirs worked.
>
> When I start up all my brokers,
Do you see any errors in the controller.log or the state change log?
Thanks,
Neha
On Sep 12, 2013 10:47 PM, "Rajasekar Elango" wrote:
> We are seeing a problem that we we try to send messages to new topic it
> fails kafka.common.LeaderNotAvailableException. But usually this problem
> will be tra
Thanks Neha
I will try applying this property and circle back.
Also, I have been attempting to execute kafka-producer-perf-test.sh and I
receive the following error
Error: Could not find or load main class
kafka.perf.ProducerPerformance
I am running against 0.8.0-beta1
Seems like perf i
Thanks Neha and Jun, It turned out to be miss configuration in our
zookeeper cluster. After correcting it everything looks good.
Thanks,
Raja.
On Fri, Sep 13, 2013 at 10:13 AM, Jun Rao wrote:
> Any error in the controller and the state-change log? Are brokers 2,3,4
> alive?
>
> Thanks,
>
> Jun
Just curious to know, what was the misconfiguration?
On Fri, Sep 13, 2013 at 10:02 AM, Rajasekar Elango
wrote:
> Thanks Neha and Jun, It turned out to be miss configuration in our
> zookeeper cluster. After correcting it everything looks good.
>
> Thanks,
> Raja.
>
>
> On Fri, Sep 13, 2013 at 10
Thanks Chris for the patches and Neha for reviewing and committing them!!!
It is great we now have support for Scala 2.10 in Kafka trunk and also 0.8
branch and without losing any existing support for anything else.
/***
Joe Stein
Founder, Principal Consu
I ran into this problem as well Prashant. The default partition key was
recently changed:
https://github.com/apache/kafka/commit/b71e6dc352770f22daec0c9a3682138666f032be
It no longer assigns a random partition to data with a null partition key.
I had to change my code to generate random partiti
Isn't this a bug?
I don't see why we would want users to have to code and generate random
partition keys to randomly distributed the data to partitions, that is
Kafka's job isn't it?
Or if supplying a null value tell the user this is not supported (throw
exception) in KeyedMessage like we do for
We are currently trying out AVRO although we're not being very fancy with it.
We're actually trying to build on top of this client code right now:
https://github.com/ExactTargetDev/kafka.git
Out of all the repos I looked at it seems to show the most promise and had some
work within this year. We
Q: Do you also use Avro?
Regards,
Henrik
On 13 sep 2013, at 08:33, Grégoire Seux wrote:
> Hi Richard,
>
> we are currently writing a C# driver as all c# drivers are not very active.
> It has started as a simple translation of the scala driver (producer only)
> and targets 0.8 protocol.
>
>
Hi,
We're considering the following network architecture:
Each kafka server will have 2 network cards:
one will be used to send/receive messages to/from consumer/producer
(external cluster traffic)
the other one will be used to send and receive replication messages between
the brokers (internal c
I am using kafka 08 version ...
On Thu, Sep 12, 2013 at 8:44 PM, Jun Rao wrote:
> Which revision of 0.8 are you using? In a recent change, a producer will
> stick to a partition for topic.metadata.refresh.interval.ms (defaults to
> 10
> mins) time before picking another partition at random.
> T
We have 3 zookeeper node in the cluster with a hardware load balancer . In
one of the zookeeper, we did not configure ensemble correctly (server.n
property in zoo.cfg) . So it ended up as like 2 nodes in one cluster, one
node in other cluster. The load balancer is randomly hitting one of 2
zookeep
Hello Joe,
The reason we make the producers to produce to a fixed partition for each
metadata-refresh interval are the following:
https://issues.apache.org/jira/browse/KAFKA-1017
https://issues.apache.org/jira/browse/KAFKA-959
So in a word the randomness is still preserved but within one
metada
Ah ok. Thanks for sharing that.
On Fri, Sep 13, 2013 at 2:50 PM, Rajasekar Elango wrote:
> We have 3 zookeeper node in the cluster with a hardware load balancer . In
> one of the zookeeper, we did not configure ensemble correctly (server.n
> property in zoo.cfg) . So it ended up as like 2 node
Hi Guozhang, Joe, Drew
In our case we have been running for the past 3 weeks and it has been
consistently writing only to to the first partition. The rest of the
partitions have empty index files.
Not sure if I am hitting any issue here.
I am using offset checker as my barometer. Also introspec
Without fixing KAFKA-1017, the issue is that the producer will maintain a
socket connection per min(#partitions, #brokers). If you have lots of
producers, the open file handlers on the broker could be an issue.
So, what KAFKA-1017 fixes is to pick a random partition and stick to it for
a configura
27 matches
Mail list logo
