Thanks for sharing.
Best Regards
Jerry
-Original Message-
From: Jay Krepslt;jay.kr...@gmail.comgt;
To: users@kafka.apache.orglt;users@kafka.apache.orggt;;
lt;d...@samza.incubator.apache.orggt;;
Cc:
Sent: 2013-12-17 (星期二) 06:00:17
Subject: Logs and distributed systems
For anyone
Hi All,
We are having kafka cluster of 2 nodes. (using 0.8.0 final release)
Replication Factor: 2
Number of partitions: 2
I have created a topic test-topic1 in kafka.
When i am listing status of that topic using bin/kafka-list-topic.sh, the
status is:
topic: test-topic1partition: 0
Hey all,
I've recently been having problems with consumer groups rebalancing. I'm
using several high level consumers which all belong to the same group.
Occasionally one or two of them will get stuck in a rebalance loop. They
attempt to rebalance, but the partitions they try to claim are
Hello,
This issue is known as in this JIRA:
https://issues.apache.org/jira/browse/KAFKA-1067
Guozhang
On Tue, Dec 17, 2013 at 8:48 AM, Gerrit Jansen van Vuuren
gerrit...@gmail.com wrote:
hi,
I've had the same issue with the kafka producer.
you need to use a different partitioner than
ZK was designed from the start as a clustered, consistent, highly available
store for this sort of data and it works extremely well. Redis wasn't and I
don't know anyone using Redis in production, including me, who doesn't have
stories of Redis losing data. I'm sticking with ZK.
On Tue, Dec 17,
I'm on Kafka 0.8 final. Both brokers are up. The behavior is my producer
produces messages just fine, then it pauses for a few seconds. Then it
continues. The brokers are not stopping and starting. The broker logs show
that another producer/consumer has a connection error at the same time my
Sorry, a typo. Correct my question. When consumer.timeout.ms is set to 0, if
there is no
message available, hasNext() throws a timeout exception, otherwise it returns
true.
Is that the right behavior?
Regards,
Libo
-Original Message-
From: Jun Rao [mailto:jun...@gmail.com]
Sent:
There are no compatibility issues. You can roll upgrades through the
cluster one node at a time.
Thanks
Neha
On Tue, Dec 17, 2013 at 9:15 AM, Drew Goya d...@gradientx.com wrote:
So I'm going to be going through the process of upgrading a cluster from
0.8.0 to the trunk (0.8.1).
I'm going
Interesting, wasn't aware of that.
Can you comment on how you go about monitoring your ZK cluster in terms of
throughput and if it is reaching its limits? Or is it even possible to do
this?
On Tue, Dec 17, 2013 at 2:01 PM, Benjamin Black b...@b3k.us wrote:
ZK was designed from the start as a
Hi,
We have been trying out the kafka 0.8.0 beta1 for awhile and recently
attempted to upgrade to 0.8.0 but noticed that the stop server script
doesn't seem to stop the broker anymore. I noticed here[1] that a commit
was made before the release to change the signal sent to stop the broker
from
It appears that consumers that do not get messages regularly are timing out
every 30 seconds. This interval coincides with the default setting for
socket.timeout.ms at the consumer. When the timeout happens it looks like
the broker socket hangs for a few seconds, causing all other connected
Sorry it's taken so long to reply, the issue went away after I reassigned
partitions. Now it's back.
I haven't checked JMX, because the brokers and zookeeper have been
reporting the same ISR for several hours.
Some more details:
The cluster/topic has
5 brokers (1, 4, 5, 7, 8)
15 partitions
You can do this indirectly by monitoring the avg/max latency of operations
on zookeeper. There is no direct way of measuring the requests/sec to
zookeeper since they don't expose the relevant jmx metrics.
Thanks,
Neha
On Tue, Dec 17, 2013 at 11:13 AM, S Ahmed sahmed1...@gmail.com wrote:
That error comes from you calling create message stream twice or the
container you are running in is causing this to be called twice
https://github.com/apache/kafka/blob/0.8/core/src/main/scala/kafka/consumer/
ZookeeperConsumerConnector.scala#L133from , don't do that and you won't get
the error.
If there is no more messages, hasNext will return false instead of throwing
an exception.
Guozhang
On Tue, Dec 17, 2013 at 11:53 AM, Yu, Libo libo...@citi.com wrote:
Sorry, a typo. Correct my question. When consumer.timeout.ms is set to 0,
if there is no
message available, hasNext() throws
It is worth mentioning you can reduce the likelyhood of loosing message by
running the controlled shutdown before killing the broker.
https://cwiki.apache.org/confluence/display/KAFKA/Replication+tools#Replicationtools-Whatiscontrolledshutdown?
The connection refused is a bit surprising though.
i am not able to find run-simulator.sh in 0.8 even after building perf.if
this tool has been deprecated what are other alternatives available now for
perf testing?
Regards,
Pushkar
Have you looked at
https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Whyaretheremanyrebalancesinmyconsumerlog
?
Thanks,
Jun
On Tue, Dec 17, 2013 at 9:24 AM, Drew Goya d...@gradientx.com wrote:
Hey all,
I've recently been having problems with consumer groups rebalancing. I'm
using
Which OS are you on?
Thanks,
Jun
On Tue, Dec 17, 2013 at 11:15 AM, Bryan Baugher bjb...@gmail.com wrote:
Hi,
We have been trying out the kafka 0.8.0 beta1 for awhile and recently
attempted to upgrade to 0.8.0 but noticed that the stop server script
doesn't seem to stop the broker
Did you change fetch.wait.max.ms in the consumer config? If so, did you
make sure that it is smaller than socket.timeout.ms? Also, if you look at
the request log, how long does it take to complete the timed out fetch
request?
Thanks,
Jun
On Tue, Dec 17, 2013 at 2:30 PM, Tom Amon
Actually, hasNext() only returns false when the consumer connector is
shutdown. Typically, you either set consumer.timeout.ms to -1 or a value
larger than 0. If it's set to 0, my guess is that it throws a timeout
exception immediately if there is no more message.
Thanks,
Jun
On Tue, Dec 17,
If a broker never joins an ISR, it could be that the fetcher died
unexpectedly. Did you see any Error due to in the log of broker 4?
Another thing to check is the max lag and the per partition lag in jmx.
Thanks,
Jun
On Tue, Dec 17, 2013 at 4:09 PM, Ryan Berdeen rberd...@hubspot.com wrote:
What's the replication factor of the topic? Is it larger than 1? You can
find out using the list topic command.
Thanks,
Jun
On Tue, Dec 17, 2013 at 2:39 PM, Francois Langelier
francois.langel...@mate1inc.com wrote:
Hi,
I installed zookeeper and kafka 8.0 following the quick start (
You can run kafka-producer-perf-test.sh and kafka-consumer-perf-test.sh.
Thanks,
Jun
On Tue, Dec 17, 2013 at 8:44 PM, pushkar priyadarshi
priyadarshi.push...@gmail.com wrote:
i am not able to find run-simulator.sh in 0.8 even after building perf.if
this tool has been deprecated what are
RHEL 6.4 64bit
Java 6u35
On Tue, Dec 17, 2013 at 10:57 PM, Jun Rao jun...@gmail.com wrote:
Which OS are you on?
Thanks,
Jun
On Tue, Dec 17, 2013 at 11:15 AM, Bryan Baugher bjb...@gmail.com wrote:
Hi,
We have been trying out the kafka 0.8.0 beta1 for awhile and recently
I explored that possibility but I'm not seeing any ZK session expirations
in the logs and it doesn't look like these rebalances complete.
They fail due to conflicts in the zookeeper data
On Tue, Dec 17, 2013 at 8:53 PM, Jun Rao jun...@gmail.com wrote:
Have you looked at
thanks Jun.
On Wed, Dec 18, 2013 at 10:47 AM, Jun Rao jun...@gmail.com wrote:
You can run kafka-producer-perf-test.sh and kafka-consumer-perf-test.sh.
Thanks,
Jun
On Tue, Dec 17, 2013 at 8:44 PM, pushkar priyadarshi
priyadarshi.push...@gmail.com wrote:
i am not able to find
27 matches
Mail list logo
