Hi,
Running some tests on 0811 and wanted to see what happens when a broker is
taken down with 'kill'. I bumped into the situation at the subject where
launching the broker back left him a bit out of the game as far as I could
see using stack driver metrics.
Trying to rebalance with verify
Dear Experts,
We recently updated to kafka v0.8.1.1 with zookeeper v3.4.5. I have of
topic with 30 partitions and 2 replicas. We are using High level consumer
api.
Each consumer process which is a storm topolofy has 5 streams which
connects to 1 or more partitions. We are not using storm's
You can run with a single node zookeeper cluster also.
See
http://zookeeper.apache.org/doc/r3.3.4/zookeeperStarted.html#sc_InstallingSingleMode
Cheers,
Erik.
Op 9 okt. 2014, om 22:52 heeft S Ahmed sahmed1...@gmail.com het volgende
geschreven:
I want kafka features (w/o the redundancy)
Hi,
We have 2 data centers that produce events. Each DC has to process events from
both DCs.
I had the following in mind:
DC 1 | DC 2
events |events
+ + + | + + +
| | | | | | |
v
Mohit,
I wonder if it is related to
https://issues.apache.org/jira/browse/KAFKA-1585. When zookeeper expires a
session, it doesn't delete the ephemeral nodes immediately. So if you end
up trying to recreate ephemeral nodes quickly, it could either be in the
valid latest session or from the
Did you ensure that your replication factor was set higher than 1? If so,
things should recover automatically after adding the killed broker back
into the cluster.
On Mon, Oct 20, 2014 at 1:32 AM, Shlomi Hazan shl...@viber.com wrote:
Hi,
Running some tests on 0811 and wanted to see what
Another way to set up this kind of mirroring is by deploying 2 clusters in
each DC - a local Kafka cluster and an aggregate Kafka cluster. The mirror
maker copies data from both the DC's local clusters into the aggregate
clusters. So if you want access to a topic with data from both DC's, you
Hi all,
This config property does not appear in the table of broker config properties.
But it appears in the example on the Web page. So I wonder if this is still a
valid config property for 0.8.1. Thanks.
Libo
Which example are you referring to?
On Mon, Oct 20, 2014 at 7:47 AM, Libo Yu yu_l...@hotmail.com wrote:
Hi all,
This config property does not appear in the table of broker config
properties. But it appears in the example on the Web page. So I wonder if
this is still a valid config property
http://kafka.apache.org/documentation.html#brokerconfigs
In section 6.3, there is an example your production server configuration.
Date: Mon, 20 Oct 2014 07:49:56 -0700
Subject: Re: log.cleanup.interval.mins still valid for 0.8.1?
From: neha.narkh...@gmail.com
To: users@kafka.apache.org
hi
Is it possible to have iOS and android to run the code needed for Kafka
producers ? I want to have mobile clients connect to Kafka broker
Thanks,
Josh
Hi Josh,
Why not have Rest api service running where you post messages
from your mobile clients. Rest api can run kafka producers
accepting these messages pushing it into kafka brokers. Here
is an example where we did similar service for kafka
Thanks for the tip. I would like to avoid hand rolling any code if
possible. For example, on Android I would like to ask if people are able to
include and use the kafka jars with no problem? And on iOS, if there is a
way to include any C or other relevant code.
On Mon, Oct 20, 2014 at 8:49 AM,
What is the use case requiring that? If you try to integrate kafka in the
two different mobile platforms you will get many separate development
cycles and none will work in many mobile networked environments. You can
HTTP/HTTPS POST the same Avro objects (or Thrift or ProtoBuf) from each
platform.
What is the use case requiring that?
I'm looking for an open source library that I can use in Android and iOS,
instead of hand rolling my own.
On Mon, Oct 20, 2014 at 10:21 AM, Joe Stein joe.st...@stealth.ly wrote:
What is the use case requiring that? If you try to integrate kafka in the
two
I may be out of date, but I believe security measures are only in the
proposal stage. Your use case most likely involves sending data from the
internet at large to the Kafka instance. This will result in all data sent
to the Kafka instance being consumable by the internet at large. This is
Yes I did. It is set to 2.
On Oct 20, 2014 5:38 PM, Neha Narkhede neha.narkh...@gmail.com wrote:
Did you ensure that your replication factor was set higher than 1? If so,
things should recover automatically after adding the killed broker back
into the cluster.
On Mon, Oct 20, 2014 at 1:32
If the whole cluster is down and you allow unclean leader election on the
broker, some exposed messages on the broker could be lost when restarting
the brokers. When that happens, the consumers may need to reset their
offset since the current offsets may no longer be valid. By default, the
offset
Hi everyone,
We run a 3 Kafka cluster using 0.8.1.1 with all topics having a replication
factor of 3 meaning every broker has a replica of every partition.
We recently ran into this issue (
https://issues.apache.org/jira/browse/KAFKA-1028) and saw data loss within
Kafka. We understand why it
Hi Kyle,
I added new documentation, which will hopefully help. Please take a look here:
https://issues.apache.org/jira/browse/KAFKA-1555
I've heard rumors that you are very very good at documenting, so I'm
looking forward to your comments.
Note that I'm completely ignoring the acks1 case since
This is another potential use-case for message metadata. i.e., if we
had a DC/environment field in the header you could easily set up a
two-way mirroring pipeline. The mirror-maker can just filter out
messages that originated in the source cluster. For this to be
efficient the mirror maker should
As Neha mentioned, with rep factor 2x, this shouldn't normally cause
an issue.
Taking the broker down will cause the leader to move to another
replica; consumers and producers will rediscover the new leader; no
rebalances should be triggered.
When you bring the broker back up, unless you run a
It has been replaced with:
log.retention.check.interval.ms
We will update the docs. Thanks for reporting this.
Joel
On Mon, Oct 20, 2014 at 11:08:38AM -0400, Libo Yu wrote:
http://kafka.apache.org/documentation.html#brokerconfigs
In section 6.3, there is an example your production server
I am running a performance test and from what I am seeing is that messages
are taking about 100ms to pop from the queue itself and hence making the
test slow. I am looking for pointers of how I can troubleshoot this issue.
There seems to be plenty of CPU and IO available. I am running 22
Hi Neha,
Yes, I understand that but when transmitting single message (I can not set
List of all topics) Only Single one. So I will to add same message in
buffer with different topic. If Kakfa protocol, allows to add multiple
topic then message does not have to be re-transmited over the wire to
25 matches
Mail list logo
