I think it will be very usefull if we can mirror to a different topic name
on destination side.
We have a use case to merge data from multiple colos to one central colo.
SunilKalva
On Mon, Mar 9, 2015 at 4:29 PM, tao xiao xiaotao...@gmail.com wrote:
I don't think you can mirror messages to a
Hi,
I created a message stream in my consumer using connector
.createMessageStreamsByFilter(new Whitelist(mm-benchmark-test\\w*), 5); I
have 5 topics in my cluster and each of the topic has only one partition.
My understanding of wildcard stream is that multiple streams are shared
between
I am wondering where does kafka cluster keep the topic metadata (name,
partition, replication, etc)? How does a server recover the topic's
metadata and messages after restart and what data will be lost?
Thanks for anyone to answer my questions.
best,
Yuheng
If a send request in the middle of the list fails, will all send requests
that follows it fail? Or only the messages that are put in the same batch
by the underneath transportation layer fail?
On Mon, Mar 9, 2015 at 1:31 AM, Manikumar Reddy ku...@nmsworks.co.in
wrote:
1. We can send list of
Yuheng,
kafka keeps cluster metadata in zookeeper along with topic metadata
as well. You can use zookeeper-shell.sh or zkCli.sh to check zk nodes,
/brokers/topics will give you the list of topics .
--
Harsha
On March 9, 2015 at 8:20:59 AM, Yuheng Du (yuheng.du.h...@gmail.com)
Harsha,
Thanks for reply. So what if the zookeeper cluster fails? Will the topics
information be lost? What fault-tolerant mechanism does zookeeper offer?
best,
On Mon, Mar 9, 2015 at 11:36 AM, Harsha ka...@harsha.io wrote:
Yuheng,
kafka keeps cluster metadata in zookeeper along
In general users are expected to run zookeeper cluster of 3 or 5 nodes.
Zookeeper requires quorum of servers running which means at least ceil(n/2)
servers need to be up. For 3 zookeeper nodes there needs to be atleast 2 zk
nodes up at any time , i.e your cluster can function fine incase of 1
Any timeline on an official 0.8.2.1 release? Were there any issues found
with rc2? Just checking in because we are anxious to update our brokers but
waiting for the patch release. Thanks.
On Thu, Mar 5, 2015 at 12:01 AM, Neha Narkhede n...@confluent.io wrote:
+1. Verified quick start, unit
I'm new to Kafka and I'm trying to understand the version semantics. We
want to use Kafka w/ Spark but our version of Spark is tied to 0.8.0. We
were wondering what guarantees are made about backwards compatbility across
0.8.x.x. At first glance, given the 3 digits used for versions, I figured
Hmm, that sounds like a bug. Can you paste the log of leader rebalance
here?
Thanks for you suggestions.
It looks like the rebalance actually happened only once soon after I started
with clean cluster and data was pushed, it didn’t happen again so far, and I
see the partitions leader counts
I was trying to see if kafka-2010 is a blocker to the 0.8.2.1 release. It
doesn't seem to be since it won't affect the common usage when the
controlled shutdown is enabled (by default). I will wrap up the 0.8.2.1
release.
Thanks,
Jun
On Mon, Mar 9, 2015 at 8:25 AM, Solon Gordon
Opened a kafka issue for rebalance happening with auto.rebalance set to false.
https://issues.apache.org/jira/browse/KAFKA-2011
Logs for rebalance:
[2015-03-07 16:52:48,969] INFO [Controller 2]: Resuming preferred replica
election for partitions: (kafka.controller.KafkaController)
Hi Sunilkalva,
We are rewriting mirror maker in KAFKA-1997 with a handful of enhancement.
With that new mirror maker, you will be able to mirror to a different
topic by using the message handler.
Jiangjie (Becket) Qin
On 3/9/15, 4:41 AM, sunil kalva sambarc...@gmail.com wrote:
I think it will
Thanks, got it!
best,
Yuheng
On Mon, Mar 9, 2015 at 11:52 AM, Harsha ka...@harsha.io wrote:
In general users are expected to run zookeeper cluster of 3 or 5 nodes.
Zookeeper requires quorum of servers running which means at least ceil(n/2)
servers need to be up. For 3 zookeeper nodes there
I'm curious what type of batching Kafka producers do at the socket layer.
For instance, if I have a partitioner that round robin's n messages to a
different partition, am I guaranteed to get n different messages sent over
the socket or is there some micro-batching going on underneath?
I am trying
Hi,
I have two separate consumer groups on different JVM processes, but both have
the same group.id. They are high level consumer groups with each group
containing 3 consumers.
Only one group consumes at a given time - and I would like both groups, with
the same id to share the load and
If you have 2 consumer groups, each group will read from all partitions
automaticcally if you are using HighLevel consumer ( In your case it would
be each consumer gets 2 partitons). You don't have to specify the
partitions it should read from.
Thanks,
Mayuresh
On Mon, Mar 9, 2015 at 9:59 AM,
Correction: Actually the rebalance happened quite until 24 hours after the
start, and thats where below errors were found. Ideally rebalance should not
have happened at all.
Thanks
Zakee
On Mar 9, 2015, at 10:28 AM, Zakee kzak...@netzero.net wrote:
Hmm, that sounds like a bug. Can you
Hi,
How to specify group name when using kafka-console-consmer.sh in 0.8.2.
Kafka 0.8.1 had --group option while running the above script.
I need group name to run offset checker after running the consumer.
Thanks,
Tushar
The following are the results of the votes.
+1 binding = 3 votes
+1 non-binding = 2 votes
-1 = 0 votes
0 = 0 votes
The vote passes.
I will release artifacts to maven central, update the dist svn and download
site. Will send out an announce after that.
Thanks everyone that contributed to the
HI Phill,
Do you mean you are using 6 consumers with the same group id? Or you have
3 consumers using one group id, and another 3 using another different
group id?
For the example you mentioned, what you can do is to run several consumers
on different physical machine with the same group id, they
Hi Tao,
That is expected behavior. You can use set
partition.assignment.strategy=roundrobin in consumer config. It will take
all the partitions from all topics and do a round robin assignment,
whereas range only take partitions for each individual topic for
assignment.
Jiangjie (Becket) Qin
On
Hi Yang,
In the code suggested by Manikumar, yes, it is possible message 3 still
got sent even message 2 failed.
There is no single line code for send a batch of message synchronously
now, but after KAFKA-1660 is checked in, you may be able to achieve this
by doing the following:
Set a callback
On Mon, Mar 9, 2015 at 10:38 AM, Phill Tomlinson philltomlin...@fico.com
wrote:
Hi,
I have two separate consumer groups on different JVM processes, but both
have the same group.id.
You've said this twice, and I think it's creating some confusion, because
the group.id is exactly what
Hi,
I would like to subscribe to the Kafka mailing list for general questions.
Please let me know what I need to do in order to submit questions to the Kafka
general mailing list. Thanks.
Regards,
Mark Flores
Project Manager, Enterprise Technology
Direct206-576-2675
Email
Looks like you subscribed. Just start a new thread and ask away.
Otis
--
Monitoring * Alerting * Anomaly Detection * Centralized Log Management
Solr Elasticsearch Support * http://sematext.com/
On Mon, Mar 9, 2015 at 4:27 PM, Mark Flores mark.flo...@expeditors.com
wrote:
Hi,
I would
If I understood your question correctly, you want to be able to read the
output of Camus in Hive and be able to know partition values. If my
understanding is right, you can do so by using the following.
Hive provides the ability to provide custom patterns for partitions. You
can use this in
I'm calling ConsumerConnector.shutdown to close a consumer connection
and kafka's log reports an error?
I don't see a similar error when using SimpleConsumer.
Is there a way to close ConsumerConnector so that the errors aren't
reported in the kafka log (this is making it very difficult to sift
Hi Mark,
You’ve already asked a question in the right place – sending email to
users@kafka.apache.org is the right way.
If it is a development question, you can send to d...@kakfa.apache.org.
Jiangjie (Becket) Qin
From: Mark Flores
mark.flo...@expeditors.commailto:mark.flo...@expeditors.com
One of our development teams is considering implementing a Kafka solution. If
the development team were to assume implementing 6 separate regional Kafka
clusters:
*How could we implement global Pub/Sub between clusters?
*Can we do bi-directional replication with MirrorMaker
Is there anything wrong with brokers around that time? E.g. Broker restart?
The log you pasted are actually from replica fetchers. Could you paste the
related logs in controller.log?
Thanks.
Jiangjie (Becket) Qin
On 3/9/15, 10:32 AM, Zakee kzak...@netzero.net wrote:
Correction: Actually the
Hi Kevin,
You can use partition.assignment.strategy=roundrobin.
This will balance all the partition of all the topics across consumer
thread.
I think the rationale behind using default consumer id is that you will
have better information to identify a consumer. But if you want to have
some
No broker restarts.
Created a kafka issue: https://issues.apache.org/jira/browse/KAFKA-2011
https://issues.apache.org/jira/browse/KAFKA-2011
Logs for rebalance:
[2015-03-07 16:52:48,969] INFO [Controller 2]: Resuming preferred replica
election for partitions:
The stickiness of partition only applies to old producer. In new producer
we have the round robin for each message. The batching in new producer is
per topic partition, the batch size it is controlled by both max batch
size and linger time config.
Jiangjie (Becket) Qin
On 3/9/15, 10:10 AM, Corey
No broker restarts.
Created a kafka issue: https://issues.apache.org/jira/browse/KAFKA-2011
https://issues.apache.org/jira/browse/KAFKA-2011
Logs for rebalance:
[2015-03-07 16:52:48,969] INFO [Controller 2]: Resuming preferred replica
election for partitions:
Yes, Kevin is right. It does not matter whether you run the consumer from
the same JVM or not, as long as the consumers has same group id, they are
in the same group. So in your case, you have 6 consumers in the same
consumer group. Since you have 6 partitions in the topic, assuming you
have only
There are various prior questions including..
http://search-hadoop.com/m/4TaT4ts2oz1/disaster+recovery/v=threaded
Is there a clear document on disaster recovery patterns for K and their
respective trade offs.
How are actual prod deployments dealing with this.
For instance I want my topics
https://github.com/apache/kafka/blob/0.8.2/core/src/main/scala/kafka/consumer/ConsumerConfig.scala#L101
suggests that 'consumer.id' should only be set explicitly for testing
purposes. Is there a reason that it would be a bad idea to set it
ourselves for production use?
The reason I am asking is
There was a typo in the question - should have been ...
I can tolerate the [replicant]
I don't think you can mirror messages to a different topic name in the
current mirror maker implementation. Mirror maker sends the message to
destination topic based on the topic name it reads from source
On Mon, Mar 9, 2015 at 5:00 PM, sunil kalva sambarc...@gmail.com wrote:
Can i configure
Hi, tao xiao and Jiangjie Qin
I encounter with the same issue, my node had recovered from high load problem
(caused by other application)
this is the kafka-topic show:
Topic:ad_click_sts PartitionCount:6ReplicationFactor:2 Configs:
Topic: ad_click_sts Partition: 0
Hi,
I have a topic with 6 partitions. I have two consumer groups with 3 consumers
each, both with the same group.id. However only one group appears to consume
from the topic.
Is this expected behaviour? I would expect to be able to concurrently use two
consumer groups on the same topic to
Hi,
Kafka 0.8.1.1 allows us to send a list of messages in sync mode:
public void send(ListKeyedMessageK,V messages);
I did not find a counter-part of this api in the new producer that is
introduced in kafka 0.8.2. It seems that we can use the following method
to do sync send in kafka
43 matches
Mail list logo
