Thanks for your response and I have some other questions as below in green
-Original Message-
From: Jun Rao [mailto:j...@confluent.io]
Sent: April 08, 2015 5:04
To: users@kafka.apache.org
Cc: d...@kafka.apache.org
Subject: Re: Is there a complete Kafka 0.8.* replication design document
Y
Hi,
I have seen this presentation:
http://www.slideshare.net/jjkoshy/offset-management-in-kafka
describing a new approach for offset management.
I wanted to know from which kafka version this option will be available and
will it be available by default
Thanks
Yosi
This is available from 0.8.2.0, and is enabled on server by default. The
consumer needs to specify offsets.storage parameter - the default is still
zookeeper, so the consumers should set it to 'kafka'.
The documentation also explain how to migrate from zookeeper offsets to
kafka offsets.
Gwen
On
Thanks Gwan,
Are there any plans to change the default from zookeeper to kafka in the
future?
On Wed, Apr 8, 2015 at 6:29 PM, Gwen Shapira wrote:
> This is available from 0.8.2.0, and is enabled on server by default. The
> consumer needs to specify offsets.storage parameter - the default is st
Hi,
I am looking at the java
class org.apache.kafka.clients.consumer.ConsumerConfig and I do not see
there a constant for offsets.storage
Am I missing something?
This is my pom dependency definition:
org.apache.kafka
kafka_2.10
0.8.2.1
On Wed, Apr 8, 2015 at 6:29 PM, Gwen Shapira wrote:
>
Thanks for the info, Todd. This is very useful. Please see my question
inline:
On Mon, Apr 6, 2015 at 10:24 AM, Todd Palino wrote:
>
> - Partition count (leader and follower combined) on each broker should
> stay under 4000
>
> As far as topic volume goes, it varies widely. We have topics
Hi all,
I have faced a strange situation with a broker this week where some
partitions of a topic with replication factor 1 became unavailable and
stayed unavailable even after the broker node that had (network?) issues
was back in the cluster. Even multiple restarts of the broker did not get
the
You are looking at the newer, and not-ready-yet consumer there.
The configuration you are looking for is defined here:
core/src/main/scala/kafka/consumer/ConsumerConfig.scala
Gwen
On Wed, Apr 8, 2015 at 9:25 AM, Yosi Botzer wrote:
> Hi,
>
> I am looking at the java
> class org.apache.kafka.cli
After initiating partition reassignment, does Kafka try to move all
partitions at once or does it do a subset at a time?
Thanks,
Wes
Hello,
I'm still trying to get to the bottom of an issue we had previously, with
an unclean shutdown during an upgrade to 0.8.2.1 (from 0.8.1.1). In that
case, the controlled shutdown was interrupted, and the node was shutdown
abruptly. This resulted in about 5 minutes of unavailability for most
FWIW, we've had good luck changing the mtime. No problems found.
On Mon, Apr 6, 2015 at 4:37 PM, Todd Palino wrote:
> I answered this in IRC, but the issue is that retention depends on the
> modification time of the log segments on disk. When you copy a partition
> from one broker to another, th
Sorry go back this far in time, I just noticed that the list had
replied accusing this email being spam, so I'll try again with better
formatting...
A few questions, hopefully you (and everyone) don't mind. Feel free to
ignore any/all.. I am trying to learn what I can from people who are
considera
Created issue https://issues.apache.org/jira/browse/KAFKA-2108
Cheers,
Thunder
-Original Message-
From: Jason Rosenberg [mailto:j...@squareup.com]
Sent: Tuesday, April 07, 2015 12:05 PM
To: users@kafka.apache.org
Subject: Re: Problem with node after restart no partitions?
Thunder, than
thanks, but can you please tell which metrics could highlight the factor
causing slow data migration by MirrorMaker?
Regards,
Nitin Kumar Sharma.
On Tue, Apr 7, 2015 at 10:10 PM, tao xiao wrote:
> You may need to look into the consumer metrics and producer metrics to
> identify the root cause.
Hi,
I am facing trouble in version 0.8.2.1. I saw this issue is closed.
https://issues.apache.org/jira/browse/KAFKA-1123
The console-producer --broker-list option from command line tool does work
with ipv6.
But we are using the 'new Producer<>(config);', and
metadata.broker.list=[FEDC:BA98:7654
Resolved ... My cilent is compiled with a old version of kafka.
Mingtao
Best Regards,
Mingtao
On Wed, Apr 8, 2015 at 3:00 PM, Mingtao Zhang
wrote:
> Hi,
>
> I am facing trouble in version 0.8.2.1. I saw this issue is closed.
> https://issues.apache.org/jira/browse/KAFKA-1123
>
> The console-pr
#2. /brokers/topics/[topic] stores the replica assignment for all
partitions in the topic.
/brokers/topics/[topic]/partitions/[partition_id]/state stores the
leader/isr per partition. We did it this way since the leader/isr need to
be updated on a per partition basis.
#4. Yes, what you observed is
Thanks.
When a failed broker recovery, it will truncate its log to the last HW. But how
to truncate? For a partition, it will store the log in many segment and one
segment is a physical file. How to truncate a file to a specified HW?
[cid:image001.png@01D072D5.BD52DFF0]
Regards,
Jason
From: Ju
So as I noted, it really does depend on what you need. In the case of a
small number of topics, I would say to make the number of partitions be a
multiple of the number of brokers. That will balance them in the cluster,
while still giving you some freedom to have larger partition counts for
larger
Hi,
I wanted to understand what kind of config settings will help me get the
best possible throughput with new producer API without increasing latency.
I tried to run the new producer shared between couple of threads and
latency seems to be increasing by increasing no. of threads. Ideally I
would
Metrics like Bytepersec, FetchRequestRateAndTimeMs can help you to check if
the consumer has problem processing messages
On Thu, Apr 9, 2015 at 2:40 AM, nitin sharma
wrote:
> thanks, but can you please tell which metrics could highlight the factor
> causing slow data migration by MirrorMaker?
>
Hello Kafka Friends,
We are considering a use-case where we'd like to have a Kafka Cluster with
potentially 1000's of partitions using a hashed key on customer userids.
We have heard that Kafka can support 1000's of partitions in a single
cluster and I wanted to find out if it's reasonable to have
The Tool Standardization KIP that Jiangjie started has been updated to contain
proposal details:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-14+-+Tools+Standardization
Any feedback is appreciated.
Thanks,
Matt
I've confirmed that the same thing happens even if it's not the controller
that's killed hard. Also, in several trials, it took between 10-30 seconds
to recover.
Jason
On Wed, Apr 8, 2015 at 1:31 PM, Jason Rosenberg wrote:
> Hello,
>
> I'm still trying to get to the bottom of an issue we had p
I am new to Kafka that's the reason asking so many question
KeyedMessage keyedMessage = new KeyedMessage(request.getRequestTopicName(),SerializationUtils.serialize(message));
producer.send(keyedMessage);
Currently,I am sending message without any key maintained as part of keyed
message , will it
Good questions. Here are the answers...
- Yes, all brokers we run are hardware. We do not use virtual systems for
Kafka or Zookeeper
- There's a number of things we have done. I covered a lot of them last
year at ApacheCon (
http://www.slideshare.net/ToddPalino/enterprise-kafka-kafka-as-a-service
26 matches
Mail list logo
