of consumer object each with a unique group-Id will
>deliver the message to both the consumer instances and I don't want that.
>
>Is there anyway to implement the use case I have mentioned or the use case
>itself does not fit kafka?
>
>
>
>
>On Tue, Mar 29, 2016 a
You could create two instances of the Consumer object, each with a unique
group ID.
On 3/29/16, 11:10 AM, "Gaurav Agarwal" wrote:
>I am not aware of any.
>
>On Mar 29, 2016 6:14 PM, "Vinod Kakad" wrote:
>>
>> But can one consumer have 2 group-id's, cause in my question I have
>> mentioned that
Well, if a partition is too large of a unit of order for your tastes, you
can add publisher ID¹s to some metadata, or force partition mapping and
use the key as an extra level of partitioning. And, pick a topicName that
describes all the traffic on that topic. An example:
topicName=³ad.click.even
The partitions enable scalability. Consumers and produces know which
partition records belong in based on their key (or manual assignment),
which makes it very easy to scale up your kafka cluster or a consuming
cluster.
The ordering problem is one that I have faced, and have a workaround.
Just kee
For some high performance environments, I would like to see microsecond or
nanosecond precision on metrics whenever possible. Even better would be
some sort of histogram of individual events so we could see the
variability.
On 12/21/15, 9:27 PM, "Alexey Pirogov" wrote:
>Ismael, thanks for reply
Hi All,
Is it possible to create a topic programmatically with a specific topic
configuration (number of partitions, replication factor, retention time, etc)
using just the new 0.9.0 client jar?
-Erik
a.a.a.a:9092,b.b.b.b:9092
>--messages
>1048576 --message-size 500 --topics part_1_repl_3_4 --show-detailed-stats
>--threads 32 --request-num-acks 1 --batch-size 1000 --request-timeout-ms
>1 --compression-codec 2 --reporting-interval 1000
>
>Regards,
>
>Prabhjot
>
>On
What happens when the new producer that is getting 70 MB/s is started on a
machine that is not part of the kafka cluster?
Can you include your topic description/configuration, producer
configuration, and broker configuration?
On 9/24/15, 1:44 AM, "Prabhjot Bharaj" wrote:
>Hi,
>
>I would like to
So, when using the high level consumer, only offset.storage needs to be
set, and the High level API will take care of committing the offsets
automatically every so often.
If using the simple consumer API, call commitOffsets and set versionId on
the OffsetCommitRequest¹s constructor to be 1 to comm
Looking at the docs here (
https://cwiki.apache.org/confluence/display/KAFKA/Committing+and+fetching+c
onsumer+offsets+in+Kafka), its possible to attach metadata as a string to
each partition for the consumer group using the PartitionData constructor
(http://apache.osuosl.org/kafka/0.8.2-beta/java-
So, this is expected behavior on the producer when its unable to
communicate with the kafka broker that is the leader for the message is
being sent.
First, if the design of your app allows, try to migrate to the new
producer API release is 0.8.1 It is fully asynchronous, and provides
callbacks
You only need one of the brokers to connect for publishing. Kafka will
tell the client about all the other brokers. But best practices state
including all of them is best.
-Erik
On 9/14/15, 2:46 PM, "Yuheng Du" wrote:
>I am writing a kafka producer application in java. I want the producer to
>
nks and Regards,
>Madhukar
>
>On Fri, Sep 11, 2015 at 8:27 PM, Prabhjot Bharaj
>wrote:
>
>> Hi,
>>
>> In addition to the parameters asked by Erik, it would be great if you
>>could
>> share your broker's server.properties as well
>>
>> R
Hi Madhukar,
Some questions that can help understand whats going on: Which kafka
version is used? Which Producer API is being used
(http://kafka.apache.org/documentation.html#producerapi)? And what are
the configs for this producer?
Also, because I know little about tomcat, is there a semantic f
artitions)
>
>So I understand that Kafka doesn't provide concurrently in the form that
>rabbitmq provides.
>
>I just can't understand why should any message delayed when I have enough
>machines and threads idle.
>
>
>
>On Thursday, September 2015, Helleren, Erik
>
So, the general scalability approach with kafka is to add more partitions
to scale. If you are using consumer groups and the High Level Consumer
API, redistribution of partitions is automatic on a failover of a member
of a consumer group. But, the High level consumer doesn¹t allow a
configuratio
So, there are several ways to do this. Lets assume the goal is to add
more topics to the application at runtime. And that this app is currently
written to be distributed via the magic of consumer groups. Sadly, I
don¹t think the High level consumer is well designed for this particular
use case.
It is possible to commit offsets using the SimpleConsumer API to kafka or
zookeeper for any GroupID, topic, and partition tuple. There are some
difficulties with the SimpleConsumer, but it should be able to make the
call within your app. See the scala Doc here:
http://apache.mirrorcatalogs.com/ka
tile
>happens in the slower partitions that are further away? Thanks.
>
>On Wed, Sep 9, 2015 at 3:15 PM, Helleren, Erik
>
>wrote:
>
>> So, I did my own latency test on a cluster of 3 nodes, and there is a
>> significant difference around the 99%’ile and higher for partiti
ope it will work.
>
>On Fri, Sep 4, 2015 at 2:03 PM, Helleren, Erik
>
>wrote:
>
>> I thing the suggestion is to have partitions/brokers >=1, so 32 should
>>be
>> enough.
>>
>> As for latency tests, there isn’t a lot of code to do a latency test.
>>
Hi King,
So, I think the issue could be which consumer you are using. Are you
using the simple consumer or the high level consumer API? And which
version of kafka are you using?
If you are using the simple consumer API, you can listen to a specific
partition. But you have to do the failover cod
Jörg,
So, I will start with some assumptions I have which effect my suggestions
below. I assume that the details you list are per cluster, and you have
3 clusters, one in each DC. Each DC¹s cluster replicates its topic ONLY
to the other DC¹s (Mirror maker configuration, otherwise you have circul
ed to
>rebuild the source if I modify the standard java test program
>ProducerPerformance provided in kafka, right? Now this standard program
>only has average latencies and percentile latencies but no per event
>latencies.
>
>Thanks.
>
>On Fri, Sep 4, 2015 at 1:42 PM, Hel
e
>the source of this jitter?
>
>Thanks.
>
>On Fri, Sep 4, 2015 at 10:54 AM, Helleren, Erik
>
>wrote:
>
>> WellŠ not to be contrarian, but latency depends much more on the latency
>> between the producer and the broker that is the leader for the partition
>
WellŠ not to be contrarian, but latency depends much more on the latency
between the producer and the broker that is the leader for the partition
you are publishing to. At least when your brokers are not saturated with
messages, and acks are set to 1. If acks are set to ALL, latency on an
non-sat
Allen,
1. The design docs state that most OS¹s make good use of memory and keep
recently written files in memory
(http://kafka.apache.org/documentation.html#maximizingefficiency). Since
everything was just written, it should still be fresh in the OS cache.
2. I cannot answer your virtualization qu
Hi Alexey,
So, a couple things. Your config seems to have some issues that would
result in long wait times,
You should try this configuration and see if you still have the issue:
acks=1
compression.type=snappy
retries=3 #Retry a few times to make it so they don¹t get dropped when a
broker fails
Shahab,
So, after a few searches, it just makes sense to paste it here. To Commit,
do something like this:
OffsetCommitRequest request = new OffsetCommitRequest(StringgroupId,
Map requestInfo, int correlationId,
String clientId, short versionId);
simpleConsumer.commitOffsets(request);
I am leavin
Hi Marc,
That describes the behavior of the kafka producer library that batches
writes to kafka. This post on confluent.io explains it pretty well:
http://kafka.apache.org/082/javadoc/index.html?org/apache/kafka/clients/pro
ducer/KafkaProducer.html
But the general idea is that the producer will g
Hi Buck,
So, if you are on 0.8.2, you can use the SimpleConsumer¹s commitOffsets
function which allows you to commit offsets to either zookeeper or a kafka
topic.
I would avoid rolling your own solution that edits data within existing
zookeeper paths because it might make upgrading to future versi
ure round trip latency using kafka-producer-perf-test.sh ?
or any other tool ?
Regards,
Prabhjot
On Tue, Aug 25, 2015 at 7:41 PM, Helleren, Erik
mailto:erik.helle...@cmegroup.com>> wrote:
Prabhjot,
When no compression is being used, it should have only a tiny impact on
performance. But whe
utomatically.
I'll also consider your point on creating real-life messages
But, I have still have one confusion -
Why would the current ProducerPerformance.scala compress an Array of Bytes with
all zeros ?
That will anyways give better throughput. correct ?
Regards,
Prabhjot
On Tu
Hi Prabhjot,
There are two important things to know about kafka compression: First
uncompression happens automatically in the consumer
(https://cwiki.apache.org/confluence/display/KAFKA/Compression) so you
should see ascii returned on the consumer side. The best way to see if
compression has happe
33 matches
Mail list logo
