I've wondered that about Azure Event Hubs as well. They both use a
different consumer offset tracking mechanism than the one in 0.8 for their
higher level consumers.
Christian
On Thu, Nov 13, 2014 at 2:32 PM, Joseph Lawson jlaw...@roomkey.com wrote:
Oh man they look similar. Any comments?
Yeah the real question is really are the products built on top of Kafka
(Kafka with a hat on). The last place I worked we ended up using Kinesis
rather than Kafka basically for the reason Niek mentions, it seemed easier
to accept the limitations and pay Amazon rather than run Kafka (small
company
I may be out of date, but I believe security measures are only in the
proposal stage. Your use case most likely involves sending data from the
internet at large to the Kafka instance. This will result in all data sent
to the Kafka instance being consumable by the internet at large. This is
Knowing that the partitioning is consistent for a given key means that
(apart from other benefits) a given consumer only deals with a partition of
the keyspace. So if you are in a system with tens of millions of users each
consumer only has to store state on a small number of them with
Wouldn't this work only for producers using random partitioning?
On Tue, Oct 14, 2014 at 1:51 PM, Kyle Banker kyleban...@gmail.com wrote:
Consider a 12-node Kafka cluster with a 200-parition topic having a
replication factor of 3. Let's assume, in addition, that we're running
Kafka v0.8.2,
.
Just to be clear on the size of each document/message, we are talking about
tweets, blog posts, ... (on 90% of cases the size is less than 50Kb)
Regards
On 9 October 2014 20:02, Christian Csar cac...@gmail.com wrote:
Apart from your data locality problem it sounds like what you want
I would say that it depends upon what you mean by persistence. I don't
believe Kafka is intended to be your permanent data store, but it would
work if you were basically write once with appropriate query patterns. It
would be an odd way to describe it though.
Christian
On Fri, Sep 12, 2014 at
:39 PM, cac...@gmail.com cac...@gmail.com
wrote:
I would say that it depends upon what you mean by persistence. I don't
believe Kafka is intended to be your permanent data store, but it would
work if you were basically write once with appropriate query patterns. It
would be an odd way
Based on the phrasing of your first question I might recommend taking
either a closer look at how Kafka works. Kafka stores data on its broker
servers in its own fashion as that is a key part of what makes it useful.
It is not written to use another database for message storage. Consumers of
the
I believe there are architectures for the chat system that can use Kafka in
a sensible fashion to achieve certain of the difficult aspects. However
doing partition per user would not be advisable, nor I imagine would
relying on Kafka's storage for checking for past or expired messages. (I've
done
Message retention in Kafka is disconnected from message consumption.
Messages are all persisted to disk and the queues do not need to fit in RAM
unlike some other systems. There are configuration values that control
maximum log size in terms of MB and the duration of retention which is
typically
This code says to send this message infinitely as fast as the machine can
thereby consuming as much of one CPU as possible. You may want to consider
an alternate test, perhaps one that records the number of messages sent in
a given time period.
public static void main(String[] args) {
wrote:
The new producer (that supports callbacks) is in trunk. It will be
released
in 0.8.2. You can look at the java doc of KafkaProducer for the api.
Thanks
Jun
On Thu, May 1, 2014 at 8:43 PM, Christian Csar cac...@gmail.com wrote:
On 05/01/2014 07:22 PM, Christian Csar wrote
13 matches
Mail list logo
