Thanks Johan,
I converted your code to vanilla java with a few small modifications
(included below in case anyone wants to use it) and ran it a few times.
Seems like it works ok for the quick peek use case, but I wouldn't
recommend anyone rely on the accuracy of it since I find, at least in our
I'm planning to upgrade a 0.8 cluster from 2 old nodes, to 3 new ones
(better hardware). I'm using a replication factor of 2.
I'm thinking the plan should be to spin up the 3 new nodes, and operate as
a 5 node cluster for a while. Then first remove 1 of the old nodes, and
wait for the
Hi,
I am planning to use Apache Kafka 0.8 to handle millions of messages per day.
Now I need to form the environment, like
(i) How many Topics to be created?
(ii) How many partitions/replications to be created?
(iii) How many Brokers to be created?
(iv) How many consumer instances in
Millions of messages per day (with each message being few bytes) is not
really 'Big Data'. Kafka has been tested for a million message per second.
The answer to all your question IMO is It depends.
You can start with a single instance (Single machine installation). Let
your producer send
This seems like the type of behavior I'd ultimately want from the
controlled shutdown tool
https://cwiki.apache.org/confluence/display/KAFKA/Replication+tools#Replicationtools-1.ControlledShutdown.
Currently, I believe the ShutdownBroker causes new leaders to be
selected for any partition the
Yeah, good point. I hadn't seen that before.
-Jay
On Mon, Jul 22, 2013 at 10:20 AM, Radek Gruchalski
radek.gruchal...@portico.io wrote:
296 looks familiar: https://www.nodejitsu.com/
Kind regards,
Radek Gruchalski
radek.gruchal...@technicolor.com (mailto:radek.gruchal...@technicolor.com)
296 looks familiar: https://www.nodejitsu.com/
Kind regards,
Radek Gruchalski
radek.gruchal...@technicolor.com (mailto:radek.gruchal...@technicolor.com) |
radek.gruchal...@portico.io (mailto:radek.gruchal...@portico.io) |
ra...@gruchalski.com
(mailto:ra...@gruchalski.com)
00447889948663
Is the kafka-reassign-partitions tool something I can experiment with now
(this will only be staging data, in the first go-round). How does it work?
Do I manually have to specify each replica I want to move? This would be
cumbersome, as I have on the order of 100's of topicsOr does the tool
Here's a ruby cli that you can use to replace brokers...it shells out to
the kafka-reassign-partitions.sh tool after figuring out broker lists from
zk. Hope its useful.
#!/usr/bin/env ruby
require 'excon'
require 'json'
require 'zookeeper'
def replace(arr, o, n)
arr.map{|v| v == o ? n : v }
Similar, yet different. I like it!
On Mon, Jul 22, 2013 at 1:25 PM, Jay Kreps jay.kr...@gmail.com wrote:
Yeah, good point. I hadn't seen that before.
-Jay
On Mon, Jul 22, 2013 at 10:20 AM, Radek Gruchalski
radek.gruchal...@portico.io wrote:
296 looks familiar:
It should be a roach in honor of Franz Kafka's Metamorphosis.
On 7/22/2013 2:55 PM, S Ahmed wrote:
Similar, yet different. I like it!
On Mon, Jul 22, 2013 at 1:25 PM, Jay Kreps jay.kr...@gmail.com wrote:
Yeah, good point. I hadn't seen that
I actually did this the last time a logo was discussed :)
https://docs.google.com/drawings/d/11WHfjkRGbSiZK6rRkedCrgmgFoP_vQ-QuWNENd4u7UY/edit
As it turns out, it was a dung beetle in the book (I thought it was
a roach as well).
-David
On 7/22/13 2:59
Hi,
We have a 3 node Kafka cluster. We want to increase the maximum amount of
time for which messages are saved in Kafka data logs.
Can we change the configuration on one node, stop it and start it and then
change the configuration of the next node?
Or should we stop all 3 nodes at a time, make
Yes, all configuration changes should be possible to do one node at a time.
-Jay
On Mon, Jul 22, 2013 at 2:03 PM, arathi maddula arathimadd...@gmail.comwrote:
Hi,
We have a 3 node Kafka cluster. We want to increase the maximum amount of
time for which messages are saved in Kafka data logs.
The beta release comes with mostly trace level logging. Is this
recommended? I notice our cluster produce way too many logs. I set all the
level to info currently.
nah. We just changed it to INFO and will monitor the log. We have GBs of logs
when it was at trace level. the kafka-request log was going crazy.
On Jul 22, 2013, at 10:54 PM, Jay Kreps jay.kr...@gmail.com wrote:
We run at info too except when debugging stuff. Are you saying that info is
too
You can try kafka-reassign-partitions now. You do have to specify the new
replica assignment manually. We are improving that tool to make it more
automatic.
Thanks,
Jun
On Mon, Jul 22, 2013 at 10:40 AM, Jason Rosenberg j...@squareup.com wrote:
Is the kafka-reassign-partitions tool something
Yes, the kafka-request log logs every request (in TRACE). It's mostly for
debugging purpose. Other than that, there is no harm to turn it off.
Thanks,
Jun
On Mon, Jul 22, 2013 at 7:59 PM, Calvin Lei ckp...@gmail.com wrote:
nah. We just changed it to INFO and will monitor the log. We have GBs
18 matches
Mail list logo
