You can find those numbers in
http://www.slideshare.net/Hadoop_Summit/building-a-realtime-data-pipeline-apache-kafka-at-linkedin?from_search=5
.
Thanks,
Jun
On Thu, Aug 15, 2013 at 4:38 PM, Vadim Keylis wrote:
> Just curious Jay. How many topics and consumers you guys have?
>
> Thanks
>
>
>
Just curious Jay. How many topics and consumers you guys have?
Thanks
On Thu, Aug 15, 2013 at 4:07 PM, Jay Kreps wrote:
> The tradeoff is there:
> Pro: more partitions means more consumer parallelism. The total
> threads/processes across all consumer machines can't exceed the consumer
> count.
The tradeoff is there:
Pro: more partitions means more consumer parallelism. The total
threads/processes across all consumer machines can't exceed the consumer
count.
Con: more partitions mean more file descriptors and hence smaller writes to
each file (so more random io).
Our setting is fairly ra
Jay. Thanks so much for explaining. What is the optimal number of
partitions per topic? What are the reasoning were behind your guys choice
of 8 partitions per topic?
Thanks,
Vadim
On Thu, Aug 15, 2013 at 1:58 PM, Jay Kreps wrote:
> Technically it is
> topics * partitions * replicas * 2 (ind
Technically it is
topics * partitions * replicas * 2 (index file and log file) + #open
sockets
-Jay
On Thu, Aug 15, 2013 at 11:49 AM, Vadim Keylis wrote:
> Good Morning Joel. Just to understand clearly how to predict number of open
> files kept by kafka.
>
> That is calculated by multiplying
Good Morning Joel. Just to understand clearly how to predict number of open
files kept by kafka.
That is calculated by multiplying number of topics * number of partitions
* number of replicas. In our case it would be 150 * 36 * 3. Am I correct?
How number of producers and consumers will influence
> One more question. What is the optimal number partition per topic to have?
>> Do you guys have hard set limit on a maximum topics Kafka can support. Are
>> there any other OS level settings I should be concerned that may cause
>> kafka to crash.
These would be highly specific to capacity planni
One more question. What is the optimal number partition per topic to have?
On Wed, Aug 14, 2013 at 9:47 AM, Vadim Keylis wrote:
> Joel thanks so much.
> Do you guys have hard set limit on a maximum topics Kafka can support. Are
> there any other OS level settings I should be concerned that may
Joel thanks so much.
Do you guys have hard set limit on a maximum topics Kafka can support. Are
there any other OS level settings I should be concerned that may cause
kafka to crash.
I am still trying to understand how to recover from failure and start
service.
The following error causes kafka not
We use 30k as the limit. It is largely driven by the number of partitions
(including replicas), retention period and number of
simultaneous producers/consumers.
In your case it seems you have 150 topics, 36 partitions, 3x replication -
with that configuration you will definitely need to up your fi
Good morning Jun. Correction in terms of open file handler limit. I was
wrong. I re-ran the command ulimit -Hn and it shows 10240. Which brings to
the next question. How appropriately calculate open files handler required
by Kafka? What is your guys settings for this field?
Thanks,
Vadim
On We
Good morning Jun. We are using Kafka 0.8 that I built from trunk in June or
early July. I forgot to mention that running ulimit on the hosts shows open
file handler set to unlimited. What are the ways to recover from last error and
restart Kafka ? How can I delete topic with Kafka service on all
The first error is caused by too many open file handlers. Kafka keeps each
of the segment files open on the broker. So, the more topics/partitions you
have, the more file handlers you need. You probably need to increase the
open file handler limit and also monitor the # of open file handlers so
tha
We have 3 node kafka cluster. I initially created 4 topics.
I wrote small shell script to create 150 topics.
TOPICS=$(< $1)
for topic in $TOPICS
do
echo "/usr/local/kafka/bin/kafka-create-topic.sh --replica 3 --topic
$topic --zookeeper $2:2181/kafka --partition 36"
/usr/local/kafka/bin/kafka
14 matches
Mail list logo
