Hello,
I'm facing a somewhat similar issue than Antonin described in
https://mail.mozilla.org/pipermail/heka/2015-April/000522.html where
Heka is slow getting stuff from Kafka (around 1,600 messages/s).
I have setup a Kafka instance where I dump a truck load of log messages.
Without tuning anything, Heka is able to read from files and to output
roughly 100,000 lines/s of logs to Kafka using the Kafka output plugin.
I have 3 distinct servers (all VMs):
- 1 heka used to feed Kafka (producer): this one does at least 100,000
messages/s with a basic config [1], running heka 0.9.2
- 1 kafka (standard config), running kafka 0.8.2.1
- 1 heka used to parse data coming from kafka (consumer), running heka
0.9.2
This looks like a convoluted setup but it is required by the way I
receive the logs (meaning I couldn't really merge the 2 heka instances,
specially if I want to add more consumers).
If I setup my heka consumer (the last server in the list above), as a
simple heka input without output or decoder or anything (cf config in
[2]), I only get around 1,600 messages/s from Kafka.
I know Kafka can do better because locally (on the kafka server) I can
easily get around 354,000 messages/s (using kafka-console-consumer.sh).
And remotely (ie on the consumer machine), using kafka-console-consumer.sh,
I get around 340,000 messages/s.
I've played with maxprocs, plugin_chansize, max_process_inject,
poolsize, default_fetch_size, event_buffer_size, max_open_reqests,
background_refresh_frequency, max_wait_time and I haven't been able to
do much better.
The only way I've managed to get more messages/s is to duplicate my
input (and to run more heka servers): ie to declare a KafkaInput2 block
similar to KafkaInput1.
Ideally I'd like to sustain at least around 80,000 messages/s or at
least be CPU bound by my decoders, right now the bottleneck is the
input.
Is this something anyone has experienced? Is there something wrong in my
setup?
Wait, I just discovered something, if I switch the offset_method to
oldest, I get around 5,000 messages/s instead of 1,600?
Cheers,
Mathieu
============================================================
[1] Producer config
[hekad]
maxprocs = 4
base_dir = "/data/heka-0_9_2-linux-amd64"
share_dir = "/data/heka-0_9_2-linux-amd64/share/heka"
poolsize = 300
[syslog]
type = "LogstreamerInput"
log_directory = "/data/syslog/log"
oldest_duration = "1h"
file_match =
'heka_(?P<Year>\d+)-(?P<Month>\d+)-(?P<Day>\d+)-(?P<Hour>\d+)-(?P<Minute>\d+).log'
priority = ["Year", "Month", "Day", "Hour", "Minute"]
[FxaKafkaOutput]
type = "KafkaOutput"
message_matcher = "TRUE"
topic = "logs"
addrs = ["10.100.100.37:9092"]
encoder = "ProtobufEncoder"
partitioner = "RoundRobin"
[DashboardOutput]
ticker_interval = 5
============================================================
[2] Consumer config
[hekad]
maxprocs = 4
base_dir = "/home/heka/heka"
share_dir = "/home/heka/heka/share/heka"
plugin_chansize = 300
max_process_inject = 40
poolsize = 2000
[KafkaInput1]
type = "KafkaInput"
topic = "logs"
addrs = ["10.100.100.37:9092"]
splitter = "KafkaSplitter"
decoder = "ProtobufDecoder"
[KafkaSplitter]
type = "NullSplitter"
use_message_bytes = true
[DashboardOutput]
ticker_interval = 5
--
Mathieu Chouquet-Stringer m+h...@thi.eu.com
The sun itself sees not till heaven clears.
-- William Shakespeare --
_______________________________________________
Heka mailing list
Heka@mozilla.org
https://mail.mozilla.org/listinfo/heka
