What sort of throughput are you seeing? On 04/14/2015 10:53 PM, Antonin Kral wrote:
Hi Rob and Gil,I am trying to do something similar, even in smaller scale. My configuration looks something like: [ModgenKafkaInput0] type = "KafkaInput" topic = "logs" addrs = ["kafka_tmp.modgen.net:9092"] splitter = "KafkaSplitter" decoder = "ProtobufDecoder" group = "kafka-client-group01" partition = 0 event_buffer_size = 512 max_open_reqests = 8 default_fetch_size = 65536 [ModgenKafkaInput1] type = "KafkaInput" topic = "logs" addrs = ["kafka_tmp.modgen.net:9092"] splitter = "KafkaSplitter" decoder = "ProtobufDecoder" group = "kafka-client-group01" partition = 1 event_buffer_size = 512 max_open_reqests = 8 default_fetch_size = 65536 [KafkaSplitter] type = "NullSplitter" use_message_bytes = true [ESJsonEncoder] es_index_from_timestamp = true type_name = "%{Type}" [ElasticSearchOutput] server = "http://localhost:9200"; message_matcher = "Type !~ /^heka/" encoder = "ESJsonEncoder" flush_interval = 100 # in ms flush_count = 50 use_buffering = true queue_max_buffer_size = 102400000 queue_full_action = "shutdown" I am currently observing quite low throughput though. Not rely sure why, but it seems that problem is between kafka and heka. Best, Antonin * Rob Miller <[email protected]> [2015-04-15 07:41] wrote: > On 04/14/2015 12:48 PM, Gil Fliker wrote: >> Thx for the quick response, >> >> I am not yet familiar with all of heka's futures specifically with >> "message_matcher". > That's one of Heka's fundamental concepts, please see http://hekad.readthedocs.org/en/v0.9.1/index.html, http://hekad.readthedocs.org/en/v0.9.1/getting_started.html and http://hekad.readthedocs.org/en/v0.9.1/message_matcher.html. >> Let me just add that the reasoning behind the high number of partitions >> is to enable parallelism to support the throughput needed. >> >> Can you please point me in a direction for a similar heka example ? > Sorry, there's no existing example that I can point you to at the moment. We're happy to answer specific questions, to the extent we're able, but every massively parallel data processing infrastructure is going to be different, you're going to have to get familiar with the building blocks that Heka provides and drill down a bit before you'll be able to get a useful response. :) > > -r > >> >> >> Thx >> >> >> >> >> Gil Fliker >> >> >> On Tue, Apr 14, 2015 at 3:22 PM, Rob Miller <[email protected] >> <mailto:[email protected]>> wrote: >> >> Yes, currently a single KafkaInput can only pull from a single Kafka >> partition. You can think of Heka's KafkaInput as analogous to a >> SimpleConsumer (see >> https://cwiki.apache.org/__confluence/display/KAFKA/0.8.__0+SimpleConsumer+Example >> <https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+SimpleConsumer+Example>). >> >> If you want to manage inter-partition coordination, along the lines >> of what is described as a "High Level Consumer" >> (https://cwiki.apache.org/__confluence/display/KAFKA/__Consumer+Group+Example >> <https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Group+Example>), >> you'd handle that at the filter layer. For instance, you might set >> up a filter plugin with a message_matcher constructed such that it >> catches all of the messages from a single topic, regardless of >> partition, and perform any necessary correlations therein. The >> delivery semantics to this filter would match that described on the >> consumer group example page linked above, i.e. all of the messages >> from a single partition will be received in the correct order, but >> the messages from across partitions would be non-deterministically >> interleaved. >> >> If there are so many partitions carrying so much data that a single >> Heka instance can't handle them all, then you might have to have one >> box handling one subset of partitions, another box processing a >> different subset, and each of *those* in turn feeding into a third >> box that performs the next level of correlation. >> >> In other words, the building blocks are there, but you have to >> actually use them to put together a more sophisticated system. We're >> unfortunately not yet at the point where there are higher level >> constructs that will automatically distribute load for you. >> >> Hope this helps! >> >> -r >> >> >> >> On 04/10/2015 02:21 PM, Gil Fliker wrote: >> >> Hi, >> >> We are about to start a poc using Heka. >> >> The plan is to pipe messages via Kafka transport and Heka being the >> endpoints speaking http with various producers and consumers. >> >> I saw in the documentation that you have to specify a partition >> number >> and only one partition number ? >> >> Our Kafka topic setup will be made of around 1000 partitions. >> >> What is the best way to approach this ? >> >> >> Thx >> >> >> Gil Fliker >> >> Outbrain Operations Manager >> >> The above terms reflect a potential business arrangement, are >> provided >> solely as a basis for further discussion, and are not intended >> to be and >> do not constitute a legally binding obligation. No legally binding >> obligations will be created, implied, or inferred until an >> agreement in >> final form is executed in writing by all parties involved. >> >> This email and any attachments hereto may be confidential or >> privileged. >> If you received this communication by mistake, please don't >> forward it >> to anyone else, please erase all copies and attachments, and >> please let >> me know that it has gone to the wrong person. Thanks. >> >> >> _________________________________________________ >> Heka mailing list >> [email protected] <mailto:[email protected]> >> https://mail.mozilla.org/__listinfo/heka >> <https://mail.mozilla.org/listinfo/heka> >> >> >> >> >> The above terms reflect a potential business arrangement, are provided >> solely as a basis for further discussion, and are not intended to be and >> do not constitute a legally binding obligation. No legally binding >> obligations will be created, implied, or inferred until an agreement in >> final form is executed in writing by all parties involved. >> >> This email and any attachments hereto may be confidential or privileged. >> If you received this communication by mistake, please don't forward it >> to anyone else, please erase all copies and attachments, and please let >> me know that it has gone to the wrong person. Thanks. > > _______________________________________________ > Heka mailing list > [email protected] > https://mail.mozilla.org/listinfo/heka >
_______________________________________________ Heka mailing list [email protected] https://mail.mozilla.org/listinfo/heka
