On Tue, Aug 28, 2018 at 17:12 +0200, Dominik Charousset wrote:
> 1) Matthias threw in memory-mapping, but I’m not so sure if this is
> actually feasible for you.
Yeah, our normal use case is different, memory-mapping won't help much
with that.
> 2) CAF already does batching. Ideally, Broker should not need to do
> any additional batching on top of that.
Yep, but (3) was the problem with that:
> Do you still remember what showed up during your investigation that
> triggered you to go with the blob?
Looking back through emails, at some point Jon replaced CAF
serialization with these blobs and got substantially better
performance. He also had a patch that reproduced the effect with the
benchmark tool you wrote. I'm pasting that in below, I'm assuming it
still applies. Looks like the conclusion at that time was that it is
indeed an issue with the serialization and/or copying the data.
> An in-depth performance analysis of Broker’s streaming layer is on my
> todo list for months at this point. I hope I get something done before
> the Bro Workshop in Europe.
That would be great. :)
Robin
```
diff --git a/tests/benchmark/broker-stream-benchmark.cc
b/tests/benchmark/broker-stream-benchmark.cc
index 821ac39..26b0778 100644
--- a/tests/benchmark/broker-stream-benchmark.cc
+++ b/tests/benchmark/broker-stream-benchmark.cc
@@ -1,6 +1,7 @@
#include <iostream>
#include <broker/broker.hh>
+#include <broker/bro.hh>
using std::cout;
using std::cerr;
@@ -55,8 +56,11 @@ void publish_mode(broker::endpoint& ep, const std::string&
topic_str) {
// nop
},
[=](caf::unit_t&, downstream<std::pair<topic, data>>& out, size_t num) {
- for (size_t i = 0; i < num; ++i)
- out.push(std::make_pair(topic_str, "Lorem ipsum dolor sit amet."));
+ for (size_t i = 0; i < num; ++i) {
+ auto ev = broker::bro::Event(std::string("event_1"),
+ std::vector<broker::data>{42, "test"});
+ out.push(std::make_pair(topic_str, std::move(ev)));
+ }
global_count += num;
},
[=](const caf::unit_t&) {
```
--
Robin Sommer * Corelight, Inc. * ro...@corelight.com * www.corelight.com
_______________________________________________
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
