> Agree. Right now a newly connecting peer gets a round of explicit > LogCreates, but that's probably not the best way forward for larger > topologies.
Okay. In the future, we probably need some form of "serialization-free" batching mechanism to ship data more efficiently. There exist technologies like Apache Arrow, flatbuffers, Cap'N'Proto, MsgPack, etc., all of which require building a set of values once, and then just copying them around as a binary blob on the wire. Deserialization is not needed because one would typically only "view" the data through light-weight accessors. We're doing something similar in VAST for performance reasons, but Bro and Broker have the exact same issues in that regard. > > In other words, can Broker currently be used if one writes a Bro > > script that publishes plain events (message type 1 in bro.hh)? > > Yes to that. Non-Bros can exchange events (assuming they know the > schema), but not logs. Got it. (Unfortunately that will make our BroCon talk pretty boring in terms of throughput analysis, because we were planning to build an end-to-end log ingestion system based on Broker. We'll probably switch gears a bit and focus more on the latency side, where a Bro script publishes something to an external application and receives feedback though an auxiliary channel.) Matthias _______________________________________________ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev