On Tue, Aug 28, 2018 at 17:12 +0200, Dominik Charousset wrote:
> 1) Matthias threw in memory-mapping, but I’m not so sure if this is
> actually feasible for you.
Yeah, our normal use case is different, memory-mapping won't help much
with that.
> 2) CAF already does batching. Ideally, Broker
>> Okay. In the future, we probably need some form of
>> "serialization-free" batching mechanism to ship data more efficiently.
>
> Do you guys have a sense of how load splits up between serialization
> and batching/communication? My hope has been that batching itself can
> take care of the
On Sat, Aug 25, 2018 at 17:42 +0200, Matthias Vallentin wrote:
> Okay. In the future, we probably need some form of
> "serialization-free" batching mechanism to ship data more efficiently.
Do you guys have a sense of how load splits up between serialization
and batching/communication? My hope
> Agree. Right now a newly connecting peer gets a round of explicit
> LogCreates, but that's probably not the best way forward for larger
> topologies.
Okay. In the future, we probably need some form of
"serialization-free" batching mechanism to ship data more efficiently.
There exist
On Fri, Aug 24, 2018 at 16:32 +0200, Matthias Vallentin wrote:
> It sounds like this is critical also for regular operation:
Agree. Right now a newly connecting peer gets a round of explicit
LogCreates, but that's probably not the best way forward for larger
topologies.
> is it currently
> I don't really see a way around that without substantially increasing
> volume. We could send LogCreate updates regularly, so that it's easier
> to synchronize with an ongoing stream.
It sounds like this is critical also for regular operation: (1) when
an endpoint bootstraps slowly and the
> Dominik, wasn't the original idea for VAST to provide an event
> description language that would create the link between the values
> coming over the wire and their interpretation? Such a specification
> could be auto-generated from Bro's knowledge about the events it
> generates.
We were
On Thu, Aug 23, 2018 at 10:01 -0500, Jonathan Siwek wrote:
> Yeah, that's one problem, but a bigger issue is you can't parse
> LogWrite because the content is a serial blob whose format is another
> thing not intended for public consumption.
I guess my earlier comment might have been
On Thu, Aug 23, 2018 at 15:32 +0200, Dominik Charousset wrote:
> Does that mean I need to receive the LogCreate even first to
> understand successive LogWrite events?
I don't really see a way around that without substantially increasing
volume. We could send LogCreate updates regularly, so
On Thu, Aug 23, 2018 at 8:32 AM Dominik Charousset
wrote:
> I’m a bit hesitant to rely on this header at the moment, because of:
>
> /// A Bro log-write message. Note that at the moment this should be used only
> /// by Bro itself as the arguments aren't publicly defined.
>
> Is the API stable
On Tue, Aug 21, 2018 at 14:05 -0500, Jonathan Siwek wrote:
> Though the Broker data corresponding to log entry content is also
> opaque at the moment (I recall that was maybe for performance or
> message volume optimization),
Yeah, but generally this is something I could see opening up. The
On Tue, Aug 21, 2018 at 1:09 PM Robin Sommer wrote:
> Also, this question is about events, not logs, right? Logs have a
> different wire format and they actually come with meta data describing
> their columns.
Though the Broker data corresponding to log entry content is also
opaque at the
On Tue, Aug 21, 2018 at 12:34 -0500, Jonathan Siwek wrote:
> Maybe there's a more standardized approach that could be worked
> towards, but likely we just need more experience in understanding and
> defining common use-cases for external Bro data consumption.
Dominik, wasn't the original idea
On Tue, Aug 21, 2018 at 8:54 AM Dominik Charousset
wrote:
> This raises a couple of questions. Primarily: where can Broker users learn
> the layouts to interpret received data?
broker/bro.hh is basically all there is right now. e.g. if you
construct a broker::bro::Event from a received
We are currently writing code for ingesting data directly using Broker’s API.
From the docs, it seems that Broker assumes that publishers and subscribers
somehow agree on one layout per topic: "senders and receivers will need to
agree on a specific data layout for the values exchanged, so that
15 matches
Mail list logo
