On 27/08/17 04:03, Seth Hall wrote: > I believe that Robin meant the intel framework instead of sumstats. > (Hopefully this avoids some confusion)
Thanks for the clarification! I was thinking about send_id() in context of the intel framework as well. As you might noticed, I enjoyed playing around with the intel framework :) Thus, some questions to make sure I got everything correctly: > On Sat, Aug 26, 2017 at 11:12 AM Robin Sommer <[email protected]> wrote: > >> Jon, replacing send_id() may indeed work better with an extension at >> the C++/Broker level. I'd like to avoid introducing new dependencies >> on Bro's serialization code, as I'm very much hoping that once the old >> communication code code goes we won't need that serialization layer >> anymore either (I know we're using it for opaque values over Broker >> too, but that's quite contained and should be easy to replace). So sending opaque values will still be possible using broker, right? >> - There's one larger problem with replacing send_id() though: the >> old communication system has logic to send large values >> incrementally, so that send_id() won't block stuff. As Seth >> reminded me the SumStats framework is relying on that quite >> extensively for sending tables around. Incremental operation is >> something we don't have with Broker. I think that's ok, we can >> replace the few existing use cases of sending large values with >> something else. For SumStats that should probably be data >> stores. As far as I understand the broker data stores (straight forward key-value stores), a data store does not fit for the intel framework, as it uses e.g. the patricia-trie implementation in tables to efficiently match subnets. Additionally I was thinking about using cuckoo-filters, implemented as opaque type, to further improve matching on workers. However, the intel framework uses send_id() only initially to transfer the current min_data_store to newly connected workers. Every further update is handled "manually". Thus I guess there would be two options: 1. Sending all data at once. Maybe ok for that use case. 2. Sending stuff incrementally using some script-layer logic. Am I right here? Thanks, Jan _______________________________________________ bro-dev mailing list [email protected] http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
