> On Nov 2, 2017, at 5:21 PM, Siwek, Jon <jsi...@illinois.edu> wrote: >> >> Mostly so that workers don't end up spending all their time sending out >> messages when they should be analyzing packets. > > Ok, I get what you want to avoid, though could be interesting to actually > have a fully-connected cluster in order to collect performance data on each > comm. pattern and see how significant the difference is for a variety of > use-cases. > > Do you have a particular example you can give where you’d use this > BIF/function to relay a broadcast from a worker to all other workers via a > proxy?
Scripts like what validate-certs does to broadcast the presence of a new intermediate cert to the other nodes in the cluster. That script does have the added optimization on the manager side for only broadcasting the value once if a new cert is seen by two workers at the same time, so maybe it's not the best example for a broadcast. With explicit event destinations and load balancing across data nodes that script could look something like function add_to_cache(key: string, value: vector of opaque of x509) { # could still do @if ( Cluster::is_enabled() ), but in a standalone setup we are the data # node, so this would just short circuit to raise the event locally. I'd rather broker do # the @if internally than have every script have to have two implementations. Broker::publish_hrw(Cluster::data_pool, key, SSL::new_intermediate, key, value); } event SSL::new_intermediate(key: string, value: vector of opaque of x509) { if ( key in intermediate_cache ) return; intermediate_cache[key] = value; # in a standalone setup this would just be a NOOP Broker::broadcast(Cluster::worker_pool, SSL:: intermediate_add, key, value); } event SSL::intermediate_add(key: string, value: vector of opaque of x509) { intermediate_cache[key] = value; } Without the added optimization you'd just have function add_to_cache(key: string, value: vector of opaque of x509) { intermediate_cache[key] = value; # in a standalone setup this would just be a NOOP Broker::broadcast(Cluster::worker_pool, SSL:: intermediate_add, key, value); } event SSL::intermediate_add(key: string, value: vector of opaque of x509) { intermediate_cache[key] = value; } The optimization could be built into broker though, something like Broker::broadcast_magic_once_whatever(Cluster::worker_pool, key, SSL:: intermediate_add, key, value); That would hash the key, send it to a data node, then have the data node broadcast the event while adding key to a 'recently broadcasted keys' table that only needs to buffer for 10s or so. This would enable you to efficiently broadcast an event (once) across all workers with a single line of code. In either case all of the manager2worker_events worker2manager_events @if ( Cluster::is_enabled() && Cluster::local_node_type() != Cluster::MANAGER ) Is no longer needed. I guess my whole point with all of this is that if the intent of a script is that an event should be seen on all workers, the script should look something like Broker::broadcast(Cluster::worker_pool, SSL:: intermediate_add, key, value); and not have a bunch of redefs and @ifs so that the script can eventually have event SSL::new_intermediate(key, value); >> How would something like policy/protocols/ssl/validate-certs.bro look with >> intermediate_cache as a data store? > > global intermediate_store: Cluster::StoreInfo; > > event bro_init() > { > intermediate_store = > Cluster::create_store(“ssl/validate-certs/intermediate_store"); > } > > And then port the rest of that script to use broker data store api > (get/put/exists calls) to access that store. > > - Jon Does that have the same performance profile as the current method? if (issuer in intermediate_cache) vs Broker::get(intermediate_cache, issuer) — Justin Azoff _______________________________________________ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev