> On Nov 2, 2017, at 5:21 PM, Siwek, Jon <[email protected]> wrote:
>>
>> Mostly so that workers don't end up spending all their time sending out
>> messages when they should be analyzing packets.
>
> Ok, I get what you want to avoid, though could be interesting to actually
> have a fully-connected cluster in order to collect performance data on each
> comm. pattern and see how significant the difference is for a variety of
> use-cases.
>
> Do you have a particular example you can give where you’d use this
> BIF/function to relay a broadcast from a worker to all other workers via a
> proxy?
Scripts like what validate-certs does to broadcast the presence of a new
intermediate cert to the other nodes in the cluster.
That script does have the added optimization on the manager side for only
broadcasting the value once if a new cert is seen by two workers at the same
time,
so maybe it's not the best example for a broadcast.
With explicit event destinations and load balancing across data nodes that
script could look something like
function add_to_cache(key: string, value: vector of opaque of x509)
{
# could still do @if ( Cluster::is_enabled() ), but in a standalone setup
we are the data
# node, so this would just short circuit to raise the event locally. I'd
rather broker do
# the @if internally than have every script have to have two
implementations.
Broker::publish_hrw(Cluster::data_pool, key, SSL::new_intermediate, key,
value);
}
event SSL::new_intermediate(key: string, value: vector of opaque of x509)
{
if ( key in intermediate_cache )
return;
intermediate_cache[key] = value;
# in a standalone setup this would just be a NOOP
Broker::broadcast(Cluster::worker_pool, SSL:: intermediate_add, key, value);
}
event SSL::intermediate_add(key: string, value: vector of opaque of x509)
{
intermediate_cache[key] = value;
}
Without the added optimization you'd just have
function add_to_cache(key: string, value: vector of opaque of x509)
{
intermediate_cache[key] = value;
# in a standalone setup this would just be a NOOP
Broker::broadcast(Cluster::worker_pool, SSL:: intermediate_add, key, value);
}
event SSL::intermediate_add(key: string, value: vector of opaque of x509)
{
intermediate_cache[key] = value;
}
The optimization could be built into broker though, something like
Broker::broadcast_magic_once_whatever(Cluster::worker_pool, key, SSL::
intermediate_add, key, value);
That would hash the key, send it to a data node, then have the data node
broadcast the
event while adding key to a 'recently broadcasted keys' table that only needs
to buffer for 10s or so.
This would enable you to efficiently broadcast an event (once) across all
workers with a single line of code.
In either case all of the
manager2worker_events
worker2manager_events
@if ( Cluster::is_enabled() && Cluster::local_node_type() != Cluster::MANAGER
)
Is no longer needed.
I guess my whole point with all of this is that if the intent of a script is
that an event should be seen on all workers, the script should look something
like
Broker::broadcast(Cluster::worker_pool, SSL:: intermediate_add, key, value);
and not have a bunch of redefs and @ifs so that the script can eventually have
event SSL::new_intermediate(key, value);
>> How would something like policy/protocols/ssl/validate-certs.bro look with
>> intermediate_cache as a data store?
>
> global intermediate_store: Cluster::StoreInfo;
>
> event bro_init()
> {
> intermediate_store =
> Cluster::create_store(“ssl/validate-certs/intermediate_store");
> }
>
> And then port the rest of that script to use broker data store api
> (get/put/exists calls) to access that store.
>
> - Jon
Does that have the same performance profile as the current method?
if (issuer in intermediate_cache)
vs
Broker::get(intermediate_cache, issuer)
—
Justin Azoff
_______________________________________________
bro-dev mailing list
[email protected]
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
