> On Nov 2, 2017, at 5:33 PM, Azoff, Justin S <[email protected]> wrote:
>
> The optimization could be built into broker though, something like
>
> Broker::broadcast_magic_once_whatever(Cluster::worker_pool, key, SSL::
> intermediate_add, key, value);
>
> That would hash the key, send it to a data node, then have the data node
> broadcast the
> event
The first thing I’d suggest is a new internal broker message type that can
publish a message via a topic and then on any receiving nodes re-publish via a
second topic. In this way, we could distribute to all workers via a single
proxy/data node as:
local one_proxy_topic = Cluster::rr_topic(Cluster::proxy_pool,
“ssl/intermediate_add”));
local e = Broker::make_event(SSL::intermediate_add, key, value);
Broker::relay(one_proxy_topic, Cluster::worker_topic, e);
Or potentially compressed into one call:
Cluster::relay(first_topic, second_topic, rr_key, event, varargs...);
> while adding key to a 'recently broadcasted keys' table that only needs to
> buffer for 10s or so.
>
> This would enable you to efficiently broadcast an event (once) across all
> workers with a single line of code.
I’m not sure about this part. Is that just for throttling potential
duplicates? Is that something you want generally or just for the this
particular example? I’m thinking maybe it can wait until we know we have
several places where it’s actually needed/used before compressing that pattern
into a single BIF/function.
> I guess my whole point with all of this is that if the intent of a script is
> that an event should be seen on all workers, the script should look something
> like
>
> Broker::broadcast(Cluster::worker_pool, SSL:: intermediate_add, key,
> value);
>
> and not have a bunch of redefs and @ifs so that the script can eventually have
>
> event SSL::new_intermediate(key, value);
Yeah, minimizing the need for people to have to constantly implement various
code paths that depend on cluster/node-type is a good goal and what I’m also
aiming at.
>>> How would something like policy/protocols/ssl/validate-certs.bro look with
>>> intermediate_cache as a data store?
>>
>> global intermediate_store: Cluster::StoreInfo;
>>
>> event bro_init()
>> {
>> intermediate_store =
>> Cluster::create_store(“ssl/validate-certs/intermediate_store");
>> }
>>
>> And then port the rest of that script to use broker data store api
>> (get/put/exists calls) to access that store.
>>
>> - Jon
>
> Does that have the same performance profile as the current method?
>
> if (issuer in intermediate_cache)
>
> vs
>
> Broker::get(intermediate_cache, issuer)
Theoretically, they’re both in-memory hash-table lookups, though the
implementations are obviously very different and I don’t know how they compare
in reality.
I think it’s true that many scripts could be ported to use either data stores
or just explicitly exchange events. Probably the preference to use a data
store would be for cases where it may be useful to keep persistent data across
crashes/restarts.
- Jon
_______________________________________________
bro-dev mailing list
[email protected]
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
