Re: [Bro-Dev] [Bro-Commits] [git/bro] topic/actor-system: First-pass broker-enabled Cluster scripting API + misc. (07ad06b)

Sounds good to me. We should probably label the new parts experimental for now, as I'm sure we'll iterate some more as people get experience with them. Robin On Wed, Nov 08, 2017 at 18:46 +, you wrote: > Just a quick summary of key points of this thread related to cluster-layout, >

Re: [Bro-Dev] [Bro-Commits] [git/bro] topic/actor-system: First-pass broker-enabled Cluster scripting API + misc. (07ad06b)

Just a quick summary of key points of this thread related to cluster-layout, messaging patterns, and API (omitting some minor stuff from Robin’s initial feedback). - "proxy" nodes will be renamed at a later point toward the end of the project ("proxy" actually makes sense to me, but "data"

Re: [Bro-Dev] [Bro-Commits] [git/bro] topic/actor-system: First-pass broker-enabled Cluster scripting API + misc. (07ad06b)

> 2) Let the developer specify constraints for the data service > distribution across data nodes and automatize the optimization. The > minimal example would be that for each data service a minimum and > maximum or default number of data nodes is specified (e.g. Intel on 1-2 > nodes and Scan

Re: [Bro-Dev] [Bro-Commits] [git/bro] topic/actor-system: First-pass broker-enabled Cluster scripting API + misc. (07ad06b)

On 03/11/17 21:05, Azoff, Justin S wrote: > I've been thinking the same thing, but I hope it doesn't come to that. > Ideally people will be able > to scale their clusters by just increasing the number of data nodes without > having to get into > the details about what node is doing what. > >

Re: [Bro-Dev] [Bro-Commits] [git/bro] topic/actor-system: First-pass broker-enabled Cluster scripting API + misc. (07ad06b)

> On Nov 3, 2017, at 2:13 PM, Jan Grashöfer wrote: > > Fully agreed! In that case it might be nice if one can define separate > special purpose data nodes, e.g. "intel data nodes". But, I am not sure > whether this is a good idea as this might lead to complex cluster

Re: [Bro-Dev] [Bro-Commits] [git/bro] topic/actor-system: First-pass broker-enabled Cluster scripting API + misc. (07ad06b)

On 03/11/17 18:07, Azoff, Justin S wrote:> Partitioning the intel data set is a little tricky since it supports subnets and hashing 10.10.0.0/16 > and 10.10.10.10 won't necessarily give you the same node. Maybe subnets need > to exist on all > nodes but everything else can be partitioned?

Re: [Bro-Dev] [Bro-Commits] [git/bro] topic/actor-system: First-pass broker-enabled Cluster scripting API + misc. (07ad06b)

> On Nov 3, 2017, at 6:51 AM, Jan Grashöfer wrote: > > At this point, if the manager functionality is distributed across > multiple data nodes, we have to make sure, that every data node has the > right part of the DataStore to deal with the incoming hit. One could

Re: [Bro-Dev] [Bro-Commits] [git/bro] topic/actor-system: First-pass broker-enabled Cluster scripting API + misc. (07ad06b)

> On Nov 3, 2017, at 5:51 AM, Jan Grashöfer wrote: > > And just to avoid misunderstandings: We won't be able to get rid of the > @if (Cluster::local_node_type() != Cluster::MANAGER/DATANODE) > statements completely as different node types have different >

Re: [Bro-Dev] [Bro-Commits] [git/bro] topic/actor-system: First-pass broker-enabled Cluster scripting API + misc. (07ad06b)

On 02/11/17 23:58, Azoff, Justin S wrote: > For an example of a purely broadcast use case, see > > scripts/base/frameworks/intel/cluster.bro > > You can see the crazy amount of complexity around the Intel::cluster_new_item > event. That's right! Took me some time to figure out how data should

Re: [Bro-Dev] [Bro-Commits] [git/bro] topic/actor-system: First-pass broker-enabled Cluster scripting API + misc. (07ad06b)

> On Nov 2, 2017, at 5:33 PM, Azoff, Justin S wrote: > > The optimization could be built into broker though, something like > >Broker::broadcast_magic_once_whatever(Cluster::worker_pool, key, SSL:: > intermediate_add, key, value); > > That would hash the key, send

Re: [Bro-Dev] [Bro-Commits] [git/bro] topic/actor-system: First-pass broker-enabled Cluster scripting API + misc. (07ad06b)

> On Nov 2, 2017, at 5:54 PM, Siwek, Jon wrote: > > Thanks, though I’m not sure this scenario maps well to this particular point. > E.g. my impression is Justin wants a single BIF/function that can send one > event from a worker to a proxy and have the proxy purely relay

Re: [Bro-Dev] [Bro-Commits] [git/bro] topic/actor-system: First-pass broker-enabled Cluster scripting API + misc. (07ad06b)

> On Nov 2, 2017, at 5:21 PM, Siwek, Jon wrote: >> >> Mostly so that workers don't end up spending all their time sending out >> messages when they should be analyzing packets. > > Ok, I get what you want to avoid, though could be interesting to actually > have a

Re: [Bro-Dev] [Bro-Commits] [git/bro] topic/actor-system: First-pass broker-enabled Cluster scripting API + misc. (07ad06b)

> On Nov 2, 2017, at 1:37 PM, Aashish Sharma wrote: > >>> In the case of broadcasting from a worker to all other workers, the reason >>> why you relay via another node is only because workers are not connected to >>> each other? Do we know that a fully-connected cluster is a

Re: [Bro-Dev] [Bro-Commits] [git/bro] topic/actor-system: First-pass broker-enabled Cluster scripting API + misc. (07ad06b)

> On Nov 2, 2017, at 12:58 PM, Azoff, Justin S wrote: > > >> On Nov 2, 2017, at 1:22 PM, Siwek, Jon wrote: >> >> >>> On Nov 1, 2017, at 6:11 PM, Azoff, Justin S wrote: >>> >>> - a bif/function for efficiently broadcasting an

Re: [Bro-Dev] [Bro-Commits] [git/bro] topic/actor-system: First-pass broker-enabled Cluster scripting API + misc. (07ad06b)

> On Nov 2, 2017, at 2:37 PM, Aashish Sharma wrote: > > > > Now, while Justins' multiple data nodes idea has specticular merits, I am not > much fan of it. Reason being having multiple data-notes results in same sets > of problems It does not have the same problems.. It

Re: [Bro-Dev] [Bro-Commits] [git/bro] topic/actor-system: First-pass broker-enabled Cluster scripting API + misc. (07ad06b)

My view: I have again and again encountered 4 types cases while doing script/pkg work: 1) manager2worker: Input-framework reads external data and all workers need to see it. examples: intel-framework, 2) worker2manager: workers see something report to manager, manager keeps aggregated

Re: [Bro-Dev] [Bro-Commits] [git/bro] topic/actor-system: First-pass broker-enabled Cluster scripting API + misc. (07ad06b)

> On Nov 2, 2017, at 1:22 PM, Siwek, Jon wrote: > > >> On Nov 1, 2017, at 6:11 PM, Azoff, Justin S wrote: >> >> - a bif/function for efficiently broadcasting an event to all other workers >> (or data nodes) >> - If the current node is a data node,

Re: [Bro-Dev] [Bro-Commits] [git/bro] topic/actor-system: First-pass broker-enabled Cluster scripting API + misc. (07ad06b)

> On Nov 1, 2017, at 6:11 PM, Azoff, Justin S wrote: > > - a bif/function for efficiently broadcasting an event to all other workers > (or data nodes) > - If the current node is a data node, just send it to all workers > - otherwise, round robin the event to a data

Re: [Bro-Dev] [Bro-Commits] [git/bro] topic/actor-system: First-pass broker-enabled Cluster scripting API + misc. (07ad06b)

> On Nov 1, 2017, at 5:23 PM, Robin Sommer wrote: > > Justin, correct me if I'm wrong, but I don't think this has ever been > fully fleshed out. If anybody wants to propose something specific, we > can discuss, otherwise I would suggest we stay with the minimum for > now that

Re: [Bro-Dev] [Bro-Commits] [git/bro] topic/actor-system: First-pass broker-enabled Cluster scripting API + misc. (07ad06b)

On Tue, Oct 31, 2017 at 22:35 +, you wrote: > My thought was they can conceptually still be used for the same type > of stuff: data sharing and offloading other misc. > analysis/calculation. Yeah, agree that we want such nodes, however I would like to switch away from the proxy name.

Re: [Bro-Dev] [Bro-Commits] [git/bro] topic/actor-system: First-pass broker-enabled Cluster scripting API + misc. (07ad06b)

> On Oct 31, 2017, at 1:16 PM, Robin Sommer wrote: > >- One thing I can't quite tell is if this is still aiming to > maintain compatibility with the old communication system, like > by keeping the proxies and also the *_events patterns. Looking > at

Re: [Bro-Dev] [Bro-Commits] [git/bro] topic/actor-system: First-pass broker-enabled Cluster scripting API + misc. (07ad06b)

This is coming together quite nicely. Not sure if it's stable yet, but I'll just go ahead with some feedback I noticed looking over the new cluster API: - One thing I can't quite tell is if this is still aiming to maintain compatibility with the old communication system, like by