Re: [Bro-Dev] Config Framework Feedback

2018-10-30 Thread Azoff, Justin S
> On Oct 30, 2018, at 2:09 PM, Hosom, Stephen M wrote: > > I bumped into a limitation that was a little frustrating to work around with > the config framework. > > > It is inconvenient that values stored in files read by adding to > Config::config_files are not available within the bro_init

Re: [Bro-Dev] 2.6-beta build error

2018-09-21 Thread Azoff, Justin S
> On Sep 21, 2018, at 11:24 AM, Jon Siwek wrote: > > On Thu, Sep 20, 2018 at 2:14 PM Michael Dopheide wrote: >> >> bro.bif:3630:47: error: cannot convert ‘stat’ to ‘__dev_t {aka long unsigned >> int}’ in initialization > > Thanks, it's fixed by a simple patch [1] now in master. > > - Jon >

Re: [Bro-Dev] 2.6-beta build error

2018-09-21 Thread Azoff, Justin S
> On Sep 20, 2018, at 3:13 PM, Michael Dopheide wrote: > > > I'm hitting the same error when trying to build release/2.6 or master: > > [ 99%] In file included from /usr/local/src/bro/src/Func.cc:702:0: > bro.bif: In constructor ‘MMDB::MMDB(const char*, stat)’: > bro.bif:3630:47: error: cannot

Re: [Bro-Dev] Bro 2.6-beta plans

2018-09-14 Thread Azoff, Justin S
> On Sep 13, 2018, at 5:57 PM, John Althouse wrote: > > Could anyone update Bro on docker to include 2.5.5, 2.6-beta and master? > https://hub.docker.com/r/broplatform/bro/tags/ > > We use this with our internal trybro instance which is fantastic for quickly > collaborating and testing script

Re: [Bro-Dev] Bro 2.6-beta plans

2018-09-12 Thread Azoff, Justin S
On Sep 5, 2018, at 6:35 PM, Jon Siwek mailto:jsi...@corelight.com>> wrote: There's no significant code changes/features planned to get added to the master branch from now until the 2.6-beta gets released (maybe in about a week). Until that happens, please help test the latest master branch and

Re: [Bro-Dev] Bro 2.6-beta plans

2018-09-10 Thread Azoff, Justin S
> On Sep 7, 2018, at 4:41 PM, Azoff, Justin S wrote: > > Before, cpu maxed out but spending 60% in user and 30% in system > After, cpu maxed out but spending 12% in user and 80% in system > I did some more testing and profiling and figured out what is going on.. The new versi

Re: [Bro-Dev] Bro 2.6-beta plans

2018-09-08 Thread Azoff, Justin S
> On Sep 8, 2018, at 11:20 AM, Azoff, Justin S wrote: > > Do many of those options do anything? I tried looking in the CAF source to > figure out how they are used, and it looks like they are all defined in > libcaf_core/caf/actor_system_config.hpp as Scratch that.. the dep

Re: [Bro-Dev] Bro 2.6-beta plans

2018-09-08 Thread Azoff, Justin S
> On Sep 7, 2018, at 6:58 PM, Jon Siwek wrote: > > On Fri, Sep 7, 2018 at 3:41 PM Azoff, Justin S wrote: > >> One thing I'm still seeing when I switch from an old version to latest >> master is that huge spike >> in Content switches/interrupts and cpu spent

Re: [Bro-Dev] Bro 2.6-beta plans

2018-09-07 Thread Azoff, Justin S
> On Sep 6, 2018, at 7:40 PM, Jon Siwek wrote: > > On Thu, Sep 6, 2018 at 3:40 PM Azoff, Justin S wrote: > >> 8842 >> broker::topic+broker::internal_comma...@u32.bro/known/certs/<$>/data/clone > > Thanks, there was an unintended forwarding loop in dat

Re: [Bro-Dev] Bro 2.6-beta plans

2018-09-06 Thread Azoff, Justin S
> On Sep 6, 2018, at 4:19 PM, Jon Siwek wrote: > > On Thu, Sep 6, 2018 at 3:14 PM Azoff, Justin S wrote: > > >> I tested an almost stock local.bro (a few additional things disabled) and >> saw the same thing. >> >> fa7fa5aa is fine, but with 452eb0cb

Re: [Bro-Dev] Bro 2.6-beta plans

2018-09-06 Thread Azoff, Justin S
> On Sep 6, 2018, at 3:41 PM, Azoff, Justin S wrote: > > I just got 2 clusters upgraded from > > fa7fa5aa to > 452eb0cb > > > I may have created a message loop replacing the relay_rr stuff, but it's kind > of hard to tell. > > I'll do some m

Re: [Bro-Dev] Bro 2.6-beta plans

2018-09-06 Thread Azoff, Justin S
> On Sep 5, 2018, at 6:35 PM, Jon Siwek wrote: > > There's no significant code changes/features planned to get added to > the master branch from now until the 2.6-beta gets released (maybe in > about a week). Until that happens, please help test the latest master > branch and provide any feedba

Re: [Bro-Dev] Bro 2.5 Packet Drop Issue

2018-08-30 Thread Azoff, Justin S
> On Aug 30, 2018, at 4:11 PM, Rajput, Jawad (CONTR) > wrote: > > Hello Everyone, > > I am reaching out with the hope that someone will be able to help us with an > issue we are having with Bro upgrade from 2.4.1 to 2.5.X. > > We have a system with 12 core (3Ghz) ,128GB RAM, and 10G NIC (

Re: [Bro-Dev] Compatibilty script for policy/protocols/smb?

2018-08-30 Thread Azoff, Justin S
> On Aug 30, 2018, at 2:26 PM, Jon Siwek wrote: > > On Thu, Aug 30, 2018 at 9:50 AM Azoff, Justin S wrote: > >> fatal error in /bro/share/bro/site/local.bro, line 88: can't open >> /bro/share/bro/policy/protocols/smb/__load__.bro >> >> I see in

[Bro-Dev] Compatibilty script for policy/protocols/smb?

2018-08-30 Thread Azoff, Justin S
Upgrading between master builds I just ran into this: fatal error in /bro/share/bro/site/local.bro, line 88: can't open /bro/share/bro/policy/protocols/smb/__load__.bro I see in NEWS we have - The SMB scripts in policy/protocols/smb are now moved into base/protocols/smb and loaded/enabled by

Re: [Bro-Dev] [Bro-Commits] [git/bro] topic/johanna/tls-more-data: Update NEWS for ssl changes. (3c7c60cf6)

2018-08-29 Thread Azoff, Justin S
> On Aug 29, 2018, at 12:02 PM, Johanna Amann wrote: > > @if ( version <= 2.6) > event 2.5-event > @else > event 2.6-event > @endif > > breaks with 2.5. Should that be < and not <= ? — Justin Azoff ___ bro-dev mailing list bro-dev@bro.org http://

Re: [Bro-Dev] Broker::publish API

2018-08-13 Thread Azoff, Justin S
> On Aug 13, 2018, at 12:24 PM, Jon Siwek wrote: > > Even Newer Worker: > > Broker::publish(Cluster::worker_topic, my_event); > > See any problems there? That's nice and simple :-) Assuming that can send the events around in the most efficient way possible, that's perfect. The one tricky

Re: [Bro-Dev] Broker::publish API

2018-08-10 Thread Azoff, Justin S
> On Aug 10, 2018, at 11:55 AM, Robin Sommer wrote: > > > Ok, let's make that change then, I think removing relay() will help > for sure making the API easier. If relay is removed how does a script writer efficiently get an event from one worker (or manager) to all of the other workers? — J

Re: [Bro-Dev] Broker::publish API

2018-08-08 Thread Azoff, Justin S
> On Aug 6, 2018, at 3:50 PM, Robin Sommer wrote: > >- Relaying is hardly used. > > >- There's a lot of checks in publishing code of the type "if I am > (not) of node type X". I think these 2 are somewhat related. Since there weren't higher level things like relaying, in order

Re: [Bro-Dev] Broker::publish API

2018-07-27 Thread Azoff, Justin S
> On Jul 27, 2018, at 6:10 PM, Jon Siwek wrote: > > On Fri, Jul 27, 2018 at 3:55 PM Azoff, Justin S wrote: > >> I do agree that there's room for a lot of simplification, for example a >> worker broadcasting a message efficiently to all >> other workers n

Re: [Bro-Dev] Broker::publish API

2018-07-27 Thread Azoff, Justin S
> On Jul 27, 2018, at 1:39 PM, Robin Sommer wrote: > > I'm wondering if we should give it another try to simply this API > while we still can (i.e., before 2.6 goes out). To me, the most > intuitive publish operation is "send to topic T and propagate to > everybody subscribed to that topic". I'd

Re: [Bro-Dev] [Bro-Commits] [git/bro] master: Removed a few more discovered UTF-8 characters in Bro scripts. (cd18d9620)

2018-06-15 Thread Azoff, Justin S
> On Jun 15, 2018, at 6:13 PM, Johanna Amann wrote: > > I actually am more concerned about the scripts that we ship with Bro itself > :). Though it is nice to have for packages too... > > Johanna Oh :-) I hadn't ran it on the scripts that come with bro.. it finds one non-ascii char: base/pr

Re: [Bro-Dev] [Bro-Commits] [git/bro] master: Removed a few more discovered UTF-8 characters in Bro scripts. (cd18d9620)

2018-06-15 Thread Azoff, Justin S
> On Jun 2, 2018, at 12:49 PM, Johanna Amann wrote: > > Hum. Would it make sense to introduce a test that checks all > script-files for non-ascii-characters? > > I can so see that happening again... > > Johanna > I added this as one of the checks that packages.bro.org does, corelight/bro-h

Re: [Bro-Dev] DHCP event removal

2018-06-15 Thread Azoff, Justin S
> On Jun 15, 2018, at 5:18 PM, Seth Hall wrote: > > On the > upside, you can handle both the old events and the new and they > shouldn't impact each other (if you want to make a script work on > multiple releases). I ran into this on a script I got from somewhere, bash-cve-2014-6271.bro The

Re: [Bro-Dev] Broker has landed in master, please test

2018-06-15 Thread Azoff, Justin S
> On Jun 1, 2018, at 11:37 AM, Azoff, Justin S wrote: > > I could never figure out what was causing the problem, and it's possible that > &synchronized not doing anything anymore is why it's better now. I'm mostly > using &synchronized for syncing inpu

Re: [Bro-Dev] Performance Issues after the fe7e1ee commit?

2018-06-06 Thread Azoff, Justin S
> On Jun 6, 2018, at 12:54 PM, McMullan, Tim wrote: > > We are running into performance issues (30x slower) since the Broker patch > (fe7e1ee) – > > We have 40G connections tapped from our storage filers feeding multiple Bro > instances which analyze specifically only NFS and SMB traffic; al

Re: [Bro-Dev] bro-devel package?

2018-05-24 Thread Azoff, Justin S
> On May 24, 2018, at 3:50 PM, Vlad Grigorescu wrote: > > There are a couple of cases where I think it'd be useful to have a bro-devel > package -- a package that I can install on a system, and then be able to > build plugins against Bro. (This is the same model as other *-devel packages, > s

Re: [Bro-Dev] Broker data store use case and questions

2018-05-14 Thread Azoff, Justin S
> On May 14, 2018, at 10:12 AM, Jon Siwek wrote: > > A short-lived cache, separate from the data store, still has problems like > the above: there can be times where the local cache contains the key and the > master store does not and so you may miss some (re)insertions. I see what you mean..

Re: [Bro-Dev] Broker data store use case and questions

2018-05-11 Thread Azoff, Justin S
> On May 11, 2018, at 10:13 AM, Jon Siwek wrote: > > > There's no check against the local cache to first see if the key exists > as going down that path leads to race conditions. What sort of race conditions? Right now I see a lot of events going around so it seems like there may be a bit o

Re: [Bro-Dev] Final Broker branch testing

2018-04-27 Thread Azoff, Justin S
> On Apr 26, 2018, at 4:25 PM, Azoff, Justin S wrote: > > Other than that things are working great. Cluster::publish_hrw is > distributing data cross proxies perfectly: > > # for x in 1 2 3; do broctl print Scan::attacks proxy-$x|grep attempts= > -c;done > 3304 >

Re: [Bro-Dev] Final Broker branch testing

2018-04-26 Thread Azoff, Justin S
> On Apr 26, 2018, at 11:16 AM, Jon Siwek wrote: > > The latest version of the new Broker-ized cluster/communication system > for Bro in 'topic/actor-system' branch is wrapping up and, in my > opinion, ready to be merged into Bro's 'master' branch. > [..] > > Though, for this round of testin

Re: [Bro-Dev] set and vector operators

2018-04-25 Thread Azoff, Justin S
> On Apr 25, 2018, at 1:40 PM, Vern Paxson wrote: > > I'm working on some scripts that use sets and vectors, sometimes together, > and am finding it clunky that Bro doesn't offer much in the way of operators > for this. To that end, I'm thinking of implementing some along the following > lines,

Re: [Bro-Dev] scheduling events vs using &expire_func ?

2018-04-19 Thread Azoff, Justin S
> On Apr 19, 2018, at 7:03 AM, Aashish Sharma wrote: > > aggregating across all proxies is still distributing data around. So the way I > see is you are moving the problem around :) But as I said, I don't know more > how > this works since I haven't tried new broker stuff just yet. I think th

Re: [Bro-Dev] scheduling events vs using &expire_func ?

2018-04-18 Thread Azoff, Justin S
> On Apr 17, 2018, at 4:04 PM, Aashish Sharma wrote: > > For now, I am resorting to &expire_func route only. I think by using some more > heuristics in worker's expire functions for more aggregated stats, I am able > to > shed load on manager where manager doesn't need to track ALL potential >

[Bro-Dev] Configuration framework on a cluster?

2018-03-26 Thread Azoff, Justin S
How is the Configuration framework intended to be used on a cluster? It has code that does if ( Cluster::is_enabled() && Cluster::local_node_type() != Cluster::MANAGER ) return; to set the input files to be only read on the manager, and appears to not allow you to use &

Re: [Bro-Dev] Broker port status

2018-03-09 Thread Azoff, Justin S
> On Mar 9, 2018, at 8:24 AM, Azoff, Justin S wrote: > > Ah.. I grabbed the latest commit from the wrong checkout and was using a > commit from a few weeks ago. > Ok.. running the latest commit now (105fc386ef0f65a91839706641abae664c7f3e49) Have noticed a problem where at the

Re: [Bro-Dev] Broker port status

2018-03-09 Thread Azoff, Justin S
> On Mar 8, 2018, at 6:32 PM, Jon Siwek wrote: > > Yeah, I realized that and fixed it yesterday. Have you pulled the > latest commits from the topic/actor-system branch? > > - Jon Ah.. I grabbed the latest commit from the wrong checkout and was using a commit from a few weeks ago. — Jus

Re: [Bro-Dev] Broker port status

2018-03-08 Thread Azoff, Justin S
> On Mar 8, 2018, at 5:46 PM, Jon Siwek wrote: > > On Thu, Mar 8, 2018 at 5:07 PM, Azoff, Justin S wrote: > >> One thing I notice is that the traffic to/from the manager box and cpu has >> increased quite a bit. >> >> Ignoring the large CPU spikes from

Re: [Bro-Dev] Broker port status

2018-03-08 Thread Azoff, Justin S
> On Mar 8, 2018, at 1:28 PM, Jon Siwek wrote: > > Hi all, this is just a status update on porting Bro to use the new > Broker communication system. I'd say the topic/actor-system branch is > now functionally complete with still a few weeks left of internal > cleanup/improvements. Awesome, I'll

Re: [Bro-Dev] Shipping CAF with Broker?

2018-02-13 Thread Azoff, Justin S
> On Feb 13, 2018, at 11:36 AM, Seth Hall wrote: > > > > On 13 Feb 2018, at 11:31, Robin Sommer wrote: > >> We could even go a step further and compile CAF statically into >> libbroker, so that in the end from a user's perspective all they deal >> with is Broker: if they link against it, they

Re: [Bro-Dev] 'async' update and proposal

2018-01-28 Thread Azoff, Justin S
> On Jan 27, 2018, at 12:25 AM, Johanna Amann wrote: > Consider e.g. an example like the following > > event protocol_event_1(...) > { > c$proto$la = function_call; > } > > event protocol_event_end(...) > { > Log::write([c$proto$la...]); > } I was thinki

Re: [Bro-Dev] Feedback on configuration framework implementation

2017-12-07 Thread Azoff, Justin S
> On Dec 7, 2017, at 5:22 PM, Johanna Amann wrote: > Indeed, that is my thought. This seems like a job for broker, instead of > trying to somehow force this into a complex ascii-representation. > > Note that this is just a limitation of the config reader - the rest of the > config framework (

Re: [Bro-Dev] Feedback on configuration framework implementation

2017-12-07 Thread Azoff, Justin S
> On Nov 29, 2017, at 5:02 PM, Johanna Amann wrote: > > The config reader provides a way to read configuration files back into > Bro. Most importantly it automatically converts values to the correct > types. This is important because it is at least inconvenient (and > sometimes near impossible)

Re: [Bro-Dev] Feedback on configuration framework implementation

2017-12-01 Thread Azoff, Justin S
> On Dec 1, 2017, at 9:58 AM, Robin Sommer wrote: > > > > On Thu, Nov 30, 2017 at 10:28 -0800, you wrote: > >> think of that. I honestly also never liked modifying the values that are >> passed in arguments; this is for example also theoretically possible for >> events, but something that w

Re: [Bro-Dev] [Bro-Commits] [git/bro] topic/actor-system: First-pass broker-enabled Cluster scripting API + misc. (07ad06b)

2017-11-03 Thread Azoff, Justin S
On Nov 3, 2017, at 3:13 PM, Jan Grashöfer mailto:jan.grashoe...@gmail.com>> wrote: On 03/11/17 18:07, Azoff, Justin S wrote:> Partitioning the intel data set is a little tricky since it supports subnets and hashing 10.10.0.0/16 and 10.10.10.10 won't necessarily give you the sam

Re: [Bro-Dev] [Bro-Commits] [git/bro] topic/actor-system: First-pass broker-enabled Cluster scripting API + misc. (07ad06b)

2017-11-03 Thread Azoff, Justin S
> On Nov 3, 2017, at 6:51 AM, Jan Grashöfer wrote: > > At this point, if the manager functionality is distributed across > multiple data nodes, we have to make sure, that every data node has the > right part of the DataStore to deal with the incoming hit. One could > keep the complete DataSto

Re: [Bro-Dev] [Bro-Commits] [git/bro] topic/actor-system: First-pass broker-enabled Cluster scripting API + misc. (07ad06b)

2017-11-02 Thread Azoff, Justin S
> On Nov 2, 2017, at 5:54 PM, Siwek, Jon wrote: > > Thanks, though I’m not sure this scenario maps well to this particular point. > E.g. my impression is Justin wants a single BIF/function that can send one > event from a worker to a proxy and have the proxy purely relay it to all > other wo

Re: [Bro-Dev] [Bro-Commits] [git/bro] topic/actor-system: First-pass broker-enabled Cluster scripting API + misc. (07ad06b)

2017-11-02 Thread Azoff, Justin S
> On Nov 2, 2017, at 5:21 PM, Siwek, Jon wrote: >> >> Mostly so that workers don't end up spending all their time sending out >> messages when they should be analyzing packets. > > Ok, I get what you want to avoid, though could be interesting to actually > have a fully-connected cluster in or

Re: [Bro-Dev] [Bro-Commits] [git/bro] topic/actor-system: First-pass broker-enabled Cluster scripting API + misc. (07ad06b)

2017-11-02 Thread Azoff, Justin S
> On Nov 2, 2017, at 2:37 PM, Aashish Sharma wrote: > > > > Now, while Justins' multiple data nodes idea has specticular merits, I am not > much fan of it. Reason being having multiple data-notes results in same sets > of problems It does not have the same problems.. It may have different p

Re: [Bro-Dev] [Bro-Commits] [git/bro] topic/actor-system: First-pass broker-enabled Cluster scripting API + misc. (07ad06b)

2017-11-02 Thread Azoff, Justin S
> On Nov 2, 2017, at 1:22 PM, Siwek, Jon wrote: > > >> On Nov 1, 2017, at 6:11 PM, Azoff, Justin S wrote: >> >> - a bif/function for efficiently broadcasting an event to all other workers >> (or data nodes) >> - If the current node is a

Re: [Bro-Dev] [Bro-Commits] [git/bro] topic/actor-system: First-pass broker-enabled Cluster scripting API + misc. (07ad06b)

2017-11-01 Thread Azoff, Justin S
> On Nov 1, 2017, at 5:23 PM, Robin Sommer wrote: > > Justin, correct me if I'm wrong, but I don't think this has ever been > fully fleshed out. If anybody wants to propose something specific, we > can discuss, otherwise I would suggest we stay with the minimum for > now that replicates the old

Re: [Bro-Dev] File Analysis Inconsistencies

2017-10-13 Thread Azoff, Justin S
> On Oct 13, 2017, at 11:01 AM, Aaron Eppert > wrote: > > Justin, > > Indeed, cutting new territory is always interesting. As for the code, > > https://github.com/aeppert/test_file_analyzer > > > File I am using for this case: > https://www.bro.org/static/exchange-2013/faf-exercise.pcap >

Re: [Bro-Dev] Performance Enhancements

2017-10-12 Thread Azoff, Justin S
> On Oct 6, 2017, at 5:59 PM, Jim Mellander wrote: > > I particularly like the idea of an allocation pool that per-packet > information can be stored, and reused by the next packet. > > There also are probably some optimizations of frequent operations now that > we're in a 64-bit world that c

Re: [Bro-Dev] File Analysis Inconsistencies

2017-10-12 Thread Azoff, Justin S
> On Oct 12, 2017, at 5:21 PM, Aaron Eppert wrote: > > I crafted a custom file analysis plugin that attaches to specific MIME types > via file_sniff and fires an appropriate event once processing has been > completed. > > I had to jump through a few hoops to make a file analysis plugin, first

Re: [Bro-Dev] Performance Enhancements

2017-10-09 Thread Azoff, Justin S
> On Oct 9, 2017, at 4:57 PM, Jim Mellander wrote: > > Well, I found pathological behavior with BaseList::remove > > Instrumenting it with a printf of num_entries & i after the for loop, running > against a test pcap then summarizing with awk gives: > > Count, num_entries, i > > 1 3583 3536

Re: [Bro-Dev] Performance Enhancements

2017-10-09 Thread Azoff, Justin S
> On Oct 9, 2017, at 4:03 PM, Clark, Gilbert wrote: > > If you look in one of the subdirectories or another, in ages past there was a > little shell script to incrementally execute bro against a specific trace, > loading one script at a time to see the effect each of them had on the > overall

Re: [Bro-Dev] design summary: porting Bro scripts to use Broker

2017-10-09 Thread Azoff, Justin S
> On Oct 9, 2017, at 2:08 PM, Siwek, Jon wrote: > > >> I got send_event_hashed to work via a bit of a hack >> (https://github.com/JustinAzoff/broker_distributed_events/blob/master/distributed_broker.bro), >> but it needs support from inside broker or at least the bro/broker >> integration to

Re: [Bro-Dev] Performance Enhancements

2017-10-09 Thread Azoff, Justin S
> On Oct 6, 2017, at 5:59 PM, Jim Mellander wrote: > > I particularly like the idea of an allocation pool that per-packet > information can be stored, and reused by the next packet. Turns out bro does this most of the time.. unless you use the next_packet event. Normal connections use the se

Re: [Bro-Dev] Performance Enhancements

2017-10-06 Thread Azoff, Justin S
> On Oct 6, 2017, at 5:59 PM, Jim Mellander wrote: > > I particularly like the idea of an allocation pool that per-packet > information can be stored, and reused by the next packet. > > There also are probably some optimizations of frequent operations now that > we're in a 64-bit world that c

Re: [Bro-Dev] design summary: porting Bro scripts to use Broker

2017-10-06 Thread Azoff, Justin S
> On Oct 6, 2017, at 12:53 PM, Siwek, Jon wrote: > > I want to check if there’s any feedback on the approach I’m planning to take > when porting over Bro’s scripts to use Broker. There’s two major areas to > consider: (1) how users specify network topology e.g. either for traditional > clust

Re: [Bro-Dev] Performance Enhancements

2017-10-06 Thread Azoff, Justin S
> On Oct 6, 2017, at 12:10 AM, Clark, Gilbert wrote: > > I'll note that one of the challenges with profiling is that there are the bro > scripts, and then there is the bro engine. The scripting layer has a > completely different set of optimizations that might make sense than the > engine do

Re: [Bro-Dev] Configuration framework syntax proposal

2017-09-21 Thread Azoff, Justin S
> On Sep 21, 2017, at 9:18 AM, Seth Hall wrote: > > There is just something about the idea of exposing variable names to > users (even if it's wrapped in a gui) that is intensely unpalatable to > me. It's pretty much unheard of among other types of software. It > would be like exposing inte

Re: [Bro-Dev] 'async' keyword

2017-09-20 Thread Azoff, Justin S
> On Sep 20, 2017, at 7:12 PM, Alan Commike wrote: > > What are your thoughts on error handling? > > Exec::run() returns an Exec::Result, which is nice in that we can recover if > something goes wrong. I would think one would want most calls of Exec::run() > in an async context, but we lose

Re: [Bro-Dev] Configuration framework syntax proposal

2017-09-20 Thread Azoff, Justin S
> On Sep 20, 2017, at 6:24 PM, Johanna Amann wrote: > > > const filter = "ip" &config="input.pcap.filter"; > > the definition could look like > > configopt filter = "ip"; Could the definition be const filter = “ip” &config; if you just wanted to use NameSpace::filter ? That kinda seems li

Re: [Bro-Dev] [Bro-Commits] [git/bro] topic/jsiwek/actor-system: Add Broker::publish_id function to replace send_id (096ec69)

2017-08-30 Thread Azoff, Justin S
> On Aug 29, 2017, at 3:58 PM, Jonathan Siwek wrote: > >A minor/unrelated detail of this change is that it also allows >'pattern' values to be exchanged over broker messages. It currently >does this via Bro's serialization, but can likely be done via the >pattern's textual repre

Re: [Bro-Dev] Check if table element exists

2017-08-08 Thread Azoff, Justin S
You're looking for 'in' http://try.bro.org/#/?example=basics-composite-types-table -- - Justin Azoff > On Aug 8, 2017, at 2:04 PM, Reinhard Gentz wrote: > > Hi, > > I would like to check if a certain table element exists and then take > corresponding action like the following: > > if (exi

Re: [Bro-Dev] 2.5.1 release?

2017-05-12 Thread Azoff, Justin S
> On May 12, 2017, at 9:48 PM, Slagell, Adam J wrote: > > > >> On May 12, 2017, at 4:09 PM, Seth Hall wrote: >> >> I'd be fine with that. I think master is quite stable right now anyway > > > I'm pretty sure Vlad is in agreement, but traveling today. I think Justin as > well, but I shoul

Re: [Bro-Dev] early performance comparisons of CAF-based run loop

2017-04-12 Thread Azoff, Justin S
> On Apr 12, 2017, at 9:05 PM, Siwek, Jon wrote: > > >> On Apr 12, 2017, at 1:35 PM, Slagell, Adam J wrote: >> >> Justin asked an interesting question today, how does this affect performance >> on the manager? That is where we are feeling a lot of pain with select(). > > If you mean the sel

Re: [Bro-Dev] Scaling out bro cluster communication

2017-03-10 Thread Azoff, Justin S
> > It implements a fake known hosts and scan detection policy. > > the main things to figure out is: > > * How to work out the proper node_count at runtime. I think on a real bro > cluster the Cluster namespace has the data I need for this, including which > nodes are reachable. > > * How t

Re: [Bro-Dev] Scaling out bro cluster communication

2017-03-08 Thread Azoff, Justin S
Hi, I was able to put together of a prototype of the functionality I had in mind. I learned a bit more about broker including the Broker::send_event function vs. the auto_event method. send_event doesn't let you pick the node you want to send data to, but it does let you pick the queue. By cr

Re: [Bro-Dev] [desired broker api as oppose to whats in known-hosts.bro]

2017-03-03 Thread Azoff, Justin S
> On Mar 3, 2017, at 4:36 PM, Aashish Sharma wrote: > > SO I came across a sample of Broker-API usage: Yeah.. there's a lot of things wrong with how that is being done. There are a few things going on here. One is that &synchronized is no longer functions. I think we should bring this back

Re: [Bro-Dev] Scaling out bro cluster communication

2017-02-10 Thread Azoff, Justin S
> On Feb 10, 2017, at 11:49 AM, Matthias Vallentin wrote: > > Concretely: can you describe (without Bro script code) what "client-side > load-balancing and failover" means? Who is the client and what state > needs to be resilient to failure? I don't think we have a working > definition of "data

Re: [Bro-Dev] Scaling out bro cluster communication

2017-02-10 Thread Azoff, Justin S
> On Feb 10, 2017, at 9:30 AM, Seth Hall wrote: > > >> On Feb 9, 2017, at 3:21 PM, Azoff, Justin S wrote: >> # define event and handle on all nodes >> global scan_attempt: event(scanner: addr, attempt: Attempt); >> event Scan::scan_attemp

[Bro-Dev] Scaling out bro cluster communication

2017-02-09 Thread Azoff, Justin S
I've been thinking about ideas for how to better scale out bro cluster communication and how that would look in scripts. Scaling out sumstats and known hosts/services/certs detection will require script language or bif changes. What I want to make possible is client side load balancing and fail

Re: [Bro-Dev] Packet Signature, Protocol, and Analyzer Relationship

2017-02-08 Thread Azoff, Justin S
> On Feb 8, 2017, at 3:26 PM, Justin Oursler wrote: > > Hello, > > I am writing a new analyzer and plugin for a TCP Application protocol. Can > someone help explain the relationship among the protocol, the analyzer, and > the dynamic signature files? Bro either attaches an analyzer to a con

Re: [Bro-Dev] Broker's remote logging (BIT-1784)

2017-01-31 Thread Azoff, Justin S
> On Jan 31, 2017, at 5:41 PM, Robin Sommer wrote: > ... > The fact that RemoteSerializer and broker::Manager are calling > different Write() functions seems to be a broader issue: we get > different semantics that way. For RemoteSerializer, log events and log > filters run only on the originati

Re: [Bro-Dev] fatal error: can't find base/init-bare.bro

2017-01-19 Thread Azoff, Justin S
> On Jan 19, 2017, at 5:32 PM, Alberto Garcia wrote: > > I've tried running: > > set exec-wrapper bash -c 'source /home/default/bro/build/bro-path-dev.sh' > within GDB but it didn't work either. > > On Thu, Jan 19, 2017 at 4:01 PM, Alberto Garcia > wrote: > It actually works if i'm in a shel

Re: [Bro-Dev] help Reading the backtrace

2017-01-18 Thread Azoff, Justin S
Yeah.. lots of expires may have something to do with it, your traceback shows TableEntryVal16ExpireAccessTimeEv But I also wonder what you are doing that is triggering Dictionary9NextEntryERP7HashKeyRP10IterCookiei which would be void* Dictionary::NextEntry(HashKey*& h, IterCookie*& cookie, in

Re: [Bro-Dev] help Reading the backtrace

2017-01-18 Thread Azoff, Justin S
> On Jan 18, 2017, at 12:29 PM, Aashish Sharma wrote: > > So I am running a new detection package It was stable before you added the new scripts? Are the new scripts publicly available? -- - Justin Azoff ___ bro-dev mailing list bro-dev@bro.or

Re: [Bro-Dev] [Proposal] Language extensions for better Broker support

2016-12-14 Thread Azoff, Justin S
> On Dec 13, 2016, at 1:52 PM, Robin Sommer wrote: > > My draft proposal extends > opaque types to support conversion to boolean to allow for more > elegant error checks. Would that mean you wouldn't be able to return a bool and check for errors at the same time? -- - Justin Azoff

Re: [Bro-Dev] broctl archive copy vs move

2016-12-13 Thread Azoff, Justin S
> On Dec 13, 2016, at 3:12 PM, Aashish Sharma wrote: > > So if we have compresscmd unset then archive-log script does a copy: > > archive-log:nice cp $file_name "$dest" > > Any reason why it doesn't do move instead ? > > I propose changing cp to mv No.. I don't think there's any good r

Re: [Bro-Dev] [Proposal] Language extensions for better Broker support

2016-12-02 Thread Azoff, Justin S
> On Dec 1, 2016, at 9:39 PM, Robin Sommer wrote: > > Bro's current Broker framework has a few pretty inelegant API parts > because Bro's scripting language doesn't support some of its > operations well currently. I've put some thoughts together on > potential language extensions to improve the s

Re: [Bro-Dev] Proposal: Operating the Bro package manager offline

2016-09-19 Thread Azoff, Justin S
Sounds great.. What you are describing is basically source and binary packages. The only thing I would do differently is that you wouldn't want bundles (at least not as the only feature) but individual source and binary packages. For example bro-pkg install sethhall/domain-tld would still inst

Re: [Bro-Dev] Bro IDS request

2016-08-12 Thread Azoff, Justin S
> On Aug 12, 2016, at 2:14 PM, Aashish Sharma wrote: > > May be try: ftp://ftp.ee.lbl.gov/cf-1.2.5.tar.gz > > eg: cf conn.log | less > Yeah.. cf should be a few times faster than bro-cut for busy log files, especially if the only thing you are doing is converting the timestamp. It has an

Re: [Bro-Dev] Making scan.bro great again.

2016-08-02 Thread Azoff, Justin S
> On Aug 2, 2016, at 6:03 PM, Slagell, Adam J wrote: > > Wow. Big difference Indeed :-) I realized one of the bigger issues in the sumstats based code is not really the detection of scans, but what happens AFTER the detection. After detection it keeps accumulating data, or possibly only sl

[Bro-Dev] Making scan.bro great again.

2016-07-29 Thread Azoff, Justin S
I took a closer look at scan-NG and at the scan.bro that shipped with 1.5 to understand how the detection could be better than what we have now. 1.5 wasn't fundamentally better, but compared to what we are doing now it has an unfair advantage :-) I found that it used tables like this: globa

Re: [Bro-Dev] testing topic/dnthayer/ticket1627

2016-07-29 Thread Azoff, Justin S
You simply need to copy/paste your manager section in node.cfg and change manager to logger, so you should end up with something like this: [manager] type=manager host=1.2.3.4 [logger] type=logger host=1.2.3.4 -- - Justin Azoff > On Jul 29, 2016, at 5:48 PM, Aashish Sharma wrote: > > HI Dan

Re: [Bro-Dev] package manager progress

2016-07-27 Thread Azoff, Justin S
> On Jul 27, 2016, at 11:44 AM, Seth Hall wrote: > > >> On Jul 25, 2016, at 4:49 PM, Azoff, Justin S wrote: >> >> In one aspect the pktsrc- prefix acts like a tag, but can also help >> disambiguate plugins... i.e., a redis log writer plugin vs. a redis

Re: [Bro-Dev] package manager progress

2016-07-25 Thread Azoff, Justin S
> On Jul 24, 2016, at 3:45 PM, Siwek, Jon wrote: > > * Add a way for package’s to define “discoverability metadata”. Kind of related to this, I think we need to define some basic rules for package naming. This can help discoverability and also namespacing issues. Right now we have plugins nam

Re: [Bro-Dev] Unified scan.bro script

2016-07-15 Thread Azoff, Justin S
A further iteration of the unified scan.bro script is now in the branch topic/jazoff/scan-unified Use of the branch isn't required though, as it is a self contained change one can just grab the https://raw.githubusercontent.com/bro/bro/31b63445ed07e2e76f98c49dd59091b1742523d1/scripts/policy/mi

Re: [Bro-Dev] Unified scan.bro script

2016-07-12 Thread Azoff, Justin S
> On Jul 12, 2016, at 5:10 PM, Seth Hall wrote: > > >> On Jul 11, 2016, at 8:44 PM, Azoff, Justin S wrote: >> >> It is.. amazing! The unified code is simpler, uses less memory, puts less >> load on sumstats, generates nicer notice messages, and dete

[Bro-Dev] Unified scan.bro script

2016-07-11 Thread Azoff, Justin S
As part of the sumstats things I've been looking into I tried refactoring scan.bro to put less load on sumstats. The refactored script is at https://gist.github.com/JustinAzoff/fe68223da6f81319d3389c605b8dfb99 It is.. amazing! The unified code is simpler, uses less memory, puts less load on s

Re: [Bro-Dev] [Bro-Blue] scan detector related sumstats load

2016-07-10 Thread Azoff, Justin S
> > On Jul 10, 2016, at 1:06 AM, Johanna Amann wrote: > > First - wouldn't this be a topic for bro-dev instead of bro-blue? I don't > really see a reason to keep this private. Indeed.. Moving this to bro-dev as there aren't any more internal logs or addresses. To catch everyone up, I noticed

Re: [Bro-Dev] New proposal (Re: CBAN naming)

2016-06-07 Thread Azoff, Justin S
> On Jun 6, 2016, at 2:50 PM, Robin Sommer wrote: > > - At install time ("cban install" or whatever) gets copied into > a subdirectory at a global location ("cp -rp > /"). Uninstallation means removing that > installation directory. One thing to think about is a distinction between insta

Re: [Bro-Dev] CBAN naming

2016-06-01 Thread Azoff, Justin S
> On Jun 1, 2016, at 5:58 PM, Slagell, Adam J wrote: > > I have gone over a bunch of these threads and want to bring this back towards > consensus, or at least consilience. Let’s pick from the following 5 options, > and then we can decide if these are bundles, ports, packages, plugins, > exte

Re: [Bro-Dev] CBAN naming

2016-05-27 Thread Azoff, Justin S
> On May 27, 2016, at 1:00 PM, Matthias Vallentin wrote: > > To find the new name for our CBAN project, it probably make sense to > brainstorm separately from the existing technical thread. I'd say let's > collect some candidates and then create survey to vote on them. > > Here are some ideas f

Re: [Bro-Dev] Option -z

2016-05-26 Thread Azoff, Justin S
> On May 26, 2016, at 10:15 AM, Robin Sommer wrote: > > > > On Wed, May 25, 2016 at 20:56 -0700, you wrote: > >> Well it's there in CHANGES, per the appended. But yeah looks like it never >> went anywhere beyond the original instigation, so I think removing it is >> okay. > > Ah, I didn't

Re: [Bro-Dev] Flare removal

2016-05-20 Thread Azoff, Justin S
> On May 20, 2016, at 4:52 PM, Seth Hall wrote: > > For the 2.5 release, we were hoping to understand why the > topic/seth/remove-flare fixes some issues that people have been seeing with > the communication code. Perhaps even more to the point we are aiming to > understand why that branch fi