[Bro-Dev] Zeek mailing list renaming and outage

[Johanna Amann]

Hello everyone,

As part of renaming Bro to Zeek, we are going to change the addresses and
names of all our mailing lists.

The important thing first: this change is going to happen tomorrow
(Thursday), 12/20. The mailing list rename will happen during a system
maintenance window at ICSI (which is hosting the mailing list). Thus the
mailing lists (including the archives) will be unavailable for a few hours
during the day on Thursday. Mails sent to the lists during this time
should be queued and delivered when the system is back up.

The current Bro mailing lists will be renamed as follows:
b...@bro.org -> z...@zeek.org
bro-dev@bro.org -> zeek-...@zeek.org
bro-comm...@bro.org -> zeek-comm...@zeek.org
bro-annou...@bro.org -> zeek-annou...@zeek.org

Similarly we will change the mailing list subject tags from Bro to Zeek.

There will be redirects from the old mailing list names to the new ones,
so mails sent to the old addresses will not be lost.

Johanna
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] New nightly builds and Bro build testing

[Johanna Amann]

Hello everyone,

as some of you might have noticed, our nightly builds have been broken for
a while. To be more exact since broker was introduced - since this opened
up a few questions on how exactly packaging should be performed.

These problems have now been cleared up and new nightly builds are
available on OBS:

Download URLs: 
https://software.opensuse.org//download.html?project=network%3Abro&package=bro-nightly
Package Information: 
https://build.opensuse.org/package/show/network:bro/bro-nightly

The new builds now link CAF and Broker statically into Bro - so it is no
longer necessary to fiddle with the ld.so.conf or similar.

Builds are now also available for a newer selection of distributions
(again). If your favorite distribution is missing for some reason, let me
know :)

If no one discovers any problems with these build I will do new package
builds of Bro 2.6 using the exact same approach. I only did somewhat
limited testing, but on the few distributions I tried everything worked
fine.

Note that the name of the project on OBS will change to
network:zeek/zeek-nightly at some point in the future - I will send
another followup email once that happens.

Johanna
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] Building bro 2.6 with static broker/caf libraries

[Johanna Amann]

FYI - there now also is a topic/johanna/static branch in the base
repository, which adds a --enable-static-broker flag that automatically
builds/links static broker/caf against Bro.

Pull request in https://github.com/zeek/zeek/pull/224.

Johanna

On Thu, Dec 06, 2018 at 10:25:50AM -0800, Johanna Amann wrote:
> Hi Craig,
> 
> I pushed another commit to the branch that passes --build-static-only through
> to CAF; if you just want the patch for that it is available at
> https://github.com/zeek/broker/commit/bf03a4246113c72d10530cc0c2729a3fa6f0b046.
> 
> (Note that repositories are currently being migrated; if you pull it from
> somewhere make sure that you actually have that commit in the branch)
> 
> Johanna
> 
> On Thu, Dec 06, 2018 at 07:01:25AM -0800, Johanna Amann wrote:
> > Hi Craig,
> > 
> > I actually recently started working on this, however I am did not quite
> > look at what you want.
> > 
> > There already is a branch called topic/johanna/static, which now makes
> > --build-static(-only) work for broker whan CAF is built statically - it
> > does not yet pass the static flags through to broker; I actually wanted to
> > take a look at that today.
> > 
> > I will let you know when I make progress there :)
> > 
> > Johanna
> > 
> > On Wed, Dec 05, 2018 at 07:03:06PM -0800, Craig Leres wrote:
> > > I've read up on cmake variable scope but I can't figure out how to build 
> > > bro with static libraries. The bro-bundled caf already has 
> > > CAF_BUILD_STATIC_ONLY which I'm pretty sure works but I'm unable to turn 
> > > it on when building caf as part of a bro build.
> > > 
> > > For example I'd like is to optionally (i.e. from a -D argument to cmake 
> > > itself) be able to turn on CAF_BUILD_STATIC_ONLY for 
> > > aux/broker/3rdparty/caf/CMakeLists.txt but nothing I've tried in the top 
> > > level CMakeLists.txt is seen when the caf CMakeLists.txt is being 
> > > evaluated.
> > > 
> > > (I'm working on updating the FreeBSD port to 2.6 and can't install 
> > > things like libcaf_io.so in /usr/local/lib because they conflict with 
> > > libraries potentially installed by the devel/caf port.)
> > > 
> > >   Craig
> > > ___
> > > bro-dev mailing list
> > > bro-dev@bro.org
> > > http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
> > > 
> > ___
> > bro-dev mailing list
> > bro-dev@bro.org
> > http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
> > 
> ___
> bro-dev mailing list
> bro-dev@bro.org
> http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
> 
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] Building bro 2.6 with static broker/caf libraries

[Johanna Amann]

Hi Craig,

I pushed another commit to the branch that passes --build-static-only through
to CAF; if you just want the patch for that it is available at
https://github.com/zeek/broker/commit/bf03a4246113c72d10530cc0c2729a3fa6f0b046.

(Note that repositories are currently being migrated; if you pull it from
somewhere make sure that you actually have that commit in the branch)

Johanna

On Thu, Dec 06, 2018 at 07:01:25AM -0800, Johanna Amann wrote:
> Hi Craig,
> 
> I actually recently started working on this, however I am did not quite
> look at what you want.
> 
> There already is a branch called topic/johanna/static, which now makes
> --build-static(-only) work for broker whan CAF is built statically - it
> does not yet pass the static flags through to broker; I actually wanted to
> take a look at that today.
> 
> I will let you know when I make progress there :)
> 
> Johanna
> 
> On Wed, Dec 05, 2018 at 07:03:06PM -0800, Craig Leres wrote:
> > I've read up on cmake variable scope but I can't figure out how to build 
> > bro with static libraries. The bro-bundled caf already has 
> > CAF_BUILD_STATIC_ONLY which I'm pretty sure works but I'm unable to turn 
> > it on when building caf as part of a bro build.
> > 
> > For example I'd like is to optionally (i.e. from a -D argument to cmake 
> > itself) be able to turn on CAF_BUILD_STATIC_ONLY for 
> > aux/broker/3rdparty/caf/CMakeLists.txt but nothing I've tried in the top 
> > level CMakeLists.txt is seen when the caf CMakeLists.txt is being evaluated.
> > 
> > (I'm working on updating the FreeBSD port to 2.6 and can't install 
> > things like libcaf_io.so in /usr/local/lib because they conflict with 
> > libraries potentially installed by the devel/caf port.)
> > 
> > Craig
> > ___
> > bro-dev mailing list
> > bro-dev@bro.org
> > http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
> > 
> ___
> bro-dev mailing list
> bro-dev@bro.org
> http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
> 
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] Building bro 2.6 with static broker/caf libraries

[Johanna Amann]

Hi Craig,

I actually recently started working on this, however I am did not quite
look at what you want.

There already is a branch called topic/johanna/static, which now makes
--build-static(-only) work for broker whan CAF is built statically - it
does not yet pass the static flags through to broker; I actually wanted to
take a look at that today.

I will let you know when I make progress there :)

Johanna

On Wed, Dec 05, 2018 at 07:03:06PM -0800, Craig Leres wrote:
> I've read up on cmake variable scope but I can't figure out how to build 
> bro with static libraries. The bro-bundled caf already has 
> CAF_BUILD_STATIC_ONLY which I'm pretty sure works but I'm unable to turn 
> it on when building caf as part of a bro build.
> 
> For example I'd like is to optionally (i.e. from a -D argument to cmake 
> itself) be able to turn on CAF_BUILD_STATIC_ONLY for 
> aux/broker/3rdparty/caf/CMakeLists.txt but nothing I've tried in the top 
> level CMakeLists.txt is seen when the caf CMakeLists.txt is being evaluated.
> 
> (I'm working on updating the FreeBSD port to 2.6 and can't install 
> things like libcaf_io.so in /usr/local/lib because they conflict with 
> libraries potentially installed by the devel/caf port.)
> 
>   Craig
> ___
> bro-dev mailing list
> bro-dev@bro.org
> http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
> 
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] [Bro-Commits] [git/bro] master: Allow loading policy/protocols/smb once again (57a505b0e)

[Johanna Amann]

To pick up the idea that you mentioned before - do we also want to make 
the new policy/protocols/smb/__load__.bro trigger a reporter warning 
that it is deprecated?

Johanna

On 30 Aug 2018, at 14:07, Jonathan Siwek wrote:

> Repository : ssh://g...@bro-ids.icir.org/bro
> On branch  : master
> Link   : 
> https://github.com/bro/bro/commit/57a505b0e46d499644a6fb3b063cece0684240b8
>
>> ---
>
> commit 57a505b0e46d499644a6fb3b063cece0684240b8
> Author: Jon Siwek 
> Date:   Thu Aug 30 16:05:36 2018 -0500
>
> Allow loading policy/protocols/smb once again
>
> It just redirects to base/protocols/smb
>
>
>> ---
>
> 57a505b0e46d499644a6fb3b063cece0684240b8
>  CHANGES   | 4 
>  NEWS  | 8 ++--
>  VERSION   | 2 +-
>  scripts/policy/protocols/smb/__load__.bro | 1 +
>  scripts/test-all-policy.bro   | 1 +
>  5 files changed, 13 insertions(+), 3 deletions(-)
>
> diff --git a/CHANGES b/CHANGES
> index af31bdea0..15184aa4a 100644
> --- a/CHANGES
> +++ b/CHANGES
> @@ -1,4 +1,8 @@
>
> +2.5-947 | 2018-08-30 16:05:36 -0500
> +
> +  * Allow loading policy/protocols/smb once again (Jon Siwek, 
> Corelight)
> +
>  2.5-946 | 2018-08-30 09:51:16 -0500
>
>* Update NEWS with more info about runtime options (Daniel Thayer)
> diff --git a/NEWS b/NEWS
> index 0af51ef60..86839427b 100644
> --- a/NEWS
> +++ b/NEWS
> @@ -267,8 +267,12 @@ New Functionality
>
>  - Added new NFS events: nfs_proc_symlink, nfs_proc_link, 
> nfs_proc_sattr.
>
> -- The SMB scripts in policy/protocols/smb are now moved into 
> base/protocols/smb
> -  and loaded/enabled by default.
> +- The SMB scripts in policy/protocols/smb are now moved into
> +  base/protocols/smb and loaded/enabled by default.  If you 
> previously
> +  loaded these scripts from their policy/ location (in local.bro or
> +  other custom scripts) you may now remove/change those although they
> +  should still work since policy/protocols/smb is simply a 
> placeholder
> +  script that redirects to the new base/ location.
>
>  - Added new SMB events: smb1_transaction_secondary_request,
>smb1_transaction2_secondary_request, smb1_transaction_response.
> diff --git a/VERSION b/VERSION
> index d522ba4d6..ecd34e707 100644
> --- a/VERSION
> +++ b/VERSION
> @@ -1 +1 @@
> -2.5-946
> +2.5-947
> diff --git a/scripts/policy/protocols/smb/__load__.bro 
> b/scripts/policy/protocols/smb/__load__.bro
> new file mode 100644
> index 0..8fd733d38
> --- /dev/null
> +++ b/scripts/policy/protocols/smb/__load__.bro
> @@ -0,0 +1 @@
> +@load base/protocols/smb
> diff --git a/scripts/test-all-policy.bro b/scripts/test-all-policy.bro
> index 11824c2c6..d31da6573 100644
> --- a/scripts/test-all-policy.bro
> +++ b/scripts/test-all-policy.bro
> @@ -82,6 +82,7 @@
>  @load protocols/modbus/track-memmap.bro
>  @load protocols/mysql/software.bro
>  @load protocols/rdp/indicate_ssl.bro
> +@load protocols/smb/__load__.bro
>  @load protocols/smb/log-cmds.bro
>  @load protocols/smtp/blocklists.bro
>  @load protocols/smtp/detect-suspicious-orig.bro
>
>
>
> ___
> bro-commits mailing list
> bro-comm...@bro.org
> http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-commits
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] [Bro-Commits] [git/bro] topic/johanna/tls-more-data: Update NEWS for ssl changes. (3c7c60cf6)

[Johanna Amann]

Sorry, yup.

Johanna

On 29 Aug 2018, at 9:10, Azoff, Justin S wrote:

>> On Aug 29, 2018, at 12:02 PM, Johanna Amann  wrote:
>>
>> @if ( version <= 2.6)
>> event 2.5-event
>> @else
>> event 2.6-event
>> @endif
>>
>> breaks with 2.5.
>
> Should that be < and not <= ?
>
> —
> Justin Azoff
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] [Bro-Commits] [git/bro] topic/johanna/tls-more-data: Update NEWS for ssl changes. (3c7c60cf6)

[Johanna Amann]

Hi Jon,

I actually tested it - and it works fine with old versions as long as 
you use the @if this way round.

So

@if ( version >= 2.6)
event 2.6-event
@else
event 2.5-event
@endif

works perfectly with 2.5 and 2.6.

@if ( version <= 2.6)
event 2.5-event
@else
event 2.6-event
@endif

breaks with 2.5.

I admittedly stopped looking for the exact reason why at some point - 
but I tested it rather thoroughly :). And I admittedly only figured that 
out after I wrote my comment to the merge-request.

So - I am tempted to put it in NEWS like this - I assume most people 
will just copy-paste it because the @if-statement is complex enough that 
you will not come up with it yourself easily...

Johanna

On 29 Aug 2018, at 8:13, Jon Siwek wrote:

> On Tue, Aug 28, 2018 at 6:35 PM Johanna Amann  
> wrote:
>
>> +  If you use these events, you can make your scripts work on old and 
>> new versions
>> +  of Bro by wrapping the event definition in an @if, for example:
>> +
>> +@if ( Version::at_least("2.6") || ( Version::number == 20500 && 
>> Version::info$commit >= [commit number of change] ) )
>> +event ssl_client_hello(c: connection, version: count, 
>> record_version: count, possible_ts: time, client_random: string, 
>> session_id: string, ciphers: index_vec, comp_methods: index_vec)
>> +@else
>> +event ssl_client_hello(c: connection, version: count, 
>> possible_ts: time, client_random: string, session_id: string, 
>> ciphers: index_vec)
>> +@endif
>
> Since the parser won't be happy with that type of @if usage in old
> releases due to [1], should we instead suggest something like:
>
> function my_ssl_client_hello_impl(c: connection, version: count,
> possible_ts: time, client_random: string, session_id: string, ciphers:
> index_vec, record_version: counter &default=0, comp_methods: index_vec
> &default=index_vec())
> {
> # Copy existing code to here
> }
>
> @if ( Version::at_least("2.6") || ( Version::number == 20500 &&
> Version::info$commit >= [commit number of change] ) )
> event ssl_client_hello(c: connection, version: count, record_version:
> count, possible_ts: time, client_random: string, session_id: string,
> ciphers: index_vec, comp_methods: index_vec)
> { my_ssl_client_hello_impl(c, version, possible_ts, client_random,
> session_id, ciphers, record_version, comp_methods); }
> @else
> event ssl_client_hello(c: connection, version: count, possible_ts:
> time, client_random: string, session_id: string, ciphers: index_vec)
> { my_ssl_client_hello_impl(c, version, possible_ts, client_random,
> session_id, ciphers); }
> @endif
>
> - Jon
>
> [1] https://bro-tracker.atlassian.net/browse/BIT-1976
> ___
> bro-dev mailing list
> bro-dev@bro.org
> http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] Jira filter results

[Johanna Amann]

Hi,

when I go to tracker.bro.org, the top-right box (Filter result) for me
shows:

"The filter configured for this gadget could not be retrieved. Please
verify it is still valid on the issue navigator.". This seems to be
independent of Browser. I think this used to show the merge-requests.

Can someone perhaps fix that again? :)

Thanks,
 Johanna
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] Use of 'any' type

[Johanna Amann]

Hi Jim,

On 16 Aug 2018, at 13:40, Jim Mellander wrote:

> It would be most convenient if the 'any' type could defer type 
> checking
> until runtime at the script level.
>
> For instance, if both A & B are defined as type 'any', a compile time 
> error
>
> "illegal comparison (A < B)"
>
> occurs upon encountering a bro statement
>
> if (A < B) do_something();
>
> even if the actual values stored in A & B at runtime are integral 
> types for
> which comparison makes sense.

I think this is a bit hard to do with how things are set up at the 
moment internally - and it also does make type-checking at startup less 
possible-helpful.

However...

>
> If the decision could be made at runtime (which could then potentially
> throw an error), a number of useful generic functions could be created 
> at
> the script level, rather than creating yet-another-bif.  A useful
> yet-another-bif would be 'typeof' to allow varying code paths based on 
> the
> type of value actually stored in 'any'.

This already exists and I think you can actually use it to write code 
like that; you just have to cast your any-type to the correct type 
first. The function you want is type_name; it is e.g. used in 
base/utils/json.bro.

Johanna
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] case-insensitive patterns

[Johanna Amann]

On Fri, Jun 29, 2018 at 12:26:34PM -0700, Vern Paxson wrote:
> > Just so I have this right: it looks like the preferred would not be
> > /(?i foo)/ but rather /(?i)foo/, yes?
> 
> Oh and to follow up on this, so in PCRE does /x((?i)bar)foo/ make the "foo"
> part case-insensitive too, or not?  It's not obvious to me from the page
> you pointed me at, and I don't have an environment set up to definitively
> test this.

It does not. So:

/x((?i)bar)foo/ is (kind-of [1]) equivalent to /x(?i:bar)foo/ and only makes 
the bar
part case insensitive.

/x(?i)barfoo/ would make barfoo case insensitive.

I am fine with implementing either (?[flags]) or (?[flags]:[pattern) or
both.

I hope this makes sense.

Johanna

[1]: Kind-of, because (?i:bar) is non-capturing, whereas ((?i)bar)
captures "bar" in the first capture-variable. Since we don't support
capturing at the moment that does not really matter for us - it might be
of importance in the future though.
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] case-insensitive patterns

[Johanna Amann]

On Fri, Jun 29, 2018 at 12:00:30PM -0700, Vern Paxson wrote:
> Once I wound up monkeying around with the internals of the pattern-matching
> code (to fix leaks, because Johanna [correctly] pushed back on adding the
> &/| operators for general use if they leaked, which an old ticket indicated
> they would) ... I thought what-the-heck, it's time for supporting
> case-insensitive patterns.

Thanks a lot for searching the memory leaks - I know that has been a pain.

> This turned out to be tricky to implement, as I gleaned from talking with
> Seth about an approach he had tried a while back but abandoned.  But I now
> have it working.

This is great - case-insensitive pattern have been something that I wanted
to have for a long time.

>   You can achieve the same functionality for a subpattern enclosed in
>   parentheses by adding "+i" to the open parenthesis, optionally followed
>   by whitespace.  So for example "/foo|(+i bar)/" will match "BaR", but
>   not "FoO".

Hum. Is there a reason why we come up with our own syntax for this? Other
implementations already have this using a just slightly different syntax.

To do the same in perl, you would use "/foo|(?i:bar)/". It also supports
turning off case insensitivity for part of a pattern by doing
"/foo|(?-i:bar)/". Furthermore you can also switch it on for the rest of
the pattern by doing (?i) - after that everything is insensitive.
https://perldoc.perl.org/perlre.html#Extended-Patterns has more details

Python supports the exact same syntax. And - to make things easier for
users I think it would be way nicer if we just also would do this.

> The funky (+i ...) syntax isn't meant for general user consumption (though
> it's okay if a user wants to use it directly), but rather is how I implemented
> /pattern/i functionality.

And this is fine - but if we support it I would actually prefer just
making it explicit and doing it like everyone else :)

Johanna
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] more set operators? (equality/subset)

[Johanna Amann]

On Fri, Jun 29, 2018 at 08:58:25AM -0700, Vern Paxson wrote:
> This in fact suggests we could implement record equality by converting the
> records to hash indices and then comparing those.

Oh, neat. If that actually works in all cases (so also with records of
records, etc) I would be totally on board with this. Hash equality was
something that I missed a few times :).

Johanna
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] more set operators? (equality/subset)

[Johanna Amann]

On Sun, Jun 24, 2018 at 07:07:06PM -0700, Vern Paxson wrote:
>   s1 == s2  iff  both sets have exactly the same members
> 
>   s1 < s2   iff  every element in s1 is in s2, but s2 has some
>  elements not in s1

[...]

> Any concerns with adding these too?

I actually have a small question when thinking about these - which I
should already have raised about the intersect operators. What happens
when sets contain records or other complex types in these cases?

>From what I can tell, Bro so far refuses to compare records - the reason
being that (I think) we do not have properly implemented comparison
operators internally. For example, the following script:


type A: record {
a: string;
};

event bro_init()
{
local i = A($a="a");
local j = A($a="a");
print i == j;
}

outputs:

$ bro test.bro
error in ./test.bro, line 9: illegal comparison (i == j)

I assume what will at the moment happen with sets is that the pointers of
records are checked for equality - not the content. Which might arguably
be a bit non-intuitive.

As I said this is more of a concern about the already added operators - in
principle I don't have a problem with ==, but I think it should work for
other complex datatypes too.

Johanna
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] patterns and &&/|| vs. &/| operators

[Johanna Amann]

> If there actually is no (longer) problems with concatenating patterns
> at run-time, I'd agree to deprecate.
> 
> I'm imagine it existed because there was such a problem with
> dynamically creating patterns at run-time, but don't know/remember
> what it was.
> 

Now that you mention it - yes, there still is a problem as far as I know.
https://bro-tracker.atlassian.net/browse/BIT-328 seems to be the
relevant ticket.

This probably means that we will either have to limit p1 & p2 to only be
allowed when Bro is not processing traffic yet too - or fix the cleanup of
the DFA data structures.

Johanna
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] set and vector operators

[Johanna Amann]

> (3) Implement "v += e" to mean "append the element e to the vector v".

Do we want to do this now, or should we potentially wait a release-cycle
with it (to prevent the situation where v + e and v+= e means something
different).

Looking at the emails I am generally not 100% sure if we reached consensus
on this one.

> (4) Wait on whether "v + e" should mean "return a vector that is v with
> the element e appended".  (And indeed we can't do this right now if
> we're #2.)

Yup.

> I'm not clear whether we reached agreement on:
> 
> (7?) Add "s1 &= s2" etc. to mean "s1 = s1 & s2".  The advantage of
>  having this as an operator is it might more easily enable efficient
>  implementation of some set operations for big sets.  I suppose
>  if we have it then we'd be expected to also have:
>   (7') "c1 &= c2" etc., i.e., bitwise assignments for "count"
>variables.

They were contained in the "minimal" list that Jon came up with. I think I
would be ok with them, but they did not really get any discussion
afterwards that I can see.

Johanna
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] patterns and &&/|| vs. &/| operators

[Johanna Amann]

On Tue, Jun 19, 2018 at 11:21:10AM -0700, Vern Paxson wrote:
> In working on adding bitwise &/| operators for counts, I've come across
> apparently undocumented && and || operators for patterns:
> 
>   p1 && p2 yields a pattern that matches a p1 followed by a p2
>   p1 || p2 yields a pattern that matches either p1 or p2
> 
> Confusingly, Bro also supports "p1 | p2", which means the same as "p1 || p2"
> above, but *only* if p1 and p2 are literal patterns, not if they are
> variables of type "pattern".  (This functionality is in common use.)
> It doesn't support "p1 & p2" in any form.
> 
> I searched a large corpus of scripts and didn't find any instances of
> "p1 && p2" or "p1 || p2" for literal patterns, so I suspect the current
> feature is basically unused.
> 
> Proposal: as part of adding bitwise &/| operators for counts, I'll
> also implement &/| operators for patterns, and remove the current
> &&/|| functionality.
> 
> This seems pretty straightforward to me - but I've mistakenly thought
> that about other things before! :-P   So if anyone has comments, plz
> speak up ...

Sounds good to me...

Johanna
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] [Bro-Commits] [git/bro] master: Removed a few more discovered UTF-8 characters in Bro scripts. (cd18d9620)

[Johanna Amann]

On 15 Jun 2018, at 14:40, Azoff, Justin S wrote:

>> On Jun 2, 2018, at 12:49 PM, Johanna Amann  wrote:
>>
>> Hum. Would it make sense to introduce a test that checks all
>> script-files for non-ascii-characters?
>>
>> I can so see that happening again...
>>
>> Johanna
>>
>
>
> I added this as one of the checks that packages.bro.org does,
>
> corelight/bro-hardware is the only one I found that has non-ascii in 
> it.. but all in the auto generated data.bro.

I actually am more concerned about the scripts that we ship with Bro 
itself :). Though it is nice to have for packages too...

Johanna
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] $history extensions - zero windows, logarithmic counts

[Johanna Amann]

I think I like these, the only small concern I have is...

> (2) A notion of "logarithmic counts" for history events: for certain
> events ('C' = checksum, 'T' = retransmission, and 'W' = zero window)
> the count is repeated on the 10th/100th/1000th/etc. occurrence.  So a
> history value of 'ttt' means that the responder sent somewhere between
> 100 and 999 retransmissions.  This is useful because for large
> connections, a single checksum error, retransmission, or zero window
> is much less significant for analyzing performance issues than a whole
> bunch of these.

Here we will not have cases where some repetitions are logarithmic, and
some (like for R) are not. I guess that makes sense, but I can see it
potentially being confusing.

Johanna
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] [Bro-Commits] [git/bro] master: Removed a few more discovered UTF-8 characters in Bro scripts. (cd18d9620)

[Johanna Amann]

Hum. Would it make sense to introduce a test that checks all 
script-files for non-ascii-characters?

I can so see that happening again...

Johanna

On 2 Jun 2018, at 1:57, Seth Hall wrote:

> Repository : ssh://g...@bro-ids.icir.org/bro
> On branch  : master
> Link   : 
> https://github.com/bro/bro/commit/cd18d96205851aa9b81bcb8f0c6960768b457f72
>
>> ---
>
> commit cd18d96205851aa9b81bcb8f0c6960768b457f72
> Author: Seth Hall 
> Date:   Sat Jun 2 04:57:48 2018 -0400
>
> Removed a few more discovered UTF-8 characters in Bro scripts.
>
>
>> ---
>
> cd18d96205851aa9b81bcb8f0c6960768b457f72
>  scripts/base/protocols/smb/const-nt-status.bro | 8 
>  1 file changed, 4 insertions(+), 4 deletions(-)
>
> diff --git a/scripts/base/protocols/smb/const-nt-status.bro 
> b/scripts/base/protocols/smb/const-nt-status.bro
> index ea7c7fe9e..f985e72a3 100644
> --- a/scripts/base/protocols/smb/const-nt-status.bro
> +++ b/scripts/base/protocols/smb/const-nt-status.bro
> @@ -494,7 +494,7 @@ redef SMB::statuses += {
>   [0xC131] = [$id="INVALID_IMAGE_WIN_16", $desc="The specified 
> image file did not have the correct format: it appears to be a 16-bit 
> Windows image."],
>   [0xC132] = [$id="LOGON_SERVER_CONFLICT", $desc="The Netlogon 
> service cannot start because another Netlogon service running in the 
> domain conflicts with the specified role."],
>   [0xC133] = [$id="TIME_DIFFERENCE_AT_DC", $desc="The time at the 
> primary domain controller is different from the time at the backup 
> domain controller or member server by too large an amount."],
> - [0xC134] = [$id="SYNCHRONIZATION_REQUIRED", $desc="The SAM 
> database on a Windows Server is significantly out of synchronization 
> with the copy on the domain controller. A complete synchronization is 
> required."],
> + [0xC134] = [$id="SYNCHRONIZATION_REQUIRED", $desc="The SAM 
> database on a Windows Server is significantly out of synchronization 
> with the copy on the domain controller. A complete synchronization is 
> required."],
>   [0xC135] = [$id="DLL_NOT_FOUND", $desc="{Unable To Locate 
> Component} This application has failed to start because %hs was not 
> found. Reinstalling the application may fix this problem."],
>   [0xC136] = [$id="OPEN_FAILED", $desc="The NtCreateFile API 
> failed. This error should never be returned to an application; it is a 
> place holder for the Windows LAN Manager Redirector to use in its 
> internal error-mapping routines."],
>   [0xC137] = [$id="IO_PRIVILEGE_FAILED", $desc="{Privilege Failed} 
> The I/O permissions for the process could not be changed."],
> @@ -536,7 +536,7 @@ redef SMB::statuses += {
>   [0xC15B] = [$id="LOGON_TYPE_NOT_GRANTED", $desc="A user has 
> requested a type of logon (for example, interactive or network) that 
> has not been granted. An administrator has control over who may logon 
> interactively and through the network."],
>   [0xC15C] = [$id="NOT_REGISTRY_FILE", $desc="The system has 
> attempted to load or restore a file into the registry, and the 
> specified file is not in the format of a registry file."],
>   [0xC15D] = [$id="NT_CROSS_ENCRYPTION_REQUIRED", $desc="An 
> attempt was made to change a user password in the security account 
> manager without providing the necessary Windows cross-encrypted 
> password."],
> - [0xC15E] = [$id="DOMAIN_CTRLR_CONFIG_ERROR", $desc="A 
> Windows Server has an incorrect configuration."],
> + [0xC15E] = [$id="DOMAIN_CTRLR_CONFIG_ERROR", $desc="A Windows 
> Server has an incorrect configuration."],
>   [0xC15F] = [$id="FT_MISSING_MEMBER", $desc="An attempt was made 
> to explicitly access the secondary copy of information via a device 
> control to the fault tolerance driver and the secondary copy is not 
> present in the system."],
>   [0xC160] = [$id="ILL_FORMED_SERVICE_ENTRY", $desc="A 
> configuration registry node that represents a driver service entry was 
> ill-formed and did not contain the required value entries."],
>   [0xC161] = [$id="ILLEGAL_CHARACTER", $desc="An illegal character 
> was encountered. For a multibyte character set, this includes a lead 
> byte without a succeeding trail byte. For the Unicode character set 
> this includes the characters 0x and 0xFFFE."],
> @@ -577,7 +577,7 @@ redef SMB::statuses += {
>   [0xC188] = [$id="LOG_FILE_FULL", $desc="The log file space is 
> insufficient to support this operation."],
>   [0xC189] = [$id="TOO_LATE", $desc="A write operation was 
> attempted to a volume after it was dismounted."],
>   [0xC18A] = [$id="NO_TRUST_LSA_SECRET", $desc="The workstation 
> does not have a trust secret for the primary domain in the local LSA 
> database."],
> - [0xC18B] = [$id="NO_TRUST_SAM_ACCOUNT", $desc="The SAM database

Re: [Bro-Dev] System upgrades

[Johanna Amann]

This is done, please let me know if you encounter any problems.

Johanna

On Thu, May 31, 2018 at 08:13:04AM -0700, Johanna Amann wrote:
> Hi,
> 
> the main bro server will go down for system upgrades a bit. This should
> mostly affect the git server, the webpage should stay accessible.
> 
> I will write another email once I am done.
> 
> Johanna
> ___
> bro-dev mailing list
> bro-dev@bro.org
> http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
> 
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] System upgrades

[Johanna Amann]

Hi,

the main bro server will go down for system upgrades a bit. This should
mostly affect the git server, the webpage should stay accessible.

I will write another email once I am done.

Johanna
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] Include C++ header file in plugin function

[Johanna Amann]

>
> Unfortunatly, to use "stringstream" I will have to include the 
>  header file. Is this possible to do in plugin functions?
>

Sure, plugins include libraries/other headers/etc. all the time.

Johanna
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] Moving to GitHub?

[Johanna Amann]

I am also more in favor of starting clean and manually letting people 
move tickets that they think are important over.

But - currently there is a lot in the tracker that are nice to have or 
potential problems that I do not ever see getting addressed.

Johanna

On 18 May 2018, at 9:17, Slagell, Adam J wrote:

> On May 18, 2018, at 11:11 AM, Robin Sommer 
> mailto:ro...@icir.org>> wrote:
>
> What I was envisioning is more or less a clean slate: we'd migrate
> over a few tickets, but essentially we'd start with an empty list. I
> realize that sounds pretty harsh. However, I hardly ever see any
> activity on older tickets in JIRA, and I generally believe that the
> less open tickets a tracker has, the easier it is for people to
> understand what's actually relevant and worth spending cycles on.
> Tagging tickets may help, but in the end if everybody just filters 95%
> out all the time anyways, I'm not sure what the value is.
>
> That said, I'm open to a real porting effort if people do believe it's
> helpful to get all the JIRA tickets into GitHub. What do others think?
>
> Start clean for BIT. Unless it is marked critical, I don’t think it 
> needs to go over. If people have tickets of their own they want to 
> move over, it should not be too hard to manually recreate a couple.
>
> And after BroCon I think we kill the Jira and wiki completely. We 
> should be done with the project management and event tickets by then.
>
> --
>
> Adam J. Slagell
> Director, Cybersecurity & Networking Division
> Chief Information Security Officer
> National Center for Supercomputing Applications
> University of Illinois at Urbana-Champaign
> www.slagell.info
>
> "Under the Illinois Freedom of Information Act (FOIA), any written 
> communication to or from University employees regarding University 
> business is a public record and may be subject to public disclosure."
>
>
>
>
>
>
>
>
> ___
> bro-dev mailing list
> bro-dev@bro.org
> http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] Moving to GitHub?

[Johanna Amann]

> What do people think? Any support, or concerns?

I am in favor. The only thing I would miss are the immediate change 
notifications by email - I really like those...

Johanna
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] Final Broker branch testing

[Johanna Amann]

On 3 May 2018, at 14:20, Jon Siwek wrote:

> On 5/2/18 9:59 AM, Johanna Amann wrote:
>
>>>> (3) I need to try to hack our CMake system more to try to get back 
>>>> down
>>>> to 2.8.12 while still being able to embed CAF.
>
> I think (hope!) I was mistaken and everything already works with 
> 2.8.12 (structure of CMake docs previously led me to think it 
> wouldn't) and just needed the version check moved back down, sorry for 
> the noise.
>

Yay, that is really good news, thanks :)

Johanna
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] Final Broker branch testing

[Johanna Amann]

On 27 Apr 2018, at 8:55, Robin Sommer wrote:

> On Thu, Apr 26, 2018 at 16:54 -0500, you wrote:
>
>> (1) Users whose OS has insufficient CMake will need to compile/obtain 
>> a  newer one.
>
>> (2) We go back to CMake 2.8.12 and have people compile CAF 
>> themselves.
>> (Or maybe we could conditionally require only 2.8.12 users to compile
>> CAF and others get the embedded CAF).
>
>> (3) I need to try to hack our CMake system more to try to get back 
>> down
>> to 2.8.12 while still being able to embed CAF.
>
> If there's something quick that ends up making (3) work, that'd be
> ideal of course, but I don't think it's worth spending much time on,
> given that there are reasonable ways to get a more recent cmake.
>
> I wouldn't want to go back to not shipping CAF at all, but if we can
> tell cmake that 2.8.12 is fine if users build CAF themselves, that
> would be the 2nd best option I think. (1) ist worst case, which still
> isn't too bad IMO, unless it does actually prevent us from building
> binary packages for RH, that would be a problem.
>

Yup, 2 would be ok I guess. One should still be able to just compile the 
CAF in the Bro subdirectory in that case, right?

1 I would rather avoid if possible.

Johanna
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] set and vector operators

[Johanna Amann]

On 30 Apr 2018, at 11:13, Robin Sommer wrote:

> On Mon, Apr 30, 2018 at 07:10 -0700, you wrote:
>
>> Okay, I can live with this as long as '|' and '-' support add-to-set and
>> remove-from-set.   But I think those have to work, given we'll enable them
>> for operations on two sets.
>
> Well, my vote then remains not adding new set operators for
> add/delete, so that we don't have multiple ways to do the same thing.
> Just looked at Python again, as a data point: That's what they do,
> too. There are '|'/'&'/'-' for set/set operations, but no versions of
> those for individual elements (they do that through methods instead;
> add/delete are kind of our version of methods). Same for Ruby. I
> looked around for a few more minutes for other languages, but didn't
> immediately find any that even have any set operators at all (only
> methods/functions for union/intersection/etc.).

Yup, I think I concur.

Johanna
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] set and vector operators

[Johanna Amann]

On Thu, Apr 26, 2018 at 09:29:24AM -0700, Vern Paxson wrote:
> > Just one more thing still: I'm actually feeling pretty strongly
> > against having multiple different operators for the same operation
> > (set union, set addition/removal).
> 
> I'm fine with removing "add" and "delete" for sets!  (But seems we gotta
> keep them for a good while for backward compatibility.  Plus, what would
> be the remove operator for tables?  "t -= index" seems pretty weird to me.)

Perhaps my thinking is too driven by how this is handled internally - but
thinking that way I am kind of opposed to get rid of add and delete for
sets. For me sets were always a special case of tables - and it made
complete sense that they operate in the same way.

I think I actually would prefer just keeping add/delete, at least for
sets, and not introduce the plus-syntax. While it is a bit shorter,
add/delete is also not that ugly. And while you did not like the argument
of Jon that that lets you more easily determine what is going on in the
code I kind of think it has a bit of merit.

Johanna
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] set and vector operators

[Johanna Amann]

On Thu, Apr 26, 2018 at 01:43:53PM -0700, Vern Paxson wrote:
> > A nice thing about "add" and "delete" for sets is that you can infer the 
> > I do also notice that you had "s + e" in the proposal and not "v + e". 
> > Isn't that weird by the same logic or is it just an accidental omission?
> 
> This is because "v + e" already has a meaning: apply "+ e" to each element
> of v.  (Note though that "v += e" is not currently allowed.)

I was actually not aware of this - and if we keep this behavior I am a bit
opposed to add a few of the others. It especially feels weird to me if v +
e and v += e are operations that perform something completely different. I
also think it is a bad idea to have s + e and v + e perform completely
different operations.

Johanna
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] pattern values and "||"/"&&" operators

[Johanna Amann]

On 26 Apr 2018, at 14:04, Vern Paxson wrote:

> In implementing bitwise operations for counts (now pretty much done!),
> I found that Bro's internals actually support "||" and "&&" operators 
> for
> patterns:
>
>   p1 || p2returns a pattern that matches either p1 or p2
>   p1 && p2returns a pattern that matches p1 followed by p2
>
> (Note that this second is not obvious.  One might instead expect it to
> return a pattern that matches only if both p1 and p2 match, where the 
> two
> could overlap.)
>
> This doesn't appear to be documented anywhere.  I discussed it with 
> Robin
> and we both agree that this seems weird and likely not used anywhere.
> Therefore we're thinking we should deprecate it and eventually remove 
> it.

Sounds fine to me - I totally was not aware of this. Though having the 
functionality of the former available in some form would be kind of 
nice-I have a few scripts where I would have used that.

Johanna
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] Final Broker branch testing

[Johanna Amann]

On 26 Apr 2018, at 14:05, Jon Siwek wrote:

> On 4/26/18 2:04 PM, Johanna Amann wrote:
>
>> With this change, we Bro cannot be compiled out of the Box on 
>> RedHat/Centos 7 anymore. Since that is the latest release of RedHat 
>> and probably used in production by quite a few people a potentially 
>> significant amount of people might not be able to (easily) compile 
>> Bro with this merge.
>>
>> It aborts in configure, with:
>>
>> -- Performing Test cxx11_header_works - Success
>> CMake Error at aux/broker/CMakeLists.txt:4 (cmake_minimum_required):
>>    CMake 3.0.2 or higher is required.  You are running version 
>> 2.8.12.2
>
> Is "use cmake3 from EPEL" an acceptable answer?
>
> The main reason for it (IIRC) is for embedding CAF as a CMake 
> ExternalProject, which I was struggling to hack around with lack of 
> features in CMake 2.8.

It might be. I am honestly not sure - I suspect that this still will 
mean that some places might not be able to easily use Bro 
anymore--adding external package sources does not seem to be a viable 
option everywhere.

As a side-note, it also looks like that means that we cannot provide 
binary packages for RedHat/CentOS anymore.

Johanna
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] Final Broker branch testing

[Johanna Amann]

Trying this I noticed a few things (ordered by urgency from my point of 
view).

With this change, we Bro cannot be compiled out of the Box on 
RedHat/Centos 7 anymore. Since that is the latest release of RedHat and 
probably used in production by quite a few people a potentially 
significant amount of people might not be able to (easily) compile Bro 
with this merge.

It aborts in configure, with:

-- Performing Test cxx11_header_works - Success
CMake Error at aux/broker/CMakeLists.txt:4 (cmake_minimum_required):
   CMake 3.0.2 or higher is required.  You are running version 2.8.12.2

--snip

Compiling on Debian 8 gives some CAF warnings that are a tad ugly:

In file included from 
/root/bro/aux/broker/3rdparty/caf/libcaf_core/caf/serializer.hpp:32:0,
  from 
/root/bro/aux/broker/3rdparty/caf/libcaf_core/caf/detail/tuple_vals.hpp:25,
  from 
/root/bro/aux/broker/3rdparty/caf/libcaf_core/caf/make_message.hpp:28,
  from 
/root/bro/aux/broker/3rdparty/caf/libcaf_core/caf/mailbox_element.hpp:27,
  from 
/root/bro/aux/broker/3rdparty/caf/libcaf_core/caf/abstract_actor.hpp:37,
  from 
/root/bro/aux/broker/3rdparty/caf/libcaf_core/caf/actor.hpp:32,
  from /root/bro/aux/broker/broker/data.hh:11,
  from /root/bro/aux/broker/broker/broker.hh:8,
  from /root/bro/src/broker/Data.h:4,
  from /root/bro/src/broker/Data.cc:1:
/root/bro/aux/broker/3rdparty/caf/libcaf_core/caf/data_processor.hpp: In 
function ‘typename std::enable_if().caf::data_processor::apply(declval()))>::value>::type
 
caf::operator&(caf::deserializer&, T&) [with T = 
std::chrono::time_point > >; 
typename std::enable_if().caf::data_processor::apply(declval()))>::value>::type
 
= void]’:
/root/bro/aux/broker/3rdparty/caf/libcaf_core/caf/data_processor.hpp:478:7: 
warning: ‘dur’ may be used uninitialized in this function 
[-Wmaybe-uninitialized]
t = std::chrono::time_point{dur};
^
/root/bro/aux/broker/3rdparty/caf/libcaf_core/caf/data_processor.hpp:476:16: 
note: ‘dur’ was declared here
Duration dur;
 ^
--snip

/root/bro/aux/broker/3rdparty/caf/libcaf_core/src/scheduled_actor.cpp:892:55: 
warning: unused parameter ‘sender’ [-Wunused-parameter]
actor_addr& sender,

--snip

I noticed one small thing while building with make -j4; in this case you 
get several different % numbers simultaneously (one for car and one for 
broker).

Example:

[ 25%] Built target plugin-Bro-BackDoor
[ 25%] Building CXX object 
src/analyzer/protocol/bittorrent/CMakeFiles/plugin-Bro-BitTorrent.dir/bittorrent_pac.cc.o
[ 85%] Building CXX object 
libcaf_io/CMakeFiles/libcaf_io_shared.dir/src/interfaces.cpp.o
[ 25%] Building CXX object 
src/analyzer/protocol/bittorrent/CMakeFiles/plugin-Bro-BitTorrent.dir/events.bif.cc.o

While this is obviously cosmetic, it still looks weird to me :).

Apart from that it compiled and ran all tests on all systems I tried it 
on.

There were a few test failures on the first run (that succeeded on a 
rerun) though.

These were (from different systems):
MacOs:
[ 76%] scripts.base.frameworks.logging.field-extension-cluster ... 
failed
[ 21%] broker.disconnect ... failed
[ 56%] broker.ssl_auth_failure ... failed
[ 89%] scripts.base.frameworks.control.shutdown ... failed
[ 99%] scripts.base.frameworks.openflow.log-cluster ... failed

There were also a couple that did not succeed after several reruns for 
me. This was on a digital ocean 4cpu optimized debian8 instance for me; 
the reruns were not parallel:

root@debian-c-4-8gib-sfo2-01:~/bro/testing/btest# ../../aux/btest/btest 
-r -d
[  0%] scripts.base.frameworks.control.configuration_update ... failed
   % 'btest-bg-wait 10' failed unexpectedly (exit code 1)
   % cat .stderr
   The following processes did not terminate:

   
BROPATH=.:/root/bro/scripts:/root/bro/scripts/policy:/root/bro/scripts/site:/root/bro/build/scripts:..
 
bro 
/root/bro/testing/btest/.tmp/scripts.base.frameworks.control.configuration_update/configuration_update.bro
 
frameworks/control/controller Control::host=127.0.0.1 
Control::host_port=65531/tcp Control::cmd=shutdown

   ---
   <<< [15700] 
BROPATH=.:/root/bro/scripts:/root/bro/scripts/policy:/root/bro/scripts/site:/root/bro/build/scripts:..
 
bro 
/root/bro/testing/btest/.tmp/scripts.base.frameworks.control.configuration_update/configuration_update.bro
 
frameworks/control/controllee Broker::default_port=65531/tcp
   , line 1: received termination signal
  >>>
   <<< [15738] 
BROPATH=.:/root/bro/scripts:/root/bro/scripts/policy:/root/bro/scripts/site:/root/bro/build/scripts:..
 
bro 
/root/bro/testing/btest/.tmp/scripts.base.frameworks.control.configuration_update/configuration_update.bro
 
test-redef frameworks/control/controller Control::host=127.0.0.1 
Control::host_port=65531/tcp Control::cmd=configuration_update
   /root/bro/scripts/polic

Re: [Bro-Dev] input-framework reporter_error vs reporter_warning events ?

[Johanna Amann]

Hi Aashish,

This changed with Bro 2.5.1. To quote NEWS:

- The input framework's Ascii reader is now more resilient. If an input
   is marked to reread a file when it changes and the file didn't exist
   during a check Bro would stop watching the file in previous versions.
   The same could happen with bad data in a line of a file.  These
   situations do not cause Bro to stop watching input files anymore. The
   old behavior is available through settings in the Ascii reader.

Johanna

On 20 Apr 2018, at 14:09, Aashish Sharma wrote:

> While testing other stuff, I realized that if input-framework cannot 
> find a file
> its now generating reporter_warning event instead of reporter_error ?
>
> Did "error" changed to "warning" for some reason ? Wasn't previously 
> this a
> error condition ?
>
>
> 0.00Reporter::WARNING   
> /DATA/feeds/BRO-feeds/WIRED.blocknet.2/Input::READER_ASCII: Init: 
> cannot open /DATA/feeds/BRO-feeds/WIRED.blocknet.2(empty)
>
>
> 1) Also in what situations input-framework would generate WARNINGS vs 
> ERRORS ?
>
> 2) Does warnings means READER_ASCII will try to read file again after 
> some time
> or does it just gives up and waits for me to tap into reporter_warning 
> event to
> handle the situation ?
>
> Thanks,
> Aashish
> ___
> bro-dev mailing list
> bro-dev@bro.org
> http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] Bro git/webserver updates

[Johanna Amann]

After a brief fight with httpd (the old config file did not work without
changes), everything seems to be up and working again.

Please let me know if you notice/encounter any problems.

Johanna

On Thu, Apr 19, 2018 at 11:41:33AM -0700, Johanna Amann wrote:
> Hi,
> 
> I will be updating the Bro git/webserver. There might be intermittent
> outages in the next few hours (mostly of git - the webserver should
> automatically fail over).
> 
> Johanna
> ___
> bro-dev mailing list
> bro-dev@bro.org
> http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
> 
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] Bro git/webserver updates

[Johanna Amann]

Hi,

I will be updating the Bro git/webserver. There might be intermittent
outages in the next few hours (mostly of git - the webserver should
automatically fail over).

Johanna
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] Configuration framework on a cluster?

[Johanna Amann]

Hi Justin,

> How is the Configuration framework intended to be used on a cluster?

It is intended to read the configuration on the manager node; events 
then should be used internally to distribute the data to all other 
nodes.

And you are right - that is something that is completely missing at the 
moment. Which is an oversight - I completely forgot to write this last 
part of the config framework. I am going to update the scripts and 
submit a merge request for them in the next few days.

Johanna
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] [Bro-Commits] [git/bro] master: Update Mozilla CA list to state of NSS 3.35. (8ea7de9)

[Johanna Amann]

This is fixed - thanks again for pointing it out.

Johanna

On 18 Feb 2018, at 19:52, Johanna Amann wrote:

> Sorry about that - I ran the SSL tests but forgot the x.509 tests.
>
> I will fix that by tuesday at the very latest. (At the moment I have a
> bit spotty Internet)
>
> Johanna
>
> On 18 Feb 2018, at 15:50, Jon Siwek wrote:
>
>> This update breaks a unit test:
>> scripts.base.files.x509.signed_certificate_timestamp
>>
>> - Jon
>>
>> On Fri, Feb 16, 2018 at 12:53 PM, Johanna Amann 
>> wrote:
>>> Repository : ssh://g...@bro-ids.icir.org/bro
>>> On branch  : master
>>> Link   :
>>> https://github.com/bro/bro/commit/8ea7de9380e2045e393b3bc22f6be0fa252a94ba
>>>
>>>> -------
>>>
>>> commit 8ea7de9380e2045e393b3bc22f6be0fa252a94ba
>>> Author: Johanna Amann 
>>> Date:   Fri Feb 16 10:52:13 2018 -0800
>>>
>>> Update Mozilla CA list to state of NSS 3.35.
>>>
>>>
>>>> ---
>>>
>>> 8ea7de9380e2045e393b3bc22f6be0fa252a94ba
>>>  scripts/base/protocols/ssl/mozilla-ca-list.bro | 58
>>> --
>>>  1 file changed, 17 insertions(+), 41 deletions(-)
>>>
>>> Diff suppressed because of size. To see it, use:
>>>
>>> git diff-tree --root --patch-with-stat --no-color
>>> --ignore-space-at-eol --textconv --cc
>>> 8ea7de9380e2045e393b3bc22f6be0fa252a94ba
>>>
>>>
>>> ___
>>> bro-commits mailing list
>>> bro-comm...@bro.org
>>> http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-commits
>> ___
>> bro-dev mailing list
>> bro-dev@bro.org
>> http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
> ___
> bro-dev mailing list
> bro-dev@bro.org
> http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] [Bro-Commits] [git/bro] master: Update Mozilla CA list to state of NSS 3.35. (8ea7de9)

[Johanna Amann]

Sorry about that - I ran the SSL tests but forgot the x.509 tests.

I will fix that by tuesday at the very latest. (At the moment I have a 
bit spotty Internet)

Johanna

On 18 Feb 2018, at 15:50, Jon Siwek wrote:

> This update breaks a unit test:
> scripts.base.files.x509.signed_certificate_timestamp
>
> - Jon
>
> On Fri, Feb 16, 2018 at 12:53 PM, Johanna Amann  
> wrote:
>> Repository : ssh://g...@bro-ids.icir.org/bro
>> On branch  : master
>> Link   : 
>> https://github.com/bro/bro/commit/8ea7de9380e2045e393b3bc22f6be0fa252a94ba
>>
>>> ---
>>
>> commit 8ea7de9380e2045e393b3bc22f6be0fa252a94ba
>> Author: Johanna Amann 
>> Date:   Fri Feb 16 10:52:13 2018 -0800
>>
>> Update Mozilla CA list to state of NSS 3.35.
>>
>>
>>> ---
>>
>> 8ea7de9380e2045e393b3bc22f6be0fa252a94ba
>>  scripts/base/protocols/ssl/mozilla-ca-list.bro | 58 
>> --
>>  1 file changed, 17 insertions(+), 41 deletions(-)
>>
>> Diff suppressed because of size. To see it, use:
>>
>> git diff-tree --root --patch-with-stat --no-color 
>> --ignore-space-at-eol --textconv --cc 
>> 8ea7de9380e2045e393b3bc22f6be0fa252a94ba
>>
>>
>> ___
>> bro-commits mailing list
>> bro-comm...@bro.org
>> http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-commits
> ___
> bro-dev mailing list
> bro-dev@bro.org
> http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] Shipping CAF with Broker?

[Johanna Amann]

On 13 Feb 2018, at 8:36, Seth Hall wrote:

> On 13 Feb 2018, at 11:31, Robin Sommer wrote:
>
>> We could even go a step further and compile CAF statically into
>> libbroker, so that in the end from a user's perspective all they deal
>> with is Broker: if they link against it, they get everything they
>> need.
>>
>> Would that make sense?
>
> I think we're quite a ways off from CAF being generally packaged with
> most operating systems (especially having the correct version to work
> with broker).  I think that including it with broker and building 
> libcaf
> into libbroker statically makes sense too.  I've been concerned about
> the difficulty of building Bro from source too.

I agree - while I see the argument of not packaging external stuff, I 
think we should make an exception here.

Johanna
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] 'async' update and proposal

[Johanna Amann]

On 13 Feb 2018, at 7:44, Robin Sommer wrote:

> On Thu, Feb 08, 2018 at 10:01 -0800, Johanna wrote:
>
>> I just wanted to quickly chime in here to say that I generally like 
>> the
>> idea of having these contexts.
>
> Sounds like we all like that idea. Now the question is if we want to
> wait for that to materialize (which will take quite a bit more
> brainstorming and then implementation, obviously), or if we want to
> get async in in the current state and then put that on the TODO list?
> I can see arguments either way, curious what others think.

I have one obvious fear if we put it in now. Which is that it will be 
more or less put off forever :).

And we might end up with a pretty annoying situation where people (and 
we) start using it and you end up with all these really hard to debug 
corner-cases with event ordering. This is already kind of hard to do, 
especially with protocols that have a bunch of events - I managed to get 
this wrong a few times in the case of SSL where some parts of the 
connections where not logged correctly in some circumstances.

Basically - I fear that this will make it really hard to write scripts 
that work correctly in all cases.

Johanna
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] 'async' update and proposal

[Johanna Amann]

I just wanted to quickly chime in here to say that I generally like the
idea of having these contexts. I have no idea how complex it would be to
implement something like this, but that seems like it might be a
relatively clean solution to our problem :)

Johanna

On Tue, Jan 30, 2018 at 07:38:42AM -0800, Robin Sommer wrote:
> 
> 
> On Mon, Jan 29, 2018 at 13:58 -0600, you wrote:
> 
> >  And if 'function_call' starts as a synchronous function and later
> >  changes, that's also kind of a problem, so you might see people
> >  cautiously implementing the same type of code patterns everywhere
> >  even if not required for some cases.
> 
> That's a good point more generally: if we require "async" at call
> sites, an internal change to a framework can break existing code.
> 
> > event protocol_event_1(c: connection ...) &context = { return c$id; } { ... 
> > }
> > 
> > I only skimmed the paper, though seemed like it outlined a similar way
> > of generalizing contexts/scopes ?
> 
> Yeah, that's pretty much the idea there. For concurrency, we'd hash
> that context value and use that to determine a target threat to
> schedule execution too, just like in a cluster the process/machine is
> determined.
> 
> An attribute can work if we're confident that the relevant information
> can always be extracted from the event parameters. In a concurrent
> prototype many years ago we instead used a hardcoded set of choices
> based on the underlying connection triggering the event (5-tuple, host
> pair, src IP, dst IP). So you'd write (iirc):
> 
> event protocol_event_1(c: connection ...) &scope = connection
> 
> That detaches the context calculation from event parameters, with the
> obvious disadvantage that it can't be customized any further. May be
> there's some middle ground where we'd get both.
> 
> (To clarify terminology: In that paper "scope" is the scheduling
> granularity, e.g., "by connection". "context" is the current
> instantiation of that scope (e.g., "1.2.3.4:1234,2.3.4.5:80" for
> connection scope).
> 
> Robin
> 
> -- 
> Robin Sommer * ICSI/LBNL * ro...@icir.org * www.icir.org/robin
> ___
> bro-dev mailing list
> bro-dev@bro.org
> http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
> 
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] (no subject)

[Johanna Amann]

Hi Martina,

you have picked out one of the more confusing parts of Bro to look at. The
logging code is sadly quite complex - mostly because it has to handle a
lot of different cases.

On Fri, Feb 02, 2018 at 04:33:55PM +, Martina Balintova wrote:
> Hi,
> Here are 2 questions, one about usage of memory after free, another seems
> to me like memory leak. Could you please either confirm that these are bugs
> and suggest fixes or help me work out why no?
> 
> Function src/loggin/Manager.cc: bool Manager::Write(EnumVal* id, RecordVal*
> columns) seems to be using memory that might have been freed already.
> CreateWriter function can delete info_and_fields and still return non null
> writer. Any suggestions how to fix it? Maybe do not delete info in
> CreateWriter if old writer still exists? Will info's memory be freed
> somewhere later?

This is not a problem. If you check the 2 cases in which
delete_info_and_fields are called, you see that one of them is when
FindStream returns a nullptr, and the other one is if the Steram already
exists in WriterMap.

If you look at the Manager::Write function, both of these cases are
checked before calling CreateStream - in both of these cases, CreateStream
will not be called. So - this write-after-use cannto happen.

I think I tried to think about a way to make this nicer looking in the
past, but was not able to find one.

> Another question is - inside src/input/Manager.cc -> a lot of places where
> ev is allocated from EnumVal, it is not being freed or Unrefed.  Eg in
> function Manager::Delete(...reader, vals) , it seems like predidx and ev
> are allocated, but are not freed if there was not convert_error. This looks
> like memory is leaked in all those cases.

The values here are only Unref'd in the error case, because they are
consumed, and will be deleted, by CallPred in the case where there is no
error. In this case they are basically "passed to scriptland" and their
freeing will be managed by the script interpreter.

Sadly it is a bit difficult to follow what happens in cases like these.
Basically, one needs to knpw if each function call that goes into the Bro
core will take ownership and free the data structure afterwards, or if
that is up to the calling function. Often this means looking the function
that is called up to see what happens there - and even then one has to run
tests afterwards to make sure one got it correct in all cases.

I hope this made a few things slightly more clear,
 Johanna


>  if ( stream->pred )
> {
> int startpos = 0;
> Val* predidx = ValueToRecordVal(i, vals,
> stream->itype, &startpos, convert_error);
> 
> if ( convert_error )
> Unref(predidx);
> else
> {
> Ref(val);
> EnumVal *ev = new
> EnumVal(BifEnum::Input::EVENT_REMOVED, BifType::Enum::Input::Event);
> 
> streamresult =
> CallPred(stream->pred, 3, ev, predidx, val);
> 
> if ( streamresult == false )
> {
> // keep it.
> Unref(idxval);
> success = true;
> }
> }
> 
> }
> 
> 
> Thank you for suggestions,
> 
> Martina

> ___
> bro-dev mailing list
> bro-dev@bro.org
> http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] Merged branches deletion

[Johanna Amann]

This is done.

Johanna

On 30 Jan 2018, at 15:36, Johanna Amann wrote:

> Hi,
>
> I am going to delete these (merged) branches thursday, unless someone
> feels especially attached to them:
>
>   topic/dnthayer/ticket1821
>   topic/dnthayer/ticket1836
>   topic/dnthayer/ticket1863
>   topic/jazoff/contentline-limit
>   topic/jazoff/fix-gridftp
>   topic/jazoff/fix-intel-error
>   topic/jazoff/speedup-for
>   topic/robin/broker-logging
>   topic/robin/event-args
>   topic/robin/plugin-version-check
>   topic/seth/add-file-lookup-functions
>   topic/seth/input-thread-behavior
>   topic/seth/remove-dns-weird
>   topic/vladg/bit-1838
>
> Johanna
> ___
> bro-dev mailing list
> bro-dev@bro.org
> http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] Merged branches deletion

[Johanna Amann]

Hi,

I am going to delete these (merged) branches thursday, unless someone
feels especially attached to them:

  topic/dnthayer/ticket1821
  topic/dnthayer/ticket1836
  topic/dnthayer/ticket1863
  topic/jazoff/contentline-limit
  topic/jazoff/fix-gridftp
  topic/jazoff/fix-intel-error
  topic/jazoff/speedup-for
  topic/robin/broker-logging
  topic/robin/event-args
  topic/robin/plugin-version-check
  topic/seth/add-file-lookup-functions
  topic/seth/input-thread-behavior
  topic/seth/remove-dns-weird
  topic/vladg/bit-1838

Johanna
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] 'async' update and proposal

[Johanna Amann]

> I don't think skipping "async" this would be a big deal for anything,
> as the cases where the new behaviour may actually lead to significant
> differences should be rare.

After pondering this for a while I am a bit afraid that skipping async
completely might lead to quite hard to debug problems in scripts.

At the moments, Bro scripts are basically predictable - e.g. once an event
is executed you know that everything in that event happens before anything
else is executed.

Having asynchronous functions (obviously) changes that - now other things
will execute while you wait for something to happen. Without an async
keyword, users basically have no direct way to determine if a function
will run to the end without interruption. This can be especially
problematic if you manipulate data structures in an event that have to be
present in later events (as it is now commonly done in scripts).

Consider e.g. an example like the following

event protocol_event_1(...)
{
c$proto$la = function_call;
}

event protocol_event_end(...)
{
Log::write([c$proto$la...]);
}

If I understand everything correctly, in this case it is not guaranteed
to be present - it still might be waiting for function_call (if it is
asynchronous). This might not be a problem for cases in which functions
that obviously need DNS lookups are used - but if this is hidden between a
few layers of indirection it will get really hard to reason about this.

I actually think that it makes sense to be more explicit there. So -
require async in front of all bifs, etc that are asynchronous. And if a
user creates a function that in turn calls an asynchronous function, I
think we should require that function to be called using async too. Either
a user knows that a function uses an asynchronous function, or the script
interpreter will raise an error message telling them that the async
keyword is required here because an async function is in the call stack.
While this might be a bit painful at times I think this still is better
because it makes the script writer aware that things might be interrupted
at this point.

So - my argument is basically exactly the reverse of yours - if an async
function is somewhere in the call stack I definitely would want to know
about this when writing my script - otherwhise I can see really nasty bugs
happening.

If we want to make it more explicit that a function is potentially
asynchronous, we also could consider requiring the function definition to
mark this explicitly, e.g. just by adding the async keyword in front:

async function lookup(name: string) : set[addr] {
local addrs = async lookup_hostname(name);
return addrs;
}

...but that is not strictly necessary and does not look that pretty.

Johanna
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] Feedback on configuration framework implementation

[Johanna Amann]

On 7 Dec 2017, at 14:56, Azoff, Justin S wrote:

>> On Dec 7, 2017, at 5:22 PM, Johanna Amann  
>> wrote:
>
>> Indeed, that is my thought. This seems like a job for broker, instead 
>> of trying to somehow force this into a complex ascii-representation.
>>
>> Note that this is just a limitation of the config reader - the rest 
>> of the config framework (that does not deal with file reading) does 
>> not care what you throw at it and will happily accept tables, etc. So 
>> if you get Broker to give you a table you should be able to just use 
>> the calls for setting options with that table afterwards.
>>
>> Johanna
>
> Cool.. so you figure something like a python script to load/organize 
> your data from whatever upstream source you have, then just call 
> Option::set using broker?

Yup.

> I think the ascii representation of data structures would still help 
> in a few places.. bro is in a weird place right now where we have json 
> and 'print' that can output an
> ascii representation of almost any data structure, but what it outputs 
> is not always valid bro code that can be parsed in the other 
> direction.

You might have a point - however that kind of is a problem that I think 
is out of scope for the config framework. Plus - if we ever introduce 
something it will work without changes with the config framework as it 
is now (and the config reader should be able to read anything new that 
the input framework supports because it just re-uses functionality)

Johanna
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] Feedback on configuration framework implementation

[Johanna Amann]

> > think of that. I honestly also never liked modifying the values that are 
> > passed in arguments; this is for example also theoretically possible for 
> > events, but something that we have avoided to use in practice so far.
> 
> Yeah, and it also won't work for atomic values, at least not since
> https://github.com/bro/bro/commit/5b889360705120c9061390214881ea376819c669
> went in. :)

And as far as I can tell that applies to hooks too, true?

This is actually a but sneaky - it should not be a problem for
Intel::extend_match that Jan mentioned earlier because it is unlikely that
someone will just assign a new value to info. But if someone does it will
fail.

Which, after thinking about it for a few moments seems like the right
choice in any case. :)

Johanna
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] Feedback on configuration framework implementation

[Johanna Amann]

On 30 Nov 2017, at 10:22, Jan Grashöfer wrote:

> On 30/11/17 19:01, Johanna Amann wrote:
>>> 1. Thinking of handlers that may change values and are associated 
>>> with a
>>> priority, hooks come to my mind (e.g. Intel::extend_match). Are
>>> functions preferable compared to hooks here?
>>
>> In this case - yes. The problem with hooks is that they cannot return 
>> a
>> value, which is used here to let user change (or reject) changes to
>> options. :)
>
> The Intel::extend_match hook allows changing values or rejecting as 
> well. If the "chain of hooks" is "broken", i.e. one hook executed a 
> break statement, the call to the hook returns false and (in that case) 
> the log write is rejected. Otherwise, all changes made to the hook 
> arguments inside the handlers are propagated allowing changes.

Ah, you have a point there it is possible to do it like that, I did not 
think of that. I honestly also never liked modifying the values that are 
passed in arguments; this is for example also theoretically possible for 
events, but something that we have avoided to use in practice so far. 
Functionally they are, however, obviously equivalent.

I think I prefer functions in this case from a stylistic point of view. 
I am happy to change it over to hooks though if there is a consensus 
that that is the more fitting approach here. :)

Johanna
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] Feedback on configuration framework implementation

[Johanna Amann]

> 1. Thinking of handlers that may change values and are associated with a 
> priority, hooks come to my mind (e.g. Intel::extend_match). Are 
> functions preferable compared to hooks here?

In this case - yes. The problem with hooks is that they cannot return a
value, which is used here to let user change (or reject) changes to
options. :)

> > config reader
> > =
> > 
> > The config reader provides a way to read configuration files back into
> > Bro. Most importantly it automatically converts values to the correct
> > types. This is important because it is at least inconvenient (and
> > sometimes near impossible) to perform the necessary type conversions in
> > Bro scripts themselves. This is especially true for sets/vectors.
> > 
> > Configuration generally look like this:
> > 
> > [option name][tab/spaces][new variable value]
> 
> 2. Are module namespaces part of the option name (e.g. 
> "Notice::reply_to" vs. "reply_to")?

Yes

Johanna
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] Feedback on configuration framework implementation

[Johanna Amann]

Hello everyone,

the branch topic/johanna/config contains an implementation of the
configuration framework as it was discussed in an earlier thread on this
list. GitHub link: https://github.com/bro/bro/compare/topic/johanna/config

The implementation is basically what we discussed in the earlier thread
with some additional components like a reader for configuration values and
a script-level framework.

It would be great if people could take a look at all of this and see if
this makes sense, or if they see any problems with the implementation as
it is at the moment.

In the rest of the mail I wil go into a bit more detail and describe the
different parts of this change. Note that the rest of this email will be
very similar to the git commit message which also describes this change :)

The configuration framework consists of three mostly distinct parts:

* option variables
* the config reader
* the script level framework

option variable
===

The option keyword allows variables to be specified as run-tine options.
Such variables cannot be changed using normal assignments. Instead, they
can be changed using Option::set. It is possible to "subscribe" to
options and be notified when an option value changes.

Change handlers can also change values before they are applied; this
gives them the opportunity to reject changes. Priorities can be
specified if there are several handlers for one option.

Example script:

option testbool: bool = T;

function option_changed(ID: string, new_value: bool): bool
  {
  print fmt("Value of %s changed from %s to %s", ID, testbool, new_value);
  return new_value;
  }

event bro_init()
  {
  print "Old value", testbool;
  Option::set_change_handler("testbool", option_changed);
  Option::set("testbool", F);
  print "New value", testbool;
  }

config reader
=

The config reader provides a way to read configuration files back into
Bro. Most importantly it automatically converts values to the correct
types. This is important because it is at least inconvenient (and
sometimes near impossible) to perform the necessary type conversions in
Bro scripts themselves. This is especially true for sets/vectors.

Configuration generally look like this:

[option name][tab/spaces][new variable value]

so, for example:

testaddr 2607:f8b0:4005:801::200e
testinterval 60
testtime 1507321987
test_set a  b   c   d   erdbeerschnitzel

The reader uses the option name to look up the type that variable has in
the Bro core and automatically converts the value to the correct type.

Example script use:

type Idx: record {
  option_name: string;
};

type Val: record {
  option_val: string;
};

global currconfig: table[string] of string = table();

event InputConfig::new_value(name: string, source: string, id: string, value: 
any)
  {
  print id, value;
  }

event bro_init()
  {
  Input::add_table([$reader=Input::READER_CONFIG, $source="../configfile", 
$name="configuration", $idx=Idx, $val=Val, $destination=currconfig, 
$want_record=F]);
  }

Script-level config framework
=

The script-level framework ties these two features together and makes
them a bit more convenient to use. Configuration files can simply be
specified by placing them into Config::config_files. The framework also
creates a config.log that shows all value changes that took place.

Usage example:

redef Config::config_files += {configfile};

export {
  option testbool : bool = F;
}

The file is now monitored for changes; when a change occurs the
respective option values are automatically updated and the value change
is written to config.log.

Other changes
=

Internally, this commit also performs a range of changes to the Input
manager; it marks a lot of functions as const and introduces a new
ValueToVal method (which could in theory replace the already existing
one - it is a bit more powerful).

This also changes SerialTypes to have a subtype for Values, just as
Fields already have it; I think it was mostly an oversight that this was
not introduced from the beginning. This should not necessitate any code
changes for people already using SerialTypes.

Johanna
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] design summary: porting Bro scripts to use Broker

[Johanna Amann]

> Script-Author Example Usage
> ---
> 
> # Script author that wants to utilize data stores doesn't have to be aware of
> # whether user is running a cluster or if they want to use persistent storage
> # backends.
> 
> const Software::tracked_store_name = "bro/framework/software/tracked" &redef;
> 
> global Software::tracked_store: opaque of Broker::Store;
> 
> event bro_init() &priority = +10
>   {
>   Software::tracked_store = Broker::InitStore(Software::tracked_store_name);
>   }

I hope that this was not already answered somewhere else and I just missed
it - after you set up a store with Broker::InitStore, how do you interact
with Software::tracked_store?

I am especially curious how this handles the strong typing of Bro.

Johanna
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] design summary: porting Bro scripts to use Broker

[Johanna Amann]

> On Fri, Oct 06, 2017 at 16:53 +, you wrote:
> 
> > # contains topic prefixes
> > const Cluster::manager_subscriptions: set[string] &redef;
> > 
> > # contains (topic string, event name) pairs
> > const Cluster::manager_publications: set[string, string] &redef;
> 
> I'm wondering if we can simplify this with Broker. With the old comm
> system we needed the event names because that's what was subscribed
> to. Now that we have topics, does the cluster framework still need to
> know about the events at all? I'm thinking we could just go with a
> topic convention and then the various scripts would publish there
> directly.
> 
> In the most simple version of this, the cluster framework would just
> hard-code a subscription to "bro/cluster/". And then scripts like the
> Intel framework would just publish all their events to "bro/cluster/"
> directly through Broker.
> 
> To allow for distinguishing by node type we can define separate topic
> hierarchies: "bro/cluster/{manager,worker,logger}/". Each node
> subscribes to the hierarchy corresponding to its type, and each script
> publishes according to where it wants to send events to (again
> directly using the Broker API).
> 
> I think we could fit in Justin's hashing here too: We add per node
> topics as well ("bro/cluster/node/worker-1/",
> "bro/cluster/node/worker-2/", etc.) and then the cluster framework can
> provide a function that maps a hash key to a topic that corresponds to
> currently active node:
> 
> local topic = Cluster:topic_for_key("abcdef"); # Returns, e.g., 
> "bro/cluster/node/worker-1"
> Broker::publish(topic, event);
> 
> And that scheme may suggest that instead of hard-coding topics on the
> sender side, the Cluster framework could generally provide a set of
> functions to retrieve the right topic:
> 
> # In SumStats framework:
> local topic = Cluster::topic_for_manager() # Returns 
> "bro/cluster/manager".
> Broker::public(topic, event);
> 
> Bottom-line: If we can find a way to steer information by setting up
> topics appropriately, we might not need much additional configuration
> at all.

Just to add my two cents here - I like this a whole lot better and agree
that mostly steering events through topics seems like a neat choice.

Johanna
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] Configuration framework syntax proposal

[Johanna Amann]

I like that - I honestly did not really like configopt from the beginning,
the only reason for choosing it is that "config" conflicts with the
input/logging framework.

I am not completely sure we should deprecate const &redefs - there might
actually be a need for constants that can really only be changed on
startup. I like the idea of Jon to also remove the current ways to
manipulate &costs at runtime though :)

Johanna

On Fri, Sep 22, 2017 at 04:31:07PM +, Robin Sommer wrote:
> I was thinking the same when discussing an earlier proposal with
> Johanna. The "configopt" idea came out of that with the observation
> that "const &redef" isn't quite fitting here (and, as you say, it's
> already blurred anyways). At that time, however, the thinking was
> still to have a 2nd namespace, and writing 'configopt X: string
> &config="a.b.c"' seemed a bit too much. But if we just go with a more
> generic display name via Broxygen, then I'm back to liking it --
> except maybe for the name, how about "option" instead of "configopt"?
> So we'd arrive at something like this (similar to what has been said
> already):
> 
> module Foo;
> 
> export {
> 
> ## The username for our new feature.
> ##
> ## Display: User Name
> option user_name: string;
> 
> }
> 
> And we could even start deprecating "const ... &redef" if we wanted.
> 
> Robin
> 
> On Fri, Sep 22, 2017 at 15:59 +, you wrote:
> 
> > 
> > > On Sep 21, 2017, at 11:50 AM, Johanna Amann  wrote:
> > > 
> > > The feature that the
> > > configuration feature wants to bring is the ability to change options
> > > during runtime - which cannot be accomplished with redefs. redef-able
> > > consts will still have their place afterwards (for everything that still
> > > cannot be changed during runtime).
> > 
> > Just had a misc. thought related to the use of ‘const’:
> > 
> > I remember first being confused/unfamiliar with Bro’s use of ‘const’ and 
> > thought it meant “never changes” only to learn it’s further qualified as 
> > “never changes at run-time” so that the ‘const’ + ‘&redef’ combo ultimately 
> > means “never changes at run-time, but initial value may be changed at 
> > parse-time”.
> > 
> > Though, technically, ‘const’ can still be modified at run-time, if you know 
> > how… e.g. send_id...
> > 
> > And that’s maybe all ok -- it’s easy to explain unfamiliar context as I did 
> > above and the means of subverting runtime modification restrictions isn’t 
> > actively advertised as such.  Though, with a new config framework, there’s 
> > opportunities:
> > 
> > 1) could remove need for the backdoor method of modifying ‘const’ values at 
> > runtime, (e.g. via send_id) as that’s done through new identifiers 
> > explicitly tagged for config purposes
> > 
> > 2) using a new ‘configopt’ access modifier may be warranted over re-using 
> > ‘const’ for configuration values as the semantics are now blatantly 
> > different than what ‘const’ is expected to mean.  i.e. config values are 
> > meant to change at run-time, but only via a restricted API and ‘const’ is 
> > still intended to never change at run-time
> > 
> > - Jon
> > 
> > ___
> > bro-dev mailing list
> > bro-dev@bro.org
> > http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
> > 
> 
> 
> 
> -- 
> Robin Sommer * ICSI/LBNL * ro...@icir.org * www.icir.org/robin
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] Configuration framework syntax proposal

[Johanna Amann]

> I think that it's important to have this behavior come with a reasonable
> default. I think that whatever we choose should just work out of the box.
> For example, I think the existing construct should continue to work:
> 
> > const user_name: string &redef

I agree; note that what I proposed preserves this compatibility (it does
not change anything at all about redefs). The feature that the
configuration feature wants to bring is the ability to change options
during runtime - which cannot be accomplished with redefs. redef-able
consts will still have their place afterwards (for everything that still
cannot be changed during runtime).

> At the end of the day, what we're discussing is how a developer should
> document and expose a feature to an end-user. If, as a developer, I choose
> a bad variable name, then I'm not providing a good experience for the
> end-user, but that's my decision. I don't think that forcing developers to
> essentially add documentation via syntactic sugar is the right approach. If
> their variable names are confusing, people are less likely to use their
> script.
> 
> I think that a lot of what users might want to re-configure is too complex
> to be explained through a variable name anyway. We already have a system in
> place to document variables, and I think that we need to rely on that
> instead of focusing so much on which name is exposed to the user.

I agree with this.

> As we look at moving Bro scripts to packages, I think we need to look at
> how other package repositories have handled similar configuration options.
> Puppet Forge, for instance, has a types tab which documents the names of
> the parameters, and what they do:
> https://forge.puppet.com/puppetlabs/mysql/types This would be pretty easy
> to do with the Broxygen documentation, and a UI could also expose this.

Yup, I also agree with this.

Johanna
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] Configuration framework syntax proposal

[Johanna Amann]

On 21 Sep 2017, at 9:10, Siwek, Jon wrote:

>> On Sep 21, 2017, at 10:37 AM, Johanna Amann  
>> wrote:
>>
>> The only thing that I would like to avoid (which is obviously 
>> separate
>> from this) is internally remapping variable names to configuration 
>> names
>> in a non-reversible manner; then one suddenly has to think about what 
>> to
>> do when names conflict (several variable names being able to 
>> automatically
>> map to the same configuration name). But - that seem to be separate
>> concern :)
>
> Still not sure how much of an issue that is, provided the display 
> names are only for display and not used to actually locate/update 
> identifier values.  E.g. if a user sees 2 “User Name” fields in a 
> UI, I think we’re still able to fall back on the broxygen 
> documentation comments to provide more context to the user.  Or if 
> theres standardized/automatic conventions for these display names that 
> are based on modules/namespacing, I’m not sure how often you’d 
> even see such conflicts, or ’d expect they’d get patched out 
> pretty rapidly by the community when they pop up.

This actually was my point - which I apparently did not make clear. As 
long as it is only for display it is not a problem - I just don't want 
it to be used for identification :)

Johanna
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] Configuration framework syntax proposal

[Johanna Amann]

On Thu, Sep 21, 2017 at 02:57:31PM +, Siwek, Jon wrote:
> 
> > I'm not sure how exposing something like "input.pcap.filter" is any 
> > different from exposing something like "Pcap::filter" from that standpoint. 
> >  Maybe there's a larger discussion here around what the user experience 
> > should look like?  I feel like two different things are being talked about 
> > now.
> 
> I’m thinking on the same lines.

Yes, I can see that argument.


> > Directly using variable names in UI elements is not unheard of though, a 
> > lot of UI frameworks will do things like present a variable like user_name 
> > as "User Name" in the UI.  This is usually a simple text transform like
> > 
>  s='My_Program::user_name'
>  s.replace("::", " - ").replace("_", " ").title()
> >'My Program - User Name’
> 
> Yeah, I’ve noticed this approach in other UIs as well.

I admittedly did not think of this - that could work as a neat default.

The only thing that I would like to avoid (which is obviously separate
from this) is internally remapping variable names to configuration names
in a non-reversible manner; then one suddenly has to think about what to
do when names conflict (several variable names being able to automatically
map to the same configuration name). But - that seem to be separate
concern :)

Johanna
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] Configuration framework syntax proposal

[Johanna Amann]

On Thu, Sep 21, 2017 at 08:22:23AM -0700, Robin Sommer wrote:
> > comments. Like Jan, I had a hard time understanding the benefit having
> > two names for the same value: the identifier and config string.
> 
> Yeah, that's been my original concern as well. What if we focused that
> new attribute just on displaying something to the user:
> 
> const user_name: string &redef &display_name="User name"

I am not sure that we do need a new language element for that at all. If
we want a new attribute for just displaying information in a different
way, that (at least to me) feels more like something broxygen would do
(i.e. something that a script writer could put into one of the ## comments
if they so desire for the respective variable).

That being said, I still think it would be nice to have something in the
Bro language to denote that a value is a configuration option, mostly for
the reasons stated in the very first email. The biggest reason from my
point of view is strong typing - we tried to implement this just as Bro
scripts and it ends up not so nice.

So - how about something like this:

## The username for our new feature
##
## Display: Feature User Name
const user_name: string &config;

or

configopt user_name: string;

The comment block identifies the display name which can be picked up by
the UI (and documentation generation). The &config attribute (or the
configopt specifier) specifies this as a configuration option. The name in
the comment blocks also can be used for other language elements and will
be used in the documentation.

Johanna
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] Configuration framework syntax proposal

[Johanna Amann]

On Wed, Sep 20, 2017 at 10:00:33PM -0500, Daniel Thayer wrote:
> On 9/20/17 5:24 PM, Johanna Amann wrote:
> > advantages or disadvantages. One idea is to have a different syntax to 
> > define
> > the configuration option. Instead of
> > 
> > const filter = "ip" &config="input.pcap.filter";
> > 
> > the definition could look like
> > 
> > configopt filter = "ip";
> > 
> If we decide to use the "const" syntax, then is the plan
> to allow a const to have both the &config and &redef attributes?
> (presumably, we wouldn't allow "&redef" with "configopt")

Actually, yes, that was the thought so far; since they do not interact,
they are combineable if someone would desire this.

Johanna
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] Configuration framework syntax proposal

[Johanna Amann]

> Could the definition be
> 
> const filter = “ip” &config;
> 
> if you just wanted to use NameSpace::filter ?  That kinda seems like the best 
> of both worlds… Especially if anything marked &redef was automatically 
> registered as a configuration variable.

technically - yes. Though I am not quite sure that I like it :).

On the redef side - this specifically does not touch the functionality of
redef and also does not aim to automatically integrate with redef. The
background is that we do not know if a variable that is currently
redef-able will work as a configuration variable, or needs additional
commands to be run (or just works if set at startup as it is actually the
case with a lot of the current consts). I don't think going the route of
intermingling that would be a good idea - if someone wants something to be
a config variable, I think it should be an explicit opt-in.

> Thinking of all my scripts that could use this feature I think I would always 
> want NameSpace::option.

Ok. That would actually more pull me to using the other syntax again
(configopt varname) and not doing &plugin at all.

Thanks a lot :)
 Johanna
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] Configuration framework syntax proposal

[Johanna Amann]

Hello bro-dev,

in this email I want to get feedback on a possible syntax for the configuration
framework. The aim of the configuration framework is to provide an easy method
for Bro users and script writers to change configuration options during the
runtime of Bro (as opposed to only on startup as already possible using redef).

Let me start with an example what a script using the configuration framework
could look like:



module PcapFilter;

export {
## Documentation goes here:
## Set the PCAP filter that will be applied.
const filter = "ip" &config="input.pcap.filter";
}

redef enum PcapFilterId += {
NewFilter
};

function install_filter()
{
precompile_pcap_filter(NewFilter, filter);
install_pcap_filter(NewFilter);
}

# hook that is called when the configuration value changes
hook config_change(name: string, old_value: string)
{
install_filter();
}

event bro_init()
{
# Register and cause the hook to be called on configuration changes
Config::register_for_change("input.pcap.filter", config_change);
install_filter();
}



The two parts here are the definition of the configuration value and
an (optional) hook that can be defined to execute when a configuration option
changes.

The line 'const filter = "ip" &config="input.pcap.filter"' binds the constant
filter to the configuration option input.pcap.filter. When a user updates the
configuration option, the const will automatically change. For simple
configuration options, this line is all that is required to have a configurable
options.

For cases in which additional actions have to be performed when a configuration
option changes (e.g. calling BIFs), a hook can be registered to listen to the
change. The hook will be executed right after the value has been updated, gets
access to the previous value, and can perform any necessary actions.

Note that this proposal preserves typing: register_for_change will enforce that
the hook that is passed in (in this case config_change) will have a signature
that fits to the configuration option.

If a user desires to change a configuration option from script-land (e.g. in
local.bro), this is possible using 'Config::set'. For example:
Config::set("input.pcap.filter", "not ip");
Config::set also enforces typing and only accepts variables with the same type
as the const that was defined.

There obviously are a few more parts to this (storage backends, etc); however
these should be very flexible as we aim to implement as much as possible in the
script-land. There will be a few more BIFs that are necessary to write the
script-part of the config-framework; these are mainly needed for introspection.

In this email I am mostly interested in seeing if people think that this syntax
is a good idea, or if there are any better way to do this. I already talked
to a few people where a few other ideas were brought up, all having different
advantages or disadvantages. One idea is to have a different syntax to define
the configuration option. Instead of

const filter = "ip" &config="input.pcap.filter";

the definition could look like

configopt filter = "ip";

In this case, the name of the option would have to be automatically deduced and
look like "NameSpace::filter". The syntax is slightly nicer in this case, but
the configuration option naming is not as easy to structure.

So - I would be happy for any feedback or ideas on this.

Johanna

___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] Check if table element exists

[Johanna Amann]

Use "in":

if ("my_dynamic_name" in mytable)

Johanna

On 8 Aug 2017, at 11:04, Reinhard Gentz wrote:

> Hi,
>
> I would like to check if a certain table element exists and then take
> corresponding action like the following:
>
> if (exists(mytable["my_dynamic_name"]))
> do something
> else
> do something else
>
>
> Can someone give me a hint?
> Reinhard
> ___
> bro-dev mailing list
> bro-dev@bro.org
> http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] 2.5.1 release?

[Johanna Amann]

Looking over Changes, there is nothing that I am especially concerned
about at a cursory glance.

Johanna

On Fri, May 12, 2017 at 05:08:51PM -0400, Seth Hall wrote:
> 
> > On May 12, 2017, at 4:34 PM, Robin Sommer  wrote:
> > 
> >  I just went through the CHANGES since 2.5 and
> > I can see snapshotting current master as 2.5.1 if folks here are fine
> > with doing a broader release (which will need a bit more testing than
> > just a bugfix update would). Any opinions?
> 
> I'd be fine with that.  I think master is quite stable right now anyway.
> 
>   .Seth
> 
> --
> Seth Hall * Corelight, Inc * s...@corelight.com * www.corelight.com
> 
> ___
> bro-dev mailing list
> bro-dev@bro.org
> http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
> 
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] can I send an opaque of bloomfilter over Cluster::manager2worker_event ?

[Johanna Amann]

>> 1493427133.170419   Reporter::ERROR incompatible hashers in 
>> BasicBloomFilter merge  (empty) -
>
>> Not sure if the error is because an opaque of bloomfilter cannot be
>> sent over worker2manager_events and manager2worker_events or if I am
>> doing something not quite right.
>
> Bloom filters should work over communication. What's the code for the
> two sides? The error messages sound like these are filters of
> different types.

Actually - I am not sure if we ever implemented consistent hashing over 
the cluster; if I remember it correctly, the bloom filters need to use 
the same seed for the hash-functions in all cluster nodes to be 
mergeable.

That might be the reason for this to fail (and the error message also 
seems to indicate this).

Johanna

___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] Splitting up init-bare?

[Johanna Amann]

On Fri, Feb 10, 2017 at 11:51:08AM -0600, Vlad Grigorescu wrote:
> What do people think about splitting up portions of init-bare into separate
> files, and having init-bare simply @load those files? Right now, it's a
> 4500+ line script that keeps growing, and it commonly results in conflicts.
> 
> For the protocols, I could see having a file such as
> protocols/kerberos/bare.bro which defines the appropriate types which are
> currently in init-bare.

That sounds like a good idea - I am not a big fan of the fact that a lot
of the protocol dependent datatypes are in init-bare currently.

I am just not sure if it might require a lot of fiddling to get the load
order of things correct; but assuming that every protocol has a bare.bro
(or perhaps a datatypes.bro or similar?), that should not be a huge issue.

Johanna
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] bro-pkg 1.0 available

[Johanna Amann]

And as a followup - this happens because Bro was not in the path.

This really should give a nicer error message though (or abort before 
even trying to install).

Johanna

On 27 Jan 2017, at 11:34, Johanna Amann wrote:

> And - second followup - this time I think I am doing things right this
> time.
>
> On os-x, when trying to install using bro-pkg, I get the following 
> output:
>
> $ bro-pkg install bro-sumstats-counttable --version master
> The following packages will be INSTALLED:
>   bro/0xxon/bro-sumstats-counttable (master)
>
> Proceed? [Y/n] y
> Running unit tests for "bro/0xxon/bro-sumstats-counttable"
> Traceback (most recent call last):
>   File "/Users/johanna/venv/bin/bro-pkg", line 1635, in 
> main()
>   File "/Users/johanna/venv/bin/bro-pkg", line 1631, in main
> args.run_cmd(manager, args, config)
>   File "/Users/johanna/venv/bin/bro-pkg", line 314, in cmd_install
> error, passed, test_dir = manager.test(name, version)
>   File 
> "/Users/johanna/venv/lib/python2.7/site-packages/bropkg/manager.py", 
> line 1622, in test
> bropath = os.path.dirname(stage_script_dir) + ':' + bropath
> TypeError: coercing to Unicode: need string or buffer, NoneType found
>
> The same happens with your bro-test-package.
>
> Is there anything obvious that I am doing wrong?
>
> Johanna
>
> On Fri, Jan 27, 2017 at 11:14:18AM -0800, Johanna Amann wrote:
>> Ah, and if you remember to specify --version master, things suddenly 
>> look
>> much better - ignore this :)
>>
>> Johanna
>>
>> On Fri, Jan 27, 2017 at 11:10:46AM -0800, Johanna Amann wrote:
>>> Hi Jon,
>>>
>>> On Wed, Jan 25, 2017 at 02:23:57AM +, Siwek, Jon wrote:
>>>> bro-pkg 1.0 is now out and supports
>>>>
>>>> * package unit testing [1]
>>>
>>> thanks for this. Are there any extra steps that one has to do for 
>>> this to
>>> work? I tried to activate it for my repository at
>>> https://github.com/0xxon/bro-sumstats-counttable, where the 
>>> bro-pkg.meta
>>> specifies
>>>
>>> test_command = cd testing && btest -d
>>>
>>> However, bro-pkg (version 1.0) seems to just ignore this:
>>>
>>> $ bro-pkg install bro-sumstats-counttable
>>> The following packages will be INSTALLED:
>>>   bro/0xxon/bro-sumstats-counttable (0.0.2)
>>>
>>> Proceed? [Y/n] y
>>> Running unit tests for "bro/0xxon/bro-sumstats-counttable"
>>> error: failed to run tests for bro/0xxon/bro-sumstats-counttable:
>>> Package does not specify a test_command
>>> Proceed to install anyway? [Y/n] n
>>>
>>> Am I doing something wrong here? Or is there a problem with the way 
>>> that I
>>> specify test_command? (The error message seems to indicate that it 
>>> is just
>>> not being identified though).
>>>
>>> Johanna
>>> ___
>>> bro-dev mailing list
>>> bro-dev@bro.org
>>> http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
>>>
> ___
> bro-dev mailing list
> bro-dev@bro.org
> http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] bro-pkg 1.0 available

[Johanna Amann]

And - second followup - this time I think I am doing things right this
time.

On os-x, when trying to install using bro-pkg, I get the following output:

$ bro-pkg install bro-sumstats-counttable --version master
The following packages will be INSTALLED:
  bro/0xxon/bro-sumstats-counttable (master)

Proceed? [Y/n] y
Running unit tests for "bro/0xxon/bro-sumstats-counttable"
Traceback (most recent call last):
  File "/Users/johanna/venv/bin/bro-pkg", line 1635, in 
main()
  File "/Users/johanna/venv/bin/bro-pkg", line 1631, in main
args.run_cmd(manager, args, config)
  File "/Users/johanna/venv/bin/bro-pkg", line 314, in cmd_install
error, passed, test_dir = manager.test(name, version)
  File "/Users/johanna/venv/lib/python2.7/site-packages/bropkg/manager.py", 
line 1622, in test
bropath = os.path.dirname(stage_script_dir) + ':' + bropath
TypeError: coercing to Unicode: need string or buffer, NoneType found

The same happens with your bro-test-package.

Is there anything obvious that I am doing wrong?

Johanna

On Fri, Jan 27, 2017 at 11:14:18AM -0800, Johanna Amann wrote:
> Ah, and if you remember to specify --version master, things suddenly look
> much better - ignore this :)
> 
> Johanna
> 
> On Fri, Jan 27, 2017 at 11:10:46AM -0800, Johanna Amann wrote:
> > Hi Jon,
> > 
> > On Wed, Jan 25, 2017 at 02:23:57AM +, Siwek, Jon wrote:
> > > bro-pkg 1.0 is now out and supports
> > > 
> > > * package unit testing [1]
> > 
> > thanks for this. Are there any extra steps that one has to do for this to
> > work? I tried to activate it for my repository at
> > https://github.com/0xxon/bro-sumstats-counttable, where the bro-pkg.meta
> > specifies
> > 
> > test_command = cd testing && btest -d
> > 
> > However, bro-pkg (version 1.0) seems to just ignore this:
> > 
> > $ bro-pkg install bro-sumstats-counttable
> > The following packages will be INSTALLED:
> >   bro/0xxon/bro-sumstats-counttable (0.0.2)
> > 
> > Proceed? [Y/n] y
> > Running unit tests for "bro/0xxon/bro-sumstats-counttable"
> > error: failed to run tests for bro/0xxon/bro-sumstats-counttable:
> > Package does not specify a test_command
> > Proceed to install anyway? [Y/n] n
> > 
> > Am I doing something wrong here? Or is there a problem with the way that I
> > specify test_command? (The error message seems to indicate that it is just
> > not being identified though).
> > 
> > Johanna
> > ___
> > bro-dev mailing list
> > bro-dev@bro.org
> > http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
> > 
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] bro-pkg 1.0 available

[Johanna Amann]

Ah, and if you remember to specify --version master, things suddenly look
much better - ignore this :)

Johanna

On Fri, Jan 27, 2017 at 11:10:46AM -0800, Johanna Amann wrote:
> Hi Jon,
> 
> On Wed, Jan 25, 2017 at 02:23:57AM +, Siwek, Jon wrote:
> > bro-pkg 1.0 is now out and supports
> > 
> > * package unit testing [1]
> 
> thanks for this. Are there any extra steps that one has to do for this to
> work? I tried to activate it for my repository at
> https://github.com/0xxon/bro-sumstats-counttable, where the bro-pkg.meta
> specifies
> 
> test_command = cd testing && btest -d
> 
> However, bro-pkg (version 1.0) seems to just ignore this:
> 
> $ bro-pkg install bro-sumstats-counttable
> The following packages will be INSTALLED:
>   bro/0xxon/bro-sumstats-counttable (0.0.2)
> 
>   Proceed? [Y/n] y
>   Running unit tests for "bro/0xxon/bro-sumstats-counttable"
>   error: failed to run tests for bro/0xxon/bro-sumstats-counttable:
>   Package does not specify a test_command
>   Proceed to install anyway? [Y/n] n
> 
> Am I doing something wrong here? Or is there a problem with the way that I
> specify test_command? (The error message seems to indicate that it is just
> not being identified though).
> 
> Johanna
> ___
> bro-dev mailing list
> bro-dev@bro.org
> http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
> 
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] bro-pkg 1.0 available

[Johanna Amann]

Hi Jon,

On Wed, Jan 25, 2017 at 02:23:57AM +, Siwek, Jon wrote:
> bro-pkg 1.0 is now out and supports
> 
> * package unit testing [1]

thanks for this. Are there any extra steps that one has to do for this to
work? I tried to activate it for my repository at
https://github.com/0xxon/bro-sumstats-counttable, where the bro-pkg.meta
specifies

test_command = cd testing && btest -d

However, bro-pkg (version 1.0) seems to just ignore this:

$ bro-pkg install bro-sumstats-counttable
The following packages will be INSTALLED:
  bro/0xxon/bro-sumstats-counttable (0.0.2)

Proceed? [Y/n] y
Running unit tests for "bro/0xxon/bro-sumstats-counttable"
error: failed to run tests for bro/0xxon/bro-sumstats-counttable:
Package does not specify a test_command
Proceed to install anyway? [Y/n] n

Am I doing something wrong here? Or is there a problem with the way that I
specify test_command? (The error message seems to indicate that it is just
not being identified though).

Johanna
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] Testing and Docs for Packages

[Johanna Amann]

On Tue, Jan 17, 2017 at 04:01:19AM +, Siwek, Jon wrote:
> > I actually think it would be neat to do this isolated, especially given
> > that this enables testing before installing.
> 
> Not sure I follow.  Can you explain further?

Sorry - what I meant is that the tests can run before the packages are put
into the bro directory, so you can see if they will work with the
installed Bro version (or potentially system configuration) before putting
the files in. So you can use it as a prerequisite check for installation.

The other way round, you have to roll back after putting them already
there - unless I misunderstood something.

Plus - even in this case, shouldn't you be able to load the user scripts
by loading local.bro? Meaning we could even run all the tests twice - once
just with the default Bro installation, and once with the user changes,
both before installing the scripts, which could even give an indication if
Bro or other packages are at fault.

> > It also makes it easier to create something like "smokers" (Bro
> > installations that just tro tu run all testsuites of all available
> > packages with a newer version to see if something went wrong).
> 
> Can you also go into more detail on what you’re thinking there?
> 
> If there's concerns about accidentally corrupting an existing/production
> bro installation, the alternative I’d suggest would be to set up a
> separate bro-pkg config file for the smoke tests that would have bro-pkg
> install stuff in an isolated location.  This allows users to explicitly
> define the testing sandbox for themselves.

No, the idea would be more along the lines that, in this case, you might
actually never want to really install the package; you just want to see if
the tests can pass. Though, admittedly, this can once again be
accomplished by just immediately uninstalling afterwards.

Johanna
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] fatal error: can't find base/init-bare.bro

[Johanna Amann]

On Thu, Jan 19, 2017 at 11:19:20AM -0600, Alberto Garcia wrote:
> Hi,
> 
> I've compiled bro from source to do some debugging. Once compiled I can't
> run bro since there is an error popping up:
> 
> default@debian:~/bro$ ./build/src/bro
> fatal error: can't find base/init-bare.bro
> 
> If I do the make install and then call bro from /usr/local/bro/bin/bro it
> works fine.
> 
> What I should do to execute bro from the build directory?

source build/bro-path-dev.sh

should set all required environment variables.

Johanna
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] Testing and Docs for Packages

[Johanna Amann]

Just to add my two cents to this, because automated testing is actually
one of the things that I really think package managers should do...

On Mon, Jan 16, 2017 at 06:45:52PM +, Siwek, Jon wrote:
> 1) Add `bro-pkg test ` command.

Might it also make sense to just run the test on installation, before the
package is actually installed, to see if it works on the environment of
the user?

This might make it much easier for users (& developers) to identify early
when it is something wrong. And bro-pkg could just abort (or ask a user if
it should continue) if a test fails.

> 2) Add “test_command” field to bro-pkg.meta
> 
> The “test_command” is more general than “test_dir" — the command could just 
> `cd test_dir` if needed and there’s no other reason bro-pkg needs to know the 
> dir where tests are stored, is there?
> 
> Other questions:
> 
> 1) Is it fine for `bro-pkg test ` to operate on the installed 
> version of the package or are there expectations of testing a package in an 
> isolated sandbox without installing it?  I think the former is more useful 
> since it may catch odd inter-package conflicts that wouldn’t show up when 
> testing in isolation.

I actually think it would be neat to do this isolated, especially given
that this enables testing before installing. It also makes it easier to
create something like "smokers" (Bro installations that just tro tu run
all testsuites of all available packages with a newer version to see if
something went wrong).

Running on the installed version might be a neat bonus, but I actually see
the other as more interesting.

Johanna
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] plugins/hooks test fail in the new year

[Johanna Amann]

Hi,

plugins/hooks currently fails because of the changed year number:

  0.00 | HookCallFunction strftime(%Y, XX.XX)
  0.00 | HookCallFunction string_to_pattern((^\.?|\.)()$, F)
  0.00 | HookCallFunction sub((^\.?|\.)(~~)$, <...>/, )
  -0.00 | HookCallFunction to_count(2016)
  +0.00 | HookCallFunction to_count(2017)

After a slight amount of digging, the culprit is the following part of
init-bare.bro:

# A bit of functionality for 2.5
global brocon:event
(x:count);event
bro_init   (){event
brocon  (  to_count
(strftime ("%Y"
,current_time(;}

While I know this is cute, currently this will make the test fail on every year
change. Do we perhaps want to either remove this, or at least move this 
somewhere
outside of base/?

Johanna
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] OpenFlow Analyzer

[Johanna Amann]

Just to add to this - there is no analyzer and so far this is also not 
planned. In addition to that - OpenFlow encourages use of TLS, so you 
(hopefully) should not actually see a lot of unencrypted OF traffic on 
the wire.

Johanna

On 17 Oct 2016, at 15:47, Slagell, Adam J wrote:

> I get you now. I am not aware of an open flow protocol analyzer in 
> Bro.
>
>> On Oct 17, 2016, at 2:45 PM, Hui Lin (Hugo)  
>> wrote:
>>
>> Actually, netcontrol is what I thought of as northbound APIs. I 
>> actually just wonder whether Bro has analyzer for openflow protocol 
>> or not (some refer them as southbound traffics). It not, I probably 
>> need to use wireshark to output the pcap and analyze them manually.
>>
>> On Mon, Oct 17, 2016 at 2:37 PM, Slagell, Adam J 
>> mailto:slag...@illinois.edu>> wrote:
>> Have you looked at the netcontrol framework in Bro, developed by 
>> Johanna?
>>
>>> On Oct 17, 2016, at 2:24 PM, Hui Lin (Hugo) >> > wrote:
>>>
>>> Hi
>>>
>>> My understanding is that Bro has some northbound API to openflow 
>>> switches or controllers. I am wondering whether any development 
>>> branch has analyzer of openflow controllers. Just try to see whether 
>>> I can use Bro to analyze some controller-to-switch traffics.
>>>
>>> Best,
>>>
>>> Hui
>>>
>>>
>>> -- 
>>> Hui Lin
>>> Ph.D. Candidate (http://hlin33.web.engr.illinois.edu/ 
>>> )
>>> DEPEND (http://depend.csl.illinois.edu/ 
>>> )
>>> ECE, Uni. of Illinois at Urbana-Champaign
>>>
>>> ___
>>> bro-dev mailing list
>>> bro-dev@bro.org 
>>> http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev 
>>> 
>>
>> --
>>
>> Adam J. Slagell
>> Chief Information Security Officer
>> Director, Cybersecurity Division
>> National Center for Supercomputing Applications
>> University of Illinois at Urbana-Champaign
>> www.slagell.info 
>> 
>>
>> "Under the Illinois Freedom of Information Act (FOIA), any written 
>> communication to or from University employees regarding University 
>> business is a public record and may be subject to public disclosure."
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> -- 
>> Hui Lin
>> Ph.D. Candidate (http://hlin33.web.engr.illinois.edu/ 
>> )
>> DEPEND (http://depend.csl.illinois.edu/ 
>> )
>> ECE, Uni. of Illinois at Urbana-Champaign
>>
>
> --
>
> Adam J. Slagell
> Chief Information Security Officer
> Director, Cybersecurity Division
> National Center for Supercomputing Applications
> University of Illinois at Urbana-Champaign
> www.slagell.info
>
> "Under the Illinois Freedom of Information Act (FOIA), any written 
> communication to or from University employees regarding University 
> business is a public record and may be subject to public disclosure."
>
> ___
> bro-dev mailing list
> bro-dev@bro.org
> http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] Potential of including TLSv1.3 support in Bro 2.5

[Johanna Amann]

As a follow-up: since all responses were positives, I filed a
merge-request for this and it should (hopefully) make it into 2.5.

Merge-request for those who want to follow it:
https://bro-tracker.atlassian.net/browse/BIT-1727

Johanna

On Fri, Oct 07, 2016 at 02:06:53PM -0700, Johanna Amann wrote:
> I just finished a branch that adds support for TLSv1.3 to Bro (branch
> topic/johanna/tls13, important commit:
> https://github.com/bro/bro/commit/fdef28ce7c3455d43267ab07dbb8ad96c9ea3890).
> 
> What do people think of the idea of adding that patch to the upcoming Bro
> 2.5 release?
> 
> I know that we are quite late in the current release process and that we
> should not really make any feature changes after releasing the beta.  It
> would, however, be neat to be able to support TLSv1.3 starting the moment
> that people actually start to use it; without that support, we will only
> have empty lines in ssl.log for these connections. Furthermore, the
> changes that are needed to support TLSv1.3 have nearly no interaction with
> the code that is used to parse earlier versions of TLS. Even if there are
> problems with the code (or if the on-the-wire format still changes), the
> only thing that should happen is that binpac throws errors. Which is
> exactly what already happens now when throwing TLSv1.3 sessions at the
> current master versions of Bro.
> 
> Thanks,
>  Johanna
> ___
> bro-dev mailing list
> bro-dev@bro.org
> http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
> 
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] Potential of including TLSv1.3 support in Bro 2.5

[Johanna Amann]

On 8 Oct 2016, at 17:38, Vlad Grigorescu wrote:

> Well, I should point out that Cloudflare enabled it a couple of weeks 
> ago:
> https://blog.cloudflare.com/introducing-tls-1-3/

You actually got that to run? I did not manage to get any client to 
successfully negotiate TLS 1.3 with them and set up my own server in the 
end. But perhaps they updated in the last few days...

> I was able to connect with my usual browser and grab a PCAP (after 
> setting
> the option in about:config). It seems to run just fine against the 
> branch
> (attached, in case it's of any use).
>
> Is there any way to detect TLS 1.3 with git master? I wouldn't expect 
> to
> see any, but I've been surprised once or twice before. I ran the PCAP
> against master, and while I did get an ssl.log, I didn't see anything 
> in
> there that would indicate it's TLS1.3.

Well, it will show up as a binpac error while parsing a specific TLS 
message. Not the best way to do it ;)

Johanna
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] Potential of including TLSv1.3 support in Bro 2.5

[Johanna Amann]

I would be happy if you test this branch - however, you are actually 
unlikely to trigger the new code paths. TLS 1.3 is still in the 
development stage, so much that I doubt that you will even encounter a 
single connection that uses it. At the moment, you have to enable it by 
hand in the development edition of browsers, and more or less compile 
your own server that is able to speak it.

(That being said, I am quite confident the on-the-wire format won't 
change significantly enough anymore that the new analyzer won't be able 
to parse it.)

Johanna

On 7 Oct 2016, at 17:03, Aashish Sharma wrote:

> I think the current feature freeze is a self-imposed limit out of 
> coding discipline - but it ok to make exceptions.  Esp since 2.6 would 
> be long way away.
>
> Risky as it is, It seems like inclusion of this code isn't going to 
> cause any significant problems. FWIW, I can run this branch on my end 
> for until release happens.
>
> Aashish
>
> On Fri, Oct 07, 2016 at 02:06:53PM -0700, Johanna Amann wrote:
>> I just finished a branch that adds support for TLSv1.3 to Bro (branch
>> topic/johanna/tls13, important commit:
>> https://github.com/bro/bro/commit/fdef28ce7c3455d43267ab07dbb8ad96c9ea3890).
>>
>> What do people think of the idea of adding that patch to the upcoming 
>> Bro
>> 2.5 release?
>>
>> I know that we are quite late in the current release process and that 
>> we
>> should not really make any feature changes after releasing the beta.  
>> It
>> would, however, be neat to be able to support TLSv1.3 starting the 
>> moment
>> that people actually start to use it; without that support, we will 
>> only
>> have empty lines in ssl.log for these connections. Furthermore, the
>> changes that are needed to support TLSv1.3 have nearly no interaction 
>> with
>> the code that is used to parse earlier versions of TLS. Even if there 
>> are
>> problems with the code (or if the on-the-wire format still changes), 
>> the
>> only thing that should happen is that binpac throws errors. Which is
>> exactly what already happens now when throwing TLSv1.3 sessions at 
>> the
>> current master versions of Bro.
>>
>> Thanks,
>>  Johanna
>> ___
>> bro-dev mailing list
>> bro-dev@bro.org
>> http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
>
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] Potential of including TLSv1.3 support in Bro 2.5

[Johanna Amann]

I just finished a branch that adds support for TLSv1.3 to Bro (branch
topic/johanna/tls13, important commit:
https://github.com/bro/bro/commit/fdef28ce7c3455d43267ab07dbb8ad96c9ea3890).

What do people think of the idea of adding that patch to the upcoming Bro
2.5 release?

I know that we are quite late in the current release process and that we
should not really make any feature changes after releasing the beta.  It
would, however, be neat to be able to support TLSv1.3 starting the moment
that people actually start to use it; without that support, we will only
have empty lines in ssl.log for these connections. Furthermore, the
changes that are needed to support TLSv1.3 have nearly no interaction with
the code that is used to parse earlier versions of TLS. Even if there are
problems with the code (or if the on-the-wire format still changes), the
only thing that should happen is that binpac throws errors. Which is
exactly what already happens now when throwing TLSv1.3 sessions at the
current master versions of Bro.

Thanks,
 Johanna
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] [Bro] broker make error

[Johanna Amann]

On 8 Aug 2016, at 8:20, Robin Sommer wrote:

> (Moving to bro-dev).
>
> On Sun, Aug 07, 2016 at 09:59 -0700, Johanna Amann wrote:
>
>> Yup, that is exactly it. There currently is a rewrite of Broker
>> underway, which will use the newer library versions, but it is not
>> quite done yet.
>
> I'm wondering if we should add a version check to Broker that clearly
> says what's is needed if it cannot find the right CAF version? People
> have run into this a few times now both ways (Broker wants newer/older
> CAF version). That seems something worth adding for 2.5 still.

Agreed; that also should not be very hard, I can try to take a look at it.

Johanna
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] package manager progress

[Johanna Amann]

And to add a me three to this - I am also with him on this one. On top 
of things - I might misremember this, but didn't we plan package names 
to include the github user name at one point of time? So a package name 
would be user/redis, for example, and there also could be user2/redis?

Johanna

On 27 Jul 2016, at 9:05, Matthias Vallentin wrote:

>> I actually don't like this that much because some of these can cross
>> boundaries and do all sorts of different things in a single plugin.
>> It makes more sense to me to leave the naming open.
>
> I'm with Seth on this one. The reason why I think we should keep the
> naming open is that it's the job of the meta data tags to take care of
> the grouping. If someone writes a redis package, then they should 
> apply
> the redis package. Encoding this meta data into the package name is
> quite limited, however.
>
> Matthias
> ___
> bro-dev mailing list
> bro-dev@bro.org
> http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] Remove application/pkix-cert from files.log?

[Johanna Amann]

I think kind of like having the certificates being handled as files by 
default. However, I see that most people who run clusters in production 
do not want that information in files.log. So - from my point of view, 
it might make sense to have a policy script that filters certificates 
from files.log and adds the hashes to x509.log; and we have that 
auto-loaded by default in local.bro.

Would that make sense?

Johanna

On 15 Jul 2016, at 8:08, Seth Hall wrote:

> What does everyone think of making some change for 2.5 so that 
> certificates from SSL aren't logged in the files.log by default?  I've 
> heard grumblings about the number of certs that show up from quite a 
> few people and personally noticed that the number of certificates will 
> dwarf all other files types pretty badly which makes the output look a 
> bit weird since very few people are ever interested in looking at 
> those files in the files.log.
>
> Certificates would still be passed through the files framework, so 
> it's not an architectural change, it would all be related to just not 
> doing the log.  There is one minor issue that this brings up though in 
> that right now certificate hashes are all given in the files.log.  We 
> could move them elsewhere like x509.log or ssl.log, but I'm curious if 
> anyone had thoughts on what they think would be most useful?
>
>   .Seth
>
> --
> Seth Hall
> International Computer Science Institute
> (Bro) because everyone has a network
> http://www.bro.org/
>
>
> ___
> bro-dev mailing list
> bro-dev@bro.org
> http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] New proposal (Re: CBAN naming)

[Johanna Amann]

On 9 Jun 2016, at 13:29, Matthias Vallentin wrote:

>> I see benefits in two separate repos:
>
> Yep.
>
>>  client: bro-pkg
>>  community packages: bro-pkg-community
>
> I'm not sure if I understand the -community suffix. The client bro-pkg
> makes sense to me. But the first association I have with
> bro-pkg-community is a community-version of bro-pkg (i.e., the 
> client).
> How about just "packages?" Forking github.com/bro/packages feels like 
> a
> natural and understandable descriptor.

And to throw another idea in that is in line with this - what about 
calling the community package repository "packages" like matthias said 
and the client repository "package-manager"?

github.com/bro/package-manager also seems quite clear :)

Johanna

___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] New proposal (Re: CBAN naming)

[Johanna Amann]

On Mon, Jun 06, 2016 at 02:14:50PM -0500, Daniel Thayer wrote:
> On 06/06/2016 01:50 PM, Robin Sommer wrote:
> > - For shipping binary plugins:
> >
> >  - Through meta information, we let the author specify a build
> >command to build all their binary stuff, such as "./configure &&
> >make && make test". The command line client runs that command
> >inside .
> >
> >  - Per default, we expect that build command to create a directory
> >/build that contains a binary plugin. We add
> >//build to BRO_PLUGIN_PATH.
> >
> >  - If one wants to locate the plugin elsewhere, optional meta
> >information can set a different BRO_PLUGIN_PATH.  For example,
> >to put it into "/compiled/cool_plugin", one would specify
> >as meta information "pluginpath=compiled/cool_plugin".
> >
> 
> 
> Could you clarify what you mean by BRO_PLUGIN_PATH here?  Are you saying
> that after I do a "cban install cool-plugin", I'd need to manually
> set an environment variable in order for Bro to find the new plugin?
> Or are we still planning to default to /lib/bro/plugins/ ?

No, you do not, that happens automatically. Users also have the ability to
specify a subdirectory inside of their repository that is then used as the
BRO_PLUGIN_PATH. That would be done by the plugin authors if they want to
change the directory structure. For the installing user, everything
remains automatic.

Johanna
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] [JIRA] (BIT-1580) Add ipv6 detection to conn.log

[Johanna Amann (JIRA)]

 [ 
https://bro-tracker.atlassian.net/browse/BIT-1580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Johanna Amann updated BIT-1580:
---
Resolution: Won't Do
Status: Closed  (was: Open)

Just to repeat the commont from the github pull request here:

Hi,

thank you very much for your pull request. Since this change only adds an 
additional field of data, which can already be deduced by the data that is 
present in conn.log, I do not think this is something we will want to add to 
the base scripts. While it might be convenient to have this for easy grepping 
in some cases, people who need this can easily add it to their own installation 
as a script that extends the conn.log

So - I would encourage you to change this to be a script that extends conn.log 
and publish it, e.g. in a bro-scripts repository in your github account. We 
will also create a easy way to add user scripts to bro in the future - things 
like these might make good candidates to be added to this.


> Add ipv6 detection to conn.log
> --
>
> Key: BIT-1580
> URL: https://bro-tracker.atlassian.net/browse/BIT-1580
> Project: Bro Issue Tracker
>  Issue Type: Patch
>  Components: Bro
>Affects Versions: 2.4
>Reporter: Malware Utkonos
>  Labels: IPv6
>
> This is an additional column added to conn.log to determine if the connection 
> is using ipv6. The address itself makes this clear, but it is much easier to 
> grep for T/F than examining the address.
> Pull request with patch:
> https://github.com/bro/bro/pull/70



--
This message was sent by Atlassian JIRA
(v1000.5.0#72002)
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] [JIRA] (BIT-1579) Please merge topic/johanna/xmpp-starttls

[Johanna Amann (JIRA)]

 [ 
https://bro-tracker.atlassian.net/browse/BIT-1579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Johanna Amann updated BIT-1579:
---
Status: Merge Request  (was: Open)

> Please  merge topic/johanna/xmpp-starttls
> -
>
> Key: BIT-1579
> URL: https://bro-tracker.atlassian.net/browse/BIT-1579
> Project: Bro Issue Tracker
>  Issue Type: New Feature
>  Components: Bro
>Affects Versions: git/master
>    Reporter: Johanna Amann
> Fix For: 2.5
>
>
> Please merge topic/johanna/xmpp-starttls
> This branch adds very basic support for XMPP, just up to the point when SSL 
> encryption starts, when it switches to the SSL analyzer. Similar to the case 
> of IMAP, this allows us to extract certificates from xmpp sessions that are 
> upgraded.



--
This message was sent by Atlassian JIRA
(v1000.5.0#72002)
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] [JIRA] (BIT-1579) Please merge topic/johanna/xmpp-starttls

[Johanna Amann (JIRA)]

Johanna Amann created BIT-1579:
--

 Summary: Please  merge topic/johanna/xmpp-starttls
 Key: BIT-1579
 URL: https://bro-tracker.atlassian.net/browse/BIT-1579
 Project: Bro Issue Tracker
  Issue Type: New Feature
  Components: Bro
Affects Versions: git/master
Reporter: Johanna Amann
 Fix For: 2.5


Please merge topic/johanna/xmpp-starttls

This branch adds very basic support for XMPP, just up to the point when SSL 
encryption starts, when it switches to the SSL analyzer. Similar to the case of 
IMAP, this allows us to extract certificates from xmpp sessions that are 
upgraded.




--
This message was sent by Atlassian JIRA
(v1000.5.0#72002)
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] [JIRA] (BIT-1577) Fix minor spelling errors

[Johanna Amann (JIRA)]

 [ 
https://bro-tracker.atlassian.net/browse/BIT-1577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Johanna Amann updated BIT-1577:
---
Resolution: Merged  (was: Fixed)
Status: Closed  (was: Merge Request)

Merged in f9db0f2e847eaeca028eac2974c709f8e2cb794f

> Fix minor spelling errors
> -
>
> Key: BIT-1577
> URL: https://bro-tracker.atlassian.net/browse/BIT-1577
> Project: Bro Issue Tracker
>  Issue Type: Task
>  Components: Bro
>Reporter: Jeannette Dopheide
>    Assignee: Johanna Amann
>
> Fixing minor spelling errors in Bro 2.4.1 found here: 
> https://lintian.debian.org/full/ben...@debian.org.html#bro_2.4.1_x2bdfsg-2
> Repository : ssh://g...@bro-ids.icir.org/bro
> On branch  : topic/jdopheid/typos
> Link   : 
> https://github.com/bro/bro/commit/635d218583014938c2ee732cb6a1dfdee0f2
> ---
> commit 635d218583014938c2ee732cb6a1dfdee0f2
> Author: Jeannette Dopheide 
> Date:   Mon Apr 25 11:49:04 2016 -0500
> Correcting spelling errors found under bro 2.4.1+dfsg-2 here:
> 
> https://lintian.debian.org/full/ben...@debian.org.html#bro_2.4.1_x2bdfsg-2
> ---
> 635d218583014938c2ee732cb6a1dfdee0f2
> src/RuleCondition.cc  | 2 +-
> src/RuleMatcher.cc| 2 +-
> src/Serializer.cc | 2 +-
> src/StateAccess.cc| 2 +-
> src/broxygen/Configuration.cc | 2 +-
> src/nb_dns.c  | 2 +-
> 6 files changed, 6 insertions(+), 6 deletions(-)
> diff --git a/src/RuleCondition.cc b/src/RuleCondition.cc
> index 68eb131..40ef5f0 100644
> --- a/src/RuleCondition.cc
> +++ b/src/RuleCondition.cc
> @@ -111,7 +111,7 @@ bool RuleConditionPayloadSize::DoMatch(Rule* rule, 
> RuleEndpointState* state,
>   return payload_size >= val;
>   default:
> - reporter->InternalError("unknown comparision type");
> + reporter->InternalError("unknown comparison type");
>   }
>   // Should not be reached
> diff --git a/src/RuleMatcher.cc b/src/RuleMatcher.cc
> index f40a5c4..f5b5b82 100644
> --- a/src/RuleMatcher.cc
> +++ b/src/RuleMatcher.cc
> @@ -21,7 +21,7 @@
> //it may fail to match. Work-around: Insert an always
> //matching "payload" pattern (not done in snort2bro yet)
> //  - tcp-state always evaluates to true
> -//   (implemented but deactivated for comparision to Snort)
> +//   (implemented but deactivated for comparison to Snort)
> uint32 RuleHdrTest::idcounter = 0;
> diff --git a/src/Serializer.cc b/src/Serializer.cc
> index 49e57c0..5c1ae60 100644
> --- a/src/Serializer.cc
> +++ b/src/Serializer.cc
> @@ -437,7 +437,7 @@ bool Serializer::UnserializeCall(UnserialInfo* info)
> bool Serializer::UnserializeStateAccess(UnserialInfo* info)
>   {
> - SetErrorDescr("unserializing state acess");
> + SetErrorDescr("unserializing state access");
>   StateAccess* s = StateAccess::Unserialize(info);
> diff --git a/src/StateAccess.cc b/src/StateAccess.cc
> index aa4a1f3..6e73c8c 100644
> --- a/src/StateAccess.cc
> +++ b/src/StateAccess.cc
> @@ -150,7 +150,7 @@ bool StateAccess::CheckOld(const char* op, ID* id, Val* 
> index,
>   if ( should && is )
>   {
> - // There's no general comparision for non-atomic vals currently.
> + // There's no general comparison for non-atomic vals currently.
>   if ( ! (is_atomic_val(is) && is_atomic_val(should)) )
>   return true;
> diff --git a/src/broxygen/Configuration.cc b/src/broxygen/Configuration.cc
> index 264e8e6..4780e6a 100644
> --- a/src/broxygen/Configuration.cc
> +++ b/src/broxygen/Configuration.cc
> @@ -65,7 +65,7 @@ Config::Config(const string& arg_file, const string& delim)
>   Target* target = target_factory.Create(tokens[0], tokens[2], 
> tokens[1]);
>   if ( ! target )
> - reporter->FatalError("unkown Broxygen target type: %s",
> + reporter->FatalError("unknown Broxygen target type: %s",
>tokens[0].c_str());
>   targets.push_back(target);
> diff --git a/src/nb_dns.c b/src/nb_dns.c
> index 1e5d427..35059ab 100644
> --- a/src/nb_dns.c
> +++ b/src/nb_dns.c
> @@ -389,7 +389,7 @@ nb_dns_addr_request2(register s

[Bro-Dev] [JIRA] (BIT-1577) Fix minor spelling errors

[Johanna Amann (JIRA)]

 [ 
https://bro-tracker.atlassian.net/browse/BIT-1577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Johanna Amann reassigned BIT-1577:
--

Assignee: Johanna Amann

> Fix minor spelling errors
> -
>
> Key: BIT-1577
> URL: https://bro-tracker.atlassian.net/browse/BIT-1577
> Project: Bro Issue Tracker
>  Issue Type: Task
>  Components: Bro
>Reporter: Jeannette Dopheide
>    Assignee: Johanna Amann
>
> Fixing minor spelling errors in Bro 2.4.1 found here: 
> https://lintian.debian.org/full/ben...@debian.org.html#bro_2.4.1_x2bdfsg-2
> Repository : ssh://g...@bro-ids.icir.org/bro
> On branch  : topic/jdopheid/typos
> Link   : 
> https://github.com/bro/bro/commit/635d218583014938c2ee732cb6a1dfdee0f2
> ---
> commit 635d218583014938c2ee732cb6a1dfdee0f2
> Author: Jeannette Dopheide 
> Date:   Mon Apr 25 11:49:04 2016 -0500
> Correcting spelling errors found under bro 2.4.1+dfsg-2 here:
> 
> https://lintian.debian.org/full/ben...@debian.org.html#bro_2.4.1_x2bdfsg-2
> ---
> 635d218583014938c2ee732cb6a1dfdee0f2
> src/RuleCondition.cc  | 2 +-
> src/RuleMatcher.cc| 2 +-
> src/Serializer.cc | 2 +-
> src/StateAccess.cc| 2 +-
> src/broxygen/Configuration.cc | 2 +-
> src/nb_dns.c  | 2 +-
> 6 files changed, 6 insertions(+), 6 deletions(-)
> diff --git a/src/RuleCondition.cc b/src/RuleCondition.cc
> index 68eb131..40ef5f0 100644
> --- a/src/RuleCondition.cc
> +++ b/src/RuleCondition.cc
> @@ -111,7 +111,7 @@ bool RuleConditionPayloadSize::DoMatch(Rule* rule, 
> RuleEndpointState* state,
>   return payload_size >= val;
>   default:
> - reporter->InternalError("unknown comparision type");
> + reporter->InternalError("unknown comparison type");
>   }
>   // Should not be reached
> diff --git a/src/RuleMatcher.cc b/src/RuleMatcher.cc
> index f40a5c4..f5b5b82 100644
> --- a/src/RuleMatcher.cc
> +++ b/src/RuleMatcher.cc
> @@ -21,7 +21,7 @@
> //it may fail to match. Work-around: Insert an always
> //matching "payload" pattern (not done in snort2bro yet)
> //  - tcp-state always evaluates to true
> -//   (implemented but deactivated for comparision to Snort)
> +//   (implemented but deactivated for comparison to Snort)
> uint32 RuleHdrTest::idcounter = 0;
> diff --git a/src/Serializer.cc b/src/Serializer.cc
> index 49e57c0..5c1ae60 100644
> --- a/src/Serializer.cc
> +++ b/src/Serializer.cc
> @@ -437,7 +437,7 @@ bool Serializer::UnserializeCall(UnserialInfo* info)
> bool Serializer::UnserializeStateAccess(UnserialInfo* info)
>   {
> - SetErrorDescr("unserializing state acess");
> + SetErrorDescr("unserializing state access");
>   StateAccess* s = StateAccess::Unserialize(info);
> diff --git a/src/StateAccess.cc b/src/StateAccess.cc
> index aa4a1f3..6e73c8c 100644
> --- a/src/StateAccess.cc
> +++ b/src/StateAccess.cc
> @@ -150,7 +150,7 @@ bool StateAccess::CheckOld(const char* op, ID* id, Val* 
> index,
>   if ( should && is )
>   {
> - // There's no general comparision for non-atomic vals currently.
> + // There's no general comparison for non-atomic vals currently.
>   if ( ! (is_atomic_val(is) && is_atomic_val(should)) )
>   return true;
> diff --git a/src/broxygen/Configuration.cc b/src/broxygen/Configuration.cc
> index 264e8e6..4780e6a 100644
> --- a/src/broxygen/Configuration.cc
> +++ b/src/broxygen/Configuration.cc
> @@ -65,7 +65,7 @@ Config::Config(const string& arg_file, const string& delim)
>   Target* target = target_factory.Create(tokens[0], tokens[2], 
> tokens[1]);
>   if ( ! target )
> - reporter->FatalError("unkown Broxygen target type: %s",
> + reporter->FatalError("unknown Broxygen target type: %s",
>tokens[0].c_str());
>   targets.push_back(target);
> diff --git a/src/nb_dns.c b/src/nb_dns.c
> index 1e5d427..35059ab 100644
> --- a/src/nb_dns.c
> +++ b/src/nb_dns.c
> @@ -389,7 +389,7 @@ nb_dns_addr_request2(register struct nb_dns_info *nd, 
> char *addrp,
>   default:
>   snprintf(errstr, NB_DN

[Bro-Dev] [JIRA] (BIT-1573) 3 useless EventHandlerPtr in the ARP Analyzer

[Johanna Amann (JIRA)]

 [ 
https://bro-tracker.atlassian.net/browse/BIT-1573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Johanna Amann updated BIT-1573:
---
Resolution: Fixed
Status: Closed  (was: Open)

> 3 useless EventHandlerPtr in the ARP Analyzer
> -
>
> Key: BIT-1573
> URL: https://bro-tracker.atlassian.net/browse/BIT-1573
> Project: Bro Issue Tracker
>  Issue Type: Improvement
>  Components: Bro
>Reporter: llh
>Priority: Trivial
>
> The class analyzer::arp::ARP_Analyzer declared in the file 
> src/analyzer/protocol/arp/ARP.h declares 3 protected EventHandlerPtr that are 
> never initialized and never used.
> What the corresponding source file refer to are the following global 
> variables :
> * bad_arp
> * arp_request
> * arp_reply
> which are declared as "extern" in the file 
> build/src/analyzer/protocol/arp/events.bif.h which is generated by bifcl from 
> src/analyzer/protocol/arp/events.bif.
> Fixing this issue is trivial : deleting the 3 lines declaring the unused 
> EventHandlerPtr.
> The expected improvement is saving 3 bytes of memory and mostly not messing 
> with those who will try to understand the code of this analyzer in the 
> future. 



--
This message was sent by Atlassian JIRA
(v1000.5.0#72002)
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] [JIRA] (BIT-1573) 3 useless EventHandlerPtr in the ARP Analyzer

[Johanna Amann (JIRA)]

[ 
https://bro-tracker.atlassian.net/browse/BIT-1573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=26001#comment-26001
 ] 

Johanna Amann commented on BIT-1573:


You are right; this is fixed in 3a70289e91b09640cda77a0534aa997a15fff40f

> 3 useless EventHandlerPtr in the ARP Analyzer
> -
>
> Key: BIT-1573
> URL: https://bro-tracker.atlassian.net/browse/BIT-1573
> Project: Bro Issue Tracker
>  Issue Type: Improvement
>  Components: Bro
>Reporter: llh
>Priority: Trivial
>
> The class analyzer::arp::ARP_Analyzer declared in the file 
> src/analyzer/protocol/arp/ARP.h declares 3 protected EventHandlerPtr that are 
> never initialized and never used.
> What the corresponding source file refer to are the following global 
> variables :
> * bad_arp
> * arp_request
> * arp_reply
> which are declared as "extern" in the file 
> build/src/analyzer/protocol/arp/events.bif.h which is generated by bifcl from 
> src/analyzer/protocol/arp/events.bif.
> Fixing this issue is trivial : deleting the 3 lines declaring the unused 
> EventHandlerPtr.
> The expected improvement is saving 3 bytes of memory and mostly not messing 
> with those who will try to understand the code of this analyzer in the 
> future. 



--
This message was sent by Atlassian JIRA
(v1000.5.0#72002)
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] [JIRA] (BIT-1574) Please merge topic/johanna/imap-starttls

[Johanna Amann (JIRA)]

Johanna Amann created BIT-1574:
--

 Summary: Please merge topic/johanna/imap-starttls
 Key: BIT-1574
 URL: https://bro-tracker.atlassian.net/browse/BIT-1574
 Project: Bro Issue Tracker
  Issue Type: New Feature
  Components: Bro
Affects Versions: git/master
Reporter: Johanna Amann
 Fix For: 2.5


Please merge topic/johanna/imap-starttls

This adds a very rudimentary IMAP analyzer (binpac based), which parses just 
enough of the protocol to recognize when a server switches to SSL using 
StartTLS, switching a connection to the SSL analyzer from this point.



--
This message was sent by Atlassian JIRA
(v1000.5.0#72002)
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] [JIRA] (BIT-1574) Please merge topic/johanna/imap-starttls

[Johanna Amann (JIRA)]

 [ 
https://bro-tracker.atlassian.net/browse/BIT-1574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Johanna Amann updated BIT-1574:
---
Status: Merge Request  (was: Open)

> Please merge topic/johanna/imap-starttls
> 
>
> Key: BIT-1574
> URL: https://bro-tracker.atlassian.net/browse/BIT-1574
> Project: Bro Issue Tracker
>  Issue Type: New Feature
>  Components: Bro
>Affects Versions: git/master
>    Reporter: Johanna Amann
> Fix For: 2.5
>
>
> Please merge topic/johanna/imap-starttls
> This adds a very rudimentary IMAP analyzer (binpac based), which parses just 
> enough of the protocol to recognize when a server switches to SSL using 
> StartTLS, switching a connection to the SSL analyzer from this point.



--
This message was sent by Atlassian JIRA
(v1000.5.0#72002)
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] [JIRA] (BIT-1573) 3 useless EventHandlerPtr in the ARP Analyzer

[Johanna Amann (JIRA)]

[ 
https://bro-tracker.atlassian.net/browse/BIT-1573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=25902#comment-25902
 ] 

Johanna Amann commented on BIT-1573:


I might miss something - but it seems that those EventHanderPtrs are 
initialized in ARP_Analyzer::ARP_Analyzer in ARP.cc (lines 13-15).

They are also used to send events later in ARP.cc

> 3 useless EventHandlerPtr in the ARP Analyzer
> -
>
> Key: BIT-1573
> URL: https://bro-tracker.atlassian.net/browse/BIT-1573
> Project: Bro Issue Tracker
>  Issue Type: Improvement
>  Components: Bro
>Reporter: llh
>Priority: Trivial
>
> The class analyzer::arp::ARP_Analyzer declared in the file 
> src/analyzer/protocol/arp/ARP.h declares 3 protected EventHandlerPtr that are 
> never initialized and never used.
> What the corresponding source file refer to are the following global 
> variables :
> * bad_arp
> * arp_request
> * arp_reply
> which are declared as "extern" in the file 
> build/src/analyzer/protocol/arp/events.bif.h which is generated by bifcl from 
> src/analyzer/protocol/arp/events.bif.
> Fixing this issue is trivial : deleting the 3 lines declaring the unused 
> EventHandlerPtr.
> The expected improvement is saving 3 bytes of memory and mostly not messing 
> with those who will try to understand the code of this analyzer in the 
> future. 



--
This message was sent by Atlassian JIRA
(v1000.5.0#72002)
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] [JIRA] (BIT-1572) Please merge topic/johanna/intel-uid-fuid

[Johanna Amann (JIRA)]

 [ 
https://bro-tracker.atlassian.net/browse/BIT-1572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Johanna Amann updated BIT-1572:
---
Status: Merge Request  (was: Open)

> Please merge topic/johanna/intel-uid-fuid
> -
>
> Key: BIT-1572
> URL: https://bro-tracker.atlassian.net/browse/BIT-1572
> Project: Bro Issue Tracker
>  Issue Type: Improvement
>  Components: Bro
>Affects Versions: git/master
>    Reporter: Johanna Amann
> Fix For: 2.5
>
>
> Please merge topic/johanna/intel-uid-fuid. 
> This patch allows users to provide the fuid or the connection id directly, in 
> case they do not have access to either in the event that they handle.
> An example for this is the handling of certificates in SSL, where the fa_file 
> record cannot be retained because this would create a cyclic data structure.
> This patch also provides file IDs for hostname matches in certificates, which 
> was not possible with the previous API.



--
This message was sent by Atlassian JIRA
(v7.2.0-OD-05-030#72002)
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] [JIRA] (BIT-1572) Please merge topic/johanna/intel-uid-fuid

[Johanna Amann (JIRA)]

Johanna Amann created BIT-1572:
--

 Summary: Please merge topic/johanna/intel-uid-fuid
 Key: BIT-1572
 URL: https://bro-tracker.atlassian.net/browse/BIT-1572
 Project: Bro Issue Tracker
  Issue Type: Improvement
  Components: Bro
Affects Versions: git/master
Reporter: Johanna Amann
 Fix For: 2.5


Please merge topic/johanna/intel-uid-fuid. 

This patch allows users to provide the fuid or the connection id directly, in 
case they do not have access to either in the event that they handle.

An example for this is the handling of certificates in SSL, where the fa_file 
record cannot be retained because this would create a cyclic data structure.

This patch also provides file IDs for hostname matches in certificates, which 
was not possible with the previous API.




--
This message was sent by Atlassian JIRA
(v7.2.0-OD-05-030#72002)
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] Deleting old branches

[Johanna Amann]

Just one more warning - if no one complains, I will go ahead and delete
all of this on Friday.

Johanna

On Wed, Apr 20, 2016 at 10:28:01AM -0700, Johanna Amann wrote:
> Hi,
> 
> we currently have a ton of branches in Bro which have been merged into
> master (some of them a long time ago). And - I would like to delete them,
> unless people think they are worth keeping around for some reason.
> 
> To be more specific, the branches I would like to delete are:
> 
> robin/topic/writer-info
> topic/dnthayer/configure
> topic/dnthayer/doc-fixes
> topic/dnthayer/doc-fixes-for-2.3
> topic/dnthayer/doc-improvements-2.4
> topic/dnthayer/doc-updates
> topic/dnthayer/fix-rdp
> topic/dnthayer/langref
> topic/dnthayer/mktemp
> topic/dnthayer/ticket1160
> topic/dnthayer/ticket1186
> topic/dnthayer/ticket1206
> topic/dnthayer/ticket1215
> topic/dnthayer/ticket1467
> topic/dnthayer/ticket1481
> topic/dnthayer/ticket1503
> topic/dnthayer/ticket856
> topic/gilbert/plugin-api-tweak
> topic/hui/dnp3-udp
> topic/hui/modbus-events
> topic/jazoff/notice_file_info
> topic/jazoff/ssl-validation-fix
> topic/jazoff/suppression
> topic/jdopheid/BIT-1242
> topic/jdopheid/bro/edits_to_installation_and_getting_started
> topic/jdopheid/bro_documentation
> topic/johanna/filter_subnet_table
> topic/johanna/function-recursion
> topic/johanna/openflow
> topic/johanna/stats_smb_leak
> topic/johanna/str-functions
> topic/jsiwek/asan-fixes
> topic/jsiwek/ascii-log-memleak-fix
> topic/jsiwek/bif-loader-scripts
> topic/jsiwek/bit-1077
> topic/jsiwek/bit-1153
> topic/jsiwek/bit-1156
> topic/jsiwek/bit-1166
> topic/jsiwek/bit-1176
> topic/jsiwek/bit-1235
> topic/jsiwek/bit-1240
> topic/jsiwek/bit-1246
> topic/jsiwek/bit-1247
> topic/jsiwek/bit-1248
> topic/jsiwek/bit-1280
> topic/jsiwek/bit-1288
> topic/jsiwek/bit-1295
> topic/jsiwek/bit-1296
> topic/jsiwek/bit-1298
> topic/jsiwek/bit-1305
> topic/jsiwek/bit-1324
> topic/jsiwek/bit-1343
> topic/jsiwek/bit-1350
> topic/jsiwek/bit-1367
> topic/jsiwek/bit-1368
> topic/jsiwek/bit-1373
> topic/jsiwek/bit-1376
> topic/jsiwek/bit-1384
> topic/jsiwek/bit-1408
> topic/jsiwek/bit-342
> topic/jsiwek/bit-348
> topic/jsiwek/bit-788
> topic/jsiwek/bit-844
> topic/jsiwek/broccoli-vectors
> topic/jsiwek/broker
> topic/jsiwek/broxygen
> topic/jsiwek/coverity
> topic/jsiwek/deprecation
> topic/jsiwek/dnp3-udp
> topic/jsiwek/dns-perf
> topic/jsiwek/dns_fake
> topic/jsiwek/faf-perf
> topic/jsiwek/faster-val-clone
> topic/jsiwek/file-reassembly-merge
> topic/jsiwek/file-signatures
> topic/jsiwek/flip-roles
> topic/jsiwek/gre
> topic/jsiwek/http-file-id-caching
> topic/jsiwek/improve-type-checks
> topic/jsiwek/improve_comm_loop
> topic/jsiwek/jemalloc
> topic/jsiwek/jj-bugs
> topic/jsiwek/libmagic-integration
> topic/jsiwek/mime-multipart-boundary-leniency
> topic/jsiwek/misc-fixes
> topic/jsiwek/missing-pac-deps
> topic/jsiwek/missing-plugin
> topic/jsiwek/new-libmagic
> topic/jsiwek/odesc-escaping
> topic/jsiwek/outer_param_binding
> topic/jsiwek/parse-only
> topic/jsiwek/pktsrc-idle
> topic/jsiwek/remove-val-attribs
> topic/jsiwek/review-rafael-bro-manual-changes
> topic/jsiwek/snmp
> topic/jsiwek/socks-authentication
> topic/jsiwek/string-slicing-fix
> topic/jsiwek/tcp-improvements
> topic/jsiwek/while
> topic/matthias/bloomfilter-fix
> topic/rafaelb/new-Bro-Manual-Development-Edition-Update1
> topic/robin/ascii-escape-normalization
> topic/robin/bit-348-merge
> topic/robin/bpf-vector
> topic/robin/dnp3-merge-v4
> topic/robin/dynamic-plugins-2.3
> topic/robin/event-dumper
> topic/robin/http-connect
> topic/robin/modbus-events-merge
> topic/robin/pacf
> topic/robin/pktsrc
> topic/robin/plugin-updates
> topic/robin/reader-writer-plugins
> topic/robin/rework-packets-merge
> topic/robin/smtp-fix
> topic/seth/ascii-escape-normalization
> topic/seth/compiler-cleanup
> topic/seth/deflate-missing-headers-fix
> topic/seth/dnp3-wrong-sizeof-argument
> topic/seth/dns-srv-fix
> topic/seth/file-analysis-exe-analyzer
> topic/seth/file-entropy
> topic/seth/files-reassembly-and-mime-updates
> topic/seth/files-tracking
> topic/seth/http-connect
> topic/seth/ie11-software-parsing
> topic/seth/json-formatter
> topic/seth/mime-updates
> topic/seth/modbus_dpd_fix
> topic/seth/more-file-type-ident-fixes
> topic/seth/radiotap
> topic/seth/rdp
> topic/seth/sip-fixes
> topic/seth/snmp
> topic/seth/socks-authentication
> topic/struck/BIT-1277
> topic/struck/BIT-1287
> topic/struck/openflow
> topic/vladg/bit-1410
> topic/vladg/bit-1458
> topic/vladg/bit-1460
> topic/vladg/bit-1466

[Bro-Dev] [JIRA] (BIT-1570) Added get_current_packet_header() bif

[Johanna Amann (JIRA)]

 [ 
https://bro-tracker.atlassian.net/browse/BIT-1570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Johanna Amann updated BIT-1570:
---
Resolution: Merged  (was: Fixed)
Status: Closed  (was: Merge Request)

> Added get_current_packet_header() bif
> -
>
> Key: BIT-1570
> URL: https://bro-tracker.atlassian.net/browse/BIT-1570
> Project: Bro Issue Tracker
>  Issue Type: Improvement
>  Components: Bro
>Reporter: Jan Grashoefer
>    Assignee: Johanna Amann
>
> [Pull request #65|https://github.com/bro/bro/pull/65] adds 
> get_current_packet_header() BIF, which returns a raw_pkt_hdr record 
> containing layer 2, 3 and 4 headers. This comes in handy e.g. for analyzing 
> ICMP.



--
This message was sent by Atlassian JIRA
(v7.2.0-OD-05-030#72002)
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] [JIRA] (BIT-1563) BrokerComm and BrokerStore namespaces should be combined

[Johanna Amann (JIRA)]

 [ 
https://bro-tracker.atlassian.net/browse/BIT-1563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Johanna Amann updated BIT-1563:
---
Resolution: Merged  (was: Fixed)
Status: Closed  (was: Merge Request)

> BrokerComm and BrokerStore namespaces should be combined
> 
>
> Key: BIT-1563
> URL: https://bro-tracker.atlassian.net/browse/BIT-1563
> Project: Bro Issue Tracker
>  Issue Type: Task
>  Components: Bro
>Reporter: Daniel Thayer
>    Assignee: Johanna Amann
> Fix For: 2.5
>
>
> The BrokerComm and BrokerStore namespaces should be combined to
> just "Broker".



--
This message was sent by Atlassian JIRA
(v7.2.0-OD-05-030#72002)
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] [JIRA] (BIT-1506) Bro fails to build on OS X 10.11 (El Capitan) due to OpenSSL header removal

[Johanna Amann (JIRA)]

 [ 
https://bro-tracker.atlassian.net/browse/BIT-1506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Johanna Amann updated BIT-1506:
---
Resolution: Merged  (was: Fixed)
Status: Closed  (was: Merge Request)

> Bro fails to build on OS X 10.11 (El Capitan) due to OpenSSL header removal
> ---
>
> Key: BIT-1506
> URL: https://bro-tracker.atlassian.net/browse/BIT-1506
> Project: Bro Issue Tracker
>  Issue Type: Problem
>  Components: Bro
>Affects Versions: 2.4
>Reporter: Vlad Grigorescu
>    Assignee: Johanna Amann
> Fix For: 2.5
>
>
> It looks like Apple removed the OpenSSL headers with El Capitan[1] (OS X
> 10.11), and now Bro fails to build on OS X. Apple's recommendation is
> that we either include a copy of OpenSSL ourselves or we use their
> Secure Transport API.
> [1] - <https://lists.apple.com/archives/macnetworkprog/2015/Jun/msg00025.html>



--
This message was sent by Atlassian JIRA
(v7.2.0-OD-05-030#72002)
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] [JIRA] (BIT-1506) Bro fails to build on OS X 10.11 (El Capitan) due to OpenSSL header removal

[Johanna Amann (JIRA)]

[ 
https://bro-tracker.atlassian.net/browse/BIT-1506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=25804#comment-25804
 ] 

Johanna Amann commented on BIT-1506:


Yup, that actually was my plan :). Thanks.

> Bro fails to build on OS X 10.11 (El Capitan) due to OpenSSL header removal
> ---
>
> Key: BIT-1506
> URL: https://bro-tracker.atlassian.net/browse/BIT-1506
> Project: Bro Issue Tracker
>  Issue Type: Problem
>  Components: Bro
>Affects Versions: 2.4
>Reporter: Vlad Grigorescu
>Assignee: Johanna Amann
> Fix For: 2.5
>
>
> It looks like Apple removed the OpenSSL headers with El Capitan[1] (OS X
> 10.11), and now Bro fails to build on OS X. Apple's recommendation is
> that we either include a copy of OpenSSL ourselves or we use their
> Secure Transport API.
> [1] - <https://lists.apple.com/archives/macnetworkprog/2015/Jun/msg00025.html>



--
This message was sent by Atlassian JIRA
(v7.2.0-OD-05-030#72002)
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev