[Bro-Dev] [Auto] Merge Status

[Merge Tracker]

Open Merge Requests
===

IDComponentReporterAssigneeUpdated   For 
Version  PrioritySummary
  ---  --  --  --  
-  --  

BIT-1125 [1]  Bro  Jon Siwek   -   2014-01-30   
 2.3  Normal  topic/jsiwek/http-file-id-caching [2]
BIT-1124 [3]  BroControl   Robin Sommer-   2014-01-30   
 2.3  Normal  process command misplaces custom scripts
BIT-1123 [4]  Bro  Jeannette Dopheide  -   2014-01-29   
 2.3  Normal  topic/jdopheid/bro/edits_to_installation_and_getting_started 
[5]
BIT-1122 [6]  Bro  Jon Siwek   Seth Hall   2014-01-30   
 2.3  Normal  topic/jsiwek/dns-improvements [7]


Open Fastpath Commits
==

Commit   ComponentAuthor  DateSummary
---  ---  --  --  
-
62b3cb0 [8]  bro  Bernhard Amann  2014-01-28  Also use exec-module test 
to check for leaks.


[1]  BIT-1125   
https://bro-tracker.atlassian.net/browse/BIT-1125
[2]  http-file-id-caching   
https://github.com/bro/bro/tree/topic/jsiwek/http-file-id-caching
[3]  BIT-1124   
https://bro-tracker.atlassian.net/browse/BIT-1124
[4]  BIT-1123   
https://bro-tracker.atlassian.net/browse/BIT-1123
[5]  edits_to_installation_and_getting_started  
https://github.com/bro/bro/tree/topic/jdopheid/bro/edits_to_installation_and_getting_started
[6]  BIT-1122   
https://bro-tracker.atlassian.net/browse/BIT-1122
[7]  dns-improvements   
https://github.com/bro/bro/tree/topic/jsiwek/dns-improvements
[8]  62b3cb0
https://github.com/bro/bro/commit/62b3cb0a5b7bdd8fed1d7d0dae3337115b2feae7

___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] [JIRA] (BIT-1125) topic/jsiwek/http-file-id-caching

[Seth Hall (JIRA)]

 [ 
https://bro-tracker.atlassian.net/browse/BIT-1125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Seth Hall updated BIT-1125:
---

Attachment: signature.asc



I've been thinking about this and I'm not sure how I feel about analyzers 
computing their own identifiers.  That actually causes inconsistent behavior 
because a user would have to know that a certain analyzer does that or that it 
does that in certain cases.  i.e. the user would have no control over how file 
chunks are tied together to form complete files.  Is this something that is 
already implemented?


> topic/jsiwek/http-file-id-caching
> -
>
> Key: BIT-1125
> URL: https://bro-tracker.atlassian.net/browse/BIT-1125
> Project: Bro Issue Tracker
>  Issue Type: Improvement
>  Components: Bro
>Affects Versions: git/master
>Reporter: Jon Siwek
> Fix For: 2.3
>
> Attachments: signature.asc
>
>
> This branch is in bro and bro-testing repos.  It adds a file ID caching / 
> "fast path" mechanism to the file analysis API and adapts HTTP to use it for 
> performance improvement.



--
This message was sent by Atlassian JIRA
(v6.2-OD-07-028#6211)
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] Dot release?

[Seth Hall]

On Jan 30, 2014, at 1:17 PM, Bernhard Amann  wrote:

> I already told Robin - but just for the record, I think it is a good 
> idea/plan.


I'm in the same boat as Bernhard here.  Looking forward to the 2.2.1 release. ;)

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] [JIRA] (BIT-1124) process command misplaces custom scripts

[Daniel Thayer (JIRA)]

 [ 
https://bro-tracker.atlassian.net/browse/BIT-1124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Thayer updated BIT-1124:
---

Status: Merge Request  (was: Open)

> process command misplaces custom scripts
> 
>
> Key: BIT-1124
> URL: https://bro-tracker.atlassian.net/browse/BIT-1124
> Project: Bro Issue Tracker
>  Issue Type: Problem
>  Components: BroControl
>Affects Versions: 2.2
>Reporter: Robin Sommer
> Fix For: 2.3
>
>
> {noformat}
> # cat test.bro
> @load base/utils/site
> print Site::local_nets;
> {noformat}
> {{broctl  process trace.pcap test.bro}} gives:
> {noformat}
> error in /usr/local/bro-2.2/share/bro/policy/misc/loaded-scripts.bro, line 4: 
> syntax error, at or near “module"
> {noformat}
> I believe it's due to test.bro being placed in the middle of the command line 
> that {{process}} builds. If I move it to the end, it works fine.



--
This message was sent by Atlassian JIRA
(v6.2-OD-07-028#6211)

___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] [JIRA] (BIT-1124) process command misplaces custom scripts

[Daniel Thayer (JIRA)]

[ 
https://bro-tracker.atlassian.net/browse/BIT-1124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15311#comment-15311
 ] 

Daniel Thayer commented on BIT-1124:


In branch topic/dnthayer/ticket1124, I've changed the order of 
scripts so that user-specified scripts are always at the end of
the Bro command, and I've improved the broctl help message
to show how the process command should be used.


> process command misplaces custom scripts
> 
>
> Key: BIT-1124
> URL: https://bro-tracker.atlassian.net/browse/BIT-1124
> Project: Bro Issue Tracker
>  Issue Type: Problem
>  Components: BroControl
>Affects Versions: 2.2
>Reporter: Robin Sommer
> Fix For: 2.3
>
>
> {noformat}
> # cat test.bro
> @load base/utils/site
> print Site::local_nets;
> {noformat}
> {{broctl  process trace.pcap test.bro}} gives:
> {noformat}
> error in /usr/local/bro-2.2/share/bro/policy/misc/loaded-scripts.bro, line 4: 
> syntax error, at or near “module"
> {noformat}
> I believe it's due to test.bro being placed in the middle of the command line 
> that {{process}} builds. If I move it to the end, it works fine.



--
This message was sent by Atlassian JIRA
(v6.2-OD-07-028#6211)

___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] [JIRA] (BIT-1124) process command misplaces custom scripts

[Daniel Thayer (JIRA)]

 [ 
https://bro-tracker.atlassian.net/browse/BIT-1124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Thayer updated BIT-1124:
---

Fix Version/s: 2.3

> process command misplaces custom scripts
> 
>
> Key: BIT-1124
> URL: https://bro-tracker.atlassian.net/browse/BIT-1124
> Project: Bro Issue Tracker
>  Issue Type: Problem
>  Components: BroControl
>Affects Versions: 2.2
>Reporter: Robin Sommer
> Fix For: 2.3
>
>
> {noformat}
> # cat test.bro
> @load base/utils/site
> print Site::local_nets;
> {noformat}
> {{broctl  process trace.pcap test.bro}} gives:
> {noformat}
> error in /usr/local/bro-2.2/share/bro/policy/misc/loaded-scripts.bro, line 4: 
> syntax error, at or near “module"
> {noformat}
> I believe it's due to test.bro being placed in the middle of the command line 
> that {{process}} builds. If I move it to the end, it works fine.



--
This message was sent by Atlassian JIRA
(v6.2-OD-07-028#6211)

___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] [JIRA] (BIT-1125) topic/jsiwek/http-file-id-caching

[Robin Sommer (JIRA)]

[ 
https://bro-tracker.atlassian.net/browse/BIT-1125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15310#comment-15310
 ] 

Robin Sommer commented on BIT-1125:
---

For the case that the core can compute the file id itself without needing the 
script-land, is the idea that it then just passes it in as the {{cached_id}}?



> topic/jsiwek/http-file-id-caching
> -
>
> Key: BIT-1125
> URL: https://bro-tracker.atlassian.net/browse/BIT-1125
> Project: Bro Issue Tracker
>  Issue Type: Improvement
>  Components: Bro
>Affects Versions: git/master
>Reporter: Jon Siwek
> Fix For: 2.3
>
>
> This branch is in bro and bro-testing repos.  It adds a file ID caching / 
> "fast path" mechanism to the file analysis API and adapts HTTP to use it for 
> performance improvement.



--
This message was sent by Atlassian JIRA
(v6.2-OD-07-028#6211)
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] [JIRA] (BIT-1119) topic/jsiwek/tcp-improvements

[Robin Sommer (JIRA)]

 [ 
https://bro-tracker.atlassian.net/browse/BIT-1119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robin Sommer updated BIT-1119:
--

Status: Open  (was: Merge Request)

> topic/jsiwek/tcp-improvements
> -
>
> Key: BIT-1119
> URL: https://bro-tracker.atlassian.net/browse/BIT-1119
> Project: Bro Issue Tracker
>  Issue Type: Improvement
>  Components: Bro
>Affects Versions: git/master
>Reporter: Jon Siwek
> Fix For: 2.3
>
> Attachments: signature.asc
>
>
> This branch is in the bro, bro-testing, and bro-testing-private repos and has 
> a few changes to improve reporting of TCP connection sizes and gaps (commit 
> messages explain in more detail).
> The baseline changes in the external repos all seemed reasonable/explainable 
> (or actually fix a problem).  There's too much changed to go through 
> case-by-case and actually check things, but I did do closer examinations of 
> unique differences as I came across them (e.g. try to corroborate Bro results 
> via wireshark).  Then for those that seem to follow the same trend as 
> something I already inspected, I wouldn't manually check.



--
This message was sent by Atlassian JIRA
(v6.2-OD-07-028#6211)
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] [JIRA] (BIT-1122) topic/jsiwek/dns-improvements

[Jon Siwek (JIRA)]

[ 
https://bro-tracker.atlassian.net/browse/BIT-1122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15309#comment-15309
 ] 

Jon Siwek commented on BIT-1122:


I just pushed another commit on this branch containing a rewrite of the 
query-reply state tracking and matching logic.  It now relies on "dns_end" 
event to pair messages and log them.  The old way of tracking the number of 
resource records seen versus the total number declared in the reply message is 
too unreliable in many cases.

> topic/jsiwek/dns-improvements
> -
>
> Key: BIT-1122
> URL: https://bro-tracker.atlassian.net/browse/BIT-1122
> Project: Bro Issue Tracker
>  Issue Type: Improvement
>  Components: Bro
>Affects Versions: git/master
>Reporter: Jon Siwek
>Assignee: Seth Hall
> Fix For: 2.3
>
>
> This branch is in bro, bro-testing, and bro-testing-private repos.
> - Fixes incorrect parsing of DNS message format for messages with empty 
> question sections.
> - Changes dns.log to only include standard queries (opcode == 1).
> - Adds "dns_unknown_reply" event for RR types that Bro doesn't know how to 
> parse, which improves accuracy of request-reply pair matching performed by 
> the default DNS scripts.



--
This message was sent by Atlassian JIRA
(v6.2-OD-07-028#6211)
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] Dot release?

[Robin Sommer]

On Thu, Jan 30, 2014 at 11:22 -0800, you wrote:

> fixed and I wonder whether 2.2.1 is the right target for that.

Yes, that would be good to get in there too if we can figure out
what's going on.

Robin

-- 
Robin Sommer * Phone +1 (510) 722-6541 * ro...@icir.org
ICSI/LBNL* Fax   +1 (510) 666-2956 * www.icir.org/robin
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] Dot release?

[Matthias Vallentin]

> I'd wait for two more things though:

Aashish also raised some potential bugs with Bro's hashing. It appears
that the Bloom filters fill up too quickly, i.e., do not meet their
false positive requirements. My hunch is that this has to do with the
construction of hash functions, perhaps they are not pairwise
independent unless parametrized in a certain way, or perhaps there's
just some other smaller bug in place. In any case, it needs to be
fixed and I wonder whether 2.2.1 is the right target for that.

Matthias
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] Dot release?

[Liam Randall]

Yes, the current master is WAY more stable on busy production sensors that
2.2.  For sites really leaning on the intel framework master is the only
way to go.

Thanks,

Liam Randall


On Thu, Jan 30, 2014 at 1:17 PM, Bernhard Amann
wrote:

> I already told Robin - but just for the record, I think it is a good
> idea/plan.
>
> Bernhard
>
> On Jan 30, 2014, at 8:57 AM, Slagell, Adam J  wrote:
>
> > I like that plan. I think there are some minor Maverick's issues too
> that Daniel found. So we might want to get those in there as well.
> >
> > On Jan 30, 2014, at 10:50 AM, Robin Sommer  wrote:
> >
> >> Folks,
> >>
> >> making a 2.2.1 release has been coming up a few times and I'm thinking
> >> we should just snapshot current master for that. We've been fixing
> >> quite a number of things since 2.2, yet there aren't any larger new
> >> features yet (GRE tunnel decapsulation being the only one I can think
> >> of right now).
> >>
> >> I'd wait for two more things though:
> >>
> >>   - Merging, and some testing, of Jon's recent file analysis
> >>   framework API changes that make the file handle management more
> >>   efficient.
> >>
> >>   - Figuring out the exec and/or sumstats problems (it looks certain
> >>   at this point that exec isn't cleaning up fully; and sumstats may
> >>   have a larger than expected CPU impact, but that's not clear yet I
> >>   believe).
> >>
> >> Once 2.2.1 is out, I'd then next work on merging my dynamic plugin
> >> code, which is mostly ready but needs cleanup, review, documentation,
> >> testing.
> >>
> >> How does that sound? If good, now would also be the time to finalize
> >> any other minor fixes that people might want to see in 2.2.1.
> >>
> >> Robin
> >>
> >> --
> >> Robin Sommer * Phone +1 (510) 722-6541 * ro...@icir.org
> >> ICSI/LBNL* Fax   +1 (510) 666-2956 * www.icir.org/robin
> >> ___
> >> bro-dev mailing list
> >> bro-dev@bro.org
> >> http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
> >
> > --
> >
> > Adam J. Slagell
> > Chief Information Security Officer
> > Assistant Director, Cybersecurity
> > National Center for Supercomputing Applications
> > University of Illinois at Urbana-Champaign
> > www.ncsa.illinois.edu/~slagell/
> >
> > "Under the Illinois Freedom of Information Act (FOIA), any written
> communication to or from University employees regarding University business
> is a public record and may be subject to public disclosure."
> >
> >
> > ___
> > bro-dev mailing list
> > bro-dev@bro.org
> > http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
>
>
> ___
> bro-dev mailing list
> bro-dev@bro.org
> http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
>



-- 
Liam Randall
Managing Partner
510-281-0760
www.Broala.com 
>From the creators of Bro 
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] [JIRA] (BIT-1124) process command misplaces custom scripts

[Daniel Thayer (JIRA)]

[ 
https://bro-tracker.atlassian.net/browse/BIT-1124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15308#comment-15308
 ] 

Daniel Thayer commented on BIT-1124:


If you run "broctl process" with a custom script, then you need
to specify "--" so that broctl knows where the Bro options end
and where the custom scripts begin.  So your example
should be:
broctl process trace.pcap -- test.bro

Doing so moves test.bro farther down towards the end
of the Bro command line.  

However, in this example we still get a different error,
because broctl adds its own script (process-trace.bro,
which contains a couple of redefs) at the very end of
the Bro command.  I will change the order so that the
process-trace.bro is before the custom scripts (which
solves the syntax error).


> process command misplaces custom scripts
> 
>
> Key: BIT-1124
> URL: https://bro-tracker.atlassian.net/browse/BIT-1124
> Project: Bro Issue Tracker
>  Issue Type: Problem
>  Components: BroControl
>Affects Versions: 2.2
>Reporter: Robin Sommer
>
> {noformat}
> # cat test.bro
> @load base/utils/site
> print Site::local_nets;
> {noformat}
> {{broctl  process trace.pcap test.bro}} gives:
> {noformat}
> error in /usr/local/bro-2.2/share/bro/policy/misc/loaded-scripts.bro, line 4: 
> syntax error, at or near “module"
> {noformat}
> I believe it's due to test.bro being placed in the middle of the command line 
> that {{process}} builds. If I move it to the end, it works fine.



--
This message was sent by Atlassian JIRA
(v6.2-OD-07-028#6211)

___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] Dot release?

[Bernhard Amann]

I already told Robin - but just for the record, I think it is a good idea/plan.

Bernhard

On Jan 30, 2014, at 8:57 AM, Slagell, Adam J  wrote:

> I like that plan. I think there are some minor Maverick's issues too that 
> Daniel found. So we might want to get those in there as well.
> 
> On Jan 30, 2014, at 10:50 AM, Robin Sommer  wrote:
> 
>> Folks,
>> 
>> making a 2.2.1 release has been coming up a few times and I'm thinking
>> we should just snapshot current master for that. We've been fixing
>> quite a number of things since 2.2, yet there aren't any larger new
>> features yet (GRE tunnel decapsulation being the only one I can think
>> of right now).
>> 
>> I'd wait for two more things though:
>> 
>>   - Merging, and some testing, of Jon's recent file analysis
>>   framework API changes that make the file handle management more
>>   efficient.
>> 
>>   - Figuring out the exec and/or sumstats problems (it looks certain
>>   at this point that exec isn't cleaning up fully; and sumstats may
>>   have a larger than expected CPU impact, but that's not clear yet I
>>   believe).
>> 
>> Once 2.2.1 is out, I'd then next work on merging my dynamic plugin
>> code, which is mostly ready but needs cleanup, review, documentation,
>> testing.
>> 
>> How does that sound? If good, now would also be the time to finalize
>> any other minor fixes that people might want to see in 2.2.1.
>> 
>> Robin
>> 
>> -- 
>> Robin Sommer * Phone +1 (510) 722-6541 * ro...@icir.org
>> ICSI/LBNL* Fax   +1 (510) 666-2956 * www.icir.org/robin
>> ___
>> bro-dev mailing list
>> bro-dev@bro.org
>> http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
> 
> --
> 
> Adam J. Slagell
> Chief Information Security Officer
> Assistant Director, Cybersecurity
> National Center for Supercomputing Applications
> University of Illinois at Urbana-Champaign
> www.ncsa.illinois.edu/~slagell/
> 
> "Under the Illinois Freedom of Information Act (FOIA), any written 
> communication to or from University employees regarding University business 
> is a public record and may be subject to public disclosure." 
> 
> 
> ___
> bro-dev mailing list
> bro-dev@bro.org
> http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev


___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] Dot release?

[Robin Sommer]

Folks,

making a 2.2.1 release has been coming up a few times and I'm thinking
we should just snapshot current master for that. We've been fixing
quite a number of things since 2.2, yet there aren't any larger new
features yet (GRE tunnel decapsulation being the only one I can think
of right now).

I'd wait for two more things though:

- Merging, and some testing, of Jon's recent file analysis
framework API changes that make the file handle management more
efficient.

- Figuring out the exec and/or sumstats problems (it looks certain
at this point that exec isn't cleaning up fully; and sumstats may
have a larger than expected CPU impact, but that's not clear yet I
believe).

Once 2.2.1 is out, I'd then next work on merging my dynamic plugin
code, which is mostly ready but needs cleanup, review, documentation,
testing.

How does that sound? If good, now would also be the time to finalize
any other minor fixes that people might want to see in 2.2.1.

Robin

-- 
Robin Sommer * Phone +1 (510) 722-6541 * ro...@icir.org
ICSI/LBNL* Fax   +1 (510) 666-2956 * www.icir.org/robin
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] Dot release?

[Slagell, Adam J]

I like that plan. I think there are some minor Maverick's issues too that 
Daniel found. So we might want to get those in there as well.

On Jan 30, 2014, at 10:50 AM, Robin Sommer  wrote:

> Folks,
> 
> making a 2.2.1 release has been coming up a few times and I'm thinking
> we should just snapshot current master for that. We've been fixing
> quite a number of things since 2.2, yet there aren't any larger new
> features yet (GRE tunnel decapsulation being the only one I can think
> of right now).
> 
> I'd wait for two more things though:
> 
>- Merging, and some testing, of Jon's recent file analysis
>framework API changes that make the file handle management more
>efficient.
> 
>- Figuring out the exec and/or sumstats problems (it looks certain
>at this point that exec isn't cleaning up fully; and sumstats may
>have a larger than expected CPU impact, but that's not clear yet I
>believe).
> 
> Once 2.2.1 is out, I'd then next work on merging my dynamic plugin
> code, which is mostly ready but needs cleanup, review, documentation,
> testing.
> 
> How does that sound? If good, now would also be the time to finalize
> any other minor fixes that people might want to see in 2.2.1.
> 
> Robin
> 
> -- 
> Robin Sommer * Phone +1 (510) 722-6541 * ro...@icir.org
> ICSI/LBNL* Fax   +1 (510) 666-2956 * www.icir.org/robin
> ___
> bro-dev mailing list
> bro-dev@bro.org
> http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

--

Adam J. Slagell
Chief Information Security Officer
Assistant Director, Cybersecurity
National Center for Supercomputing Applications
University of Illinois at Urbana-Champaign
www.ncsa.illinois.edu/~slagell/

"Under the Illinois Freedom of Information Act (FOIA), any written 
communication to or from University employees regarding University business is 
a public record and may be subject to public disclosure." 


___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev