[Bro-Dev] [JIRA] (BIT-985) 'tail -f' functionality for file reading in input framework

[scampbell (JIRA)]

 [ 
https://bro-tracker.atlassian.net/browse/BIT-985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

scampbell updated BIT-985:
--

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Yes - sorry about that!

scott


-BEGIN PGP SIGNATURE-
Version: GnuPG/MacGPG2 v2.0.22 (Darwin)
Comment: GPGTools - http://gpgtools.org

iEYEARECAAYFAlUI1MoACgkQK2Plq8B7ZBxKawCgpxUNSI21dDqcDg5o49g8JKUq
Q3AAoKFtR//MMCSyCke5670RdA1nGfEw
=HHK7
-END PGP SIGNATURE-


> 'tail -f' functionality for file reading in input framework
> ---
>
> Key: BIT-985
> URL: https://bro-tracker.atlassian.net/browse/BIT-985
> Project: Bro Issue Tracker
>  Issue Type: New Feature
>  Components: Bro
>Affects Versions: git/master
>Reporter: scampbell
>Assignee: Johanna Amann
>Priority: Low
> Fix For: 2.4
>
> Attachments: input.diff, PATCH
>
>
> With the current input framework, file data \-> event translation requires 
> that the entire data file be read at bro start time.  This can be prohibitive 
> when the file sizes become large ( > 1GB ).  
> It would be great to see a file open option that would start reading at the 
> end of the file.



--
This message was sent by Atlassian JIRA
(v6.4-OD-15-055#64014)
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] [JIRA] (BIT-985) 'tail -f' functionality for file reading in input framework

[Johanna Amann (JIRA)]

[ 
https://bro-tracker.atlassian.net/browse/BIT-985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=20018#comment-20018
 ] 

Johanna Amann commented on BIT-985:
---

Thank you for that explanation. I assume that raw_unescape_URI function made it 
into the patch by accident?

> 'tail -f' functionality for file reading in input framework
> ---
>
> Key: BIT-985
> URL: https://bro-tracker.atlassian.net/browse/BIT-985
> Project: Bro Issue Tracker
>  Issue Type: New Feature
>  Components: Bro
>Affects Versions: git/master
>Reporter: scampbell
>Assignee: Johanna Amann
>Priority: Low
> Fix For: 2.4
>
> Attachments: input.diff, PATCH
>
>
> With the current input framework, file data \-> event translation requires 
> that the entire data file be read at bro start time.  This can be prohibitive 
> when the file sizes become large ( > 1GB ).  
> It would be great to see a file open option that would start reading at the 
> end of the file.



--
This message was sent by Atlassian JIRA
(v6.4-OD-15-055#64014)
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] [JIRA] (BIT-985) 'tail -f' functionality for file reading in input framework

[scampbell (JIRA)]

 [ 
https://bro-tracker.atlassian.net/browse/BIT-985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

scampbell updated BIT-985:
--

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Absolutely - the key issues that I ran into with the first patch were
dealing with file rotation under the reader and leaks in the data
copying scheme.  After spending a few days on the mem leak issues
modifying the single use linear buffers (and mostly de-stabilizing
everything), I reimplemented the whole thing as a ring buffer.

My use case - reading a very rapidly moving log file - might be far
enough away from the original design pattern of small reasonably
static files that it is worth another type?  On the other hand I might
have just messed up the original work.

If this makes no sense please let me know and I will look over my
notes re the changes.

thanks for looking into this,
scott


-BEGIN PGP SIGNATURE-
Version: GnuPG/MacGPG2 v2.0.22 (Darwin)
Comment: GPGTools - http://gpgtools.org

iEYEARECAAYFAlUIyPMACgkQK2Plq8B7ZByVQwCghwbGlmgetHNMkxicrms6wl69
d2EAoIXsHbv1JWPeXJ5rpWv2rAlfWpPQ
=bKTE
-END PGP SIGNATURE-


> 'tail -f' functionality for file reading in input framework
> ---
>
> Key: BIT-985
> URL: https://bro-tracker.atlassian.net/browse/BIT-985
> Project: Bro Issue Tracker
>  Issue Type: New Feature
>  Components: Bro
>Affects Versions: git/master
>Reporter: scampbell
>Assignee: Johanna Amann
>Priority: Low
> Fix For: 2.4
>
> Attachments: input.diff, PATCH
>
>
> With the current input framework, file data \-> event translation requires 
> that the entire data file be read at bro start time.  This can be prohibitive 
> when the file sizes become large ( > 1GB ).  
> It would be great to see a file open option that would start reading at the 
> end of the file.



--
This message was sent by Atlassian JIRA
(v6.4-OD-15-055#64014)
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] [JIRA] (BIT-1331) BroControl manager crashes when logs rotate

[Aaron Eppert (JIRA)]

[ 
https://bro-tracker.atlassian.net/browse/BIT-1331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=20008#comment-20008
 ] 

Aaron Eppert edited comment on BIT-1331 at 3/17/15 6:38 PM:


Encountering the same problem in a clustered configuration with only a single 
worker and proxy at the moment. Confirming the same error and the linked 
BIT-1253 ticket.

More data:

{noformat}
Bro 2.3-451-debug
Linux 2.6.32-504.8.1.el6.x86_64


 reporter.log
{"ts":0.0,"level":"Reporter::ERROR","message":"no such index 
(Cluster::nodes[Intel::p$descr])","location":"/usr/local/bro/share/bro/base/frameworks/intel/./cluster.bro,
 line 37"}
{"ts":1426622062.691619,"level":"Reporter::ERROR","message":"extra base64 
groups after \u0027=\u0027 padding are ignored","location":""}
{"ts":1426622062.691619,"level":"Reporter::ERROR","message":"incomplete base64 
group, padding with 12 bits of 0","location":""}
{"ts":1426622072.075103,"level":"Reporter::ERROR","message":"extra base64 
groups after \u0027=\u0027 padding are ignored","location":""}
{"ts":1426622072.075103,"level":"Reporter::ERROR","message":"incomplete base64 
group, padding with 6 bits of 0","location":""}
{"ts":0.0,"level":"Reporter::ERROR","message":"no such index 
(Cluster::nodes[Intel::p$descr])","location":"/usr/local/bro/share/bro/base/frameworks/intel/./cluster.bro,
 line 37"}
{"ts":1426622135.535154,"level":"Reporter::ERROR","message":"extra base64 
groups after \u0027=\u0027 padding are ignored","location":""}
{"ts":1426622140.709589,"level":"Reporter::ERROR","message":"extra base64 
groups after \u0027=\u0027 padding are ignored","location":""}
{"ts":1426622140.709589,"level":"Reporter::ERROR","message":"incomplete base64 
group, padding with 6 bits of 0","location":""}
{"ts":0.0,"level":"Reporter::ERROR","message":"no such index 
(Cluster::nodes[Intel::p$descr])","location":"/usr/local/bro/share/bro/base/frameworks/intel/./cluster.bro,
 line 37"}

 stderr.log
warning in 
/usr/local/bro/spool/installed-scripts-do-not-touch/site/ps-detections/credit-card-exposure/./main.bro,
 line 83: deprecated (split_all)
warning in 
/usr/local/bro/spool/installed-scripts-do-not-touch/site/ps-detections/credit-card-exposure/./main.bro,
 line 93: deprecated (join_string_array)
fatal error in : Val::CONVERTER (string/port) (80/tcp)

 stdout.log
PacketFilter::LOG
X509::LOG
Software::LOG
SSH::LOG
DHCP::LOG
DNS::LOG
HTTP::LOG
SOCKS::LOG
DNP3::LOG
Known::HOSTS_LOG

 .cmdline
-U .status -p broctl -p broctl-live -p local -p manager local.bro broctl 
base/frameworks/cluster local-manager.bro broctl/auto -B threading

 .env_vars
PATH=/usr/local/bro/bin:/usr/local/bro/share/broctl/scripts:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/tokumx/bin:/root/bin
BROPATH=/usr/local/bro/spool/installed-scripts-do-not-touch/site::/usr/local/bro/spool/installed-scripts-do-not-touch/auto:/usr/local/bro/share/bro:/usr/local/bro/share/bro/policy:/usr/local/bro/share/bro/site
CLUSTER_NODE=manager

 .status
TERMINATED [atexit]

 No prof.log

{noformat}



was (Author: aeppert):
Encountering the same problem in a clustered configuration with only a single 
worker and proxy at the moment. Confirming the same error and the linked 
BIT-1253 ticket.

Also getting this:

fatal error in : val::CONVERTER (string/port) (80/tcp)

> BroControl manager crashes when logs rotate
> ---
>
> Key: BIT-1331
> URL: https://bro-tracker.atlassian.net/browse/BIT-1331
> Project: Bro Issue Tracker
>  Issue Type: Problem
>  Components: Bro
>Affects Versions: git/master, 2.4
> Environment: Ubuntu 12.04.5 LTS, PF_RING lb_method
>Reporter: Josh Liburdi
>Priority: High
> Fix For: 2.4
>
>
> The BroControl manager crashes when the logs rotate. Workers run fine through 
> this process. 
> stderr.log output:
> internal error: finish missing
> /usr/local/bro/share/broctl/scripts/run-bro: line 100: 157357 Aborted 
> (core dumped) nohup "$mybro" "$@"
> send-mail: SENDMAIL-NOTFOUND not found



--
This message was sent by Atlassian JIRA
(v6.4-OD-15-055#64014)
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] [JIRA] (BIT-1303) pysubnettree tests should be changed to use btest

[Daniel Thayer (JIRA)]

 [ 
https://bro-tracker.atlassian.net/browse/BIT-1303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Thayer updated BIT-1303:
---
Status: Merge Request  (was: Open)

> pysubnettree tests should be changed to use btest
> -
>
> Key: BIT-1303
> URL: https://bro-tracker.atlassian.net/browse/BIT-1303
> Project: Bro Issue Tracker
>  Issue Type: Problem
>  Components: pysubnettree
>Reporter: Daniel Thayer
> Fix For: 2.4
>
>
> The test cases in pysubnettree should be changed to use btest
> so that the tests are easier to run and can be better organized
> by splitting them into multiple test files.



--
This message was sent by Atlassian JIRA
(v6.4-OD-15-055#64014)
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] [JIRA] (BIT-1303) pysubnettree tests should be changed to use btest

[Daniel Thayer (JIRA)]

 [ 
https://bro-tracker.atlassian.net/browse/BIT-1303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Thayer updated BIT-1303:
---
Fix Version/s: (was: 2.5)
   2.4

The branch topic/dnthayer/ticket1303 in pysubnettree repo contains these 
changes.

> pysubnettree tests should be changed to use btest
> -
>
> Key: BIT-1303
> URL: https://bro-tracker.atlassian.net/browse/BIT-1303
> Project: Bro Issue Tracker
>  Issue Type: Problem
>  Components: pysubnettree
>Reporter: Daniel Thayer
> Fix For: 2.4
>
>
> The test cases in pysubnettree should be changed to use btest
> so that the tests are easier to run and can be better organized
> by splitting them into multiple test files.



--
This message was sent by Atlassian JIRA
(v6.4-OD-15-055#64014)
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] [JIRA] (BIT-1330) topic/python3-compat

[Robin Sommer (JIRA)]

 [ 
https://bro-tracker.atlassian.net/browse/BIT-1330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robin Sommer updated BIT-1330:
--
Resolution: Merged  (was: Fixed)
Status: Closed  (was: Merge Request)

> topic/python3-compat
> 
>
> Key: BIT-1330
> URL: https://bro-tracker.atlassian.net/browse/BIT-1330
> Project: Bro Issue Tracker
>  Issue Type: Improvement
>  Components: pysubnettree
>Reporter: Jon Siwek
>Assignee: Robin Sommer
> Fix For: 2.4
>
>
> Updates to pysubnettree for Python 3 compatibility: have to now consider that 
> bytes are a distinct type from strings and allow the API to accept either.



--
This message was sent by Atlassian JIRA
(v6.4-OD-15-055#64014)
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] [JIRA] (BIT-1199) Better error messages for input file errors in READER_ASCII

[Robin Sommer (JIRA)]

 [ 
https://bro-tracker.atlassian.net/browse/BIT-1199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robin Sommer updated BIT-1199:
--
Status: Open  (was: Merge Request)

> Better error messages for input file errors in READER_ASCII
> ---
>
> Key: BIT-1199
> URL: https://bro-tracker.atlassian.net/browse/BIT-1199
> Project: Bro Issue Tracker
>  Issue Type: Problem
>  Components: Bro
>Reporter: grigorescu
>Assignee: Johanna Amann
> Fix For: 2.4
>
> Attachments: test.intel
>
>
> This came up on the mailing list a few weeks ago. If one tries to load the 
> attached file as Intelligence, Bro will error out, with:
> {code}
> internal error: Value not found in enum mappimg. Module: GLOBAL, var: , var 
> size: 0
> {code}
> The attached file contains an extra tab after downloader.com.
> It'd be nice if Bro would tell you that this was an issue with the input 
> reader, which file it occurred in, and a line number.
> I think generally speaking, if there's an issue with an input file, it'd be 
> nice to know the line number.
> (Also, there's a typo in mappimg in the error message that's currently 
> displayed).



--
This message was sent by Atlassian JIRA
(v6.4-OD-15-055#64014)
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] [JIRA] (BIT-1342) Occasional test failures

[Robin Sommer (JIRA)]

Robin Sommer created BIT-1342:
-

 Summary: Occasional test failures
 Key: BIT-1342
 URL: https://bro-tracker.atlassian.net/browse/BIT-1342
 Project: Bro Issue Tracker
  Issue Type: Problem
  Components: BroControl
Reporter: Robin Sommer
 Fix For: 2.4


Two tests in current master fail for me occasionally (usually when I run the 
full broctl test-suite but not when I rerun just these failing tests). Diag 
output below.

{code}
command.start-stop-standalone ... failed
  % 'btest-diff stop.out' failed unexpectedly (exit code 1)
  % cat .diag
  == File ===
  stopping bro ...
  Exception in thread Thread-1 (most likely raised during interpreter shutdown):
  Traceback (most recent call last):
  File "/usr/lib64/python2.7/threading.py", line 811, in __bootstrap_inner
  File 
"/home/robin/bro/master/aux/broctl/testing/../build/testing/test.21123/lib/broctl/BroControl/ssh_runner.py",
 line
  File 
"/home/robin/bro/master/aux/broctl/testing/../build/testing/test.21123/lib/broctl/BroControl/ssh_runner.py",
 line
  File "/usr/lib64/python2.7/Queue.py", line 177, in get
  File "/usr/lib64/python2.7/threading.py", line 354, in wait
  : 'NoneType' object is not callable
  == Diff ===
  --- 
/home/robin/bro/master/aux/broctl/testing/Baseline/command.start-stop-standalone/stop.out
 2013-06-01 00:29:07.
  +++ stop.out  2015-03-17 22:50:01.857838625 +
  @@ -1 +1,9 @@
  stopping bro ...
  +Exception in thread Thread-1 (most likely raised during interpreter 
shutdown):
  +Traceback (most recent call last):
  +  File "/usr/lib64/python2.7/threading.py", line 811, in __bootstrap_inner
  +  File 
"/home/robin/bro/master/aux/broctl/testing/../build/testing/test.21123/lib/broctl/BroControl/ssh_runner.py",
 l
  +  File 
"/home/robin/bro/master/aux/broctl/testing/../build/testing/test.21123/lib/broctl/BroControl/ssh_runner.py",
 l
  +  File "/usr/lib64/python2.7/Queue.py", line 177, in get
  +  File "/usr/lib64/python2.7/threading.py", line 354, in wait
  +: 'NoneType' object is not callable
  ===
[...]
command.start-cluster-slowstart ... failed
  % 'btest-diff status2.out' failed unexpectedly (exit code 1)
  % cat .diag
  == File ===
  Getting process status ...
  Getting peer status ...
  Name TypeHost StatusPidPeers  Started
  manager  manager localhoststopped
  proxy-1  proxy   localhoststopped
  worker-1 worker  localhoststopped
  worker-2 worker  localhoststopped
  == Diff ===
  --- 
/home/robin/bro/master/aux/broctl/testing/Baseline/command.start-cluster-slowstart/status2.out
2015-03-04 20:16
  +++ status2.out   2015-03-17 22:50:26.578618684 +
  @@ -3,5 +3,5 @@
  Name TypeHost StatusPidPeers  Started
  manager  manager localhoststopped
  proxy-1  proxy   localhoststopped
  -worker-1 worker  localhostcrashed
  +worker-1 worker  localhoststopped
  worker-2 worker  localhoststopped
  ===
{code}




--
This message was sent by Atlassian JIRA
(v6.4-OD-15-055#64014)
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] [JIRA] (BIT-1077) fix policy/protocols/http/header-names.bro

[Robin Sommer (JIRA)]

 [ 
https://bro-tracker.atlassian.net/browse/BIT-1077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robin Sommer updated BIT-1077:
--
Resolution: Merged  (was: Fixed)
Status: Closed  (was: Merge Request)

> fix policy/protocols/http/header-names.bro
> --
>
> Key: BIT-1077
> URL: https://bro-tracker.atlassian.net/browse/BIT-1077
> Project: Bro Issue Tracker
>  Issue Type: Problem
>  Components: Bro
>Affects Versions: git/master
>Reporter: Jon Siwek
>Assignee: Robin Sommer
> Fix For: 2.4
>
>
> This script is wrong for the {{log_server_header_names}} case.



--
This message was sent by Atlassian JIRA
(v6.4-OD-15-055#64014)
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] [JIRA] (BIT-1305) Consider marking some attributes as deprecated

[Robin Sommer (JIRA)]

 [ 
https://bro-tracker.atlassian.net/browse/BIT-1305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robin Sommer updated BIT-1305:
--
Resolution: Merged  (was: Fixed)
Status: Closed  (was: Merge Request)

> Consider marking some attributes as deprecated
> --
>
> Key: BIT-1305
> URL: https://bro-tracker.atlassian.net/browse/BIT-1305
> Project: Bro Issue Tracker
>  Issue Type: Problem
>  Components: Bro
>Reporter: Jon Siwek
>Assignee: Robin Sommer
> Fix For: 2.4
>
>
> Likely candidates for deprecation:
> &rotate_interval
> &rotate_size
> &encrypt
> &mergeable
> &synchronize
> &persistent
> &group
> While the mechanism I added in BIT-757 can't be used to mark attributes as 
> deprecated, I'm thinking it's not difficult to just hard code the scanner to 
> emit a warning when encountering certain attributes.



--
This message was sent by Atlassian JIRA
(v6.4-OD-15-055#64014)
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] [JIRA] (BIT-1332) Please merge topic/johanna/cert-validation

[Robin Sommer (JIRA)]

 [ 
https://bro-tracker.atlassian.net/browse/BIT-1332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robin Sommer updated BIT-1332:
--
Resolution: Merged  (was: Fixed)
Status: Closed  (was: Merge Request)

> Please merge topic/johanna/cert-validation
> --
>
> Key: BIT-1332
> URL: https://bro-tracker.atlassian.net/browse/BIT-1332
> Project: Bro Issue Tracker
>  Issue Type: Improvement
>  Components: Bro
>Affects Versions: git/master
>Reporter: Johanna Amann
>Assignee: Robin Sommer
> Fix For: 2.4
>
>
> Please merge topic/johanna/cert-validation. This is an update to the script 
> used to validate certificates in SSL/TLS connections. Description from main 
> commit:
> {quote}
> Update certificate validation script - new version will cache valid
> intermediate chains that it encounters on the wire and use those to try
> to validate chains that might be missing intermediate certificates.
> This vastly improves the number of certificates that Bro can validate.
> The only drawback is that now validation behavior is not entirely
> predictable anymore - the certificate of a server can fail to validate
> when Bro just started up (due to the intermediate missing), and succeed
> later, when the intermediate can be found in the cache.
> Has been tested on big-ish clusters and should not introduce any
> performance problems.
> {quote}



--
This message was sent by Atlassian JIRA
(v6.4-OD-15-055#64014)
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] [JIRA] (BIT-1341) topic/dnthayer/fixes-for-2.4beta

[Robin Sommer (JIRA)]

 [ 
https://bro-tracker.atlassian.net/browse/BIT-1341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robin Sommer updated BIT-1341:
--
Resolution: Merged  (was: Fixed)
Status: Closed  (was: Merge Request)

> topic/dnthayer/fixes-for-2.4beta
> 
>
> Key: BIT-1341
> URL: https://bro-tracker.atlassian.net/browse/BIT-1341
> Project: Bro Issue Tracker
>  Issue Type: Problem
>  Components: BroControl
>Reporter: Daniel Thayer
>Assignee: Robin Sommer
> Fix For: 2.4
>
>
> Branch topic/dnthayer/fixes-for-2.4beta in the broctl repo addresses the 
> following issues:
> -Improved test setup scripts to specify correct bro install prefix.
> -Fix bug where "./configure --conf-files-dir" did not work
> -Fix bug where "./configure --scriptdir" did not work
> -Print error messages without showing Python stack trace
> -Improved processing of node input args, to remove duplicates and sort
> -Improved sorting of the output by node type and name
> -Added the "deploy" command
> -Update docs for the deploy command



--
This message was sent by Atlassian JIRA
(v6.4-OD-15-055#64014)
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] [JIRA] (BIT-985) 'tail -f' functionality for file reading in input framework

[Johanna Amann (JIRA)]

[ 
https://bro-tracker.atlassian.net/browse/BIT-985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=20015#comment-20015
 ] 

Johanna Amann commented on BIT-985:
---

Thanks for the new patch. Cursory looking at it, it seems that this patch 
changes a lot of functionality in the Raw reader that seems to have nothing to 
do with skipping parts of the input file.

Can you perhaps just sketch what else this patch changes? It seems to change 
something about how the buffering is done in the raw reader, but I am not quite 
sure what all this does on a first glance.

> 'tail -f' functionality for file reading in input framework
> ---
>
> Key: BIT-985
> URL: https://bro-tracker.atlassian.net/browse/BIT-985
> Project: Bro Issue Tracker
>  Issue Type: New Feature
>  Components: Bro
>Affects Versions: git/master
>Reporter: scampbell
>Assignee: Johanna Amann
>Priority: Low
> Fix For: 2.4
>
> Attachments: input.diff, PATCH
>
>
> With the current input framework, file data \-> event translation requires 
> that the entire data file be read at bro start time.  This can be prohibitive 
> when the file sizes become large ( > 1GB ).  
> It would be great to see a file open option that would start reading at the 
> end of the file.



--
This message was sent by Atlassian JIRA
(v6.4-OD-15-055#64014)
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] [JIRA] (BIT-672) Bring POP3 back into the distribution

[Matthias Vallentin (JIRA)]

[ 
https://bro-tracker.atlassian.net/browse/BIT-672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=20014#comment-20014
 ] 

Matthias Vallentin commented on BIT-672:


We had student refactoring the code, but his changes never got merged: 
https://github.com/albert-magyar/bro/tree/topic/pop3. He refactored the scripts 
and I find their quality is good enough for us to ship them with the 
distribution, albeit disabled.

> Bring POP3 back into the distribution
> -
>
> Key: BIT-672
> URL: https://bro-tracker.atlassian.net/browse/BIT-672
> Project: Bro Issue Tracker
>  Issue Type: Task
>  Components: Bro
>Affects Versions: git/master
>Reporter: Matthias Vallentin
>Assignee: Seth Hall
> Fix For: 2.5
>
>
> The current master has no longer support for POP3. It lingers around but we 
> need to bring it back into the distribution.



--
This message was sent by Atlassian JIRA
(v6.4-OD-15-055#64014)
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] [JIRA] (BIT-1199) Better error messages for input file errors in READER_ASCII

[Robin Sommer (JIRA)]

 [ 
https://bro-tracker.atlassian.net/browse/BIT-1199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robin Sommer reassigned BIT-1199:
-

Assignee: Johanna Amann  (was: Robin Sommer)

> Better error messages for input file errors in READER_ASCII
> ---
>
> Key: BIT-1199
> URL: https://bro-tracker.atlassian.net/browse/BIT-1199
> Project: Bro Issue Tracker
>  Issue Type: Problem
>  Components: Bro
>Reporter: grigorescu
>Assignee: Johanna Amann
> Fix For: 2.4
>
> Attachments: test.intel
>
>
> This came up on the mailing list a few weeks ago. If one tries to load the 
> attached file as Intelligence, Bro will error out, with:
> {code}
> internal error: Value not found in enum mappimg. Module: GLOBAL, var: , var 
> size: 0
> {code}
> The attached file contains an extra tab after downloader.com.
> It'd be nice if Bro would tell you that this was an issue with the input 
> reader, which file it occurred in, and a line number.
> I think generally speaking, if there's an issue with an input file, it'd be 
> nice to know the line number.
> (Also, there's a typo in mappimg in the error message that's currently 
> displayed).



--
This message was sent by Atlassian JIRA
(v6.4-OD-15-055#64014)
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] [JIRA] (BIT-1229) loading a non-existant enum from an input file terminates bro

[Johanna Amann (JIRA)]

 [ 
https://bro-tracker.atlassian.net/browse/BIT-1229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Johanna Amann updated BIT-1229:
---
Fix Version/s: (was: 2.5)
   2.4

> loading a non-existant enum from an input file terminates bro
> -
>
> Key: BIT-1229
> URL: https://bro-tracker.atlassian.net/browse/BIT-1229
> Project: Bro Issue Tracker
>  Issue Type: Problem
>  Components: Bro
>Affects Versions: git/master
>Reporter: Justin Azoff
>Assignee: Johanna Amann
> Fix For: 2.4
>
> Attachments: ignored_notices.csv, ignore-notices.bro
>
>
> If you have an input file with an enum in it and it does not exist, bro 
> terminates:
> internal error: Value not found in enum mappimg. Module: NoSuch, var: Notice, 
> var size: 6



--
This message was sent by Atlassian JIRA
(v6.4-OD-15-055#64014)
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] [JIRA] (BIT-1229) loading a non-existant enum from an input file terminates bro

[Johanna Amann (JIRA)]

[ 
https://bro-tracker.atlassian.net/browse/BIT-1229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=20013#comment-20013
 ] 

Johanna Amann commented on BIT-1229:


Just talked to Robin about this - there is probably a different way to solve 
this problem which will make it into 2.4 after all.

> loading a non-existant enum from an input file terminates bro
> -
>
> Key: BIT-1229
> URL: https://bro-tracker.atlassian.net/browse/BIT-1229
> Project: Bro Issue Tracker
>  Issue Type: Problem
>  Components: Bro
>Affects Versions: git/master
>Reporter: Justin Azoff
>Assignee: Johanna Amann
> Fix For: 2.4
>
> Attachments: ignored_notices.csv, ignore-notices.bro
>
>
> If you have an input file with an enum in it and it does not exist, bro 
> terminates:
> internal error: Value not found in enum mappimg. Module: NoSuch, var: Notice, 
> var size: 6



--
This message was sent by Atlassian JIRA
(v6.4-OD-15-055#64014)
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] [JIRA] (BIT-985) 'tail -f' functionality for file reading in input framework

[scampbell (JIRA)]

 [ 
https://bro-tracker.atlassian.net/browse/BIT-985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

scampbell updated BIT-985:
--
Attachment: input.diff

> 'tail -f' functionality for file reading in input framework
> ---
>
> Key: BIT-985
> URL: https://bro-tracker.atlassian.net/browse/BIT-985
> Project: Bro Issue Tracker
>  Issue Type: New Feature
>  Components: Bro
>Affects Versions: git/master
>Reporter: scampbell
>Assignee: Johanna Amann
>Priority: Low
> Fix For: 2.4
>
> Attachments: input.diff, PATCH
>
>
> With the current input framework, file data \-> event translation requires 
> that the entire data file be read at bro start time.  This can be prohibitive 
> when the file sizes become large ( > 1GB ).  
> It would be great to see a file open option that would start reading at the 
> end of the file.



--
This message was sent by Atlassian JIRA
(v6.4-OD-15-055#64014)
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] [JIRA] (BIT-1199) Better error messages for input file errors in READER_ASCII

[Robin Sommer (JIRA)]

 [ 
https://bro-tracker.atlassian.net/browse/BIT-1199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robin Sommer reassigned BIT-1199:
-

Assignee: Robin Sommer

> Better error messages for input file errors in READER_ASCII
> ---
>
> Key: BIT-1199
> URL: https://bro-tracker.atlassian.net/browse/BIT-1199
> Project: Bro Issue Tracker
>  Issue Type: Problem
>  Components: Bro
>Reporter: grigorescu
>Assignee: Robin Sommer
> Fix For: 2.4
>
> Attachments: test.intel
>
>
> This came up on the mailing list a few weeks ago. If one tries to load the 
> attached file as Intelligence, Bro will error out, with:
> {code}
> internal error: Value not found in enum mappimg. Module: GLOBAL, var: , var 
> size: 0
> {code}
> The attached file contains an extra tab after downloader.com.
> It'd be nice if Bro would tell you that this was an issue with the input 
> reader, which file it occurred in, and a line number.
> I think generally speaking, if there's an issue with an input file, it'd be 
> nice to know the line number.
> (Also, there's a typo in mappimg in the error message that's currently 
> displayed).



--
This message was sent by Atlassian JIRA
(v6.4-OD-15-055#64014)
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] [JIRA] (BIT-1171) misc/app-stats/main.bro broken for a few sites

[Jon Siwek (JIRA)]

 [ 
https://bro-tracker.atlassian.net/browse/BIT-1171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jon Siwek reassigned BIT-1171:
--

Assignee: (was: Jon Siwek)

> misc/app-stats/main.bro broken for a few sites
> --
>
> Key: BIT-1171
> URL: https://bro-tracker.atlassian.net/browse/BIT-1171
> Project: Bro Issue Tracker
>  Issue Type: Problem
>  Components: Bro
>Affects Versions: git/master
>Reporter: Johanna Amann
> Fix For: 2.5
>
>
> Currently the reporting of misc/app-stats/main.bro seems to be quite wrong 
> for some of the sites it monitors.
> At the very least the numbers for youtube and netflix are completely off, 
> gmail also seems slightly unbelievable.



--
This message was sent by Atlassian JIRA
(v6.4-OD-15-055#64014)
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] [JIRA] (BIT-985) 'tail -f' functionality for file reading in input framework

[scampbell (JIRA)]

[ 
https://bro-tracker.atlassian.net/browse/BIT-985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=20012#comment-20012
 ] 

scampbell commented on BIT-985:
---

I have a significantly improved patch from the one that I previously attached.  
That one leaked memory rather enthusiastically will send over in a moment.  

> 'tail -f' functionality for file reading in input framework
> ---
>
> Key: BIT-985
> URL: https://bro-tracker.atlassian.net/browse/BIT-985
> Project: Bro Issue Tracker
>  Issue Type: New Feature
>  Components: Bro
>Affects Versions: git/master
>Reporter: scampbell
>Assignee: Johanna Amann
>Priority: Low
> Fix For: 2.4
>
> Attachments: PATCH
>
>
> With the current input framework, file data \-> event translation requires 
> that the entire data file be read at bro start time.  This can be prohibitive 
> when the file sizes become large ( > 1GB ).  
> It would be great to see a file open option that would start reading at the 
> end of the file.



--
This message was sent by Atlassian JIRA
(v6.4-OD-15-055#64014)
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] [JIRA] (BIT-985) 'tail -f' functionality for file reading in input framework

[Johanna Amann (JIRA)]

 [ 
https://bro-tracker.atlassian.net/browse/BIT-985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Johanna Amann updated BIT-985:
--
Fix Version/s: (was: 2.5)
   2.4

> 'tail -f' functionality for file reading in input framework
> ---
>
> Key: BIT-985
> URL: https://bro-tracker.atlassian.net/browse/BIT-985
> Project: Bro Issue Tracker
>  Issue Type: New Feature
>  Components: Bro
>Affects Versions: git/master
>Reporter: scampbell
>Assignee: Johanna Amann
>Priority: Low
> Fix For: 2.4
>
> Attachments: PATCH
>
>
> With the current input framework, file data \-> event translation requires 
> that the entire data file be read at bro start time.  This can be prohibitive 
> when the file sizes become large ( > 1GB ).  
> It would be great to see a file open option that would start reading at the 
> end of the file.



--
This message was sent by Atlassian JIRA
(v6.4-OD-15-055#64014)
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] [JIRA] (BIT-985) 'tail -f' functionality for file reading in input framework

[Johanna Amann (JIRA)]

[ 
https://bro-tracker.atlassian.net/browse/BIT-985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=20011#comment-20011
 ] 

Johanna Amann commented on BIT-985:
---

This is such a small thing that I might try to really still do it for 2.4.

> 'tail -f' functionality for file reading in input framework
> ---
>
> Key: BIT-985
> URL: https://bro-tracker.atlassian.net/browse/BIT-985
> Project: Bro Issue Tracker
>  Issue Type: New Feature
>  Components: Bro
>Affects Versions: git/master
>Reporter: scampbell
>Assignee: Johanna Amann
>Priority: Low
> Fix For: 2.4
>
> Attachments: PATCH
>
>
> With the current input framework, file data \-> event translation requires 
> that the entire data file be read at bro start time.  This can be prohibitive 
> when the file sizes become large ( > 1GB ).  
> It would be great to see a file open option that would start reading at the 
> end of the file.



--
This message was sent by Atlassian JIRA
(v6.4-OD-15-055#64014)
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] [JIRA] (BIT-1253) Bro 2.3 - 2.3.1 manager dieing on Bivio hardware

[Johanna Amann (JIRA)]

 [ 
https://bro-tracker.atlassian.net/browse/BIT-1253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Johanna Amann updated BIT-1253:
---
Resolution: Cannot Reproduce
Status: Closed  (was: Reopened)

I will just close this because we have not gotten any more feedback / 
information on it and it is currently not actionable.

If you ever have more information on this, please feel free to re-open the 
ticket.

> Bro 2.3 - 2.3.1 manager dieing on Bivio hardware
> 
>
> Key: BIT-1253
> URL: https://bro-tracker.atlassian.net/browse/BIT-1253
> Project: Bro Issue Tracker
>  Issue Type: Problem
>  Components: Bro
>Affects Versions: 2.3
> Environment: Bro 2.3 and Bro 2.3.1 
> bivio hardwareLinux CPU.2.6.31-45 has curl 7.36 gperftools 2.2 flex 2.5.39 
> bison 3.0.2 libpcap 1.1 swig 2.0.8
>Reporter: Larry Leviton
>Assignee: Johanna Amann
> Fix For: 2.4
>
>
> After starting bro up, the bro manager crashes in less than 60 seconds.
> Thanks for any help you can give.
> Sent stack trace to vendor (at bottom), and here was their response:
> Comment(s):   Hello Larry,
> We have duplicated a crash in our lab setup that seems to be identical to 
> that experienced by you. The code has changed quite a bit from 2.1 to 2.3.1, 
> and we suspect a bug was introduced.
> What is going on, seems to be that a writer thread is being terminated, and 
> the destructor for the Ascii writer is called eventually. However, the 
> destructor code does some checks and finds out that proper cleanup has not 
> been done, so it aborts. This does not seem to be due to any library 
> incompatibility, and looks more like maybe a race condition was introduced.
> Since you knows the Bro developers, can you please ask them to take a look 
> this and get back to us? We think it requires their expertise at this point.
> Thank You,
> Hassan.
>   
> Bivio Case Information:   
> Bivio Case #: 4566243 
> Date Created: 9/02/2014 08:02 AM PDT  
> Stack trace below:
> GNU gdb (GDB) Fedora (6.8.50.20090302-40.fc11) Copyright (C) 2009 Free 
> Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later 
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
> and "show warranty" for details.
> This GDB was configured as "ppc-redhat-linux-gnu".
> For bug reporting instructions, please see:
> ...
> backtrace
> [New Thread 25501]
> [New Thread 25328]
> [New Thread 25378]
> [New Thread 25379]
> [New Thread 25380]
> [New Thread 25381]
> [New Thread 25382]
> [New Thread 25383]
> [New Thread 25384]
> [New Thread 25385]
> [New Thread 25386]
> [New Thread 25389]
> [New Thread 25442]
> warning: Can't read pathname for load map: Input/output error.
> Missing separate debuginfo for /usr/local/lib/libz.so.1
> Try: yum --enablerepo='*-debuginfo' install 
> /usr/lib/debug/.build-id/a2/0a0d1fc0d48c2a303af1417ccc03308b9de04a
> Missing separate debuginfo for /usr/local/lib/libtcmalloc.so.4
> Try: yum --enablerepo='*-debuginfo' install 
> /usr/lib/debug/.build-id/27/eaf56bc64810920d55b9530156c1e8ffbfd43e
> Missing separate debuginfo for /usr/local/lib/libcurl.so.4
> Try: yum --enablerepo='*-debuginfo' install 
> /usr/lib/debug/.build-id/a7/9a2cebb4abc156495ec0806b1c18015c8eba01
> Reading symbols from /usr/lib/libpcap.so.1...done.
> Loaded symbols for /usr/lib/libpcap.so.1 Reading symbols from 
> /usr/lib/libssl.so.10...done.
> Loaded symbols for /usr/lib/libssl.so.10 Reading symbols from 
> /usr/lib/libcrypto.so.10...done.
> Loaded symbols for /usr/lib/libcrypto.so.10 Reading symbols from 
> /usr/lib/libbind.so.4...done.
> Loaded symbols for /usr/lib/libbind.so.4 Reading symbols from 
> /usr/local/lib/libz.so.1...done.
> Loaded symbols for /usr/local/lib/libz.so.1 Reading symbols from 
> /usr/local/lib/libtcmalloc.so.4...done.
> Loaded symbols for /usr/local/lib/libtcmalloc.so.4 Reading symbols from 
> /usr/local/lib/libcurl.so.4...done.
> Loaded symbols for /usr/local/lib/libcurl.so.4 Reading symbols from 
> /lib/libpthread.so.0...done.
> Loaded symbols for /lib/libpthread.so.0
> Reading symbols from /lib/libdl.so.2...done.
> Loaded symbols for /lib/libdl.so.2
> Reading symbols from /usr/lib/libstdc++.so.6...done.
> Loaded symbols for /usr/lib/libstdc++.so.6 Reading symbols from 
> /lib/libm.so.6...done.
> Loaded symbols for /lib/libm.so.6
> Reading symbols from /lib/libgcc_s.so.1...done.
> Loaded symbols for /lib/libgcc_s.so.1
> Reading symbols from /lib/libc.so.6...done.
> Loaded symbols for /lib/libc.so.6
> Reading symbols from /usr/lib/libzcp.so...done.
> Loaded symbols for /usr/lib/libzcp.so
> Reading symbols from /lib/libgssapi_krb5.so.2...done.
> Loaded symbols for /lib/libgssa

[Bro-Dev] [JIRA] (BIT-1199) Better error messages for input file errors in READER_ASCII

[Johanna Amann (JIRA)]

[ 
https://bro-tracker.atlassian.net/browse/BIT-1199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=20009#comment-20009
 ] 

Johanna Amann commented on BIT-1199:


Addressed in topic/johanna/bit-1199 - error messages now contain the stream 
name:

{quote}
internal error: Value not 'NoSuch::Notice' for stream 'ignored_notices' is not 
a valid enum.
{quote}

> Better error messages for input file errors in READER_ASCII
> ---
>
> Key: BIT-1199
> URL: https://bro-tracker.atlassian.net/browse/BIT-1199
> Project: Bro Issue Tracker
>  Issue Type: Problem
>  Components: Bro
>Reporter: grigorescu
>Assignee: Johanna Amann
> Fix For: 2.4
>
> Attachments: test.intel
>
>
> This came up on the mailing list a few weeks ago. If one tries to load the 
> attached file as Intelligence, Bro will error out, with:
> {code}
> internal error: Value not found in enum mappimg. Module: GLOBAL, var: , var 
> size: 0
> {code}
> The attached file contains an extra tab after downloader.com.
> It'd be nice if Bro would tell you that this was an issue with the input 
> reader, which file it occurred in, and a line number.
> I think generally speaking, if there's an issue with an input file, it'd be 
> nice to know the line number.
> (Also, there's a typo in mappimg in the error message that's currently 
> displayed).



--
This message was sent by Atlassian JIRA
(v6.4-OD-15-055#64014)
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] [JIRA] (BIT-1199) Better error messages for input file errors in READER_ASCII

[Johanna Amann (JIRA)]

 [ 
https://bro-tracker.atlassian.net/browse/BIT-1199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Johanna Amann reassigned BIT-1199:
--

Assignee: (was: Johanna Amann)

> Better error messages for input file errors in READER_ASCII
> ---
>
> Key: BIT-1199
> URL: https://bro-tracker.atlassian.net/browse/BIT-1199
> Project: Bro Issue Tracker
>  Issue Type: Problem
>  Components: Bro
>Reporter: grigorescu
> Fix For: 2.4
>
> Attachments: test.intel
>
>
> This came up on the mailing list a few weeks ago. If one tries to load the 
> attached file as Intelligence, Bro will error out, with:
> {code}
> internal error: Value not found in enum mappimg. Module: GLOBAL, var: , var 
> size: 0
> {code}
> The attached file contains an extra tab after downloader.com.
> It'd be nice if Bro would tell you that this was an issue with the input 
> reader, which file it occurred in, and a line number.
> I think generally speaking, if there's an issue with an input file, it'd be 
> nice to know the line number.
> (Also, there's a typo in mappimg in the error message that's currently 
> displayed).



--
This message was sent by Atlassian JIRA
(v6.4-OD-15-055#64014)
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] [JIRA] (BIT-1199) Better error messages for input file errors in READER_ASCII

[Johanna Amann (JIRA)]

 [ 
https://bro-tracker.atlassian.net/browse/BIT-1199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Johanna Amann updated BIT-1199:
---
Status: Merge Request  (was: Open)

> Better error messages for input file errors in READER_ASCII
> ---
>
> Key: BIT-1199
> URL: https://bro-tracker.atlassian.net/browse/BIT-1199
> Project: Bro Issue Tracker
>  Issue Type: Problem
>  Components: Bro
>Reporter: grigorescu
>Assignee: Johanna Amann
> Fix For: 2.4
>
> Attachments: test.intel
>
>
> This came up on the mailing list a few weeks ago. If one tries to load the 
> attached file as Intelligence, Bro will error out, with:
> {code}
> internal error: Value not found in enum mappimg. Module: GLOBAL, var: , var 
> size: 0
> {code}
> The attached file contains an extra tab after downloader.com.
> It'd be nice if Bro would tell you that this was an issue with the input 
> reader, which file it occurred in, and a line number.
> I think generally speaking, if there's an issue with an input file, it'd be 
> nice to know the line number.
> (Also, there's a typo in mappimg in the error message that's currently 
> displayed).



--
This message was sent by Atlassian JIRA
(v6.4-OD-15-055#64014)
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] [JIRA] (BIT-1253) Bro 2.3 - 2.3.1 manager dieing on Bivio hardware

[Jon Siwek (JIRA)]

 [ 
https://bro-tracker.atlassian.net/browse/BIT-1253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jon Siwek updated BIT-1253:
---
Status: Reopened  (was: Closed)
Resolution: (was: Fixed)

> Bro 2.3 - 2.3.1 manager dieing on Bivio hardware
> 
>
> Key: BIT-1253
> URL: https://bro-tracker.atlassian.net/browse/BIT-1253
> Project: Bro Issue Tracker
>  Issue Type: Problem
>  Components: Bro
>Affects Versions: 2.3
> Environment: Bro 2.3 and Bro 2.3.1 
> bivio hardwareLinux CPU.2.6.31-45 has curl 7.36 gperftools 2.2 flex 2.5.39 
> bison 3.0.2 libpcap 1.1 swig 2.0.8
>Reporter: Larry Leviton
>Assignee: Johanna Amann
> Fix For: 2.4
>
>
> After starting bro up, the bro manager crashes in less than 60 seconds.
> Thanks for any help you can give.
> Sent stack trace to vendor (at bottom), and here was their response:
> Comment(s):   Hello Larry,
> We have duplicated a crash in our lab setup that seems to be identical to 
> that experienced by you. The code has changed quite a bit from 2.1 to 2.3.1, 
> and we suspect a bug was introduced.
> What is going on, seems to be that a writer thread is being terminated, and 
> the destructor for the Ascii writer is called eventually. However, the 
> destructor code does some checks and finds out that proper cleanup has not 
> been done, so it aborts. This does not seem to be due to any library 
> incompatibility, and looks more like maybe a race condition was introduced.
> Since you knows the Bro developers, can you please ask them to take a look 
> this and get back to us? We think it requires their expertise at this point.
> Thank You,
> Hassan.
>   
> Bivio Case Information:   
> Bivio Case #: 4566243 
> Date Created: 9/02/2014 08:02 AM PDT  
> Stack trace below:
> GNU gdb (GDB) Fedora (6.8.50.20090302-40.fc11) Copyright (C) 2009 Free 
> Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later 
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
> and "show warranty" for details.
> This GDB was configured as "ppc-redhat-linux-gnu".
> For bug reporting instructions, please see:
> ...
> backtrace
> [New Thread 25501]
> [New Thread 25328]
> [New Thread 25378]
> [New Thread 25379]
> [New Thread 25380]
> [New Thread 25381]
> [New Thread 25382]
> [New Thread 25383]
> [New Thread 25384]
> [New Thread 25385]
> [New Thread 25386]
> [New Thread 25389]
> [New Thread 25442]
> warning: Can't read pathname for load map: Input/output error.
> Missing separate debuginfo for /usr/local/lib/libz.so.1
> Try: yum --enablerepo='*-debuginfo' install 
> /usr/lib/debug/.build-id/a2/0a0d1fc0d48c2a303af1417ccc03308b9de04a
> Missing separate debuginfo for /usr/local/lib/libtcmalloc.so.4
> Try: yum --enablerepo='*-debuginfo' install 
> /usr/lib/debug/.build-id/27/eaf56bc64810920d55b9530156c1e8ffbfd43e
> Missing separate debuginfo for /usr/local/lib/libcurl.so.4
> Try: yum --enablerepo='*-debuginfo' install 
> /usr/lib/debug/.build-id/a7/9a2cebb4abc156495ec0806b1c18015c8eba01
> Reading symbols from /usr/lib/libpcap.so.1...done.
> Loaded symbols for /usr/lib/libpcap.so.1 Reading symbols from 
> /usr/lib/libssl.so.10...done.
> Loaded symbols for /usr/lib/libssl.so.10 Reading symbols from 
> /usr/lib/libcrypto.so.10...done.
> Loaded symbols for /usr/lib/libcrypto.so.10 Reading symbols from 
> /usr/lib/libbind.so.4...done.
> Loaded symbols for /usr/lib/libbind.so.4 Reading symbols from 
> /usr/local/lib/libz.so.1...done.
> Loaded symbols for /usr/local/lib/libz.so.1 Reading symbols from 
> /usr/local/lib/libtcmalloc.so.4...done.
> Loaded symbols for /usr/local/lib/libtcmalloc.so.4 Reading symbols from 
> /usr/local/lib/libcurl.so.4...done.
> Loaded symbols for /usr/local/lib/libcurl.so.4 Reading symbols from 
> /lib/libpthread.so.0...done.
> Loaded symbols for /lib/libpthread.so.0
> Reading symbols from /lib/libdl.so.2...done.
> Loaded symbols for /lib/libdl.so.2
> Reading symbols from /usr/lib/libstdc++.so.6...done.
> Loaded symbols for /usr/lib/libstdc++.so.6 Reading symbols from 
> /lib/libm.so.6...done.
> Loaded symbols for /lib/libm.so.6
> Reading symbols from /lib/libgcc_s.so.1...done.
> Loaded symbols for /lib/libgcc_s.so.1
> Reading symbols from /lib/libc.so.6...done.
> Loaded symbols for /lib/libc.so.6
> Reading symbols from /usr/lib/libzcp.so...done.
> Loaded symbols for /usr/lib/libzcp.so
> Reading symbols from /lib/libgssapi_krb5.so.2...done.
> Loaded symbols for /lib/libgssapi_krb5.so.2 Reading symbols from 
> /lib/libkrb5.so.3...done.
> Loaded symbols for /lib/libkrb5.so.3
> Reading symbols from /lib/libcom_err.so.2...done.
> Loaded symbols for /lib/libcom_err.so.2
> Reading symbols from 

[Bro-Dev] [JIRA] (BIT-1253) Bro 2.3 - 2.3.1 manager dieing on Bivio hardware

[Jon Siwek (JIRA)]

 [ 
https://bro-tracker.atlassian.net/browse/BIT-1253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jon Siwek updated BIT-1253:
---
Status: Closed  (was: Reopened)

> Bro 2.3 - 2.3.1 manager dieing on Bivio hardware
> 
>
> Key: BIT-1253
> URL: https://bro-tracker.atlassian.net/browse/BIT-1253
> Project: Bro Issue Tracker
>  Issue Type: Problem
>  Components: Bro
>Affects Versions: 2.3
> Environment: Bro 2.3 and Bro 2.3.1 
> bivio hardwareLinux CPU.2.6.31-45 has curl 7.36 gperftools 2.2 flex 2.5.39 
> bison 3.0.2 libpcap 1.1 swig 2.0.8
>Reporter: Larry Leviton
>Assignee: Johanna Amann
> Fix For: 2.4
>
>
> After starting bro up, the bro manager crashes in less than 60 seconds.
> Thanks for any help you can give.
> Sent stack trace to vendor (at bottom), and here was their response:
> Comment(s):   Hello Larry,
> We have duplicated a crash in our lab setup that seems to be identical to 
> that experienced by you. The code has changed quite a bit from 2.1 to 2.3.1, 
> and we suspect a bug was introduced.
> What is going on, seems to be that a writer thread is being terminated, and 
> the destructor for the Ascii writer is called eventually. However, the 
> destructor code does some checks and finds out that proper cleanup has not 
> been done, so it aborts. This does not seem to be due to any library 
> incompatibility, and looks more like maybe a race condition was introduced.
> Since you knows the Bro developers, can you please ask them to take a look 
> this and get back to us? We think it requires their expertise at this point.
> Thank You,
> Hassan.
>   
> Bivio Case Information:   
> Bivio Case #: 4566243 
> Date Created: 9/02/2014 08:02 AM PDT  
> Stack trace below:
> GNU gdb (GDB) Fedora (6.8.50.20090302-40.fc11) Copyright (C) 2009 Free 
> Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later 
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
> and "show warranty" for details.
> This GDB was configured as "ppc-redhat-linux-gnu".
> For bug reporting instructions, please see:
> ...
> backtrace
> [New Thread 25501]
> [New Thread 25328]
> [New Thread 25378]
> [New Thread 25379]
> [New Thread 25380]
> [New Thread 25381]
> [New Thread 25382]
> [New Thread 25383]
> [New Thread 25384]
> [New Thread 25385]
> [New Thread 25386]
> [New Thread 25389]
> [New Thread 25442]
> warning: Can't read pathname for load map: Input/output error.
> Missing separate debuginfo for /usr/local/lib/libz.so.1
> Try: yum --enablerepo='*-debuginfo' install 
> /usr/lib/debug/.build-id/a2/0a0d1fc0d48c2a303af1417ccc03308b9de04a
> Missing separate debuginfo for /usr/local/lib/libtcmalloc.so.4
> Try: yum --enablerepo='*-debuginfo' install 
> /usr/lib/debug/.build-id/27/eaf56bc64810920d55b9530156c1e8ffbfd43e
> Missing separate debuginfo for /usr/local/lib/libcurl.so.4
> Try: yum --enablerepo='*-debuginfo' install 
> /usr/lib/debug/.build-id/a7/9a2cebb4abc156495ec0806b1c18015c8eba01
> Reading symbols from /usr/lib/libpcap.so.1...done.
> Loaded symbols for /usr/lib/libpcap.so.1 Reading symbols from 
> /usr/lib/libssl.so.10...done.
> Loaded symbols for /usr/lib/libssl.so.10 Reading symbols from 
> /usr/lib/libcrypto.so.10...done.
> Loaded symbols for /usr/lib/libcrypto.so.10 Reading symbols from 
> /usr/lib/libbind.so.4...done.
> Loaded symbols for /usr/lib/libbind.so.4 Reading symbols from 
> /usr/local/lib/libz.so.1...done.
> Loaded symbols for /usr/local/lib/libz.so.1 Reading symbols from 
> /usr/local/lib/libtcmalloc.so.4...done.
> Loaded symbols for /usr/local/lib/libtcmalloc.so.4 Reading symbols from 
> /usr/local/lib/libcurl.so.4...done.
> Loaded symbols for /usr/local/lib/libcurl.so.4 Reading symbols from 
> /lib/libpthread.so.0...done.
> Loaded symbols for /lib/libpthread.so.0
> Reading symbols from /lib/libdl.so.2...done.
> Loaded symbols for /lib/libdl.so.2
> Reading symbols from /usr/lib/libstdc++.so.6...done.
> Loaded symbols for /usr/lib/libstdc++.so.6 Reading symbols from 
> /lib/libm.so.6...done.
> Loaded symbols for /lib/libm.so.6
> Reading symbols from /lib/libgcc_s.so.1...done.
> Loaded symbols for /lib/libgcc_s.so.1
> Reading symbols from /lib/libc.so.6...done.
> Loaded symbols for /lib/libc.so.6
> Reading symbols from /usr/lib/libzcp.so...done.
> Loaded symbols for /usr/lib/libzcp.so
> Reading symbols from /lib/libgssapi_krb5.so.2...done.
> Loaded symbols for /lib/libgssapi_krb5.so.2 Reading symbols from 
> /lib/libkrb5.so.3...done.
> Loaded symbols for /lib/libkrb5.so.3
> Reading symbols from /lib/libcom_err.so.2...done.
> Loaded symbols for /lib/libcom_err.so.2
> Reading symbols from /lib/libk5crypto.so.3...done.
> Loade

[Bro-Dev] [JIRA] (BIT-1229) loading a non-existant enum from an input file terminates bro

[Jon Siwek (JIRA)]

 [ 
https://bro-tracker.atlassian.net/browse/BIT-1229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jon Siwek updated BIT-1229:
---
Status: Closed  (was: Reopened)

> loading a non-existant enum from an input file terminates bro
> -
>
> Key: BIT-1229
> URL: https://bro-tracker.atlassian.net/browse/BIT-1229
> Project: Bro Issue Tracker
>  Issue Type: Problem
>  Components: Bro
>Affects Versions: git/master
>Reporter: Justin Azoff
>Assignee: Johanna Amann
> Fix For: 2.5
>
> Attachments: ignored_notices.csv, ignore-notices.bro
>
>
> If you have an input file with an enum in it and it does not exist, bro 
> terminates:
> internal error: Value not found in enum mappimg. Module: NoSuch, var: Notice, 
> var size: 6



--
This message was sent by Atlassian JIRA
(v6.4-OD-15-055#64014)
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] [JIRA] (BIT-1229) loading a non-existant enum from an input file terminates bro

[Jon Siwek (JIRA)]

 [ 
https://bro-tracker.atlassian.net/browse/BIT-1229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jon Siwek updated BIT-1229:
---
Status: Reopened  (was: Closed)
Resolution: (was: Fixed)

> loading a non-existant enum from an input file terminates bro
> -
>
> Key: BIT-1229
> URL: https://bro-tracker.atlassian.net/browse/BIT-1229
> Project: Bro Issue Tracker
>  Issue Type: Problem
>  Components: Bro
>Affects Versions: git/master
>Reporter: Justin Azoff
>Assignee: Johanna Amann
> Fix For: 2.5
>
> Attachments: ignored_notices.csv, ignore-notices.bro
>
>
> If you have an input file with an enum in it and it does not exist, bro 
> terminates:
> internal error: Value not found in enum mappimg. Module: NoSuch, var: Notice, 
> var size: 6



--
This message was sent by Atlassian JIRA
(v6.4-OD-15-055#64014)
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] [JIRA] (BIT-1331) BroControl manager crashes when logs rotate

[Aaron Eppert (JIRA)]

[ 
https://bro-tracker.atlassian.net/browse/BIT-1331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=20008#comment-20008
 ] 

Aaron Eppert edited comment on BIT-1331 at 3/17/15 3:06 PM:


Encountering the same problem in a clustered configuration with only a single 
worker and proxy at the moment. Confirming the same error and the linked 
BIT-1253 ticket.

Also getting this:

fatal error in : val::CONVERTER (string/port) (80/tcp)


was (Author: aeppert):
Encountering the same problem in a clustered configuration with only a single 
worker and proxy at the moment. Confirming the same error and the linked 
BIT-1253 ticket.

> BroControl manager crashes when logs rotate
> ---
>
> Key: BIT-1331
> URL: https://bro-tracker.atlassian.net/browse/BIT-1331
> Project: Bro Issue Tracker
>  Issue Type: Problem
>  Components: Bro
>Affects Versions: git/master, 2.4
> Environment: Ubuntu 12.04.5 LTS, PF_RING lb_method
>Reporter: Josh Liburdi
>Priority: High
> Fix For: 2.4
>
>
> The BroControl manager crashes when the logs rotate. Workers run fine through 
> this process. 
> stderr.log output:
> internal error: finish missing
> /usr/local/bro/share/broctl/scripts/run-bro: line 100: 157357 Aborted 
> (core dumped) nohup "$mybro" "$@"
> send-mail: SENDMAIL-NOTFOUND not found



--
This message was sent by Atlassian JIRA
(v6.4-OD-15-055#64014)
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] [JIRA] (BIT-1331) BroControl manager crashes when logs rotate

[Aaron Eppert (JIRA)]

[ 
https://bro-tracker.atlassian.net/browse/BIT-1331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=20008#comment-20008
 ] 

Aaron Eppert commented on BIT-1331:
---

Encountering the same problem in a clustered configuration with only a single 
worker and proxy at the moment. Confirming the same error and the linked 
BIT-1253 ticket.

> BroControl manager crashes when logs rotate
> ---
>
> Key: BIT-1331
> URL: https://bro-tracker.atlassian.net/browse/BIT-1331
> Project: Bro Issue Tracker
>  Issue Type: Problem
>  Components: Bro
>Affects Versions: git/master, 2.4
> Environment: Ubuntu 12.04.5 LTS, PF_RING lb_method
>Reporter: Josh Liburdi
>Priority: High
> Fix For: 2.4
>
>
> The BroControl manager crashes when the logs rotate. Workers run fine through 
> this process. 
> stderr.log output:
> internal error: finish missing
> /usr/local/bro/share/broctl/scripts/run-bro: line 100: 157357 Aborted 
> (core dumped) nohup "$mybro" "$@"
> send-mail: SENDMAIL-NOTFOUND not found



--
This message was sent by Atlassian JIRA
(v6.4-OD-15-055#64014)
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] [JIRA] (BIT-528) Python 3 compatibility

[Jon Siwek (JIRA)]

 [ 
https://bro-tracker.atlassian.net/browse/BIT-528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jon Siwek updated BIT-528:
--
   Resolution: Fixed
Fix Version/s: 2.4
   Status: Closed  (was: Open)

I think Bro 2.4 is now Python 3 compatible.  Daniel, was there any remaining 
pieces?

> Python 3 compatibility
> --
>
> Key: BIT-528
> URL: https://bro-tracker.atlassian.net/browse/BIT-528
> Project: Bro Issue Tracker
>  Issue Type: Task
>  Components: BroControl
>Reporter: Robin Sommer
> Fix For: 2.4
>
>
> We should make sure that BroControl (and other Pytjon pieces we ship
> run fine with Python 3.x).



--
This message was sent by Atlassian JIRA
(v6.4-OD-15-055#64014)
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] [JIRA] (BIT-667) Non-deterministic behavior when deleting current set element during iteration

[Jon Siwek (JIRA)]

 [ 
https://bro-tracker.atlassian.net/browse/BIT-667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jon Siwek updated BIT-667:
--
   Resolution: Duplicate
Fix Version/s: 2.4
   Status: Closed  (was: Open)

As part of BIT-978, I'll be adding documentation to the for-each loop that 
explains you can't modify a container's membership while iterating over it.  
(There is code that in Bro's core that would allow this, "MakeRobustCookie", 
but I assume it's not used for performance reasons).

> Non-deterministic behavior when deleting current set element during iteration
> -
>
> Key: BIT-667
> URL: https://bro-tracker.atlassian.net/browse/BIT-667
> Project: Bro Issue Tracker
>  Issue Type: Problem
>  Components: Bro
>Reporter: david.bianco
> Fix For: 2.4
>
> Attachments: part3.bro
>
>
> As we discussed during the Bro Workshop at NCSA, the attached code shows some 
> non-deterministic results while deleting the current element while iterating 
> through a set.  Most of the time, it works.  Some times, it doesn't.
> Have a look at subexercise 3's output.



--
This message was sent by Atlassian JIRA
(v6.4-OD-15-055#64014)
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] [JIRA] (BIT-1338) http response mime types uninitialized in file_over_new_connection event

[Paul Pearce (JIRA)]

[ 
https://bro-tracker.atlassian.net/browse/BIT-1338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=20005#comment-20005
 ] 

Paul Pearce commented on BIT-1338:
--

bq. The "conns" field of fa_file should hold all the connection records over 
which the file was transferred, if any. Does that help simplify your analysis

Hmm. Yes this does help. I believe it will yield subtly different semantics on 
files that span multiple connections, but that is not a problem for me.

Thanks.

> http response mime types uninitialized in file_over_new_connection event
> 
>
> Key: BIT-1338
> URL: https://bro-tracker.atlassian.net/browse/BIT-1338
> Project: Bro Issue Tracker
>  Issue Type: Problem
>  Components: Bro
>Affects Versions: git/master
>Reporter: Paul Pearce
>  Labels: mime
> Fix For: 2.4
>
>
> http resp_mime_types (accessed via: connection$http$resp_mime_types) are no 
> longer initialized during the file_over_new_connection event. This is new 
> behavior between Bro v2.3 and git/master.
> The following snippet shows the new behavior on one of the included bro test 
> traces.
> {code:bash}
> $ bro_v23 -e 'event file_over_new_connection(f: fa_file, c:connection, 
> is_orig:bool){ print c$http?$resp_mime_types; }' -r 
> bro/testing/btest/Traces/http/get.trace 
> T
> $ bro_git -e 'event file_over_new_connection(f: fa_file, c:connection, 
> is_orig:bool){ print c$http?$resp_mime_types; }' -r 
> bro/testing/btest/Traces/http/get.trace 
> F
> {code}
> It's worth pointing out that ultimately the resp_mime_types field does get 
> set for subsequent events.
> {code:bash}
> $ bro_v23 -e 'event http_message_done (c: connection, is_orig: bool,  stat: 
> http_message_stat){ if (!is_orig) print c$http?$resp_mime_types; }' -r 
> bro/testing/btest/Traces/http/get.trace 
> T
> $ bro_git -e 'event http_message_done (c: connection, is_orig: bool,  stat: 
> http_message_stat){ if (!is_orig) print c$http?$resp_mime_types; }' -r 
> bro/testing/btest/Traces/http/get.trace 
> T
> {code}



--
This message was sent by Atlassian JIRA
(v6.4-OD-15-055#64014)
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] [JIRA] (BIT-1154) Formatters restructed in: topic/seth/json-formatter

[Jon Siwek (JIRA)]

 [ 
https://bro-tracker.atlassian.net/browse/BIT-1154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jon Siwek updated BIT-1154:
---
Fix Version/s: 2.4

> Formatters restructed in: topic/seth/json-formatter
> ---
>
> Key: BIT-1154
> URL: https://bro-tracker.atlassian.net/browse/BIT-1154
> Project: Bro Issue Tracker
>  Issue Type: New Feature
>  Components: Bro
>Affects Versions: 2.4
>Reporter: Seth Hall
> Fix For: 2.4
>
>
> topic/seth/json-formatter has an abstraction for Formatters and I created a 
> formatters directory under threading.  There is also a new JSON formatter and 
> support in the Ascii and ElasticSearch writers for the JSON formatter.
> I went ahead and threw in per-filter configuration options for the Ascii 
> writer for all of the options that were exposed globally too.



--
This message was sent by Atlassian JIRA
(v6.4-OD-15-055#64014)
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] [JIRA] (BIT-1181) Input-framework errors should be fatal (or Notice_Alarm) instead of silent reporter::error failures

[Jon Siwek (JIRA)]

 [ 
https://bro-tracker.atlassian.net/browse/BIT-1181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jon Siwek updated BIT-1181:
---
Fix Version/s: 2.5

> Input-framework errors should be fatal (or Notice_Alarm) instead of silent 
> reporter::error failures
> ---
>
> Key: BIT-1181
> URL: https://bro-tracker.atlassian.net/browse/BIT-1181
> Project: Bro Issue Tracker
>  Issue Type: Problem
>  Components: Bro
>Affects Versions: 2.2
>Reporter: Aashish Sharma
>Assignee: Johanna Amann
>  Labels: input-framework
> Fix For: 2.5
>
>
> I noticed many times that if there is a problem in a feed file (syntax, or 
> some other issue) and input-framework is unable to read the file, it 
> generates a Reporter::Error. This is a silent failure condition ie bro 
> continues to operate as normal and the error is logged into reporter log. 
> Ideally above is the right thing to do. However, This failure results in no 
> data in the tables getting updated any more while I continue to operate 
> under-impression that Bro is working fine (unless I have explicitly been 
> looking at reporter log for this issue , which now I do). 
> If input-framework is unable to read/digest data from a feed, I believe that 
> should be a (configurable) fatal error or something which at least triggers 
> an alarm/alert/email. 



--
This message was sent by Atlassian JIRA
(v6.4-OD-15-055#64014)
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] [JIRA] (BIT-1199) Better error messages for input file errors in READER_ASCII

[Johanna Amann (JIRA)]

 [ 
https://bro-tracker.atlassian.net/browse/BIT-1199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Johanna Amann updated BIT-1199:
---
Priority: Normal  (was: Low)

> Better error messages for input file errors in READER_ASCII
> ---
>
> Key: BIT-1199
> URL: https://bro-tracker.atlassian.net/browse/BIT-1199
> Project: Bro Issue Tracker
>  Issue Type: Problem
>  Components: Bro
>Reporter: grigorescu
>Assignee: Johanna Amann
> Fix For: 2.4
>
> Attachments: test.intel
>
>
> This came up on the mailing list a few weeks ago. If one tries to load the 
> attached file as Intelligence, Bro will error out, with:
> {code}
> internal error: Value not found in enum mappimg. Module: GLOBAL, var: , var 
> size: 0
> {code}
> The attached file contains an extra tab after downloader.com.
> It'd be nice if Bro would tell you that this was an issue with the input 
> reader, which file it occurred in, and a line number.
> I think generally speaking, if there's an issue with an input file, it'd be 
> nice to know the line number.
> (Also, there's a typo in mappimg in the error message that's currently 
> displayed).



--
This message was sent by Atlassian JIRA
(v6.4-OD-15-055#64014)
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] [JIRA] (BIT-1199) Better error messages for input file errors in READER_ASCII

[Jon Siwek (JIRA)]

[ 
https://bro-tracker.atlassian.net/browse/BIT-1199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=20003#comment-20003
 ] 

Jon Siwek commented on BIT-1199:


Johanna, though it may not be trivial to continue processing input on errors 
like these as mentioned in BIT-1229, is it easy to improve the error message to 
include file name and line number info as suggested by this ticket?

> Better error messages for input file errors in READER_ASCII
> ---
>
> Key: BIT-1199
> URL: https://bro-tracker.atlassian.net/browse/BIT-1199
> Project: Bro Issue Tracker
>  Issue Type: Problem
>  Components: Bro
>Reporter: grigorescu
>Assignee: Johanna Amann
>Priority: Low
> Fix For: 2.4
>
> Attachments: test.intel
>
>
> This came up on the mailing list a few weeks ago. If one tries to load the 
> attached file as Intelligence, Bro will error out, with:
> {code}
> internal error: Value not found in enum mappimg. Module: GLOBAL, var: , var 
> size: 0
> {code}
> The attached file contains an extra tab after downloader.com.
> It'd be nice if Bro would tell you that this was an issue with the input 
> reader, which file it occurred in, and a line number.
> I think generally speaking, if there's an issue with an input file, it'd be 
> nice to know the line number.
> (Also, there's a typo in mappimg in the error message that's currently 
> displayed).



--
This message was sent by Atlassian JIRA
(v6.4-OD-15-055#64014)
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] [JIRA] (BIT-1199) Better error messages for input file errors in READER_ASCII

[Johanna Amann (JIRA)]

[ 
https://bro-tracker.atlassian.net/browse/BIT-1199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=20004#comment-20004
 ] 

Johanna Amann commented on BIT-1199:


I will take a look. It might be possible to output the file name and the 
offending entry - line numbers are sadly not available in the core.

> Better error messages for input file errors in READER_ASCII
> ---
>
> Key: BIT-1199
> URL: https://bro-tracker.atlassian.net/browse/BIT-1199
> Project: Bro Issue Tracker
>  Issue Type: Problem
>  Components: Bro
>Reporter: grigorescu
>Assignee: Johanna Amann
>Priority: Low
> Fix For: 2.4
>
> Attachments: test.intel
>
>
> This came up on the mailing list a few weeks ago. If one tries to load the 
> attached file as Intelligence, Bro will error out, with:
> {code}
> internal error: Value not found in enum mappimg. Module: GLOBAL, var: , var 
> size: 0
> {code}
> The attached file contains an extra tab after downloader.com.
> It'd be nice if Bro would tell you that this was an issue with the input 
> reader, which file it occurred in, and a line number.
> I think generally speaking, if there's an issue with an input file, it'd be 
> nice to know the line number.
> (Also, there's a typo in mappimg in the error message that's currently 
> displayed).



--
This message was sent by Atlassian JIRA
(v6.4-OD-15-055#64014)
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] [JIRA] (BIT-1199) Better error messages for input file errors in READER_ASCII

[Jon Siwek (JIRA)]

 [ 
https://bro-tracker.atlassian.net/browse/BIT-1199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jon Siwek reassigned BIT-1199:
--

Assignee: Johanna Amann

> Better error messages for input file errors in READER_ASCII
> ---
>
> Key: BIT-1199
> URL: https://bro-tracker.atlassian.net/browse/BIT-1199
> Project: Bro Issue Tracker
>  Issue Type: Problem
>  Components: Bro
>Reporter: grigorescu
>Assignee: Johanna Amann
>Priority: Low
> Fix For: 2.4
>
> Attachments: test.intel
>
>
> This came up on the mailing list a few weeks ago. If one tries to load the 
> attached file as Intelligence, Bro will error out, with:
> {code}
> internal error: Value not found in enum mappimg. Module: GLOBAL, var: , var 
> size: 0
> {code}
> The attached file contains an extra tab after downloader.com.
> It'd be nice if Bro would tell you that this was an issue with the input 
> reader, which file it occurred in, and a line number.
> I think generally speaking, if there's an issue with an input file, it'd be 
> nice to know the line number.
> (Also, there's a typo in mappimg in the error message that's currently 
> displayed).



--
This message was sent by Atlassian JIRA
(v6.4-OD-15-055#64014)
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] [JIRA] (BIT-1198) Input framework's READER_ASCII can't handle DOS files

[Jon Siwek (JIRA)]

 [ 
https://bro-tracker.atlassian.net/browse/BIT-1198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jon Siwek updated BIT-1198:
---
Fix Version/s: 2.5

> Input framework's READER_ASCII can't handle DOS files
> -
>
> Key: BIT-1198
> URL: https://bro-tracker.atlassian.net/browse/BIT-1198
> Project: Bro Issue Tracker
>  Issue Type: Problem
>  Components: Bro
>Affects Versions: git/master
>Reporter: grigorescu
>Priority: Low
> Fix For: 2.5
>
> Attachments: test.intel
>
>
> DOS files use CR+LF for newlines, while Linux uses only LF. I've heard of a 
> number of cases where people generate files designed to be read with the 
> input framework on Windows (e.g. exporting from Excel). It'd be nice if we 
> could support that.
> Trying to load the attached sample file results in:
> {code}
> error: test.intel/Input::READER_ASCII: Did not find requested field 
> meta.source in input data file test.intel.
> error: test.intel/Input::READER_ASCII: Init: cannot open test.intel; headers 
> are incorrect
> error: test.intel/Input::READER_ASCII: Init failed
> error: test.intel/Input::READER_ASCII: terminating thread
> {code}



--
This message was sent by Atlassian JIRA
(v6.4-OD-15-055#64014)
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] [JIRA] (BIT-1199) Better error messages for input file errors in READER_ASCII

[Jon Siwek (JIRA)]

 [ 
https://bro-tracker.atlassian.net/browse/BIT-1199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jon Siwek updated BIT-1199:
---
Fix Version/s: 2.4

> Better error messages for input file errors in READER_ASCII
> ---
>
> Key: BIT-1199
> URL: https://bro-tracker.atlassian.net/browse/BIT-1199
> Project: Bro Issue Tracker
>  Issue Type: Problem
>  Components: Bro
>Reporter: grigorescu
>Priority: Low
> Fix For: 2.4
>
> Attachments: test.intel
>
>
> This came up on the mailing list a few weeks ago. If one tries to load the 
> attached file as Intelligence, Bro will error out, with:
> {code}
> internal error: Value not found in enum mappimg. Module: GLOBAL, var: , var 
> size: 0
> {code}
> The attached file contains an extra tab after downloader.com.
> It'd be nice if Bro would tell you that this was an issue with the input 
> reader, which file it occurred in, and a line number.
> I think generally speaking, if there's an issue with an input file, it'd be 
> nice to know the line number.
> (Also, there's a typo in mappimg in the error message that's currently 
> displayed).



--
This message was sent by Atlassian JIRA
(v6.4-OD-15-055#64014)
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] [JIRA] (BIT-1237) Bro script declaration ordering

[Jon Siwek (JIRA)]

 [ 
https://bro-tracker.atlassian.net/browse/BIT-1237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jon Siwek updated BIT-1237:
---
Description: 
During one of the scripting exercises I noticed odd behavior with items 
declared in the global scope:

{code}
# error.bro
not working:

local test_var = "test_var";

function test_1()
{
print "test_1";
}

print test_var;
test_1();

>>> Output:
error in ./test.bro, line 3: syntax error, at or near "test_1"


# working.bro
working:


function test_1()
{
print "test_1";
}

local test_var = "test_var";
print test_var;
test_1();

>>> Output:
test_var
test_1

#
{code}

To declare a function, bro 2.3 forced me to do it at the top of the file. On 
the exercise with the redef of the grid ftp size variable I noticed the same 
issue with redef, it required me to put the redef at the very top of the file. 

Robin asked me to open a ticket and mentioned this was low priority.

  was:
During one of the scripting exercises I noticed odd behavior with items 
declared in the global scope:

# error.bro
not working:

local test_var = "test_var";

function test_1()
{
print "test_1";
}

print test_var;
test_1();

>>> Output:
error in ./test.bro, line 3: syntax error, at or near "test_1"


# working.bro
working:


function test_1()
{
print "test_1";
}

local test_var = "test_var";
print test_var;
test_1();

>>> Output:
test_var
test_1

#

To declare a function, bro 2.3 forced me to do it at the top of the file. On 
the exercise with the redef of the grid ftp size variable I noticed the same 
issue with redef, it required me to put the redef at the very top of the file. 

Robin asked me to open a ticket and mentioned this was low priority.


> Bro script declaration ordering
> ---
>
> Key: BIT-1237
> URL: https://bro-tracker.atlassian.net/browse/BIT-1237
> Project: Bro Issue Tracker
>  Issue Type: Problem
>  Components: Bro
>Affects Versions: 2.3
> Environment: Bro con training VM
>Reporter: Peter Kaloroumakis
>Priority: Trivial
>  Labels: BroScript
>
> During one of the scripting exercises I noticed odd behavior with items 
> declared in the global scope:
> {code}
> # error.bro
> not working:
> 
> local test_var = "test_var";
> function test_1()
> {
> print "test_1";
> }
> print test_var;
> test_1();
> >>> Output:
> error in ./test.bro, line 3: syntax error, at or near "test_1"
> # working.bro
> working:
> 
> function test_1()
> {
> print "test_1";
> }
> local test_var = "test_var";
> print test_var;
> test_1();
> >>> Output:
> test_var
> test_1
> #
> {code}
> To declare a function, bro 2.3 forced me to do it at the top of the file. On 
> the exercise with the redef of the grid ftp size variable I noticed the same 
> issue with redef, it required me to put the redef at the very top of the 
> file. 
> Robin asked me to open a ticket and mentioned this was low priority.



--
This message was sent by Atlassian JIRA
(v6.4-OD-15-055#64014)
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] [JIRA] (BIT-1263) Implementing three event handlers for supported data structure in Modbus Analyzer

[Jon Siwek (JIRA)]

[ 
https://bro-tracker.atlassian.net/browse/BIT-1263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=20002#comment-20002
 ] 

Jon Siwek commented on BIT-1263:


Seems like this is nearly done, but just didn't get merged due to lack of test 
case?  Can we get that for 2.4 ?

> Implementing three event handlers for supported data structure in Modbus 
> Analyzer
> -
>
> Key: BIT-1263
> URL: https://bro-tracker.atlassian.net/browse/BIT-1263
> Project: Bro Issue Tracker
>  Issue Type: Improvement
>  Components: Bro
>Reporter: hui
>Assignee: hui
>Priority: Low
>  Labels: analyzer, modbus
> Fix For: 2.4
>
>
> Three support data structures are defined in Modbus analyzer:
> FileRecordRequest,
> FileRecordResponse,
> ReferenceWithData
> Three event handlers are declared for them. 
> The changes are already made and pushed into the branch:
> topic/hui/modbus-events2



--
This message was sent by Atlassian JIRA
(v6.4-OD-15-055#64014)
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] [JIRA] (BIT-1263) Implementing three event handlers for supported data structure in Modbus Analyzer

[Jon Siwek (JIRA)]

 [ 
https://bro-tracker.atlassian.net/browse/BIT-1263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jon Siwek updated BIT-1263:
---
Fix Version/s: 2.4

> Implementing three event handlers for supported data structure in Modbus 
> Analyzer
> -
>
> Key: BIT-1263
> URL: https://bro-tracker.atlassian.net/browse/BIT-1263
> Project: Bro Issue Tracker
>  Issue Type: Improvement
>  Components: Bro
>Reporter: hui
>Assignee: hui
>Priority: Low
>  Labels: analyzer, modbus
> Fix For: 2.4
>
>
> Three support data structures are defined in Modbus analyzer:
> FileRecordRequest,
> FileRecordResponse,
> ReferenceWithData
> Three event handlers are declared for them. 
> The changes are already made and pushed into the branch:
> topic/hui/modbus-events2



--
This message was sent by Atlassian JIRA
(v6.4-OD-15-055#64014)
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] [JIRA] (BIT-1274) Moving GeoIP Code to Plugin

[Jon Siwek (JIRA)]

 [ 
https://bro-tracker.atlassian.net/browse/BIT-1274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jon Siwek updated BIT-1274:
---
Fix Version/s: 2.5

> Moving GeoIP Code to Plugin
> ---
>
> Key: BIT-1274
> URL: https://bro-tracker.atlassian.net/browse/BIT-1274
> Project: Bro Issue Tracker
>  Issue Type: Improvement
>  Components: Bro
>Reporter: AK
> Fix For: 2.5
>
> Attachments: ak.patch
>
>
> I've started moving the GeoIP code to a plugin. The branch of Bro I'm working 
> from is here: 
> https://github.com/anthonykasza/bro/tree/topic/akasza/geoplugin. 
> The source for the plugin is here: https://github.com/anthonykasza/Bro_GeoIP.
> Any pointers would be appreciated.



--
This message was sent by Atlassian JIRA
(v6.4-OD-15-055#64014)
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] [JIRA] (BIT-1265) Single sided HTTP POST split

[Jon Siwek (JIRA)]

 [ 
https://bro-tracker.atlassian.net/browse/BIT-1265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jon Siwek updated BIT-1265:
---
Fix Version/s: 2.5

> Single sided HTTP POST split
> 
>
> Key: BIT-1265
> URL: https://bro-tracker.atlassian.net/browse/BIT-1265
> Project: Bro Issue Tracker
>  Issue Type: Problem
>  Components: Bro
>Affects Versions: git/master
> Environment: CentOS 6
>Reporter: Jimmy Jones
> Fix For: 2.5
>
> Attachments: sample-upload2-all.pcap, sample-upload2-req.pcap
>
>
> Attached two pcap samples, one is a single sided version of the other, an 
> HTTP POST.
> When I process the single sided version (sample-upload2-req) conn.log shows 
> two sessions (the HTTP POST tcp connection that has been split) and http.log 
> shows a partial upload. However processing the original sample 
> (sample-upload2-all) everything is as expected - one connection in conn.log 
> and a complete http.log
> Are there any parameters I can tweak to make this work?



--
This message was sent by Atlassian JIRA
(v6.4-OD-15-055#64014)
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] [JIRA] (BIT-1306) bro process would get stuck/freeze with myricom drivers

[Jon Siwek (JIRA)]

 [ 
https://bro-tracker.atlassian.net/browse/BIT-1306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jon Siwek updated BIT-1306:
---
Fix Version/s: 2.4

> bro process would get stuck/freeze with myricom drivers
> ---
>
> Key: BIT-1306
> URL: https://bro-tracker.atlassian.net/browse/BIT-1306
> Project: Bro Issue Tracker
>  Issue Type: Problem
>  Components: Bro
>Affects Versions: git/master
> Environment:  OS: FreeBSD 9.3-RELEASE-p5 OS
> bro version 2.3-328
> git log -1 --format="%H"
> 379593c7fded0f9791ae71a52dd78a4c9d5a2c1f
>Reporter: Aashish Sharma
>  Labels: bro-git, myricom
> Fix For: 2.4
>
>
> When I stop bro (in cluster mode), one of the bro worker process (random) 
> would get stuck and wouldn't shutdown, stop or even be killed using kill -s 
> 9. 
> System has to be ultimately rebooted to remove stuck bro process. 
> On running  myri_start_stop I see:
> # /usr/local/opt/snf/sbin/myri_start_stop stop
> Removing myri_snf.ko
> kldunload: can't unload file: Device busy
> It appears that the myri_snf.ko driver cannot be unloaded because of the 
> stuck bro process.  That process still has an open descriptor on the Sniffer 
> device/driver and bro process freezes 
> More details:
> The bro process is stuck in RNE state
> R   Marks a runnable process.
> N   The process has reduced CPU scheduling priority (see setpriority(2)).
> E   The process is trying to exit.
> Here is an example:
> ### stuck process:
> [bro@01 ~]$ ps auxwww | fgrep 1616
> bro1616  100.0  0.0 758040 60480 ??  RNE   2:57PM   53:50.04 
> /usr/local/bro-git/bin/bro -i myri0 -U .status -p broctl -p broctl-live -p 
> local -p worker-1-1 mgr.bro broctl base/frameworks/cluster local-worker.bro 
> broctl/auto
> when checking for process in proc:
> [bro@c ~]$ ls -l /proc/1616
> ls: /proc/1616: No such file or directory



--
This message was sent by Atlassian JIRA
(v6.4-OD-15-055#64014)
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] [JIRA] (BIT-788) Good analysis of unidirectional DNS flows

[Jon Siwek (JIRA)]

 [ 
https://bro-tracker.atlassian.net/browse/BIT-788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jon Siwek reassigned BIT-788:
-

Assignee: Jon Siwek

> Good analysis of unidirectional DNS flows
> -
>
> Key: BIT-788
> URL: https://bro-tracker.atlassian.net/browse/BIT-788
> Project: Bro Issue Tracker
>  Issue Type: Patch
>  Components: Bro
>Affects Versions: git/master
>Reporter: juliensentier
>Assignee: Jon Siwek
> Fix For: 2.4
>
> Attachments: 
> 0011-Good-analysis-of-unidirectional-answer-DNS-traffic-f.patch
>
>
> Some use port udp 53 as a source port for dns requests.
> And sometimes, we can miss the DNS request.
> In this case, we can rely on the DNS field QR to identify the direction of 
> the flow.



--
This message was sent by Atlassian JIRA
(v6.4-OD-15-055#64014)
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] [JIRA] (BIT-944) @bro-meta index in ES writer

[Jon Siwek (JIRA)]

[ 
https://bro-tracker.atlassian.net/browse/BIT-944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=20001#comment-20001
 ] 

Jon Siwek commented on BIT-944:
---

Vlad or Seth, up to you whether to re-schedule this ticket for 2.5.

> @bro-meta index in ES writer
> 
>
> Key: BIT-944
> URL: https://bro-tracker.atlassian.net/browse/BIT-944
> Project: Bro Issue Tracker
>  Issue Type: Problem
>  Components: Bro
>Affects Versions: git/master
>Reporter: Seth Hall
>Priority: Low
> Fix For: 2.4
>
>
> The elasticsearch writer isn't creating/modifying the required (for Brownian) 
> @bro-meta index when using the ReLog script to import old logs because 
> rotation is disabled when importing logs.  For now the right answer is to 
> probably just leave off out the start and end fields and write to the index 
> in the UpdateIndex method if rotation is disabled.



--
This message was sent by Atlassian JIRA
(v6.4-OD-15-055#64014)
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] [JIRA] (BIT-947) Incorrect size calculation for SSH failed/successful heuristic

[Jon Siwek (JIRA)]

[ 
https://bro-tracker.atlassian.net/browse/BIT-947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=2#comment-2
 ] 

Jon Siwek commented on BIT-947:
---

Vlad, any SSH changes you have coming that address this ticket?

> Incorrect size calculation for SSH failed/successful heuristic
> --
>
> Key: BIT-947
> URL: https://bro-tracker.atlassian.net/browse/BIT-947
> Project: Bro Issue Tracker
>  Issue Type: Problem
>  Components: Bro
>Affects Versions: git/master
>Reporter: grigorescu
>Priority: Low
> Fix For: 2.4
>
>
> We're getting a lot of false positives for successful SSH logins from a 
> source that we recently blackholed. I suspect what's happening is that the 
> retransmissions keep bumping up the size of the connection, until it crosses 
> the threshold for a "successful" connection. 
> With the changes from BIT-730: Find and fix tcp sequence counting bugs, is it 
> possible to improve the accuracy of the reported size?



--
This message was sent by Atlassian JIRA
(v6.4-OD-15-055#64014)
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] [JIRA] (BIT-1327) broctl status output is not sorted correctly

[Jon Siwek (JIRA)]

 [ 
https://bro-tracker.atlassian.net/browse/BIT-1327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jon Siwek updated BIT-1327:
---
Resolution: Fixed
Status: Closed  (was: Open)

Seems like this is addressed once BIT-1341 is merged.

> broctl status output is not sorted correctly
> 
>
> Key: BIT-1327
> URL: https://bro-tracker.atlassian.net/browse/BIT-1327
> Project: Bro Issue Tracker
>  Issue Type: Problem
>  Components: BroControl
>Affects Versions: git/master
>Reporter: Johanna Amann
> Fix For: 2.4
>
>
> With the current version of BroControl, broctl status is no longer sorted in 
> the traditional order that we had in old versions (master, proxy, workers). 
> Instead, the order seems to be more-or-less-random, but static (it does not 
> change inbetween runs).
> I think we should revert this to the old behavior - having sorted output is 
> nice and makes it more convenient to see what is going on.



--
This message was sent by Atlassian JIRA
(v6.4-OD-15-055#64014)
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] [JIRA] (BIT-1328) BroControl displays backtrace for all failed / mistyped commands

[Jon Siwek (JIRA)]

 [ 
https://bro-tracker.atlassian.net/browse/BIT-1328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jon Siwek updated BIT-1328:
---
Resolution: Fixed
Status: Closed  (was: Open)

Seems like this will be fixed when BIT-1341 is merged, please re-open if that's 
not the case.

> BroControl displays backtrace for all failed / mistyped commands
> 
>
> Key: BIT-1328
> URL: https://bro-tracker.atlassian.net/browse/BIT-1328
> Project: Bro Issue Tracker
>  Issue Type: Problem
>  Components: BroControl
>Affects Versions: git/master
>Reporter: Johanna Amann
> Fix For: 2.4
>
>
> BroControl shows a backtrace for failing commands, instead of just an error 
> message.
> Example:
> {code}
> [BroControl] > status sdd
> Traceback (most recent call last):
>   File "/xa/bro/master/lib/broctl/BroControl/brocmd.py", line 49, in cmdloop
> success = self.onecmd(line)
>   File "/usr/local/lib/python2.7/cmd.py", line 221, in onecmd
> return func(arg)
>   File "/xa/bro/master/bin/broctl", line 190, in do_status
> results = self.broctl.status(node_list=args)
>   File "/xa/bro/master/lib/broctl/BroControl/broctl.py", line 36, in wrapper
> return func(self, *args, **kwargs)
>   File "/xa/bro/master/lib/broctl/BroControl/broctl.py", line 231, in status
> nodes = self.node_args(node_list)
>   File "/xa/bro/master/lib/broctl/BroControl/broctl.py", line 98, in node_args
> raise InvalidNodeError("unknown node '%s'" % arg)
> InvalidNodeError: unknown node 'sdd'
> [BroControl] > 
> {code}



--
This message was sent by Atlassian JIRA
(v6.4-OD-15-055#64014)
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] [JIRA] (BIT-334) Portmapper.bro documentation and script interaction

[Jon Siwek (JIRA)]

 [ 
https://bro-tracker.atlassian.net/browse/BIT-334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jon Siwek updated BIT-334:
--
Fix Version/s: (was: 2.4)
   2.5

> Portmapper.bro documentation and script interaction
> ---
>
> Key: BIT-334
> URL: https://bro-tracker.atlassian.net/browse/BIT-334
> Project: Bro Issue Tracker
>  Issue Type: Problem
>  Components: Bro
>Affects Versions: git/master
>Reporter: gregor
> Fix For: 2.5
>
> Attachments: portmapper-logging.txt
>
>
> Hi, 
> just adding this ticket so the the information doesn't get lost. Notes on how 
> portmapper.bro does its logging and interaction with other scripts. Hopefully 
> helpful for the script documentation / cleanup push. 
> See also: http://bro.icir.org/devel/rpc-portmap-nfs-notes.html



--
This message was sent by Atlassian JIRA
(v6.4-OD-15-055#64014)
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] [JIRA] (BIT-327) Binding attributes to values/variables

[Jon Siwek (JIRA)]

 [ 
https://bro-tracker.atlassian.net/browse/BIT-327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jon Siwek updated BIT-327:
--
Fix Version/s: (was: 2.4)
   2.5

> Binding attributes to values/variables
> --
>
> Key: BIT-327
> URL: https://bro-tracker.atlassian.net/browse/BIT-327
> Project: Bro Issue Tracker
>  Issue Type: Problem
>  Components: Bro
>Affects Versions: git/master
>Reporter: Robin Sommer
> Fix For: 2.5
>
>
> From Vern:
> In abstract terms, we need to marry two notions: per-variable
> attributes (those introduced when defining the variable) and
> per-value attributes (those introduced when creating a value).
> These both exist under-the-hood, but the rules for propagating
> them are ad hoc.
> I'm attaching the follow-up email thread with further thoughts on
> streamlining this.
> Robin
> [^"None"]
> Did we ever reach resolution regarding the appended thread (from, um,
> a year ago\!), or at least put something in the Tracker so we don't lose
> sight of it?
>   Vern
> [^"None-1"]
> On Tue, Nov 03, 2009 at 17:25 \-0800, you wrote:
> > In abstract terms, we need to marry two notions: per-variable attributes
> > (those introduced when defining the variable) and per-value attributes
> > (those introduced when creating a value).  These both exist under-the-hood,
> > but the rules for propagating them are ad hoc.
> This is something I've wondered about a few times already what the
> right thing to do is. The keyword at the moment is indeed "ad-hoc":
> I remember that a number of times I've been running into problems
> with propagating (or not propagating) attributes, and while I was
> always able to fix the immediate problem in some way, we don't have
> a clear system at the moment when that happens and when not. 
> That said, I'm not really sure that this should ideally look like.
> Intuitively, I'd actually say attributes belong to values, not
> variables, because transfer-on-assignment can lead to subtle effects
> (values are passed around, and what if the receiving function
> happens to assign the value to the wrong variable?. Also what if you
> assign a value with attribute X to a variable without X; shouldn't
> the value then be *deleted* for consistency reasons?). 
> If we accept for a moment that attributes belong only to values,
> then we can think about how to set them. A global definition such as
>   const log_file = open_log_file("foo") &rotate_interval 
>
> can be interpreted as assigning the attribute to the value returned
> from the function (more generaly to whatever what the assigned
> expression yields). 
> We can use the "add foo &raw_output" syntax you suggested for adding
> attributes to the value of foo dynamically. 
> A declaration such as 
>   const foo = F &redef;
>   
> can be interpreted as "we can rebind foo if it's current value has
> the &redef attribute". 
> I haven't thought this through actually but I guess my question is
> whether we need per-variable attributes at all? 
> Robin
> [^"None-2"]
> On Nov 4, 2009, at 7:54 PM, Robin Sommer wrote:
> >That said, I'm not really sure that this should ideally look like.
> >Intuitively, I'd actually say attributes belong to values, not
> >variables, because transfer-on-assignment can lead to subtle effects
> >(values are passed around, and what if the receiving function
> >happens to assign the value to the wrong variable?. Also what if you
> >assign a value with attribute X to a variable without X; shouldn't
> >the value then be *deleted* for consistency reasons?).
> Attributes being attached to value really seems to make sense.
> >If we accept for a moment that attributes belong only to values,
> >then we can think about how to set them. A global definition such as
> >
> > const log_file = open_log_file("foo") &rotate_interval
> It works in this case, but this has typically been where trouble was 
> encountered.  What about cases where there isn't a value assigned yet?  
> Something like...
> const bad_addrs_with_description: table[addr] of string &redef 
> &write_expire=10mins;
> There isn't a value yet, but it has an attribute applied to it.  Would that 
> style still be supported?  It would seem to conflict with having only value 
> attributes.
> Even for my database backed variable stuff I'm working on, it created a 
> stumbling block.  What I'm doing internally is creating a copy of the value 
> including attributes to a separate internal value when a query is being run.  
> That value is then filled from the database and the script level variable is 
> rebound to my newly filled internal value and the old value is deleted.  I 
> think that would be the right way to do it in this case even if only value 
> attributes exist because it's an internal detail and the 

[Bro-Dev] [JIRA] (BIT-329) Optimizing detect-protocols-http.bro

[Jon Siwek (JIRA)]

 [ 
https://bro-tracker.atlassian.net/browse/BIT-329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jon Siwek updated BIT-329:
--
Fix Version/s: (was: 2.4)
   2.5

> Optimizing detect-protocols-http.bro
> 
>
> Key: BIT-329
> URL: https://bro-tracker.atlassian.net/browse/BIT-329
> Project: Bro Issue Tracker
>  Issue Type: Task
>  Components: Bro
>Affects Versions: git/master
>Reporter: Seth Hall
> Fix For: 2.5
>
>
> This script does a for loop over a 7 element table for every http_header and 
> http_request event.  In my opinion, I'd say that the benefit does not 
> outweigh the cost and it should be removed from the default local.bro scripts.



--
This message was sent by Atlassian JIRA
(v6.4-OD-15-055#64014)
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] [JIRA] (BIT-340) Cleanup: unify where global consts are defined (access from policy layer and event engine)

[Jon Siwek (JIRA)]

 [ 
https://bro-tracker.atlassian.net/browse/BIT-340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jon Siwek updated BIT-340:
--
Fix Version/s: (was: 2.4)
   2.5

> Cleanup: unify where global consts are defined (access from policy layer and 
> event engine)
> --
>
> Key: BIT-340
> URL: https://bro-tracker.atlassian.net/browse/BIT-340
> Project: Bro Issue Tracker
>  Issue Type: New Feature
>  Components: Bro
>Reporter: gregor
>Priority: Low
>  Labels: cleanup
> Fix For: 2.5
>
>
> {noformat}
> #!rst
> Global ``const``'s that are accessible from the policy layer and event engine 
> (e.g., to configure features) are currently defined in different ways: 
> 1. in ``bro.init`` and ``NetVar.{cc|h}``
> 2. in a specific .bro policy script and ``NetVar.{cc|h}``
> 3. in ``const.bif`` 
> According to our discussion on bro-dev, we should change it to only use 
> ``const.bif``. 
> For case 2. we should add a ``redef`` in the .bro policy scripts, so that 
> users looking at the script see that the const exists (TODO: how to best 
> auto-doucment these). 
> Setting milestone to 1.6 as it seems this can be done together with the 
> general policy script overhaul, but can also be pushed backed. 
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4-OD-15-055#64014)
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] [JIRA] (BIT-342) Add payload to ICMP analyzer

[Jon Siwek (JIRA)]

[ 
https://bro-tracker.atlassian.net/browse/BIT-342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=19997#comment-19997
 ] 

Jon Siwek commented on BIT-342:
---

I'll just add a new {{icmp_sent_payload}} event for this to address the 
overhead concern.

> Add payload to ICMP analyzer
> 
>
> Key: BIT-342
> URL: https://bro-tracker.atlassian.net/browse/BIT-342
> Project: Bro Issue Tracker
>  Issue Type: Patch
>  Components: Bro
>Affects Versions: 1.5.2
>Reporter: Seth Hall
>Assignee: Jon Siwek
> Fix For: 2.4
>
> Attachments: ICMP-add-payload.diff
>
>
> This is a patch from Julien Sentier on the mailing list that makes ICMP 
> payloads available at the scripting layer.  Is there a reason this isn't 
> already available?  I would have committed it to fastpath except I don't know 
> if it's not already doing this due to the potential overhead of creating a 
> lot of strings in ICMP floods.  At the very least, I suppose it could be 
> optional (which the patch doesn't currently do).



--
This message was sent by Atlassian JIRA
(v6.4-OD-15-055#64014)
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] [JIRA] (BIT-342) Add payload to ICMP analyzer

[Jon Siwek (JIRA)]

 [ 
https://bro-tracker.atlassian.net/browse/BIT-342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jon Siwek reassigned BIT-342:
-

Assignee: Jon Siwek

> Add payload to ICMP analyzer
> 
>
> Key: BIT-342
> URL: https://bro-tracker.atlassian.net/browse/BIT-342
> Project: Bro Issue Tracker
>  Issue Type: Patch
>  Components: Bro
>Affects Versions: 1.5.2
>Reporter: Seth Hall
>Assignee: Jon Siwek
> Fix For: 2.4
>
> Attachments: ICMP-add-payload.diff
>
>
> This is a patch from Julien Sentier on the mailing list that makes ICMP 
> payloads available at the scripting layer.  Is there a reason this isn't 
> already available?  I would have committed it to fastpath except I don't know 
> if it's not already doing this due to the potential overhead of creating a 
> lot of strings in ICMP floods.  At the very least, I suppose it could be 
> optional (which the patch doesn't currently do).



--
This message was sent by Atlassian JIRA
(v6.4-OD-15-055#64014)
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] [JIRA] (BIT-351) Incorrect bounds checking with truncated TCP options

[Jon Siwek (JIRA)]

 [ 
https://bro-tracker.atlassian.net/browse/BIT-351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jon Siwek updated BIT-351:
--
Fix Version/s: (was: 2.4)
   2.5

> Incorrect bounds checking with truncated TCP options
> 
>
> Key: BIT-351
> URL: https://bro-tracker.atlassian.net/browse/BIT-351
> Project: Bro Issue Tracker
>  Issue Type: Problem
>  Components: Bro
>Affects Versions: git/master
>Reporter: gregor
> Fix For: 2.5
>
>
> {noformat}
> #!rst
> (from an e-mail I sent a while ago)
> (setting milestone to 1.6. Can probably be pushed back)
> Hi,
> there is a potential problem in Bro when it receives a packet with
> truncated TCP options (i.e., the packet isn't long enough to accommodate
> all options).
> This can happen:
> a) in the ConnCompressor: it calls ParseTCPOptions without checking
>whether the packet is long enough to contain all options.
>ConnCompressor needs to parse the TCP options to know the window
>scaling factor.
> b) in TCP.cc, when caplen < len and len is long enough for the TCP
>options but caplen is not.
> ExtractTCPHeader() ensures that the len is long enough to contain the
> options and that caplen is long enough to contain a struct tcphdr (but
> doesn't check for options). Presumably this is done to enable parsing of
> header only traces that truncate options.
> Nevertheless, the TCP Analyzer correctly checks caplen before
> ParseTCPOptions().
> But there are also situations when options are parsed without checking
> for caplen:
> * BuildSYNVal(), which is called on every SYN to get the window
>   scaling options.
> * BuildOSVal(), which is only called when the OS_version_found event
>   has a handler
> * TCP TraceRewriter (presumably we can ignore this, as we were going
>   to remove it anyway)
> So, question is: what's the best way to tackle this? One option is to
> not parse packets that are truncated. But that's probably not a good
> idea wrt header traces.
> The other option is to check for the caplen whenever we parse options.
> That might be cumbersome as this information needs to be passed to many
> functions, e.g. in TCP_Analyzer: ProcessFlags -> ProcessSYN ->
> BuildSYNPacketVal.
> (In any case, truncated packets mean that we can't learn the window
> scaling)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4-OD-15-055#64014)
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] [JIRA] (BIT-423) Additional dynamic init time pattern construction

[Jon Siwek (JIRA)]

 [ 
https://bro-tracker.atlassian.net/browse/BIT-423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jon Siwek updated BIT-423:
--
Fix Version/s: (was: 2.4)
   2.5

> Additional dynamic init time pattern construction
> -
>
> Key: BIT-423
> URL: https://bro-tracker.atlassian.net/browse/BIT-423
> Project: Bro Issue Tracker
>  Issue Type: Problem
>  Components: Bro
>Reporter: Seth Hall
>  Labels: language
> Fix For: 2.5
>
>
> The pattern building BiF are now working, but I think that pattern 
> construction operators need to be supported with variables too.
> {noformat}
> const a = /abc/;
> const b = /def/;
> const c = a | b;
> {noformat}
> This doesn't seem to work currently.



--
This message was sent by Atlassian JIRA
(v6.4-OD-15-055#64014)
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] [JIRA] (BIT-410) Extension to init time pattern construction

[Jon Siwek (JIRA)]

 [ 
https://bro-tracker.atlassian.net/browse/BIT-410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jon Siwek updated BIT-410:
--
Fix Version/s: (was: 2.4)
   2.5

> Extension to init time pattern construction
> ---
>
> Key: BIT-410
> URL: https://bro-tracker.atlassian.net/browse/BIT-410
> Project: Bro Issue Tracker
>  Issue Type: New Feature
>  Components: Bro
>Affects Versions: git/master
>Reporter: Seth Hall
>  Labels: language
> Fix For: 2.5
>
>
> I'd like to be able to do this...
> {noformat}
> const pattern_a = /A/;
> const pattern_b = /B/;
> const pattern_ab = pattern_a | pattern_b;
> {noformat}
> This doesn't currently work.



--
This message was sent by Atlassian JIRA
(v6.4-OD-15-055#64014)
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] [JIRA] (BIT-423) Additional dynamic init time pattern construction

[Jon Siwek (JIRA)]

 [ 
https://bro-tracker.atlassian.net/browse/BIT-423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jon Siwek updated BIT-423:
--
   Resolution: Duplicate
Fix Version/s: (was: 2.5)
   Status: Closed  (was: Open)

Duplicate of BIT-410

> Additional dynamic init time pattern construction
> -
>
> Key: BIT-423
> URL: https://bro-tracker.atlassian.net/browse/BIT-423
> Project: Bro Issue Tracker
>  Issue Type: Problem
>  Components: Bro
>Reporter: Seth Hall
>  Labels: language
>
> The pattern building BiF are now working, but I think that pattern 
> construction operators need to be supported with variables too.
> {noformat}
> const a = /abc/;
> const b = /def/;
> const c = a | b;
> {noformat}
> This doesn't seem to work currently.



--
This message was sent by Atlassian JIRA
(v6.4-OD-15-055#64014)
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] [JIRA] (BIT-465) Fix up the MIME analyzer

[Jon Siwek (JIRA)]

 [ 
https://bro-tracker.atlassian.net/browse/BIT-465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jon Siwek updated BIT-465:
--
Fix Version/s: (was: 2.4)
   2.5

> Fix up the MIME analyzer
> 
>
> Key: BIT-465
> URL: https://bro-tracker.atlassian.net/browse/BIT-465
> Project: Bro Issue Tracker
>  Issue Type: Problem
>  Components: Bro
>Affects Versions: git/master
>Reporter: Seth Hall
>  Labels: analyzer
> Fix For: 2.5
>
>
> The mime analyzer has a lot of inconsistency issues and is broken in a few 
> places.
> * mime_all_headers loops and could potentially be a bad idea. More prone to 
> DoS as well.  Delete it?
> * mime_all_data is probably also a bad idea.  Especially for large files.  
> Delete it?
> * mime_entity_data seems very similar to mime_all_data and is not chunked as 
> the similarity to the http_entity_data would imply.  The current 
> mime_entity_data should be removed and the current mime_all_data should be 
> renamed to mime_entity_data.
> * mime_next_entity is never generated by the core or policy scripts and 
> should either be fixed or deleted.
> * mime_one_header should probably be renamed to mime_header for consistency.
> * I have no clue what mime_event is for.  Is it necessary?
> * mime_content_hash gives a non printable hash value and it could be removed 
> since hash generation is done in the script now and eventually will be done 
> in the file analyzer.
> * The wrong ifdef is used in the source: #ifdef DEBUG_BRO used instead of 
> #ifdef DEBUG
> * mime_end_entity is generated generated multiple times in some cases when it 
> shouldn't be.  It's something to keep an eye out for, I never dug into it 
> enough to find out what caused it.



--
This message was sent by Atlassian JIRA
(v6.4-OD-15-055#64014)
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] [JIRA] (BIT-465) Fix up the MIME analyzer

[Jon Siwek (JIRA)]

[ 
https://bro-tracker.atlassian.net/browse/BIT-465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=19995#comment-19995
 ] 

Jon Siwek commented on BIT-465:
---

Related to BIT-698 (maybe some duplicates, didn't check closely).

> Fix up the MIME analyzer
> 
>
> Key: BIT-465
> URL: https://bro-tracker.atlassian.net/browse/BIT-465
> Project: Bro Issue Tracker
>  Issue Type: Problem
>  Components: Bro
>Affects Versions: git/master
>Reporter: Seth Hall
>  Labels: analyzer
> Fix For: 2.5
>
>
> The mime analyzer has a lot of inconsistency issues and is broken in a few 
> places.
> * mime_all_headers loops and could potentially be a bad idea. More prone to 
> DoS as well.  Delete it?
> * mime_all_data is probably also a bad idea.  Especially for large files.  
> Delete it?
> * mime_entity_data seems very similar to mime_all_data and is not chunked as 
> the similarity to the http_entity_data would imply.  The current 
> mime_entity_data should be removed and the current mime_all_data should be 
> renamed to mime_entity_data.
> * mime_next_entity is never generated by the core or policy scripts and 
> should either be fixed or deleted.
> * mime_one_header should probably be renamed to mime_header for consistency.
> * I have no clue what mime_event is for.  Is it necessary?
> * mime_content_hash gives a non printable hash value and it could be removed 
> since hash generation is done in the script now and eventually will be done 
> in the file analyzer.
> * The wrong ifdef is used in the source: #ifdef DEBUG_BRO used instead of 
> #ifdef DEBUG
> * mime_end_entity is generated generated multiple times in some cases when it 
> shouldn't be.  It's something to keep an eye out for, I never dug into it 
> enough to find out what caused it.



--
This message was sent by Atlassian JIRA
(v6.4-OD-15-055#64014)
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] [JIRA] (BIT-678) Fix and test Bro's debugger

[Jon Siwek (JIRA)]

[ 
https://bro-tracker.atlassian.net/browse/BIT-678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=19994#comment-19994
 ] 

Jon Siwek commented on BIT-678:
---

I don't recall anything specific that's broken about the script debugger.  I 
did a couple minor improvements to it a couple years back, but was also 
generally able to use it at least for simple tasks.  Regardless, I guess the 
ticket should remain open w/ emphasis on creating regression tests for the 
debugger -- I don't think many people use it so would be easy at the moment to 
break it completely and not be aware of it.

> Fix and test Bro's debugger
> ---
>
> Key: BIT-678
> URL: https://bro-tracker.atlassian.net/browse/BIT-678
> Project: Bro Issue Tracker
>  Issue Type: Problem
>  Components: Bro
>Reporter: Seth Hall
> Fix For: 2.5
>
>
> Vern commented a while back that the debugger is currently broken in Bro.  
> Let's get it working again and tested.  If someone feels like fixing this up 
> for the 2.1 release we can certainly bump it forward but I'm going to target 
> it at 2.2 for now.



--
This message was sent by Atlassian JIRA
(v6.4-OD-15-055#64014)
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] [JIRA] (BIT-672) Bring POP3 back into the distribution

[Johanna Amann (JIRA)]

[ 
https://bro-tracker.atlassian.net/browse/BIT-672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=19993#comment-19993
 ] 

Johanna Amann commented on BIT-672:
---

As far as I remember, the general feeling was not to bring back pop3 by 
default, but to leave it to sites if they want to activate it by default, 
because the analyzer is of doubtable quality (it is one of the hand-written 
ones that does not use binpac at all).

So - I would just close the ticket.

> Bring POP3 back into the distribution
> -
>
> Key: BIT-672
> URL: https://bro-tracker.atlassian.net/browse/BIT-672
> Project: Bro Issue Tracker
>  Issue Type: Task
>  Components: Bro
>Affects Versions: git/master
>Reporter: Matthias Vallentin
>Assignee: Seth Hall
> Fix For: 2.5
>
>
> The current master has no longer support for POP3. It lingers around but we 
> need to bring it back into the distribution.



--
This message was sent by Atlassian JIRA
(v6.4-OD-15-055#64014)
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] [JIRA] (BIT-610) topic/seth/syslog-analyzer-updates - Updates for syslog analyzer

[Jon Siwek (JIRA)]

 [ 
https://bro-tracker.atlassian.net/browse/BIT-610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jon Siwek updated BIT-610:
--
Fix Version/s: (was: 2.4)
   2.5

> topic/seth/syslog-analyzer-updates - Updates for syslog analyzer
> 
>
> Key: BIT-610
> URL: https://bro-tracker.atlassian.net/browse/BIT-610
> Project: Bro Issue Tracker
>  Issue Type: Task
>  Components: Bro
>Reporter: Seth Hall
>  Labels: analyzer
> Fix For: 2.5
>
>
> \\- Supports "Octet Stuffing" mode for Syslog over TCP (untested\!).  If
>   someone has a tracefile with TCP syslog, I'd appreciate getting a
>   few packets.
> 
> \\- DPD support for syslog.  Calls ProtocolConfirmation when detected and 
>includes signatures for UDP and TCP syslog.
> 
> \\- Removing newlines and nulls from EOL when syslog implementation has
>   included those in the actual message.



--
This message was sent by Atlassian JIRA
(v6.4-OD-15-055#64014)
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] [JIRA] (BIT-1341) topic/dnthayer/fixes-for-2.4beta

[Robin Sommer (JIRA)]

 [ 
https://bro-tracker.atlassian.net/browse/BIT-1341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robin Sommer reassigned BIT-1341:
-

Assignee: Robin Sommer

> topic/dnthayer/fixes-for-2.4beta
> 
>
> Key: BIT-1341
> URL: https://bro-tracker.atlassian.net/browse/BIT-1341
> Project: Bro Issue Tracker
>  Issue Type: Problem
>  Components: BroControl
>Reporter: Daniel Thayer
>Assignee: Robin Sommer
> Fix For: 2.4
>
>
> Branch topic/dnthayer/fixes-for-2.4beta in the broctl repo addresses the 
> following issues:
> -Improved test setup scripts to specify correct bro install prefix.
> -Fix bug where "./configure --conf-files-dir" did not work
> -Fix bug where "./configure --scriptdir" did not work
> -Print error messages without showing Python stack trace
> -Improved processing of node input args, to remove duplicates and sort
> -Improved sorting of the output by node type and name
> -Added the "deploy" command
> -Update docs for the deploy command



--
This message was sent by Atlassian JIRA
(v6.4-OD-15-055#64014)
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] [JIRA] (BIT-519) policy/protocols/http/headers.bro only logs client headers

[Jon Siwek (JIRA)]

 [ 
https://bro-tracker.atlassian.net/browse/BIT-519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jon Siwek updated BIT-519:
--
Resolution: Fixed
Status: Closed  (was: Reopened)

This should be done (and working when BIT-1077 is merged).

> policy/protocols/http/headers.bro only logs client headers
> --
>
> Key: BIT-519
> URL: https://bro-tracker.atlassian.net/browse/BIT-519
> Project: Bro Issue Tracker
>  Issue Type: Problem
>  Components: Bro
>Reporter: Vern Paxson
>Assignee: Seth Hall
> Fix For: 2.4
>
>
> In Bro 1.5, policy/http-header.bro logs both client and server headers.  The 
> new http/headers.bro only logs client headers, which breaks some forms of 
> analysis.



--
This message was sent by Atlassian JIRA
(v6.4-OD-15-055#64014)
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] [JIRA] (BIT-1330) topic/python3-compat

[Robin Sommer (JIRA)]

 [ 
https://bro-tracker.atlassian.net/browse/BIT-1330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robin Sommer reassigned BIT-1330:
-

Assignee: Robin Sommer

> topic/python3-compat
> 
>
> Key: BIT-1330
> URL: https://bro-tracker.atlassian.net/browse/BIT-1330
> Project: Bro Issue Tracker
>  Issue Type: Improvement
>  Components: pysubnettree
>Reporter: Jon Siwek
>Assignee: Robin Sommer
> Fix For: 2.4
>
>
> Updates to pysubnettree for Python 3 compatibility: have to now consider that 
> bytes are a distinct type from strings and allow the API to accept either.



--
This message was sent by Atlassian JIRA
(v6.4-OD-15-055#64014)
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] [JIRA] (BIT-560) Child analyzer Init() problem

[Jon Siwek (JIRA)]

 [ 
https://bro-tracker.atlassian.net/browse/BIT-560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jon Siwek updated BIT-560:
--
Fix Version/s: (was: 2.4)
   2.5

> Child analyzer Init() problem
> -
>
> Key: BIT-560
> URL: https://bro-tracker.atlassian.net/browse/BIT-560
> Project: Bro Issue Tracker
>  Issue Type: Problem
>  Components: Bro
>Affects Versions: git/master
>Reporter: gregor
>Assignee: Robin Sommer
> Fix For: 2.5
>
>
> {noformat}
> #!rst
> I think there is an inherent problem in the way analyzers and child analyzers 
> are initialized. If analyzers are added by BuildInitialAnalyzerTree() they 
> are not initialized at first but in a batch by calling::
> 
> root->Init();
> root->InitChildren(); 
> If an analyzer wants to add a child in its Init(), the parent doesn't know 
> whether it needs to init this child or not. If the parent was added by 
> ``BuildInitialAnalyzerTree()``, it *must not* ``Init()`` the child, because 
> ``BuildInitialAnalyzerTree()`` will do it. OTOH, if the parent was added 
> dynamically, e.g., by DPD signatures, then it *must* ``Init()`` the child.
> What was the reason for ``BuildInitialAnalyzerTree()`` to defer 
> initialization of the tree until the end of the function?  Initializing when 
> they are added would solve the problem but I guess there was a good reason to 
> do it this way. 
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4-OD-15-055#64014)
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] [JIRA] (BIT-631) Special message for broctl locking when done by cron

[Jon Siwek (JIRA)]

[ 
https://bro-tracker.atlassian.net/browse/BIT-631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=19991#comment-19991
 ] 

Jon Siwek commented on BIT-631:
---

Daniel or Justin, any improvements made in this area that mean we can close the 
ticket?  If not, just re-schedule Fix Version for 2.5.

> Special message for broctl locking when done by cron
> 
>
> Key: BIT-631
> URL: https://bro-tracker.atlassian.net/browse/BIT-631
> Project: Bro Issue Tracker
>  Issue Type: New Feature
>  Components: BroControl
>Reporter: Seth Hall
> Fix For: 2.4
>
>
> If the broctl lock is being held by the cron command it would be nice if the 
> message that indicates a lock is already held would indicate if it is the 
> cron command.  If multiple people are working with broctl the person that 
> gets a lock doesn't know if it's because of another user or because they 
> happened to be trying to do something while the cron command is running.



--
This message was sent by Atlassian JIRA
(v6.4-OD-15-055#64014)
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] [JIRA] (BIT-634) CouchDB writer

[Jon Siwek (JIRA)]

 [ 
https://bro-tracker.atlassian.net/browse/BIT-634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jon Siwek updated BIT-634:
--
Fix Version/s: (was: 2.4)
   2.5

> CouchDB writer
> --
>
> Key: BIT-634
> URL: https://bro-tracker.atlassian.net/browse/BIT-634
> Project: Bro Issue Tracker
>  Issue Type: Patch
>  Components: Bro
>Reporter: jeff.baumes
> Fix For: 2.5
>
> Attachments: 0001-Adding-couchdb-writer.patch
>
>
> Attached is a git patch for logging information to CouchDB. It has a new 
> dependence on libcurl which it searches for with a find_package CMake 
> command, and JsonCpp (MIT license), whose code is included directly in the 
> source tree.



--
This message was sent by Atlassian JIRA
(v6.4-OD-15-055#64014)
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] [JIRA] (BIT-678) Fix and test Bro's debugger

[grigorescu (JIRA)]

[ 
https://bro-tracker.atlassian.net/browse/BIT-678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=19990#comment-19990
 ] 

grigorescu commented on BIT-678:


Do we know what the issues are? I was able to use the debugger seemingly just 
fine recently.

> Fix and test Bro's debugger
> ---
>
> Key: BIT-678
> URL: https://bro-tracker.atlassian.net/browse/BIT-678
> Project: Bro Issue Tracker
>  Issue Type: Problem
>  Components: Bro
>Reporter: Seth Hall
> Fix For: 2.5
>
>
> Vern commented a while back that the debugger is currently broken in Bro.  
> Let's get it working again and tested.  If someone feels like fixing this up 
> for the 2.1 release we can certainly bump it forward but I'm going to target 
> it at 2.2 for now.



--
This message was sent by Atlassian JIRA
(v6.4-OD-15-055#64014)
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] [JIRA] (BIT-1305) Consider marking some attributes as deprecated

[Robin Sommer (JIRA)]

[ 
https://bro-tracker.atlassian.net/browse/BIT-1305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=19988#comment-19988
 ] 

Robin Sommer commented on BIT-1305:
---

I'll remove &mergeable from the list, as that goes with &synchronized.

> Consider marking some attributes as deprecated
> --
>
> Key: BIT-1305
> URL: https://bro-tracker.atlassian.net/browse/BIT-1305
> Project: Bro Issue Tracker
>  Issue Type: Problem
>  Components: Bro
>Reporter: Jon Siwek
>Assignee: Robin Sommer
> Fix For: 2.4
>
>
> Likely candidates for deprecation:
> &rotate_interval
> &rotate_size
> &encrypt
> &mergeable
> &synchronize
> &persistent
> &group
> While the mechanism I added in BIT-757 can't be used to mark attributes as 
> deprecated, I'm thinking it's not difficult to just hard code the scanner to 
> emit a warning when encountering certain attributes.



--
This message was sent by Atlassian JIRA
(v6.4-OD-15-055#64014)
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] [JIRA] (BIT-646) Cleanup interpreter error handling.

[Jon Siwek (JIRA)]

 [ 
https://bro-tracker.atlassian.net/browse/BIT-646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jon Siwek updated BIT-646:
--
Resolution: Fixed
Status: Closed  (was: Open)

Not sure how useful keeping this open is, so closing, but feel free to re-open 
if anyone feels it needs a more thorough audit.

> Cleanup interpreter error handling.
> ---
>
> Key: BIT-646
> URL: https://bro-tracker.atlassian.net/browse/BIT-646
> Project: Bro Issue Tracker
>  Issue Type: Task
>  Components: Bro
>Reporter: Robin Sommer
>  Labels: language
> Fix For: 2.4
>
>
> From 15ab2874369b5d7a3e6a14df24b141fa75bb (which has been merged into 
> master):
> {noformat}
>  Currently, a lot of interpreter runtime errors, such as an access to
> an unset optional record field, cause Bro to abort with an internal
> error. This is an experimental branch that turns such errors into
> non-fatal runtime errors by internally raising exceptions. These are
> caught upstream and processing continues afterwards.
> 
> For now, not many errors actually raise exceptions (the example above
> does though). We'll need to go through them eventually and adapt the
> current Internal() calls (and potentially others). More generally, at
> some point we should cleanup the interpreter error handling (unifying
> errors reported at parse- and runtime; and switching to exceptions for
> all Expr/Stmt/Vals). But that's a larger change and left for later.
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4-OD-15-055#64014)
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] [JIRA] (BIT-671) Test Bro core and script layer simultaneously

[Jon Siwek (JIRA)]

 [ 
https://bro-tracker.atlassian.net/browse/BIT-671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jon Siwek updated BIT-671:
--
   Resolution: Rejected
Fix Version/s: (was: 2.4)
   Status: Closed  (was: Open)

I think this approach may complicate the unit test framework more than it's 
worth -- there's already a not-negligible maintenance cost of Bro's unit 
testing that will continue to grow as we add tests and I think they currently 
do well enough at detecting potential problems.

> Test Bro core and script layer simultaneously
> -
>
> Key: BIT-671
> URL: https://bro-tracker.atlassian.net/browse/BIT-671
> Project: Bro Issue Tracker
>  Issue Type: Problem
>  Components: BTest
>Affects Versions: git/master
>Reporter: Matthias Vallentin
>
> If we record all events during testing, say by adding {{events.bst}} to each 
> Bro run, we can simultaneously test the core. Moreover, we instantly know 
> whether a bug manifests at script land or at the core.



--
This message was sent by Atlassian JIRA
(v6.4-OD-15-055#64014)
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] [JIRA] (BIT-647) Extend HTTP analyzer to support multiply encoded content.

[Jon Siwek (JIRA)]

 [ 
https://bro-tracker.atlassian.net/browse/BIT-647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jon Siwek reassigned BIT-647:
-

Assignee: (was: Jon Siwek)

> Extend HTTP analyzer to support multiply encoded content.
> -
>
> Key: BIT-647
> URL: https://bro-tracker.atlassian.net/browse/BIT-647
> Project: Bro Issue Tracker
>  Issue Type: Problem
>  Components: Bro
>Reporter: Seth Hall
> Attachments: http-sdch-gzip.trace
>
>
> When Chrome and other SDCH supporting http clients request content from SDCH 
> compatible HTTP servers the response includes a header that looks like this:
> {noformat}
> Content-Encoding: sdch,gzip
> {noformat}
> Bro's HTTP analyzer doesn't currently do substring matches on the 
> content-encoding header so the resulting sdch/gzip content is identified as 
> gzip only.  Two things need to happen here:
> 1. Support substring matches on the content-encoding header to identify 
> that the content is gzip encoded.
> 2. Support some notion of the SDCH protocol.
> I think that point 1 should be done for the 2.0 release  but point 2 can wait 
> until later when we have a better notion of what SDCH support would entail.



--
This message was sent by Atlassian JIRA
(v6.4-OD-15-055#64014)
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] [JIRA] (BIT-647) Extend HTTP analyzer to support multiply encoded content.

[Jon Siwek (JIRA)]

 [ 
https://bro-tracker.atlassian.net/browse/BIT-647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jon Siwek updated BIT-647:
--
Fix Version/s: (was: 2.4)

> Extend HTTP analyzer to support multiply encoded content.
> -
>
> Key: BIT-647
> URL: https://bro-tracker.atlassian.net/browse/BIT-647
> Project: Bro Issue Tracker
>  Issue Type: Problem
>  Components: Bro
>Reporter: Seth Hall
> Attachments: http-sdch-gzip.trace
>
>
> When Chrome and other SDCH supporting http clients request content from SDCH 
> compatible HTTP servers the response includes a header that looks like this:
> {noformat}
> Content-Encoding: sdch,gzip
> {noformat}
> Bro's HTTP analyzer doesn't currently do substring matches on the 
> content-encoding header so the resulting sdch/gzip content is identified as 
> gzip only.  Two things need to happen here:
> 1. Support substring matches on the content-encoding header to identify 
> that the content is gzip encoded.
> 2. Support some notion of the SDCH protocol.
> I think that point 1 should be done for the 2.0 release  but point 2 can wait 
> until later when we have a better notion of what SDCH support would entail.



--
This message was sent by Atlassian JIRA
(v6.4-OD-15-055#64014)
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] [JIRA] (BIT-1077) fix policy/protocols/http/header-names.bro

[Robin Sommer (JIRA)]

 [ 
https://bro-tracker.atlassian.net/browse/BIT-1077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robin Sommer reassigned BIT-1077:
-

Assignee: Robin Sommer

> fix policy/protocols/http/header-names.bro
> --
>
> Key: BIT-1077
> URL: https://bro-tracker.atlassian.net/browse/BIT-1077
> Project: Bro Issue Tracker
>  Issue Type: Problem
>  Components: Bro
>Affects Versions: git/master
>Reporter: Jon Siwek
>Assignee: Robin Sommer
> Fix For: 2.4
>
>
> This script is wrong for the {{log_server_header_names}} case.



--
This message was sent by Atlassian JIRA
(v6.4-OD-15-055#64014)
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] [JIRA] (BIT-672) Bring POP3 back into the distribution

[Jon Siwek (JIRA)]

 [ 
https://bro-tracker.atlassian.net/browse/BIT-672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jon Siwek updated BIT-672:
--
Fix Version/s: (was: 2.4)
   2.5

> Bring POP3 back into the distribution
> -
>
> Key: BIT-672
> URL: https://bro-tracker.atlassian.net/browse/BIT-672
> Project: Bro Issue Tracker
>  Issue Type: Task
>  Components: Bro
>Affects Versions: git/master
>Reporter: Matthias Vallentin
>Assignee: Seth Hall
> Fix For: 2.5
>
>
> The current master has no longer support for POP3. It lingers around but we 
> need to bring it back into the distribution.



--
This message was sent by Atlassian JIRA
(v6.4-OD-15-055#64014)
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] [JIRA] (BIT-1332) Please merge topic/johanna/cert-validation

[Robin Sommer (JIRA)]

 [ 
https://bro-tracker.atlassian.net/browse/BIT-1332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robin Sommer reassigned BIT-1332:
-

Assignee: Robin Sommer

> Please merge topic/johanna/cert-validation
> --
>
> Key: BIT-1332
> URL: https://bro-tracker.atlassian.net/browse/BIT-1332
> Project: Bro Issue Tracker
>  Issue Type: Improvement
>  Components: Bro
>Affects Versions: git/master
>Reporter: Johanna Amann
>Assignee: Robin Sommer
> Fix For: 2.4
>
>
> Please merge topic/johanna/cert-validation. This is an update to the script 
> used to validate certificates in SSL/TLS connections. Description from main 
> commit:
> {quote}
> Update certificate validation script - new version will cache valid
> intermediate chains that it encounters on the wire and use those to try
> to validate chains that might be missing intermediate certificates.
> This vastly improves the number of certificates that Bro can validate.
> The only drawback is that now validation behavior is not entirely
> predictable anymore - the certificate of a server can fail to validate
> when Bro just started up (due to the intermediate missing), and succeed
> later, when the intermediate can be found in the cache.
> Has been tested on big-ish clusters and should not introduce any
> performance problems.
> {quote}



--
This message was sent by Atlassian JIRA
(v6.4-OD-15-055#64014)
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] [JIRA] (BIT-672) Bring POP3 back into the distribution

[Jon Siwek (JIRA)]

[ 
https://bro-tracker.atlassian.net/browse/BIT-672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=19986#comment-19986
 ] 

Jon Siwek commented on BIT-672:
---

What's the status here, the analyzer exists w/ a signature to activate it, but 
there's no scripts for it?  Is there plans to add scripts?  Close the ticket if 
not.

> Bring POP3 back into the distribution
> -
>
> Key: BIT-672
> URL: https://bro-tracker.atlassian.net/browse/BIT-672
> Project: Bro Issue Tracker
>  Issue Type: Task
>  Components: Bro
>Affects Versions: git/master
>Reporter: Matthias Vallentin
>Assignee: Seth Hall
> Fix For: 2.5
>
>
> The current master has no longer support for POP3. It lingers around but we 
> need to bring it back into the distribution.



--
This message was sent by Atlassian JIRA
(v6.4-OD-15-055#64014)
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] [JIRA] (BIT-683) Some BiFs should return a vector instead of a set/table

[Jon Siwek (JIRA)]

 [ 
https://bro-tracker.atlassian.net/browse/BIT-683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jon Siwek updated BIT-683:
--
Resolution: Fixed
Status: Closed  (was: Open)

The "split" family of functions now return vectors based at index 0, see 
BIT-757.

Don't quite understand the comment about changing the "find_all" function, now 
that "split_string" returns a vector, "find_all" returning a set is serving a 
different use-case -- the one where you care about inspecting membership 
instead of ordering.  Open another ticket if there's still something left to do 
regarding "find_all".

> Some BiFs should return a vector instead of a set/table
> ---
>
> Key: BIT-683
> URL: https://bro-tracker.atlassian.net/browse/BIT-683
> Project: Bro Issue Tracker
>  Issue Type: Problem
>  Components: Bro
>Affects Versions: git/master
>Reporter: Matthias Vallentin
>  Labels: language
> Fix For: 2.4
>
>
> The following functions should have a {{vector}} as yield value rather than a 
> set/table:
> {noformat}
> split.*(...)
> find_all(...)
> {noformat}
> Moreover, {{split}} & friends have the yield value {{table[count] of string}} 
> which starts at index 1. This is counterintuitive, as regular vectors start 
> with a index at 0. I suggest replacing the yield value with {{vector of 
> string}}.



--
This message was sent by Atlassian JIRA
(v6.4-OD-15-055#64014)
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] [JIRA] (BIT-678) Fix and test Bro's debugger

[Jon Siwek (JIRA)]

 [ 
https://bro-tracker.atlassian.net/browse/BIT-678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jon Siwek updated BIT-678:
--
Fix Version/s: (was: 2.4)
   2.5

> Fix and test Bro's debugger
> ---
>
> Key: BIT-678
> URL: https://bro-tracker.atlassian.net/browse/BIT-678
> Project: Bro Issue Tracker
>  Issue Type: Problem
>  Components: Bro
>Reporter: Seth Hall
> Fix For: 2.5
>
>
> Vern commented a while back that the debugger is currently broken in Bro.  
> Let's get it working again and tested.  If someone feels like fixing this up 
> for the 2.1 release we can certainly bump it forward but I'm going to target 
> it at 2.2 for now.



--
This message was sent by Atlassian JIRA
(v6.4-OD-15-055#64014)
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] [JIRA] (BIT-688) [Fwd] Re: content_gap vs. ack_above_hole

[Jon Siwek (JIRA)]

 [ 
https://bro-tracker.atlassian.net/browse/BIT-688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jon Siwek updated BIT-688:
--
Fix Version/s: (was: 2.4)
   2.5

> [Fwd] Re: [Bro-Dev] content_gap vs. ack_above_hole
> --
>
> Key: BIT-688
> URL: https://bro-tracker.atlassian.net/browse/BIT-688
> Project: Bro Issue Tracker
>  Issue Type: Task
>  Components: Bro
>Affects Versions: git/master
>Reporter: Robin Sommer
>  Labels: cleanup
> Fix For: 2.5
>
>
> From: Vern Paxson 
> Subject: Re: [Bro-Dev] content_gap vs. ack_above_hole 
> > Can somebody remind me what exactly the difference between these two
> > is (and/or why we have both?).
> Yeah, my fault :-P.  As best as I can tell (from revisiting the code),
> content-gap is a superset of ack-above-hole.  Content gaps can also occur
> in situations where we're not expecting to see ACKs (for example, due to
> split routing, or because we're not processing traffic from the receiver).
> I think merging the two into a single content_gap event would make sense.
>   Vern



--
This message was sent by Atlassian JIRA
(v6.4-OD-15-055#64014)
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] [JIRA] (BIT-698) HTTP vs MIME events

[Jon Siwek (JIRA)]

 [ 
https://bro-tracker.atlassian.net/browse/BIT-698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jon Siwek updated BIT-698:
--
Fix Version/s: (was: 2.4)
   2.5

> HTTP vs MIME events
> ---
>
> Key: BIT-698
> URL: https://bro-tracker.atlassian.net/browse/BIT-698
> Project: Bro Issue Tracker
>  Issue Type: Problem
>  Components: Bro
>Reporter: Robin Sommer
>  Labels: analyzer, cleanup
> Fix For: 2.5
>
>
> The way the {{mime_}} and {{http_entity_}} events are structure is a mess:
> - First, I think we should have only a single set of events for all MIME data.
> - the {{mime_}} events don't come with {{is_orig}}.
> - {{http_header}} vs. {{mime_one_header}}
> - {{http_entity_data}} delivers segments of size 
> {{http_entity_data_delivery_size}} while {{mime_entity_data}} delivers 
> complete entities (and {{mime_segment_data}} delivers segments…)
> - There are further inconsistencies I didn't record.



--
This message was sent by Atlassian JIRA
(v6.4-OD-15-055#64014)

___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] [JIRA] (BIT-697) Equivalent of capture-events.bro in 2.x

[Jon Siwek (JIRA)]

 [ 
https://bro-tracker.atlassian.net/browse/BIT-697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jon Siwek updated BIT-697:
--
Fix Version/s: (was: 2.4)

> Equivalent of capture-events.bro in 2.x
> ---
>
> Key: BIT-697
> URL: https://bro-tracker.atlassian.net/browse/BIT-697
> Project: Bro Issue Tracker
>  Issue Type: Problem
>  Components: Bro
>Affects Versions: git/master
>Reporter: Matthias Vallentin
>
> How should we handle the functionality provided by the 1.5 script 
> {{capture-events.bro}} in 2.x? It currently does not exist. Since it's 
> implementation only consists of this one-liner
> {noformat}
> event bro_init()
> {
> capture_events("events.bst");
> }
> {noformat}
> I think we make that a redefinable script variable rather than shipping a 
> separate script.



--
This message was sent by Atlassian JIRA
(v6.4-OD-15-055#64014)
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] [JIRA] (BIT-698) HTTP vs MIME events

[Jon Siwek (JIRA)]

 [ 
https://bro-tracker.atlassian.net/browse/BIT-698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jon Siwek updated BIT-698:
--
Description: 
The way the {{mime_}} and {{http_entity_}} events are structure is a mess:

- First, I think we should have only a single set of events for all MIME data.

- the {{mime_}} events don't come with {{is_orig}}.

- {{http_header}} vs. {{mime_one_header}}

- {{http_entity_data}} delivers segments of size 
{{http_entity_data_delivery_size}} while {{mime_entity_data}} delivers complete 
entities (and {{mime_segment_data}} delivers segments…)

- There are further inconsistencies I didn't record.

  was:
The way the `{{mime_*}} and {{http_entity_*}} events are structure is a mess:

\\- First, I think we should have only a single set of events for all MIME data.

\\- the }}{{mime_*}}{{ events don't come with }}{{is_orig}}{{.

\\- }}{{http_header}}{{ vs. }}{{mime_one_header}}{{

\\- }}{{http_entity_data}}{{ delivers segments of size 
}}{{http_entity_data_delivery_size}}{{ while }}{{mime_entity_data}}{{ delivers 
complete entities (and }}{{mime_segment_data}}` delivers segments…)

\\- There are further inconsistencies I didn't record.


> HTTP vs MIME events
> ---
>
> Key: BIT-698
> URL: https://bro-tracker.atlassian.net/browse/BIT-698
> Project: Bro Issue Tracker
>  Issue Type: Problem
>  Components: Bro
>Reporter: Robin Sommer
>  Labels: analyzer, cleanup
> Fix For: 2.5
>
>
> The way the {{mime_}} and {{http_entity_}} events are structure is a mess:
> - First, I think we should have only a single set of events for all MIME data.
> - the {{mime_}} events don't come with {{is_orig}}.
> - {{http_header}} vs. {{mime_one_header}}
> - {{http_entity_data}} delivers segments of size 
> {{http_entity_data_delivery_size}} while {{mime_entity_data}} delivers 
> complete entities (and {{mime_segment_data}} delivers segments…)
> - There are further inconsistencies I didn't record.



--
This message was sent by Atlassian JIRA
(v6.4-OD-15-055#64014)

___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] [JIRA] (BIT-698) HTTP vs MIME events

[Jon Siwek (JIRA)]

 [ 
https://bro-tracker.atlassian.net/browse/BIT-698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jon Siwek updated BIT-698:
--
Description: 
The way the `{{mime_*}} and {{http_entity_*}} events are structure is a mess:

\\- First, I think we should have only a single set of events for all MIME data.

\\- the }}{{mime_*}}{{ events don't come with }}{{is_orig}}{{.

\\- }}{{http_header}}{{ vs. }}{{mime_one_header}}{{

\\- }}{{http_entity_data}}{{ delivers segments of size 
}}{{http_entity_data_delivery_size}}{{ while }}{{mime_entity_data}}{{ delivers 
complete entities (and }}{{mime_segment_data}}` delivers segments…)

\\- There are further inconsistencies I didn't record.

  was:
The way the `{{mime_*}}{{ and }}{{http_entity_*}}{{ events are structure is a 
mess:

\\- First, I think we should have only a single set of events for all MIME data.

\\- the }}{{mime_*}}{{ events don't come with }}{{is_orig}}{{.

\\- }}{{http_header}}{{ vs. }}{{mime_one_header}}{{

\\- }}{{http_entity_data}}{{ delivers segments of size 
}}{{http_entity_data_delivery_size}}{{ while }}{{mime_entity_data}}{{ delivers 
complete entities (and }}{{mime_segment_data}}` delivers segments…)

\\- There are further inconsistencies I didn't record.


> HTTP vs MIME events
> ---
>
> Key: BIT-698
> URL: https://bro-tracker.atlassian.net/browse/BIT-698
> Project: Bro Issue Tracker
>  Issue Type: Problem
>  Components: Bro
>Reporter: Robin Sommer
>  Labels: analyzer, cleanup
> Fix For: 2.4
>
>
> The way the `{{mime_*}} and {{http_entity_*}} events are structure is a mess:
> \\- First, I think we should have only a single set of events for all MIME 
> data.
> \\- the }}{{mime_*}}{{ events don't come with }}{{is_orig}}{{.
> \\- }}{{http_header}}{{ vs. }}{{mime_one_header}}{{
> \\- }}{{http_entity_data}}{{ delivers segments of size 
> }}{{http_entity_data_delivery_size}}{{ while }}{{mime_entity_data}}{{ 
> delivers complete entities (and }}{{mime_segment_data}}` delivers segments…)
> \\- There are further inconsistencies I didn't record.



--
This message was sent by Atlassian JIRA
(v6.4-OD-15-055#64014)

___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] [JIRA] (BIT-735) Clean up and merge the TCPStats analyzer

[Jon Siwek (JIRA)]

 [ 
https://bro-tracker.atlassian.net/browse/BIT-735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jon Siwek updated BIT-735:
--
Fix Version/s: (was: 2.4)
   2.5

> Clean up and merge the TCPStats analyzer
> 
>
> Key: BIT-735
> URL: https://bro-tracker.atlassian.net/browse/BIT-735
> Project: Bro Issue Tracker
>  Issue Type: Task
>  Components: Bro
>Reporter: Seth Hall
> Fix For: 2.5
>
>
> Katrina wants to get her TCPStats analyzer merged.  Let's aim for getting it 
> cleaned up and ready for the 2.1 release.



--
This message was sent by Atlassian JIRA
(v6.4-OD-15-055#64014)
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] [JIRA] (BIT-742) Maintain constant order for hostname notice email extension

[Jon Siwek (JIRA)]

 [ 
https://bro-tracker.atlassian.net/browse/BIT-742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jon Siwek updated BIT-742:
--
Fix Version/s: (was: 2.4)
   2.5

> Maintain constant order for hostname notice email extension
> ---
>
> Key: BIT-742
> URL: https://bro-tracker.atlassian.net/browse/BIT-742
> Project: Bro Issue Tracker
>  Issue Type: Problem
>  Components: Bro
>Reporter: Seth Hall
> Fix For: 2.5
>
>
> The orig and resp field names will be ordered differently at times which is 
> confusing when reading emails.  Figure out a way to maintain constant 
> ordering.



--
This message was sent by Atlassian JIRA
(v6.4-OD-15-055#64014)
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] [JIRA] (BIT-748) Allow creation of blank patterns

[Jon Siwek (JIRA)]

 [ 
https://bro-tracker.atlassian.net/browse/BIT-748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jon Siwek updated BIT-748:
--
Fix Version/s: (was: 2.4)
   2.5

> Allow creation of blank patterns
> 
>
> Key: BIT-748
> URL: https://bro-tracker.atlassian.net/browse/BIT-748
> Project: Bro Issue Tracker
>  Issue Type: New Feature
>  Components: Bro
>Reporter: Seth Hall
> Fix For: 2.5
>
>
> Right now, it's not possible to create blank patterns of // but it would be 
> helpful in cases where patterns are used as configuration variables but there 
> is no default.



--
This message was sent by Atlassian JIRA
(v6.4-OD-15-055#64014)
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] [JIRA] (BIT-768) Inline monitoring of modified scripts.

[Jon Siwek (JIRA)]

 [ 
https://bro-tracker.atlassian.net/browse/BIT-768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jon Siwek updated BIT-768:
--
Resolution: Fixed
Status: Closed  (was: Open)

Don't think there's anything left to do here.

> Inline monitoring of modified scripts.
> --
>
> Key: BIT-768
> URL: https://bro-tracker.atlassian.net/browse/BIT-768
> Project: Bro Issue Tracker
>  Issue Type: Problem
>  Components: BroControl
>Affects Versions: git/master
>Reporter: Seth Hall
>Assignee: Daniel Thayer
> Fix For: 2.4
>
>
> We need to train users to do check, install, restart through broctl better.  
> I'd like to reduce the barrier to entry a bit more and if broctl can coach 
> new users through the process better and remind existing users of the process 
> it would be great.
> Here are my suggestions for what I think needs to be done:
> \\- Track hashes for all copied scripts (maybe in broctl.dat?) and watch for 
> changes to notify the user.  I think it would be ok to only notify the user 
> when they are in broctl but I can see that people may want that to also check 
> and occasionally email from broctl cron (let's save emailing for later 
> though, inline notification in broctl may be enough).
> \\- Track hashes for scripts that have been "checked" because then we can 
> coach people about what step in the process they are at.  If someone has 
> already run "check" on the current scripts we can recommend that they need to 
> \\- Create variables to turn off various suggestions.  I think the various 
> suggestions would be "need to check scripts", "need to install scripts", and 
> "ready to restart" or something along those lines.  I'm not even sure I like 
> this idea though.



--
This message was sent by Atlassian JIRA
(v6.4-OD-15-055#64014)
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] [JIRA] (BIT-788) Good analysis of unidirectional DNS flows

[Jon Siwek (JIRA)]

[ 
https://bro-tracker.atlassian.net/browse/BIT-788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=19983#comment-19983
 ] 

Jon Siwek commented on BIT-788:
---

Patch also seems fine to me, except in the condition where the analyzer is told 
to flip roles, maybe the values of "is_query" and "msg.is_query" should also 
flip.

> Good analysis of unidirectional DNS flows
> -
>
> Key: BIT-788
> URL: https://bro-tracker.atlassian.net/browse/BIT-788
> Project: Bro Issue Tracker
>  Issue Type: Patch
>  Components: Bro
>Affects Versions: git/master
>Reporter: juliensentier
> Fix For: 2.4
>
> Attachments: 
> 0011-Good-analysis-of-unidirectional-answer-DNS-traffic-f.patch
>
>
> Some use port udp 53 as a source port for dns requests.
> And sometimes, we can miss the DNS request.
> In this case, we can rely on the DNS field QR to identify the direction of 
> the flow.



--
This message was sent by Atlassian JIRA
(v6.4-OD-15-055#64014)
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] [JIRA] (BIT-1229) loading a non-existant enum from an input file terminates bro

[Johanna Amann (JIRA)]

[ 
https://bro-tracker.atlassian.net/browse/BIT-1229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=19982#comment-19982
 ] 

Johanna Amann commented on BIT-1229:


The fix does not fix the problem completely. I basically figured out after 
committing it that it fixes a specific problem, but not the underlying issue 
(which can still be triggered by a slight variation of the testcase) and that I 
have to do it a different and much more invasive way instead. Because of that, 
no merge request was ever filed.

> loading a non-existant enum from an input file terminates bro
> -
>
> Key: BIT-1229
> URL: https://bro-tracker.atlassian.net/browse/BIT-1229
> Project: Bro Issue Tracker
>  Issue Type: Problem
>  Components: Bro
>Affects Versions: git/master
>Reporter: Justin Azoff
>Assignee: Johanna Amann
> Fix For: 2.5
>
> Attachments: ignored_notices.csv, ignore-notices.bro
>
>
> If you have an input file with an enum in it and it does not exist, bro 
> terminates:
> internal error: Value not found in enum mappimg. Module: NoSuch, var: Notice, 
> var size: 6



--
This message was sent by Atlassian JIRA
(v6.4-OD-15-055#64014)
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev