[Bro-Dev] [JIRA] (BIT-1103) Memory leak in Bro Intel framework
[
https://bro-tracker.atlassian.net/browse/BIT-1103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robin Sommer updated BIT-1103:
--
Resolution: Merged (was: Fixed)
Status: Closed (was: Merge Request)
Memory leak in Bro Intel framework
--
Key: BIT-1103
URL: https://bro-tracker.atlassian.net/browse/BIT-1103
Project: Bro Issue Tracker
Issue Type: Problem
Components: Bro
Affects Versions: 2.2
Environment: Red Hat Enterprise Linux Server release 6.5
Reporter: Andrew Hoying
Assignee: Bernhard Amann
Priority: High
Labels: intel, leak
The policy/frameworks/intel/seen bro scripts have a memory leak. On my
moderately busy Bro installation I am leaking about a gig of memory a day per
worker process with the Intel framework enabled. I can replicate by adding
the following to the local.bro default script and then running through a
small PCAP with primarily dns, dhcp and syslog traffic.
{{
@load policy/frameworks/intel/seen
redef Intel::read_files += {
/usr/local/bro/spool/domain_suspicious.txt,
};
}}
The intel file is in the following format, here's a few sample lines. It is
generated automatically by CIF:
{{
#fields indicator indicator_type meta.source meta.desc
meta.urlmeta.cif_impact meta.cif_severity meta.cif_confidence
mete-tools.biz Intel::DOMAIN CIF - need-to-know spammed domain
http://www.spamhaus.org/query/dbl?domain=mete-tools.biz (public)-
- 95
rttvxygkmwlqmq.net Intel::DOMAIN CIF - need-to-know spammed
domain http://www.spamhaus.org/query/dbl?domain=rttvxygkmwlqmq.net (public)
- - 95
podserveruho.comIntel::DOMAIN CIF - need-to-know spammed
domain http://www.spamhaus.org/query/dbl?domain=podserveruho.com (public)
- - 95
wwfcogdgntlxw.biz Intel::DOMAIN CIF - need-to-know spammed
domain http://www.spamhaus.org/query/dbl?domain=wwfcogdgntlxw.biz (public)
- - 95
}}
I compiled bro with gperftool debug support and followed the instructions
here: http://www.bro.org/development/howtos/leaks.html. (Note, the
instructions are wrong on the flags for ./configure, you need to add
--enable-perftools-debug to get the -m option for bro)
Here's the output from pprof top after running a PCAP trace with 10,000
packets. Running traces with more packets show a greater number of lost
objects in the same code locations.
{{
# pprof bin/bro /tmp/bro.24541.net_run-end.heap --inuse_objects --lines
--heapcheck --edgefraction=1e-10 --nodefraction=1e-10
Using local file bin/bro.
Using local file /tmp/bro.24541.net_run-end.heap.
Welcome to pprof! For help, type 'help'.
(pprof) top
Total: 4295 objects
2150 50.1% 50.1% 2150 50.1% AsciiFormatter::ParseValue
/usr/src/bro-2.2/src/threading/AsciiFormatter.cc:186
2141 49.8% 99.9% 2141 49.8% copy_string
/usr/src/bro-2.2/src/util.cc:155
2 0.0% 100.0%2 0.0% re_alloc
/usr/src/bro-2.2/build/src/re-scan.cc:2287
1 0.0% 100.0%1 0.0% RE_parse
/usr/src/bro-2.2/build/src/re-parse.y:110
1 0.0% 100.0%1 0.0% RE_parse
/usr/src/bro-2.2/build/src/re-parse.y:133
0 0.0% 100.0% 2141 49.8% AsciiFormatter::ParseValue
/usr/src/bro-2.2/src/threading/AsciiFormatter.cc:195
0 0.0% 100.0%4 0.1% Connection::NextPacket
/usr/src/bro-2.2/src/Conn.cc:259
0 0.0% 100.0%4 0.1% NetSessions::DispatchPacket
/usr/src/bro-2.2/src/Sessions.cc:189
0 0.0% 100.0%4 0.1% NetSessions::DoNextPacket
/usr/src/bro-2.2/src/Sessions.cc:709
0 0.0% 100.0%4 0.1% NetSessions::NextPacket
/usr/src/bro-2.2/src/Sessions.cc:247
}}
--
This message was sent by Atlassian JIRA
(v6.2-OD-03#6206)
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
[Bro-Dev] [JIRA] (BIT-1103) Memory leak in Bro Intel framework
[
https://bro-tracker.atlassian.net/browse/BIT-1103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Seth Hall updated BIT-1103:
---
Status: In Progress (was: Open)
Memory leak in Bro Intel framework
--
Key: BIT-1103
URL: https://bro-tracker.atlassian.net/browse/BIT-1103
Project: Bro Issue Tracker
Issue Type: Problem
Components: Bro
Affects Versions: 2.2
Environment: Red Hat Enterprise Linux Server release 6.5
Reporter: Andrew Hoying
Priority: High
Labels: intel, leak
The policy/frameworks/intel/seen bro scripts have a memory leak. On my
moderately busy Bro installation I am leaking about a gig of memory a day per
worker process with the Intel framework enabled. I can replicate by adding
the following to the local.bro default script and then running through a
small PCAP with primarily dns, dhcp and syslog traffic.
{{
@load policy/frameworks/intel/seen
redef Intel::read_files += {
/usr/local/bro/spool/domain_suspicious.txt,
};
}}
The intel file is in the following format, here's a few sample lines. It is
generated automatically by CIF:
{{
#fields indicator indicator_type meta.source meta.desc
meta.urlmeta.cif_impact meta.cif_severity meta.cif_confidence
mete-tools.biz Intel::DOMAIN CIF - need-to-know spammed domain
http://www.spamhaus.org/query/dbl?domain=mete-tools.biz (public)-
- 95
rttvxygkmwlqmq.net Intel::DOMAIN CIF - need-to-know spammed
domain http://www.spamhaus.org/query/dbl?domain=rttvxygkmwlqmq.net (public)
- - 95
podserveruho.comIntel::DOMAIN CIF - need-to-know spammed
domain http://www.spamhaus.org/query/dbl?domain=podserveruho.com (public)
- - 95
wwfcogdgntlxw.biz Intel::DOMAIN CIF - need-to-know spammed
domain http://www.spamhaus.org/query/dbl?domain=wwfcogdgntlxw.biz (public)
- - 95
}}
I compiled bro with gperftool debug support and followed the instructions
here: http://www.bro.org/development/howtos/leaks.html. (Note, the
instructions are wrong on the flags for ./configure, you need to add
--enable-perftools-debug to get the -m option for bro)
Here's the output from pprof top after running a PCAP trace with 10,000
packets. Running traces with more packets show a greater number of lost
objects in the same code locations.
{{
# pprof bin/bro /tmp/bro.24541.net_run-end.heap --inuse_objects --lines
--heapcheck --edgefraction=1e-10 --nodefraction=1e-10
Using local file bin/bro.
Using local file /tmp/bro.24541.net_run-end.heap.
Welcome to pprof! For help, type 'help'.
(pprof) top
Total: 4295 objects
2150 50.1% 50.1% 2150 50.1% AsciiFormatter::ParseValue
/usr/src/bro-2.2/src/threading/AsciiFormatter.cc:186
2141 49.8% 99.9% 2141 49.8% copy_string
/usr/src/bro-2.2/src/util.cc:155
2 0.0% 100.0%2 0.0% re_alloc
/usr/src/bro-2.2/build/src/re-scan.cc:2287
1 0.0% 100.0%1 0.0% RE_parse
/usr/src/bro-2.2/build/src/re-parse.y:110
1 0.0% 100.0%1 0.0% RE_parse
/usr/src/bro-2.2/build/src/re-parse.y:133
0 0.0% 100.0% 2141 49.8% AsciiFormatter::ParseValue
/usr/src/bro-2.2/src/threading/AsciiFormatter.cc:195
0 0.0% 100.0%4 0.1% Connection::NextPacket
/usr/src/bro-2.2/src/Conn.cc:259
0 0.0% 100.0%4 0.1% NetSessions::DispatchPacket
/usr/src/bro-2.2/src/Sessions.cc:189
0 0.0% 100.0%4 0.1% NetSessions::DoNextPacket
/usr/src/bro-2.2/src/Sessions.cc:709
0 0.0% 100.0%4 0.1% NetSessions::NextPacket
/usr/src/bro-2.2/src/Sessions.cc:247
}}
--
This message was sent by Atlassian JIRA
(v6.2-OD-03#6206)
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
[Bro-Dev] [JIRA] (BIT-1103) Memory leak in Bro Intel framework
[
https://bro-tracker.atlassian.net/browse/BIT-1103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Seth Hall reassigned BIT-1103:
--
Assignee: Bernhard Amann
Memory leak in Bro Intel framework
--
Key: BIT-1103
URL: https://bro-tracker.atlassian.net/browse/BIT-1103
Project: Bro Issue Tracker
Issue Type: Problem
Components: Bro
Affects Versions: 2.2
Environment: Red Hat Enterprise Linux Server release 6.5
Reporter: Andrew Hoying
Assignee: Bernhard Amann
Priority: High
Labels: intel, leak
The policy/frameworks/intel/seen bro scripts have a memory leak. On my
moderately busy Bro installation I am leaking about a gig of memory a day per
worker process with the Intel framework enabled. I can replicate by adding
the following to the local.bro default script and then running through a
small PCAP with primarily dns, dhcp and syslog traffic.
{{
@load policy/frameworks/intel/seen
redef Intel::read_files += {
/usr/local/bro/spool/domain_suspicious.txt,
};
}}
The intel file is in the following format, here's a few sample lines. It is
generated automatically by CIF:
{{
#fields indicator indicator_type meta.source meta.desc
meta.urlmeta.cif_impact meta.cif_severity meta.cif_confidence
mete-tools.biz Intel::DOMAIN CIF - need-to-know spammed domain
http://www.spamhaus.org/query/dbl?domain=mete-tools.biz (public)-
- 95
rttvxygkmwlqmq.net Intel::DOMAIN CIF - need-to-know spammed
domain http://www.spamhaus.org/query/dbl?domain=rttvxygkmwlqmq.net (public)
- - 95
podserveruho.comIntel::DOMAIN CIF - need-to-know spammed
domain http://www.spamhaus.org/query/dbl?domain=podserveruho.com (public)
- - 95
wwfcogdgntlxw.biz Intel::DOMAIN CIF - need-to-know spammed
domain http://www.spamhaus.org/query/dbl?domain=wwfcogdgntlxw.biz (public)
- - 95
}}
I compiled bro with gperftool debug support and followed the instructions
here: http://www.bro.org/development/howtos/leaks.html. (Note, the
instructions are wrong on the flags for ./configure, you need to add
--enable-perftools-debug to get the -m option for bro)
Here's the output from pprof top after running a PCAP trace with 10,000
packets. Running traces with more packets show a greater number of lost
objects in the same code locations.
{{
# pprof bin/bro /tmp/bro.24541.net_run-end.heap --inuse_objects --lines
--heapcheck --edgefraction=1e-10 --nodefraction=1e-10
Using local file bin/bro.
Using local file /tmp/bro.24541.net_run-end.heap.
Welcome to pprof! For help, type 'help'.
(pprof) top
Total: 4295 objects
2150 50.1% 50.1% 2150 50.1% AsciiFormatter::ParseValue
/usr/src/bro-2.2/src/threading/AsciiFormatter.cc:186
2141 49.8% 99.9% 2141 49.8% copy_string
/usr/src/bro-2.2/src/util.cc:155
2 0.0% 100.0%2 0.0% re_alloc
/usr/src/bro-2.2/build/src/re-scan.cc:2287
1 0.0% 100.0%1 0.0% RE_parse
/usr/src/bro-2.2/build/src/re-parse.y:110
1 0.0% 100.0%1 0.0% RE_parse
/usr/src/bro-2.2/build/src/re-parse.y:133
0 0.0% 100.0% 2141 49.8% AsciiFormatter::ParseValue
/usr/src/bro-2.2/src/threading/AsciiFormatter.cc:195
0 0.0% 100.0%4 0.1% Connection::NextPacket
/usr/src/bro-2.2/src/Conn.cc:259
0 0.0% 100.0%4 0.1% NetSessions::DispatchPacket
/usr/src/bro-2.2/src/Sessions.cc:189
0 0.0% 100.0%4 0.1% NetSessions::DoNextPacket
/usr/src/bro-2.2/src/Sessions.cc:709
0 0.0% 100.0%4 0.1% NetSessions::NextPacket
/usr/src/bro-2.2/src/Sessions.cc:247
}}
--
This message was sent by Atlassian JIRA
(v6.2-OD-03#6206)
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
