[Bro-Dev] [JIRA] (BIT-1356) Bro process sticks around after broctl stop
[
https://bro-tracker.atlassian.net/browse/BIT-1356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robin Sommer updated BIT-1356:
--
Resolution: Fixed
Status: Closed (was: Open)
> Bro process sticks around after broctl stop
> ---
>
> Key: BIT-1356
> URL: https://bro-tracker.atlassian.net/browse/BIT-1356
> Project: Bro Issue Tracker
> Issue Type: Problem
> Components: BroControl
>Affects Versions: git/master
>Reporter: Johanna Amann
>Assignee: Daniel Thayer
> Fix For: 2.4
>
>
> It seems that after running a "broctl stop" not all bro processes are killed
> immediately. On our cluster, one of the processes keeps running; I seems like
> it eventually terminates after all log-compression is done. Is that on
> purpose or is that a bug?
> Ps output (on the node running the manager, bro process in first line,
> including the running compression jobs for completeness):
> {code}
> $ ps -ax | grep bro
> 23353 - IN 20:06.96 /xa/bro/master/bin/bro -U .status -p broctl -p
> broctl-live -p local -p manager local.bro broctl base/frameworks/cluster
> local-manager.bro broctl/auto
> 24979 - I 0:00.01 bash
> /xa/bro/master/share/broctl/scripts/archive-log http.2015-03-25-14-40-30.log
> http 15-03-25_14.40.30 15-03-25_16.29.29 1 ascii
> 25047 - I 0:00.01 bash
> /xa/bro/master/share/broctl/scripts/archive-log conn.2015-03-25-14-40-30.log
> conn 15-03-25_14.40.30 15-03-25_16.29.29 1 ascii
> 25841 - S 0:00.59 bash
> /xa/bro/master/share/broctl/scripts/post-terminate
> /xa/bro/master/spool/manager
> 29204 0 D+0:00.00 grep bro
> {code}
--
This message was sent by Atlassian JIRA
(v6.5-OD-01-120#65000)
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
[Bro-Dev] [JIRA] (BIT-1356) Bro process sticks around after broctl stop
[
https://bro-tracker.atlassian.net/browse/BIT-1356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=20263#comment-20263
]
Daniel Thayer commented on BIT-1356:
I've tested this, and 0620bc97 fixed the problem for me.
> Bro process sticks around after broctl stop
> ---
>
> Key: BIT-1356
> URL: https://bro-tracker.atlassian.net/browse/BIT-1356
> Project: Bro Issue Tracker
> Issue Type: Problem
> Components: BroControl
>Affects Versions: git/master
>Reporter: Johanna Amann
>Assignee: Daniel Thayer
> Fix For: 2.4
>
>
> It seems that after running a "broctl stop" not all bro processes are killed
> immediately. On our cluster, one of the processes keeps running; I seems like
> it eventually terminates after all log-compression is done. Is that on
> purpose or is that a bug?
> Ps output (on the node running the manager, bro process in first line,
> including the running compression jobs for completeness):
> {code}
> $ ps -ax | grep bro
> 23353 - IN 20:06.96 /xa/bro/master/bin/bro -U .status -p broctl -p
> broctl-live -p local -p manager local.bro broctl base/frameworks/cluster
> local-manager.bro broctl/auto
> 24979 - I 0:00.01 bash
> /xa/bro/master/share/broctl/scripts/archive-log http.2015-03-25-14-40-30.log
> http 15-03-25_14.40.30 15-03-25_16.29.29 1 ascii
> 25047 - I 0:00.01 bash
> /xa/bro/master/share/broctl/scripts/archive-log conn.2015-03-25-14-40-30.log
> conn 15-03-25_14.40.30 15-03-25_16.29.29 1 ascii
> 25841 - S 0:00.59 bash
> /xa/bro/master/share/broctl/scripts/post-terminate
> /xa/bro/master/spool/manager
> 29204 0 D+0:00.00 grep bro
> {code}
--
This message was sent by Atlassian JIRA
(v6.4-OD-16-006#64014)
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
[Bro-Dev] [JIRA] (BIT-1356) Bro process sticks around after broctl stop
[
https://bro-tracker.atlassian.net/browse/BIT-1356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=20258#comment-20258
]
Robin Sommer commented on BIT-1356:
---
Can somebody see if 0620bc97 helps?
> Bro process sticks around after broctl stop
> ---
>
> Key: BIT-1356
> URL: https://bro-tracker.atlassian.net/browse/BIT-1356
> Project: Bro Issue Tracker
> Issue Type: Problem
> Components: BroControl
>Affects Versions: git/master
>Reporter: Johanna Amann
>Assignee: Daniel Thayer
> Fix For: 2.4
>
>
> It seems that after running a "broctl stop" not all bro processes are killed
> immediately. On our cluster, one of the processes keeps running; I seems like
> it eventually terminates after all log-compression is done. Is that on
> purpose or is that a bug?
> Ps output (on the node running the manager, bro process in first line,
> including the running compression jobs for completeness):
> {code}
> $ ps -ax | grep bro
> 23353 - IN 20:06.96 /xa/bro/master/bin/bro -U .status -p broctl -p
> broctl-live -p local -p manager local.bro broctl base/frameworks/cluster
> local-manager.bro broctl/auto
> 24979 - I 0:00.01 bash
> /xa/bro/master/share/broctl/scripts/archive-log http.2015-03-25-14-40-30.log
> http 15-03-25_14.40.30 15-03-25_16.29.29 1 ascii
> 25047 - I 0:00.01 bash
> /xa/bro/master/share/broctl/scripts/archive-log conn.2015-03-25-14-40-30.log
> conn 15-03-25_14.40.30 15-03-25_16.29.29 1 ascii
> 25841 - S 0:00.59 bash
> /xa/bro/master/share/broctl/scripts/post-terminate
> /xa/bro/master/spool/manager
> 29204 0 D+0:00.00 grep bro
> {code}
--
This message was sent by Atlassian JIRA
(v6.4-OD-16-006#64014)
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
[Bro-Dev] [JIRA] (BIT-1356) Bro process sticks around after broctl stop
[
https://bro-tracker.atlassian.net/browse/BIT-1356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=20234#comment-20234
]
Justin Azoff commented on BIT-1356:
---
I wonder if that process is just left over from when bro calls system() to run
the child process...
I'm not sure what to do about this. killing that process is not the best idea,
but there may be a way to wait for it.
I think there is a larger issue here in that log rotation has a number of
problems:
* All logs get rotated+compressed at the same time, causing a CPU/IO Storm
* Logs are compressed on the fly to their destination, then the originals are
removed
* If compression is not in use, logs are copied and then removed (rather than
moved)
* If using something like the sftp handler and sftp fails, nothing is retried.
* Bro is the parent process to all of this.
* If bro crashes logs often end up in a crash directory rather than the proper
location.
I think that the only thing bro should be doing is atomically moving the
current logs to an archive directory or an archive staging directory. The
compression,moving,copying,uploading would be done by an external tool. There
are a number of benefits to this:
* If bro crashes recovering the logs is easy: on startup just move any existing
log files to the staging dir. A bro crash could never result in a partially
compressed/rotated log file
* Compression can be done serially or with limited parallelism rather than all
at once
* You could even delay the compression to idle periods
* Bugs like this would not occur since stopping bro would just require the logs
to be moved, not compressed
> Bro process sticks around after broctl stop
> ---
>
> Key: BIT-1356
> URL: https://bro-tracker.atlassian.net/browse/BIT-1356
> Project: Bro Issue Tracker
> Issue Type: Problem
> Components: BroControl
>Affects Versions: git/master
>Reporter: Johanna Amann
>Assignee: Daniel Thayer
> Fix For: 2.4
>
>
> It seems that after running a "broctl stop" not all bro processes are killed
> immediately. On our cluster, one of the processes keeps running; I seems like
> it eventually terminates after all log-compression is done. Is that on
> purpose or is that a bug?
> Ps output (on the node running the manager, bro process in first line,
> including the running compression jobs for completeness):
> {code}
> $ ps -ax | grep bro
> 23353 - IN 20:06.96 /xa/bro/master/bin/bro -U .status -p broctl -p
> broctl-live -p local -p manager local.bro broctl base/frameworks/cluster
> local-manager.bro broctl/auto
> 24979 - I 0:00.01 bash
> /xa/bro/master/share/broctl/scripts/archive-log http.2015-03-25-14-40-30.log
> http 15-03-25_14.40.30 15-03-25_16.29.29 1 ascii
> 25047 - I 0:00.01 bash
> /xa/bro/master/share/broctl/scripts/archive-log conn.2015-03-25-14-40-30.log
> conn 15-03-25_14.40.30 15-03-25_16.29.29 1 ascii
> 25841 - S 0:00.59 bash
> /xa/bro/master/share/broctl/scripts/post-terminate
> /xa/bro/master/spool/manager
> 29204 0 D+0:00.00 grep bro
> {code}
--
This message was sent by Atlassian JIRA
(v6.4-OD-16-006#64014)
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
[Bro-Dev] [JIRA] (BIT-1356) Bro process sticks around after broctl stop
[
https://bro-tracker.atlassian.net/browse/BIT-1356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adam Slagell reassigned BIT-1356:
-
Assignee: Daniel Thayer
> Bro process sticks around after broctl stop
> ---
>
> Key: BIT-1356
> URL: https://bro-tracker.atlassian.net/browse/BIT-1356
> Project: Bro Issue Tracker
> Issue Type: Problem
> Components: BroControl
>Affects Versions: git/master
>Reporter: Johanna Amann
>Assignee: Daniel Thayer
> Fix For: 2.4
>
>
> It seems that after running a "broctl stop" not all bro processes are killed
> immediately. On our cluster, one of the processes keeps running; I seems like
> it eventually terminates after all log-compression is done. Is that on
> purpose or is that a bug?
> Ps output (on the node running the manager, bro process in first line,
> including the running compression jobs for completeness):
> {code}
> $ ps -ax | grep bro
> 23353 - IN 20:06.96 /xa/bro/master/bin/bro -U .status -p broctl -p
> broctl-live -p local -p manager local.bro broctl base/frameworks/cluster
> local-manager.bro broctl/auto
> 24979 - I 0:00.01 bash
> /xa/bro/master/share/broctl/scripts/archive-log http.2015-03-25-14-40-30.log
> http 15-03-25_14.40.30 15-03-25_16.29.29 1 ascii
> 25047 - I 0:00.01 bash
> /xa/bro/master/share/broctl/scripts/archive-log conn.2015-03-25-14-40-30.log
> conn 15-03-25_14.40.30 15-03-25_16.29.29 1 ascii
> 25841 - S 0:00.59 bash
> /xa/bro/master/share/broctl/scripts/post-terminate
> /xa/bro/master/spool/manager
> 29204 0 D+0:00.00 grep bro
> {code}
--
This message was sent by Atlassian JIRA
(v6.4-OD-16-005#64014)
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
[Bro-Dev] [JIRA] (BIT-1356) Bro process sticks around after broctl stop
Johanna Amann created BIT-1356:
--
Summary: Bro process sticks around after broctl stop
Key: BIT-1356
URL: https://bro-tracker.atlassian.net/browse/BIT-1356
Project: Bro Issue Tracker
Issue Type: Problem
Components: BroControl
Affects Versions: git/master
Reporter: Johanna Amann
Fix For: 2.4
It seems that after running a "broctl stop" not all bro processes are killed
immediately. On our cluster, one of the processes keeps running; I seems like
it eventually terminates after all log-compression is done. Is that on purpose
or is that a bug?
Ps output (on the node running the manager, bro process in first line,
including the running compression jobs for completeness):
{code}
$ ps -ax | grep bro
23353 - IN 20:06.96 /xa/bro/master/bin/bro -U .status -p broctl -p
broctl-live -p local -p manager local.bro broctl base/frameworks/cluster
local-manager.bro broctl/auto
24979 - I 0:00.01 bash
/xa/bro/master/share/broctl/scripts/archive-log http.2015-03-25-14-40-30.log
http 15-03-25_14.40.30 15-03-25_16.29.29 1 ascii
25047 - I 0:00.01 bash
/xa/bro/master/share/broctl/scripts/archive-log conn.2015-03-25-14-40-30.log
conn 15-03-25_14.40.30 15-03-25_16.29.29 1 ascii
25841 - S 0:00.59 bash
/xa/bro/master/share/broctl/scripts/post-terminate /xa/bro/master/spool/manager
29204 0 D+0:00.00 grep bro
{code}
--
This message was sent by Atlassian JIRA
(v6.4-OD-16-005#64014)
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
