Trying this I noticed a few things (ordered by urgency from my point of
view).
With this change, we Bro cannot be compiled out of the Box on
RedHat/Centos 7 anymore. Since that is the latest release of RedHat and
probably used in production by quite a few people a potentially
significant amount of people might not be able to (easily) compile Bro
with this merge.
It aborts in configure, with:
-- Performing Test cxx11_header_works - Success
CMake Error at aux/broker/CMakeLists.txt:4 (cmake_minimum_required):
CMake 3.0.2 or higher is required. You are running version 2.8.12.2
--snip
Compiling on Debian 8 gives some CAF warnings that are a tad ugly:
In file included from
/root/bro/aux/broker/3rdparty/caf/libcaf_core/caf/serializer.hpp:32:0,
from
/root/bro/aux/broker/3rdparty/caf/libcaf_core/caf/detail/tuple_vals.hpp:25,
from
/root/bro/aux/broker/3rdparty/caf/libcaf_core/caf/make_message.hpp:28,
from
/root/bro/aux/broker/3rdparty/caf/libcaf_core/caf/mailbox_element.hpp:27,
from
/root/bro/aux/broker/3rdparty/caf/libcaf_core/caf/abstract_actor.hpp:37,
from
/root/bro/aux/broker/3rdparty/caf/libcaf_core/caf/actor.hpp:32,
from /root/bro/aux/broker/broker/data.hh:11,
from /root/bro/aux/broker/broker/broker.hh:8,
from /root/bro/src/broker/Data.h:4,
from /root/bro/src/broker/Data.cc:1:
/root/bro/aux/broker/3rdparty/caf/libcaf_core/caf/data_processor.hpp: In
function ‘typename std::enable_if<std::is_same<caf::error, decltype
(declval<caf::deserializer&>().caf::data_processor<caf::deserializer>::apply(declval<T&>()))>::value>::type
caf::operator&(caf::deserializer&, T&) [with T =
std::chrono::time_point<std::chrono::_V2::system_clock,
std::chrono::duration<long int, std::ratio<1l, 1000000000l> > >;
typename std::enable_if<std::is_same<caf::error, decltype
(declval<caf::deserializer&>().caf::data_processor<caf::deserializer>::apply(declval<T&>()))>::value>::type
= void]’:
/root/bro/aux/broker/3rdparty/caf/libcaf_core/caf/data_processor.hpp:478:7:
warning: ‘dur’ may be used uninitialized in this function
[-Wmaybe-uninitialized]
t = std::chrono::time_point<std::chrono::system_clock,
Duration>{dur};
^
/root/bro/aux/broker/3rdparty/caf/libcaf_core/caf/data_processor.hpp:476:16:
note: ‘dur’ was declared here
Duration dur;
^
--snip
/root/bro/aux/broker/3rdparty/caf/libcaf_core/src/scheduled_actor.cpp:892:55:
warning: unused parameter ‘sender’ [-Wunused-parameter]
actor_addr& sender,
--snip
I noticed one small thing while building with make -j4; in this case you
get several different % numbers simultaneously (one for car and one for
broker).
Example:
[ 25%] Built target plugin-Bro-BackDoor
[ 25%] Building CXX object
src/analyzer/protocol/bittorrent/CMakeFiles/plugin-Bro-BitTorrent.dir/bittorrent_pac.cc.o
[ 85%] Building CXX object
libcaf_io/CMakeFiles/libcaf_io_shared.dir/src/interfaces.cpp.o
[ 25%] Building CXX object
src/analyzer/protocol/bittorrent/CMakeFiles/plugin-Bro-BitTorrent.dir/events.bif.cc.o
While this is obviously cosmetic, it still looks weird to me :).
Apart from that it compiled and ran all tests on all systems I tried it
on.
There were a few test failures on the first run (that succeeded on a
rerun) though.
These were (from different systems):
MacOs:
[ 76%] scripts.base.frameworks.logging.field-extension-cluster ...
failed
[ 21%] broker.disconnect ... failed
[ 56%] broker.ssl_auth_failure ... failed
[ 89%] scripts.base.frameworks.control.shutdown ... failed
[ 99%] scripts.base.frameworks.openflow.log-cluster ... failed
There were also a couple that did not succeed after several reruns for
me. This was on a digital ocean 4cpu optimized debian8 instance for me;
the reruns were not parallel:
root@debian-c-4-8gib-sfo2-01:~/bro/testing/btest# ../../aux/btest/btest
-r -d
[ 0%] scripts.base.frameworks.control.configuration_update ... failed
% 'btest-bg-wait 10' failed unexpectedly (exit code 1)
% cat .stderr
The following processes did not terminate:
BROPATH=.:/root/bro/scripts:/root/bro/scripts/policy:/root/bro/scripts/site:/root/bro/build/scripts:..
bro
/root/bro/testing/btest/.tmp/scripts.base.frameworks.control.configuration_update/configuration_update.bro
frameworks/control/controller Control::host=127.0.0.1
Control::host_port=65531/tcp Control::cmd=shutdown
-----------
<<< [15700]
BROPATH=.:/root/bro/scripts:/root/bro/scripts/policy:/root/bro/scripts/site:/root/bro/build/scripts:..
bro
/root/bro/testing/btest/.tmp/scripts.base.frameworks.control.configuration_update/configuration_update.bro
frameworks/control/controllee Broker::default_port=65531/tcp
<params>, line 1: received termination signal
>>>
<<< [15738]
BROPATH=.:/root/bro/scripts:/root/bro/scripts/policy:/root/bro/scripts/site:/root/bro/build/scripts:..
bro
/root/bro/testing/btest/.tmp/scripts.base.frameworks.control.configuration_update/configuration_update.bro
test-redef frameworks/control/controller Control::host=127.0.0.1
Control::host_port=65531/tcp Control::cmd=configuration_update
/root/bro/scripts/policy/frameworks/control/controller.bro, line 136:
Control framework sent 330 IDs
<params>, line 1: received termination signal
>>>
<<< [15779]
BROPATH=.:/root/bro/scripts:/root/bro/scripts/policy:/root/bro/scripts/site:/root/bro/build/scripts:..
bro
/root/bro/testing/btest/.tmp/scripts.base.frameworks.control.configuration_update/configuration_update.bro
frameworks/control/controller Control::host=127.0.0.1
Control::host_port=65531/tcp Control::cmd=shutdown
<params>, line 1: received termination signal
>>>
[ 20%] scripts.base.frameworks.control.id_value ... failed
% 'btest-diff controller/.stdout' failed unexpectedly (exit code 1)
% cat .diag
== File ===============================
== Diff ===============================
--- /tmp/test-diff.15967.controller..stdout.baseline.tmp 2018-04-26
19:02:46.156000000 +0000
+++ /tmp/test-diff.15967.controller..stdout.tmp 2018-04-26
19:02:46.156000000 +0000
@@ -1 +0,0 @@
-Got an id_value_response(test_var, This is the value from the
controllee) event
=======================================
% cat .stderr
<<< [15865]
BROPATH=.:/root/bro/scripts:/root/bro/scripts/policy:/root/bro/scripts/site:/root/bro/build/scripts:..
bro
/root/bro/testing/btest/.tmp/scripts.base.frameworks.control.id_value/id_value.bro
only-for-controllee frameworks/control/controllee
Broker::default_port=65532/tcp
<params>, line 1: received termination signal
>>>
<<< [15893]
BROPATH=.:/root/bro/scripts:/root/bro/scripts/policy:/root/bro/scripts/site:/root/bro/build/scripts:..
bro
/root/bro/testing/btest/.tmp/scripts.base.frameworks.control.id_value/id_value.bro
frameworks/control/controller Control::host=127.0.0.1
Control::host_port=65532/tcp Control::cmd=id_value Control::arg=test_var
<params>, line 1: received termination signal
>>>
[ 60%] scripts.base.frameworks.intel.remove-item-cluster ... failed
% 'TEST_DIFF_CANONIFIER=$SCRIPTS/diff-sort btest-diff
worker-1/.stdout' failed unexpectedly (exit code 1)
% cat .diag
== File ===============================
Purging 192.168.0.1.
Purging 192.168.0.2.
Removing 192.168.1.2 (source: source1).
Removing 192.168.1.2 (source: source2).
== Diff ===============================
--- /tmp/test-diff.16234.worker-1..stdout.baseline.tmp 2018-04-26
19:02:50.396000000 +0000
+++ /tmp/test-diff.16234.worker-1..stdout.tmp 2018-04-26
19:02:50.404000000 +0000
@@ -1,6 +1,5 @@
### NOTE: This file has been sorted with diff-sort.
Purging 192.168.0.1.
Purging 192.168.0.2.
-Purging 192.168.1.2.
Removing 192.168.1.2 (source: source1).
Removing 192.168.1.2 (source: source2).
=======================================
% cat .stderr
<<< [16113]
BROPATH=.:/root/bro/scripts:/root/bro/scripts/policy:/root/bro/scripts/site:/root/bro/build/scripts:..
CLUSTER_NODE=manager-1 bro
/root/bro/testing/btest/.tmp/scripts.base.frameworks.intel.remove-item-cluster/remove-item-cluster.bro
received termination signal
>>>
<<< [16142]
BROPATH=.:/root/bro/scripts:/root/bro/scripts/policy:/root/bro/scripts/site:/root/bro/build/scripts:..
CLUSTER_NODE=worker-1 bro
/root/bro/testing/btest/.tmp/scripts.base.frameworks.intel.remove-item-cluster/remove-item-cluster.bro
received termination signal
>>>
[ 80%] scripts.base.frameworks.logging.field-extension-cluster ...
failed
% 'btest-diff manager-1/http.log' failed unexpectedly (exit code 1)
% cat .diag
== File ===============================
#separator \x09
#set_separator ,
#empty_field (empty)
#unset_field -
#path http
#open 2018-04-26-19-02-54
#fields _write_ts _stream _system_name ts uid
id_orig_h id_orig_p id_resp_h id_resp_p trans_depth
method host uri referrer version user_agent
request_body_len response_body_len status_code status_msg
info_code info_msg tags username password proxied
orig_fuids orig_filenames orig_mime_types resp_fuids resp_filenames
resp_mime_types
#types time string string time string addr port addr
port count string string string string string string count count
count string count string set[enum] string string set[string]
vector[string] vector[string] vector[string] vector[string] vector[string]
vector[string]
1524769374.222098 http worker-1 1524769374.162224
CUM0KZ3MLUfNB0cl11 141.142.220.118 48649 208.80.152.118 80 1
GET bits.wikimedia.org /skins-1.5/monobook/main.css
http://www.wikipedia.org/ 1.1 Mozilla/5.0
(X11; U; Linux x86_64; en-US; rv:1.9.2.15) Gecko/20110303 Ubuntu/10.04
(lucid) Firefox/3.6.15 0 0 304 Not
Modified - - (empty) - - - - -
- - - -
#close 2018-04-26-19-03-00
== Diff ===============================
--- /tmp/test-diff.16396.manager-1.http.log.baseline.tmp 2018-04-26
19:03:00.504000000 +0000
+++ /tmp/test-diff.16396.manager-1.http.log.tmp 2018-04-26
19:03:00.508000000 +0000
@@ -7,17 +7,4 @@
#fields _write_ts _stream _system_name ts uid
id_orig_h id_orig_p id_resp_h id_resp_p trans_depth
method host uri referrer version user_agent
request_body_len response_body_len status_code status_msg
info_code info_msg tags username password proxied
orig_fuids orig_filenames orig_mime_types resp_fuids resp_filenames
resp_mime_types
#types time string string time string addr port addr
port count string string string string string string count count
count string count string set[enum] string string set[string]
vector[string] vector[string] vector[string] vector[string] vector[string]
vector[string]
XXXXXXXXXX.XXXXXX http worker-1 XXXXXXXXXX.XXXXXX
CUM0KZ3MLUfNB0cl11 141.142.220.118 48649 208.80.152.118 80 1
GET bits.wikimedia.org /skins-1.5/monobook/main.css
http://www.wikipedia.org/ 1.1 Mozilla/5.0
(X11; U; Linux x86_64; en-US; rv:1.9.2.15) Gecko/20110303 Ubuntu/10.04
(lucid) Firefox/3.6.15 0 0 304 Not
Modified - - (empty) - - - - -
- - - -
-XXXXXXXXXX.XXXXXX http worker-1 XXXXXXXXXX.XXXXXX
CwjjYJ2WqgTbAqiHl6 141.142.220.118 49997 208.80.152.3 80 1
GET upload.wikimedia.org /wikipedia/commons/6/63/Wikipedia-logo.png
http://www.wikipedia.org/ 1.0 Mozilla/5.0
(X11; U; Linux x86_64; en-US; rv:1.9.2.15) Gecko/20110303 Ubuntu/10.04
(lucid) Firefox/3.6.15 0 0 304 Not
Modified - - (empty) - - - - -
- - - -
-XXXXXXXXXX.XXXXXX http worker-1 XXXXXXXXXX.XXXXXX
C3eiCBGOLw3VtHfOj 141.142.220.118 49996 208.80.152.3 80 1
GET upload.wikimedia.org
/wikipedia/commons/thumb/b/bb/Wikipedia_wordmark.svg/174px-Wikipedia_wordmark.svg.png
http://www.wikipedia.org/ 1.0 Mozilla/5.0
(X11; U; Linux x86_64; en-US; rv:1.9.2.15) Gecko/20110303 Ubuntu/10.04
(lucid) Firefox/3.6.15 0 0 304 Not
Modified - - (empty) - - - - -
- - - -
-XXXXXXXXXX.XXXXXX http worker-1 XXXXXXXXXX.XXXXXX
Ck51lg1bScffFj34Ri 141.142.220.118 49998 208.80.152.3 80 1
GET upload.wikimedia.org /wikipedia/commons/b/bd/Bookshelf-40x201_6.png
http://www.wikipedia.org/ 1.0 Mozilla/5.0
(X11; U; Linux x86_64; en-US; rv:1.9.2.15) Gecko/20110303 Ubuntu/10.04
(lucid) Firefox/3.6.15 0 0 304 Not
Modified - - (empty) - - - - -
- - - -
-XXXXXXXXXX.XXXXXX http worker-1 XXXXXXXXXX.XXXXXX
CtxTCR2Yer0FR1tIBg 141.142.220.118 50000 208.80.152.3 80 1
GET upload.wikimedia.org
/wikipedia/commons/thumb/8/8a/Wikinews-logo.png/35px-Wikinews-logo.png
http://www.wikipedia.org/ 1.0 Mozilla/5.0
(X11; U; Linux x86_64; en-US; rv:1.9.2.15) Gecko/20110303 Ubuntu/10.04
(lucid) Firefox/3.6.15 0 0 304 Not
Modified - - (empty) - - - - -
- - - -
-XXXXXXXXXX.XXXXXX http worker-1 XXXXXXXXXX.XXXXXX
CykQaM33ztNt0csB9a 141.142.220.118 49999 208.80.152.3 80 1
GET upload.wikimedia.org
/wikipedia/commons/4/4a/Wiktionary-logo-en-35px.png
http://www.wikipedia.org/ 1.0 Mozilla/5.0
(X11; U; Linux x86_64; en-US; rv:1.9.2.15) Gecko/20110303 Ubuntu/10.04
(lucid) Firefox/3.6.15 0 0 304 Not
Modified - - (empty) - - - - -
- - - -
-XXXXXXXXXX.XXXXXX http worker-1 XXXXXXXXXX.XXXXXX
CLNN1k2QMum1aexUK7 141.142.220.118 50001 208.80.152.3 80 1
GET upload.wikimedia.org
/wikipedia/commons/thumb/f/fa/Wikiquote-logo.svg/35px-Wikiquote-logo.svg.png
http://www.wikipedia.org/ 1.0 Mozilla/5.0
(X11; U; Linux x86_64; en-US; rv:1.9.2.15) Gecko/20110303 Ubuntu/10.04
(lucid) Firefox/3.6.15 0 0 304 Not
Modified - - (empty) - - - - -
- - - -
-XXXXXXXXXX.XXXXXX http worker-1 XXXXXXXXXX.XXXXXX
CiyBAq1bBLNaTiTAc 141.142.220.118 35642 208.80.152.2 80 1
GET meta.wikimedia.org /images/wikimedia-button.png
http://www.wikipedia.org/ 1.0 Mozilla/5.0
(X11; U; Linux x86_64; en-US; rv:1.9.2.15) Gecko/20110303 Ubuntu/10.04
(lucid) Firefox/3.6.15 0 0 304 Not
Modified - - (empty) - - - - -
- - - -
-XXXXXXXXXX.XXXXXX http worker-1 XXXXXXXXXX.XXXXXX
CwjjYJ2WqgTbAqiHl6 141.142.220.118 49997 208.80.152.3 80 2
GET upload.wikimedia.org
/wikipedia/commons/thumb/f/fa/Wikibooks-logo.svg/35px-Wikibooks-logo.svg.png
http://www.wikipedia.org/ 1.0 Mozilla/5.0
(X11; U; Linux x86_64; en-US; rv:1.9.2.15) Gecko/20110303 Ubuntu/10.04
(lucid) Firefox/3.6.15 0 0 304 Not
Modified - - (empty) - - - - -
- - - -
-XXXXXXXXXX.XXXXXX http worker-1 XXXXXXXXXX.XXXXXX
C3eiCBGOLw3VtHfOj 141.142.220.118 49996 208.80.152.3 80 2
GET upload.wikimedia.org
/wikipedia/commons/thumb/d/df/Wikispecies-logo.svg/35px-Wikispecies-logo.svg.png
http://www.wikipedia.org/ 1.0 Mozilla/5.0
(X11; U; Linux x86_64; en-US; rv:1.9.2.15) Gecko/20110303 Ubuntu/10.04
(lucid) Firefox/3.6.15 0 0 304 Not
Modified - - (empty) - - - - -
- - - -
-XXXXXXXXXX.XXXXXX http worker-1 XXXXXXXXXX.XXXXXX
Ck51lg1bScffFj34Ri 141.142.220.118 49998 208.80.152.3 80 2
GET upload.wikimedia.org
/wikipedia/commons/thumb/4/4c/Wikisource-logo.svg/35px-Wikisource-logo.svg.png
http://www.wikipedia.org/ 1.0 Mozilla/5.0
(X11; U; Linux x86_64; en-US; rv:1.9.2.15) Gecko/20110303 Ubuntu/10.04
(lucid) Firefox/3.6.15 0 0 304 Not
Modified - - (empty) - - - - -
- - - -
-XXXXXXXXXX.XXXXXX http worker-1 XXXXXXXXXX.XXXXXX
CtxTCR2Yer0FR1tIBg 141.142.220.118 50000 208.80.152.3 80 2
GET upload.wikimedia.org
/wikipedia/commons/thumb/4/4a/Commons-logo.svg/35px-Commons-logo.svg.png
http://www.wikipedia.org/ 1.0 Mozilla/5.0
(X11; U; Linux x86_64; en-US; rv:1.9.2.15) Gecko/20110303 Ubuntu/10.04
(lucid) Firefox/3.6.15 0 0 304 Not
Modified - - (empty) - - - - -
- - - -
-XXXXXXXXXX.XXXXXX http worker-1 XXXXXXXXXX.XXXXXX
CykQaM33ztNt0csB9a 141.142.220.118 49999 208.80.152.3 80 2
GET upload.wikimedia.org
/wikipedia/commons/thumb/9/91/Wikiversity-logo.svg/35px-Wikiversity-logo.svg.png
http://www.wikipedia.org/ 1.0 Mozilla/5.0
(X11; U; Linux x86_64; en-US; rv:1.9.2.15) Gecko/20110303 Ubuntu/10.04
(lucid) Firefox/3.6.15 0 0 304 Not
Modified - - (empty) - - - - -
- - - -
-XXXXXXXXXX.XXXXXX http worker-1 XXXXXXXXXX.XXXXXX
CLNN1k2QMum1aexUK7 141.142.220.118 50001 208.80.152.3 80 2
GET upload.wikimedia.org
/wikipedia/commons/thumb/7/75/Wikimedia_Community_Logo.svg/35px-Wikimedia_Community_Logo.svg.png
http://www.wikipedia.org/ 1.0 Mozilla/5.0
(X11; U; Linux x86_64; en-US; rv:1.9.2.15) Gecko/20110303 Ubuntu/10.04
(lucid) Firefox/3.6.15 0 0 304 Not
Modified - - (empty) - - - - -
- - - -
#close XXXX-XX-XX-XX-XX-XX
=======================================
% cat .stderr
<<< [16284] cp ../cluster-layout.bro . && CLUSTER_NODE=manager-1 bro
/root/bro/testing/btest/.tmp/scripts.base.frameworks.logging.field-extension-cluster/field-extension-cluster.bro
received termination signal
>>>
<<< [16325] cp ../cluster-layout.bro . && CLUSTER_NODE=worker-1 bro
--pseudo-realtime -C -r /root/bro/testing/btest/Traces/wikipedia.trace
/root/bro/testing/btest/.tmp/scripts.base.frameworks.logging.field-extension-cluster/field-extension-cluster.bro
>>>
Johanna
On 26 Apr 2018, at 8:16, Jon Siwek wrote:
> The latest version of the new Broker-ized cluster/communication system
> for Bro in 'topic/actor-system' branch is wrapping up and, in my
> opinion, ready to be merged into Bro's 'master' branch.
>
> However, since it's such a big change, I'd like a last round of
> feedback
> before merging. If you want to test, the build process should now be
> as
> simple as:
>
> $ git clone --recursive --branch=topic/actor-system
> git://git.bro.org/bro
> $ cd bro && ./configure && make
>
> Configuring BroControl is not any different from before.
>
> If you had custom scripts, they may require porting. There's a guide
> and examples for that at [1] and [2] (hyperlinks in those docs will
> render more nicely when it's up on bro.org).
>
> Though, for this round of testing, I'd be most interested just in any
> general stability issues or major feature breakages from a vanilla Bro
> installation. Mild performance issues, minor bugs, or other issues w/
> porting custom scripts are things I think we can iron out even after
> merging into 'master'.
>
> - Jon
>
> [1]
> https://github.com/bro/bro/blob/topic/actor-system/doc/frameworks/broker.rst
> [2]
> https://github.com/bro/bro/tree/topic/actor-system/doc/frameworks/broker
> _______________________________________________
> bro-dev mailing list
> [email protected]
> http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
_______________________________________________
bro-dev mailing list
[email protected]
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
