Re: [Bro-Dev] Dynamic plugin model (Re: [Bro-Commits] [git/bro] topic/robin/dynamic-plugins-2.3: Start of a plugin writing how-to. (87a1618))
On Thu, Dec 19, 2013 at 18:55 -0500, you wrote: What's the reason for supporting both static and dynamic plugin types? That's exactly what I haven't really made up my mind about yet. :) I think there's benefit to having a single Bro binary that comes with all the standard functionality. One piece is portability: dynamic linking may not be feasible/possible on some platforms (like tiny devices, or exotic OSs where our cmake setup may fail to do the right thing). And I generally like the notion of having just a single binary with all the standard code included; means less can go wrong (like version mismatches, etc.) In terms of performance, I wouldn't be too worried actually, although it's something that needs testing. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * ro...@icir.org ICSI/LBNL* Fax +1 (510) 666-2956 * www.icir.org/robin ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
Re: [Bro-Dev] Dynamic plugin model (Re: [Bro-Commits] [git/bro] topic/robin/dynamic-plugins-2.3: Start of a plugin writing how-to. (87a1618))
Hi:
Still running through the plugin generation stuff ... seemed to build the
binary distribution okay for me after I set BRO (FC 19, x86_64), but haven't
tried to really load / run anything yet.
Personally, I'm a fan of this kind of development model. I like not having to
worry about synchronizing with master to work on a plugin. I also like that
the compilation / link times are much shorter for plugin development than they
are when trying to add code to bro directly.
+1 Seth's comment that the init-plugin script is very nice.
A few thoughts:
* Would a section on testing be appropriate? Both btest and unit testing might
be useful for plugins.
* A short section explaining how / when to modify CMakeLists.txt might be
useful. I had to look at one of the existing analyzer plugins to double-check
that all CC files needed to go into bro_plugin_cc(). Not hard, but still might
be nice to explicitly document.
* Should plugins be allowed to link to additional libraries? If so, how? I
believe this could become an issue if I write a plugin that links against
libXYZ, but libXYZ isn't available on the system that's trying to load the
plugin.
Cheers,
Gilbert
From: bro-dev-boun...@bro.org [bro-dev-boun...@bro.org] On Behalf Of Robin
Sommer [ro...@icir.org]
Sent: Monday, December 16, 2013 3:34 PM
To: bro-dev@bro.org
Subject: [Bro-Dev] Dynamic plugin model (Re: [Bro-Commits] [git/bro]
topic/robin/dynamic-plugins-2.3: Start of a plugin writing how-to.
(87a1618))
I'd appreciate feedback on the model for dynamic plugins described
below.
The document contains a short intro on writing a simple plugin, as
well as some background on how/when Bro integrates plugins. The code
is in topic/robin/dynamic-plugins-2.3. It hasn't seen much testing yet
but the basic infrastructure seems to work on Linux and Darwin at
least.
Robin
On Mon, Dec 16, 2013 at 12:09 -0800, I wrote:
+===
+Writing Bro Plugins
+===
+
+Bro is internally moving to a plugin structure that enables extending
+the system dynamically, without modifying the core code base. That way
+custom code remains self-contained and can be maintained, compiled,
+and installed independently. Currently, plugins can add the following
+functionality to Bro:
+
+- Bro scripts.
+
+- Builtin functions/events/types for the scripting language.
+
+- Protocol and file analyzers.
+
+- Packet sources and packet dumpers. TODO: Not yet.
+
+- Logging framework backends. TODO: Not yet.
+
+- Input framework readers. TODO: Not yet.
+
+A plugin's functionality is available to the user just as if Bro had
+the corresponding code built-in. Indeed, internally many of Bro's
+pieces are structured as plugins as well, they are just statically
+compiled into the binary rather than loaded dynamically at runtime, as
+external plugins are.
+
+Quick Start
+===
+
+Writing a basic plugin is quite straight-forward as long as one
+follows a few conventions. In the following we walk through adding a
+new built-in function (bif) to Bro; we'll add `a `rot13(s: string) :
+string``, a function that rotates every character in a string by 13
+places.
+
+A plugin comes in the form of a directory following a certain
+structure. To get started, Bro's distribution provides a helper script
+``aux/bro-aux/plugin-support/init-plugin`` that creates a skeleton
+plugin that can then be customized. Let's use that::
+
+# mkdir rot13-plugin
+# cd rot13-plugin
+# init-plugin Demo Rot13
+
+As you can see the script takes two arguments. The first is a
+namespace the plugin will live in, and the second a descriptive name
+for the plugin itself. Bro uses the combination of the two to identify
+a plugin. The namespace serves to avoid naming conflicts between
+plugins written by independent developers; pick, e.g., the name of
+your organisation (and note that the namespace ``Bro`` is reserved for
+functionality distributed by the Bro Project). In our example, the
+plugin will be called ``Demo::Rot13``.
+
+The ``init-plugin`` script puts a number of files in place. The full
+layout is described later. For now, all we need is
+``src/functions.bif``. It's initially empty, but we'll add our new bif
+there as follows::
+
+# cat scripts/functions.bif
+module CaesarCipher;
+
+function camel_case%(s: string%) : string
+%{
+char* rot13 = copy_string(s-CheckString());
+
+for ( char* p = rot13; *p; p++ )
+{
+char b = islower(*p) ? 'a' : 'A';
+*p = (*p - b + 13) % 26 + b;
+}
+
+return new StringVal(strlen(rot13), rot13);
+%}
+
+The syntax of this file is just like any other ``*.bif`` file; we
+won't go into it here.
+
+Now we can already compile our plugin, we just need to tell the
+Makefile put in place by ``init
Re: [Bro-Dev] Dynamic plugin model (Re: [Bro-Commits] [git/bro] topic/robin/dynamic-plugins-2.3: Start of a plugin writing how-to. (87a1618))
On Wed, Dec 18, 2013 at 12:20 -0500, you wrote: I just build bro, cd into the build directory, source in the bro-path-dev.sh script and run Bro. Ah, I see. It's something else than I thought: a left-over from the earlier version that isn't needed anymore. Removed. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * ro...@icir.org ICSI/LBNL* Fax +1 (510) 666-2956 * www.icir.org/robin ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
Re: [Bro-Dev] Dynamic plugin model (Re: [Bro-Commits] [git/bro] topic/robin/dynamic-plugins-2.3: Start of a plugin writing how-to. (87a1618))
On Wed, Dec 18, 2013 at 21:07 -0500, you wrote: /tmp/bro/src/util.h:24:10: fatal error: 'magic.h' file not found I didn't consider Bro's CXX_FLAGS. I think I've fixed that, please try again. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * ro...@icir.org ICSI/LBNL* Fax +1 (510) 666-2956 * www.icir.org/robin ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
Re: [Bro-Dev] Dynamic plugin model (Re: [Bro-Commits] [git/bro] topic/robin/dynamic-plugins-2.3: Start of a plugin writing how-to. (87a1618))
On Thu, Dec 19, 2013 at 10:23 -0500, you wrote: * Would a section on testing be appropriate? Both btest and unit testing might be useful for plugins. Ack, that's a good point, the init-plugin script could put a basic setup in place for that, maybe even with a first test making sure things compile. * A short section explaining how / when to modify CMakeLists.txt might be useful. Yeah, likewise agreed. Indeed tthe documentation needs quite a bit more material to get people actually started without having to browse a ton of other code first. I'll leave that for later though once we've fleshed this all fully out. * Should plugins be allowed to link to additional libraries? Yes, definitly. My thinking is that the plugin author will extend the CMakeIndex.txt with the corresponding pieces, including compile-time logic to figure out if it's available. However, if the binary module aims to link against a lib that's not available at runtime where Bro executes, then I don't think there's much more we can do than fail loading the plugin: the dlopen will fail (iirc, Bro currently aborts in that case, I'm not sure if it should proceed without?) Thanks for the feedback. From chatting with Seth the other day, I took two more suggestions away: - I'm coming around that the BRO_PLUGIN_* macros aren't the best way of doing things. My main motivation for using them was hiding implementation details of the plugin API so that we can more easily change things without breaking existing code. However, it seems they are putting too much constraints on the plugin writer and/or, if one needs to get around them, require a lot of digging into the internals. So I'm mulling over creating a (simpler) C++ API to the Plugin class that can be used directly. - The static and dynamic plugins could be unified further. It's unclear what the right default is for shipping plugins that provide standard functionality, but it would be nice in any case if we could just flip a switch to change between static and dynamic builds for the in-tree stuff. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * ro...@icir.org ICSI/LBNL* Fax +1 (510) 666-2956 * www.icir.org/robin ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
Re: [Bro-Dev] Dynamic plugin model (Re: [Bro-Commits] [git/bro] topic/robin/dynamic-plugins-2.3: Start of a plugin writing how-to. (87a1618))
The static and dynamic plugins could be unified further. It's unclear what the right default is for shipping plugins that provide standard functionality, but it would be nice in any case if we could just flip a switch to change between static and dynamic builds for the in-tree stuff. What's the reason for supporting both static and dynamic plugin types? Assuming that eliminating the static plugin type entirely would simplify the build / link process, and that the static plugin type exists largely due to performance concerns: if we could prove that dynamic linking didn't have a significant impact on performance, could we use that as rationale for eliminating static plugin linkage entirely? Cheers, Gilbert ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
Re: [Bro-Dev] Dynamic plugin model (Re: [Bro-Commits] [git/bro] topic/robin/dynamic-plugins-2.3: Start of a plugin writing how-to. (87a1618))
On Dec 16, 2013, at 3:34 PM, Robin Sommer ro...@icir.org wrote: I'd appreciate feedback on the model for dynamic plugins described below. I'm getting this error: error in ./plugins, line 1: read failed with Is a directory It looks like the build/ directory ends up with a directory named plugins in it which doesn't play well for when I run Bro in the build directory. :) Anyway, easy to deal with since I can get rid of the plugins directory but maybe something to consider. Maybe just drop a __load__.bro file in there by default? .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ signature.asc Description: Message signed with OpenPGP using GPGMail ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
Re: [Bro-Dev] Dynamic plugin model (Re: [Bro-Commits] [git/bro] topic/robin/dynamic-plugins-2.3: Start of a plugin writing how-to. (87a1618))
On Wed, Dec 18, 2013 at 08:10 -0500, you wrote: error in ./plugins, line 1: read failed with Is a directory Doh. :) Not sure how to reproducee though. How exactly are you running it? Are you setting BRO_PLUGIN_PATH, and if so, how? Robin -- Robin Sommer * Phone +1 (510) 722-6541 * ro...@icir.org ICSI/LBNL* Fax +1 (510) 666-2956 * www.icir.org/robin ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
Re: [Bro-Dev] Dynamic plugin model (Re: [Bro-Commits] [git/bro] topic/robin/dynamic-plugins-2.3: Start of a plugin writing how-to. (87a1618))
On Dec 18, 2013, at 10:53 AM, Robin Sommer ro...@icir.org wrote: On Wed, Dec 18, 2013 at 08:10 -0500, you wrote: error in ./plugins, line 1: read failed with Is a directory Doh. :) Not sure how to reproducee though. How exactly are you running it? Are you setting BRO_PLUGIN_PATH, and if so, how? I just build bro, cd into the build directory, source in the bro-path-dev.sh script and run Bro. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ signature.asc Description: Message signed with OpenPGP using GPGMail ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
Re: [Bro-Dev] Dynamic plugin model (Re: [Bro-Commits] [git/bro] topic/robin/dynamic-plugins-2.3: Start of a plugin writing how-to. (87a1618))
On Dec 16, 2013, at 3:34 PM, Robin Sommer ro...@icir.org wrote: I'd appreciate feedback on the model for dynamic plugins described below. Couple more observations… - The init-plugin script is neat. :) - I got the following error when I tried to build my plugin... [ 40%] Building CXX object CMakeFiles/Bro-LanguageDetect.darwin-x86_64.dir/src/Plugin.cc.o In file included from /tmp/detect-language/src/Plugin.cc:2: In file included from /tmp/bro/src/plugin/Plugin.h:9: In file included from /tmp/bro/src/plugin/Macros.h:11: In file included from /tmp/bro/src/analyzer/Component.h:6: In file included from /tmp/bro/src/analyzer/Tag.h:7: /tmp/bro/src/util.h:24:10: fatal error: 'magic.h' file not found #include magic.h ^ 1 error generated. make[3]: *** [CMakeFiles/Bro-LanguageDetect.darwin-x86_64.dir/src/Plugin.cc.o] Error 1 make[2]: *** [CMakeFiles/Bro-LanguageDetect.darwin-x86_64.dir/all] Error 2 make[1]: *** [all] Error 2 make: *** [build-it] Error 2 .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ signature.asc Description: Message signed with OpenPGP using GPGMail ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
[Bro-Dev] Dynamic plugin model (Re: [Bro-Commits] [git/bro] topic/robin/dynamic-plugins-2.3: Start of a plugin writing how-to. (87a1618))
I'd appreciate feedback on the model for dynamic plugins described
below.
The document contains a short intro on writing a simple plugin, as
well as some background on how/when Bro integrates plugins. The code
is in topic/robin/dynamic-plugins-2.3. It hasn't seen much testing yet
but the basic infrastructure seems to work on Linux and Darwin at
least.
Robin
On Mon, Dec 16, 2013 at 12:09 -0800, I wrote:
+===
+Writing Bro Plugins
+===
+
+Bro is internally moving to a plugin structure that enables extending
+the system dynamically, without modifying the core code base. That way
+custom code remains self-contained and can be maintained, compiled,
+and installed independently. Currently, plugins can add the following
+functionality to Bro:
+
+- Bro scripts.
+
+- Builtin functions/events/types for the scripting language.
+
+- Protocol and file analyzers.
+
+- Packet sources and packet dumpers. TODO: Not yet.
+
+- Logging framework backends. TODO: Not yet.
+
+- Input framework readers. TODO: Not yet.
+
+A plugin's functionality is available to the user just as if Bro had
+the corresponding code built-in. Indeed, internally many of Bro's
+pieces are structured as plugins as well, they are just statically
+compiled into the binary rather than loaded dynamically at runtime, as
+external plugins are.
+
+Quick Start
+===
+
+Writing a basic plugin is quite straight-forward as long as one
+follows a few conventions. In the following we walk through adding a
+new built-in function (bif) to Bro; we'll add `a `rot13(s: string) :
+string``, a function that rotates every character in a string by 13
+places.
+
+A plugin comes in the form of a directory following a certain
+structure. To get started, Bro's distribution provides a helper script
+``aux/bro-aux/plugin-support/init-plugin`` that creates a skeleton
+plugin that can then be customized. Let's use that::
+
+# mkdir rot13-plugin
+# cd rot13-plugin
+# init-plugin Demo Rot13
+
+As you can see the script takes two arguments. The first is a
+namespace the plugin will live in, and the second a descriptive name
+for the plugin itself. Bro uses the combination of the two to identify
+a plugin. The namespace serves to avoid naming conflicts between
+plugins written by independent developers; pick, e.g., the name of
+your organisation (and note that the namespace ``Bro`` is reserved for
+functionality distributed by the Bro Project). In our example, the
+plugin will be called ``Demo::Rot13``.
+
+The ``init-plugin`` script puts a number of files in place. The full
+layout is described later. For now, all we need is
+``src/functions.bif``. It's initially empty, but we'll add our new bif
+there as follows::
+
+# cat scripts/functions.bif
+module CaesarCipher;
+
+function camel_case%(s: string%) : string
+%{
+char* rot13 = copy_string(s-CheckString());
+
+for ( char* p = rot13; *p; p++ )
+{
+char b = islower(*p) ? 'a' : 'A';
+*p = (*p - b + 13) % 26 + b;
+}
+
+return new StringVal(strlen(rot13), rot13);
+%}
+
+The syntax of this file is just like any other ``*.bif`` file; we
+won't go into it here.
+
+Now we can already compile our plugin, we just need to tell the
+Makefile put in place by ``init-plugin`` where the Bro source tree is
+located (Bro needs to have been built there first)::
+
+# make BRO=/path/to/bro/dist
+[... cmake output ...]
+
+Now our ``rot13-plugin`` directory has everything that it needs
+for Bro to recognize it as a dynamic plugin. Once we point Bro to it,
+it will pull it in automatically, as we can check with the ``-N``
+option:
+
+# export BRO_PLUGIN_PATH=/path/to/rot13-plugin
+# bro -N
+[...]
+Plugin: Demo::Rot13 - Insert brief description of plugin (dynamic,
version 1)
+[...]
+
+That looks quite good, except for the dummy description that we should
+replace with something nicer so that users will know what our plugin
+is about. We do this by editing the ``BRO_PLUGIN_DESCRIPTION`` line
+in ``src/Plugin.cc``, like this:
+
+# cat src/Plugin.cc
+
+#include plugin/Plugin.h
+
+BRO_PLUGIN_BEGIN(Demo, Rot13)
+BRO_PLUGIN_VERSION(1);
+BRO_PLUGIN_DESCRIPTION(Caesar cipher rotating a string's characters
by 13 places.);
+BRO_PLUGIN_BIF_FILE(consts);
+BRO_PLUGIN_BIF_FILE(events);
+BRO_PLUGIN_BIF_FILE(functions);
+BRO_PLUGIN_END
+
+# make
+[...]
+# bro -N | grep Rot13
+Plugin: Demo::Rot13 - Caesar cipher rotating a string's characters by 13
places. (dynamic, version 1)
+
+Better. Bro can also show us what exactly the plugin provides with the
+more verbose option ``-NN``::
+
+# bro -NN
+[...]
+Plugin: Demo::Rot13 - Caesar cipher rotating a string's
