Hi: Still running through the plugin generation stuff ... seemed to build the binary distribution okay for me after I set BRO (FC 19, x86_64), but haven't tried to really load / run anything yet.
Personally, I'm a fan of this kind of development model. I like not having to worry about synchronizing with master to work on a plugin. I also like that the compilation / link times are much shorter for plugin development than they are when trying to add code to bro directly. +1 Seth's comment that the init-plugin script is very nice. A few thoughts: * Would a section on testing be appropriate? Both btest and unit testing might be useful for plugins. * A short section explaining how / when to modify CMakeLists.txt might be useful. I had to look at one of the existing analyzer plugins to double-check that all CC files needed to go into bro_plugin_cc(). Not hard, but still might be nice to explicitly document. * Should plugins be allowed to link to additional libraries? If so, how? I believe this could become an issue if I write a plugin that links against libXYZ, but libXYZ isn't available on the system that's trying to load the plugin. Cheers, Gilbert ________________________________________ From: bro-dev-boun...@bro.org [bro-dev-boun...@bro.org] On Behalf Of Robin Sommer [ro...@icir.org] Sent: Monday, December 16, 2013 3:34 PM To: bro-dev@bro.org Subject: [Bro-Dev] Dynamic plugin model (Re: [Bro-Commits] [git/bro] topic/robin/dynamic-plugins-2.3: Start of a plugin writing how-to. (87a1618)) I'd appreciate feedback on the model for dynamic plugins described below. The document contains a short intro on writing a simple plugin, as well as some background on how/when Bro integrates plugins. The code is in topic/robin/dynamic-plugins-2.3. It hasn't seen much testing yet but the basic infrastructure seems to work on Linux and Darwin at least. Robin On Mon, Dec 16, 2013 at 12:09 -0800, I wrote: > +=================== > +Writing Bro Plugins > +=================== > + > +Bro is internally moving to a plugin structure that enables extending > +the system dynamically, without modifying the core code base. That way > +custom code remains self-contained and can be maintained, compiled, > +and installed independently. Currently, plugins can add the following > +functionality to Bro: > + > + - Bro scripts. > + > + - Builtin functions/events/types for the scripting language. > + > + - Protocol and file analyzers. > + > + - Packet sources and packet dumpers. TODO: Not yet. > + > + - Logging framework backends. TODO: Not yet. > + > + - Input framework readers. TODO: Not yet. > + > +A plugin's functionality is available to the user just as if Bro had > +the corresponding code built-in. Indeed, internally many of Bro's > +pieces are structured as plugins as well, they are just statically > +compiled into the binary rather than loaded dynamically at runtime, as > +external plugins are. > + > +Quick Start > +=========== > + > +Writing a basic plugin is quite straight-forward as long as one > +follows a few conventions. In the following we walk through adding a > +new built-in function (bif) to Bro; we'll add `a `rot13(s: string) : > +string``, a function that rotates every character in a string by 13 > +places. > + > +A plugin comes in the form of a directory following a certain > +structure. To get started, Bro's distribution provides a helper script > +``aux/bro-aux/plugin-support/init-plugin`` that creates a skeleton > +plugin that can then be customized. Let's use that:: > + > + # mkdir rot13-plugin > + # cd rot13-plugin > + # init-plugin Demo Rot13 > + > +As you can see the script takes two arguments. The first is a > +namespace the plugin will live in, and the second a descriptive name > +for the plugin itself. Bro uses the combination of the two to identify > +a plugin. The namespace serves to avoid naming conflicts between > +plugins written by independent developers; pick, e.g., the name of > +your organisation (and note that the namespace ``Bro`` is reserved for > +functionality distributed by the Bro Project). In our example, the > +plugin will be called ``Demo::Rot13``. > + > +The ``init-plugin`` script puts a number of files in place. The full > +layout is described later. For now, all we need is > +``src/functions.bif``. It's initially empty, but we'll add our new bif > +there as follows:: > + > + # cat scripts/functions.bif > + module CaesarCipher; > + > + function camel_case%(s: string%) : string > + %{ > + char* rot13 = copy_string(s->CheckString()); > + > + for ( char* p = rot13; *p; p++ ) > + { > + char b = islower(*p) ? 'a' : 'A'; > + *p = (*p - b + 13) % 26 + b; > + } > + > + return new StringVal(strlen(rot13), rot13); > + %} > + > +The syntax of this file is just like any other ``*.bif`` file; we > +won't go into it here. > + > +Now we can already compile our plugin, we just need to tell the > +Makefile put in place by ``init-plugin`` where the Bro source tree is > +located (Bro needs to have been built there first):: > + > + # make BRO=/path/to/bro/dist > + [... cmake output ...] > + > +Now our ``rot13-plugin`` directory has everything that it needs > +for Bro to recognize it as a dynamic plugin. Once we point Bro to it, > +it will pull it in automatically, as we can check with the ``-N`` > +option: > + > + # export BRO_PLUGIN_PATH=/path/to/rot13-plugin > + # bro -N > + [...] > + Plugin: Demo::Rot13 - <Insert brief description of plugin> (dynamic, > version 1) > + [...] > + > +That looks quite good, except for the dummy description that we should > +replace with something nicer so that users will know what our plugin > +is about. We do this by editing the ``BRO_PLUGIN_DESCRIPTION`` line > +in ``src/Plugin.cc``, like this: > + > + # cat src/Plugin.cc > + > + #include <plugin/Plugin.h> > + > + BRO_PLUGIN_BEGIN(Demo, Rot13) > + BRO_PLUGIN_VERSION(1); > + BRO_PLUGIN_DESCRIPTION("Caesar cipher rotating a string's characters > by 13 places."); > + BRO_PLUGIN_BIF_FILE(consts); > + BRO_PLUGIN_BIF_FILE(events); > + BRO_PLUGIN_BIF_FILE(functions); > + BRO_PLUGIN_END > + > + # make > + [...] > + # bro -N | grep Rot13 > + Plugin: Demo::Rot13 - Caesar cipher rotating a string's characters by 13 > places. (dynamic, version 1) > + > +Better. Bro can also show us what exactly the plugin provides with the > +more verbose option ``-NN``:: > + > + # bro -NN > + [...] > + Plugin: Demo::Rot13 - Caesar cipher rotating a string's characters by 13 > places. (dynamic, version 1) > + [Function] CaesarCipher::rot13 > + [...] > + > +There's our function. Now let's use it:: > + > + # bro -e 'print CaesarCipher::rot13("Hello")' > + Uryyb > + > +It works. We next install the plugin along with Bro itself, so that it > +will find it directly without needing the ``BRO_PLUGIN_PATH`` > +environment variable. If we first unset the variable, the function > +will no longer be available:: > + > + # unset BRO_PLUGIN_PATH > + # bro -e 'print CaesarCipher::rot13("Hello")' > + error in <command line>, line 1: unknown identifier CaesarCipher::rot13, > at or near "CaesarCipher::rot13" > + > +Once we install it, it works again:: > + > + # make install > + # bro -e 'print CaesarCipher::rot13("Hello")' > + Uryyb > + > +The installed version went into > +``<bro-install-prefix>/lib/bro/plugins/Demo_Rot13``. > + > +We can distribute the plugin in either source or binary form by using > +the Makefile's ``sdist`` and ``bdist`` target, respectively. Both > +create corrsponding tarballs:: > + > + # make sdist > + [...] > + Source distribution in build/sdist/Demo_Rot13.tar.gz > + > + # make bdist > + [...] > + Binary distribution in build/Demo_Rot13-darwin-x86_64.tar.gz > + > +The source archive will contain everything in the plugin directory > +except any generated files. The binary archive will contain anything > +needed to install and run the plugin, i.e., just what ``make install`` > +puts into place as well. As the binary distribution is > +platform-dependent, its name includes the OS and architecture the > +plugin was built on. > + > +Plugin Directory Layout > +======================= > + > +A plugin's directory needs to follow a set of conventions so that Bro > +(1) recognizes it as a plugin, and (2) knows what to load. While > +``init-plugin`` takes care of most of this, the following is the full > +story. We'll use ``<base>`` to represent a plugin's top-level > +directory. > + > +``<base>/__bro_plugin__`` > + A file that marks a directory as containing a Bro plugin. The file > + must exist, and its content must consist of a single line with the > + qualified name of the plugin (e.g., "Demo::Rot13"). > + > +``<base>/lib/<plugin-name>-<os>-<arch>.so`` > + The shared library containing the plugin's compiled code. Bro will > + load this in dynamically at run-time if OS and architecture match > + the current platform. > + > +``lib/bif/`` > + Directory with auto-generated Bro scripts that declare the plugins > + bif elements. The files here are produced by ``bifcl``. > + > +``scripts/`` > + A directory with the plugin's custom Bro scripts. When the plugin > + gets activated, this directory will be automatically added to > + ``BROPATH``, so that any scripts/modules inside can be > + ``@load``ed. > + > +``scripts``/__load__.bro > + A Bro script that will be loaded immediately when the plugin gets > + activated. See below for more information on activating plugins. > + > +By convention, a plugin should put its custom scripts into sub folders > +of ``scripts/``, i.e., ``scripts/<script-namespace>/<script>.bro`` to > +avoid conflicts. As usual, you can then put a ``__load__.bro`` in > +there as well so that, e.g., ``@load Demo/Rot13`` could load a whole > +module in the form of multiple individual scripts. > + > +Note that in addition to the paths above, the ``init-plugin`` helper > +puts some more files and directories in place that help with > +development and installation (e.g., ``CMakeLists.txt``, ``Makefile``, > +and source code in ``src/``). However, all these do not have a special > +meaning for Bro at runtime and aren't necessary for a plugin to > +function. > + > +``init-plugin`` > +=============== > + > +``init-plugin`` puts a basic plugin structure in place that follows > +the above layout and augments it with a CMake build and installation > +system. Note that plugins with this structure can be used both > +directly out of their source directory (after ``make`` and setting > +Bro's ``BRO_PLUGIN_PATH``), and when installed alongside Bro (after > +``make install``). > + > +``make install`` copies over the ``lib`` and ``scripts`` directories, > +as well as the ``__bro_plugin__`` magic file and the ``README`` (which > +you should customize). One can add further CMake ``install`` rules to > +install additional files if neeed. > + > +.. todo:: > + > + Describe the other files that the script puts in place. > + > +Activating a Plugin > +=================== > + > +A plugin needs to be *activated* to make it available to the user. > +Activating a plugin will: > + > + 1. Load the dynamic module > + 2. Make any bif items available > + 3. Add the ``scripts/`` directory to ``BROPATH`` > + 4. Load ``scripts/__load__.bro`` > + > +By default, Bro will automatically activate all dynamic plugins found > +in its search path ``BRO_PLUGIN_PATH``. However, in bare mode (``bro > +-b``), no dynamic plugins will be activated by default; instead the > +user can selectively enable individual plugins in scriptland using the > +``@load-plugin <qualified-plugin-name>`` directive (e.g., > +``@load-plugin Demo::Rot13``). > + > +``bro -N`` shows activated and found yet unactivated plugins > +separately. Note that plugins compiled statically into Bro are always > +activated, and hence show up as such even in bare mode. > + > +.. todo:: > + > + Is this the right activation model? > + > + > +Plugin Component > +================ > + > +The following gives additional information about providing individual > +types of functionality via plugins. Note that a single plugin can > +provide more than one type. For example, a plugin could provide > +multiple protocol analyzers at once; or both a logging backend and > +input reader at the same time. > + > +We now walk briefly through the specifics of providing a specific type > +of functionality (a *component*) through plugin. We'll focus on their > +interfaces to the plugin system, rather than specifics on writing the > +corresponding logic (usually the best way to get going on that is to > +start with an existing plugin providing a corresponding component and > +adapt that). We'll also point out how the CMake infrastructure put in > +place by the ``init-plugin`` helper script ties the various pieces > +together. > + > +Bro Scripts > +----------- > + > +Scripts are easy: just put them into ``scripts/``, as described above. > +The CMake infrastructure will automatically install them, as well > +include them into the source and binary plugin distributions. > + > +Builtin Language Elements > +------------------------- > + > +Functions > + TODO > + > +Events > + TODO > + > +Types > + TODO > + > +Protocol Analyzers > +------------------ > + > +TODO. > + > +File Analyzers > +-------------- > + > +TODO. > + > +Logging Writer > +-------------- > + > +Not yet implemented. > + > +Input Reader > +------------ > + > +Not yet implemented. > + > +Packet Sources > +-------------- > + > +Not yet implemented. > + > +Packet Dumpers > +-------------- > + > +Not yet implemented. > + > +Documenting Plugins > +=================== > + > +..todo:: > + > + Integrate all this with Broxygen. > + > + > + > > _______________________________________________ > bro-commits mailing list > bro-comm...@bro.org > http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-commits > -- Robin Sommer * Phone +1 (510) 722-6541 * ro...@icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org/robin _______________________________________________ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev _______________________________________________ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev