Re: [Bro-Dev] Performance Enhancements

[Clark, Gilbert]

Howdy:


Not sure about the content of the BroCon talk ... but a few years back, I did a 
bit of work on this.  There was a plugin here:


https://github.com/cubic1271/bro-plugin-instrumentation


that allowed me to profile the execution of various bro scripts and figure out 
what was eating the most time.  It also added a pretty braindead mechanism to 
expose script variables through a REST interface, which I wrapped in an HTML5 
UI to get some real-time statistics ... though I have no idea where that code 
went.


I also threw together this:


https://github.com/cubic1271/pybrig


which was intended to benchmark Bro on a specific platform, the idea being to 
get results that were relatively consistent.  It could make some pretty 
pictures, which was kind of neat ... but I'd probably do things a lot 
differently if I had it to do over again :)

I'll note that one of the challenges with profiling is that there are the bro 
scripts, and then there is the bro engine.  The scripting layer has a 
completely different set of optimizations that might make sense than the engine 
does: turning off / turning on / tweaking different scripts can have a huge 
impact on Bro's relative performance depending on the frequency with which 
those script fragments are executed.  Thus, one way to look at speeding things 
up might be to take a look at the scripts that are run most often and seeing 
about ways to accelerate core pieces of them ... possibly by moving pieces of 
those scripts to builtins (as C methods).


If I had to guess at one engine-related thing that would've sped things up when 
I was profiling this stuff back in the day, it'd probably be rebuilding the 
memory allocation strategy / management.  From what I remember, Bro does do 
some malloc / free in the data path, which hurts quite a bit when one is trying 
to make things go fast.  It also means that the selection of a memory allocator 
and NUMA / per-node memory management is going to be important.  That's 
probably not going to qualify as something *small*, though ...


On a related note, a fun experiment is always to try running bro with a 
different allocator and seeing what happens ...


Another thing that (I found) got me a few percentage points for more-or-less 
free was profile-guided optimization: I ran bro first with profiling enabled 
against a representative data set, then rebuilt it against the profile I 
collected.  Of course, your mileage may vary ...


Anyway, hope something in there was useful.


Cheers,

Gilbert Clark



From: bro-dev-boun...@bro.org  on behalf of Jim 
Mellander 
Sent: Thursday, October 5, 2017 3:45:21 PM
To: bro-dev@bro.org
Subject: [Bro-Dev] Performance Enhancements

Hi all:

One item of particular interest to me from Brocon was this tidbit from 
Packetsled's lightning talk:

"Optimizing core loops (like net_run() ) with preprocessor branch prediction 
macros likely() and unlikely() for ~3% speedup. We optimize for maximum load."

After conversing with Leo Linsky of Packetsled, I wanted to initiate a 
conversation about easy performance improvements that may be within fairly easy 
reach:

1. Obviously, branch prediction, as mentioned above.  3% speedup for (almost) 
free is nothing to sneeze at.
2. Profiling bro to identify other hot spots that could benefit from 
optimization.
3. Best practices for compiling Bro (compiler options, etc.)
4. Data structure revisit (hash functions, perhaps?)
etc.

Perhaps the Bro core team is working on some, all, or a lot more in this area.  
It might be nice to get the Bro community involved too.  Is anyone else 
interested?

Jim Mellander
ESNet




___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] early performance comparisons of CAF-based run loop

[Clark, Gilbert]

$0.02 USD:


As I recall, Bro's per-packet processing overhead can vary significantly as a 
result of timers and triggers that execute on a situational basis.  Also, 
relative overhead of packet ingest is going to vary based on the set of loaded 
scripts in addition to the specific trace used to run the tests.  That's not 
trying to argue that these results are not useful / interesting, but instead 
*only* that the specific percentages might not be representative of the general 
case (just because I'm convinced that there really is not a general case to 
objectively measure).


Also ... if the overhead of the polling / ingest itself turns out to be a huge 
problem at high rates, one idea would be to separate that and pass packets (in 
bulk) through a ring / high-speed IPC to the process that needs to ingest them. 
 That's worked pretty well for me in DPDK, and has the benefit of being able to 
distribute packets from one ingest to multiple processors (which is something 
I've had to do for process-heavy workloads ... which I would argue is something 
that Bro tends to be).


Along those lines, rather than spending much time on packet ingest mechanics in 
bro (or pieces thereof), one idea might be to instead focus on integrating 
packet bricks as a standard ingest / distribution mechanic for everything 
packet-related in the general case.  The idea would be that fetching packets 
from bro (and its related processes) would become less about calls to epoll and 
select, and more about high-speed IPC that went out of its way to avoid 
kernel-space entirely.  The nice thing about that is that it'd be a little 
easier to standardize on the bro side of things, and would take a step toward 
separating bro as a scripting / event engine from bro as a (relative) monolith.


Of course, the down side is that packet bricks could add some serious 
(mandatory) complexity to bro, so maybe it's not the right answer ... but maybe 
a more lightweight, specialized distribution channel might be doable, or maybe 
there would be a way to embed packet bricks inside of an application in the 
event that folks didn't want to run the two separately, or ... etc.


As always, just for what it's worth :)


-Gilbert



From: bro-dev-boun...@bro.org  on behalf of Siwek, Jon 

Sent: Tuesday, April 11, 2017 8:41:23 PM
To: 
Subject: [Bro-Dev] early performance comparisons of CAF-based run loop

I recently got a minimal CAF-based run loop for Bro working, did crude 
performance comparisons, and wanted to share.

The approach was to measure average time between calls of net_packet_dispatch() 
and also the average time it takes to analyze a packet.  The former attempts to 
measure the overhead imposed by the loop implementation and the later just 
gives an idea of how significant a chunk of time that is in relation to Bro’s 
main workload.  I found that the overhead of the loop can be ~5-10% of the 
packet processing time, so it does seem worthwhile to try and keep the run loop 
overhead low.

Initial testing of the CAF-based loop showed the overhead increased by ~1.8x, 
but there was still a major difference in the implementations: the standard Bro 
loop only invokes its IOSource polling mechanism (select) once every 25 cycles 
of the loop, while the CAF implementation’s polling mechanism (actor/thread 
scheduling + messaging + epoll) is used for every cycle/packet.  As one would 
expect, by just trivially spinning the main process() function in a loop for 25 
iterations, the overhead of the CAF-based loop comes back into line with the 
standard run loop.

To try and better measure the actual differences related to the polling 
mechanism implementation, I quickly hacked Bro’s standard runloop to select() 
on every packet instead of once every 25th and found that the overhead measures 
+/- 10% within the 1.8x overhead increase of the initial CAF-based loop.  So is 
the cost of the extra system call for epoll/select per packet the main thing to 
avoid?  Sort of.  I again hacked Bro’s standard loop to be able to use either 
epoll or poll instead of select and found that those do better, with the 
overhead increase being about 1.3x (still doing one “poll” per packet) in 
relation to the standard run loop.  Meaning there is some measurable trend in 
polling mechanism performance (for sparse # of FDs/sources): poll comes in 
first, epoll second, with CAF and select about tied for third.

Takeaways:

(1) Regardless of runloop implementation or polling mechanism choices, 
performing the polling operation once per packet should probably be avoided.  
In concept, it’s an easy way to get a 2-5% speedup in relation to total packet 
processing time.

(2) Related to (1), but not in the sense of performance, is that even w/ a 
CAF-based loop it still seems somewhat difficult to reason about the reality of 
how IOSources are prioritized.  In the standard loop, the 

[Bro-Dev] Fw: Broker raw throughput

[Clark, Gilbert]

Hi:

Forwarding reply to the bro-dev list: original was mistakenly posted elsewhere 
(sorry about that).  Leaving original message content inline for context.

From: Matthias Vallentin <matth...@vallentin.net> on behalf of Matthias 
Vallentin <vallen...@icir.org>
Sent: Thursday, March 10, 2016 1:46 PM
To: Clark, Gilbert
Subject: Re: [Bro-Dev] Broker raw throughput

> > Sorry if this is a stupid question, but what are the performance
> > requirements for broker, exactly?
>
>We have too little experience to tell what we need 

Right, this was my question.  The less that broker's requirements are 
documented and understood, the more difficult it becomes to evaluate whether or 
not broker will fit with an intended use-case, I think.  To me, the performance 
numbers themselves don't matter as much as managing expectations does: should I 
*expect* to be able to pass all of my events through broker?

>and where we hit bottlenecks. 
> 
> This is why we compare a worst-case scenario across
>multiple libraries and see where we find low-hanging fruits for
>optimization potential.

Got it.  I think that's what confused me ...

>> That's *good enough* for most things, but it's also still quite
>> possible to do better: I've built DSP applications on top of DPDK that
>> do fine with ~14 Mpps, which is itself pretty slow when compared to
>> results reported by e.g. [2].
>
>It's not about going as fast as possible. We're looking to achieve good
>performance in the common case, *without* reducing a high level of
>abstraction.

... because some of those messaging libraries operate at different levels of 
abstraction than others, which is going to drive the performance to some extent 
...

>
>> Realistically, depending on what broker is intended to support, maybe
>> 200k messages / second is fine:
>
>Agreed. At this point, this rate is certainly fine for our
>worst-case-single-int-per-message benchmarks.
>
>> TL/DR: I'm of the opinion that optimization is fun, but I also would
>> feel kind of bad watching CAF go too far down a (very scary) rabbit
>> hole just to support one (albeit very large and rather cool)
>> application ...
>
>I don't think we want to do that either, this must be a
>misunderstanding. The optimizations we've been looking at are
>application-independent. 

... so it looks like it was indeed a misunderstanding on my part.  Sorry about 
that.

Trying to express things a slightly different way, I was concerned that the 
different numbers from the different libraries were being interpreted as an 
apples-to-apples comparison.  Modifying CAF to achieve the same results as e.g. 
0mq would, at some point and in some way, eventually require modifying CAF to 
be more like 0mq.  I don't think that would be good, because 0mq and CAF aren't 
(and shouldn't be, in my humble opinion) the same thing.

> We have looked at a very specific workload to
>bound worst-case performance. I'm very happy with the recent
>improvements that will ship with CAF 0.15.

Definitely.  Performance improvements are always good :)

Cheers,
Gilbert

___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] Geo Location Plugin

[Clark, Gilbert]

+1 that core plugins be built / distributed with bro.

Re: package / plugin management, coming from a Java EE background, one of the 
things I don't like about some of the OSGi software (read: highly modular, 
plugin-driven) I've worked with is how confusing it can get to track versions 
of each specific bundle / plugin to know how / when to update them.  Binding a 
specific set of core plugin versions to application releases mitigates some of 
that complexity in my experience.  In other words, I'd be a bit hesitant to 
agree that bro should individually package *core* plugins and offer them 
independently through a repo, just because it will probably require a great 
deal of care to update those (since there could be e.g. side effects / 
dependencies that aren't immediately obvious).

An external repository / package manager seems like a good fit for non-core 
plugins, though.

-Gilbert

On Oct 16, 2014 11:41 AM, anthony kasza anthony.ka...@gmail.com wrote:

I think having some plugins distributed and built with Bro is reasonable, too. 
As long as functionality and interfaces don't change current users shouldn't 
notice a difference.
How eventually moving the plugin to a package manager will affect users is 
something else to consider.

-AK

On Oct 16, 2014 6:21 AM, Slagell, Adam J 
slag...@illinois.edumailto:slag...@illinois.edu wrote:
I think that is reasonable for some things.

On Oct 16, 2014, at 8:02 AM, Robin Sommer ro...@icir.orgmailto:ro...@icir.org 
wrote:

 (unless we decide to
 build (some) plugins by default, which currently isn't happening).

--

Adam J. Slagell
Chief Information Security Officer
Assistant Director, Cybersecurity Directorate
National Center for Supercomputing Applications
University of Illinois at Urbana-Champaign
www.slagell.infohttp://www.slagell.info

Under the Illinois Freedom of Information Act (FOIA), any written 
communication to or from University employees regarding University business is 
a public record and may be subject to public disclosure.










___
bro-dev mailing list
bro-dev@bro.orgmailto:bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] Plugins providing threads?

[Clark, Gilbert]

Would it make sense to replace the existing inter-thread communication code 
with the broker / porting the existing writers and readers to use the actor 
framework?  This way, there would only be a single, shared message-passing 
mechanism inside of bro, instead of having one for core - threads, and 
another for core - external

-Gilbert

From: bro-dev-boun...@bro.org bro-dev-boun...@bro.org on behalf of Siwek, Jon 
jsi...@illinois.edu
Sent: Wednesday, October 08, 2014 11:29 AM
To: Robin Sommer
Cc: bro-dev@bro.org
Subject: Re: [Bro-Dev] Plugins providing threads?

On Oct 7, 2014, at 5:43 PM, Robin Sommer ro...@icir.org wrote:

 Jon, how are you planing to integrate Broker into Bro? Would this help
 there as well if you could just follow a similar structure with Broker
 running inside its own thread?

Mostly planning to integrate as IOSource(s) in the main thread as I expect many 
of the messages either originate or terminate there due to script-land 
interactivity.  Broker’s internal processing is to be scheduled to threads 
automatically by actor-framework.  Wrapping all Broker-related messaging as a 
Bro-thread that does its own message passing seems like an unneeded indirection 
as Broker already facilitates its own message passing between endpoints (as 
it’s built on top of the more-generalized actor-framework message passing).

- Jon
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] Geo Location Plugin

[Clark, Gilbert]

Hi:

A mismatched function definition in .h / implementation in .cc usually causes 
this issue for me.  Maybe try running the mangled symbol that's reported as 
missing through c++filt to see what it really is (if you haven't already) and 
use nm or the like to make sure the plugin library file contains that symbol?

-Gilbert


From: bro-dev-boun...@bro.org [bro-dev-boun...@bro.org] on behalf of anthony 
kasza [anthony.ka...@gmail.com]
Sent: Tuesday, October 07, 2014 12:46 AM
To: Robin Sommer
Cc: bro-dev@bro.org
Subject: Re: [Bro-Dev] Geo Location Plugin

Thanks Robin. Everything on the Writing Bro Plugins page is clear although 
the debugging section seems thin to me. Then again, I've have never compiled 
Bro with debugging enabled before.

I'm running into issues with my plugin's shared object. Bro is complaining 
about an undefined symbol (likely due to a bug in my source). Are there plans 
to expand the Types section of that page? I think I'm not declaring a 
scriptland record type correctly.

-AK

On Oct 6, 2014 8:10 AM, Robin Sommer ro...@icir.orgmailto:ro...@icir.org 
wrote:
On Sun, Oct 05, 2014 at 23:00 -0700, you wrote:

 I can at least now see why my plugin isn't working as excepted.

Great, let me know if it's anything that the plugin documentation
could make more clear.

Robin

--
Robin Sommer * Phone +1 (510) 722-6541tel:%2B1%20%28510%29%20722-6541 * 
ro...@icir.orgmailto:ro...@icir.org
ICSI/LBNL* Fax   +1 (510) 666-2956tel:%2B1%20%28510%29%20666-2956 * 
www.icir.org/robinhttp://www.icir.org/robin

___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] Dynamic plugin model (Re: [Bro-Commits] [git/bro] topic/robin/dynamic-plugins-2.3: Start of a plugin writing how-to. (87a1618))

[Clark, Gilbert]

Hi:

Still running through the plugin generation stuff ... seemed to build the 
binary distribution okay for me after I set BRO (FC 19, x86_64), but haven't 
tried to really load / run anything yet.

Personally, I'm a fan of this kind of development model.  I like not having to 
worry about synchronizing with master to work on a plugin.  I also like that 
the compilation / link times are much shorter for plugin development than they 
are when trying to add code to bro directly.

+1 Seth's comment that the init-plugin script is very nice.

A few thoughts:

* Would a section on testing be appropriate?  Both btest and unit testing might 
be useful for plugins.
* A short section explaining how / when to modify CMakeLists.txt might be 
useful.  I had to look at one of the existing analyzer plugins to double-check 
that all CC files needed to go into bro_plugin_cc().  Not hard, but still might 
be nice to explicitly document.
* Should plugins be allowed to link to additional libraries?  If so, how?  I 
believe this could become an issue if I write a plugin that links against 
libXYZ, but libXYZ isn't available on the system that's trying to load the 
plugin.

Cheers,
Gilbert

From: bro-dev-boun...@bro.org [bro-dev-boun...@bro.org] On Behalf Of Robin 
Sommer [ro...@icir.org]
Sent: Monday, December 16, 2013 3:34 PM
To: bro-dev@bro.org
Subject: [Bro-Dev] Dynamic plugin model (Re: [Bro-Commits]  [git/bro]   
topic/robin/dynamic-plugins-2.3: Start of a plugin  writing how-to. 
(87a1618))

I'd appreciate feedback on the model for dynamic plugins described
below.

The document contains a short intro on writing a simple plugin, as
well as some background on how/when Bro integrates plugins. The code
is in topic/robin/dynamic-plugins-2.3. It hasn't seen much testing yet
but the basic infrastructure seems to work on Linux and Darwin at
least.

Robin

On Mon, Dec 16, 2013 at 12:09 -0800, I wrote:

 +===
 +Writing Bro Plugins
 +===
 +
 +Bro is internally moving to a plugin structure that enables extending
 +the system dynamically, without modifying the core code base. That way
 +custom code remains self-contained and can be maintained, compiled,
 +and installed independently. Currently, plugins can add the following
 +functionality to Bro:
 +
 +- Bro scripts.
 +
 +- Builtin functions/events/types for the scripting language.
 +
 +- Protocol and file analyzers.
 +
 +- Packet sources and packet dumpers. TODO: Not yet.
 +
 +- Logging framework backends. TODO: Not yet.
 +
 +- Input framework readers. TODO: Not yet.
 +
 +A plugin's functionality is available to the user just as if Bro had
 +the corresponding code built-in. Indeed, internally many of Bro's
 +pieces are structured as plugins as well, they are just statically
 +compiled into the binary rather than loaded dynamically at runtime, as
 +external plugins are.
 +
 +Quick Start
 +===
 +
 +Writing a basic plugin is quite straight-forward as long as one
 +follows a few conventions. In the following we walk through adding a
 +new built-in function (bif) to Bro; we'll add `a `rot13(s: string) :
 +string``, a function that rotates every character in a string by 13
 +places.
 +
 +A plugin comes in the form of a directory following a certain
 +structure. To get started, Bro's distribution provides a helper script
 +``aux/bro-aux/plugin-support/init-plugin`` that creates a skeleton
 +plugin that can then be customized. Let's use that::
 +
 +# mkdir rot13-plugin
 +# cd rot13-plugin
 +# init-plugin Demo Rot13
 +
 +As you can see the script takes two arguments. The first is a
 +namespace the plugin will live in, and the second a descriptive name
 +for the plugin itself. Bro uses the combination of the two to identify
 +a plugin. The namespace serves to avoid naming conflicts between
 +plugins written by independent developers; pick, e.g., the name of
 +your organisation (and note that the namespace ``Bro`` is reserved for
 +functionality distributed by the Bro Project). In our example, the
 +plugin will be called ``Demo::Rot13``.
 +
 +The ``init-plugin`` script puts a number of files in place. The full
 +layout is described later. For now, all we need is
 +``src/functions.bif``. It's initially empty, but we'll add our new bif
 +there as follows::
 +
 +# cat scripts/functions.bif
 +module CaesarCipher;
 +
 +function camel_case%(s: string%) : string
 +%{
 +char* rot13 = copy_string(s-CheckString());
 +
 +for ( char* p = rot13; *p; p++ )
 +{
 +char b = islower(*p) ? 'a' : 'A';
 +*p  = (*p - b + 13) % 26 + b;
 +}
 +
 +return new StringVal(strlen(rot13), rot13);
 +%}
 +
 +The syntax of this file is just like any other ``*.bif`` file; we
 +won't go into it here.
 +
 +Now we can already compile our plugin, we just need to tell the
 +Makefile put in place by 

Re: [Bro-Dev] Dynamic plugin model (Re: [Bro-Commits] [git/bro] topic/robin/dynamic-plugins-2.3: Start of a plugin writing how-to. (87a1618))

[Clark, Gilbert]

The static and dynamic plugins could be unified further. It's unclear what the 
right default is for shipping plugins that provide standard functionality, but 
it would be nice in any case if we could just flip a switch to change between 
static and dynamic builds for the in-tree stuff.

What's the reason for supporting both static and dynamic plugin types?  

Assuming that eliminating the static plugin type entirely would simplify the 
build / link process, and that the static plugin type exists largely due to 
performance concerns: if we could prove that dynamic linking didn't have a 
significant impact on performance, could we use that as rationale for 
eliminating static plugin linkage entirely?

Cheers,
Gilbert

___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] [Bro-Commits] [git/bro] master: Fix for input readers occasionally dead-locking. (c980d10)

[Clark, Gilbert]

// Thoughts?

/* 
 * Since we know that read_ctr is only incremented after a successful read, and 
write_ctr is only incremented after a successful write, the two values should 
be equal iff the queue is empty.  I think it could also help if these two items 
were 
*  declared volatile.
*/
bool MaybeReady() { return read_ctr != write_ctr; }

// Alternatively, replacing random() with a per-instance counter and saying the 
function returns true every Xth time it is called might mean you weren't having 
to hit the RNG every time you checked on the status of a queue.
// Also, should the call be to random_r() instead of random() ?

// --Gilbert

From: bro-commits-boun...@bro.org [bro-commits-boun...@bro.org] On Behalf Of 
Robin Sommer [ro...@icir.org]
Sent: Thursday, October 24, 2013 9:28 PM
To: bro-comm...@bro.org
Subject: [Bro-Commits] [git/bro] master: Fix for input readers occasionally 
dead-locking. (c980d10)

Repository : ssh://g...@bro-ids.icir.org/bro

On branch  : master
Link   : 
https://github.com/bro/bro/commit/c980d1055e1e17da4867e3fab1ee10f604b242b0

---

commit c980d1055e1e17da4867e3fab1ee10f604b242b0
Author: Robin Sommer ro...@icir.org
Date:   Thu Oct 24 18:16:49 2013 -0700

Fix for input readers occasionally dead-locking.

Bernhard and I tracked it down we believe: the thread queue could
deadlock in certain cases. As a fix we tuned the heuristic for telling
if a queue might have input to occasionaly err on the safe side by
flagging yes, so that processing will proceed.

It's a bit unfortunate to apply this fix last minute before the
release as it could potentially impact performance if the heuristic
fails to often. We believe the chosen parmaterization should be fine ...


---

c980d1055e1e17da4867e3fab1ee10f604b242b0
 CHANGES   |  4 
 VERSION   |  2 +-
 src/threading/Queue.h | 10 ++
 3 files changed, 11 insertions(+), 5 deletions(-)

diff --git a/CHANGES b/CHANGES
index 10bc187..1ae192f 100644
--- a/CHANGES
+++ b/CHANGES
@@ -1,4 +1,8 @@

+2.2-beta-152 | 2013-10-24 18:16:49 -0700
+
+  * Fix for input readers occasionally dead-locking. (Robin Sommer)
+
 2.2-beta-151 | 2013-10-24 16:52:26 -0700

   * Updating submodule(s).
diff --git a/VERSION b/VERSION
index 2b5702f..26d1beb 100644
--- a/VERSION
+++ b/VERSION
@@ -1 +1 @@
-2.2-beta-151
+2.2-beta-152
diff --git a/src/threading/Queue.h b/src/threading/Queue.h
index 792fb63..c4f2bfa 100644
--- a/src/threading/Queue.h
+++ b/src/threading/Queue.h
@@ -61,11 +61,13 @@ public:
bool Ready();

/**
-* Returns true if the next Get() operation might succeed.
-* This function may occasionally return a value not
-* indicating the actual state, but won't do so very often.
+* Returns true if the next Get() operation might succeed. This
+* function may occasionally return a value not indicating the actual
+* state, but won't do so very often. Occasionally we also return a
+* true unconditionally to avoid a deadlock when both pointers happen
+* to be equal even though there's stuff queued.
 */
-   bool MaybeReady() { return ( ( read_ptr - write_ptr) != 0 ); }
+   bool MaybeReady() { return (read_ptr != write_ptr) || (random() % 1 
== 0); }

/** Wake up the reader if it's currently blocked for input. This is
 primarily to give it a chance to check termination quickly.

___
bro-commits mailing list
bro-comm...@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-commits
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] Draft API for new communication library

[Clark, Gilbert]

FWIW, nanopb is a lightweight (read: targeted at embedded devices) C library 
that is compatible with protobuf.

http://koti.kapsi.fi/~jpa/nanopb/

--Gilbert

From: bro-dev-boun...@bro.org [bro-dev-boun...@bro.org] On Behalf Of Robin 
Sommer [ro...@icir.org]
Sent: Thursday, October 17, 2013 6:03 PM
To: Siwek, Jonathan Luke
Cc: bro-dev@bro.org
Subject: Re: [Bro-Dev] Draft API for new communication library

On Thu, Oct 17, 2013 at 21:54 +, you wrote:

 Would something like Cap'n Proto or Protocol Buffers help in
 defining/maintaining a serialization format?

I didn't know Cap’n Proto so far but I have been wondering about using
Protocol Buffers already as well. We'd have to add another dependency
but it would make this stuff quite a bit less cumbersome. Do you know
if their C version is well maintained? It looks rather old compared to
the standard protobuf distribution. Does Cap'n Proto have a C API?

Robin

--
Robin Sommer * Phone +1 (510) 722-6541 * ro...@icir.org
ICSI/LBNL* Fax   +1 (510) 666-2956 * www.icir.org/robin
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev