Hi all,

This is a quick email more reporting a "gotcha": an effect that can cause 
gmetric updates to be lost.  This may be "well known" and, in retrospect, 
should be expected.  Apologies if this is already known, but I've not seen it 
documented, so I thought I'd risk sending an email.

When a program is "listening" for UDP packets (i.e. has bound [via bind(2)] a 
socket to some part of the udp-namespace), the kernel will queue only a 
finite number of packets.  Once that limit is reached, the kernel will not 
queue any additional packets so the application will never see them.

I've thrown together some code that tests this.  To see this for yourself, 
please download the source from:

http://ppewww.physics.gla.ac.uk/~paul/RndCode/pitch-catch.c

Here are some test results:
        Machine         Kernel                  Mem     Max queued UDP
        SL 3.0.8                2.4.21-47.ELhugemem     1GiB    221
        SL 4.4          2.6.9-42.0.10.EL                1GiB    337
        Debian sid      2.6.21-2-686            1GiB    406

SL is "Scientific Linux", a rebuild of the corresponding RedHat Enterprise 
Linux.

The consequence is that, when sending a batch of ~400 metric update messages 
from MonAMI to gmond (running on an SL-3.0.8 machine) there's a fairly good 
chance that the number of queued UDP packets will hit the 221 maximum.  When 
this happens last few packets are lost.

I guess forwarned is forarmed, but following NickG's suggestion of allowing 
the XDR content for multiple gmetric updates to be packed into a single 
packet would greatly reduce the likelihood of this problem being encountered.  
This is what ApMon (part of the MonaLisa project) does.  However, I guess 
switching to a TCP transport is the better long-term solution.

Cheers,

Paul.

-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
_______________________________________________
Ganglia-developers mailing list
Ganglia-developers@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ganglia-developers

Reply via email to