I'm in the process of trying to debug a somewhat sluggish set of backends. The backends are ~20 python tornado web servers that implement a simple blocking db call to mongodb. I would theorize that the request rate can overload the number of backends and their ability to service periodically when outside forces slow the app down slightly and in turn cause connections to sit in each app servers listen queue briefly while it clears out requests.
I'm pretty sure halog can help me figure this out for certain, but I can't seem to either invoke the correct cmd line args or I'm not comprehending the output properly. For example: <CMD TO PIPE ~3k relavent requests lines>| halog -ad 10 gives me lines like: 21:02:21.020 75741020 11 1 21:02:21.050 75741050 13 1 21:02:21.313 75741313 17 1 21:02:21.522 75741522 10 1 21:02:21.549 75741549 13 1 21:02:21.661 75741661 11 2 21:02:21.704 75741704 12 1 21:02:21.762 75741762 15 1 I expect this to mean: 'filter out any lines that indicate an accept time below 10ms and show me anything greater'. However -ad is an input filter, so I have no idea what the output means.
