On Fri, Feb 12, 2021 at 1:47 PM Michael Goulish <[email protected]> wrote:
> Well, *this* certainly made a difference! > I tried this test: > > *message size:* 200000 bytes > *client-pairs:* 10 > *sender pause between messages:* 10 msec > *messages per sender:* 10,000 > * credit window:* 1000 > > > > > *Results:* > > router buffer size > 512 bytes 4K bytes > ----------------------------------- > CPU 517% 102% > Mem 711 MB 59 MB > Latency 26.9 *seconds* 2.486 *msec* > > > So with the large messages and our normal buffer size of 1/2 K, the router > just got overwhelmed. What I recorded was average memory usage, but looking > at the time sequence I see that its memory kept increasing steadily until > the end of the test. > With the large messages, the credit window is not sufficient to protect the memory of the router. I think this test needs to use a limited session window as well. This will put back-pressure on the senders much earlier in the test. With 200Kbyte messages x 1000 credits x 10 senders, there's a theoretical maximum of 2Gig of proton buffer memory that can be consumed before the router core ever moves any data. It's interesting that in the 4K-buffer case, the router core keeps up with the flow and in the 512-byte case, it does not. It appears that increasing the buffer size is a good idea. I don't think we've figured out how much the increase should be, however. We should look at interim sizes: 1K, 2K, maybe 1.5K and 3K. We want the smallest buffer size that gives us acceptable performance. If throughput, CPU, and memory use improve sharply with buffer size then level off, let's identify the "knee of the curve" and see what buffer size that represents. > > Messages just sat there waiting to get processed, which is maybe why their > average latency was *10,000 times longer* than when I used the large > buffers. > > And Nothing Bad Happened in the 4K buffer test. No crash, all messages > delivered, normal shutdown. > > Now I will try a long-duration test to see if it survives that while using > the large buffers. > > If it does survive OK, we need to see what happens with large buffers as > message size varies from small to large. >
