On Wed, Sep 09, 2026 at 01:00:33PM +0100, Daniel P. Berrangé wrote: > The migration code caches vmstate/ram writes into an iovec and > flushes this periodically. The total amount of data to sent > may be 100's of KB, but split across many iovec, each of which > is potentially quite small. > > The QIOChannelTLS receives the iovec, but since GNUTLS cannot > accept iovec data, it iterates calling send for each element. > > As a result of the migration data pattern, this results in > GNUTLS putting lots of small TLS records on the wire. This > has shown writes alternate between about 4k and 30 bytes > in some tests. > > This is triggering the nagle algorithm on migration-test for > many of the TLS test cases, resulting in a "go slow" for I/O > that eventually hits the migration timeout configured by the > test. > > Not every contributor reports seeing the "go slow" but for > those who do see it, it hits >= 95% of the time for at least > one of the migration TLS test cases run by 'make check'. > > While we could disable the nagle algorithm (and multifd > channels already do this), that would stil result in lots > of small TLS records hitting the wire which is not good for > throughput. The migration code flushes in batches because > it wants large writes for high throughput. > > To achieve this we must tell GNUTLS to encrypt data but not > immediately sent TLS records by "corking" its output. Once > the complete iovec has been written to GNUTLS, it can be > uncorked allowing it to hit the wire. > > There is added complexity with uncorking on non-blocking > channels as not all encrypted data can be sent at once. > qio_channel_write() will return what was sent, but some > data might remain pending inside GNUTLS buffers. A later > call to qio_channel_write() will provide the same plain > text data buffers that have already been cached. Thus the > code must attempt to uncork GNUTLS again to clear pending > data and then deduct the equivalent amount of plain text. > > Signed-off-by: Daniel P. Berrangé <[email protected]>
I wanted to test across two hosts on this patch, not for the issue because I can't reproduce, but for perf, due to the expected reduce on write syscalls. But then one of my hosts died.. so I only did a run locally with PSK, running high dirty loads and sample the bandwidth without multifd. vanilla: 2203.89 2061.09 2243.31 2241.19 2183.49 2186.59 (±74.59) patched: 3185.10 3223.91 2877.11 2987.71 2632.30 2981.23 (±241.40) That's 36% improvement. I still think we can do more things on top when we will care about perf (ktls might be the next to think about), but it looks a good start besides fixing the extremely (kind of a bug) slowness. Reviewed-by: Peter Xu <[email protected]> Thanks, -- Peter Xu
