Hi Daniel, On 2026-09-09 13:00, Daniel P. Berrangé wrote: > The migration code caches vmstate/ram writes into an iovec and > flushes this periodically. The total amount of data to sent > may be 100's of KB, but split across many iovec, each of which > is potentially quite small. > > The QIOChannelTLS receives the iovec, but since GNUTLS cannot > accept iovec data, it iterates calling send for each element. > > As a result of the migration data pattern, this results in > GNUTLS putting lots of small TLS records on the wire. This > has shown writes alternate between about 4k and 30 bytes > in some tests. > > This is triggering the nagle algorithm on migration-test for > many of the TLS test cases, resulting in a "go slow" for I/O > that eventually hits the migration timeout configured by the > test. > > Not every contributor reports seeing the "go slow" but for > those who do see it, it hits >= 95% of the time for at least > one of the migration TLS test cases run by 'make check'. > > While we could disable the nagle algorithm (and multifd > channels already do this), that would stil result in lots > of small TLS records hitting the wire which is not good for > throughput. The migration code flushes in batches because > it wants large writes for high throughput. > > To achieve this we must tell GNUTLS to encrypt data but not > immediately sent TLS records by "corking" its output. Once > the complete iovec has been written to GNUTLS, it can be > uncorked allowing it to hit the wire. > > There is added complexity with uncorking on non-blocking > channels as not all encrypted data can be sent at once. > qio_channel_write() will return what was sent, but some > data might remain pending inside GNUTLS buffers. A later > call to qio_channel_write() will provide the same plain > text data buffers that have already been cached. Thus the > code must attempt to uncork GNUTLS again to clear pending > data and then deduct the equivalent amount of plain text. > > Signed-off-by: Daniel P. Berrangé <[email protected]> > --- > crypto/tlssession.c | 33 ++++++++++++++ > include/crypto/tlssession.h | 25 +++++++++++ > include/io/channel-tls.h | 1 + > io/channel-tls.c | 87 ++++++++++++++++++++++++++++++++++--- > 4 files changed, 140 insertions(+), 6 deletions(-)
I have tested the patch and after applying I could no longer reproduce the "go slow" issue. Thanks! Tested-by: Juraj Marcin <[email protected]>
