Yes, from a user’s point of view the unbuffered send operation is blocking. I just meant that unbuffered send (the kernel side) does not wait for each TCP packet to be acknowledged. E.g. apps/netutils/iperf uses a user buffer with size of 16384 bytes. While these 16384 bytes are being sent, the performance is fine. The performance is reduced only at the end of the buffer because the TCP stack waits for the ACK acknowledging the last segment of the buffer. I want to say that unbuffered send mode does not look bad to me for some cases.
On Thu, 2021-10-14 at 13:25 -0600, Gregory Nutt wrote: > > Why does the send operation block > > It has to, at least for TCP. The data resides in a user provided > buffered > .. that is why it is unbuffered. In TCP, it may need to retransmit > if the > data is not ACKed. Hence the user buffer must stay intact until the > ACK is > received. > > The fully buffered logic does not have to wait for the ACK but can > return > as soon as the data is copied out of the user buffer. > > On Thu, Oct 14, 2021 at 12:54 PM Alexander Lunev > <alexanderlu...@mail.ru.invalid> wrote: > > > Hi Gregory, > > > > > In the unbuffered send case, the send operation blocks until the > > > packet is > > > sent and ACKed. Per RFC 1122, the peer may delay sending the ACK > > > for > > > up to > > > 500 Msec. So the performance of the unbuffered send is abysmal > > > when > > > sending to an RFC 1122 client. > > > > Why does the send operation block? > > As I can see in the source code of the unbuffered send operation, > > TCP > > packets are sent w/o waiting for ACKing each sent packet. The > > packets > > are sent from NuttX until the remote TCP receiver's window covers > > the > > total size of not acknowledged sent data. > > > > > I think that the unbuffered send logic is generally very bad and > > > I > > > have > > > suggested removing it in the past. The buffered send performance > > > is > > > much > > > better. We decided to retain the unbuffered send only because it > > > does use > > > less memory since no write buffering is required. > > > > Currently I'm using "unbuffered" send mode as in my case it > > surprisingly provides twice as high throughput as "buffered" one. > > Though, I initially expected that "buffered" send mode should have > > better performance compared to "unbuffered" one. > > > > BTW, could you please say where I could read that discussion about > > removing / retaining the unbuffered send in more details? > > > > Best regards, > > Alexander Lunev. > > > > > > On Thu, 2021-10-14 at 16:49 +0800, Gregory Nutt wrote: > > > For people that need some background, this Wikipedia article may > > > be > > > helpful: > > > https://en.wikipedia.org/wiki/TCP_delayed_acknowledgment. > > > The > > > SPLIT packet change is intended to work around issues when the > > > unbuffered > > > send is used with a peer that supports RFC 1122. > > > > > > In the unbuffered send case, the send operation blocks until the > > > packet is > > > sent and ACKed. Per RFC 1122, the peer may delay sending the ACK > > > for > > > up to > > > 500 Msec. So the performance of the unbuffered send is abysmal > > > when > > > sending to an RFC 1122 client. > > > > > > The split basically breaks each TCP send into two packets. RFC > > > 1122 > > > does > > > *require* that an ACK be sent after two packets are received. If > > > you > > > are > > > seeing different behavior, then those peers are not conforming to > > > RFC > > > 1122. Perhaps there is some other standard. > > > > > > I think that the unbuffered send logic is generally very bad and > > > I > > > have > > > suggested removing it in the past. The buffered send performance > > > is > > > much > > > better. We decided to retain the unbuffered send only because it > > > does use > > > less memory since no write buffering is required. > > > > > > I am not 100% sure of this, but I still think that it would be > > > better > > > to > > > remove the unbuffered TCP send logic rather than remove the > > > packet > > > split > > > logic. I don't see how removing the split makes the unbuffered > > > send > > > better. It does not fix anything and certainly breaks the RFC > > > 1122 > > > case. > > > The unbuffered TCP send behavior is bad without the split and, > > > according to > > > your PR, it is also bad with the split. But my feeling is not > > > strong > > > so > > > just take that as my 2 cents worth. > > > > > > Greg > > > > > > On Tue, Oct 12, 2021 at 8:10 PM Takashi Yamamoto > > > <yamam...@midokura.com.invalid> wrote: > > > > > > > hi, > > > > > > > > i submitted a PR to remove NET_TCP_SPLIT. > > > > https://github.com/apache/incubator-nuttx/pull/4660 > > > > > > > > i'm posting this here because a feature removal should be > > > > reviewed > > > > by > > > > wider audience. > > > > especially, if you are relying on it, please speak up. > > > >
