As I see, sendfile uses MSS instead of MTU. Wouldn't it be better to scale also MSS along with MTU when PMTUD is enabled?
On Sun, Jun 4, 2023 at 5:40 PM Fotis Panagiotopoulos <f.j.pa...@gmail.com> wrote: > As said, this is sendfile(). I do not have control on the size of the > chunks sent. sendfile is also using TCP. > > So, sendfile cannot take advantage of PMTUD? > > On Sun, Jun 4, 2023 at 5:33 PM Xiang Xiao <xiaoxiang781...@gmail.com> > wrote: > >> On Sun, Jun 4, 2023 at 10:23 PM Fotis Panagiotopoulos < >> f.j.pa...@gmail.com> >> wrote: >> >> > I am trying this new PMTUD functionality, but it seems it doesn't work. >> > >> > I have configured: >> > CONFIG_NET_ICMP_PMTU_ENTRIES = 10 >> > CONFIG_NET_ICMP_PMTU_TIMEOUT = 10 >> > >> > Again I see lots of segments being sent (all with size 1400), and all of >> > them are responded with the same ICMP reply. >> > A couple of retransmissions are attempted, and then the connection is >> > reset. >> > >> >> The option can discover the minimal MTU from the source to the >> destination. >> You need to use PMTU to split your data into small packets(<= PMTU) by >> yourself to improve the efficiency. >> >> >> > >> > I try again the same procedure (now that the system had the chance to >> > discover the maximum PMTU), but it still fails. >> > Again all segments have a size of 1400, instead of less. >> > >> >> If you don't split your package to fit MTU by yourself, you have >> to enable NET_IPFRAG. But I would suggest that.you switch UDP to TCP >> because the protocol you implement on top of UDP is likely very >> inefficient. >> >> >> > >> > I am using sendfile() in case this matters. >> > >> > On Tue, May 30, 2023 at 12:28 AM Fotis Panagiotopoulos < >> > f.j.pa...@gmail.com> >> > wrote: >> > >> > > I can see that there are two different problems with MTU. >> > > They are completely independent from each other, so let's start with >> the >> > > simple case first. >> > > >> > > I am testing on an STM32F427, using Ethernet. >> > > >> > > As previously noted, the following code will cause the running task to >> > > hang. >> > > >> > > netlib_set_mtu(CONFIG_NETIF_DEV_NAME, 1500); >> > > >> > > int sd = socket(AF_INET, SOCK_DGRAM, 0); >> > > >> > > struct sockaddr_in server; >> > > server.sin_family = AF_INET; >> > > server.sin_port = 1000; >> > > server.sin_addr.s_addr = inet_addr("192.168.1.235"); >> > > >> > > uint8_t * data = malloc(2048); >> > > memset(data, 0xAA, 2048); >> > > >> > > sendto(sd, data, 2048, 0, (struct sockaddr*)&server, sizeof(server)); >> > > >> > > close(sd); >> > > >> > > As you can see, the MTU is set to 1500, and then I try to send a UDP >> > > datagram with a larger size (2048). >> > > Indeed `devif_send()` fails, and the aforementioned semaphore is never >> > > posted. >> > > >> > > (This is without buffering in UTP, in case this is important). >> > > >> > > This draft PR, provides a solution to the issue. >> > > https://github.com/apache/nuttx/pull/9423 >> > > >> > > If this is correct, I will also check buffered UDP, and other uses of >> > > devif_send(). >> > > >> > > Alternatively, devif_send() may be changed to actually return an error >> > > code (instead of returning void), so improved error handling can take >> > place. >> > > >> > > >> > > On Mon, May 29, 2023 at 1:41 PM Fotis Panagiotopoulos < >> > f.j.pa...@gmail.com> >> > > wrote: >> > > >> > >> The failure scenario is a bit more complicated... >> > >> >> > >> Give me some time and I will provide a correct and reproducible >> example, >> > >> with a clear explanation. >> > >> >> > >> On Mon, May 29, 2023, 13:27 Fotis Panagiotopoulos < >> f.j.pa...@gmail.com> >> > >> wrote: >> > >> >> > >>> > sendfile should return an error in this case, but senfile should >> only >> > >>> be >> > >>> > used with TCP, not UDP, since sendfile doesn't have any logic to >> ack >> > or >> > >>> > retry.. >> > >>> >> > >>> Sorry if this wasn't clear. This last test was with plain old >> > `send()`... >> > >>> >> > >>> I opened a UDP socket, and used `send()` to transmit a buffer larger >> > >>> than the MTU. >> > >>> Instead of getting an error, the application hangs indefinitely. >> > >>> `devif_send()` is called periodically, but of course it always >> fails. >> > >>> >> > >>> >> > >>> >> > >>> On Mon, May 29, 2023 at 1:13 PM Xiang Xiao < >> xiaoxiang781...@gmail.com> >> > >>> wrote: >> > >>> >> > >>>> On Mon, May 29, 2023 at 5:02 PM Fotis Panagiotopoulos < >> > >>>> f.j.pa...@gmail.com> >> > >>>> wrote: >> > >>>> >> > >>>> > > You need to enable IP fragmentation in this case, which is also >> > >>>> added >> > >>>> > > recently and disabled by default: >> > >>>> > > https://github.com/apache/nuttx/pull/8059 >> > >>>> > <https://github.com/apache/nuttx/pull/8059> >> > >>>> > > Otherwise, any packet bigger than MTU will be dropped silently. >> > >>>> > >> > >>>> > Yes, this is the expected behavior. >> > >>>> > But, instead of dropping the packet, the system hangs because the >> > >>>> semaphore >> > >>>> > is never posted. >> > >>>> > It just tries endlessly to call devif_send() which always fails. >> > >>>> > >> > >>>> > >> > >>>> sendfile should return an error in this case, but senfile should >> only >> > be >> > >>>> used with TCP, not UDP, since sendfile doesn't have any logic to >> ack >> > or >> > >>>> retry.. >> > >>>> >> > >>>> >> > >>>> > >> > >>>> > >> > >>>> > On Mon, May 29, 2023 at 11:42 AM Xiang Xiao < >> > >>>> xiaoxiang781...@gmail.com> >> > >>>> > wrote: >> > >>>> > >> > >>>> > > On Sun, May 28, 2023 at 11:55 PM Fotis Panagiotopoulos < >> > >>>> > > f.j.pa...@gmail.com> >> > >>>> > > wrote: >> > >>>> > > >> > >>>> > > > While experimenting with MTU, and checking the stability of >> my >> > >>>> system, >> > >>>> > I >> > >>>> > > > noticed the following. >> > >>>> > > > >> > >>>> > > > I try to send a UDP datagram that is larger than the >> configured >> > >>>> MTU. >> > >>>> > > > In this case, the offending thread seems to hang indefinitely >> > (or >> > >>>> at >> > >>>> > > least >> > >>>> > > > waiting for a very long timeout?) >> > >>>> > > > >> > >>>> > > >> > >>>> > > You need to enable IP fragmentation in this case, which is also >> > >>>> added >> > >>>> > > recently and disabled by default: >> > >>>> > > https://github.com/apache/nuttx/pull/8059 >> > >>>> > > Otherwise, any packet bigger than MTU will be dropped silently. >> > >>>> > > >> > >>>> > > >> > >>>> > > > The problem seems to be this line: >> > >>>> > > > >> > >>>> > > > >> > >>>> > > >> > >>>> > >> > >>>> >> > >> https://github.com/apache/nuttx/blob/master/net/udp/udp_sendto_unbuffered.c#L197 >> > >>>> > > > `devif_send()` fails because the datagram is too large, but >> > >>>> > > > `pstate->st_sem` is never posted (the code returns >> immediately). >> > >>>> > > > >> > >>>> > > > This leaves the sending task to be blocked here: >> > >>>> > > > >> > >>>> > > > >> > >>>> > > >> > >>>> > >> > >>>> >> > >> https://github.com/apache/nuttx/blob/master/net/udp/udp_sendto_unbuffered.c#L469 >> > >>>> > > > >> > >>>> > > > Shouldn't this failure also post the semaphore? >> > >>>> > > > And let the code proceed returning an error in `send()`? >> > >>>> > > > >> > >>>> > > > >> > >>>> > > > On Sun, May 28, 2023 at 5:26 PM Fotis Panagiotopoulos < >> > >>>> > > f.j.pa...@gmail.com >> > >>>> > > > > >> > >>>> > > > wrote: >> > >>>> > > > >> > >>>> > > > > >> > >>>> > > > > On Sat, May 27, 2023 at 5:35 PM Xiang Xiao < >> > >>>> > xiaoxiang781...@gmail.com> >> > >>>> > > > > wrote: >> > >>>> > > > > >> > >>>> > > > >> On Sat, May 27, 2023 at 8:19 PM Fotis Panagiotopoulos < >> > >>>> > > > >> f.j.pa...@gmail.com> >> > >>>> > > > >> wrote: >> > >>>> > > > >> >> > >>>> > > > >> > Hello, >> > >>>> > > > >> > >> > >>>> > > > >> > I encounter some problems using sendfile(). >> > >>>> > > > >> > >> > >>>> > > > >> > I am using sendfile to... send a file to a remote >> server, >> > >>>> with my >> > >>>> > > own >> > >>>> > > > >> > implementation of an FTP client. >> > >>>> > > > >> > sendfile() indeed starts to transmit chunks of the file, >> > but >> > >>>> as I >> > >>>> > > see >> > >>>> > > > in >> > >>>> > > > >> > Wireshark, I get an ICMP response "Destination >> unreachable >> > >>>> > > > >> (Fragmentation >> > >>>> > > > >> > needed)". >> > >>>> > > > >> > I have verified that the Ethrenet MTU is correctly set >> to >> > >>>> 1500. >> > >>>> > > > >> > >> > >>>> > > > >> > I tried lowering the MTU a lot (1000 bytes), and the >> > problem >> > >>>> is >> > >>>> > > > solved. >> > >>>> > > > >> > Communication succeeds. >> > >>>> > > > >> > >> > >>>> > > > >> > This raises some questions, and indicates some potential >> > >>>> bugs: >> > >>>> > > > >> > >> > >>>> > > > >> > 1. Why is there a problem with MTU in the first place? >> > >>>> Shouldn't >> > >>>> > MTU >> > >>>> > > > be >> > >>>> > > > >> > negotiated? (Is this functionality available in NuttX?) >> > >>>> > > > >> > >> > >>>> > > > >> >> > >>>> > > > >> MTU isn't negotiated but a physical attribute of your >> > >>>> > > transport(netdev). >> > >>>> > > > >> On >> > >>>> > > > >> the other hand, PMTU could be discovered from ICMP. >> > >>>> > > > >> >> > >>>> > > > > >> > >>>> > > > > I am not very familiar with MTU negotiation, so it seems >> that >> > it >> > >>>> > > doesn't >> > >>>> > > > > happen in the network layer that I thought... >> > >>>> > > > > >> > >>>> > > > > >> > >>>> > > > >> >> > >>>> > > > >> >> > >>>> > > > >> > 2. Why is the ICMP response not handled? It seems that >> > >>>> sendfile() >> > >>>> > > just >> > >>>> > > > >> > ignores it and continues to send chunks, nevertheless. >> > >>>> > > > >> > >> > >>>> > > > >> >> > >>>> > > > >> It is handled by the recent addition here: >> > >>>> > > > >> https://github.com/apachey/nuttx/pull/9254 >> > >>>> > > > >> <https://github.com/apache/nuttx/pull/9254> >> > >>>> > > > >> but this feature is disabled by default, you have to >> enable >> > it >> > >>>> > > > manually.. >> > >>>> > > > >> >> > >>>> > > > > >> > >>>> > > > > I will definitely take a look at this. Thank you. >> > >>>> > > > > >> > >>>> > > > > >> > >>>> > > > >> >> > >>>> > > > >> >> > >>>> > > > > >> > >>>> > > > >> > 3. Why sendfile() sends TCP segments without receiving >> any >> > >>>> ACKs >> > >>>> > > back? >> > >>>> > > > >> > AFAIK, depending on the configuration, TCP allows at >> most >> > two >> > >>>> > > pending >> > >>>> > > > >> > segments on the wire. But I see dozens of them, till >> > sendfile >> > >>>> > > finally >> > >>>> > > > >> > fails. >> > >>>> > > > >> > >> > >>>> > > > >> > >> > >>>> > > > >> Why only two segments? TCP can send packages until the >> slide >> > >>>> window >> > >>>> > is >> > >>>> > > > >> full. >> > >>>> > > > >> >> > >>>> > > > >> Disregard this. I was confused with delayed ACKs. Which >> is a >> > >>>> > > receiver's >> > >>>> > > > > functionality, not a sender's... >> > >>>> > > > > >> > >>>> > > > > >> > >>>> > > > >> >> > >>>> > > > >> > This last point is also verified in my MQTT client. >> > >>>> > > > >> > I have seen NuttX TCP allowing sending lots of TCP >> segments >> > >>>> > without >> > >>>> > > > >> ACKing >> > >>>> > > > >> > the previous data. >> > >>>> > > > >> > >> > >>>> > > > >> > So, is there any insight on the above? >> > >>>> > > > >> > Is my configuration wrong, or is there anything wrong >> with >> > >>>> TCP? >> > >>>> > > > >> > >> > >>>> > > > >> > Thank you. >> > >>>> > > > >> > >> > >>>> > > > >> >> > >>>> > > > > >> > >>>> > > > >> > >>>> > > >> > >>>> > >> > >>>> >> > >>> >> > >> >