Hi Joel,
I agree that setting the window update threshold below 1 MSS would be
bad. I didn't realize your window was being updated in such small
amounts. Is there a reason why your application isn't continuing to
drain the buffer?
Data being sent to the device in fact are commands. Commands have
different sizes and different execution time. Device reads commands from
the buffer one by one and executes them. So if the buffer was really
full and after that device read a long time command with size less than
the TCP window update threshold I get a described situation.
In terms of receiving a zero-window probe. Is LwIP ACKing the probe
because you technically do have window space available because your
application read some data, but not enough to trigger the threshold?
If that's the case, then it sounds like LwIP is behaving correctly.
Yes, this is my case. And I think that LwIP is correct too.
If LwIP is continuing to accept data even though the receive buffer is
full and real window is 0, then that's a problem.
No, real windows is not zero.
A zero window probe that is borrowing the probe byte of the next
unsent segment is normal and in this case, the it sounds like the
window has space, but we aren't advertising that space. Eventually
the window should fully fill up and LwIP would (hopefully) stop ACKing
zero window probe.
Yes, you right, but the problem a little deeper. LwIP knows about window
size, but really has not own buffer for received bytes. Instead it sends
every FRAGMENT (as pointer) to application's mail box.
In other words I must set window size in bytes for LwIP, but must make
mailbox operated with fragments of unknown length. In worst case my
mailbox must be able to store every 1-byte fragment. This wastes RAM.
Moreover communication will completely hangup on mailbox overflow. See
an example below:
Current LwIP behavior:
-> Seq=1, L=100
<- Ack=101, W=0
-> Seq=101, L=1 (Zero window probe)
<- Ack=102, W=0 - this is a current LwIP behavior even if fragment was
not really accepted by mailbox. This case the fragment hangs in
refused_data pointer and no further ACKs are possible.
-> Seq=102, L=1
-> Seq=102, L=1
-> Seq=102, L=1
-> RST
It would be better if LwIP didn't send ACK for the fragment it really
can't accept:
-> Seq=1, L=100
<- Ack=101, W=0
-> Seq=101, L=1 (Zero window probe)
<- Ack=101, W=0 !!! Note that ACK=101, not 102 as in previous example. 1
byte was rejected by mailbox and stored to refused_data pointer. We will
ignore all further data but ACK incoming packets
-> Seq=102, L=1
<- Ack=101, W=0
-> Seq=102, L=1
<- Ack=101, W=0
....
Such behavior will not cause connection reset.
Joel
On Nov 07, 2016, at 01:16 PM, Oleg Gladyshev <gla...@mail.ru> wrote:
Joel, thank you for your reply.
Oleg,
In your original post, you mentioned that your application reads
some data out of the buffer, but this doesn't result in a window
update. This is because LwIP waits until 1/4 of the window has
been opened up. There is a setting that controls the update
threshold in opt.h. You can adjust this to match your use case.
Yes, it is normal behavior to aviod silly window if I right. I know
that I can reduce threshold, but IMHO it is not good solution. I can
change threshold from N to M bytes, but for M bytes the problem still
here. In worst case I need exactly M-1 mbox capacity. Obviosly I
can't set threshold to few bytes because it will disable silly window
avoidance.
I'm also surprised that when LwIP ACKs the zero window probe, it
doesn't contain the updated window value.
Yes, it's true for 1.4.1 version. And it looks right for me because
this time real (internal) window size still less than the threshold.
What top-level API is your application using, sockets or Raw?
With the Raw API, I believe you have to call tcp_recved() after
your application receives data out of the buffer to update the
window.
No, I use netconn API which calls tcp_recved() internally.
Joel
On Nov 07, 2016, at 10:03 AM, Oleg Gladyshev <gla...@mail.ru
<mailto:gla...@mail.ru>> wrote:
Hi Noam,
Thank you very much for your reply but I think you misunderstood
me.
Look at the article: https://wiki.wireshark.org/TCP%20ZeroWindow
Of course my host (controlled by Windows now) sends data using
quite large packets. But when device's buffer becomes full
host starts periodical checking of the buffer. Windows performs
it using 1 byte packets called "Zero Window Probe packet" and we
can't affect this behavior.
Hi,
First of all if you have control over the host machine avoid
sending 1 byte. This is not efficient.
I suggest adding a dual mechanism. Defining size and/or timeout.
I mean adding some mechanism that counts bytes ready to be
sent. If you have more than X
Bytes you send them out. Also add a timeout. Meaning if you
have collected less then X bytes
but a time period has elapsed since the last send, send it
anyway. This way you force the sender
to send a minimum sized buffer and not send out 1 byte at a
time with a large overhead...
At your device side you probably have a DMA collecting data.
You may modify the code and check
How many bytes have you received. Similar to the above
mechanism. If you got less than some limit
you do not allocate a pBuf ... only if you have a minimum
data size or a time difference from last
data handling you allocate a pBuf, copy data to it and let
the TCP stack handle it.
BR,
Noam.
-----Original Message-----
From: lwip-users
[mailto:lwip-users-bounces+noam=silrd....@nongnu.org] On
Behalf Of Oleg Gladyshev
Sent: Monday, November 07, 2016 3:23 PM
To: lwip-users@nongnu.org <mailto:lwip-users@nongnu.org>
Subject: [lwip-users] Zero window and refused data problem
Hi all,
I develop a device which receives data from host machine
using TCP (LwIP 1.4.1). Host sends data continuously queuing
it to the controller.
Controller process data as fast as possible, but data
process time is not determined. it can be from milliseconds
to hours. And I wish to use TCP/IP incoming buffer as buffer
for the data.
In my case TCP window on the device becomes full very often.
Sometimes device's application receives from the TCP stack
portion of data less than TCP windows size. This case LwIP
doesn't update receive window leaving it zero. But the
device able to receive some bytes now. And when the host
sends 1 byte ZeroProbe packet device receives it.
For this one byte LwIP allocates new pbuf structure and
sends it to application using sys_mbox_xxx call.
I use fixed-size mbox queues so after some probe packets
from the host mbox becomes full and refuses new data.
tcp_input notices this and tries to deliver refused data to
the application every time it called.
But if application still refuses new data tcp_input()
doesn't send ACK to the host! And host disconnects after
some tries.
Making my mboxes dynamically doesn't solve the problem
because in general we can waste all RAM with (TCP_WND-1)
pbuf structures for every incoming byte from window probe
packets.
I think it would be better to ACK with previous value to the
host instead of silently dropping new data. What do you
think about the issue? What is the best way to avoid
disconnects in my project?
--
Best regards, Oleg
_______________________________________________
lwip-users mailing list
lwip-users@nongnu.org <mailto:lwip-users@nongnu.org>
https://lists.nongnu.org/mailman/listinfo/lwip-users
_______________________________________________
lwip-users mailing list
lwip-users@nongnu.org
https://lists.nongnu.org/mailman/listinfo/lwip-users
--
С уважением,
Олег Гладышев
_______________________________________________
lwip-users mailing list
lwip-users@nongnu.org
https://lists.nongnu.org/mailman/listinfo/lwip-users
