Re: [lwip-users] http server problems/bug

[Sergio R. Caprile]

> 1. We got the web server original code from   STM32 provided example
> "LwIP_HTTP_Server_Raw" , but also mixed with 1.4.1 contrib, which are 
> quite the same.


Then go to ST, "mixed" and "quite the same" is not the thoroughly tested 
server in the contrib tree. It is your code, your debugging. If you need 
help, please post the changes.


> some html files include javascript(jquery1.12.4) ,thus the browser can
> call the function, send and get back required infomation from MCU,
> this is our "dynamic" part of web processing, which is very simple,
> nothing more.

it might be "very simple" but it is your CGI, your code, your debugging. 
If you need help, please post your code (in a synthetic format, please, 
we just need to see where you call lwIP functions and your return values).


You seem to have a memory trashing problem, it is either one of the 
tasks, your server, your CGI, or as Simon pointed out your (ST's ?) DMA 
driver.




___
lwip-users mailing list
lwip-users@nongnu.org
https://lists.nongnu.org/mailman/listinfo/lwip-users

Re: [lwip-users] http server problems/bug

[lampo]

right, but one thing I am pretty sure,  all of the communication  between PC
browser and our  MCU , processed  in one single thread, ie tcpip_thread.

1. We got the web server original code from   STM32 provided example
"LwIP_HTTP_Server_Raw" , but also mixed with 1.4.1 contrib, which are quite
the same.

2. Here is the design of webserver part, 

   we use makefsdata.exe/makefsdata.pl  to generate the file '"fsdata.c" 
from original web pages  containg html, pictures,etc. 
"fsdata.c"  is nothing more but some arrays to store related contens of web
pages.

  some html files include javascript(jquery1.12.4) ,thus the browser can
call the function, send and get back required infomation from MCU,
this is our "dynamic" part of web processing, which is very simple, nothing
more.

3. we just modified the upper layer of " http server raw" example, no pbuf
related memory is changed. 
besides, we have opened SYS_LIGHTWEIGHT_PROT.

Anyway, I will try the webserver alone without other tcp/ip related threads. 



--
View this message in context: 
http://lwip.100.n7.nabble.com/http-server-problems-bug-tp26999p27053.html
Sent from the lwip-users mailing list archive at Nabble.com.

___
lwip-users mailing list
lwip-users@nongnu.org
https://lists.nongnu.org/mailman/listinfo/lwip-users

Re: [lwip-users] http server problems/bug

[lampo]

ok,  maybe it's  different in our situation. For the browser had already sent 
the 'ack' in fact, and it will not  send it again.   So it's meaningless to  
make  
the receiver part of  web server open to wait for 'ack' from the  browser.

"the upper layer", I mean relative to low_level_input,  for I  set breakpoint 
in 'case FIN_WAIT_2" in function "tcp_process", and occasionally  it will not 
reache here after
the web server sends [FIN,PSH].  

 that means somehow the ack message is lost or ignored in the level "tcp_input" 
(in low_level_input, we have count the number of received message, 
and compare it to the number of message sent by the browser with the help of  
wireshark, it's the same.)

Thus, the connection of webserver is still open(stay in FIN_WAIT_1 state)  
while the the connection of browser is closed.
Then, bad things will happen when the webserver retransmit [FIN,PSH], and  the 
browser just responds [RST], resulting in all tcp connections blocked or failed.

we cannot figure out how to avoid all tcp connections breaking down when lwip 
receives [RST]  (our web server is inside the tcpip_thread).
we don't know what makes the connection not entering FIN_WAIT_2 state.  we had 
debug and trace the code, just not that easy to find out.






 
From: Noam weissman [via lwIP]
Date: 2016-08-07 01:41
To: lampo
Subject: Re: http server problems/bug
Hi,

Every socket has a sender part and a receiver part. Normally it is full duplex 
!.

TCP has layers. Some data is handled inside the TCP stack MAC + L1 and some is
handled in L2,L3 etc...

When an ACK or some other low level message arrives it is handled inside the TCP
and never get to the upper layers.

I do not know the exact internal mechanism but what I know that when you call 
tcp_close
it shuts down the sender and receiver handling inside the TCP stack for the 
open connection.
It frees any internal memory and deletes the related PCB.

If you shut down the sender only using tcp shutdown function the connection is 
still is open for 
the receiver part. That means that is the remote side closes your tcp_recv call 
back will be triggered
with data that has ZERO length. Also the TCP stack will respond (low level) to 
that message by acknowledging it.

http://stackoverflow.com/questions/4160347/close-vs-shutdown-socket


BR,
Noam.
c - close vs shutdown socket? - Stack Overflow
stackoverflow.com
In C, I understood that if we close a socket, it means the socket will be 
destroyed and can be re-used later. How about shutdown? The description said it 
closes half ...





From: lwip-users <lwip-users-bounces+noam=[hidden email]> on behalf of lampo 
<[hidden email]>
Sent: Saturday, August 6, 2016 6:37 AM
To: [hidden email]
Subject: Re: [lwip-users] http server problems/bug 
 

> On LwIP side when you close a connection using the tcp_close it closes
> ALL.

Does that mean it closes all other connections including other independent
threads,like tcp client ,tcp server in our multithreaded system? 


> From having problems in some TCP module that I wrote I found that if I
> call tcp_shutdown and only
> define shutdown the TX side the connection is closed only after getting
> the ACK.

sorry,I can't quite get it. In our example, you mean TX side is our web
server, RX side is browser from PC, right?But browser has already 
got the ack from web server after it sends out FIN,and close the
connection,while in fact the TX side does not closed totally. 

can you show the process in following format ,so I can get it precisely?

 web server   <-- GET  HTTP/1.1-   web Client 
 web server (FIN_WAIT_1)   FIN,PSH--->   web Client 
 web server (FIN_WAIT_2)  < ACK-  web Client 
 (web server will stay in FIN_WAIT_2 for we find it receives no ACK here )
 web server < FIN--  web Client 
 web server   ACK->  web Client 
(web client receives the ack, so it  close the RX side connection, thus it
cannot respond to a  FIN now)




--
View this message in context: 
http://lwip.100.n7.nabble.com/http-server-problems-bug-tp27017p27032.html 
lwip-users - http server problems/bug
lwip.100.n7.nabble.com
http server problems/bug. With more infomation provided, I re-post it here(I 
post it through nabble before), sorry for that. Problem, web server <-- GET 
HTTP/1.1- ...

Sent from the lwip-users mailing list archive at Nabble.com.

___
lwip-users mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/lwip-users

___ 
lwip-users mailing list 
[hidden email] 
https://lists.nongnu.org/mailman/listinfo/lwip-users 



If you reply to this email, your message will be added to the discussion below:
http://lwip.100.n7.nabble.com/http-server-p

Re: [lwip-users] http server problems/bug

[Noam Weissman]

Hi,


Every socket has a sender part and a receiver part. Normally it is full duplex 
!.


TCP has layers. Some data is handled inside the TCP stack MAC + L1 and some is

handled in L2,L3 etc...


When an ACK or some other low level message arrives it is handled inside the TCP

and never get to the upper layers.


I do not know the exact internal mechanism but what I know that when you call 
tcp_close

it shuts down the sender and receiver handling inside the TCP stack for the 
open connection.

It frees any internal memory and deletes the related PCB.


If you shut down the sender only using tcp shutdown function the connection is 
still is open for

the receiver part. That means that is the remote side closes your tcp_recv call 
back will be triggered

with data that has ZERO length. Also the TCP stack will respond (low level) to 
that message by acknowledging it.


http://stackoverflow.com/questions/4160347/close-vs-shutdown-socket


<http://stackoverflow.com/questions/4160347/close-vs-shutdown-socket>


<http://stackoverflow.com/questions/4160347/close-vs-shutdown-socket>

BR,<http://stackoverflow.com/questions/4160347/close-vs-shutdown-socket>

Noam.
<http://stackoverflow.com/questions/4160347/close-vs-shutdown-socket>

[http://cdn.sstatic.net/Sites/stackoverflow/img/apple-touch-i...@2.png?v=73d79a89bded]<http://stackoverflow.com/questions/4160347/close-vs-shutdown-socket>

c - close vs shutdown socket? - Stack 
Overflow<http://stackoverflow.com/questions/4160347/close-vs-shutdown-socket>
stackoverflow.com
In C, I understood that if we close a socket, it means the socket will be 
destroyed and can be re-used later. How about shutdown? The description said it 
closes half ...





From: lwip-users <lwip-users-bounces+noam=silrd@nongnu.org> on behalf of 
lampo <lbg...@163.com>
Sent: Saturday, August 6, 2016 6:37 AM
To: lwip-users@nongnu.org
Subject: Re: [lwip-users] http server problems/bug


> On LwIP side when you close a connection using the tcp_close it closes
> ALL.

Does that mean it closes all other connections including other independent
threads,like tcp client ,tcp server in our multithreaded system?


> From having problems in some TCP module that I wrote I found that if I
> call tcp_shutdown and only
> define shutdown the TX side the connection is closed only after getting
> the ACK.

sorry,I can't quite get it. In our example, you mean TX side is our web
server, RX side is browser from PC, right?But browser has already
got the ack from web server after it sends out FIN,and close the
connection,while in fact the TX side does not closed totally.

can you show the process in following format ,so I can get it precisely?

 web server   <-- GET  HTTP/1.1-   web Client
 web server (FIN_WAIT_1)   FIN,PSH--->   web Client
 web server (FIN_WAIT_2)  < ACK-  web Client
 (web server will stay in FIN_WAIT_2 for we find it receives no ACK here )
 web server < FIN--  web Client
 web server   ACK->  web Client
(web client receives the ack, so it  close the RX side connection, thus it
cannot respond to a  FIN now)




--
View this message in context: 
http://lwip.100.n7.nabble.com/http-server-problems-bug-tp27017p27032.html
lwip-users - http server 
problems/bug<http://lwip.100.n7.nabble.com/http-server-problems-bug-tp27017p27032.html>
lwip.100.n7.nabble.com
http server problems/bug. With more infomation provided, I re-post it here(I 
post it through nabble before), sorry for that. Problem, web server <-- GET 
HTTP/1.1- ...


Sent from the lwip-users mailing list archive at Nabble.com.

___
lwip-users mailing list
lwip-users@nongnu.org
https://lists.nongnu.org/mailman/listinfo/lwip-users
___
lwip-users mailing list
lwip-users@nongnu.org
https://lists.nongnu.org/mailman/listinfo/lwip-users

Re: [lwip-users] http server problems/bug

[lampo]

> On LwIP side when you close a connection using the tcp_close it closes
> ALL.

Does that mean it closes all other connections including other independent
threads,like tcp client ,tcp server in our multithreaded system? 


> From having problems in some TCP module that I wrote I found that if I
> call tcp_shutdown and only
> define shutdown the TX side the connection is closed only after getting
> the ACK.

sorry,I can't quite get it. In our example, you mean TX side is our web
server, RX side is browser from PC, right?But browser has already 
got the ack from web server after it sends out FIN,and close the
connection,while in fact the TX side does not closed totally. 

can you show the process in following format ,so I can get it precisely?

 web server   <-- GET  HTTP/1.1-   web Client 
 web server (FIN_WAIT_1)   FIN,PSH--->   web Client 
 web server (FIN_WAIT_2)  < ACK-  web Client 
 (web server will stay in FIN_WAIT_2 for we find it receives no ACK here )
 web server < FIN--  web Client 
 web server   ACK->  web Client 
(web client receives the ack, so it  close the RX side connection, thus it
cannot respond to a  FIN now)




--
View this message in context: 
http://lwip.100.n7.nabble.com/http-server-problems-bug-tp27017p27032.html
Sent from the lwip-users mailing list archive at Nabble.com.

___
lwip-users mailing list
lwip-users@nongnu.org
https://lists.nongnu.org/mailman/listinfo/lwip-users

Re: [lwip-users] http server problems/bug

[goldsi...@gmx.de]

Sergio R. Caprile wrote:

  From what I've seen, the connection gets closed, and tens of
connections later the web server starts resending TCP stuff like data
and ACKs. It looks like trashed TCP memory or a queue somewhere.


Just from reading these lines: this could be caused by a buggy 
DMA-enabled driver, as well...


Simon

___
lwip-users mailing list
lwip-users@nongnu.org
https://lists.nongnu.org/mailman/listinfo/lwip-users

Re: [lwip-users] http server problems/bug

[Sergio R. Caprile]

Hi Noam,
the OP seems to be using lwIP's web server.
From what I've seen, the connection gets closed, and tens of 
connections later the web server starts resending TCP stuff like data 
and ACKs. It looks like trashed TCP memory or a queue somewhere.




___
lwip-users mailing list
lwip-users@nongnu.org
https://lists.nongnu.org/mailman/listinfo/lwip-users

Re: [lwip-users] http server problems/bug

[Sergio R. Caprile]

... and by looking at your capture file I discover you might not be just 
"using the web server"...


You are serving XML files.
I tend to think that means those are not static files, I may infer that 
you are generating those pages dinamically. So:

- Are you serving static XML pages ? I don't think so
- Are you using SSI ? I see some floats, I don't think so
- Are you using a CGI to generate the XML ?
* Then, How is your code ? I'd like to see your pbuf and memory usage.
- Where did you get the seb server from ? Is it the 1.4.1 in the contrib 
tree ? Is it the 2.0.0RC2 or git head ? Did your vendor provide it ? Did 
you check for mods ?


Anyway, please try the server alone. What I see in the capture file is 
pretty interesting, looks like connections closed long ago suddenly 
revive, as if TCP had a time jump to the past. Looks like someone is 
messing with lwIP's memory.




___
lwip-users mailing list
lwip-users@nongnu.org
https://lists.nongnu.org/mailman/listinfo/lwip-users

Re: [lwip-users] http server problems/bug

[Noam Weissman]

Hi,


Although I have not implemented the following in the HTTP server I am using (I 
think I should)

think about it.


Normal HTTP flow:

Browser sends a request to server

Server responds and closes the connection.


On LwIP side when you close a connection using the tcp_close it closes ALL.


From having problems in some TCP module that I wrote I found that if I call 
tcp_shutdown and only

define shutdown the TX side the connection is closed only after getting the ACK.


This way transmit is closed but RX is a live and can respond to a FIN etc...


Try replacing the tcp_close with  tcp_shutdown(pcb, FALSE, TRUE);

The above will close the TX side only.


Here is a similar problem:  
http://www.reactos.org/pipermail/ros-diffs/2013-April/048568.html



Hope that helped... please update us []


BR,

Noam.




From: lwip-users  on behalf of 
花生壳用户 
Sent: Friday, August 5, 2016 5:56 AM
To: lwip-users@nongnu.org
Subject: [lwip-users] http server problems/bug

With more infomation provided, I re-post it here(I post it through nabble 
before), sorry for that.

Problem,

 web server   <-- GET  HTTP/1.1-   web Client
 web server (FIN_WAIT_1)   FIN,PSH--->   web Client
 web server (FIN_WAIT_2)  < ACK-  web Client
 (sometimes ACK not received by web server !!  we find it does not enter 
FIN_WAIT_2 state, but the low_level_input actually received the ‘ACK’ )

 web server < FIN--  web Client
 web server   ACK->  web Client
 (..some time like half minutes later, re-transmission occurs)
 web serverFIN,PSH--->   web client
 web server   <--RST- web client

 (it seems the whole pcb(tcp_pcb) is reset ! Because all other connections fail 
,including tcp client which is an independent thread )
Then after a long time(even minutes later), all connection recovered.


Testing Conditions,
Lwip 1.4.1 ,   web server is based on example of ‘httpserver_raw’, with much 
more communication throughput, also with some dynamic web pages.
web server is inside the tcpip_thread  rather than an independent thread.

we can ensure that all the APIs is called properly in our multithreaded system.

we have tested all of our lwip related applications, like  tcp client, tcp 
server, udp, sntp, which are running in our system at the same time, for 
several weeks , all ok, except for web server. we also have tried the web 
server alone,without other applications in our system,but result is the same.

Some PC occurs much more frequently while others seldom occurs.

web server(MCU,STM32) IP :  192.168.0.10
web client(PC,IE browser) IP:  192.168.0.14

wireshark capture file 'wireshark.pcapng' is attached.



Our doubts,
1. Why the upper level didn’t receive ‘ACK’ while low_level_input did?
2. In http server application, how to avoid reseting the whole tcp pcb(if my 
analysis is right) when ACK is really missing from the low level hardware(it 
happens under poor environment, right?)

Any advice is really appreciated !





___
lwip-users mailing list
lwip-users@nongnu.org
https://lists.nongnu.org/mailman/listinfo/lwip-users

Re: [lwip-users] http server problems/bug

[lampo]

sorry,  I　post it  through mail  lwip-users@nongnu.org now.



--
View this message in context: 
http://lwip.100.n7.nabble.com/http-server-problems-bug-tp26999p27016.html
Sent from the lwip-users mailing list archive at Nabble.com.

___
lwip-users mailing list
lwip-users@nongnu.org
https://lists.nongnu.org/mailman/listinfo/lwip-users

Re: [lwip-users] http server problems/bug

[Sergio R. Caprile]

> 2.we did post the capture files “wireshark.pcapng” ,which is just
> above the snapshots.

Not in this list, maybe in nabble, but many of us don't use nabble.

I would try the server alone, you might have some task leaking.

___
lwip-users mailing list
lwip-users@nongnu.org
https://lists.nongnu.org/mailman/listinfo/lwip-users

Re: [lwip-users] http server problems/bug

[lampo]

yeah, 
1.web server is RAW API, and we do call the API in the same thread ,say
tcpip_thread; we use socket APIs
in other threads.  we can ensure that all the APIs is called properly in our
multithreaded system.

2.we did post the capture files “wireshark.pcapng” ,which is just above the
snapshots.

3.right, NO_SYS = 0;  we have tested all of our lwip related applications,
like  tcp client, tcp server, udp, sntp, which are running in our system at
the same time, for several weeks , all ok, except for web server.

4.The problem I described happens not all the time, but occasionally. Thus
the original simple example may work all the time. Our web server is based
on that example just with much more communication throughput.

thanks. 



--
View this message in context: 
http://lwip.100.n7.nabble.com/http-server-problems-bug-tp26999p27006.html
Sent from the lwip-users mailing list archive at Nabble.com.

___
lwip-users mailing list
lwip-users@nongnu.org
https://lists.nongnu.org/mailman/listinfo/lwip-users

Re: [lwip-users] http server problems/bug

[Sergio R. Caprile]

I saw this
> fail ,including tcp client which is an independent thread )
and then
> *Testing Conditions*
> Lwip 1.4.1 ,   web server is based on example of ‘httpserver_raw’,
> that means the web server is inside the tcpip_thread  rather than an
> independent thread.

All lwIP RAW API functions must be called on the very same
thread/execution context. That web server is RAW API.
Applications in other threads must use netconn or socket APIs
Frame reception must happen in the same thread the web server runs

My advices:

 please do not post capture snapshots (or text copies...), post capture
files. And please post the whole session...

Please describe your environment, yours is a NO_SYS = 0 system, is it
that way ? have you checked the driver ? the port ? Did you happen to
try a simple known-to-work app before jumping into a full multithreaded
system ?



___
lwip-users mailing list
lwip-users@nongnu.org
https://lists.nongnu.org/mailman/listinfo/lwip-users