Re: [twsocket] Pause and Resume just not working!
Hi Francois Piette, - By the way: If you have no Line Mode then ICS don't do buffering at all. It just uses send of Winsock! This sentence is wrong. TWSocket use a FIFO buffer when you call Send. Whatever data size you send, TWSocket will use a list of buffers (Each having a BufSize length wich by default is the largest data packet accepted on Ethernet so that fragmentation is the least possible). Each time you cann Send, TWSocket tries to deliver data from his FIFO to winsock. Each time winsock notifies that he has room into his own buffer, then TWSocket extract data from his FIFO and give it to winsock. Eventually, TWSocket empty his buffer and then triggers OnDataSent event. Winsock has his own buffers, usually 8KB for send and another 8KB for read. I didn't say anything about splitting and so on, but you're right in an other thing: I missed to tell that my sentence is only valid if you do TWSocket.Flush;. And there I am true that V6 ICS directly runs send on Winsock to immediatlely send the data. And that was the point: It not buffers internaly but forces to send it immediately to winsock or it blocks. Regards, Markus Mueller -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] Pause and Resume just not working!
Hi RTT, which Operationgsystem and Pachlevel are you using? Do you use ICS Version 6 RC1 ? I am using Windows XP with Service Pack3, tested my Client now on 3 Clients with this config. I am using ICS Versoin 6 at RC1. I added my client.exe and server.exe on my server http://projekte.priv.de/ics_localhost_bug/, maybe you want to try with that. Any idear why on my Computers it shows http://projekte.priv.de/ics_localhost_bug/client_localhost.png but on your computer it blocks much earlier? Regards, Markus Mueller If you reproduce my test, simply by compiling and starting server.pas and then compiling and starting client.pas with attached delphi debugger, looking on code line executed step, you can see that socket.flush forces to write out the data via the send call of winsock before it proceeds. Of course not only on Version 6 of ICS but also on any other Framework or TCP/IP Server listening on localhost. Tested right now, and in my case it blocks normally at socket.Flush line since first cycle, no 'Sent # Bytes!' message show, so something is different at your system. RTT -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] Pause and Resume just not working!
Markus Mueller wrote: Hello ICS Mailinglist, this is a bugreport that on localhost the windows kernel reads megabytes of data before a TWSocket client can block. This is not specific to TWSocket and it's not a bug but a winsock feature! Beside the fact that you should send smaller data chunks and use OnDataSent event to send next chunk the provided sample shows that winsock is buffering much more incomming data than current size of the socket receive buffer. BTW: The same happens with LAN connections as well. I tried it in the LAN to be able to trace data actually sent over the wire with WireShark. In my test approximately 80 KB are actually sent and buffered by the receiving peer with a LAN connection and ~40 KB with a localhost connection. -- Arno Garrels -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] Pause and Resume just not working!
Hi Arno Garrels, this is a bugreport that on localhost the windows kernel reads megabytes of data before a TWSocket client can block. This is not specific to TWSocket and it's not a bug but a winsock feature! Beside the fact that you should send smaller data chunks and use OnDataSent event to send next chunk the provided sample shows that winsock is buffering much more incomming data than current size of the socket receive buffer. BTW: The same happens with LAN connections as well. I tried it in the LAN to be able to trace data actually sent over the wire with WireShark. In my test approximately 80 KB are actually sent and buffered by the receiving peer with a LAN connection and ~40 KB with a localhost connection. Thank you so very much for confirming! I have the problem that if I have a FTP Transfer over my TWSocket Proxy, then the FTP Client flushes all the data (up tu 16 MBytes!) and then waits only less than a minute to get a positive Feedback about this transfer from the FTP Server. But the data is only in Winsocks! Do you have an idear how to resolv this problem? Much regards! Markus Mueller -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] Pause and Resume just not working!
Markus Mueller wrote: I missed to tell that my sentence is only valid if you do TWSocket.Flush;. And there I am true that V6 ICS directly runs send on Winsock to immediatlely send the data. And that was the point: It not buffers internaly but forces to send it immediately to winsock or it blocks. To be sent data is always put into TWSocket's internal send buffer first. Flush simply calls TryToSend and MessagePump in a loop until TWSocket's internal send buffer is empty. -- Arno Garrels -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] Pause and Resume just not working!
And there I am true that V6 ICS directly runs send on Winsock to immediatlely send the data. And that was the point: It not buffers internaly but forces to send it immediately to winsock or it blocks. I don't agree, unless a bug has been introduced recently. Send only TRIES to send data. Send pass to winsock as many bytes as winsock is will to accept but never block nor try to send everything in the buffer. Of course in normal operation, when you send small chunk of data slower than the link speed or smaller than what winsock own buffer can accept, then TWSocket send data immediately and there is nothing queued. As soon as you send enough data (combination of speed and volume) to overflow winsock buffer, then TWSocket starts buffering. -- francois.pie...@overbyte.be The author of the freeware multi-tier middleware MidWare The author of the freeware Internet Component Suite (ICS) http://www.overbyte.be -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] Pause and Resume just not working!
Any idear why on my Computers it shows but on your computer it blocks much earlier? Maybe be you some kind of proxy, firewall, anti-virus, anti-spam, malware, LSP, VPN and other kind of similar piece of software intercepting TCP trafic and buffering data somewhere. -- francois.pie...@overbyte.be The author of the freeware multi-tier middleware MidWare The author of the freeware Internet Component Suite (ICS) http://www.overbyte.be -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] Pause and Resume just not working!
Francois PIETTE schrieb: And there I am true that V6 ICS directly runs send on Winsock to immediatlely send the data. And that was the point: It not buffers internaly but forces to send it immediately to winsock or it blocks. I don't agree, unless a bug has been introduced recently. Send only TRIES to send data. Send pass to winsock as many bytes as winsock is will to accept but never block nor try to send everything in the buffer. Of course in normal operation, when you send small chunk of data slower than the link speed or smaller than what winsock own buffer can accept, then TWSocket send data immediately and there is nothing queued. As soon as you send enough data (combination of speed and volume) to overflow winsock buffer, then TWSocket starts buffering. I think we say both the same: TWSocket.send buffers but cause of .Flush befor any further TWSocket.send it can't further buffer without having sent. I just don't want to say TWSocket can't buffer, only in the way I used it it is not possible cause of using TWSocket.Flush! Peace! :-) Francois PIETTE schrieb: Any idear why on my Computers it shows but on your computer it blocks much earlier? Maybe be you some kind of proxy, firewall, anti-virus, anti-spam, malware, LSP, VPN and other kind of similar piece of software intercepting TCP trafic and buffering data somewhere. I thin this can't be cause on any Windows XP with Patchlevel 3 I tested it does this buffering! Did you try on your Operating System? What Version do you have and does it block there? Regards, Markus Mueller -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] Pause and Resume just not working!
Markus Mueller wrote: Hi Arno Garrels, this is a bugreport that on localhost the windows kernel reads megabytes of data before a TWSocket client can block. This is not specific to TWSocket and it's not a bug but a winsock feature! Beside the fact that you should send smaller data chunks and use OnDataSent event to send next chunk the provided sample shows that winsock is buffering much more incomming data than current size of the socket receive buffer. BTW: The same happens with LAN connections as well. I tried it in the LAN to be able to trace data actually sent over the wire with WireShark. In my test approximately 80 KB are actually sent and buffered by the receiving peer with a LAN connection and ~40 KB with a localhost connection. Thank you so very much for confirming! I have the problem that if I have a FTP Transfer over my TWSocket Proxy, then the FTP Client flushes all the data (up tu 16 MBytes!) and then waits only less than a minute to get a positive Feedback about this transfer from the FTP Server. But the data is only in Winsocks! I doubt that very much. As I wrote before just ~ 40-80 KB have been cached in my test until the sender paused sending. -- Arno Garrels -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] Pause and Resume just not working!
Hi RTT, which Operationgsystem and Pachlevel are you using? Do you use ICS Version 6 RC1 ? I am using Windows XP with Service Pack3, tested my Client now on 3 Clients with this config. I am using ICS Versoin 6 at RC1. I added my client.exe and server.exe on my server http://projekte.priv.de/ics_localhost_bug/, maybe you want to try with that. Any idear why on my Computers it shows http://projekte.priv.de/ics_localhost_bug/client_localhost.png but on your computer it blocks much earlier? Hi Markus, My original test was done in Vista32+SP1. But I have now repeated the test on XP2002+SP3. My compiled Client and Server, your Client and Server, and combination of both, and they work exactly has in Vista. Try your testes in a clean install OS, in a VirtualPC as example, if you can. Maybe you installed something on those machines, that replaced your winsocks DLL's by some patched/corrupted one. But in this list there are other guys that know much more to tell you what can be the cause. I'm a newbie in this subjects. RTT -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] Pause and Resume just not working!
Hello ICS, I found the problem, it was HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\TcpWindowSize set to 16 Mbyte on my Windows Systems. Thanks for your support and your time! You have helped me very very much! Much regards, Markus Mueller Hi RTT, which Operationgsystem and Pachlevel are you using? Do you use ICS Version 6 RC1 ? I am using Windows XP with Service Pack3, tested my Client now on 3 Clients with this config. I am using ICS Versoin 6 at RC1. I added my client.exe and server.exe on my server http://projekte.priv.de/ics_localhost_bug/, maybe you want to try with that. Any idear why on my Computers it shows http://projekte.priv.de/ics_localhost_bug/client_localhost.png but on your computer it blocks much earlier? Hi Markus, My original test was done in Vista32+SP1. But I have now repeated the test on XP2002+SP3. My compiled Client and Server, your Client and Server, and combination of both, and they work exactly has in Vista. Try your testes in a clean install OS, in a VirtualPC as example, if you can. Maybe you installed something on those machines, that replaced your winsocks DLL's by some patched/corrupted one. But in this list there are other guys that know much more to tell you what can be the cause. I'm a newbie in this subjects. RTT -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] Pause and Resume just not working!
The default is that it is not configured. Cause tcp window scaling is also deactivated, I asume it uses the normal tcp maximum of 64 Kbyte. Without the Tcp1323Opts for Windows scaling a higher TCP Window Size don't make sense at all. For more informations see http://proj.sunet.se/E2E/tcptune.html Much regards, Markus Mueller Yes I learned some things reading your thread. Anyway, just checked with regedit and my Vista x64 registry has no such key. What is the default value? Regards, SZ On Sat, Jan 24, 2009 at 1:41 PM, Markus Mueller twsoc...@priv.de wrote: Hello ICS, I found the problem, it was HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\TcpWindowSize set to 16 Mbyte on my Windows Systems. Thanks for your support and your time! You have helped me very very much! Much regards, Markus Mueller Hi RTT, which Operationgsystem and Pachlevel are you using? Do you use ICS Version 6 RC1 ? I am using Windows XP with Service Pack3, tested my Client now on 3 Clients with this config. I am using ICS Versoin 6 at RC1. I added my client.exe and server.exe on my server http://projekte.priv.de/ics_localhost_bug/, maybe you want to try with that. Any idear why on my Computers it shows http://projekte.priv.de/ics_localhost_bug/client_localhost.png but on your computer it blocks much earlier? Hi Markus, My original test was done in Vista32+SP1. But I have now repeated the test on XP2002+SP3. My compiled Client and Server, your Client and Server, and combination of both, and they work exactly has in Vista. Try your testes in a clean install OS, in a VirtualPC as example, if you can. Maybe you installed something on those machines, that replaced your winsocks DLL's by some patched/corrupted one. But in this list there are other guys that know much more to tell you what can be the cause. I'm a newbie in this subjects. RTT -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] Pause and Resume just not working!
Hi Arno Garrels, Markus Mueller wrote: The default is that it is not configured. Cause tcp window scaling is also deactivated, I asume it uses the normal tcp maximum of 64 Kbyte. Without the Tcp1323Opts for Windows scaling a higher TCP Window Size don't make sense at all. For more informations see http://proj.sunet.se/E2E/tcptune.html TCPWindowSize This value determines the maximum amount of data (in bytes) that can be outstanding on the network at any given time. It can be set to any value from 1 to 65,535 bytes by using the following registry entry: HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\Tcpip\Parameters\TcpWindowSize (REG_DWORD) The default for a gigabit interface is set to approximately 65,535 (rounded down to the nearest multiple of full TCP packets), 16,384 for a 100 Mbps link, and 8,192 for all interfaces of lower speeds (for example, modems), again rounded down. Ideally, this value should be set to the product of end-to-end network bandwidth (in bytes/s) and the round-trip delay (in seconds), also referred to as the bandwidth-delay product. This value should be set according to the amount of TCP data expected to be received by the computer. Correct, beside the maximum limit of 65,353 in detail: With TCP Window Scaling (RFC 1323) you can use values up to two Gigabytes. But, as I sayed, you have to enable This option via Tcp1323Opts. And both, the client as well as the server, must support this TCP option. See my sent link: http://proj.sunet.se/E2E/tcptune.html Regards, Markus Mueller -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] Pause and Resume just not working!
Markus Mueller wrote: The default is that it is not configured. Cause tcp window scaling is also deactivated, I asume it uses the normal tcp maximum of 64 Kbyte. Without the Tcp1323Opts for Windows scaling a higher TCP Window Size don't make sense at all. For more informations see http://proj.sunet.se/E2E/tcptune.html TCPWindowSize This value determines the maximum amount of data (in bytes) that can be outstanding on the network at any given time. It can be set to any value from 1 to 65,535 bytes by using the following registry entry: HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\Tcpip\Parameters\TcpWindowSize (REG_DWORD) The default for a gigabit interface is set to approximately 65,535 (rounded down to the nearest multiple of full TCP packets), 16,384 for a 100 Mbps link, and 8,192 for all interfaces of lower speeds (for example, modems), again rounded down. Ideally, this value should be set to the product of end-to-end network bandwidth (in bytes/s) and the round-trip delay (in seconds), also referred to as the bandwidth-delay product. This value should be set according to the amount of TCP data expected to be received by the computer. -- Arno Garrels -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] Pause and Resume just not working!
Markus Mueller wrote: Correct, beside the maximum limit of 65,353 in detail: With TCP Window Scaling (RFC 1323) you can use values up to two Gigabytes. But, as I sayed, you have to enable This option via Tcp1323Opts. And both, the client as well as the server, must support this TCP option. See my sent link: http://proj.sunet.se/E2E/tcptune.html That explains it, I wasn't aware that Window Scaling was available before Vista :( -- Arno Garrels -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] Pause and Resume just not working!
Correct, beside the maximum limit of 65,353 in detail: With TCP Window Scaling (RFC 1323) you can use values up to two Gigabytes. I read 1GB From RFC1323: Since the max window is 2**S (where S is the scaling shift count) times at most 2**16 - 1 (the maximum unscaled window), the maximum window is guaranteed to be 2*30 if S = 14. Thus, the shift count must be limited to 14 (which allows windows of 2**30 = 1 Gbyte). If a Window Scale option is received with a shift.cnt value exceeding 14, the TCP should log the error but use 14 instead of the specified value. -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] Pause and Resume just not working!
Markus Mueller wrote: it buffers sometime within 2-4 MByte before socket.pause does help, my 3 Ghz Dualcore seems to buffer also hunderts of MByte... procedure TCustomWSocket.Pause calls winsock API WSAAsyncSelect. It simply tells winsock to stop posting notification messages. Although WSAAsyncSelect immediately disables event message posting for the socket in this instance, it is possible that messages could be waiting in the application message queue. Therefore, the application must be prepared to receive network event messages even after cancellation. That's most likely what you observe, isn't it? BTW: The THttpCli component supports bandwidth control, search for UseBandwidthControl. -- Arno Garrels -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] Pause and Resume just not working!
Hi Arno Garrels, I looked over the code of THttpCli and if you do not have a connection from localhost to localhost it may work... I expect this in particular cause the read of Megabytes is done within split seconds, also if my program is sleeping (for example on a breakpoint in Delphi). The problem is that the read occures within miliseconds because localhost does not have any latency. In the case of THttpCli ther there is a 1000ms timer: I think this is much too slow to prevent the reading of so many data (Megabytes!) within this small time, isn't it? In my Client ICS Proxy software I have my own Message Loop, I don't juse socket.MessageLoop but process the messages in .Execute of my Thread by myself. If the problem would be the messages, then I would be able to ignore them there... But the read is done if my Client sleeps (for example on a breakpoint) or not! So I think the messages are not the problem... Maybe it is a general Winsock problem about Async read? Is it possible to put a Socket into a Sync read? Thanks in advance for any hints... I am completely frustrated about this problem... I just wanted to implement Flowcontrol into my Application, which I have developed over a month... And now I must search for a completely new network framework and must begin from root? :-( Much regards, Markus Mueller Markus Mueller wrote: it buffers sometime within 2-4 MByte before socket.pause does help, my 3 Ghz Dualcore seems to buffer also hunderts of MByte... procedure TCustomWSocket.Pause calls winsock API WSAAsyncSelect. It simply tells winsock to stop posting notification messages. Although WSAAsyncSelect immediately disables event message posting for the socket in this instance, it is possible that messages could be waiting in the application message queue. Therefore, the application must be prepared to receive network event messages even after cancellation. That's most likely what you observe, isn't it? BTW: The THttpCli component supports bandwidth control, search for UseBandwidthControl. -- Arno Garrels -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] Pause and Resume just not working!
Markus Müller wrote: Hi Arno Garrels, I looked over the code of THttpCli and if you do not have a connection from localhost to localhost it may work... I expect this in particular cause the read of Megabytes is done within split seconds, Yes, it is very fast. also if my program is sleeping (for example on a breakpoint in Delphi). How do you manage this? If the application stops at a breakpoint it can no longer receive data. The problem is that the read occures within miliseconds because localhost does not have any latency. In the case of THttpCli ther there is a 1000ms timer: I think this is much too slow to prevent the reading of so many data (Megabytes!) within this small time, isn't it? Yes, that's why THttpCli has a property BandwidthSampling which sets the timer interval. In my Client ICS Proxy software I have my own Message Loop, I don't juse socket.MessageLoop but process the messages in .Execute of my Thread by myself. If the problem would be the messages, then I would be able to ignore them there... But the read is done if my Client sleeps (for example on a breakpoint) or not! So I think the messages are not the problem... That's correct, a little bit later, after I wrote my previous replay, I recalled that after Pause a flag is set internaly and the component ignores those notification messages still waiting in the application message queue when this flag is set. Maybe it is a general Winsock problem about Async read? I don't think so, the component only reads from the socket when it gets a notification message. Is it possible to put a Socket into a Sync read? Possible yes ( with winsock API ), however I would not recomment doing that. Have you tried option wsoNoReceiveLoop yet? -- Arno Garrels Thanks in advance for any hints... I am completely frustrated about this problem... I just wanted to implement Flowcontrol into my Application, which I have developed over a month... And now I must search for a completely new network framework and must begin from root? :-( Much regards, Markus Mueller Markus Mueller wrote: it buffers sometime within 2-4 MByte before socket.pause does help, my 3 Ghz Dualcore seems to buffer also hunderts of MByte... procedure TCustomWSocket.Pause calls winsock API WSAAsyncSelect. It simply tells winsock to stop posting notification messages. Although WSAAsyncSelect immediately disables event message posting for the socket in this instance, it is possible that messages could be waiting in the application message queue. Therefore, the application must be prepared to receive network event messages even after cancellation. That's most likely what you observe, isn't it? BTW: The THttpCli component supports bandwidth control, search for UseBandwidthControl. -- Arno Garrels -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] Pause and Resume just not working!
Hello (again :-)) Arno Garrels, I looked over the code of THttpCli and if you do not have a connection from localhost to localhost it may work... I expect this in particular cause the read of Megabytes is done within split seconds, Yes, it is very fast. so you agree a (at least 1000ms) timer can't help on localhost if it is done in very less ms? I am going to try and write an response to the mailing ist if I can produced the problem with the THttpClient Object. I am also going to write a simple sample TWSocket Daemon which just tries to read only a fixed number of bytes. In my theory thats impossible and I'll post this in case as an bug report (to this maling list?). also if my program is sleeping (for example on a breakpoint in Delphi). How do you manage this? If the application stops at a breakpoint it can no longer receive data. This is exactly the point! I just don't get who reads the data, but there is defintively something reading all the available data (into a buffer?)! Later I see the data flowing through the WSocket_Synchronized_recv: So I think it can only be Winsock who is buffering, cause the actual data reading call is FRecv which is a Winsock function! Am I right at this? For your information: I have beside the GUI two threads: One server and one client thread. Hmmm... Delphi does stop all threads if I am at a breakpoint, doesn't it? The problem is that the read occures within miliseconds because localhost does not have any latency. In the case of THttpCli ther there is a 1000ms timer: I think this is much too slow to prevent the reading of so many data (Megabytes!) within this small time, isn't it? Yes, that's why THttpCli has a property BandwidthSampling which sets the timer interval. I use the wsoNoReceiveLoop Flag, but this problem also occures with this flat. I run the pause in OnDataAvailable in the case that I read too much data. I programmed (for debugging) a fixed size I only want to read: If I read (in 4096 bytes chunks) over this value then I run TWSocket.pause. But this don't work! I don't get further OnDataAvailable (with wsoNoReceiveLoop Flag), but the data has been read from the FTP Client at all! I had one idear about this wsoNoReceiveLoop-Flag in Source Code on the FD_CLOSE handler: It removes the flag and reads all available data cause it maybe is impossible later... I changed here that wsoNoReceiveLoop is not removed, so it should NOT read all the data. But also this don't change anything! In my Client ICS Proxy software I have my own Message Loop, I don't juse socket.MessageLoop but process the messages in .Execute of my Thread by myself. If the problem would be the messages, then I would be able to ignore them there... But the read is done if my Client sleeps (for example on a breakpoint) or not! So I think the messages are not the problem... That's correct, a little bit later, after I wrote my previous replay, I recalled that after Pause a flag is set internaly and the component ignores those notification messages still waiting in the application message queue when this flag is set. Where is the problem then? Do you have any idear who could it be who reads all the data? Or are you realy shure that this MUST be the AsyncRead Function? I also set a breakpoint to the DoRecv in the AsyncRead Function: Delphi never halts on this call... I investigated and found out that this is caused that by not using LineMode: It seems not to need to read here at all. So where else could the read have beeen done? Who reads the data if it is not AsyncRead? Maybe it is a general Winsock problem about Async read? I don't think so, the component only reads from the socket when it gets a notification message. There is realy nobody else than ICS who could buffere the data, isn't it? I think the problem points still to Winsocks... Is it possible to put a Socket into a Sync read? Possible yes ( with winsock API ), however I would not recomment doing that. Have you tried option wsoNoReceiveLoop yet? Yes, sure, I found the option by investigating the cause in the source. But for me it doesn't change anything beside that I don't get OnDataAvailable many times without data after I got one or more OnDataAvailable Events with readable data. Thank you very much for your help! If you need you get payed for your help please tell me. There is a budget for this! Regards, Markus Mueller -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] Pause and Resume just not working!
Markus Mueller wrote: Hello (again :-)) Arno Garrels, I looked over the code of THttpCli and if you do not have a connection from localhost to localhost it may work... I expect this in particular cause the read of Megabytes is done within split seconds, Yes, it is very fast. so you agree a (at least 1000ms) timer can't help on localhost if it is done in very less ms? Yes, there are probably better solutions that using a timer for localhost connections. I am going to try and write an response to the mailing ist if I can produced the problem with the THttpClient Object. I am also going to write a simple sample TWSocket Daemon which just tries to read only a fixed number of bytes. In my theory thats impossible and I'll post this in case as an bug report (to this maling list?). You should upload it to some webserver and post the link here, The list doesn't allow larger attachments. also if my program is sleeping (for example on a breakpoint in Delphi). How do you manage this? If the application stops at a breakpoint it can no longer receive data. This is exactly the point! I just don't get who reads the data, but there is defintively something reading all the available data (into a buffer?)! Later I see the data flowing through the WSocket_Synchronized_recv: So I think it can only be Winsock who is buffering, cause the actual data reading call is FRecv which is a Winsock function! Am I right at this? Yes, that's correct. For your information: I have beside the GUI two threads: One server and one client thread. Hmmm... Delphi does stop all threads if I am at a breakpoint, doesn't it? Yes, correct. The problem is that the read occures within miliseconds because localhost does not have any latency. In the case of THttpCli ther there is a 1000ms timer: I think this is much too slow to prevent the reading of so many data (Megabytes!) within this small time, isn't it? Yes, that's why THttpCli has a property BandwidthSampling which sets the timer interval. I use the wsoNoReceiveLoop Flag, but this problem also occures with this flat. I run the pause in OnDataAvailable in the case that I read too much data. I programmed (for debugging) a fixed size I only want to read: If I read (in 4096 bytes chunks) over this value then I run TWSocket.pause. But this don't work! I don't get further OnDataAvailable (with wsoNoReceiveLoop Flag), but the data has been read from the FTP Client at all! Do you use LineMode? I do not think so but should be asked anyway. If LineMode is turned on the component indeed buffers data internally. I had one idear about this wsoNoReceiveLoop-Flag in Source Code on the FD_CLOSE handler: It removes the flag and reads all available data cause it maybe is impossible later... I changed here that wsoNoReceiveLoop is not removed, so it should NOT read all the data. But also this don't change anything! In my Client ICS Proxy software I have my own Message Loop, I don't juse socket.MessageLoop but process the messages in .Execute of my Thread by myself. If the problem would be the messages, then I would be able to ignore them there... But the read is done if my Client sleeps (for example on a breakpoint) or not! So I think the messages are not the problem... That's correct, a little bit later, after I wrote my previous replay, I recalled that after Pause a flag is set internaly and the component ignores those notification messages still waiting in the application message queue when this flag is set. Where is the problem then? Do you have any idear who could it be who reads all the data? Or are you realy shure that this MUST be the AsyncRead Function? Hmm, should you realy use LineMode the set a breakpoint in (OverbyteIcs) WSocket.pas: function TCustomLineWSocket.TriggerDataAvailable(ErrCode : Word) : Boolean; At the line where Receive is called. If you do not use LineMode winsock is most likely the reason. Or maybe you have you increased the default Socket receive buffer size, property SocketRcvBufSize? -- Arno Garrels -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] Pause and Resume just not working!
Markus Mueller wrote: Ok, this proves that Winsocks is buffering megabytes of data before TWSocket.pause becomes active! Not quite correct, TWSocket.Pause becomes active immediately, however winsock obviously continues to buffer incomming data, in this case I think there's not much you can do and that won't be related to async sockets IMO. TWSocket.Pause just stops winsock event notification. Do you use LineMode? I do not think so but should be asked anyway. If LineMode is turned on the component indeed buffers data internally. I am not using LineMode. If you look at TriggerDataAvailable you'll see that it never gaves the DoRecv in AsyncReceive a change to read. So I think it is proved that the problem is in Winsock, do you also think so? I doubt that it was a problem at all. Winsock does a lot of things in the background you cannot control, even when the socket handle is already closed. -- Arno garrels -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] Pause and Resume just not working!
Hello! Hello ICS Mailinglist, this is a bugreport that on localhost the windows kernel reads megabytes of data before a TWSocket client can block. As you can see on the following explanation and the images on http://projekte.priv.de/ics_localhost_bug/ , the problem is the windows kernel: No. The problem is that you don't understand how ICS work. By this: transfered := socket.Send(@buf, sizeof(buf)); assuming buf: array[0..102400] of byte; you command ICS TWsocket to send 102401 bytes. It doesn't necessarily mean that it goes straight to Windows Sockets (or TCP/IP stack, whatever). Now, in TWSocket.Send, component splits your data (these 102401 bytes) into ~1,5 kilobyte chunks, puts every chunk into FIFO (first-in-first-out) queue, enlarging it, then sends every chunk one-by-one, shrinking the queue. Component sends data to TCP/IP stack as long as this doesn't err out (ie. send() doesn't return error Operation would block - or any other error - what may happen when, for example, destination socket doesn't consume sent data). When you do .pause on receiving (*NOT* sending) socket, you say ok, now I don't want to know that new data have arrived. Meanwhile, sending TWSocket component *still* send data to TCP/IP on his side, TCP/IP sends data to destination socket, TCP/IP on the destination side sees that socket doesn't consume data so it buffers them as long as there's enough space for this (see SO_RCVBUF), and when buffer is full, it responds to sending side with some kind of sorry, i cannot handle more data, you're talking too fast. Sending socket, seeing this, responds to queue pumping code (within TWSocket) with Operation Would Block, what makes FIFO in TWSocket stop shrinking. That's why memory usage of sending application skyrockets: receiving side doesn't receive data (because it doesn't want to know that data have arrived), and sending side is not aware of this. Why? Because you just say send! send! send! send it goddamnit! send it!!! not checking whether something has actually been sent *and* received. You have OnDataSent for this purpose - it's fired when TWSocket sending FIFO queue is emptied successfully. To cut things short - by using TWSocket's send method, you say put these data in your buffers, send it to the other side, and tell me, by firing OnDataSent, when you've dealt with that.. If you use send again - before having received OnDataSent event - you say hey, I have some more data to send, send them when you'll deal with all previous data, and tell me, by firing OnDataSent, when you'll send *all* data - the previously ordered to send *and* the new data. HTH. -- Piotr Dałek enigmati...@interia.pl -- Kochanka - kobieta drugiej kategorii? sprawdz http://link.interia.pl/f2026 -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] Pause and Resume just not working!
HI Piotr Dałek, yes you are right with your argumentation. I am using TWSocket in a manner generally not allowed, cause message processing and so on is not assured. BUT! - The clients wrote via send to somewhere about 16 Mbytes and if you were true and it would be in buffer of the client then the process would must have in Taskmanagers at least 16 MB of memory size. Please look at my screenshot, the server and the client does have just 3.5 MB! - I tried now with Putty, which is an SSH Client able to provide TCP Tunnels on localhost. I forwarded it to a Socket not accepting any data. If I run my Client from localhost to the opened port on localhost from Putty, I have the same 16 MBytes of caching! And also there is only 3 MBs of RAM used by the client. - The same problem is with Mails sending via Thunderbird: I get a timeout if I send a big mail via an SSH Tunnel cause the Data is immediatley in the big buffer but the real Transfer via Putty consumes much more time. So the Thunderbird throws an error. - By the way: If you have no Line Mode then ICS don't do buffering at all. It just uses send of Winsock! Because of this if at connecting to the linux host it immediatlely blocks, as shown in the screenshot. I hope you agree with me that my programming was not perfect but anyway proves the problem with TCP Read bufering on localhost on Windows. Do you agree? Regards, Markus Mueller Hello! Hello ICS Mailinglist, this is a bugreport that on localhost the windows kernel reads megabytes of data before a TWSocket client can block. As you can see on the following explanation and the images on http://projekte.priv.de/ics_localhost_bug/ , the problem is the windows kernel: No. The problem is that you don't understand how ICS work. By this: transfered := socket.Send(@buf, sizeof(buf)); assuming buf: array[0..102400] of byte; you command ICS TWsocket to send 102401 bytes. It doesn't necessarily mean that it goes straight to Windows Sockets (or TCP/IP stack, whatever). Now, in TWSocket.Send, component splits your data (these 102401 bytes) into ~1,5 kilobyte chunks, puts every chunk into FIFO (first-in-first-out) queue, enlarging it, then sends every chunk one-by-one, shrinking the queue. Component sends data to TCP/IP stack as long as this doesn't err out (ie. send() doesn't return error Operation would block - or any other error - what may happen when, for example, destination socket doesn't consume sent data). When you do .pause on receiving (*NOT* sending) socket, you say ok, now I don't want to know that new data have arrived. Meanwhile, sending TWSocket component *still* send data to TCP/IP on his side, TCP/IP sends data to destination socket, TCP/IP on the destination side sees that socket doesn't consume data so it buffers them as long as there's enough space for this (see SO_RCVBUF), and when buffer is full, it responds to sending side with some kind of sorry, i cannot handle more data, you're talking too fast. Sending socket, seeing this, responds to queue pumping code (within TWSocket) with Operation Would Block, what makes FIFO in TWSocket stop shrinking. That's why memory usage of sending application skyrockets: receiving side doesn't receive data (because it doesn't want to know that data have arrived), and sending side is not aware of this. Why? Because you just say send! send! send! send it goddamnit! send it!!! not checking whether something has actually been sent *and* received. You have OnDataSent for this purpose - it's fired when TWSocket sending FIFO queue is emptied successfully. To cut things short - by using TWSocket's send method, you say put these data in your buffers, send it to the other side, and tell me, by firing OnDataSent, when you've dealt with that.. If you use send again - before having received OnDataSent event - you say hey, I have some more data to send, send them when you'll deal with all previous data, and tell me, by firing OnDataSent, when you'll send *all* data - the previously ordered to send *and* the new data. HTH. -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] Pause and Resume just not working!
Hello! HI Piotr Dałek, yes you are right with your argumentation. I am using TWSocket in a manner generally not allowed, cause message processing and so on is not assured. And we can stop right here. Since I'm a ICS V5 user, I cannot answer to your but's - maybe something has changed in ICS V6, but anyway don't expect any predictible behavior when you're not using things right. -- Piotr Dałek enigmati...@interia.pl -- Sprawdz, co wyswietlaja kina w Twoim miescie! http://link.interia.pl/f2028 -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] Pause and Resume just not working!
- The clients wrote via send to somewhere about 16 Mbytes and if you were true and it would be in buffer of the client then the process would must have in Taskmanagers at least 16 MB of memory size. Please look at my screenshot, the server and the client does have just 3.5 MB! I'm not an expert but I think you don't see the memory usage because data is not cached in the client. Piotr clearly say that if the RCV client is not accepting data, Winsocks cache it until its internal buffer is full and only then report can't accept more because client is too slow consuming it. RTT -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] Pause and Resume just not working!
- The clients wrote via send to somewhere about 16 Mbytes and if you were true and it would be in buffer of the client then the process would must have in Taskmanagers at least 16 MB of memory size. Please look at my screenshot, the server and the client does have just 3.5 MB! I'm not an expert but I think you don't see the memory usage because data is not cached in the client. Piotr clearly say that if the RCV client is not accepting data, Winsocks cache it until its internal buffer is full and only then report can't accept more because client is too slow consuming it. If you reproduce my test, simply by compiling and starting server.pas and then compiling and starting client.pas with attached delphi debugger, looking on code line executed step, you can see that socket.flush forces to write out the data via the send call of winsock before it proceeds. Of course not only on Version 6 of ICS but also on any other Framework or TCP/IP Server listening on localhost. It is impossible that ICS is buffering here, because Winsock is a DLL from Microsoft not from ICS. But also the Winsock DLL can not be the one who is buffering, because the Winsock DLL also is running in userland and must be accounted on Taskmanager on the running process. The only thing not beeing accounted in Taskmanager memory is kernel memory allocated internal for one or more processes. And it makes sense, on localhost you can buffer much more than on network so I simply think Microsoft implemented this to increase performance on localhost connections. This is also traceable by Putty: Here is the same effect about buffering (see my last email about this) with Thunderbird sending a big email. What can I else do to prove that the kernel (on loopback connections) is the causer? Regards, Markus Mueller -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] Pause and Resume just not working!
If you reproduce my test, simply by compiling and starting server.pas and then compiling and starting client.pas with attached delphi debugger, looking on code line executed step, you can see that socket.flush forces to write out the data via the send call of winsock before it proceeds. Of course not only on Version 6 of ICS but also on any other Framework or TCP/IP Server listening on localhost. Tested right now, and in my case it blocks normally at socket.Flush line since first cycle, no 'Sent # Bytes!' message show, so something is different at your system. RTT -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] Pause and Resume just not working!
I have not read all the discussions, but at least this: - By the way: If you have no Line Mode then ICS don't do buffering at all. It just uses send of Winsock! This sentence is wrong. TWSocket use a FIFO buffer when you call Send. Whatever data size you send, TWSocket will use a list of buffers (Each having a BufSize length wich by default is the largest data packet accepted on Ethernet so that fragmentation is the least possible). Each time you cann Send, TWSocket tries to deliver data from his FIFO to winsock. Each time winsock notifies that he has room into his own buffer, then TWSocket extract data from his FIFO and give it to winsock. Eventually, TWSocket empty his buffer and then triggers OnDataSent event. Winsock has his own buffers, usually 8KB for send and another 8KB for read. -- francois.pie...@overbyte.be The author of the freeware multi-tier middleware MidWare The author of the freeware Internet Component Suite (ICS) http://www.overbyte.be -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
[twsocket] Pause and Resume just not working!
Hi ICS Mailinglist,
I use the V6 Rev1 ICS TWSocket and have a issue with buffering somewhere in
ICS and/or Winsock.
On localhost or in my local LAN, if I have there a stream to my ICS
Server which
just pushes data, like the DATA-TCP Connection of ftp, then all data
is sent just
in some seconds with a amazing transferrate and is buffered somewhere. I
can see
via Breakpoint hat the data is then received by the function
{* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *
* *}
function WSocket_Synchronized_recv(s: TSocket; var Buf: TWSocketData;
len, flags: Integer): Integer;
begin
if @FRecv= nil then
@FRecv := WSocketGetProc('recv');
Result := FRecv(s, Buf^, len, flags);
end;
{* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *
* *}
of ICS. Cause FRecv is the Winsock recv Function, wich is NOT from
ICS, so I
think it must be something inside of Windows and/or Winsock that causes this
problem. Does anybody have an idear what here does Buffer Megabytes of TCP
Data? Any idear why I can prevent it from that? I investigated the
Source, and it
is NOT a problem of the TCustomWSocket.ASyncReceive function!
The problem with this is, that if I make FTP over my Proxyserver (= ICS
Server) the
FTP Clients runs into a timeout: For the FTP Client it looks that all
data arrived and
the positive Answer is missing, but all the Data is just in some local
buffer...
My Goal is to slow down the connection via TWSocket.pause;. But if I
do this
I just don't get further data from the big Buffer of Winsock The
data is in it
if I want or not.
Thanks for any help!
Regards,
Markus Mueller
--
To unsubscribe or change your settings for TWSocket mailing list
please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket
Visit our website at http://www.overbyte.be
Re: [twsocket] Pause and Resume just not working!
Hello again
I realy don't know why, but on a different machine there is this buffering
with unchanged source code NOT. Maybe its my virusscanner or something
like this... Sorry for any convenience
ende?lp=endep=thMx..search=convenience this may cause...
Regards,
Markus Mueller
Hi ICS Mailinglist,
I use the V6 Rev1 ICS TWSocket and have a issue with buffering somewhere in
ICS and/or Winsock.
On localhost or in my local LAN, if I have there a stream to my ICS
Server which
just pushes data, like the DATA-TCP Connection of ftp, then all data
is sent just
in some seconds with a amazing transferrate and is buffered somewhere. I
can see
via Breakpoint hat the data is then received by the function
{* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *
* *}
function WSocket_Synchronized_recv(s: TSocket; var Buf: TWSocketData;
len, flags: Integer): Integer;
begin
if @FRecv= nil then
@FRecv := WSocketGetProc('recv');
Result := FRecv(s, Buf^, len, flags);
end;
{* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *
* *}
of ICS. Cause FRecv is the Winsock recv Function, wich is NOT from
ICS, so I
think it must be something inside of Windows and/or Winsock that causes this
problem. Does anybody have an idear what here does Buffer Megabytes of TCP
Data? Any idear why I can prevent it from that? I investigated the
Source, and it
is NOT a problem of the TCustomWSocket.ASyncReceive function!
The problem with this is, that if I make FTP over my Proxyserver (= ICS
Server) the
FTP Clients runs into a timeout: For the FTP Client it looks that all
data arrived and
the positive Answer is missing, but all the Data is just in some local
buffer...
My Goal is to slow down the connection via TWSocket.pause;. But if I
do this
I just don't get further data from the big Buffer of Winsock The
data is in it
if I want or not.
Thanks for any help!
Regards,
Markus Mueller
--
To unsubscribe or change your settings for TWSocket mailing list
please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket
Visit our website at http://www.overbyte.be
Re: [twsocket] Pause and Resume just not working!
Sorry... I must correct myself again... The problem is still there, on
machine with 1,4 Ghz
it buffers sometime within 2-4 MByte before socket.pause does help, my
3 Ghz Dualcore
seems to buffer also hunderts of MByte...
Hello again
I realy don't know why, but on a different machine there is this buffering
with unchanged source code NOT. Maybe its my virusscanner or something
like this... Sorry for any convenience
ende?lp=endep=thMx..search=convenience this may cause...
Regards,
Markus Mueller
Hi ICS Mailinglist,
I use the V6 Rev1 ICS TWSocket and have a issue with buffering somewhere in
ICS and/or Winsock.
On localhost or in my local LAN, if I have there a stream to my ICS
Server which
just pushes data, like the DATA-TCP Connection of ftp, then all data
is sent just
in some seconds with a amazing transferrate and is buffered somewhere. I
can see
via Breakpoint hat the data is then received by the function
{* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *
* *}
function WSocket_Synchronized_recv(s: TSocket; var Buf: TWSocketData;
len, flags: Integer): Integer;
begin
if @FRecv= nil then
@FRecv := WSocketGetProc('recv');
Result := FRecv(s, Buf^, len, flags);
end;
{* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *
* *}
of ICS. Cause FRecv is the Winsock recv Function, wich is NOT from
ICS, so I
think it must be something inside of Windows and/or Winsock that causes this
problem. Does anybody have an idear what here does Buffer Megabytes of TCP
Data? Any idear why I can prevent it from that? I investigated the
Source, and it
is NOT a problem of the TCustomWSocket.ASyncReceive function!
The problem with this is, that if I make FTP over my Proxyserver (= ICS
Server) the
FTP Clients runs into a timeout: For the FTP Client it looks that all
data arrived and
the positive Answer is missing, but all the Data is just in some local
buffer...
My Goal is to slow down the connection via TWSocket.pause;. But if I
do this
I just don't get further data from the big Buffer of Winsock The
data is in it
if I want or not.
Thanks for any help!
Regards,
Markus Mueller
--
To unsubscribe or change your settings for TWSocket mailing list
please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket
Visit our website at http://www.overbyte.be
