Re: [twsocket] Flow control
Francois PIETTE wrote: > > IMO, you have all the required tools ! Yes, no question, you realy have all the required tools. But you also have the required tools to write a SMTP client using TWSocket, and nevertheless there's a SMTP client in ICS. What I mean is that it won't hurt to put that in, but I can live happily w/o such a feature. --- Arno Garrels [TeamICS] http://www.overbyte.be/eng/overbyte/teamics.html -- To unsubscribe or change your settings for TWSocket mailing list please goto http://www.elists.org/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] Flow control
There is nothing to understand in the buffering. The buffer is automatic and only limited by available virtual memory. You get an exception when you run out of memory, just like you have - for example - when you run out of memory trying to add strings to a TStringList or anything else. That is just plain basic Delphi programming. Your program has access to BufferedByteCount at anytime to know what the size of the buffer is. I don't see any advantage to have the component check the value against some limit and make Send fail, trigger an event or an exception. Just check the property before calling Send ! -- [EMAIL PROTECTED] http://www.overbyte.be - Original Message - From: "Angus Robertson - Magenta Systems Ltd" <[EMAIL PROTECTED]> To: Sent: Monday, November 20, 2006 2:02 AM Subject: Re: [twsocket] Flow control > > Have you notice BufferedByteCount property ? > > Yes, I mentioned in my first message, but since it's undocumented (not > mentioned in the Wiki, FAQ or Help) it took me a while to understand how > the wsocket buffering works. > > But in this case, TWSocket is poorly designed and network issues can > allow an application to crash out of memory without any earlier errors. > Failing Send if the buffered text reaches a limit, say 32K or something, > seems sensible in the component, rather than expected every user to try > and understand the buffering. > > Angus > -- > To unsubscribe or change your settings for TWSocket mailing list > please goto http://www.elists.org/mailman/listinfo/twsocket > Visit our website at http://www.overbyte.be -- To unsubscribe or change your settings for TWSocket mailing list please goto http://www.elists.org/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] Flow control
Hello Francois, > IMO, you have all the required tools ! Agree for 100% ! Definitively no need to put wishels and bells into TWSocket ! All the tools are there, this is up to derrived components and or applications. --- Rgds, Wilfried [TeamICS] http://www.overbyte.be/eng/overbyte/teamics.html http://www.mestdagh.biz Sunday, November 19, 2006, 21:06, Francois PIETTE wrote: >>> I have also implemented a property called MaxSendBuffer >> >> I think this needs to go into TWSocket. > The component doesn't know what to do if too much data is sent. It is the > application responsability to take any appropriate action: throw data away, > pause the data source, overflow the buffer to disk, ... > The component offer anything needed to implement the required processing by > the application. Have you notice BufferedByteCount property ? Just examine > his value after Send() or PutDataInSendBuffer(). You'll know if you reached > your limit. > You may also use OnDataSent event which is triggered when BufferedByteCount > becomes 0. > Use OnSendData (do not confuse it with OnDataSent) which is triggered each > time the component write something into winsock own buffer. > IMO, you have all the required tools ! > Contribute to the SSL Effort. Visit > http://www.overbyte.be/eng/ssl.html > -- > [EMAIL PROTECTED] > The author for the freeware multi-tier middleware MidWare > The author of the freeware Internet Component Suite (ICS) > http://www.overbyte.be -- To unsubscribe or change your settings for TWSocket mailing list please goto http://www.elists.org/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] Flow control
> Have you notice BufferedByteCount property ? Yes, I mentioned in my first message, but since it's undocumented (not mentioned in the Wiki, FAQ or Help) it took me a while to understand how the wsocket buffering works. But in this case, TWSocket is poorly designed and network issues can allow an application to crash out of memory without any earlier errors. Failing Send if the buffered text reaches a limit, say 32K or something, seems sensible in the component, rather than expected every user to try and understand the buffering. Angus -- To unsubscribe or change your settings for TWSocket mailing list please goto http://www.elists.org/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] Flow control
>>> I have also implemented a property called MaxSendBuffer >> >> I think this needs to go into TWSocket. > > Agreed. I don't. See my response to Angus. -- Contribute to the SSL Effort. Visit http://www.overbyte.be/eng/ssl.html -- [EMAIL PROTECTED] http://www.overbyte.be -- To unsubscribe or change your settings for TWSocket mailing list please goto http://www.elists.org/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] Flow control
>> I have also implemented a property called MaxSendBuffer > > I think this needs to go into TWSocket. The component doesn't know what to do if too much data is sent. It is the application responsability to take any appropriate action: throw data away, pause the data source, overflow the buffer to disk, ... The component offer anything needed to implement the required processing by the application. Have you notice BufferedByteCount property ? Just examine his value after Send() or PutDataInSendBuffer(). You'll know if you reached your limit. You may also use OnDataSent event which is triggered when BufferedByteCount becomes 0. Use OnSendData (do not confuse it with OnDataSent) which is triggered each time the component write something into winsock own buffer. IMO, you have all the required tools ! Contribute to the SSL Effort. Visit http://www.overbyte.be/eng/ssl.html -- [EMAIL PROTECTED] The author for the freeware multi-tier middleware MidWare The author of the freeware Internet Component Suite (ICS) http://www.overbyte.be -- To unsubscribe or change your settings for TWSocket mailing list please goto http://www.elists.org/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] Flow control
> The problem you are having is with you software CamCollect? No, CamCollect just receives HTTP images and more rarely streams. The video servers are for something completely different, a bespoke application. The application locking up under intense CPU stress is ComCap, just moving multiple random streams of data around. But I've now created a new component to do much the same thing, which is why I'm trying to 'bullet proof' it, I'll be announcing it in a couple of days once I've integrated it into a few of my applications, replacing lots of horrible code. Angus -- To unsubscribe or change your settings for TWSocket mailing list please goto http://www.elists.org/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] Flow control
> I have four 8-channel video servers in my office (but only 15 cameras) > which offer different types of streaming, one if Linux based and streams > HTTP, two are Windows based and use UDP, the last is an embedded system > with hardware MPEG encoding on all channels and also streams eight > channels of UDP. I guess UDP is used to avoid flow control issues, > packets are lost if not received. Yes, UDP in this case can be used like this, so you can loose packets if the bandwidth doesn´t acommodate the data. But to use the UDP you have to have some protocol like RTP and RTSP, in most IP cameras and Video Servers HTTP Push is implemented, so, images came from TCP/HTTP Protocol, and some other video servers implements UDP transmission over RTP/RTSP >> I have also implemented a property called MaxSendBuffer > > I think this needs to go into TWSocket. I also think that this should go into TWSocket, but as I can´t change it, I have made a derived class to accomodate my needs, the ICS Team can think of implementing this functionality into TWSocket, I think that it is very useful, at least for me is The problem you are having is with you software CamCollect? Did you figured out how to solve it? Éric - Original Message - From: "Angus Robertson - Magenta Systems Ltd" <[EMAIL PROTECTED]> To: Sent: Sunday, November 19, 2006 2:13 PM Subject: Re: [twsocket] Flow control >> but this is what I have concluded >> reading the source, because the send routine always put your data >> into internal buffer, growing it as data is incoming. > > Agree. > >> I have an Client / Server application for IP Camera Surveillance >> (www.digifort.com.br), and one feature of it works like this, the >> server receives images from the IP Cameras and retransmits those >> images to the clients connected to this server > > I have four 8-channel video servers in my office (but only 15 cameras) > which offer different types of streaming, one if Linux based and streams > HTTP, two are Windows based and use UDP, the last is an embedded system > with hardware MPEG encoding on all channels and also streams eight > channels of UDP. I guess UDP is used to avoid flow control issues, > packets are lost if not received. > >> I have also implemented a property called MaxSendBuffer > > I think this needs to go into TWSocket. > > Angus > -- > To unsubscribe or change your settings for TWSocket mailing list > please goto http://www.elists.org/mailman/listinfo/twsocket > Visit our website at http://www.overbyte.be > -- To unsubscribe or change your settings for TWSocket mailing list please goto http://www.elists.org/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] Flow control
Angus Robertson - Magenta Systems Ltd wrote: > I have four 8-channel video servers in my office (but only 15 cameras) > which offer different types of streaming, one if Linux based and > streams HTTP, two are Windows based and use UDP, the last is an > embedded system with hardware MPEG encoding on all channels and also > streams eight channels of UDP. I guess UDP is used to avoid flow > control issues, packets are lost if not received. > >> I have also implemented a property called MaxSendBuffer > > I think this needs to go into TWSocket. Agreed. Also, for streaming real time data there are special protocols available: RTP (RFC 1889) and RTCP (RFC 1890). They are used for instance in VOIP/SIP. May be interesting? --- Arno Garrels [TeamICS] http://www.overbyte.be/eng/overbyte/teamics.html -- To unsubscribe or change your settings for TWSocket mailing list please goto http://www.elists.org/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] Flow control
> but this is what I have concluded > reading the source, because the send routine always put your data > into internal buffer, growing it as data is incoming. Agree. > I have an Client / Server application for IP Camera Surveillance > (www.digifort.com.br), and one feature of it works like this, the > server receives images from the IP Cameras and retransmits those > images to the clients connected to this server I have four 8-channel video servers in my office (but only 15 cameras) which offer different types of streaming, one if Linux based and streams HTTP, two are Windows based and use UDP, the last is an embedded system with hardware MPEG encoding on all channels and also streams eight channels of UDP. I guess UDP is used to avoid flow control issues, packets are lost if not received. > I have also implemented a property called MaxSendBuffer I think this needs to go into TWSocket. Angus -- To unsubscribe or change your settings for TWSocket mailing list please goto http://www.elists.org/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] Flow control
> But you said in your case you cannot control the rate your > random is arriving at (or did I misinterpret?) so what else can you > do except keep buffering it, or error? My main concern is to stop the application crashing, which has happened three times this month, so I need to ignore new data to send once the internal buffers reach a certain limit by checking BufferedByteCount (ignoring BuffSize which is clearly nothing to do with total buffer size). Angus -- To unsubscribe or change your settings for TWSocket mailing list please goto http://www.elists.org/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] Flow control
> I have been working with ICS for some time and I had the same problem as > you, I have read the ICS source code and I saw that it doesn´t have this > flow control on send, what they have is the internal buffer where data is > stored and sent in background, so, for example, if you have a 56kbps > connection to send the data and you try to send for example 1mbps of data > using the send routine, the internal buffer will keep growing and growing > and growing, I don´t know if this is correct, but this is what I have > concluded reading the source, because the send routine always put your > data > into internal buffer, growing it as data is incoming. It is correct. The idea of flow control with TWSocket is to send a data chunk (for example 4KB) and then from the OnDataSent event get the next data chunk. This way the buffer never grow beyond 4KB. This is what the FTP and HTTP component do. They could read files very fast but this would simply load most of the file into the buffer (RAM). So it read one chunk from the file and then from OnDataSent read the next chunk. On the network, the sending doesn't really send since when TWSocket has emptyed his buffer (OnDataSent is triggered at that time), winsock has his own buffer (8KB by default) filled. So while the application refill TWSocket buffer, winsock still send data to the network from his own buffer. -- Contribute to the SSL Effort. Visit http://www.overbyte.be/eng/ssl.html -- [EMAIL PROTECTED] http://www.overbyte.be -- To unsubscribe or change your settings for TWSocket mailing list please goto http://www.elists.org/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] Flow control
>> Use event OnDataSent to control the flow, and to avoid grow of >> TWSocket's send buffer. > That is impossible for random data, unless an extra FIFO buffer is used > externally to TWSocket. I don't understand what you mean. You can use OnDataSent to fetch more data to send. > Trying to understand how TIcsBufferHandler works, it appears to be > multiple 1,460 byte blocks (TIcsBuffer), with new blocks being added if > data can not be sent sufficiently fast. 1460 is just the default. You have a property to change it. It has been selected because it correspond to the data size available in the largest Ethernet packet, thus optimizing packet fragmentation when using Ethernet. > So I guess the answer to my question is there is no flow control and the > buffers will just keep growing if data can not be sent fast enough, > until the application runs out of memory. I still don't understand your problem. Are you willing to have a blocking Send method ? Could you re-explain your problem as seen from the highest level ? -- Contribute to the SSL Effort. Visit http://www.overbyte.be/eng/ssl.html -- [EMAIL PROTECTED] http://www.overbyte.be -- To unsubscribe or change your settings for TWSocket mailing list please goto http://www.elists.org/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] Flow control
Hello Angus, I have been working with ICS for some time and I had the same problem as you, I have read the ICS source code and I saw that it doesn´t have this flow control on send, what they have is the internal buffer where data is stored and sent in background, so, for example, if you have a 56kbps connection to send the data and you try to send for example 1mbps of data using the send routine, the internal buffer will keep growing and growing and growing, I don´t know if this is correct, but this is what I have concluded reading the source, because the send routine always put your data into internal buffer, growing it as data is incoming. I have an Client / Server application for IP Camera Surveillance (www.digifort.com.br), and one feature of it works like this, the server receives images from the IP Cameras and retransmits those images to the clients connected to this server, but I have have an input from the camera at any rate (for example 30FPS that will give me an avg of 900KB/s) and the clients can be connected over the internet, but the client internet bandwidth will not be able to receive 900KB/s of data, nor the server internet connection will be able to transmit the images to the client at this rate, so, if I just receive the images and try to send it without any verification to the client, my server internal buffer will grow, and grow and grow because I can´t send this data that I´m receiveing, fast enought. So, to solve this I have made a server client class derived from TWSocketClient and implements the following procedure: function TSocketConexao.GetCanSendWithoutBuffering: Boolean; begin FBufHandler.Lock; try //If the internal buffer is empty, return TRUE, else, return FALSE Result := FBufferedByteCount = 0; finally FBufHandler.UnLock; end; end; And I have overriden the Send function to be like this: function TSocketConexao.Send(const Data: TWSocketData; Len: Integer): Integer; begin //Default return Result := 0; //Check if we have enouth space on send buffer if (MaxSendBuffer > 0) and (SendBufferSize >= FMaxSendBuffer) then exit; //Send the data using the Send routine from base class Result := inherited Send(Data, Len); end; As you can see, I have also implemented a property called MaxSendBuffer, with this I can specify how much data I can store into ICS internal buffer, if I store data on it (It may store more data than MaxSendBuffer permits into one call, but on the next call if I have data into internal buffer that is bigger than MaxSendBuffer, then, this routine simply doesn´t send the data and returns 0 (Data cannot be sent) So externally, when I call my new send routine it can returns me the len of buffered data or 0 if data cannot be put into internal buffer, so, if I try to send and it returns me 0, I have to wait to try to send data again. On my particular case, I can just discart data that cannot be sent to the clients, because my data is camera images, and what will happen is that the client will just receive images at a lower frame rate, because the server is discarting the images that it cannot send to the client, but in your case I think that you have to send all the data, so, you should implement some external checking, to make this flow control. Ps. My data is also random... Hope I could help you Éric Fleming Bonilha Digifort - Original Message - From: "Angus Robertson - Magenta Systems Ltd" <[EMAIL PROTECTED]> To: Sent: Sunday, November 19, 2006 8:22 AM Subject: Re: [twsocket] Flow control >> Use event OnDataSent to control the flow, and to avoid grow of >> TWSocket's send buffer. > > That is impossible for random data, unless an extra FIFO buffer is used > externally to TWSocket. > > Trying to understand how TIcsBufferHandler works, it appears to be > multiple 1,460 byte blocks (TIcsBuffer), with new blocks being added if > data can not be sent sufficiently fast. > > So I guess the answer to my question is there is no flow control and the > buffers will just keep growing if data can not be sent fast enough, > until the application runs out of memory. > > Angus > -- > To unsubscribe or change your settings for TWSocket mailing list > please goto http://www.elists.org/mailman/listinfo/twsocket > Visit our website at http://www.overbyte.be > -- To unsubscribe or change your settings for TWSocket mailing list please goto http://www.elists.org/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] Flow control
Partially correct. You don't need an extra FIFO buffer externally to TWSocket because TWSockets buffers are FIFO buffers. You are right that these will grow until you run out of memory if you cant send data as fast as you receive it. But you said in your case you cannot control the rate your random is arriving at (or did I misinterpret?) so what else can you do except keep buffering it, or error? You could write your own buffering to disk I suppose... Dan -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Angus Robertson - Magenta Systems Ltd Sent: 19 November 2006 11:22 To: twsocket@elists.org Subject: Re: [twsocket] Flow control > Use event OnDataSent to control the flow, and to avoid grow of > TWSocket's send buffer. That is impossible for random data, unless an extra FIFO buffer is used externally to TWSocket. Trying to understand how TIcsBufferHandler works, it appears to be multiple 1,460 byte blocks (TIcsBuffer), with new blocks being added if data can not be sent sufficiently fast. So I guess the answer to my question is there is no flow control and the buffers will just keep growing if data can not be sent fast enough, until the application runs out of memory. Angus -- To unsubscribe or change your settings for TWSocket mailing list please goto http://www.elists.org/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be -- To unsubscribe or change your settings for TWSocket mailing list please goto http://www.elists.org/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] Flow control
> Use event OnDataSent to control the flow, and to avoid grow of > TWSocket's send buffer. That is impossible for random data, unless an extra FIFO buffer is used externally to TWSocket. Trying to understand how TIcsBufferHandler works, it appears to be multiple 1,460 byte blocks (TIcsBuffer), with new blocks being added if data can not be sent sufficiently fast. So I guess the answer to my question is there is no flow control and the buffers will just keep growing if data can not be sent fast enough, until the application runs out of memory. Angus -- To unsubscribe or change your settings for TWSocket mailing list please goto http://www.elists.org/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] Flow control
>> I don't understand what you want. Obviously you can't make a TCP >> session running fatser than a TCP session ! > > But I can supply data faster than the session can support. The data > might be coming from another application and I'm sending it over a slow > modem, so there must be flow control somewhere when the TCP buffers > overflow. But I can not find it. This flow control is built into the TCP protocol. You don't have to do anything to handle it. TWSocket will buffer your data automatically. If you don't want to fill memory with lots of data, use OnDataSent event to "get more data". OnDataSent is triggered each time TWSocket has emptyed his buffer. FTP component use OnDataSent event top read more data from the file to send. > Send does not currently appear to check data has previously been sent > and there is space in the buffer. Send put data into TWSocket internal buffer. Data is removed from that buffer when winsock notifies the component it can send something. -- Contribute to the SSL Effort. Visit http://www.overbyte.be/eng/ssl.html -- [EMAIL PROTECTED] http://www.overbyte.be -- To unsubscribe or change your settings for TWSocket mailing list please goto http://www.elists.org/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] Flow control
Angus Robertson - Magenta Systems Ltd wrote: >> I don't understand what you want. Obviously you can't make a TCP >> session running fatser than a TCP session ! > > But I can supply data faster than the session can support. The data > might be coming from another application and I'm sending it over a > slow modem, so there must be flow control somewhere when the TCP > buffers overflow. But I can not find it. > > Send does not currently appear to check data has previously been sent > and there is space in the buffer. TWSocket has a built-in, dynamic send buffer, that's why calling TWSocket.Send always succeeds, real sending is done in the background. The component handles winsock buffer overflows (WSAEWOULDBLOCK)internaly. Use event OnDataSent to control the flow, and to avoid grow of TWSocket's send buffer. --- Arno Garrels [TeamICS] http://www.overbyte.be/eng/overbyte/teamics.html -- To unsubscribe or change your settings for TWSocket mailing list please goto http://www.elists.org/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] Flow control
> I don't understand what you want. Obviously you can't make a TCP > session running fatser than a TCP session ! But I can supply data faster than the session can support. The data might be coming from another application and I'm sending it over a slow modem, so there must be flow control somewhere when the TCP buffers overflow. But I can not find it. Send does not currently appear to check data has previously been sent and there is space in the buffer. Angus -- To unsubscribe or change your settings for TWSocket mailing list please goto http://www.elists.org/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] Flow control
> If I'm trying to send random data at a speed faster than a TCP socket > can support, I don't understand what you want. Obviously you can't make a TCP session running fatser than a TCP session ! You can make it run slower but not faster ! Or maybe you want to use UDP which is faster than TCP but lacks the error detection and correction TCP has. -- Contribute to the SSL Effort. Visit http://www.overbyte.be/eng/ssl.html -- [EMAIL PROTECTED] http://www.overbyte.be -- To unsubscribe or change your settings for TWSocket mailing list please goto http://www.elists.org/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
