[twsocket] ICS - TWSocketThrdServer Client
Hi, This is a follow up on my posts in the same thread, mostly made in March 2015). After further extensive testing I have found the following bugs/issues (in addition to the ones mentioned in my previous posts) and have the following solutions: 1. After the TWSockets multithreaded server is created and has entered the wsListening state, it will stay in this state until it closed. However, if it has not actually had any connections, then even when it has been closed, it will stay in the wsListening state. The solution to this in your ServerClose function is to DisconnectAll, CloseDelayed, and then wait (Sleep) until either the state becomes wsClosed or the ClientCount is zero. By this point there should be no connections, but if there were many lots of connections then whilst closing the existing connections (8000), new ones may have been established. To combat this I re-issue a DisconnectAll and then Free then server. It's a messy solution to shutting down a multithreaded server, and there is potential for error conditions to creep in, but it does work for me. 2. In my server implementation, I have and atomic counter to keep track of the number of active sessions. When a client was authenticated (my own validation of initial data received on the connection) I would call InterlockedIncrement to update the session counter. Then when the client disconnected (and had authenticated) I then called OnClientDisconnect. My OnClientDisconnect also tidied up my own classes objects (and freed associated memory). This worked perfectly for 10's of millions of connect/disconnect cycles. But occasionally, with all clients disconnected, I noticed my session counter would suddenly read a positive value ie. 57 and that over time, it would increment in chunks. In my application, memory resources were also not being freed. After further investigation I found that in fact OnClientDisconnect does not actually get called for small blocks of disconnections, and that this happens only very occasionally, but randomly. My guess on what is happening is that the disconnect event is trying to fire after the client has been freed. In summary, do not use OnClientDisconnect to do anything you need to have done i.e. freeing memory, updating counters - instead use the OnClientDetach event to tidy up. Note to Angus: In one of your responses you mentioned that you had an unpredictable but very slow memory leak in one of your apps using the ICS multithreaded server - I suspect that this is the issue. Regards, Tim -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] ICS - TWSocketThrdServer Client
to Angus: In one of your responses you mentioned that you had an unpredictable but very slow memory leak in one of your apps using the ICS multithreaded server - I suspect that this is the issue. I don't used TWSocketThrdServer or threads for ICS sockets, and don't have any memory leaks. But I'll check what happens OnClientDisconnect to see if it causes other problems. Angus -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] ICS - TWSocketThrdServer Client
due to the BannerTooBusy string that is sent These banners do cause trouble, I might clear them, I got caught a long time ago with a new server. It is therefore necessary to change the 'Client.Close;' line to 'Client.CloseDelayed;' in 'OverbyteIcsWSocketS.pas' I'll have a look at that, maxclients is not something often reached. Angus -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] ICS - TWSocketThrdServer Client
Tim What is your application actually doing. I may need your services Joe Benyola UTI Solutions, Inc. 11 Sayer Ave, Suite 200 N Cherry Hill, NJ 08002 856.488.8448 x102 856.255.8412 fax Billing: UTI Solutions, Inc. P.O. Box 1056 Mount Laurel NJ 08054 Website: www.utisolutions.com On Mar 9, 2015, at 7:16 AM, Tim Hyde timh...@c21systems.com wrote: Update on extended stress testing: After four days of testing with 7200 clients everything is still looking good with 29 million connect/disconnect cycles, 16.5TB transmitted and peak memory consumption for the process at 500MB. I feel that there is a slight memory leak but nothing I can't live with (5MB over 29M connections and 4 days, and this may be due to some other unrelated issue within my application process) as even 5MB over 29 million connect/disconnects theoretically equates to 10 years of uninterrupted operation in my application. Additionally the situation could easily be monitored. Regards, Tim -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
[twsocket] ICS - TWSocketThrdServer Client
To recap and conclude the message thread that I started 20th Feb: - It is a real shame that that the Wiki at http://wiki.overbyte.be is still not working (3 weeks later). I appreciate that this service is provided for free by someone. Although only offering a limited amount of information, it was however still an invaluable reference. Maybe this could be provided as a download somewhere? - I am creating a TWSocketThrdServer object and setting the Multithreaded property to TRUE, and am not calling MessagePump. This does appear to be working fine, and I assume that this is the correct way to do this as I have had no comments back on this question. - First 2 thousand connections are really quick, remaining connections are increasingly slow to establish a connection. My server implementation is now significantly improved. The key to this has been to reduce the number of events (messages on the message queue). Less messages, each doing more work, means much more time getting things done. My initial implementation would fall over at 4K connections and I would get a 'message queue full' exception which was unrecoverable. By pushing data in larger chunks and handling all received data immediately, the number of events generated for a connection has been reduced dramatically. It is very important to note that when a OnDataAvalable event is triggered, it will continue to trigger until all the data has been read, and it has been shown to be very inefficient to only read some of the data waiting and allow another event to be generated. It is also significant that until all the data has been read, it is not even possible to do anything else - even close the connection! - Exceptions raised in OnClientDataSend. These were persistent and it is evident that properties are not the best way to determine the state of a connection in the OnClientDataSend event. Essentially there were two issues, events were generated, but only actually handled after the socket had been closed. This was in part due to the length of the message queue and in part due to the way data was being sent. In my initial post, I described how I was kick starting the send using the Server.Client[xx].Resume function. This is bad, very bad, even when checking if pointers are valid and State=wsConnected. This seems to create a whole flurry of event messages (most of which are redundant) and some of which occasionally generate events on objects that are no longer valid. Instead, to kick start a send, use SendData(nil,0) from your controlling thread. Similarly, it is not obvious where to create/destroy your own client thread classes. It seemed that Connect/Disconnect events would be an appropriate place, as my class would not be required when not connected. The reality of the situation is that even after a disconnect event has occurred, it is still possible to receive OnClientDataSent events. No amount of pointer/variable validity checking could resolve this (probably because of the multithreaded nature of the app) and so this constantly generated memory leaks, access violations etc. These issues were not obvious and only occurred under load, and as things just locked, it was impossible to find out exactly what was even happening. To resolve this, I have created my own class objects in the OnClientCreate (could also be in OnClientThreadAttached), and I have destroyed them in the OnClientThreadDetached. It would be nice if there was a OnClientDestroy function pointer as this is more logical. Obviously my own class is thread safe. - Finally, there were yet more problems, sometimes shown as exceptions, other times the app would just lock completely. This was impossible to track and there was nothing I could do to trace the cause of this. However by chance I did come across the cause. In my mind it is logical to read data from the buffers in the OnClientDataReceived event using a loop i.e. see how much data is available (RcvdData property) and then read it in as chunks using the Receive function. Strangely, this does not work and was the cause of the lockups. In fact, you can find yourself in a situation where there is 6K to read (reported by RcvdData) and yet the receive function will not give you this amount of data from the buffer (hence the apparent application lock-up with no exception events being generated. I must admit that I have not explored the exactly conditions under which this happens, but I do have a solution. Simply read data using the Receive function until you get back less data than you have asked for. This works faultlessly. Where am I now? Well, I currently have the bones of an application that I have completed initial testing on, which looks pretty good. I have tested (as before) using Win7 and Server 2008R2 and have a custom client which I run multiple copies. Through testing, I have discovered that even with the above issues unresolved, the server will operate quite happily
Re: [twsocket] ICS - TWSocketThrdServer Client
- What OS - I am using Win7 Ultimate and a Win Server 2008 for testing. Both 64-bit with at least 8GB? Windows Server 2008 (aka Vista) is getting old now, from 2008 R2 onwards 32-bit was abandoned. Local testing is however representative of both operation on a Win Server installation and using completely asynchronous clients. Maybe, maybe not. A fairer test would be several clients each running a few hundred sessions against the server, more real world. Better to test this now rather than when the project is live. Angus -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
[twsocket] ICS - TWSocketThrdServer Client
Hi Everyone I am trying to use the multithreaded version of ICS and have read all the mailing list archives, and searched the searchable archives (when they were working a month or so ago) and looked at the Wiki (when that was working some time ago) but have a few questions. By the way it would be great if the Wiki was fixed even though this is a basic reference, it is better than nothing when you're a newbie. My application consists of a producer, and a multithread server accepting client connections. As a client is created, they are allocated RX and TX FIFO's. As a client receivs data the Client.OnDataAvailable event is used to push data from the stack into the RX FIFO and then the data (which is a stream of packets) is processed (part packets are left in the FIFO until the rest of the packet arrives from the client). On the TX side the producer (actually a throttled stream of data, also packetised) needs to send data to all of the TWSocketThrdServer connected clients. Currently I do this by iterating through the list of clients in another thread, and access each client through the Server.Client[xx] property of the TWSocketThrdServer. For each client, I first push data (packetised stream of data) into the TX FIFO, and then use the Client.OnDataSent event copy take data from the TX FIFO and actually place it in the stack buffers, using Client.Send. As the OnDataSent event only fires after data has been sent, and there will be times that the FIFO is empty and so there is nothing to send, or cause the event to fire again, I force the event to be fired (as data is being pushed into the TX FIFO) by using the Server.Client[xx].Resume function. All of this is working, and as far as I can tell it is stable. I can connect and stream to approximately 3500 clients (500 per thread) simultaneously on an I7 machine (local loopback) which gives a total server to client data rate of approximately 450Mb/s and uses approximately 30% CPU. I am quite pleased with this! So here are my immediate questions: - When I push this to 4000 clients, things start to go a bit pear shaped and the server grinds to a halt and locks. Up until that point everything seems okay, clients connecting/disconnecting on mass is fine, data rate is maintained etc etc. Approx 4000 clients just kills it. Is there something I am not aware of? Any suggestions of things to look at? - In the documentation/mailing list (not really talking about the TWSocketThrdServer) it says that you need to ensure that the MessagePump is called. My Windows Service application creates the TWSocketThrdServer in a background thread and I have set Multithreaded to TRUE. Everything is working, but am I doing this correctly? - When my clients connect, the first 2 thousand connect quite quickly (within a couple of seconds), however the next 1 thousand take a lot longer to be accepted (up to 50 additional seconds) are there any options for the listen queue? - When I disconnect clients from the client end I sometimes get an exception raised in the OnClientDataSend event handler - [ESocketException][Socket is not connected (#10057 in Send)]. The event is obviously firing, but the socket has already closed. Is there a way to test for this condition so that the exception does not occur. I have tried using the Connected client property but still the exception is raised. Again, I can trap the exception and everything works, but is there a better way. I guess this is related to me using Resume to kick start the Data Send. - I was hoping to get nearer towards supporting 5K clients at 16KB/s per client but this is simply not possible at the moment. Are there any suggestions as to what can be done to achieve this or improve on what I have already done. Any other comments on what I am doing will also be appreciated. Regards, Tim -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] ICS - TWSocketThrdServer Client
looked at the Wiki (when that was working The SVN over night zips last successfully uploaded to the wiki server a week ago, I've asked François to chase his ISP. The SVN zips can also be downloaded from: http://www.magsys.co.uk/delphi/magics.asp but have not changed in the last week. I can connect and stream to approximately 3500 clients (500 per thread) simultaneously on an I7 machine What OS is this? Windows desktop systems are performance crippled to make you buy proper Windows Servers. There is only restricted non-paged memory in Windows, and I suspect you are hitting that limit. This has been discussed years here ago, but is not specifically an ICS issue, Googling for socket based applications using thousands of connections may bring some suggestions. I've personally only tested several hundred connections, never any more. Angus -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be