To recap and conclude the message thread that I started 20th Feb:
- It is a real shame that that the Wiki at http://wiki.overbyte.be is still not working (3 weeks later). I appreciate that this service is provided for free by someone. Although only offering a limited amount of information, it was however still an invaluable reference. Maybe this could be provided as a download somewhere? - I am creating a TWSocketThrdServer object and setting the Multithreaded property to TRUE, and am not calling MessagePump. This does appear to be working fine, and I assume that this is the correct way to do this as I have had no comments back on this question. - First 2 thousand connections are really quick, remaining connections are increasingly slow to establish a connection. My server implementation is now significantly improved. The key to this has been to reduce the number of events (messages on the message queue). Less messages, each doing more work, means much more time getting things done. My initial implementation would fall over at 4K connections and I would get a 'message queue full' exception which was unrecoverable. By pushing data in larger chunks and handling all received data immediately, the number of events generated for a connection has been reduced dramatically. It is very important to note that when a OnDataAvalable event is triggered, it will continue to trigger until all the data has been read, and it has been shown to be very inefficient to only read some of the data waiting and allow another event to be generated. It is also significant that until all the data has been read, it is not even possible to do anything else - even close the connection! - Exceptions raised in OnClientDataSend. These were persistent and it is evident that properties are not the best way to determine the state of a connection in the OnClientDataSend event. Essentially there were two issues, events were generated, but only actually handled after the socket had been closed. This was in part due to the length of the message queue and in part due to the way data was being sent. In my initial post, I described how I was kick starting the send using the Server.Client[xx].Resume function. This is bad, very bad, even when checking if pointers are valid and State=wsConnected. This seems to create a whole flurry of event messages (most of which are redundant) and some of which occasionally generate events on objects that are no longer valid. Instead, to kick start a send, use SendData(nil,0) from your controlling thread. Similarly, it is not obvious where to create/destroy your own client thread classes. It seemed that Connect/Disconnect events would be an appropriate place, as my class would not be required when not connected. The reality of the situation is that even after a disconnect event has occurred, it is still possible to receive OnClientDataSent events. No amount of pointer/variable validity checking could resolve this (probably because of the multithreaded nature of the app) and so this constantly generated memory leaks, access violations etc. These issues were not obvious and only occurred under load, and as things just locked, it was impossible to find out exactly what was even happening. To resolve this, I have created my own class objects in the OnClientCreate (could also be in OnClientThreadAttached), and I have destroyed them in the OnClientThreadDetached. It would be nice if there was a OnClientDestroy function pointer as this is more logical. Obviously my own class is thread safe. - Finally, there were yet more problems, sometimes shown as exceptions, other times the app would just lock completely. This was impossible to track and there was nothing I could do to trace the cause of this. However by chance I did come across the cause. In my mind it is logical to read data from the buffers in the OnClientDataReceived event using a loop i.e. see how much data is available (RcvdData property) and then read it in as chunks using the Receive function. Strangely, this does not work and was the cause of the lockups. In fact, you can find yourself in a situation where there is 6K to read (reported by RcvdData) and yet the receive function will not give you this amount of data from the buffer (hence the apparent application lock-up with no exception events being generated. I must admit that I have not explored the exactly conditions under which this happens, but I do have a solution. Simply read data using the Receive function until you get back less data than you have asked for. This works faultlessly. Where am I now? Well, I currently have the bones of an application that I have completed initial testing on, which looks pretty good. I have tested (as before) using Win7 and Server 2008R2 and have a custom client which I run multiple copies. Through testing, I have discovered that even with the above issues unresolved, the server will operate quite happily sending data to clients over an extended test period. The real point of stress is when clients Connect and Disconnect on mass (particularly on disconnect). With this in mind my test setup evolved such that my clients would connect and disconnect in batches. All 7200 clients would disconnect in batches of 1200 clients over a total period of 1 second. Currently I have a test running (no exceptions / anomalies - and I have extensive error reporting built into the application) on an I7 machine using local loopback (server and clients running on the same machine) and both setups reliably allow 7200K connections, deliver (and also receive on the test clients) 115MB per second (just short of 1 gigabit in and out), and over the last 24 hours have performed 7 million connect/disconnect cycles with no memory leaks (and even show an even distribution of load across each of the 8 processor cores). Everything is looking pretty good! Summing Up What a fantastic bit of software the ICS suite is. I have rooted through quite a bit of its code, and its methodical and extensive functionality is as accomplished as its documentation is non-existent - to be clear, fantastic library, and so much functionality - but none of it is documented. I reviewed every network stack available (commercial and non-commercial) and chose this one to move the project forward (and I now know this was a good choice). I have used Indy for many years, which is so good in so many ways, but will never be able to do what ICS can do, purely because of its blocking architecture. I can honestly say that the responses from this message board have missed the mark. Only received responses from Angus, and only one aspect of his response (possible memory allocation limit reached) was relevant to one part of my initial set of questions, and this turned out to be a red herring. I do appreciate your time on this (Angus), and I also really do understand that a large proportion of the questions asked come from people without any knowledge of sockets etc and are kind of expecting you to educate them (I like to think that I do not fall into this category, but respect your opinion). This must be incredibly frustrating, and a complete waste of your time, however there are some questions that have been asked via the mailing list that are very specific, and just need someone who is 'in the know' to point us in the right direction. As expressed above, I have used Indy in the past and have read many a post by Remy Lebeau (who has an amazing and far reaching knowledge of, well everything) and this is what I was hoping for from this mailing list, but this was not my experience. But I really do thank you for your time, and appreciate your dedication to ICS. I know there has been a call for assistance in documenting everything and that it has been pretty much ignored, but I think that much of the documentation must come from the author. I know I have spent some time using ICS, but I still don't know about the intricacies and implications of each functions functionality - hence why I have spent such a long time resolving the issues that I have documented above. François Piette, I fully understand that you may not have the inclination to do this (documentation), but have you ever considered a KickStarter project in order to give you a financial incentive? Why the comments? I chose to use ICS, because in my view it was the best non-blocking suite available. I think I have read most posts relating to a requirement for non-blocking sockets and felt that there were only two options - either ICS or using windows sockets. I have also read every post I could find regarding ICS and decided this was worth a try, despite there being very little evidence to give me confidence that ICS could support thousands of clients in a streaming data (as opposed to transactional) paradigm. Additionally, I have come across some gems of info in this mailing list, and felt that I should reciprocate. So yes this is great software, and it can be used to build substantial sockets based apps (see stats above). It is definitely worth the hassle! Regards, Tim -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
