We have seen similar behaviour in a fairly large managed-code server application. The application has about five AppDomains. The default AppDomain uses the .NET TCP remoting channel to talk to a single local process, and one of the non-default AppDomains uses System.Net.Socket instances to talk to many remote client processes. All remoting operations are synchronous and all bare socket operations are asynchronous. With low numbers of simultaneous connections (i.e. less than the 300), the application performs well and correctly. With constant socket traffic, the memory usage (private bytes) rises to not-unreasonable levels (from 50 MB up to 300 MB or so).
However above a certain threshold (roughly 500 simultaneous socket connections) performance degrades badly and the process size rises to 500-800 MB. Examining the heap with WinDbg/SOS we see that the vast majority of objects on the heap are send and receive buffers (byte arrays). The obvious thing to do would be to check all the objects to see what their GC roots are... writing a WinDbg plugin to automate this process was more than we had time for though (is there an easier way?). We did check a number of objects and found that their GC roots were OverlappedAsyncResult objects or chains of buffers from the remoting channel. (OverlappedAsyncResult is the derived type that's returned from the asynchronous socket operations.) The large number of OverlappedAsyncResult objects is troubling, because they're finalizable. As an aside, can anyone give the rationale for making the base LazyAsyncResult class finalizable? It defines a finalizer so it can provide a convenience 'Cleanup' method to derived classes. OverlappedAsyncResult is the only class that takes advantage of this convenience method, but neither class does any useful cleanup... so it seems that incurring the performance hit of having a finalizable LazyAsyncResult is entirely unnecessary. Our current hypothesis is that at a certain point the single finalizer thread can't keep up with the influx of OverlappedAsyncResult objects, and those objects are causing buffers to remain alive, causing the heap to grow. We haven't developed a small reproduction case yet, or been able to verify this hypothesis. Any insight appreciated. Mike. -----Original Message----- From: Unmoderated discussion of advanced .NET topics. [mailto:[EMAIL PROTECTED] On Behalf Of Garry Barclay Sent: July 21, 2004 8:10 AM To: [EMAIL PROTECTED] Subject: [ADVANCED-DOTNET] Missing callbacks from asynchronous sockets? It appears that we are NOT receiving the (completion port) callback from an asynchronous socket (BeginSend/EndSend) call. The sockets are clients and the failure seems to be related to shutdowns on the socket. This has the effect of leaving a pinned GCHandle on the heap (pinning the send buffer). We've checked the process memory dump with WinDbg and have seen that the sockets are shutdown successfully. Checking the Socket class implementation (v1.1) with Reflector it's seems clear that IF the callback is made then the buffer should always be unpinned. We've added in static counters using interlocked.xxx to verify that the calls are definitely not reaching our code, and this confirms our suspicions. Because of the pinned buffers, heap compaction can't take place and we end up with high memory usage figures even though most of the heap is free space between those buffers. Anyone seen something like this before? =================================== This list is hosted by DevelopMentorR http://www.develop.com Some .NET courses you may be interested in: NEW! Guerrilla ASP.NET, 17 May 2004, in Los Angeles http://www.develop.com/courses/gaspdotnetls View archives and manage your subscription(s) at http://discuss.develop.com =================================== This list is hosted by DevelopMentor� http://www.develop.com Some .NET courses you may be interested in: >>> Error in line 15 of ADVANCED-DOTNET.MAILTPL: unknown formatting command <<< -> .NET Architecture and Design: Designing Distributed Applications with <-
