This is a bit of a long story, but I felt it necessary to post the detail just in case someone has seen similar behaviour (or this is a known issue with parts). I've cross posted because there's a general ASP.Net theme as well as a SharePoint flavour.
I've been helping a customer diagnose performance issues on their SharePoint 2010 site. Specifically, the w3wp.exe process goes to 100% CPU for for an hour or so. Before digging too deeply, I looked into some of the code. Now, I'm no SharePoint developer but some things were obvious and were clearly the result of System.NullReferenceException based ASP.Net warnings in the event log and other things scared me. Once was disposing of system created objects, particularly the SP RootWeb object because they were surrounded the reference by a using(). Another thing that did not sit well was the 'hack' necessary to get LINQ working on sites with anonymous access - this is a public Internet facing web site. Microsoft's own documentation states that one should use SPQuery in these circumstances (until they improve the functionality in Microsoft.SharePoint.Linq anyway). In particular, there's a need to elevate privileges because SPLists are secured but anonymous access is required. So my theory is they did software development by Google and found this: http://jcapka.blogspot.com/2010/05/making-linq-to-sharepoint-work-for.html And: http://blogs.msdn.com/b/sowmyancs/archive/2010/09/19/linq-to-sharepoint-and-runwithelevatedprivileges.aspx I wish they'd found the latter but, sadly, they chose the former code to implement as a *static *helper method. There's a very subtle but important difference - the former doesn't behave nicely when you get an exception in the anonymous delegate code passed in because it does not set the HttpContext back to the original value and you get a NullReferenceException in the logs. This was the cause of one of the issues. Obvious. Fixed that. Job done? Nope. *Lesson learnt* - don't "develop by Google" unless you know what the code is doing, line by line. I included this part because I think there's an issue with the *static helper* ,but I will come back to that. Oddly, the exception that is thrown while executing the anonymous delegate is "An item with the same key has already been added.". Huh? All they were doing was enumerating the list. This has been reported by others. eg: http://social.technet.microsoft.com/Forums/en-US/sharepoint2010programming/thread/ff3c4212-0372-4088-972b-108c1eda2ee6/ No idea what's going on there (at this point). So I finally get access to the server when the issue is happening and managed to get a full memory dump of w3wp.exe. Immediately upon looking at the CLR stack of long running threads the problem was obvious but the cause is not so. System.Collections.Generic.Dictionary`2[[System.__Canon, mscorlib],[System.__Canon, mscorlib]].*FindEntry*(System.__Canon) System.Collections.Generic.Dictionary`2[[System.__Canon, mscorlib],[System.__Canon, mscorlib]].TryGetValue(System.__Canon, System.__Canon ByRef) Microsoft.SharePoint.Linq.Rules.ToEnumerableProcessor.GetEnumerableOperator(System.Reflection.MethodInfo) Microsoft.SharePoint.Linq.Rules.ToEnumerableProcessor.ConvertMethod(System.Linq.Expressions.MethodCallExpression, Context) Microsoft.SharePoint.Linq.Rules.ToEnumerableProcessor.<.cctor>b__13(System.Linq.Expressions.MethodCallExpression, Context) Microsoft.SharePoint.Linq.Rules.GuardedRule`4+<>c__DisplayClass3[[System.__Canon, mscorlib],[System.__Canon, mscorlib],[Microsoft.SharePoint.Linq.Rules.ToEnumerableProcessor+Context, Microsoft.SharePoint.Linq],[System.__Canon, mscorlib]].<.ctor>b__1(System.__Canon, Context) Microsoft.SharePoint.Linq.Rules.SwitchRule`3[[System.__Canon, mscorlib],[Microsoft.SharePoint.Linq.Rules.ToEnumerableProcessor+Context, Microsoft.SharePoint.Linq],[System.__Canon, mscorlib]].Apply(System.__Canon, Context) Microsoft.SharePoint.Linq.Rules.ToEnumerableProcessor.Process(System.Linq.Expressions.Expression, System.Collections.Generic.List`1> ByRef) Microsoft.SharePoint.Linq.SPLinqProvider.Rewrite(System.Linq.Expressions.Expression, System.Collections.Generic.List`1> ByRef) Microsoft.SharePoint.Linq.SPLinqProvider.RewriteAndCompile[[System.__Canon, mscorlib]](System.Linq.Expressions.Expression, System.Collections.Generic.List`1> ByRef) Microsoft.SharePoint.Linq.LinqQuery`1[[System.__Canon, mscorlib]].GetEnumerator() Dictionary objects aren't thread safe (if you're modifying them) and you can read more about that from Tess' blog. http://blogs.msdn.com/b/tess/archive/2009/12/21/high-cpu-in-net-app-using-a-static-generic-dictionary.aspx But my customer's not using Dictionary objects, anywhere! So I pulled out Reflector to follow this through. Turns out that Microsoft.SharePoint.Linq is, and a *static *one at that. But why did this happen in the first place??? I can see from the CLR stack traces that 8 long running threads are all stuck doing the same think (above) and the dump of stack objects shows that each thread is referencing the same object. In fact it looks like they are all executing the same code. My suggested fix it to follow the MSDN guidance and NOT use LINQ for anonymous access and use SPQuery and write the CAML. They are doing this now and we will see if that stops the CPU "spikes" (plateaus). Hope so, because I'm sick of looking at My questions are: - Is this a known or previously observed issue? - The "An item with the same key has already been added" is caused by the use of the static Dictionary and static helper? - Can it be that the use of a static helper combined with anonymous delegate which might get compiled into a totally static method be the cause (I don't know enough about the innards of the IL to know or check)? You probably want to see code, but I thought the blog posts should do the trick as there's not much more to it. -- *Richard Carde* E: rich...@carde.id.au
