Well, this is a user's list, not a paid support channel. People here volunteer their time/expertise.
First of all, Solr 4.2 is very old. From what you're showing, you've simply grown too big for the server and are running into memory issues. Your choices are: 1> get a bigger machine and allocate more space to the JVM 2> use SolrCloud and split the index into 2 shards. 3> use docValues for all fields that are used for sorting, grouping or faceting (although I frankly don't remember if docValues were available in 4.2). Second, your stack traces reference SolrNET, which is a separate project. Not many people on this list cah nelp with that. Best, Erick On Mon, Jun 25, 2018 at 3:49 AM, Jagdeeshwar S <jagdeeshw...@revalsys.com> wrote: > Can you please update on this? > > > > *From:* Jagdeeshwar S [mailto:jagdeeshw...@revalsys.com] > *Sent:* 22 June 2018 10:41 > *To:* 'solr-user@lucene.apache.org' <solr-user@lucene.apache.org> > *Cc:* 'Raj Samala' <r...@revalsys.com> > *Subject:* Solr objects consuming more GC (Garbage collector) on our > application > > > > Hi Support, > > > > We are using Solr 4.2.0 version for one of our ecommerce application where > we are storing entire our catalogue / products. > > > > Application developed in ASP.Net 4.5.2 and hosted in IIS 8.5 > > > > Ecommerce application flow is > > > > 1. Home page > 2. Product Listing page > 3. Product Detail page > 4. Search products listing page > 5. Cart flow (Add to cart, login and payment) > > > > In above flow, We are using Solr in all first 4 steps where users come and > browse for the products and will calling the DB from 5th step. > > > > We are using the solr from last 4 years but recently we have encountered a > High CPU for servers. > > > > We have captured logs in the same time and found that Solr objects are > consuming more memory (GC). > > > > Below are the log details . Can you please help us to identify the issue. > > > > From the below analysis its pretty clear that the high cpu issue is due to > the garbage collection being triggered very frequently due to the HIGH > allocation rate from within your application. > > > > The objects whose allocations are on the higher side is mentioned below. > They are all rooted to the above highlighted *SolrNet* component which is > highlighted in GREEN. > > > > > > > > 0:105> lmvm SolrNet > > Browse full module list > > start end module name > > 0000004c`57d50000 0000004c`57db0000 SolrNet (deferred) > > Image path: SolrNet.dll > > Image name: SolrNet.dll > > Browse all global symbols functions data > > Using CLR debugging support for all symbols > > Has CLR image header, track-debug-data flag not set > > Timestamp: Tue Apr 16 03:52:53 2013 (516C7DBD) > > CheckSum: 00000000 > > ImageSize: 00060000 > > File version: 0.4.0.2002 > > Product version: 0.4.0.2002 > > File flags: 0 (Mask 3F) > > File OS: 4 Unknown Win32 > > File type: 2.0 Dll > > File date: 00000000.00000000 > > Translations: 0000.04b0 > > ProductName: SolrNet > > InternalName: SolrNet.dll > > OriginalFilename: SolrNet.dll > > ProductVersion: 0.4.0.2002 > > FileVersion: 0.4.0.2002 > > FileDescription: SolrNet > > LegalCopyright: Copyright Mauricio Scheffer 2007-2013 > > Comments: SolrNet > > > > CPU utilization: 100% > > Worker Thread: Total: 53 Running: 18 Idle: 35 MaxLimit: 800 MinLimit: 8 > > Work Request in Queue: 0 > > -------------------------------------- > > Number of Timers: 2 > > -------------------------------------- > > Completion Port Thread:Total: 4 Free: 4 MaxFree: 16 CurrentLimit: 4 > MaxLimit: 800 MinLimit: 200 > > > > > > Top 10 threads which are consuming HIGH CPU cycles are below: > > > > Showing top 10 threads > > Thread ID User Time > > ============================== > > 58 | 0 days 0:00:26.812 > > 64 | 0 days 0:00:23.750 > > 55 | 0 days 0:00:23.718 > > 75 | 0 days 0:00:22.546 > > 47 | 0 days 0:00:21.875 > > 46 | 0 days 0:00:21.625 > > 63 | 0 days 0:00:18.953 > > 22 | 0 days 0:00:18.921 > > 24 | 0 days 0:00:18.453 > > 28 | 0 days 0:00:18.359 > > ============================== > > Thread ID User Time > > > > > > Taking one of the random thread from above, I could see the below > callstack: > > > > 0:064> kL > > # Child-SP RetAddr Call Site > > 00 0000004c`5ea1ab38 00007ffa`057e1118 ntdll!ZwWaitForSingleObject+0xa > > 01 0000004c`5ea1ab40 00007ff9`fdc07a1f KERNELBASE! > WaitForSingleObjectEx+0x94 > > 02 0000004c`5ea1abe0 00007ff9`fdc079d7 clr!CLREventWaitHelper2+0x3c > > 03 0000004c`5ea1ac20 00007ff9`fdc07958 clr!CLREventWaitHelper+0x1f > > 04 0000004c`5ea1ac80 00007ff9`fdc14c2d clr!CLREventBase::WaitEx+0x7c > > 05 (Inline Function) --------`-------- clr!CLREventBase::Wait+ > 0xffffffff`fffa63f1 > > 06 0000004c`5ea1ad10 00007ff9`fdc14ef4 clr!SVR::gc_heap::wait_for_gc_done > +0x66 > > 07 0000004c`5ea1ad40 00007ff9`fdc06709 clr!SVR::GCHeap:: > GarbageCollectGeneration+0x108 > > 08 (Inline Function) --------`-------- clr!SVR::gc_heap::try_ > allocate_more_space+0x535 > > 09 (Inline Function) --------`-------- clr!SVR::gc_heap::allocate_ > more_space+0x54a > > 0a (Inline Function) --------`-------- clr!SVR::gc_heap::allocate+0x5a1 > > 0b (Inline Function) --------`-------- clr!SVR::GCHeap::Alloc+0x601 > > 0c (Inline Function) --------`-------- clr!Alloc+0x961 > > 0d (Inline Function) --------`-------- clr!AllocateObject+0x9e3 > > 0e 0000004c`5ea1ada0 00007ff9`a0190d0a clr!JIT_New+0xac9 > > 0f 0000004c`5ea1b1e0 00007ff9`a018fb43 SolrNet!SolrNet.Impl.FieldParsers. > AggregateFieldParser.CanHandleType(System.Type)+0x3a > > 10 0000004c`5ea1b220 00007ff9`a018f9c6 SolrNet!SolrNet.Impl. > DocumentPropertyVisitors.RegularDocumentVisitor.Visit(System.Object, > System.String, System.Xml.Linq.XElement)+0xe3 > > 11 0000004c`5ea1b290 00007ff9`a018f7d1 SolrNet!SolrNet.Impl. > DocumentPropertyVisitors.AggregateDocumentVisitor.Visit(System.Object, > System.String, System.Xml.Linq.XElement)+0x76 > > 12 0000004c`5ea1b2f0 00007ff9`a0179c3c SolrNet!SolrNet.Impl. > SolrDocumentResponseParser`1[[System.__Canon, mscorlib]].ParseDocument( > System.Xml.Linq.XElement)+0x121 > > 13 0000004c`5ea1b350 00007ff9`a017959e SolrNet!SolrNet.Impl. > SolrDocumentResponseParser`1[[System.__Canon, mscorlib]].ParseResults( > System.Xml.Linq.XElement)+0xec > > 14 0000004c`5ea1b3b0 00007ff9`a0178b56 SolrNet!SolrNet.Impl. > ResponseParsers.ResultsResponseParser`1[[System.__Canon, > mscorlib]].Parse(System.Xml.Linq.XDocument, SolrNet.Impl. > AbstractSolrQueryResults`1<System.__Canon>)+0x28e > > 15 0000004c`5ea1b430 00007ff9`a0032c1e SolrNet!SolrNet.Impl. > ResponseParsers.AggregateResponseParser`1[[System.__Canon, > mscorlib]].Parse(System.Xml.Linq.XDocument, SolrNet.Impl. > AbstractSolrQueryResults`1<System.__Canon>)+0xc6 > > 16 0000004c`5ea1b4a0 00007ff9`9f97ceef SolrNet!SolrNet.Impl. > SolrQueryExecuter`1[[System.__Canon, mscorlib]].Execute(SolrNet.ISolrQuery, > SolrNet.Commands.Parameters.QueryOptions)+0xae > > 17 0000004c`5ea1b500 00007ff9`9f973145 RevalBusinessLogic!Revalsys. > BusinessLogic.ProductBAL.SearchProducts(JockeyAPI. > Models.RequestQueryListDTO)+0x1aff > > 18 0000004c`5ea1c3d0 00007ff9`9f4f1c0b JockeyGetAPIs!JockeyGetAPI. > Controllers.SearchProductsController.GET(JockeyAPI.Models. > RequestQueryListDTO)+0x165 > > 19 0000004c`5ea1c540 00007ff9`9f96da55 DynamicClass.lambda_method( > System.Runtime.CompilerServices.Closure, System.Object, > System.Object[])+0x8b > > 1a 0000004c`5ea1c580 00007ff9`9f95f680 System_Web_Http!System.Web. > Http.Controllers.ReflectedHttpActionDescriptor+ActionExecutor+<>c__ > DisplayClass10.<GetExecutor>b__9(System.Object, System.Object[])+0x15 > > 1b 0000004c`5ea1c5c0 00007ff9`9f95e494 System_Web_Http!System.Web. > Http.Controllers.ReflectedHttpActionDescriptor. > ExecuteAsync(System.Web.Http.Controllers.HttpControllerContext, > System.Collections.Generic.IDictionary`2<System.String,System.Object>, > System.Threading.CancellationToken)+0x80 > > 1c 0000004c`5ea1c620 00007ff9`9f95deb5 System_Web_Http!System.Web. > Http.Controllers.ApiControllerActionInvoker+<InvokeActionAsyncCore>d__0. > MoveNext()+0xb4 > > 1d 0000004c`5ea1c690 00007ff9`9f95de02 mscorlib!System.Runtime. > CompilerServices.AsyncTaskMethodBuilder`1[[System.Int32, > mscorlib]].Start[[System.IO.StreamReader+<ReadBufferAsync>d__97, > mscorlib]](<ReadBufferAsync>d__97 ByRef)+0x4ddb45 > > 1e 0000004c`5ea1c740 00007ff9`9f95dd05 System_Web_Http!System.Web. > Http.Controllers.ApiControllerActionInvoker.InvokeActionAsyncCore(System. > Web.Http.Controllers.HttpActionContext, System.Threading. > CancellationToken)+0xa2 > > 1f 0000004c`5ea1c820 00007ff9`9f95d7d4 System_Web_Http!System.Web. > Http.Controllers.ApiControllerActionInvoker.InvokeActionAsync(System.Web. > Http.Controllers.HttpActionContext, System.Threading. > CancellationToken)+0x15 > > 20 0000004c`5ea1c850 00007ff9`9f95d5a5 System_Web_Http!System.Web. > Http.Controllers.ActionFilterResult+<ExecuteAsync>d__2.MoveNext()+0x1c4 > > 21 0000004c`5ea1c8f0 00007ff9`9f95d4f9 mscorlib!System.Runtime. > CompilerServices.AsyncTaskMethodBuilder`1[[System.Int32, > mscorlib]].Start[[System.IO.StreamReader+<ReadBufferAsync>d__97, > mscorlib]](<ReadBufferAsync>d__97 ByRef)+0x4dd235 > > 22 0000004c`5ea1c9a0 00007ff9`9f95ca3d System_Web_Http!System.Web. > Http.Controllers.ActionFilterResult.ExecuteAsync(System.Threading. > CancellationToken)+0x99 > > 23 0000004c`5ea1ca70 00007ff9`9f95c8a5 System_Web_Http!System.Web. > Http.Filters.AuthorizationFilterAttribute+<ExecuteAuthorizationFilterAsyn > cCore>d__2.MoveNext()+0x12d > > 24 0000004c`5ea1cac0 00007ff9`9f95c7f3 mscorlib!System.Runtime. > CompilerServices.AsyncTaskMethodBuilder`1[[System.Int32, > mscorlib]].Start[[System.IO.StreamReader+<ReadBufferAsync>d__97, > mscorlib]](<ReadBufferAsync>d__97 ByRef)+0x4dc535 > > 25 0000004c`5ea1cb70 00007ff9`9f95c6b4 System_Web_Http!System.Web. > Http.Filters.AuthorizationFilterAttribute.ExecuteAuthorizationFilterAsyn > cCore(System.Web.Http.Controllers.HttpActionContext, > System.Threading.CancellationToken, > System.Func`1<System.Threading.Tasks.Task`1<System. > Net.Http.HttpResponseMessage>>)+0xa3 > > 26 0000004c`5ea1cc30 00007ff9`9f95c572 System_Web_Http!System.Web. > Http.Filters.AuthorizationFilterAttribute.System.Web.Http.Filters. > IAuthorizationFilter.ExecuteAuthorizationFilterAsyn > c(System.Web.Http.Controllers.HttpActionContext, > System.Threading.CancellationToken, > System.Func`1<System.Threading.Tasks.Task`1<System. > Net.Http.HttpResponseMessage>>)+0x14 > > 27 0000004c`5ea1cc60 00007ff9`9f95c160 System_Web_Http!System.Web. > Http.Controllers.AuthorizationFilterResult.ExecuteAsync(System.Threading. > CancellationToken)+0xe2 > > 28 0000004c`5ea1ccc0 00007ff9`9f4896b5 > *JockeyGetAPIs!JockeyGetAPI.Filters.AddChallengeOnUnauthorizedResult+<ExecuteAsync>d__7.MoveNext* > ()+0x90 > > 29 0000004c`5ea1cdb0 00007ff9`9f95c031 mscorlib!System.Runtime. > CompilerServices.AsyncTaskMethodBuilder`1[[System.Int32, > mscorlib]].Start[[System.IO.StreamReader+<ReadBufferAsync>d__97, > mscorlib]](<ReadBufferAsync>d__97 ByRef)+0x9345 > > 2a 0000004c`5ea1ce60 00007ff9`9f5e0db3 JockeyGetAPIs!JockeyGetAPI.Filters. > AddChallengeOnUnauthorizedResult.ExecuteAsync(System. > Threading.CancellationToken)+0xf1 > > 2b 0000004c`5ea1cef0 00007ff9`9f5e0865 System_Web_Http!System.Web. > Http.Controllers.AuthenticationFilterResult+<ExecuteAsync>d__0.MoveNext()+ > 0x4e3 > > > > > > From the above callstack, we see that this thread is allocating the memory > and it has triggered the GC. > > > > > > GC (alloc) > > ---------- > > Allocating Memory True > > Causing GC True > > Allocating Large Object False > > Waiting for GC True > > > > Because of this thread there are all the below threads which are waiting > for the GC to finish. > > > > > > GC Kind: Server > > Finalizer thread: 29 > > > > GC triggered by thread: 64 > > > > Threads waiting for GC: > > 46, 56, 66, 79, 80, 82, 84, 93, 94, 96, 101, 102, 103, 104, 109, 118, 119 > > > > GC Threads: 21, 22, 23, 24, 25, 26, 27, 28, 67, 68, 69, 70, 71, 72, 73, 74 > > > > > > Randomly picking one of the waiting thread it looks like below: > > > > 0:082> kL > > # Child-SP RetAddr Call Site > > 00 0000004c`5f42af78 00007ffa`057e1118 ntdll!ZwWaitForSingleObject+0xa > > 01 0000004c`5f42af80 00007ff9`fdc07a1f KERNELBASE! > WaitForSingleObjectEx+0x94 > > 02 0000004c`5f42b020 00007ff9`fdc079d7 clr!CLREventWaitHelper2+0x3c > > 03 0000004c`5f42b060 00007ff9`fdc07958 clr!CLREventWaitHelper+0x1f > > 04 0000004c`5f42b0c0 00007ff9`fdc14cae clr!CLREventBase::WaitEx+0x7c > > 05 (Inline Function) --------`-------- clr!CLREventBase::Wait+0x15 > > 06 0000004c`5f42b150 00007ff9`fdc0d2c3 clr!SVR::GCHeap::WaitUntilGCCo > mplete+0x2f > > 07 0000004c`5f42b180 00007ff9`fdc11ddd clr!Thread:: > RareDisablePreemptiveGC+0x180 > > 08 (Inline Function) --------`-------- clr!Thread:: > DisablePreemptiveGC+0x17bafc > > 09 (Inline Function) --------`-------- clr!GCHolderBase:: > EnterInternalCoop+0x17bb0c > > 0a 0000004c`5f42b210 00007ff9`fdb3cfee clr!GCCoop::GCCoop+0x3d > > 0b 0000004c`5f42b240 00007ff9`9e8c37d5 clr!RuntimeTypeHandle:: > GetGenericTypeDefinition+0xc6 > > 0c 0000004c`5f42b310 00007ff9`9e8ecf6c mscorlib! > DomainNeutralILStubClass.IL_STUB_PInvoke(System.RuntimeTypeHandle, > System.Runtime.CompilerServices.ObjectHandleOnStack)+0x85 > > 0d 0000004c`5f42b3c0 00007ff9`a0190eec mscorlib!System.RuntimeTypeHandle. > GetGenericTypeDefinition(System.RuntimeType)+0x4c > > 0e 0000004c`5f42b400 00007ff9`a0190e71 SolrNet!SolrNet.Utils. > TypeHelper.IsNullableType(System.Type)+0x2c > > 0f 0000004c`5f42b430 00007ff9`a0190dd2 SolrNet!SolrNet.Utils.TypeHelper. > GetUnderlyingNullableType(System.Type)+0x11 > > 10 0000004c`5f42b470 00007ff9`9eb063b0 SolrNet!SolrNet.Impl.FieldParsers. > NullableFieldParser.CanHandleType(System.Type)+0x32 > > 11 0000004c`5f42b4b0 00007ff9`a018fb43 System_Core!System.Linq. > Enumerable.Any[[System.__Canon, mscorlib]](System.Collections. > Generic.IEnumerable`1<System.__Canon>, System.Func`2<System.__Canon, > Boolean>)+0xb0 > > 12 0000004c`5f42b520 00007ff9`a018f9c6 SolrNet!SolrNet.Impl. > DocumentPropertyVisitors.RegularDocumentVisitor.Visit(System.Object, > System.String, System.Xml.Linq.XElement)+0xe3 > > 13 0000004c`5f42b590 00007ff9`a018f7d1 SolrNet!SolrNet.Impl. > DocumentPropertyVisitors.AggregateDocumentVisitor.Visit(System.Object, > System.String, System.Xml.Linq.XElement)+0x76 > > 14 0000004c`5f42b5f0 00007ff9`a0179c3c SolrNet!SolrNet.Impl. > SolrDocumentResponseParser`1[[System.__Canon, mscorlib]].ParseDocument( > System.Xml.Linq.XElement)+0x121 > > 15 0000004c`5f42b650 00007ff9`a017959e SolrNet!SolrNet.Impl. > SolrDocumentResponseParser`1[[System.__Canon, mscorlib]].ParseResults( > System.Xml.Linq.XElement)+0xec > > 16 0000004c`5f42b6b0 00007ff9`a0178b56 SolrNet!SolrNet.Impl. > ResponseParsers.ResultsResponseParser`1[[System.__Canon, > mscorlib]].Parse(System.Xml.Linq.XDocument, SolrNet.Impl. > AbstractSolrQueryResults`1<System.__Canon>)+0x28e > > 17 0000004c`5f42b730 00007ff9`a0032c1e SolrNet!SolrNet.Impl. > ResponseParsers.AggregateResponseParser`1[[System.__Canon, > mscorlib]].Parse(System.Xml.Linq.XDocument, SolrNet.Impl. > AbstractSolrQueryResults`1<System.__Canon>)+0xc6 > > 18 0000004c`5f42b7a0 00007ff9`9f97ce51 SolrNet!SolrNet.Impl. > SolrQueryExecuter`1[[System.__Canon, mscorlib]].Execute(SolrNet.ISolrQuery, > SolrNet.Commands.Parameters.QueryOptions)+0xae > > 19 0000004c`5f42b800 00007ff9`9f973145 > *RevalBusinessLogic!Revalsys.BusinessLogic.ProductBAL.SearchProducts* > (JockeyAPI.Models.RequestQueryListDTO)+0x1a61 > > 1a 0000004c`5f42c6d0 00007ff9`9f4f1c0b > *JockeyGetAPIs!JockeyGetAPI.Controllers.SearchProductsController.GET*( > JockeyAPI.Models.RequestQueryListDTO)+0x165 > > 1b 0000004c`5f42c840 00007ff9`9f96da55 DynamicClass.lambda_method( > System.Runtime.CompilerServices.Closure, System.Object, > System.Object[])+0x8b > > 1c 0000004c`5f42c880 00007ff9`9f95f680 System_Web_Http!System.Web. > Http.Controllers.ReflectedHttpActionDescriptor+ActionExecutor+<>c__ > DisplayClass10.<GetExecutor>b__9(System.Object, System.Object[])+0x15 > > 1d 0000004c`5f42c8c0 00007ff9`9f95e494 System_Web_Http!System.Web. > Http.Controllers.ReflectedHttpActionDescriptor. > ExecuteAsync(System.Web.Http.Controllers.HttpControllerContext, > System.Collections.Generic.IDictionary`2<System.String,System.Object>, > System.Threading.CancellationToken)+0x80 > > 1e 0000004c`5f42c920 00007ff9`9f95deb5 System_Web_Http!System.Web. > Http.Controllers.ApiControllerActionInvoker+<InvokeActionAsyncCore>d__0. > MoveNext()+0xb4 > > 1f 0000004c`5f42c990 00007ff9`9f95de02 mscorlib!System.Runtime. > CompilerServices.AsyncTaskMethodBuilder`1[[System.Int32, > mscorlib]].Start[[System.IO.StreamReader+<ReadBufferAsync>d__97, > mscorlib]](<ReadBufferAsync>d__97 ByRef)+0x4ddb45 > > 20 0000004c`5f42ca40 00007ff9`9f95dd05 System_Web_Http!System.Web. > Http.Controllers.ApiControllerActionInvoker.InvokeActionAsyncCore(System. > Web.Http.Controllers.HttpActionContext, System.Threading. > CancellationToken)+0xa2 > > 21 0000004c`5f42cb20 00007ff9`9f95d7d4 System_Web_Http!System.Web. > Http.Controllers.ApiControllerActionInvoker.InvokeActionAsync(System.Web. > Http.Controllers.HttpActionContext, System.Threading. > CancellationToken)+0x15 > > 22 0000004c`5f42cb50 00007ff9`9f95d5a5 System_Web_Http!System.Web. > Http.Controllers.ActionFilterResult+<ExecuteAsync>d__2.MoveNext()+0x1c4 > > > > Note that when the GC is running you will tend to see the HIGH CPU issue > for your application. > > > > > > Below is the statistical view of the managed heap statistics: > > > > Statistics: > > MT Count TotalSize Class Name > > 00007ff9a02f3a60 1020573 65316672 System.Linq.Enumerable+ > WhereSelectArrayIterator`2[[SolrNet.Impl.ISolrFieldParser, > SolrNet],[System.Object, mscorlib]] > > 00007ff9a02f34e0 1020574 65316736 > System.Func`2[[SolrNet.Impl.ISolrFieldParser, > SolrNet],[System.Object, mscorlib]] > > 00007ff99e57c670 1587569 93134312 System.String[] > > 00007ff99f319370 96828 108447360 Products > > 00007ff9a02f2c10 2722155 174217920 > System.Func`2[[SolrNet.Impl.ISolrFieldParser, > SolrNet],[System.Boolean, mscorlib]] > > 00007ff99e57c828 147913 290037238 System.Char[] > > 00000047c3a93130 1047425 905560344 Free > > 00007ff99ed7f310 16817454 941777424 System.Xml.Linq.XAttribute > > 00007ff99ed7df48 21076275 1348881600 System.Xml.Linq.XElement > > 00007ff99e574da8 33954691 2851256584 System.String > > > > > > > > I could see the majority of the allocation is happening within the above > highlighted objects: > > > > Seeing where the objects System.Xml.Linq.XAttribute and > System.Xml.Linq.XElement are rooted, I could see below: > > > > > > Thread a24: > > 00000047c423b000 00007ff9a018fb43 SolrNet.Impl. > DocumentPropertyVisitors.RegularDocumentVisitor.Visit(System.Object, > System.String, System.Xml.Linq.XElement) > > rbx: > > -> 00000049f5938328 System.Xml.Linq.XElement > > -> 00000049f5931508 System.Xml.Linq.XElement > > -> 00000047c4c19060 System.Xml.Linq.XElement > > -> 00000047c4c19178 System.Xml.Linq.XElement > > -> 00000047c4c192b8 System.Xml.Linq.XElement > > -> 00000047c4c19378 System.Xml.Linq.XElement > > -> 00000047c4c19660 System.Xml.Linq.XElement > > -> 00000047c4c19740 System.Xml.Linq.XElement > > -> 00000047c4c19818 System.Xml.Linq.XElement > > -> 00000047c4c198e0 System.Xml.Linq.XElement > > -> 00000047c4631bc8 *System.Xml.Linq.XElement* > > -> 00000047c4631c08 *System.Xml.Linq.XAttribute* > > > > > > Checking how Products object is rooted: > > > > Thread 14bc: > > 00000047c414b680 00007ff9a0179c3c SolrNet.Impl. > SolrDocumentResponseParser`1[[System.__Canon, mscorlib]].ParseResults( > System.Xml.Linq.XElement) > > rbp-20: 00000047c414b6b0 > > -> 0000004b5d5d6718 System.Collections.Generic.List`1[[Products, > RevalProperties]] > > -> 0000004944595230 Products[] > > -> 00000047c4432d40 Products > > > > > > > > Majority of the allocations are happening within the Gen 2 space: > > > > GEN 0 Heap Size 1,234,204,736 > > GEN 1 Heap Size 612,922,856 > > GEN 2 Heap Size 4,731,535,392 > > LOH Size 1,131,433,408 > > % Time in GC 38.83% > > > > > > > > In the other dump *10-210-2-6w3wp.exe_180614_134512.dmp* , I see the > below thread is the one which is triggering the GC due to allocation. > > > > > > 0:084> kL > > # Child-SP RetAddr Call Site > > 00 0000004c`60e7b178 00007ffa`057e1118 ntdll!ZwWaitForSingleObject+0xa > > 01 0000004c`60e7b180 00007ff9`fdc07a1f KERNELBASE! > WaitForSingleObjectEx+0x94 > > 02 0000004c`60e7b220 00007ff9`fdc079d7 clr!CLREventWaitHelper2+0x3c > > 03 0000004c`60e7b260 00007ff9`fdc07958 clr!CLREventWaitHelper+0x1f > > 04 0000004c`60e7b2c0 00007ff9`fdc14c2d clr!CLREventBase::WaitEx+0x7c > > 05 (Inline Function) --------`-------- clr!CLREventBase::Wait+ > 0xffffffff`fffa63f1 > > 06 0000004c`60e7b350 00007ff9`fdc14ef4 clr!SVR::gc_heap::wait_for_gc_ > done+0x66 > > 07 0000004c`60e7b380 00007ff9`fdc06709 clr!SVR::GCHeap:: > GarbageCollectGeneration+0x108 > > 08 (Inline Function) --------`-------- clr!SVR::gc_heap::try_ > allocate_more_space+0x535 > > 09 (Inline Function) --------`-------- clr!SVR::gc_heap::allocate_ > more_space+0x54a > > 0a (Inline Function) --------`-------- clr!SVR::gc_heap::allocate+0x5a1 > > 0b (Inline Function) --------`-------- clr!SVR::GCHeap::Alloc+0x601 > > 0c (Inline Function) --------`-------- clr!Alloc+0x961 > > 0d (Inline Function) --------`-------- clr!AllocateObject+0x9e3 > > 0e 0000004c`60e7b3e0 00007ff9`9ec6c8d7 clr!JIT_New+0xac9 > > 0f 0000004c`60e7b820 00007ff9`9ec6bd31 System_Xml_Linq!System.Xml. > Linq.XContainer.ReadContentFrom(System.Xml.XmlReader)+0xb7 > > 10 0000004c`60e7b8b0 00007ff9`9ec6bb25 System_Xml_Linq!System.Xml. > Linq.XContainer.ReadContentFrom(System.Xml.XmlReader, > System.Xml.Linq.LoadOptions)+0x41 > > 11 0000004c`60e7b9f0 00007ff9`9ec6a32c System_Xml_Linq!System.Xml. > Linq.XDocument.Load(System.Xml.XmlReader, System.Xml.Linq.LoadOptions)+ > 0x165 > > 12 0000004c`60e7ba40 00007ff9`a0032beb System_Xml_Linq!System.Xml. > Linq.XDocument.Parse(System.String, System.Xml.Linq.LoadOptions)+0x9c > > 13 0000004c`60e7bab0 00007ff9`9f97ce51 > *SolrNet!SolrNet.Impl.SolrQueryExecuter*`1[[System.__Canon, > mscorlib]].Execute(SolrNet.ISolrQuery, SolrNet.Commands.Parameters. > QueryOptions)+0x7b > > 14 0000004c`60e7bb10 00007ff9`9f973145 RevalBusinessLogic!Revalsys. > BusinessLogic.ProductBAL.SearchProducts(JockeyAPI. > Models.RequestQueryListDTO)+0x1a61 > > 15 0000004c`60e7c9e0 00007ff9`9f4f1c0b > *JockeyGetAPIs!JockeyGetAPI.Controllers.SearchProductsController.GET*( > JockeyAPI.Models.RequestQueryListDTO)+0x165 > > 16 0000004c`60e7cb50 00007ff9`9f96da55 DynamicClass.lambda_method( > System.Runtime.CompilerServices.Closure, System.Object, > System.Object[])+0x8b > > 17 0000004c`60e7cb90 00007ff9`9f95f680 System_Web_Http!System.Web. > Http.Controllers.ReflectedHttpActionDescriptor+ActionExecutor+<>c__ > DisplayClass10.<GetExecutor>b__9(System.Object, System.Object[])+0x15 > > 18 0000004c`60e7cbd0 00007ff9`9f95e494 System_Web_Http!System.Web. > Http.Controllers.ReflectedHttpActionDescriptor. > ExecuteAsync(System.Web.Http.Controllers.HttpControllerContext, > System.Collections.Generic.IDictionary`2<System.String,System.Object>, > System.Threading.CancellationToken)+0x80 > > 19 0000004c`60e7cc30 00007ff9`9f95deb5 System_Web_Http!System.Web. > Http.Controllers.ApiControllerActionInvoker+<InvokeActionAsyncCore>d__0. > MoveNext()+0xb4 > > 1a 0000004c`60e7cca0 00007ff9`9f95de02 mscorlib!System.Runtime. > CompilerServices.AsyncTaskMethodBuilder`1[[System.Int32, > mscorlib]].Start[[System.IO.StreamReader+<ReadBufferAsync>d__97, > mscorlib]](<ReadBufferAsync>d__97 ByRef)+0x4ddb45 > > 1b 0000004c`60e7cd50 00007ff9`9f95dd05 System_Web_Http!System.Web. > Http.Controllers.ApiControllerActionInvoker.InvokeActionAsyncCore(System. > Web.Http.Controllers.HttpActionContext, System.Threading. > CancellationToken)+0xa2 > > 1c 0000004c`60e7ce30 00007ff9`9f95d7d4 System_Web_Http!System.Web. > Http.Controllers.ApiControllerActionInvoker.InvokeActionAsync(System.Web. > Http.Controllers.HttpActionContext, System.Threading. > CancellationToken)+0x15 > > 1d 0000004c`60e7ce60 00007ff9`9f95d5a5 System_Web_Http!System.Web. > Http.Controllers.ActionFilterResult+<ExecuteAsync>d__2.MoveNext()+0x1c4 > > 1e 0000004c`60e7cf00 00007ff9`9f95d4f9 mscorlib!System.Runtime. > CompilerServices.AsyncTaskMethodBuilder`1[[System.Int32, > mscorlib]].Start[[System.IO.StreamReader+<ReadBufferAsync>d__97, > mscorlib]](<ReadBufferAsync>d__97 ByRef)+0x4dd235 > > 1f 0000004c`60e7cfb0 00007ff9`9f95ca3d System_Web_Http!System.Web. > Http.Controllers.ActionFilterResult.ExecuteAsync(System.Threading. > CancellationToken)+0x99 > > 20 0000004c`60e7d080 00007ff9`9f95c8a5 System_Web_Http!System.Web. > Http.Filters.AuthorizationFilterAttribute+<ExecuteAuthorizationFilterAsyn > cCore>d__2.MoveNext()+0x12d > > 21 0000004c`60e7d0d0 00007ff9`9f95c7f3 mscorlib!System.Runtime. > CompilerServices.AsyncTaskMethodBuilder`1[[System.Int32, > mscorlib]].Start[[System.IO.StreamReader+<ReadBufferAsync>d__97, > mscorlib]](<ReadBufferAsync>d__97 ByRef)+0x4dc535 > > 22 0000004c`60e7d180 00007ff9`9f95c6b4 System_Web_Http!System.Web. > Http.Filters.AuthorizationFilterAttribute.ExecuteAuthorizationFilterAsyn > cCore(System.Web.Http.Controllers.HttpActionContext, > System.Threading.CancellationToken, > System.Func`1<System.Threading.Tasks.Task`1<System. > Net.Http.HttpResponseMessage>>)+0xa3 > > 23 0000004c`60e7d240 00007ff9`9f95c572 System_Web_Http!System.Web. > Http.Filters.AuthorizationFilterAttribute.System.Web.Http.Filters. > IAuthorizationFilter.ExecuteAuthorizationFilterAsyn > c(System.Web.Http.Controllers.HttpActionContext, > System.Threading.CancellationToken, > System.Func`1<System.Threading.Tasks.Task`1<System. > Net.Http.HttpResponseMessage>>)+0x14 > > 24 0000004c`60e7d270 00007ff9`9f95c160 System_Web_Http!System.Web. > Http.Controllers.AuthorizationFilterResult.ExecuteAsync(System.Threading. > CancellationToken)+0xe2 > > 25 0000004c`60e7d2d0 00007ff9`9f4896b5 JockeyGetAPIs!JockeyGetAPI.Filters. > AddChallengeOnUnauthorizedResult+<ExecuteAsync>d__7.MoveNext()+0x90 > > 26 0000004c`60e7d3c0 00007ff9`9f95c031 mscorlib!System.Runtime. > CompilerServices.AsyncTaskMethodBuilder`1[[System.Int32, > mscorlib]].Start[[System.IO.StreamReader+<ReadBufferAsync>d__97, > mscorlib]](<ReadBufferAsync>d__97 ByRef)+0x9345 > > 27 0000004c`60e7d470 00007ff9`9f5e0db3 JockeyGetAPIs!JockeyGetAPI.Filters. > AddChallengeOnUnauthorizedResult.ExecuteAsync(System. > Threading.CancellationToken)+0xf1 > > 28 0000004c`60e7d500 00007ff9`9f5e0865 System_Web_Http!System.Web. > Http.Controllers.AuthenticationFilterResult+<ExecuteAsync>d__0.MoveNext()+ > 0x4e3 > > > > > > Thanks & Regards > > Jagadeeshwar Sidhurala > > > > [image: > http://responsivedev.revalweb.com/malires-new/revalsys-02Jan2017/02Jan2017-reval-logo.png] > <http://www.revalsys.com/> > > > > *P* > > : > > +91 9391607783 > > *L* > > : > > +91 040 2355 0141 > > *E* > > : > > jagdeeshw...@revalsys.com > > *A* > > : > > 8-2-293/82/1/83-A, 1st Floor, Banjara Hills, Road No.12, Hyderabad - > 500034 > <https://maps.google.com/?q=Banjara+Hills,+Road+No.12,+Hyderabad+-+500034&entry=gmail&source=g> > . > > *S* > > : > > [image: revalsys] <https://www.facebook.com/revalsys> [image: revalsys] > <https://twitter.com/revalsys> [image: revalsys] > <https://www.linkedin.com/company/revalsys-technologies> [image: > revalsys] <https://in.pinterest.com/revalsys/> [image: revalsys] > <https://www.instagram.com/revalsys_tech/> [image: revalsys] > <http://blog.revalsys.com/>[image: revalsys] > <https://www.google.co.in/maps/place/Revalsys+Technologies,+8-2-293%2F82%2F1%2F83-A,+Huda+Heights+Rd,+MLA+Colony,+Banjara+Hills,+Hyderabad,+Telangana+500034/@17.408496,78.4294639,16z/data=!4m2!3m1!1s0x3bcb915aef67e60d:0xbcb89e60b1e7ecb8> > > *E* > > : > > www.revalsys.com > > > > > > *IMPORTANT: *The contents of this email and any attachments are > confidential. They are intended for the named recipient(s) only. If you > have received this email by mistake, please notify the sender immediately > and do not disclose the contents to anyone or make copies thereof. > > > > Revalsys Technologies, creates possibilities by providing end-to-end IT > solutions for different types of technological needs and challenges. With > the most talented and dedicated pool of designers and developers, we work > on diverse solutions like Business Process Consulting, Web Designing, > Digital Marketing, Custom Application Development, Hosting Services and > Network Security > > >