Thanks for the confirmation. On Mon, Jul 24, 2023 at 4:40 PM Pavel Tupitsyn <ptupit...@apache.org> wrote:
> > If this flag is true will Ignite proactively allocate and use all pages > in a data region, rather than incrementally? > > LazyMemoryAllocation means whether memory for DataRegion will be allocated > only when the first cache is created in that region (when true), or > immediately (when false) > > > https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/configuration/DataRegionConfiguration.html#isLazyMemoryAllocation-- > > > On Wed, Jul 19, 2023 at 12:50 PM Raymond Wilson < > raymond_wil...@trimble.com> wrote: > >> Just FYI, we have held off any memory pressure changes in the meantime >> while we continue to investigate the memory issues we have. >> >> On Tue, 18 Jul 2023 at 9:07 AM, Raymond Wilson < >> raymond_wil...@trimble.com> wrote: >> >>> Hi Pavel, >>> >>> This area is confusing. There is no indication that the memory pressure >>> applies to any individual object or allocation, so there is clearly no >>> association between memory pressure and any particular resource. >>> >>> I get your argument that .Net can 'see' allocated memory. What is >>> unclear is whether it cares about actually allocated and used pages, or >>> committed pages. >>> >>> I see there is a LazyMemoryAllocation (default: true) for data regions. >>> Some data regions set this to false, eg: >>> >>> ^-- sysMemPlc region [type=internal, persistence=true, >>> lazyAlloc=false, >>> ^-- metastoreMemPlc region [type=internal, persistence=true, >>> lazyAlloc=false, >>> ^-- TxLog region [type=internal, persistence=true, lazyAlloc=false, >>> >>> The documentation is not clear on the effect of this flag other than to >>> say it is for 'Lazy memory allocation'. If this flag is true will Ignite >>> proactively allocate and use all pages in a data region, rather than >>> incrementally? >>> >>> Thanks, >>> Raymond. >>> >>> >>> On Tue, Jul 11, 2023 at 10:55 PM Pavel Tupitsyn <ptupit...@apache.org> >>> wrote: >>> >>>> > I can’t see another way of letting . Net know that it can’t have >>>> access to all the ‘free’ memory in the process >>>> >>>> You don't need to tell .NET how much memory is currently available. It >>>> is the job of the OS. .NET can "see" the size of the unmanaged heap. >>>> >>>> To quote another explanation [1]: >>>> >>>> > The point of AddMemoryPressure is to tell the garbage collector that >>>> there's a large amount of memory allocated with that object. >>>> > If it's unmanaged, the garbage collector doesn't know about it; only >>>> the managed portion. >>>> > Since the managed portion is relatively small, the GC may let it pass >>>> for garbage collection several times, essentially wasting memory that might >>>> need to be freed. >>>> >>>> I really don't think AddMemoryPressure is the right thing to do in your >>>> case. >>>> If you run into OOM issues, then look into Ignite memory region >>>> settings [2] and/or adjust application memory usage on the .NET side, so >>>> that the sum of those is not bigger than available RAM. >>>> >>>> [1] >>>> https://stackoverflow.com/questions/1149181/what-is-the-point-of-using-gc-addmemorypressure-with-an-unmanaged-resource >>>> [2] >>>> https://ignite.apache.org/docs/latest/memory-configuration/data-regions#configuring-default-data-region >>>> >>>> On Tue, Jul 11, 2023 at 11:48 AM Raymond Wilson < >>>> raymond_wil...@trimble.com> wrote: >>>> >>>>> How do Ignite .Net server nodes manage this memory issue in other >>>>> projects? >>>>> >>>>> On Tue, Jul 11, 2023 at 5:32 PM Raymond Wilson < >>>>> raymond_wil...@trimble.com> wrote: >>>>> >>>>>> Oops, commutes => committed >>>>>> >>>>>> On Tue, 11 Jul 2023 at 4:34 PM, Raymond Wilson < >>>>>> raymond_wil...@trimble.com> wrote: >>>>>> >>>>>>> I can’t see another way of letting . Net know that it can’t have >>>>>>> access to all the ‘free’ memory in the process when a large slab of >>>>>>> that is >>>>>>> spoken for in terms of memory commutes to Ignite data regions. >>>>>>> >>>>>>> In the current setup, as time goes on and Ignite progressively fills >>>>>>> the allocated cache ram then system behaviour changes and can result in >>>>>>> out >>>>>>> of memory issues. I think I would prefer consistent system behaviour >>>>>>> wrt to >>>>>>> allocated resources from the start. >>>>>>> >>>>>>> Raymond. >>>>>>> >>>>>>> On Tue, 11 Jul 2023 at 3:57 PM, Pavel Tupitsyn <ptupit...@apache.org> >>>>>>> wrote: >>>>>>> >>>>>>>> Are you sure this is necessary? >>>>>>>> >>>>>>>> GC.AddMemoryPressure documentation [1] states that this will >>>>>>>> "improve performance only for types that exclusively depend on >>>>>>>> finalizers". >>>>>>>> >>>>>>>> [1] >>>>>>>> https://learn.microsoft.com/en-us/dotnet/api/system.gc.addmemorypressure?view=net-7.0 >>>>>>>> >>>>>>>> On Tue, Jul 11, 2023 at 1:02 AM Raymond Wilson < >>>>>>>> raymond_wil...@trimble.com> wrote: >>>>>>>> >>>>>>>>> I'm making changes to add memory pressure to the GC to take into >>>>>>>>> account memory committed to the Ignite data regions as this will be >>>>>>>>> unmanaged memory allocations from the perspective of the GC. >>>>>>>>> >>>>>>>>> I don't call seeing anything related to this for .Net clients in >>>>>>>>> the documentation. Are you aware of any? >>>>>>>>> >>>>>>>>> Raymond. >>>>>>>>> >>>>>>>>> On Mon, Jul 10, 2023 at 9:41 PM Raymond Wilson < >>>>>>>>> raymond_wil...@trimble.com> wrote: >>>>>>>>> >>>>>>>>>> Thanks Pavel, this makes sense. >>>>>>>>>> >>>>>>>>>> Querying the .Net Process instance shows this as the difference >>>>>>>>>> between PagesMemorySize (includes committed) versus WorkingSet >>>>>>>>>> (includes >>>>>>>>>> uses/written to) size. >>>>>>>>>> Raymond. >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> <http://www.trimble.com/> >>>>>>>>> Raymond Wilson >>>>>>>>> Trimble Distinguished Engineer, Civil Construction Software (CCS) >>>>>>>>> 11 Birmingham Drive | >>>>>>>>> <https://www.google.com/maps/search/11+Birmingham+Drive%C2%A0%7C%C2%A0+Christchurch,+New+Zealand?entry=gmail&source=g>Christchurch, >>>>>>>>> New Zealand >>>>>>>>> <https://www.google.com/maps/search/11+Birmingham+Drive%C2%A0%7C%C2%A0+Christchurch,+New+Zealand?entry=gmail&source=g> >>>>>>>>> raymond_wil...@trimble.com >>>>>>>>> >>>>>>>>> >>>>>>>>> <https://worksos.trimble.com/?utm_source=Trimble&utm_medium=emailsign&utm_campaign=Launch> >>>>>>>>> >>>>>>>> -- >>>>>>> <http://www.trimble.com/> >>>>>>> Raymond Wilson >>>>>>> Trimble Distinguished Engineer, Civil Construction Software (CCS) >>>>>>> 11 Birmingham Drive | Christchurch, New Zealand >>>>>>> raymond_wil...@trimble.com >>>>>>> >>>>>>> >>>>>>> <https://worksos.trimble.com/?utm_source=Trimble&utm_medium=emailsign&utm_campaign=Launch> >>>>>>> >>>>>> -- >>>>>> <http://www.trimble.com/> >>>>>> Raymond Wilson >>>>>> Trimble Distinguished Engineer, Civil Construction Software (CCS) >>>>>> 11 Birmingham Drive | Christchurch, New Zealand >>>>>> raymond_wil...@trimble.com >>>>>> >>>>>> >>>>>> <https://worksos.trimble.com/?utm_source=Trimble&utm_medium=emailsign&utm_campaign=Launch> >>>>>> >>>>> >>>>> >>>>> -- >>>>> <http://www.trimble.com/> >>>>> Raymond Wilson >>>>> Trimble Distinguished Engineer, Civil Construction Software (CCS) >>>>> 11 Birmingham Drive | Christchurch, New Zealand >>>>> raymond_wil...@trimble.com >>>>> >>>>> >>>>> <https://worksos.trimble.com/?utm_source=Trimble&utm_medium=emailsign&utm_campaign=Launch> >>>>> >>>> >>> >>> -- >>> <http://www.trimble.com/> >>> Raymond Wilson >>> Trimble Distinguished Engineer, Civil Construction Software (CCS) >>> 11 Birmingham Drive | Christchurch, New Zealand >>> raymond_wil...@trimble.com >>> >>> >>> <https://worksos.trimble.com/?utm_source=Trimble&utm_medium=emailsign&utm_campaign=Launch> >>> >> -- >> <http://www.trimble.com/> >> Raymond Wilson >> Trimble Distinguished Engineer, Civil Construction Software (CCS) >> 11 Birmingham Drive | Christchurch, New Zealand >> raymond_wil...@trimble.com >> >> >> <https://worksos.trimble.com/?utm_source=Trimble&utm_medium=emailsign&utm_campaign=Launch> >> > -- <http://www.trimble.com/> Raymond Wilson Trimble Distinguished Engineer, Civil Construction Software (CCS) 11 Birmingham Drive | Christchurch, New Zealand raymond_wil...@trimble.com <https://worksos.trimble.com/?utm_source=Trimble&utm_medium=emailsign&utm_campaign=Launch>
