Denis, > BTW, who is considered to be the release manager of this release? I'll do it.
On Mon, Aug 28, 2017 at 3:54 PM, Seliverstov Igor <gvvinbl...@gmail.com> wrote: > Ok, the check happens at the node start time or on NODE_JOIN event > > in general it looks like: > > 1) calculate expected used memory = heap max + system cache max + all > custom policies max + default policy size and put it into a node attribute > > 2) get total physycal memory, calculate expected safe to be used memory > amount (leave 4 gb min or 20% of available memory for OS) > > 3) if expected used memory + expected used memory of other nodes on the > host > than safe to be used memory amount, start calculating suggestions > > 4) Each ignite instance needs at least 512mb heap + 40mb system cache + > 100mb default polycy, if available memory is less we cannot suggest > anything reasonable, print warning, stop calculation. > > 5) check heap size (shouldn't exceed 30% of available memory (total_memory > - reserved for OS memory) * 30% for all JVMs, if it exeedes, suggest just > calculated value or 512MB minimal) > > 6) check if system cache size changed, suggest default value if it's so > > 7) in case 100 mb * policies count < available memory, suggest using > default policy with max size equals to remaining memory (available - heap - > system cache) > > 8) calculate new size for each memory policy ( it's user defined size * > (remaining / (all_policies_size * nodes_cnt)); in proportion to > remaining memory, devided by nodes number on the host or 100 mb minimal) > > 9) print suggestions > > > > 2017-08-28 15:10 GMT+03:00 Dmitriy Setrakyan <dsetrak...@apache.org>: > > > Igor, can you please describe the algorithm with all the thresholds? > > > > On Mon, Aug 28, 2017 at 4:56 AM, Seliverstov Igor <gvvinbl...@gmail.com> > > wrote: > > > > > The suggestion here is based on initial settings, and it's so because > > there > > > is no other nodes on the host in the example. > > > > > > The algorithm tries to preserve the original ratio of memory policies > > > keeping numbers reasonable (for example after some thresshold it will > > > suggest not to use several memory policies if there is not enough of > > memory > > > for all of them) and taking into consideration nodes count on the host, > > > each jvm heap, needed memory for OS, etc > > > > > > 2017-08-28 14:38 GMT+03:00 Dmitriy Setrakyan <dsetrak...@apache.org>: > > > > > > > Looks good, but why in the example provided are we suggesting 8GB? 2 > > > nodes > > > > with 8GB will completely exhaust the available memory. I would > suggest > > 6 > > > or > > > > 7GB. > > > > > > > > Also, why 100MB for default policy. Anything under 1GB seems too > small. > > > > > > > > Can you please comment? > > > > > > > > D. > > > > > > > > On Mon, Aug 28, 2017 at 3:31 AM, Seliverstov Igor < > > gvvinbl...@gmail.com> > > > > wrote: > > > > > > > > > One more example of possible warning: > > > > > > > > > > ----------------------------------------------------- > > > > > Excessive memory usage by Ignite node process (performance may > drop) > > > > > [requested=44613MB, available=15942MB]. > > > > > > > > > > Please tune the folowing settings as suggested: > > > > > MemoryPolicyConfiguration.initialSize for bigPlc: 8102MB > > > > > MemoryPolicyConfiguration.maxSize for bigPlc: 8102MB > > > > > MemoryPolicyConfiguration.initialSize for dfltPlc: 100MB > > > > > MemoryPolicyConfiguration.maxSize for dfltPlc: 100MB > > > > > > > > > > Current settings: > > > > > Java Heap maxSize: 3543MB > > > > > Java Heap initSize: 250MB > > > > > MemoryPolicyConfiguration.initialSize for bigPlc: 256MB > > > > > MemoryPolicyConfiguration.maxSize for bigPlc: 40960MB > > > > > MemoryPolicyConfiguration.initialSize for dfltPlc: 10MB > > > > > MemoryPolicyConfiguration.maxSize for dfltPlc: 10MB > > > > > The overall expected memory usage by all Ignite nodes on the > host: > > > > > 44613MB > > > > > ----------------------------------------------------- > > > > > > > > > > Your thoughts? > > > > > > > > > > 2017-08-28 5:06 GMT+03:00 Denis Magda <dma...@apache.org>: > > > > > > > > > > > Guys, > > > > > > > > > > > > ML lib profile is missing in 2.1 release! That must be fixed and > > > rolled > > > > > > out in this emergency release: > > > > > > https://issues.apache.org/jira/browse/IGNITE-6193 < > > > > > > https://issues.apache.org/jira/browse/IGNITE-6193> > > > > > > > > > > > > Oleg, Yuri, please step in and handle the issue. > > > > > > > > > > > > BTW, who is considered to be the release manager of this release? > > > > > > > > > > > > — > > > > > > Denis > > > > > > > > > > > > > On Aug 25, 2017, at 2:29 PM, Dmitriy Setrakyan < > > > > dsetrak...@apache.org> > > > > > > wrote: > > > > > > > > > > > > > > I like the format proposed by Denis, very clear. > > > > > > > > > > > > > > However, I also do not understand why a user should change the > > size > > > > of > > > > > > some > > > > > > > system cache. How would a user ever know what value to put > there? > > > > This > > > > > > > value should be configured by Ignite automatically. > > > > > > > > > > > > > > D. > > > > > > > > > > > > > > On Fri, Aug 25, 2017 at 2:24 PM, Denis Magda < > dma...@apache.org> > > > > > wrote: > > > > > > > > > > > > > >> Igor, > > > > > > >> > > > > > > >> Let me suggest this format. > > > > > > >> > > > > > > >> --------------------------------------------- > > > > > > >> Excessive memory usage by Ignite node process (performance may > > > drop) > > > > > > >> [requested=29251MB, available=15942MB] > > > > > > >> > > > > > > >> Please tune the following settings: > > > > > > >> [MemoryConfiguration.defaultMemoryPolicySize = suggested > > value] > > > > > > >> MemoryConfiguration.systemCacheMaxSize = suggested value > > > > > > >> [MemoryPolicyConfiguration.maxSize for {policy_name_1} = > > > suggested > > > > > > >> value] > > > > > > >> [MemoryPolicyConfiguration.maxSize for {policy_name_2} = > > > suggested > > > > > > >> value] > > > > > > >> > > > > > > >> Current settings: > > > > > > >> [DefaultMemoryPolicySize = value] > > > > > > >> [{policy_name_1} size = value] > > > > > > >> [{policy_name_1} size = value] > > > > > > >> SystemCacheInitialSize = value > > > > > > >> SystemCacheMaxSize = value > > > > > > >> Java Heap Init Size = value > > > > > > >> Java Heap Max Size = value > > > > > > >> > > > > > > >> The overall memory usage by all Ignite nodes on the host: > value > > > > > > >> ------------------------------------------- > > > > > > >> > > > > > > >> Records in […] are optional. If custom memory policy is not > set > > or > > > > the > > > > > > >> default memory policy is overridden the output will miss some > of > > > the > > > > > > rows. > > > > > > >> > > > > > > >> As for systemCacheMaxSize, it should be show ONLY if the > > parameter > > > > was > > > > > > set > > > > > > >> explicitly by user code. Otherwise, the platform should be > wise > > > > enough > > > > > > to > > > > > > >> instantiate it properly depending on the host memory usage. > > > > > > >> > > > > > > >> — > > > > > > >> Denis > > > > > > >> > > > > > > >>> On Aug 25, 2017, at 1:49 PM, Seliverstov Igor < > > > > gvvinbl...@gmail.com> > > > > > > >> wrote: > > > > > > >>> > > > > > > >>> The message without logging layout: > > > > > > >>> > > > > > > >>> Not enough memory for current process [required=29251MB, > > > > > > >> available=15942MB]. > > > > > > >>> Please change MemoryConfiguration.systemCacheMaxSize and > > > > > > >>> MemoryConfiguration.defaultMemoryPolicySize to decrease > memory > > > > > > allocated > > > > > > >>> for each node. > > > > > > >>> > > > > > > >>> Current settings: > > > > > > >>> HeapInit=250MB > > > > > > >>> HeapMax=3543MB > > > > > > >>> DefaultMemoryPolicySize=12753MB > > > > > > >>> SystemCacheInitialSize=40MB > > > > > > >>> SystemCacheMaxSize=100MB > > > > > > >>> > > > > > > >>> Other ignite instanses on the server require: 12853MB > > > > > > >>> > > > > > > >>> I think it's make sense to describe what these numbers are > > > consist > > > > > of. > > > > > > >>> > > > > > > >>> We simple say which parameters have an impact on how much > > memory > > > > the > > > > > > >>> instance needs and their (parameters) actual values. > > > > > > >>> > > > > > > >>> Also we notice that more than one Ignite instance are ran on > > the > > > > > server > > > > > > >> or > > > > > > >>> workstation and it also consumes memory. > > > > > > >>> > > > > > > >>> 25 авг. 2017 г. 21:30 пользователь "Dmitriy Setrakyan" < > > > > > > >>> dsetrak...@apache.org> написал: > > > > > > >>> > > > > > > >>>> Igor, what is this flood of WARN messaging coming after the > > > text? > > > > > Are > > > > > > we > > > > > > >>>> really going to print this whole thing out? > > > > > > >>>> > > > > > > >>>> On Fri, Aug 25, 2017 at 9:49 AM, Seliverstov Igor < > > > > > > gvvinbl...@gmail.com > > > > > > >>> > > > > > > >>>> wrote: > > > > > > >>>> > > > > > > >>>>> This message appears on topology change in case the > available > > > > > memory > > > > > > is > > > > > > >>>>> exceeded > > > > > > >>>>> > > > > > > >>>>> 2017-08-25 19:47 GMT+03:00 Seliverstov Igor < > > > > gvvinbl...@gmail.com > > > > > >: > > > > > > >>>>> > > > > > > >>>>>> An example of current impl: > > > > > > >>>>>> > > > > > > >>>>>> > > > > > > >>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29% > > > > internal. > > > > > > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] > > > > > > >>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29% > > > > internal. > > > > > > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] Not enough > > > memory > > > > > for > > > > > > >>>>>> current process [required=29251MB, available=15942MB]. > > > > > > >>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29% > > > > internal. > > > > > > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] Please > change > > > > > > >>>>>> MemoryConfiguration.systemCacheMaxSize and > > > MemoryConfiguration. > > > > > > >>>>> defaultMemoryPolicySize > > > > > > >>>>>> to decrease memory allocated for each node. > > > > > > >>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29% > > > > internal. > > > > > > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] > > > > > > >>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29% > > > > internal. > > > > > > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] Current > > > settings: > > > > > > >>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29% > > > > internal. > > > > > > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] > > > HeapInit=250MB > > > > > > >>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29% > > > > internal. > > > > > > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] > > > HeapMax=3543MB > > > > > > >>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29% > > > > internal. > > > > > > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] > > > > > > >>>> DefaultMemoryPolicySize= > > > > > > >>>>>> 12753MB > > > > > > >>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29% > > > > internal. > > > > > > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] > > > > > > >>>>>> SystemCacheInitialSize=40MB > > > > > > >>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29% > > > > internal. > > > > > > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] > > > > > > >>>> SystemCacheMaxSize=100MB > > > > > > >>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29% > > > > internal. > > > > > > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] > > > > > > >>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29% > > > > internal. > > > > > > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] Other > ignite > > > > > > instanses > > > > > > >>>> on > > > > > > >>>>>> the server require: 12853MB > > > > > > >>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29% > > > > internal. > > > > > > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] > > > > > > >>>>>> > > > > > > >>>>>> > > > > > > >>>>>> 2017-08-25 17:40 GMT+03:00 Sergey Kozlov < > > > skoz...@gridgain.com > > > > >: > > > > > > >>>>>> > > > > > > >>>>>>> I suppose we should not forget JVM heap size and suggest > to > > > > > reduce > > > > > > >>>> both > > > > > > >>>>>>> options > > > > > > >>>>>>> > > > > > > >>>>>>> On Fri, Aug 25, 2017 at 5:24 PM, Dmitriy Setrakyan < > > > > > > >>>>> dsetrak...@apache.org > > > > > > >>>>>>>> > > > > > > >>>>>>> wrote: > > > > > > >>>>>>> > > > > > > >>>>>>>> Igor, I would change the message. How about this: > > > > > > >>>>>>>> > > > > > > >>>>>>>> Required RAM size is larger than total physical memory > > > > available > > > > > > for > > > > > > >>>>> OS. > > > > > > >>>>>>>>> Please change MemoryConfiguration.WhichProperty and > > > > > > >>>>>>>>> MemoryPolicyConfiguration.WhichProperty to decrease > > memory > > > > > > >>>>> allocated > > > > > > >>>>>>> for > > > > > > >>>>>>>>> each node. > > > > > > >>>>>>>> > > > > > > >>>>>>>> > > > > > > >>>>>>>> Also, can we calculate what the memory size allocated > for > > > each > > > > > > node > > > > > > >>>>>>> should > > > > > > >>>>>>>> be? In that case we should suggest it. > > > > > > >>>>>>>> > > > > > > >>>>>>>> D. > > > > > > >>>>>>>> > > > > > > >>>>>>>> > > > > > > >>>>>>>> On Fri, Aug 25, 2017 at 7:20 AM, Seliverstov Igor < > > > > > > >>>>> gvvinbl...@gmail.com > > > > > > >>>>>>>> > > > > > > >>>>>>>> wrote: > > > > > > >>>>>>>> > > > > > > >>>>>>>>> What do you, guys think about next warning? > > > > > > >>>>>>>>> > > > > > > >>>>>>>>> [2017-08-25 17:17:04,718][INFO > > > > > > >>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][ > > > GridHomePat > > > > > > >>>>>>> hSelfTest0] > > > > > > >>>>>>>>> System cache's MemoryPolicy size is configured to 40 > MB. > > > Use > > > > > > >>>>>>>>> MemoryConfiguration.systemCacheMemorySize property to > > > change > > > > > the > > > > > > >>>>>>>> setting. > > > > > > >>>>>>>>> [2017-08-25 17:17:04,718][WARN > > > > > > >>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][ > > > GridHomePat > > > > > > >>>>>>> hSelfTest0] > > > > > > >>>>>>>>> > > > > > > >>>>>>>>>>>> Required RAM size is larger than total physical > memory > > > > > > >>>> available > > > > > > >>>>>>> for > > > > > > >>>>>>>>> OS. > > > > > > >>>>>>>>>>>> Check your configuration to avoid swap partition > > usage. > > > > > > >>>>>>>>>>>> Use MemoryConfiguration and > MemoryPolicyConfiguration > > to > > > > > > >>>> change > > > > > > >>>>>>> the > > > > > > >>>>>>>>> settings. > > > > > > >>>>>>>>>>>> Physical memory [required=16397MB, > available=15942MB] > > > > > > >>>>>>>>> > > > > > > >>>>>>>>> [2017-08-25 17:17:04,726][WARN > > > > > > >>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][ > > > GridHomePat > > > > > > >>>>>>> hSelfTest0] > > > > > > >>>>>>>>> Peer class loading is enabled (disable it in production > > for > > > > > > >>>>>>> performance > > > > > > >>>>>>>> and > > > > > > >>>>>>>>> deployment consistency reasons) > > > > > > >>>>>>>>> [2017-08-25 17:17:04,726][INFO > > > > > > >>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][ > > > GridHomePat > > > > > > >>>>>>> hSelfTest0] > > > > > > >>>>>>>>> Configured caches [in 'sysMemPlc' memoryPolicy: > > > > > > >>>>> ['ignite-sys-cache']] > > > > > > >>>>>>>>> [2017-08-25 17:17:04,731][INFO > > > > > > >>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][ > > > GridHomePat > > > > > > >>>>>>> hSelfTest0] > > > > > > >>>>>>>>> 3-rd party licenses can be found at: > > > > > > >>>>>>>>> /home/gvvinblade/projects/ > ignite/incubator-ignite/libs/ > > > > > licenses > > > > > > >>>>>>>>> > > > > > > >>>>>>>>> > > > > > > >>>>>>>>> 2017-08-25 13:26 GMT+03:00 Yakov Zhdanov < > > > > yzhda...@apache.org > > > > > >: > > > > > > >>>>>>>>> > > > > > > >>>>>>>>>> Agree, let's release new version including tickets > > > mentioned > > > > > by > > > > > > >>>>>>> Denis > > > > > > >>>>>>>> and > > > > > > >>>>>>>>>> Ivan. > > > > > > >>>>>>>>>> > > > > > > >>>>>>>>>> --Yakov > > > > > > >>>>>>>>>> > > > > > > >>>>>>>>> > > > > > > >>>>>>>> > > > > > > >>>>>>> > > > > > > >>>>>>> > > > > > > >>>>>>> > > > > > > >>>>>>> -- > > > > > > >>>>>>> Sergey Kozlov > > > > > > >>>>>>> GridGain Systems > > > > > > >>>>>>> www.gridgain.com > > > > > > >>>>>>> > > > > > > >>>>>> > > > > > > >>>>>> > > > > > > >>>>> > > > > > > >>>> > > > > > > >> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > >
