Hello! 10M is very very low-ball for testing performance of disk, considering how Ignite's wal/checkpoints are structured. As already told, it does not even work properly.
I recommend using 2G value instead. Just load enough data so that you can observe constant checkpoints. Regards, -- Ilya Kasnacheev ср, 4 дек. 2019 г. в 03:16, Mitchell Rathbun (BLOOMBERG/ 731 LEX) < mrathb...@bloomberg.net>: > For the requested full ignite log, where would this be found if we are > running using local mode? We are not explicitly running a separate ignite > node, and our WorkDirectory does not seem to have any logs > > From: user@ignite.apache.org At: 12/03/19 19:00:18 > To: user@ignite.apache.org > Subject: Re: IgniteOutOfMemoryException in LOCAL cache mode with > persistence enabled > > For our configuration properties, our DataRegion initialSize and MaxSize > was set to 11 MB and persistence was enabled. For DataStorage, our pageSize > was set to 8192 instead of 4096. For Cache, write behind is disabled, on > heap cache is disabled, and Atomicity Mode is Atomic > > From: user@ignite.apache.org At: 12/03/19 13:40:32 > To: user@ignite.apache.org > Subject: Re: IgniteOutOfMemoryException in LOCAL cache mode with > persistence enabled > > Hi Mitchell, > > Looks like it could be easily reproduced on low off-heap sizes, I tried > with > simple puts and got the same exception: > > class org.apache.ignite.internal.mem.IgniteOutOfMemoryException: Failed to > find a page for eviction [segmentCapacity=1580, loaded=619, > maxDirtyPages=465, dirtyPages=619, cpPages=0, pinnedInSegment=0, > failedToPrepare=620] > Out of memory in data region [name=Default_Region, initSize=10.0 MiB, > maxSize=10.0 MiB, persistenceEnabled=true] Try the following: > ^-- Increase maximum off-heap memory size > (DataRegionConfiguration.maxSize) > ^-- Enable Ignite persistence (DataRegionConfiguration.persistenceEnabled) > ^-- Enable eviction or expiration policies > > It looks like Ignite must issue a proper warning in this case and couple of > issues must be filed against Ignite JIRA. > > Check out this article on persistent store available in Ignite confluence > as > well: > > https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Persistent+Store+-+und > er+the+hood#IgnitePersistentStore-underthehood-Checkpointing > > I've managed to make kind of similar example working with 20 Mb region with > a bit of tuning, added following properties to > org.apache.ignite.configuration.DataStorageConfiguration: > /<property name="checkpointFrequency" value="1500"/> > <property name="writeThrottlingEnabled" value="true"/>/ > > The whole idea behind this is to trigger checkpoint on timeout rather than > on too much dirty pages percentage threshold. The checkpoint page buffer > size may not exceed data region size, which is 10 Mb, which might be > overflown during checkpoint as well. > > I assume that checkpoint is never triggered in this case because of > per-partition overhead: Ignite writes some meta per partition and it looks > like that it is at least 1 meta page utilized for each which results in > some > amount of off-heap devoured by these meta pages. In the case with the > lowest > possible region size, this might consume more than 3 Mb for cache with 1k > partitions and 70% dirty data pages threshold would never be reached. > > However, I found another issue when it is not possible to save meta page on > checkpoint begin, this reproduces on 10 Mb data region with mentioned > storage configuration options. > > Could you please describe the configuration if you have anything different > from defaults (page size, wal mode, partitions count) and types of > key/value > that you use? And if it is possible, could you please attach full Ignite > log > from the node that has suffered from IOOM? > > As for the data region/cache, in reality you do also have cache groups here > playing a role. But generally I would recommend you to go with one data > region for all caches unless you have a particular reason to have multiple > regions. As for example, you have some really important data in some cache > that always needs to be available in durable off-heap memory, then you > should have a separate data region for this cache as I'm not aware if there > is possibility to disallow evicting pages for a specific cache. > > Cache groups documentation link: > https://apacheignite.readme.io/docs/cache-groups > > By default (a cache doesn't have cacheGroup property defined) each cache > has > it's own cache group with the very same name, that lives in the data region > specified or default data region. You might use them or not, depending on > the goal you have: use when you want to reduce meta > overhead/checkpoints/partition exchanges and share internal structures to > save up space a bit, or do not use them if you want to speed up > inserts/lookups by having it's own dedicated partition maps and B+ trees. > > Best regards, > Anton > > > > -- > Sent from: http://apache-ignite-users.70518.x6.nabble.com/ > > > >
