[
https://issues.apache.org/jira/browse/IGNITE-16922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17540101#comment-17540101
]
Denis Chudov edited comment on IGNITE-16922 at 5/20/22 12:30 PM:
-----------------------------------------------------------------
This happens because Ignite stores TTL in row itself and needs to update row,
and in-place update doesn't happen because of the condition in
{{IgniteCacheOffheapManagerImpl.CacheDataStoreImpl#createRow}}:
{code:java}
if (canUpdateOldRow(cctx, oldRow, dataRow) &&
rowStore.updateRow(oldRow.link(), dataRow, grp.statisticsHolderData()))
{code}
which returns false and makes Ignite add new row first, and then remove old one.
Problem is that for large entries, fragmented by several pages, Ignite can't
update the row in-place, because in-place operations are made under write lock
on single page (see AbstractDataPageIO#updateRow )
was (Author: denis chudov):
This happens because Ignite stores TTL in row itself and needs to update row,
and in-place update doesn't happen because of the condition in
{{IgniteCacheOffheapManagerImpl.CacheDataStoreImpl#createRow}}:
{code:java}
if (canUpdateOldRow(cctx, oldRow, dataRow) &&
rowStore.updateRow(oldRow.link(), dataRow, grp.statisticsHolderData()))
{code}
which returns false and makes Ignite add new row first, and then remove old one.
Problem is that for large entries, fragmented by several pages, Ignite can't
update the row in-place, because in-place operations are made under write lock
on single page (see {{ AbstractDataPageIO#updateRow }} )
> Getting an entry with expiry policy causes IgniteOutOfMemoryException
> ---------------------------------------------------------------------
>
> Key: IGNITE-16922
> URL: https://issues.apache.org/jira/browse/IGNITE-16922
> Project: Ignite
> Issue Type: Bug
> Affects Versions: 2.13
> Reporter: Alexey Kukushkin
> Priority: Major
> Labels: cggg
> Original Estimate: 64h
> Remaining Estimate: 64h
>
> {{IgniteCache#get(key)}} operation causes {{IgniteOutOfMemoryException}} if
> {{AccessedExpiryPolicy}} or {{TouchedExpiryPolicy}} is enabled for the
> {{key}} and Ignite has not enough storage for another entry of the same or
> bigger size.
> This happens because:
> # Ignite needs to update TTL
> # TTL is part of the entry and Ignite overwrites full entry to update the TTL
> # The problem is Ignite runs common code that checks if Ignite has enough
> storage to write the entry with updated TTL back. The check fails causing the
> {{IgniteCache#get(key)}} operation to throw {{IgniteOutOfMemoryException}}.
> # This behavior is very confusing for Ignite users: why would a "read"
> operation throw Ignite OOM?
> Can we update the TTL atomically and skip the storage size check?
> Please enhance Ignite not to throw Ignite OOM on {{get}}.
> Stack trace:
> {code:java}
> [2022-05-20
> 15:08:20,025][ERROR][sys-stripe-6-#8%ignite.IgniteOOMOnGet0%][IgniteTestResources]
> Critical system error detected. Will be handled accordingly to configured
> handler [hnd=NoOpFailureHandler [super=AbstractFailureHandler
> [ignoredFailureTypes=UnmodifiableSet [SYSTEM_WORKER_BLOCKED,
> SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], failureCtx=FailureContext
> [type=CRITICAL_ERROR, err=class o.a.i.i.mem.IgniteOutOfMemoryException: Out
> of memory in data region [name=default, initSize=18.1 MiB, maxSize=18.1 MiB,
> persistenceEnabled=false] Try the following:
> ^-- Increase maximum off-heap memory size (DataRegionConfiguration.maxSize)
> ^-- Enable Ignite persistence (DataRegionConfiguration.persistenceEnabled)
> ^-- Enable eviction or expiration policies]]
> class org.apache.ignite.internal.mem.IgniteOutOfMemoryException: Out of
> memory in data region [name=default, initSize=18.1 MiB, maxSize=18.1 MiB,
> persistenceEnabled=false] Try the following:
> ^-- Increase maximum off-heap memory size (DataRegionConfiguration.maxSize)
> ^-- Enable Ignite persistence (DataRegionConfiguration.persistenceEnabled)
> ^-- Enable eviction or expiration policies
> at
> org.apache.ignite.internal.processors.cache.persistence.IgniteCacheDatabaseSharedManager.ensureFreeSpaceForInsert(IgniteCacheDatabaseSharedManager.java:1234)
> at
> org.apache.ignite.internal.processors.cache.persistence.RowStore.addRow(RowStore.java:108)
> at
> org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.createRow(IgniteCacheOffheapManagerImpl.java:1962)
> at
> org.apache.ignite.internal.processors.cache.GridCacheMapEntry$UpdateClosure.call(GridCacheMapEntry.java:5767)
> at
> org.apache.ignite.internal.processors.cache.GridCacheMapEntry$UpdateClosure.call(GridCacheMapEntry.java:5695)
> at
> org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Invoke.invokeClosure(BPlusTree.java:4131)
> at
> org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.invokeDown(BPlusTree.java:2121)
> at
> org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.invoke(BPlusTree.java:1997)
> at
> org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.invoke0(IgniteCacheOffheapManagerImpl.java:1860)
> at
> org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.invoke(IgniteCacheOffheapManagerImpl.java:1843)
> at
> org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.invoke(IgniteCacheOffheapManagerImpl.java:471)
> at
> org.apache.ignite.internal.processors.cache.GridCacheMapEntry.storeValue(GridCacheMapEntry.java:4164)
> at
> org.apache.ignite.internal.processors.cache.GridCacheMapEntry.storeValue(GridCacheMapEntry.java:4140)
> at
> org.apache.ignite.internal.processors.cache.GridCacheMapEntry.updateTtl(GridCacheMapEntry.java:2961)
> at
> org.apache.ignite.internal.processors.cache.GridCacheMapEntry.updateTtl(GridCacheMapEntry.java:2934)
> at
> org.apache.ignite.internal.processors.cache.GridCacheMapEntry.innerGet0(GridCacheMapEntry.java:825)
> at
> org.apache.ignite.internal.processors.cache.GridCacheMapEntry.innerGetVersioned(GridCacheMapEntry.java:704)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtCacheAdapter.getAllAsync0(GridDhtCacheAdapter.java:851)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtCacheAdapter.getDhtAllAsync(GridDhtCacheAdapter.java:691)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtGetSingleFuture.getAsync(GridDhtGetSingleFuture.java:413)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtGetSingleFuture.map0(GridDhtGetSingleFuture.java:289)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtGetSingleFuture.map(GridDhtGetSingleFuture.java:270)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtGetSingleFuture.init(GridDhtGetSingleFuture.java:186)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtCacheAdapter.getDhtSingleAsync(GridDhtCacheAdapter.java:1156)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtCacheAdapter.processNearSingleGetRequest(GridDhtCacheAdapter.java:1174)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.access$200(GridDhtAtomicCache.java:151)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$4.apply(GridDhtAtomicCache.java:278)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$4.apply(GridDhtAtomicCache.java:273)
> at
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1150)
> at
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:591)
> at
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:392)
> at
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:318)
> at
> org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:308)
> at
> org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1727)
> at
> org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1334)
> at
> org.apache.ignite.internal.managers.communication.GridIoManager$8.execute(GridIoManager.java:1218)
> at
> org.apache.ignite.internal.managers.communication.TraceRunnable.run(TraceRunnable.java:54)
> at
> org.apache.ignite.internal.util.StripedExecutor$Stripe.body(StripedExecutor.java:567)
> at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:119)
> at java.base/java.lang.Thread.run(Thread.java:833)
> {code}
> Reproducer:
> {code:java}
> package org.apache.ignite;
> import java.util.concurrent.ThreadLocalRandom;
> import java.util.concurrent.TimeUnit;
> import java.util.concurrent.atomic.AtomicBoolean;
> import java.util.concurrent.atomic.AtomicInteger;
> import javax.cache.expiry.Duration;
> import javax.cache.expiry.TouchedExpiryPolicy;
> import org.apache.ignite.configuration.CacheConfiguration;
> import org.apache.ignite.configuration.DataRegionConfiguration;
> import org.apache.ignite.configuration.DataStorageConfiguration;
> import org.apache.ignite.configuration.IgniteConfiguration;
> import org.apache.ignite.internal.IgniteInternalFuture;
> import org.apache.ignite.testframework.junits.common.GridCommonAbstractTest;
> import org.junit.Test;
> import static java.lang.Math.max;
> import static
> org.apache.ignite.testframework.GridTestUtils.runMultiThreadedAsync;
> public class IgniteOOMOnGet extends GridCommonAbstractTest {
> @Override protected IgniteConfiguration getConfiguration(String
> igniteInstanceName) throws Exception {
> return super.getConfiguration(igniteInstanceName)
> .setDataStorageConfiguration(
> new
> DataStorageConfiguration().setDefaultDataRegionConfiguration(
> new DataRegionConfiguration().setMaxSize(19_000_000)
> )
> );
> }
> @Test
> public void test() throws Exception{
> Ignite ignite = startGrid(0);
> Ignite client = startClientGrid(1);
> IgniteCache<Integer, byte[]> cache = client.createCache(
> new CacheConfiguration<Integer, byte[]>(DEFAULT_CACHE_NAME)
> .setExpiryPolicyFactory(
> TouchedExpiryPolicy.factoryOf(new
> Duration(TimeUnit.SECONDS, 100_000))
> )
> );
> AtomicInteger maxKey = new AtomicInteger();
> for (int i = 0; i < 85; i++) {
> cache.put(i, new byte[10_000]);
> while (true) {
> int mk = maxKey.get();
> if (maxKey.compareAndSet(mk, max(mk, i)))
> break;
> }
> }
> AtomicBoolean enabled = new AtomicBoolean(true);
> IgniteInternalFuture fut = runMultiThreadedAsync(() -> {
> ThreadLocalRandom rnd = ThreadLocalRandom.current();
> while (enabled.get()) {
> cache.get(rnd.nextInt(maxKey.get()));
> }
> }, 8, "getterThread");
> doSleep(10_000);
> enabled.set(false);
> fut.get();
> }
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
