[
https://issues.apache.org/jira/browse/IGNITE-16118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pavel Pereslegin updated IGNITE-16118:
--------------------------------------
Description:
In a rare case, you might observe a confusing {{GridCacheEntryRemovedException
"Failed to send TTL update request"}} in logs while *reading a non-expired*
cache value.
{noformat}
[ERROR][sys-#258%expiry.EntryRemovedOnReadTest2%][root] <default> Failed to
send TTL update request.
org.apache.ignite.internal.processors.cache.GridCacheEntryRemovedException
at
org.apache.ignite.internal.processors.cache.GridCacheMapEntry.checkObsolete(GridCacheMapEntry.java:3052)
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtCacheEntry.checkReadersLocked(GridDhtCacheEntry.java:732)
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtCacheEntry.checkReaders(GridDhtCacheEntry.java:708)
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtCacheEntry.readers(GridDhtCacheEntry.java:416)
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtCacheAdapter$8.run(GridDhtCacheAdapter.java:1122)
at
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:7329)
at
org.apache.ignite.internal.processors.closure.GridClosureProcessor$1.body(GridClosureProcessor.java:827)
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:125)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
{noformat}
It looks like there is a race between {{sendTtlUpdateRequest(UUID,
GridCacheTtlUpdateRequest)}} and heap entry eviction.
This situation is currently not handled correctly, resulting in a hidden
NullPointerException that aborts TTL updates.
Due to the nature of the TTL updates in Ignite, this problem can only occur
when the TTL is updated from the backup node.
!ttl-updates.png!
Thus, in a very rare case, we may notice a lack of updates in the near-cache
and on "non-initiator" backups
was:
In a rare case, you might observe a confusing {{GridCacheEntryRemovedException
"Failed to send TTL update request"}} in logs while *reading a non-expired*
cache value.
{noformat}
[ERROR][sys-#258%expiry.EntryRemovedOnReadTest2%][root] <default> Failed to
send TTL update request.
org.apache.ignite.internal.processors.cache.GridCacheEntryRemovedException
at
org.apache.ignite.internal.processors.cache.GridCacheMapEntry.checkObsolete(GridCacheMapEntry.java:3052)
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtCacheEntry.checkReadersLocked(GridDhtCacheEntry.java:732)
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtCacheEntry.checkReaders(GridDhtCacheEntry.java:708)
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtCacheEntry.readers(GridDhtCacheEntry.java:416)
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtCacheAdapter$8.run(GridDhtCacheAdapter.java:1122)
at
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:7329)
at
org.apache.ignite.internal.processors.closure.GridClosureProcessor$1.body(GridClosureProcessor.java:827)
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:125)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
{noformat}
It looks like there is a race between {{sendTtlUpdateRequest(UUID,
GridCacheTtlUpdateRequest)}} and heap entry eviction.
This situation is currently not handled correctly, resulting in a hidden
NullPointerException that aborts TTL updates.
Due to the nature of the TTL updates in Ignite, this problem can only occur
when the TTL is updated from the backup node.
Thus, in a very rare case, we may notice a lack of updates in the near-cache
and on "non-initiator" backups
> Near cache entry can miss a TTL update if an NPE occurs on the primary node.
> ----------------------------------------------------------------------------
>
> Key: IGNITE-16118
> URL: https://issues.apache.org/jira/browse/IGNITE-16118
> Project: Ignite
> Issue Type: Bug
> Reporter: Pavel Pereslegin
> Assignee: Pavel Pereslegin
> Priority: Minor
> Labels: ise
> Fix For: 2.13
>
> Attachments: ttl-updates.png
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> In a rare case, you might observe a confusing
> {{GridCacheEntryRemovedException "Failed to send TTL update request"}} in
> logs while *reading a non-expired* cache value.
> {noformat}
> [ERROR][sys-#258%expiry.EntryRemovedOnReadTest2%][root] <default> Failed to
> send TTL update request.
> org.apache.ignite.internal.processors.cache.GridCacheEntryRemovedException
> at
> org.apache.ignite.internal.processors.cache.GridCacheMapEntry.checkObsolete(GridCacheMapEntry.java:3052)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtCacheEntry.checkReadersLocked(GridDhtCacheEntry.java:732)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtCacheEntry.checkReaders(GridDhtCacheEntry.java:708)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtCacheEntry.readers(GridDhtCacheEntry.java:416)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtCacheAdapter$8.run(GridDhtCacheAdapter.java:1122)
> at
> org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:7329)
> at
> org.apache.ignite.internal.processors.closure.GridClosureProcessor$1.body(GridClosureProcessor.java:827)
> at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:125)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
> It looks like there is a race between {{sendTtlUpdateRequest(UUID,
> GridCacheTtlUpdateRequest)}} and heap entry eviction.
> This situation is currently not handled correctly, resulting in a hidden
> NullPointerException that aborts TTL updates.
> Due to the nature of the TTL updates in Ignite, this problem can only occur
> when the TTL is updated from the backup node.
> !ttl-updates.png!
> Thus, in a very rare case, we may notice a lack of updates in the near-cache
> and on "non-initiator" backups
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
