[ https://issues.apache.org/jira/browse/GEODE-8278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17250673#comment-17250673 ]
Barrett Oglesby commented on GEODE-8278: ---------------------------------------- I see some code that doesn't look quite right. First, this code in AbstractRegionMap.initialImagePut: {noformat} if (oldIsTombstone) { owner.unscheduleTombstone(oldRe); if (newValue != Token.TOMBSTONE) { lruEntryCreate(oldRe); } else { lruEntryUpdate(oldRe); } } {noformat} Its only updating the LRU statistics if the previous entry was a tombstone. That doesn't seem correct. I changed it to: {noformat} if (oldIsTombstone) { owner.unscheduleTombstone(oldRe); } if (newValue != Token.TOMBSTONE) { lruEntryCreate(oldRe); } else { lruEntryUpdate(oldRe); } {noformat} Then, VMLRURegionMap.resetThreadLocals is called twice. Once by AbstractRegionMap.initialImagePut in the finally block: {noformat} java.lang.Exception at org.apache.geode.internal.cache.VMLRURegionMap.resetThreadLocals(VMLRURegionMap.java:609) at org.apache.geode.internal.cache.AbstractRegionMap.initialImagePut(AbstractRegionMap.java:949) at org.apache.geode.internal.cache.InitialImageOperation.processChunk(InitialImageOperation.java:941) {noformat} And once in VMLRURegionMap.lruUpdateCallback (after the entry has been processed by InitialImageOperation.processChunk): {noformat} java.lang.Exception at org.apache.geode.internal.cache.VMLRURegionMap.resetThreadLocals(VMLRURegionMap.java:609) at org.apache.geode.internal.cache.VMLRURegionMap.lruUpdateCallback(VMLRURegionMap.java:374) at org.apache.geode.internal.cache.InitialImageOperation.processChunk(InitialImageOperation.java:954) {noformat} VMLRURegionMap.resetThreadLocals method clears a few thread locals including lruDelta which is used by the lruUpdateCallback to determine whether to evict or not. I think the first call in the finally block of AbstractRegionMap.initialImagePut is not correct. Here is that code: {noformat} } finally { if (done && !deferLRUCallback) { lruUpdateCallback(); } else if (!cleared) { resetThreadLocals(); } } {noformat} I changed it to: {noformat} if (!deferLRUCallback) { if (done) { lruUpdateCallback(); } else if (!cleared) { resetThreadLocals(); } } {noformat} I'm not sure these are valid changes, but with these changes, during GII, I see eviction occurring. With these changes, histograms after recovery show: {noformat} num #instances #bytes class name ---------------------------------------------- 1: 8740 113626040 [B 2: 47952 4749896 [C 15: 5000 320000 org.apache.geode.internal.cache.entries.VersionedThinDiskRegionEntryHeapStringKey1 19: 5000 280000 org.apache.geode.internal.cache.entries.VMThinDiskLRURegionEntryHeapLongKey 32: 5003 120072 org.apache.geode.internal.cache.VMCachedDeserializable 46: 505 52520 org.apache.geode.internal.cache.wan.GatewaySenderEventImpl Total 443319 133451360 num #instances #bytes class name ---------------------------------------------- 1: 12201 184764920 [B 2: 47927 4770984 [C 15: 5000 320000 org.apache.geode.internal.cache.entries.VersionedThinDiskRegionEntryHeapStringKey1 19: 5000 280000 org.apache.geode.internal.cache.entries.VMThinDiskLRURegionEntryHeapLongKey 23: 8796 211104 org.apache.geode.internal.cache.VMCachedDeserializable 47: 514 53456 org.apache.geode.internal.cache.wan.GatewaySenderEventImpl Total 449229 204864160 {noformat} The top histogram shows the GII provider; the bottom histogram shows the GII requester. These show that the GII provider has only recovered keys since there are 5003 VMCachedDeserializables. These also show that the GII requester has evicted entries since there are only 8796 VMCachedDeserializables. Here is some logging that shows an entry being processed that doesn't cause eviction. It does update the total bytes (total=39151873): {noformat} Pooled High Priority Message Processor 27: InitialImageOperation.processChunk about to initialImagePut key=3000; value=VMCachedDeserializable@2010910258 Pooled High Priority Message Processor 27: VMLRURegionMap.lruEntryUpdate region=/__PR/_B__ny__PARALLEL__GATEWAY__SENDER__QUEUE_62; re=VMThinDiskLRURegionEntryHeapLongKey@45652743 (key=3000) Pooled High Priority Message Processor 27: VMLRURegionMap.setDelta region=/__PR/_B__ny__PARALLEL__GATEWAY__SENDER__QUEUE_62; lruDelta=20666 Pooled High Priority Message Processor 27: InitialImageOperation.processChunk done initialImagePut key=3000; value=VMCachedDeserializable@2010910258 Pooled High Priority Message Processor 27: InitialImageOperation.processChunk about to lruUpdateCallback key=3000 Pooled High Priority Message Processor 27: VMLRURegionMap.getDelta value=20666 Pooled High Priority Message Processor 27: VMLRURegionMap.lruUpdateCallback region=/__PR/_B__ny__PARALLEL__GATEWAY__SENDER__QUEUE_62; bytesToEvict=20666 Pooled High Priority Message Processor 27: VMLRURegionMap.changeTotalEntrySize region=/__PR/_B__ny__PARALLEL__GATEWAY__SENDER__QUEUE_62; delta=20666 Pooled High Priority Message Processor 27: MemoryLRUStatistics.updateCounter delta=20666; total=39151873 Pooled High Priority Message Processor 27: InitialImageOperation.processChunk done lruUpdateCallback key=3000 Pooled High Priority Message Processor 27: InitialImageOperation.processChunk done processing key=3000 {noformat} Here is some logging during GII where the entry does cause eviction (VMLRURegionMap.lruUpdateCallback evicted...): {noformat} Pooled High Priority Message Processor 27: InitialImageOperation.processChunk about to initialImagePut key=2900; value=VMCachedDeserializable@1084176828 Pooled High Priority Message Processor 27: VMLRURegionMap.lruEntryUpdate region=/__PR/_B__ny__PARALLEL__GATEWAY__SENDER__QUEUE_75; re=VMThinDiskLRURegionEntryHeapLongKey@527429f5 (key=2900) Pooled High Priority Message Processor 27: VMLRURegionMap.setDelta region=/__PR/_B__ny__PARALLEL__GATEWAY__SENDER__QUEUE_75; lruDelta=20666 Pooled High Priority Message Processor 27: InitialImageOperation.processChunk done initialImagePut key=2900; value=VMCachedDeserializable@1084176828 Pooled High Priority Message Processor 27: InitialImageOperation.processChunk about to lruUpdateCallback key=2900 Pooled High Priority Message Processor 27: VMLRURegionMap.getDelta value=20666 Pooled High Priority Message Processor 27: VMLRURegionMap.lruUpdateCallback region=/__PR/_B__ny__PARALLEL__GATEWAY__SENDER__QUEUE_75; bytesToEvict=20666 Pooled High Priority Message Processor 27: MemoryLRUStatistics.updateCounter delta=-20674; total=78634526 Pooled High Priority Message Processor 27: VMLRURegionMap.lruUpdateCallback evicted region=/__PR/_B__ny__PARALLEL__GATEWAY__SENDER__QUEUE_75; key=2900 Pooled High Priority Message Processor 27: VMLRURegionMap.changeTotalEntrySize region=/__PR/_B__ny__PARALLEL__GATEWAY__SENDER__QUEUE_75; delta=20666 Pooled High Priority Message Processor 27: MemoryLRUStatistics.updateCounter delta=20666; total=78655184 Pooled High Priority Message Processor 27: InitialImageOperation.processChunk done lruUpdateCallback key=2900 Pooled High Priority Message Processor 27: InitialImageOperation.processChunk done processing key=2900 {noformat} > Gateway sender queues using heap memory way above configured value after > server restart > --------------------------------------------------------------------------------------- > > Key: GEODE-8278 > URL: https://issues.apache.org/jira/browse/GEODE-8278 > Project: Geode > Issue Type: Bug > Components: eviction > Reporter: Alberto Gomez > Assignee: Alberto Gomez > Priority: Major > > In a Geode system with the following characteristics: > * WAN replication > * partition redundant regions > * overflow configured for the gateway senders queues by means of persistence > and maximum queue memory set. > * gateway receivers stopped in one site (B) > * Operations sent to the site that does not have the gateway receivers > stopped (A) > When operations are sent to site A, the gateway sender queues start to grow > as expected and the heap memory consumed by the queues does not grow > indefinitely given that there is overflow to disk when the limit is reached. > But, if a server is restarted, the restarted server will show a much higher > heap memory used than the memory used by this server before it was restarted > or by the other servers. > This can even provoke that the server cannot be restarted if the heap memory > it requires is above the limit configured. > According to the memory analyzer the entries taking up the memory are > subclasses of ```VMThinDiskLRURegionEntryHeap```. > The number of instances of this type are the same in the restarted server > than in the not restarted servers but on the restarted server they take much > more memory. The reason seems to be that the ```value``` member attribute of > the instances, in the case of the restarted server contains > ```VMCachedDeserializable``` objects while in the case of the not restarted > server the attribute contains either ```null``` or > ```GatewaySenderEventImpl``` objects that use much less memory than the > ```VMCachedDeserializable``` ones. > If redundancy is not configured for the region then the problem is not > manifested, i.e. the heap memory used by the restarted server is similar to > the one prior to the restart. > If the node not restarted is restarted then the previously restarted node > seems to release the extra memory (my guess is that it is processing the > other process queue). > Also, if traffic is sent again to the Geode cluster, then it seems eviction > kicks in and after some short time, the memory of the restarted server goes > down to the level it had before it had been restarted. > As a summary, the problem seems to be that if a server does GII > (getInitialImage) from another server, eviction does not occur for gateway > sender queue entries. -- This message was sent by Atlassian Jira (v8.3.4#803005)