[
https://issues.apache.org/jira/browse/GEODE-7920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17433107#comment-17433107
]
ASF subversion and git services commented on GEODE-7920:
--------------------------------------------------------
Commit 6fe63cc3dc586824ea827300728688fafcd3d002 in geode's branch
refs/heads/support/1.12 from Bill Burcham
[ https://gitbox.apache.org/repos/asf?p=geode.git;h=6fe63cc ]
GEODE-7920: Geode UDP INT thread found processing cache operations (#7018)
Modified DistributionMessage to look for JGroups "internal" executor
threads. We thought we'd turned off all JGroups thread pools but this
one is still around. We don't want to process DistributionMessages in
these threads unless absolutely necessary since they're needed when
processing incoming messages.
(cherry picked from commit 8f24abbc7bade3f8e65a52a1a0402343e12ba2b5)
Co-authored-by: Bruce Schuchardt <[email protected]>
> Geode UDP INT thread found processing cache operations
> ------------------------------------------------------
>
> Key: GEODE-7920
> URL: https://issues.apache.org/jira/browse/GEODE-7920
> Project: Geode
> Issue Type: Bug
> Components: membership, messaging
> Affects Versions: 1.12.6
> Reporter: Bruce J Schuchardt
> Assignee: Bill Burcham
> Priority: Major
> Labels: pull-request-available
> Fix For: 1.12.6, 1.13.0
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> While looking into another problem in a test run with disable-tcp=true I
> found this JGroups thread processing a cache operation in-line. These
> threads should never process cache ops. They exist just to read messages and
> hand them off to an Executor.
> Fixing this should improve UDP messaging performance somewhat.
> {noformat}
> "Geode UDP INT-2,rs-Awesome-781-1145-1a0i32xlarge-hydra-client-4-27004" #54
> daemon prio=5 os_prio=0 tid=0x00007f202c05b800 nid=0x778d runnable
> [0x00007f20b0bb6000]
> java.lang.Thread.State: RUNNABLE
> at java.net.SocketInputStream.socketRead0(Native Method)
> at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
> at java.net.SocketInputStream.read(SocketInputStream.java:171)
> at java.net.SocketInputStream.read(SocketInputStream.java:141)
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:265)
> - locked <0x00000000c292f560> (a java.io.BufferedInputStream)
> at java.io.DataInputStream.readByte(DataInputStream.java:265)
> at
> sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:240)
> at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:163)
> at
> java.rmi.server.RemoteObjectInvocationHandler.invokeRemoteMethod(RemoteObjectInvocationHandler.java:235)
> at
> java.rmi.server.RemoteObjectInvocationHandler.invoke(RemoteObjectInvocationHandler.java:180)
> at com.sun.proxy.$Proxy53.increment(Unknown Source)
> at
> hydra.blackboard.AnySharedCountersImpl.increment(AnySharedCountersImpl.java:159)
> at
> util.AbstractListener.incrementAfterDestroyCounters(AbstractListener.java:450)
> at event.ETListener.afterDestroy(ETListener.java:88)
> at
> org.apache.geode.internal.cache.EnumListenerEvent$AFTER_DESTROY.dispatchEvent(EnumListenerEvent.java:178)
> at
> org.apache.geode.internal.cache.LocalRegion.dispatchEvent(LocalRegion.java:8242)
> at
> org.apache.geode.internal.cache.LocalRegion.dispatchListenerEvent(LocalRegion.java:6952)
> at
> org.apache.geode.internal.cache.LocalRegion.invokeDestroyCallbacks(LocalRegion.java:6760)
> at
> org.apache.geode.internal.cache.EntryEventImpl.invokeCallbacks(EntryEventImpl.java:2443)
> at
> org.apache.geode.internal.cache.entries.AbstractRegionEntry.dispatchListenerEvents(AbstractRegionEntry.java:164)
> at
> org.apache.geode.internal.cache.LocalRegion.basicDestroyPart2(LocalRegion.java:6701)
> at
> org.apache.geode.internal.cache.map.RegionMapDestroy.removeEntryOrLeaveTombstone(RegionMapDestroy.java:509)
> at
> org.apache.geode.internal.cache.map.RegionMapDestroy.retryRemoveWithTombstone(RegionMapDestroy.java:373)
> - locked <0x00000000fe71d6e0> (a
> org.apache.geode.internal.cache.entries.VersionedStatsRegionEntryOffHeapStringKey2)
> at
> org.apache.geode.internal.cache.map.RegionMapDestroy.checkTombstoneAndConcurrency(RegionMapDestroy.java:202)
> at
> org.apache.geode.internal.cache.map.RegionMapDestroy.destroy(RegionMapDestroy.java:147)
> at
> org.apache.geode.internal.cache.AbstractRegionMap.destroy(AbstractRegionMap.java:980)
> at
> org.apache.geode.internal.cache.LocalRegion.mapDestroy(LocalRegion.java:6490)
> at
> org.apache.geode.internal.cache.LocalRegion.mapDestroy(LocalRegion.java:6464)
> at
> org.apache.geode.internal.cache.LocalRegionDataView.destroyExistingEntry(LocalRegionDataView.java:58)
> at
> org.apache.geode.internal.cache.LocalRegion.basicDestroy(LocalRegion.java:6415)
> at
> org.apache.geode.internal.cache.DistributedRegion.basicDestroy(DistributedRegion.java:1720)
> at
> org.apache.geode.internal.cache.DestroyOperation$DestroyMessage.operateOnRegion(DestroyOperation.java:88)
> at
> org.apache.geode.internal.cache.DistributedCacheOperation$CacheOperationMessage.basicProcess(DistributedCacheOperation.java:1208)
> at
> org.apache.geode.internal.cache.DistributedCacheOperation$CacheOperationMessage.process(DistributedCacheOperation.java:1110)
> at
> org.apache.geode.distributed.internal.DistributionMessage.scheduleAction(DistributionMessage.java:376)
> at
> org.apache.geode.distributed.internal.DistributionMessage.schedule(DistributionMessage.java:431)
> at
> org.apache.geode.distributed.internal.ClusterDistributionManager.scheduleIncomingMessage(ClusterDistributionManager.java:2072)
> at
> org.apache.geode.distributed.internal.ClusterDistributionManager.handleIncomingDMsg(ClusterDistributionManager.java:1847)
> at
> org.apache.geode.distributed.internal.ClusterDistributionManager$$Lambda$74/1548842157.messageReceived(Unknown
> Source)
> at
> org.apache.geode.distributed.internal.membership.gms.GMSMembership.dispatchMessage(GMSMembership.java:931)
> {noformat}
> This can be fixed by re-enabling the commented out code in
> DistributionMessage.isPreciousThread(). There are two thread-name checks
> already in that method that need to be retained as well, for the
> unicast-reader thread and the multicast-reader thread.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
