[
https://issues.apache.org/jira/browse/IGNITE-3465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Semen Boikov closed IGNITE-3465.
--------------------------------
Assignee: (was: Semen Boikov)
> Java crash: problematic frame
> org.apache.ignite.internal.binary.BinaryObjectImpl.hashCode()
> -------------------------------------------------------------------------------------------
>
> Key: IGNITE-3465
> URL: https://issues.apache.org/jira/browse/IGNITE-3465
> Project: Ignite
> Issue Type: Bug
> Reporter: Ksenia Rybakova
> Attachments: hs_err_pid12976.log, ignite-base-load-config.xml,
> run-load.properties, run-load.xml
>
>
> During streaming load test some of JVMs crashed while logging info about
> "Failed to wait for partition map exchange" problem
> {noformat}
> [01:05:23,949][WARN
> ][exchange-worker-#96%null%][GridCachePartitionExchangeManager] Failed to
> wait for partition map exchange [topVer=AffinityTopologyVersion [topVer=61,
> minorTopVer=0], node=4f7b195a-62e3-4c32-b496-4a98de50b277]. Dumping
> [01:05:23,949][WARN
> ][exchange-worker-#96%null%][GridCachePartitionExchangeManager] Ready
> affinity version: AffinityTopologyVersion [topVer=60, minorTopVer=0]
> [01:05:24,107][WARN
> ][exchange-worker-#96%null%][GridCachePartitionExchangeManager] Last exchange
> future: GridDhtPartitionsExchangeFuture [dummy=false, forcePreload=false,
> reassign=false, discoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode
> [01:05:24,108][WARN
> ][exchange-worker-#96%null%][GridCachePartitionExchangeManager] Pending
> exchange futures:
> [01:05:24,108][WARN
> ][exchange-worker-#96%null%][GridCachePartitionExchangeManager] Last 10
> exchange futures (total: 48):
> [01:05:24,109][WARN
> ][exchange-worker-#96%null%][GridCachePartitionExchangeManager] >>>
> GridDhtPartitionsExchangeFuture [dummy=false, forcePreload=false,
> reassign=false, discoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode
> [id=7291b03d-ce91
> [01:05:24,110][WARN
> ][exchange-worker-#96%null%][GridCachePartitionExchangeManager] >>>
> GridDhtPartitionsExchangeFuture [dummy=false, forcePreload=false,
> reassign=false, discoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode
> [id=739ab624-7bce
> [01:05:24,111][WARN
> ][exchange-worker-#96%null%][GridCachePartitionExchangeManager] >>>
> GridDhtPartitionsExchangeFuture [dummy=false, forcePreload=false,
> reassign=false, discoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode
> [id=93572383-07e8
> [01:05:24,111][WARN
> ][exchange-worker-#96%null%][GridCachePartitionExchangeManager] >>>
> GridDhtPartitionsExchangeFuture [dummy=false, forcePreload=false,
> reassign=false, discoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode
> [id=3c24caa1-d4b2
> [01:05:24,112][WARN
> ][exchange-worker-#96%null%][GridCachePartitionExchangeManager] >>>
> GridDhtPartitionsExchangeFuture [dummy=false, forcePreload=false,
> reassign=false, discoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode
> [id=e762044c-8ccc
> [01:05:24,118][WARN
> ][exchange-worker-#96%null%][GridCachePartitionExchangeManager] >>>
> GridDhtPartitionsExchangeFuture [dummy=false, forcePreload=false,
> reassign=false, discoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode
> [id=5ffbe144-f843
> [01:05:24,119][WARN
> ][exchange-worker-#96%null%][GridCachePartitionExchangeManager] >>>
> GridDhtPartitionsExchangeFuture [dummy=false, forcePreload=false,
> reassign=false, discoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode
> [id=28199123-fa5b
> [01:05:24,119][WARN
> ][exchange-worker-#96%null%][GridCachePartitionExchangeManager] >>>
> GridDhtPartitionsExchangeFuture [dummy=false, forcePreload=false,
> reassign=false, discoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode
> [id=14d27ae8-ee26
> [01:05:24,120][WARN
> ][exchange-worker-#96%null%][GridCachePartitionExchangeManager] >>>
> GridDhtPartitionsExchangeFuture [dummy=false, forcePreload=false,
> reassign=false, discoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode
> [id=fea5c90e-0467
> [01:05:24,120][WARN
> ][exchange-worker-#96%null%][GridCachePartitionExchangeManager] >>>
> GridDhtPartitionsExchangeFuture [dummy=false, forcePreload=false,
> reassign=false, discoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode
> [id=a5370f23-42bc
> [01:05:24,122][WARN
> ][exchange-worker-#96%null%][GridCachePartitionExchangeManager] Pending
> transactions:
> [01:05:24,767][WARN
> ][exchange-worker-#96%null%][GridCachePartitionExchangeManager] >>>
> GridDhtTxRemote [nearNodeId=fea5c90e-0467-462a-8755-928b539fb278,
> rmtFutId=a52022ed551-938a204f-bf03-451b-8425-be7047673bf6,
> nearXidVer=GridCacheVers
> [01:05:24,769][WARN
> ][exchange-worker-#96%null%][GridCachePartitionExchangeManager] >>>
> GridDhtTxRemote [nearNodeId=739ab624-7bce-4e4b-81cc-3696f65247bd,
> rmtFutId=ddfe12ed551-15e1d48c-8146-44ac-8d7b-370ea34fa44b,
> nearXidVer=GridCacheVers
> [01:05:24,771][WARN
> ][exchange-worker-#96%null%][GridCachePartitionExchangeManager] >>>
> GridDhtTxRemote [nearNodeId=5ffbe144-f843-4ad1-9ade-688a18f0f872,
> rmtFutId=401f12ed551-62727d5b-0dd3-48a0-bb04-72f7363ec19b,
> nearXidVer=GridCacheVers
> [01:05:24,771][WARN
> ][exchange-worker-#96%null%][GridCachePartitionExchangeManager] Pending
> explicit locks:
> [01:05:24,771][WARN
> ][exchange-worker-#96%null%][GridCachePartitionExchangeManager] Pending cache
> futures:
> [01:05:24,771][WARN
> ][exchange-worker-#96%null%][GridCachePartitionExchangeManager] Pending
> atomic cache futures:
> [01:05:24,786][WARN ][exchange-worker-#96%null%][TcpCommunicationSpi]
> Communication SPI recovery descriptors:
> [key=ClientKey [nodeId=14d27ae8-ee26-49a3-a7b5-6d1482850f21, order=54],
> msgsSent=181, msgsAckedByRmt=176, msgsRcvd=234, lastAcked=224, reserveCnt=1,
> descIdHash=952738531]
> [key=ClientKey [nodeId=5ffbe144-f843-4ad1-9ade-688a18f0f872, order=56],
> msgsSent=131, msgsAckedByRmt=128, msgsRcvd=168, lastAcked=160, reserveCnt=1,
> descIdHash=1038418485]
> [key=ClientKey [nodeId=876bec01-deea-4db8-a5ad-9159864ecae9, order=13],
> msgsSent=88, msgsAckedByRmt=88, msgsRcvd=88, lastAcked=88, reserveCnt=1,
> descIdHash=99878226]
> [key=ClientKey [nodeId=e762044c-8ccc-45d4-bd15-703baced868e, order=57],
> msgsSent=122, msgsAckedByRmt=112, msgsRcvd=128, lastAcked=128, reserveCnt=1,
> descIdHash=1468616075]
> [key=ClientKey [nodeId=e0454859-339d-4dd4-9766-bb96e71200c5, order=41],
> msgsSent=136, msgsAckedByRmt=128, msgsRcvd=165, lastAcked=160, reserveCnt=1,
> descIdHash=2000817554]
> [key=ClientKey [nodeId=93572383-07e8-4a6b-9482-642af1b87281, order=59],
> msgsSent=177, msgsAckedByRmt=176, msgsRcvd=186, lastAcked=176, reserveCnt=1,
> descIdHash=294201705]
> [key=ClientKey [nodeId=a5370f23-42bc-449d-93f7-83b04ea13668, order=52],
> msgsSent=282, msgsAckedByRmt=272, msgsRcvd=279, lastAcked=272, reserveCnt=1,
> descIdHash=1140700541]
> [key=ClientKey [nodeId=75627e99-8f2c-40e3-a0ba-79e28c1b4c32, order=44],
> msgsSent=139, msgsAckedByRmt=128, msgsRcvd=196, lastAcked=192, reserveCnt=1,
> descIdHash=1167475818]
> [key=ClientKey [nodeId=52f6e634-4ab7-4018-802f-c6f6cf7c9d3f, order=16],
> msgsSent=195, msgsAckedByRmt=192, msgsRcvd=197, lastAcked=192, reserveCnt=1,
> descIdHash=207990487]
> [key=ClientKey [nodeId=cd11646a-d9df-4611-bcc5-9b9625dd588b, order=17],
> msgsSent=187, msgsAckedByRmt=176, msgsRcvd=187, lastAcked=176, reserveCnt=3,
> descIdHash=1740221340]
> [key=ClientKey [nodeId=408d0657-3342-48c4-9ba0-883fba5ecea8, order=42],
> msgsSent=179, msgsAckedByRmt=176, msgsRcvd=205, lastAcked=192, reserveCnt=1,
> descIdHash=1712625195]
> [key=ClientKey [nodeId=aa52c200-1686-4985-acd7-7aadad71d874, order=50],
> msgsSent=89, msgsAckedByRmt=80, msgsRcvd=122, lastAcked=112, reserveCnt=1,
> descIdHash=1368331624]
> [key=ClientKey [nodeId=3c24caa1-d4b2-4827-9a12-be3472c09f2a, order=58],
> msgsSent=70, msgsAckedByRmt=64, msgsRcvd=55, lastAcked=48, reserveCnt=1,
> descIdHash=147228845]
> [key=ClientKey [nodeId=47935631-2877-4d7d-be19-08333ae87942, order=49],
> msgsSent=86, msgsAckedByRmt=80, msgsRcvd=112, lastAcked=112, reserveCnt=1,
> descIdHash=574036571]
> [key=ClientKey [nodeId=28199123-fa5b-4fa3-98d3-ee5c449662b0, order=55],
> msgsSent=151, msgsAckedByRmt=144, msgsRcvd=221, lastAcked=208, reserveCnt=1,
> descIdHash=1336106473]
> [key=ClientKey [nodeId=2c1893ad-ffe9-4f5f-92d2-5ecf38080337, order=43],
> msgsSent=76, msgsAckedByRmt=64, msgsRcvd=205, lastAcked=192, reserveCnt=1,
> descIdHash=1852478679]
> [key=ClientKey [nodeId=fff86516-4034-43af-b417-0d845f49c142, order=47],
> msgsSent=134, msgsAckedByRmt=128, msgsRcvd=135, lastAcked=128, reserveCnt=1,
> descIdHash=391243747]
> [key=ClientKey [nodeId=796d2fe8-4598-4794-9ec3-53ff3e4f1630, order=1],
> msgsSent=62, msgsAckedByRmt=61, msgsRcvd=112, lastAcked=112, reserveCnt=1,
> descIdHash=38728715]
> [key=ClientKey [nodeId=e4d6fe30-29da-4958-8a8b-f50c5c8c8f3f, order=2],
> msgsSent=62, msgsAckedByRmt=62, msgsRcvd=62, lastAcked=62, reserveCnt=1,
> descIdHash=225967964]
> [key=ClientKey [nodeId=c7955df7-d6ef-4a8e-976a-059eb644fa10, order=45],
> msgsSent=185, msgsAckedByRmt=176, msgsRcvd=244, lastAcked=240, reserveCnt=1,
> descIdHash=1092483052]
> [key=ClientKey [nodeId=6d9b4404-561d-475e-a9e2-4c854b2d10e7, order=28],
> msgsSent=159, msgsAckedByRmt=144, msgsRcvd=159, lastAcked=144, reserveCnt=1,
> descIdHash=377473114]
> [key=ClientKey [nodeId=bc0e61c1-fe0e-432e-bafa-6c159fcab4d9, order=6],
> msgsSent=0, msgsAckedByRmt=0, msgsRcvd=20, lastAcked=20, reserveCnt=1,
> descIdHash=523933760]
> [key=ClientKey [nodeId=739ab624-7bce-4e4b-81cc-3696f65247bd, order=60],
> msgsSent=43, msgsAckedByRmt=43, msgsRcvd=12, lastAcked=12, reserveCnt=1,
> descIdHash=1071751620]
> [key=ClientKey [nodeId=38ffbc41-33ac-4c5f-9569-197cefa678b3, order=48],
> msgsSent=161, msgsAckedByRmt=160, msgsRcvd=190, lastAcked=176, reserveCnt=1,
> descIdHash=1984129951]
> [key=ClientKey [nodeId=fea5c90e-0467-462a-8755-928b539fb278, order=53],
> msgsSent=151, msgsAckedByRmt=144, msgsRcvd=179, lastAcked=176, reserveCnt=1,
> descIdHash=1146015841]
> [key=ClientKey [nodeId=78d09665-9ded-4d1c-a46d-11a212af90fa, order=46],
> msgsSent=162, msgsAckedByRmt=160, msgsRcvd=192, lastAcked=192, reserveCnt=1,
> descIdHash=307409744]
> [key=ClientKey [nodeId=072afa77-8869-446a-8454-36e47956ede9, order=51],
> msgsSent=49, msgsAckedByRmt=49, msgsRcvd=18, lastAcked=18, reserveCnt=1,
> descIdHash=1937396373]
> #
> # A fatal error has been detected by the Java Runtime Environment:
> #
> # SIGSEGV (0xb) at pc=0x00007fb6853771c4, pid=12976, tid=140414314960640
> #
> # JRE version: Java(TM) SE Runtime Environment (7.0_80-b15) (build
> 1.7.0_80-b15)
> # Java VM: Java HotSpot(TM) 64-Bit Server VM (24.80-b11 mixed mode
> linux-amd64 compressed oops)
> # Problematic frame:
> # J 1501 C2 org.apache.ignite.internal.binary.BinaryObjectImpl.hashCode()I
> (15 bytes) @ 0x00007fb6853771c4 [0x00007fb685377180+0x44]
> #
> # Failed to write core dump. Core dumps have been disabled. To enable core
> dumping, try "ulimit -c unlimited" before starting Java again
> #
> # An error report file with more information is saved as:
> # hs_err_pid12976.log
> [thread 140414317065984 also had an error]
> #
> # If you would like to submit a bug report, please visit:
> # http://bugreport.java.com/bugreport/crash.jsp
> #
> {noformat}
> Load test config:
> - Benchmark name: IgniteCacheRandomOperationBenchmark
> - Operations: preloading (streaming)
> - Heap: 8Gb for servers, 4Gb for clients
> - 20 clients, 40 servers
> - Preload amount: 8M
> - Number of caches: 12
> - Types of caches (atomicity mode): different (atomic, transactional)
> - Types of caches (tiered storage mode) : different (onheap without
> eviction, onheap with eviction, offheap_tired, offheap_values)
> - Types of caches (indexing): different (with and without indexes)
> - Types of caches (cache mode): partitioned
> - Backups count: 1
> Attachments:
> - Error report file hs_err_pid12976.log
> - xml configs and property file that were used to run yardstick
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
