Hi guys. I've got a reproducer that can be related. See comments http://apache-ignite-users.70518.x6.nabble.com/Partition-eviction-failed-this-can-cause-grid-hang-Caused-by-java-lang-IllegalStateException-Failed--tp19122p20524.html
Sergey Sergeev, just for reference, what kind of file system do you use with Ignite's persistence? Arseny Kovalchuk Senior Software Engineer at Synesis skype: arseny.kovalchuk mobile: +375 (29) 666-16-16 LinkedIn Profile <http://www.linkedin.com/in/arsenykovalchuk/en> On 9 March 2018 at 12:31, Sergey Sergeev <webs.supp...@gmail.com> wrote: > Hi Mikhail, > > Unfortunately, the problem has repeated itself on ignite-core-2.3.3 > > 27.02.18 00:27:55 ERROR GridCacheIoManager - Failed to process message > [senderId=8f99c887-cd4b-4c38-a649-ca430040d535, messageType=class > o.a.i.i.processors.cache.distributed.dht.atomic.GridNearAtom > icUpdateResponse] > org.apache.ignite.IgniteException: Runtime failure on bounds: > [lower=null, upper=PendingRow []] > at org.apache.ignite.internal.processors.cache.persistence.tree > .BPlusTree.find(BPlusTree.java:954) ~[ignite-core-2.3.3.jar:2.3.3] > at org.apache.ignite.internal.processors.cache.persistence.tree > .BPlusTree.find(BPlusTree.java:933) ~[ignite-core-2.3.3.jar:2.3.3] > at org.apache.ignite.internal.processors.cache.IgniteCacheOffhe > apManagerImpl.expire(IgniteCacheOffheapManagerImpl.java:979) > ~[ignite-core-2.3.3.jar:2.3.3] > at org.apache.ignite.internal.processors.cache. > *GridCacheTtlManager.expire*(GridCacheTtlManager.java:197) > ~[ignite-core-2.3.3.jar:2.3.3] > at org.apache.ignite.internal.processors.cache.GridCacheUtils. > unwindEvicts(GridCacheUtils.java:833) ~[ignite-core-2.3.3.jar:2.3.3] > at org.apache.ignite.internal.processors.cache.GridCacheIoManag > er.onMessageProcessed(GridCacheIoManager.java:1099) > ~[ignite-core-2.3.3.jar:2.3.3] > at org.apache.ignite.internal.processors.cache.GridCacheIoManag > er.processMessage(GridCacheIoManager.java:1072) > ~[ignite-core-2.3.3.jar:2.3.3] > at org.apache.ignite.internal.processors.cache.GridCacheIoManag > er.onMessage0(GridCacheIoManager.java:579) ~[ignite-core-2.3.3.jar:2.3.3] > at org.apache.ignite.internal.processors.cache.GridCacheIoManag > er.handleMessage(GridCacheIoManager.java:378) > ~[ignite-core-2.3.3.jar:2.3.3] > at org.apache.ignite.internal.processors.cache.GridCacheIoManag > er.handleMessage(GridCacheIoManager.java:304) > ~[ignite-core-2.3.3.jar:2.3.3] > at org.apache.ignite.internal.processors.cache.GridCacheIoManag > er.access$100(GridCacheIoManager.java:99) ~[ignite-core-2.3.3.jar:2.3.3] > at org.apache.ignite.internal.processors.cache.GridCacheIoManag > er$1.onMessage(GridCacheIoManager.java:293) ~[ignite-core-2.3.3.jar:2.3.3] > at org.apache.ignite.internal.managers.communication.GridIoMana > ger.invokeListener(GridIoManager.java:1555) ~[ignite-core-2.3.3.jar:2.3.3] > at org.apache.ignite.internal.managers.communication.GridIoMana > ger.processRegularMessage0(GridIoManager.java:1183) > ~[ignite-core-2.3.3.jar:2.3.3] > at org.apache.ignite.internal.managers.communication.GridIoMana > ger.access$4200(GridIoManager.java:126) ~[ignite-core-2.3.3.jar:2.3.3] > at org.apache.ignite.internal.managers.communication.GridIoMana > ger$9.run(GridIoManager.java:1090) ~[ignite-core-2.3.3.jar:2.3.3] > at > org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505) > ~[ignite-core-2.3.3.jar:2.3.3] > at java.lang.Thread.run(Thread.java:748) [?:1.8.0_151] > Caused by: java.lang.IllegalStateException: Failed to get page IO > instance (page content is corrupted) > at org.apache.ignite.internal.processors.cache.persistence.tree > .io.IOVersions.forVersion(IOVersions.java:83) > ~[ignite-core-2.3.3.jar:2.3.3] > at org.apache.ignite.internal.processors.cache.persistence.tree > .io.IOVersions.forPage(IOVersions.java:95) ~[ignite-core-2.3.3.jar:2.3.3] > at org.apache.ignite.internal.processors.cache.persistence.Cach > eDataRowAdapter.initFromLink(CacheDataRowAdapter.java:148) > ~[ignite-core-2.3.3.jar:2.3.3] > at org.apache.ignite.internal.processors.cache.persistence.Cach > eDataRowAdapter.initFromLink(CacheDataRowAdapter.java:102) > ~[ignite-core-2.3.3.jar:2.3.3] > at > org.apache.ignite.internal.processors.cache.tree.PendingRow.initKey(PendingRow.java:72) > ~[ignite-core-2.3.3.jar:2.3.3] > at org.apache.ignite.internal.processors.cache.tree.PendingEntr > iesTree.getRow(PendingEntriesTree.java:118) ~[ignite-core-2.3.3.jar:2.3.3] > at org.apache.ignite.internal.processors.cache.tree.PendingEntr > iesTree.getRow(PendingEntriesTree.java:31) ~[ignite-core-2.3.3.jar:2.3.3] > at org.apache.ignite.internal.processors.cache.persistence.tree > .BPlusTree$ForwardCursor.fillFromBuffer(BPlusTree.java:4539) > ~[ignite-core-2.3.3.jar:2.3.3] > at org.apache.ignite.internal.processors.cache.persistence.tree > .BPlusTree$ForwardCursor.init(BPlusTree.java:4441) > ~[ignite-core-2.3.3.jar:2.3.3] > at org.apache.ignite.internal.processors.cache.persistence.tree > .BPlusTree$ForwardCursor.access$5300(BPlusTree.java:4380) > ~[ignite-core-2.3.3.jar:2.3.3] > at org.apache.ignite.internal.processors.cache.persistence.tree > .BPlusTree.findLowerUnbounded(BPlusTree.java:910) > ~[ignite-core-2.3.3.jar:2.3.3] > at org.apache.ignite.internal.processors.cache.persistence.tree > .BPlusTree.find(BPlusTree.java:942) ~[ignite-core-2.3.3.jar:2.3.3] > ... 17 more > > > And if we are reading from cache... > > > 27.02.18 00:27:56 ERROR MessagePartProcessingHandler - error processing > incoming messagePartDeliveryReceipt=MessagePartDeliveryReceipt{id=null, > abonent=7****, externalId='7586F73B'} > org.springframework.dao.DataAccessResourceFailureException: Runtime > failure on bounds: [lower=null, upper=PendingRow []]; nested exception is > class org.apache.ignite.IgniteException: Runtime failure on bounds: > [lower=null, upper=PendingRow []] > at com.a1s.sdp.data.dao.ignite.IgniteExceptionTranslator.transl > ateExceptionIfPossible(IgniteExceptionTranslator.java:35) > ~[core-data-1.5-SNAPSHOT.jar:3752] > at > com.a1s.sdp.data.dao.ignite.IgniteDaoHelper.wrapIgnite(IgniteDaoHelper.java:41) > ~[core-data-1.5-SNAPSHOT.jar:3752] > at com.a1s.sdp.data.dao.ignite.BaseIgniteCassandraRepository.fi > ndOne(BaseIgniteCassandraRepository.java:136) > ~[core-data-1.5-SNAPSHOT.jar:3752] > at com.a1s.sdp.common.service.TransportServiceImpl.processMessa > gePartDelivered(TransportServiceImpl.java:213) > ~[core-common-1.5-SNAPSHOT.jar:3752] > at com.a1s.sdp.common.service.TransportServiceImpl.messagePartD > elivered(TransportServiceImpl.java:202) ~[core-common-1.5-SNAPSHOT.jar > :3752] > at com.a1s.sdp.common.transport.handler.MessagePartProcessingHa > ndler.lambda$channelReadMessagePartDeliveryReceipt$1(MessagePartProcessingHandler.java:91) > ~[core-common-1.5-SNAPSHOT.jar:3752] > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > ~[?:1.8.0_151] > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > ~[?:1.8.0_151] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > ~[?:1.8.0_151] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > ~[?:1.8.0_151] > at java.lang.Thread.run(Thread.java:748) [?:*1.8.0_151*] > Caused by: org.apache.ignite.IgniteException: *Runtime failure on bounds: > [lower=null, upper=PendingRow []]* > at org.apache.ignite.internal.processors.cache.persistence.tree > .BPlusTree.find(BPlusTree.java:954) ~[ignite-core-2.3.3.jar:2.3.3] > at org.apache.ignite.internal.processors.cache.persistence.tree > .BPlusTree.find(BPlusTree.java:933) ~[ignite-core-2.3.3.jar:2.3.3] > at org.apache.ignite.internal.processors.cache. > *IgniteCacheOffheapManagerImpl.**expire*(IgniteCacheOffheapManagerImpl.java:979) > ~[ignite-core-2.3.3.jar:2.3.3] > at org.apache.ignite.internal.processors.cache.GridCacheTtlMana > ger.expire(GridCacheTtlManager.java:197) ~[ignite-core-2.3.3.jar:2.3.3] > at org.apache.ignite.internal.processors.cache.GridCacheUtils. > unwindEvicts(GridCacheUtils.java:833) ~[ignite-core-2.3.3.jar:2.3.3] > at org.apache.ignite.internal.processors.cache.GridCacheGateway > .leaveNoLock(GridCacheGateway.java:240) ~[ignite-core-2.3.3.jar:2.3.3] > at org.apache.ignite.internal.processors.cache.GridCacheGateway > .leave(GridCacheGateway.java:225) ~[ignite-core-2.3.3.jar:2.3.3] > at org.apache.ignite.internal.processors.cache.GatewayProtected > CacheProxy.onLeave(GatewayProtectedCacheProxy.java:1510) > ~[ignite-core-2.3.3.jar:2.3.3] > at org.apache.ignite.internal.processors.cache.GatewayProtected > CacheProxy.get(GatewayProtectedCacheProxy.java:608) > ~[ignite-core-2.3.3.jar:2.3.3] > at com.a1s.sdp.data.dao.ignite.BaseIgniteCassandraRepository.la > mbda$findOne$7(BaseIgniteCassandraRepository.java:136) > ~[core-data-1.5-SNAPSHOT.jar:3752] > at > com.a1s.sdp.data.dao.ignite.IgniteDaoHelper.wrapIgnite(IgniteDaoHelper.java:39) > ~[core-data-1.5-SNAPSHOT.jar:3752] > ... 9 more > Caused by: java.lang.IllegalStateException: Failed to get page IO > instance (page content is corrupted) > at org.apache.ignite.internal.processors.cache.persistence.tree > .io.IOVersions.forVersion(IOVersions.java:83) > ~[ignite-core-2.3.3.jar:2.3.3] > at org.apache.ignite.internal.processors.cache.persistence.tree > .io.IOVersions.forPage(IOVersions.java:95) ~[ignite-core-2.3.3.jar:2.3.3] > at org.apache.ignite.internal.processors.cache.persistence.Cach > eDataRowAdapter.initFromLink(CacheDataRowAdapter.java:148) > ~[ignite-core-2.3.3.jar:2.3.3] > at org.apache.ignite.internal.processors.cache.persistence.Cach > eDataRowAdapter.initFromLink(CacheDataRowAdapter.java:102) > ~[ignite-core-2.3.3.jar:2.3.3] > at > org.apache.ignite.internal.processors.cache.tree.PendingRow.initKey(PendingRow.java:72) > ~[ignite-core-2.3.3.jar:2.3.3] > at org.apache.ignite.internal.processors.cache.tree.PendingEntr > iesTree.getRow(PendingEntriesTree.java:118) ~[ignite-core-2.3.3.jar:2.3.3] > at org.apache.ignite.internal.processors.cache.tree.PendingEntr > iesTree.getRow(PendingEntriesTree.java:31) ~[ignite-core-2.3.3.jar:2.3.3] > at org.apache.ignite.internal.processors.cache.persistence.tree > .BPlusTree$ForwardCursor.fillFromBuffer(BPlusTree.java:4539) > ~[ignite-core-2.3.3.jar:2.3.3] > at org.apache.ignite.internal.processors.cache.persistence.tree > .BPlusTree$ForwardCursor.init(BPlusTree.java:4441) > ~[ignite-core-2.3.3.jar:2.3.3] > at org.apache.ignite.internal.processors.cache.persistence.tree > .BPlusTree$ForwardCursor.access$5300(BPlusTree.java:4380) > ~[ignite-core-2.3.3.jar:2.3.3] > at org.apache.ignite.internal.processors.cache.persistence.tree > .BPlusTree.findLowerUnbounded(BPlusTree.java:910) > ~[ignite-core-2.3.3.jar:2.3.3] > at org.apache.ignite.internal.processors.cache.persistence.tree > .BPlusTree.find(BPlusTree.java:942) ~[ignite-core-2.3.3.jar:2.3.3] > at org.apache.ignite.internal.processors.cache.persistence.tree > .BPlusTree.find(BPlusTree.java:933) ~[ignite-core-2.3.3.jar:2.3.3] > at org.apache.ignite.internal.processors.cache. > *IgniteCacheOffheapManagerImpl.**expire*(IgniteCacheOffheapManagerImpl.java:979) > ~[ignite-core-2.3.3.jar:2.3.3] > at org.apache.ignite.internal.processors.cache.GridCacheTtlMana > ger.expire(GridCacheTtlManager.java:197) ~[ignite-core-2.3.3.jar:2.3.3] > at org.apache.ignite.internal.processors.cache.GridCacheUtils. > unwindEvicts(GridCacheUtils.java:833) ~[ignite-core-2.3.3.jar:2.3.3] > at org.apache.ignite.internal.processors.cache.GridCacheGateway > .leaveNoLock(GridCacheGateway.java:240) ~[ignite-core-2.3.3.jar:2.3.3] > at org.apache.ignite.internal.processors.cache.GridCacheGateway > .leave(GridCacheGateway.java:225) ~[ignite-core-2.3.3.jar:2.3.3] > at org.apache.ignite.internal.processors.cache.GatewayProtected > CacheProxy.onLeave(GatewayProtectedCacheProxy.java:1510) > ~[ignite-core-2.3.3.jar:2.3.3] > at org.apache.ignite.internal.processors.cache.GatewayProtected > CacheProxy.get(GatewayProtectedCacheProxy.java:608) > ~[ignite-core-2.3.3.jar:2.3.3] > at com.a1s.sdp.data.dao.ignite.BaseIgniteCassandraRepository.la > mbda$findOne$7(BaseIgniteCassandraRepository.java:136) > ~[core-data-1.5-SNAPSHOT.jar:3752] > at > com.a1s.sdp.data.dao.ignite.IgniteDaoHelper.wrapIgnite(IgniteDaoHelper.java:39) > ~[core-data-1.5-SNAPSHOT.jar:3752] > ... 9 more > > > The attachment (https://yadi.sk/d/u7fSK8Y53SpCpA) contains a part of > cache obtained from node of cluster - /var/ignite/***/db/node00- > 66fd86f5-4e2d-4f1b-8fe1-993bf5937d99/cache-MessagePartSent. > This cache contains objects of class MessagePartSent: > > @Table("message_part_sent_external") > public final class MessagePartSent { > @Id > private MessagePartSentPk id; > private MessagePartPk partId; > > public MessagePartSentPk getId() { > return id; > } > > public void setId(MessagePartSentPk id) { > this.id = id; > } > > public MessagePartPk getPartId() { > return partId; > } > > public void setPartId(MessagePartPk partId) { > this.partId = partId; > } > > @Override > public String toString() { > final StringBuilder sb = new StringBuilder("MessagePartSent{"); > sb.append("id=").append(id); > sb.append(", partId=").append(partId); > sb.append('}'); > return sb.toString(); > } > > public static final class MessagePartSentPk implements Serializable { > private Integer linkId; > private String externalId; > > public MessagePartSentPk() { > // > } > > public MessagePartSentPk(Integer linkId, String externalId) { > this.linkId = linkId; > this.externalId = externalId; > } > > public Integer getLinkId() { > return linkId; > } > > public void setLinkId(Integer linkId) { > this.linkId = linkId; > } > > public String getExternalId() { > return externalId; > } > > public void setExternalId(String externalId) { > this.externalId = externalId; > } > > @Override > public boolean equals(Object o) { > if (this == o) return true; > if (!(o instanceof MessagePartSentPk)) return false; > > final MessagePartSentPk that = (MessagePartSentPk) o; > > if (externalId != null ? !externalId.equals(that.externalId) > : that.externalId != null) return false; > if (linkId != null ? !linkId.equals(that.linkId) : that.linkId > != null) return false; > > return true; > } > > @Override > public int hashCode() { > int result = linkId != null ? linkId.hashCode() : 0; > result = 31 * result + (externalId != null ? > externalId.hashCode() : 0); > return result; > } > > @Override > public String toString() { > final StringBuilder sb = new StringBuilder("MessagePartSent > Pk{"); > sb.append("linkId=").append(linkId); > sb.append(", externalId='").append(externalId).append('\''); > sb.append('}'); > return sb.toString(); > } > } > > } > > > public final class MessagePartPk implements Serializable { > private UUID transactionId; > private Short partNumber; > > public MessagePartPk() { > //default > } > > public MessagePartPk(UUID transactionId, Short partNumber) { > this.transactionId = transactionId; > this.partNumber = partNumber; > } > > public UUID getTransactionId() { > return transactionId; > } > > public void setTransactionId(UUID transactionId) { > this.transactionId = transactionId; > } > > public Short getPartNumber() { > return partNumber; > } > > public void setPartNumber(Short partNumber) { > this.partNumber = partNumber; > } > > @Override > public boolean equals(Object o) { > if (this == o) return true; > if (!(o instanceof MessagePartPk)) return false; > > final MessagePartPk messagePartPk = (MessagePartPk) o; > > if (partNumber != null ? !partNumber.equals(messagePartPk.partNumber) > : > messagePartPk.partNumber != null) return false; > if (transactionId != null ? > !transactionId.equals(messagePartPk.transactionId) > : > messagePartPk.transactionId != null) > return false; > > return true; > } > > @Override > public int hashCode() { > int result = transactionId != null ? transactionId.hashCode() : 0; > result = 31 * result + (partNumber != null ? partNumber.hashCode() > : 0); > return result; > } > > @Override > public String toString() { > final StringBuilder sb = new StringBuilder("MessagePartPk{"); > sb.append("transactionId=").append(transactionId); > sb.append(", partNumber=").append(partNumber); > sb.append('}'); > return sb.toString(); > } > > On Thu, Mar 8, 2018 at 11:03 PM, Dmitriy Setrakyan <dsetrak...@apache.org> > wrote: > >> Hi Sergey, >> >> The 2.4 release is about to be voted for. You can use the RC1 in the mean >> time: >> http://apache-ignite-developers.2346864.n4.nabble.com/VOTE- >> Apache-Ignite-2-4-0-RC1-td27687.html >> >> D. >> >> On Mon, Feb 19, 2018 at 6:43 AM, Mikhail <michael.cherka...@gmail.com> >> wrote: >> >>> Hi Sergey, >>> >>> The release of 2.4 should be soon, in a week or couple, however, there's >>> no >>> strong schedule for Apache releases. >>> >>> Could you please share a reproducer for the issue? Might be you can >>> share a >>> storage on which the issue can be reproduced? >>> >>> Thanks, >>> Mike. >>> >>> >>> >>> -- >>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/ >>> >> >> > > > -- > Best regards, > Sergey S. Sergeev (aka SSSerj) > Notes: https://twitter.com/ssserj > exit(0); >
