Hi, Arseny We are using ext4 with defaults,noatime,lazytime options on Virtuozzo Linux release 7.3.
We have refactored parts of logic and abandoned the use of any expired policies(( On Thu, Mar 15, 2018 at 7:34 PM, Arseny Kovalchuk < arseny.kovalc...@synesis.ru> wrote: > Hi guys. > > I've got a reproducer that can be related. See comments > http://apache-ignite-users.70518.x6.nabble.com/Partition-eviction-failed- > this-can-cause-grid-hang-Caused-by-java-lang- > IllegalStateException-Failed--tp19122p20524.html > > Sergey Sergeev, just for reference, what kind of file system do you use > with Ignite's persistence? > > > Arseny Kovalchuk > > Senior Software Engineer at Synesis > skype: arseny.kovalchuk > mobile: +375 (29) 666-16-16 > LinkedIn Profile <http://www.linkedin.com/in/arsenykovalchuk/en> > > On 9 March 2018 at 12:31, Sergey Sergeev <webs.supp...@gmail.com> wrote: > >> Hi Mikhail, >> >> Unfortunately, the problem has repeated itself on ignite-core-2.3.3 >> >> 27.02.18 00:27:55 ERROR GridCacheIoManager - Failed to process message >> [senderId=8f99c887-cd4b-4c38-a649-ca430040d535, messageType=class >> o.a.i.i.processors.cache.distributed.dht.atomic.GridNearAtom >> icUpdateResponse] >> org.apache.ignite.IgniteException: Runtime failure on bounds: >> [lower=null, upper=PendingRow []] >> at org.apache.ignite.internal.processors.cache.persistence.tree >> .BPlusTree.find(BPlusTree.java:954) ~[ignite-core-2.3.3.jar:2.3.3] >> at org.apache.ignite.internal.processors.cache.persistence.tree >> .BPlusTree.find(BPlusTree.java:933) ~[ignite-core-2.3.3.jar:2.3.3] >> at org.apache.ignite.internal.processors.cache.IgniteCacheOffhe >> apManagerImpl.expire(IgniteCacheOffheapManagerImpl.java:979) >> ~[ignite-core-2.3.3.jar:2.3.3] >> at org.apache.ignite.internal.processors.cache. >> *GridCacheTtlManager.expire*(GridCacheTtlManager.java:197) >> ~[ignite-core-2.3.3.jar:2.3.3] >> at org.apache.ignite.internal.processors.cache.GridCacheUtils.u >> nwindEvicts(GridCacheUtils.java:833) ~[ignite-core-2.3.3.jar:2.3.3] >> at org.apache.ignite.internal.processors.cache.GridCacheIoManag >> er.onMessageProcessed(GridCacheIoManager.java:1099) >> ~[ignite-core-2.3.3.jar:2.3.3] >> at org.apache.ignite.internal.processors.cache.GridCacheIoManag >> er.processMessage(GridCacheIoManager.java:1072) >> ~[ignite-core-2.3.3.jar:2.3.3] >> at org.apache.ignite.internal.processors.cache.GridCacheIoManag >> er.onMessage0(GridCacheIoManager.java:579) ~[ignite-core-2.3.3.jar:2.3.3] >> at org.apache.ignite.internal.processors.cache.GridCacheIoManag >> er.handleMessage(GridCacheIoManager.java:378) >> ~[ignite-core-2.3.3.jar:2.3.3] >> at org.apache.ignite.internal.processors.cache.GridCacheIoManag >> er.handleMessage(GridCacheIoManager.java:304) >> ~[ignite-core-2.3.3.jar:2.3.3] >> at org.apache.ignite.internal.processors.cache.GridCacheIoManag >> er.access$100(GridCacheIoManager.java:99) ~[ignite-core-2.3.3.jar:2.3.3] >> at org.apache.ignite.internal.processors.cache.GridCacheIoManag >> er$1.onMessage(GridCacheIoManager.java:293) >> ~[ignite-core-2.3.3.jar:2.3.3] >> at org.apache.ignite.internal.managers.communication.GridIoMana >> ger.invokeListener(GridIoManager.java:1555) >> ~[ignite-core-2.3.3.jar:2.3.3] >> at org.apache.ignite.internal.managers.communication.GridIoMana >> ger.processRegularMessage0(GridIoManager.java:1183) >> ~[ignite-core-2.3.3.jar:2.3.3] >> at org.apache.ignite.internal.managers.communication.GridIoMana >> ger.access$4200(GridIoManager.java:126) ~[ignite-core-2.3.3.jar:2.3.3] >> at org.apache.ignite.internal.managers.communication.GridIoMana >> ger$9.run(GridIoManager.java:1090) ~[ignite-core-2.3.3.jar:2.3.3] >> at >> org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505) >> ~[ignite-core-2.3.3.jar:2.3.3] >> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_151] >> Caused by: java.lang.IllegalStateException: Failed to get page IO >> instance (page content is corrupted) >> at org.apache.ignite.internal.processors.cache.persistence.tree >> .io.IOVersions.forVersion(IOVersions.java:83) >> ~[ignite-core-2.3.3.jar:2.3.3] >> at org.apache.ignite.internal.processors.cache.persistence.tree >> .io.IOVersions.forPage(IOVersions.java:95) ~[ignite-core-2.3.3.jar:2.3.3] >> at org.apache.ignite.internal.processors.cache.persistence.Cach >> eDataRowAdapter.initFromLink(CacheDataRowAdapter.java:148) >> ~[ignite-core-2.3.3.jar:2.3.3] >> at org.apache.ignite.internal.processors.cache.persistence.Cach >> eDataRowAdapter.initFromLink(CacheDataRowAdapter.java:102) >> ~[ignite-core-2.3.3.jar:2.3.3] >> at >> org.apache.ignite.internal.processors.cache.tree.PendingRow.initKey(PendingRow.java:72) >> ~[ignite-core-2.3.3.jar:2.3.3] >> at org.apache.ignite.internal.processors.cache.tree.PendingEntr >> iesTree.getRow(PendingEntriesTree.java:118) >> ~[ignite-core-2.3.3.jar:2.3.3] >> at org.apache.ignite.internal.processors.cache.tree.PendingEntr >> iesTree.getRow(PendingEntriesTree.java:31) ~[ignite-core-2.3.3.jar:2.3.3] >> at org.apache.ignite.internal.processors.cache.persistence.tree >> .BPlusTree$ForwardCursor.fillFromBuffer(BPlusTree.java:4539) >> ~[ignite-core-2.3.3.jar:2.3.3] >> at org.apache.ignite.internal.processors.cache.persistence.tree >> .BPlusTree$ForwardCursor.init(BPlusTree.java:4441) >> ~[ignite-core-2.3.3.jar:2.3.3] >> at org.apache.ignite.internal.processors.cache.persistence.tree >> .BPlusTree$ForwardCursor.access$5300(BPlusTree.java:4380) >> ~[ignite-core-2.3.3.jar:2.3.3] >> at org.apache.ignite.internal.processors.cache.persistence.tree >> .BPlusTree.findLowerUnbounded(BPlusTree.java:910) >> ~[ignite-core-2.3.3.jar:2.3.3] >> at org.apache.ignite.internal.processors.cache.persistence.tree >> .BPlusTree.find(BPlusTree.java:942) ~[ignite-core-2.3.3.jar:2.3.3] >> ... 17 more >> >> >> And if we are reading from cache... >> >> >> 27.02.18 00:27:56 ERROR MessagePartProcessingHandler - error processing >> incoming messagePartDeliveryReceipt=MessagePartDeliveryReceipt{id=null, >> abonent=7****, externalId='7586F73B'} >> org.springframework.dao.DataAccessResourceFailureException: Runtime >> failure on bounds: [lower=null, upper=PendingRow []]; nested exception is >> class org.apache.ignite.IgniteException: Runtime failure on bounds: >> [lower=null, upper=PendingRow []] >> at com.a1s.sdp.data.dao.ignite.IgniteExceptionTranslator.transl >> ateExceptionIfPossible(IgniteExceptionTranslator.java:35) >> ~[core-data-1.5-SNAPSHOT.jar:3752] >> at >> com.a1s.sdp.data.dao.ignite.IgniteDaoHelper.wrapIgnite(IgniteDaoHelper.java:41) >> ~[core-data-1.5-SNAPSHOT.jar:3752] >> at com.a1s.sdp.data.dao.ignite.BaseIgniteCassandraRepository.fi >> ndOne(BaseIgniteCassandraRepository.java:136) >> ~[core-data-1.5-SNAPSHOT.jar:3752] >> at com.a1s.sdp.common.service.TransportServiceImpl.processMessa >> gePartDelivered(TransportServiceImpl.java:213) >> ~[core-common-1.5-SNAPSHOT.jar:3752] >> at com.a1s.sdp.common.service.TransportServiceImpl.messagePartD >> elivered(TransportServiceImpl.java:202) ~[core-common-1.5-SNAPSHOT.jar >> :3752] >> at com.a1s.sdp.common.transport.handler.MessagePartProcessingHa >> ndler.lambda$channelReadMessagePartDeliveryReceipt$1(MessagePartProcessingHandler.java:91) >> ~[core-common-1.5-SNAPSHOT.jar:3752] >> at >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) >> ~[?:1.8.0_151] >> at java.util.concurrent.FutureTask.run(FutureTask.java:266) >> ~[?:1.8.0_151] >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) >> ~[?:1.8.0_151] >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) >> ~[?:1.8.0_151] >> at java.lang.Thread.run(Thread.java:748) [?:*1.8.0_151*] >> Caused by: org.apache.ignite.IgniteException: *Runtime failure on >> bounds: [lower=null, upper=PendingRow []]* >> at org.apache.ignite.internal.processors.cache.persistence.tree >> .BPlusTree.find(BPlusTree.java:954) ~[ignite-core-2.3.3.jar:2.3.3] >> at org.apache.ignite.internal.processors.cache.persistence.tree >> .BPlusTree.find(BPlusTree.java:933) ~[ignite-core-2.3.3.jar:2.3.3] >> at org.apache.ignite.internal.processors.cache. >> *IgniteCacheOffheapManagerImpl.**expire*(IgniteCacheOffheapManagerImpl.java:979) >> ~[ignite-core-2.3.3.jar:2.3.3] >> at org.apache.ignite.internal.processors.cache.GridCacheTtlMana >> ger.expire(GridCacheTtlManager.java:197) ~[ignite-core-2.3.3.jar:2.3.3] >> at org.apache.ignite.internal.processors.cache.GridCacheUtils.u >> nwindEvicts(GridCacheUtils.java:833) ~[ignite-core-2.3.3.jar:2.3.3] >> at org.apache.ignite.internal.processors.cache.GridCacheGateway >> .leaveNoLock(GridCacheGateway.java:240) ~[ignite-core-2.3.3.jar:2.3.3] >> at org.apache.ignite.internal.processors.cache.GridCacheGateway >> .leave(GridCacheGateway.java:225) ~[ignite-core-2.3.3.jar:2.3.3] >> at org.apache.ignite.internal.processors.cache.GatewayProtected >> CacheProxy.onLeave(GatewayProtectedCacheProxy.java:1510) >> ~[ignite-core-2.3.3.jar:2.3.3] >> at org.apache.ignite.internal.processors.cache.GatewayProtected >> CacheProxy.get(GatewayProtectedCacheProxy.java:608) >> ~[ignite-core-2.3.3.jar:2.3.3] >> at com.a1s.sdp.data.dao.ignite.BaseIgniteCassandraRepository.la >> mbda$findOne$7(BaseIgniteCassandraRepository.java:136) >> ~[core-data-1.5-SNAPSHOT.jar:3752] >> at >> com.a1s.sdp.data.dao.ignite.IgniteDaoHelper.wrapIgnite(IgniteDaoHelper.java:39) >> ~[core-data-1.5-SNAPSHOT.jar:3752] >> ... 9 more >> Caused by: java.lang.IllegalStateException: Failed to get page IO >> instance (page content is corrupted) >> at org.apache.ignite.internal.processors.cache.persistence.tree >> .io.IOVersions.forVersion(IOVersions.java:83) >> ~[ignite-core-2.3.3.jar:2.3.3] >> at org.apache.ignite.internal.processors.cache.persistence.tree >> .io.IOVersions.forPage(IOVersions.java:95) ~[ignite-core-2.3.3.jar:2.3.3] >> at org.apache.ignite.internal.processors.cache.persistence.Cach >> eDataRowAdapter.initFromLink(CacheDataRowAdapter.java:148) >> ~[ignite-core-2.3.3.jar:2.3.3] >> at org.apache.ignite.internal.processors.cache.persistence.Cach >> eDataRowAdapter.initFromLink(CacheDataRowAdapter.java:102) >> ~[ignite-core-2.3.3.jar:2.3.3] >> at >> org.apache.ignite.internal.processors.cache.tree.PendingRow.initKey(PendingRow.java:72) >> ~[ignite-core-2.3.3.jar:2.3.3] >> at org.apache.ignite.internal.processors.cache.tree.PendingEntr >> iesTree.getRow(PendingEntriesTree.java:118) >> ~[ignite-core-2.3.3.jar:2.3.3] >> at org.apache.ignite.internal.processors.cache.tree.PendingEntr >> iesTree.getRow(PendingEntriesTree.java:31) ~[ignite-core-2.3.3.jar:2.3.3] >> at org.apache.ignite.internal.processors.cache.persistence.tree >> .BPlusTree$ForwardCursor.fillFromBuffer(BPlusTree.java:4539) >> ~[ignite-core-2.3.3.jar:2.3.3] >> at org.apache.ignite.internal.processors.cache.persistence.tree >> .BPlusTree$ForwardCursor.init(BPlusTree.java:4441) >> ~[ignite-core-2.3.3.jar:2.3.3] >> at org.apache.ignite.internal.processors.cache.persistence.tree >> .BPlusTree$ForwardCursor.access$5300(BPlusTree.java:4380) >> ~[ignite-core-2.3.3.jar:2.3.3] >> at org.apache.ignite.internal.processors.cache.persistence.tree >> .BPlusTree.findLowerUnbounded(BPlusTree.java:910) >> ~[ignite-core-2.3.3.jar:2.3.3] >> at org.apache.ignite.internal.processors.cache.persistence.tree >> .BPlusTree.find(BPlusTree.java:942) ~[ignite-core-2.3.3.jar:2.3.3] >> at org.apache.ignite.internal.processors.cache.persistence.tree >> .BPlusTree.find(BPlusTree.java:933) ~[ignite-core-2.3.3.jar:2.3.3] >> at org.apache.ignite.internal.processors.cache. >> *IgniteCacheOffheapManagerImpl.**expire*(IgniteCacheOffheapManagerImpl.java:979) >> ~[ignite-core-2.3.3.jar:2.3.3] >> at org.apache.ignite.internal.processors.cache.GridCacheTtlMana >> ger.expire(GridCacheTtlManager.java:197) ~[ignite-core-2.3.3.jar:2.3.3] >> at org.apache.ignite.internal.processors.cache.GridCacheUtils.u >> nwindEvicts(GridCacheUtils.java:833) ~[ignite-core-2.3.3.jar:2.3.3] >> at org.apache.ignite.internal.processors.cache.GridCacheGateway >> .leaveNoLock(GridCacheGateway.java:240) ~[ignite-core-2.3.3.jar:2.3.3] >> at org.apache.ignite.internal.processors.cache.GridCacheGateway >> .leave(GridCacheGateway.java:225) ~[ignite-core-2.3.3.jar:2.3.3] >> at org.apache.ignite.internal.processors.cache.GatewayProtected >> CacheProxy.onLeave(GatewayProtectedCacheProxy.java:1510) >> ~[ignite-core-2.3.3.jar:2.3.3] >> at org.apache.ignite.internal.processors.cache.GatewayProtected >> CacheProxy.get(GatewayProtectedCacheProxy.java:608) >> ~[ignite-core-2.3.3.jar:2.3.3] >> at com.a1s.sdp.data.dao.ignite.BaseIgniteCassandraRepository.la >> mbda$findOne$7(BaseIgniteCassandraRepository.java:136) >> ~[core-data-1.5-SNAPSHOT.jar:3752] >> at >> com.a1s.sdp.data.dao.ignite.IgniteDaoHelper.wrapIgnite(IgniteDaoHelper.java:39) >> ~[core-data-1.5-SNAPSHOT.jar:3752] >> ... 9 more >> >> >> The attachment (https://yadi.sk/d/u7fSK8Y53SpCpA) contains a part of >> cache obtained from node of cluster - /var/ignite/***/db/node00-66fd >> 86f5-4e2d-4f1b-8fe1-993bf5937d99/cache-MessagePartSent. >> This cache contains objects of class MessagePartSent: >> >> @Table("message_part_sent_external") >> public final class MessagePartSent { >> @Id >> private MessagePartSentPk id; >> private MessagePartPk partId; >> >> public MessagePartSentPk getId() { >> return id; >> } >> >> public void setId(MessagePartSentPk id) { >> this.id = id; >> } >> >> public MessagePartPk getPartId() { >> return partId; >> } >> >> public void setPartId(MessagePartPk partId) { >> this.partId = partId; >> } >> >> @Override >> public String toString() { >> final StringBuilder sb = new StringBuilder("MessagePartSent{"); >> sb.append("id=").append(id); >> sb.append(", partId=").append(partId); >> sb.append('}'); >> return sb.toString(); >> } >> >> public static final class MessagePartSentPk implements Serializable { >> private Integer linkId; >> private String externalId; >> >> public MessagePartSentPk() { >> // >> } >> >> public MessagePartSentPk(Integer linkId, String externalId) { >> this.linkId = linkId; >> this.externalId = externalId; >> } >> >> public Integer getLinkId() { >> return linkId; >> } >> >> public void setLinkId(Integer linkId) { >> this.linkId = linkId; >> } >> >> public String getExternalId() { >> return externalId; >> } >> >> public void setExternalId(String externalId) { >> this.externalId = externalId; >> } >> >> @Override >> public boolean equals(Object o) { >> if (this == o) return true; >> if (!(o instanceof MessagePartSentPk)) return false; >> >> final MessagePartSentPk that = (MessagePartSentPk) o; >> >> if (externalId != null ? !externalId.equals(that.externalId) >> : that.externalId != null) return false; >> if (linkId != null ? !linkId.equals(that.linkId) : >> that.linkId != null) return false; >> >> return true; >> } >> >> @Override >> public int hashCode() { >> int result = linkId != null ? linkId.hashCode() : 0; >> result = 31 * result + (externalId != null ? >> externalId.hashCode() : 0); >> return result; >> } >> >> @Override >> public String toString() { >> final StringBuilder sb = new StringBuilder("MessagePartSent >> Pk{"); >> sb.append("linkId=").append(linkId); >> sb.append(", externalId='").append(externalId).append('\''); >> sb.append('}'); >> return sb.toString(); >> } >> } >> >> } >> >> >> public final class MessagePartPk implements Serializable { >> private UUID transactionId; >> private Short partNumber; >> >> public MessagePartPk() { >> //default >> } >> >> public MessagePartPk(UUID transactionId, Short partNumber) { >> this.transactionId = transactionId; >> this.partNumber = partNumber; >> } >> >> public UUID getTransactionId() { >> return transactionId; >> } >> >> public void setTransactionId(UUID transactionId) { >> this.transactionId = transactionId; >> } >> >> public Short getPartNumber() { >> return partNumber; >> } >> >> public void setPartNumber(Short partNumber) { >> this.partNumber = partNumber; >> } >> >> @Override >> public boolean equals(Object o) { >> if (this == o) return true; >> if (!(o instanceof MessagePartPk)) return false; >> >> final MessagePartPk messagePartPk = (MessagePartPk) o; >> >> if (partNumber != null ? !partNumber.equals(messagePartPk.partNumber) >> : >> messagePartPk.partNumber != null) return false; >> if (transactionId != null ? >> !transactionId.equals(messagePartPk.transactionId) >> : >> messagePartPk.transactionId != null) >> return false; >> >> return true; >> } >> >> @Override >> public int hashCode() { >> int result = transactionId != null ? transactionId.hashCode() : 0; >> result = 31 * result + (partNumber != null ? >> partNumber.hashCode() : 0); >> return result; >> } >> >> @Override >> public String toString() { >> final StringBuilder sb = new StringBuilder("MessagePartPk{"); >> sb.append("transactionId=").append(transactionId); >> sb.append(", partNumber=").append(partNumber); >> sb.append('}'); >> return sb.toString(); >> } >> >> On Thu, Mar 8, 2018 at 11:03 PM, Dmitriy Setrakyan <dsetrak...@apache.org >> > wrote: >> >>> Hi Sergey, >>> >>> The 2.4 release is about to be voted for. You can use the RC1 in the >>> mean time: >>> http://apache-ignite-developers.2346864.n4.nabble.com/VOTE-A >>> pache-Ignite-2-4-0-RC1-td27687.html >>> >>> D. >>> >>> On Mon, Feb 19, 2018 at 6:43 AM, Mikhail <michael.cherka...@gmail.com> >>> wrote: >>> >>>> Hi Sergey, >>>> >>>> The release of 2.4 should be soon, in a week or couple, however, >>>> there's no >>>> strong schedule for Apache releases. >>>> >>>> Could you please share a reproducer for the issue? Might be you can >>>> share a >>>> storage on which the issue can be reproduced? >>>> >>>> Thanks, >>>> Mike. >>>> >>>> >>>> >>>> -- >>>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/ >>>> >>> >>> >> >> >> -- >> Best regards, >> Sergey S. Sergeev (aka SSSerj) >> Notes: https://twitter.com/ssserj >> exit(0); >> > > -- Best regards, Sergey S. Sergeev (aka SSSerj) exit(0);
