Good day. Thanks again for all the feedback.
I hope the bug will get sorted out. For now I have removed the eviction policies and there error is no more after a restart. I assume that if one uses persistent regions, then the eviction+overflow is not that critical as the data will be "backed" in the store/disk. One just need enough memory. Eviction+Overflow I suspect is quite critical when one has a full in-memory grid and running out of memory could cause issues if there is no overflow to disk? I am thinking that for now I could look at *expiration* rather on the region? To keep only *relevant* data in the in-memory regions for now to prevent running out of memory. Will try and keep data in memory for as long as possible. Currently we cannot remove the transactions that we use with the persistent regions. We might in the future. Kindly Pieter On Thu, May 3, 2018 at 1:16 AM, Dan Smith <[email protected]> wrote: > > I assume this will happen on partitioned regions as well as the issue is > the combination of transactions on persistent regions and overflow. > > Unfortunately yes, this bug also affects partitioned regions > > > Also I see this bug is marked as *major* so is there any chance this > will be fixed in the next couple of months? > > I'm not sure. Geode is an open source project, so we don't really promise > fixes in any specific timeframe. > > > If I do change the region to not use overflow what will happen when it > reaches the "heap percentage"? > > The data will stay in memory. Oveflow lets you avoid running out of memory > by overflowing data to disk. Without that you could end up running out of > memory if your region gets to large. > > -Dan > > On Wed, May 2, 2018 at 2:07 PM, Pieter van Zyl <[email protected]> > wrote: > >> Hi Dan, >> >> Thanks for tracking this down! >> >> Much appreciated. >> >> This might also be why I didn't see it at first as we didn't activate the >> transactions on the persistent regions when we started with this evaluation. >> >> Based on this discussion >> >> https://markmail.org/message/jsabcdvyzsdrkvba?q=list:org%2Ea >> pache%2Egeode%2Euser+order:date-backward+pieter#query:list% >> 3Aorg.apache.geode.user%20order%3Adate-backward%20pieter+ >> page:1+mid:n25nznu7zur4xmar+state:results >> >> We are currently using -Dgemfire.ALLOW_PERSISTENT_TRANSACTIONS=true >> >> Once we have the basics up and running we will still look at the >> TransactionWriter as recommended. >> >> We are currently trying to import our old data from Berkeley into Geode >> and for now I have one node locally with a replicated region. >> But we are planning to move to more nodes and partition/sharded regions. >> >> I assume this will happen on partitioned regions as well as the issue is >> the combination of transactions on persistent regions and overflow. >> >> Also I see this bug is marked as *major* so is there any chance this >> will be fixed in the next couple of months? >> Or is our use of transactions across persistent regions just to out of >> the norm? >> >> If I do change the region to not use overflow what will happen when it >> reaches the "heap percentage"? >> >> Kindly >> Pieter >> >> >> >> On Wed, May 2, 2018 at 10:14 PM, Dan Smith <[email protected]> wrote: >> >>> I created GEODE-5173 for this issue. >>> >>> Thanks, >>> -Dan >>> >>> On Wed, May 2, 2018 at 12:17 PM, Dan Smith <[email protected]> wrote: >>> >>>> Hi Pieter, >>>> >>>> I was able to reproduce this problem. It looks like it is an issue with >>>> doing a get inside of a transaction along with a replicated region using >>>> persistence and overflow. The value is still on disk, and for whatever >>>> reason if you do the get inside of a transaction it is returning you this >>>> bogus NOT_AVAILABLE token instead of reading the value off disk. >>>> >>>> I'll create a JIRA and attach my test. In the meantime, you could do >>>> the get outside of a transaction, or you could change your region to not >>>> use overflow. If you try changing the region to not use overflow, I think >>>> you'll also have to set the system property gemfire.disk.recoverValuesSync >>>> to true to make sure that in all cases you never have to read from disk. >>>> >>>> Thanks, >>>> -Dan >>>> >>>> On Mon, Apr 30, 2018 at 3:47 AM, Pieter van Zyl < >>>> [email protected]> wrote: >>>> >>>>> Good day. >>>>> >>>>> I am constantly seeing this error below when we stop and start Geode >>>>> server after a data import. >>>>> >>>>> When the client connects the second time after the restart we get >>>>> NotSerializableException: >>>>> org.apache.geode.internal.cache.Token$NotAvailable >>>>> >>>>> Any ideas why we are getting this error or why it would state >>>>> "NotAvailable"? >>>>> >>>>> *Versions:* >>>>> >>>>> compile 'org.springframework.data:spring-data-geode:2.1.0.M2' >>>>> compile group: 'org.apache.geode', name: 'geode-core', version: '1.5.0' >>>>> >>>>> Trying to access this region on startup: >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> *<gfe:replicated-region id="ClassID-ClassName-LookUp" >>>>> disk-store-ref="tauDiskStore" >>>>> persistent="true"> <gfe:eviction type="HEAP_PERCENTAGE" >>>>> action="OVERFLOW_TO_DISK"/></gfe:replicated-region>* >>>>> >>>>> *Server config:* >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> *<util:properties id="gemfire-props"><prop >>>>> key="log-level">info</prop><prop >>>>> key="locators">pvz-dell[10334]</prop><prop >>>>> key="start-locator">pvz-dell[10334]</prop><prop >>>>> key="mcast-port">0</prop><prop key="http-service-port">0</prop><prop >>>>> key="jmx-manager">true</prop><prop key="jmx-manager-port">1099</prop><prop >>>>> key="jmx-manager-start">true</prop></util:properties>* >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> *<gfe:cache properties-ref="gemfire-props" >>>>> pdx-serializer-ref="pdxSerializer" >>>>> pdx-persistent="true"pdx-disk-store="pdx-disk-store" /><gfe:cache-server >>>>> port="40404" max-connections="300" socket-buffer-size="65536" >>>>> max-threads="200"/><gfe:transaction-manager id="txManager"/><bean >>>>> id="pdxSerializer" >>>>> class="org.rdb.geode.mapping.RDBGeodeSerializer"> <constructor-arg >>>>> value="org.rdb.*,net.lautus.*"/></bean>* >>>>> >>>>> The server seems to be up and running >>>>> *Cache server connection listener bound to address >>>>> pvz-dell-/0:0:0:0:0:0:0:0:40404 with backlog 1,000.* >>>>> >>>>> *[info 2018/04/30 12:32:30.483 SAST <main> tid=0x1] >>>>> ClientHealthMonitorThread maximum allowed time between pings: 60,000* >>>>> >>>>> *[warn 2018/04/30 12:32:30.485 SAST <main> tid=0x1] Handshaker max >>>>> Pool size: 4* >>>>> >>>>> *[info 2018/04/30 12:32:30.486 SAST <Cache Server Selector >>>>> /0:0:0:0:0:0:0:0:40404 local port: 40404> tid=0x4f] SELECTOR enabled* >>>>> >>>>> *[info 2018/04/30 12:32:30.491 SAST <main> tid=0x1] CacheServer >>>>> Configuration: port=40404 max-connections=300 max-threads=200 >>>>> notify-by-subscription=true socket-buffer-size=65536 >>>>> maximum-time-between-pings=60000 maximum-message-count=230000 >>>>> message-time-to-live=180 eviction-policy=none capacity=1 overflow >>>>> directory=. groups=[] loadProbe=ConnectionCountProbe loadPollInterval=5000 >>>>> tcpNoDelay=true* >>>>> >>>>> *server running on port 40404* >>>>> *Press <Enter> to terminate the server* >>>>> >>>>> >>>>> Exception in thread "main" >>>>> org.apache.geode.cache.client.ServerOperationException: >>>>> remote server on pvz-dell(23128:loner):38042:2edf1c16: >>>>> org.apache.geode.SerializationException: failed serializing object >>>>> at org.apache.geode.cache.client.internal.OpExecutorImpl.handle >>>>> Exception(OpExecutorImpl.java:669) >>>>> at org.apache.geode.cache.client.internal.OpExecutorImpl.handle >>>>> Exception(OpExecutorImpl.java:742) >>>>> at org.apache.geode.cache.client.internal.OpExecutorImpl.handle >>>>> Exception(OpExecutorImpl.java:611) >>>>> at org.apache.geode.cache.client.internal.OpExecutorImpl.execut >>>>> eOnServer(OpExecutorImpl.java:373) >>>>> at org.apache.geode.cache.client.internal.OpExecutorImpl.execut >>>>> eWithServerAffinity(OpExecutorImpl.java:220) >>>>> at org.apache.geode.cache.client.internal.OpExecutorImpl.execut >>>>> e(OpExecutorImpl.java:129) >>>>> at org.apache.geode.cache.client.internal.OpExecutorImpl.execut >>>>> e(OpExecutorImpl.java:116) >>>>> at org.apache.geode.cache.client.internal.PoolImpl.execute(Pool >>>>> Impl.java:774) >>>>> at org.apache.geode.cache.client.internal.GetOp.execute(GetOp.java:91) >>>>> at org.apache.geode.cache.client.internal.ServerRegionProxy.get >>>>> (ServerRegionProxy.java:113) >>>>> at org.apache.geode.internal.cache.tx.ClientTXRegionStub.findOb >>>>> ject(ClientTXRegionStub.java:72) >>>>> at org.apache.geode.internal.cache.TXStateStub.findObject(TXSta >>>>> teStub.java:453) >>>>> at org.apache.geode.internal.cache.TXStateProxyImpl.findObject( >>>>> TXStateProxyImpl.java:496) >>>>> at org.apache.geode.internal.cache.LocalRegion.get(LocalRegion. >>>>> java:1366) >>>>> at org.apache.geode.internal.cache.LocalRegion.get(LocalRegion. >>>>> java:1300) >>>>> at org.apache.geode.internal.cache.LocalRegion.get(LocalRegion. >>>>> java:1285) >>>>> at org.apache.geode.internal.cache.AbstractRegion.get(AbstractR >>>>> egion.java:320) >>>>> ...... >>>>> Caused by: org.apache.geode.SerializationException: failed >>>>> serializing object >>>>> at org.apache.geode.internal.cache.tier.sockets.Message.seriali >>>>> zeAndAddPart(Message.java:399) >>>>> at org.apache.geode.internal.cache.tier.sockets.Message.addPart >>>>> InAnyForm(Message.java:360) >>>>> at org.apache.geode.internal.cache.tier.sockets.command.Get70.w >>>>> riteResponse(Get70.java:424) >>>>> at org.apache.geode.internal.cache.tier.sockets.command.Get70.c >>>>> mdExecute(Get70.java:211) >>>>> at org.apache.geode.internal.cache.tier.sockets.BaseCommand.exe >>>>> cute(BaseCommand.java:157) >>>>> at org.apache.geode.internal.cache.tier.sockets.ServerConnectio >>>>> n.doNormalMsg(ServerConnection.java:797) >>>>> at org.apache.geode.internal.cache.tier.sockets.LegacyServerCon >>>>> nection.doOneMessage(LegacyServerConnection.java:85) >>>>> at org.apache.geode.internal.cache.tier.sockets.ServerConnectio >>>>> n.run(ServerConnection.java:1148) >>>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool >>>>> Executor.java:1149) >>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo >>>>> lExecutor.java:624) >>>>> at org.apache.geode.internal.cache.tier.sockets.AcceptorImpl$4$ >>>>> 1.run(AcceptorImpl.java:641) >>>>> at java.lang.Thread.run(Thread.java:748) >>>>> *Caused by: java.io <http://java.io>.NotSerializableException: >>>>> org.apache.geode.internal.cache.Token$NotAvailable* >>>>> at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.j >>>>> ava:1184) >>>>> at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348) >>>>> at org.apache.geode.internal.InternalDataSerializer.writeSerial >>>>> izableObject(InternalDataSerializer.java:2341) >>>>> at org.apache.geode.internal.InternalDataSerializer.basicWriteO >>>>> bject(InternalDataSerializer.java:2216) >>>>> at org.apache.geode.DataSerializer.writeObject(DataSerializer.j >>>>> ava:2936) >>>>> at org.apache.geode.internal.util.BlobHelper.serializeTo(BlobHe >>>>> lper.java:66) >>>>> at org.apache.geode.internal.cache.tier.sockets.Message.seriali >>>>> zeAndAddPart(Message.java:397) >>>>> >>>>> >>>>> Kindly >>>>> Pieter >>>>> >>>> >>>> >>> >> >
