read through cache refresh
I have an ignite cache with hbase read-through. Data is fully reconstructed on a daily basis on the underlying hbase table. My question is, when the hbase table is totally changed, calling cache.clear() is enough to get the latest data? And, during processing cache.clear(), all reads to the cache will be blocked? I mean, the cache is locked? -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Heap exhaustion when querying
I am using ignite with a native persistence and loading ~60k rows into a new cache via the jdbc driver. When I attempt to run a query against the cache, I can see my heap memory usage grow (using the JMX console) until it is exhausted and the H2 database crashes. I am using DBeaver to connect and query the cache via the SQL interface. I do not have any data regions configured other than the default and here is the configuration I have for that: I do not have on-heap caching specifically enabled anywhere in my config. The server running ignite has 64GB of memory total and 1TB of space for persisted data. My understanding was that I should be using off-heap memory since I have persistence enabled but that doesn't appear to be the behavior when querying. The cache utilizes about 25GB on disk when persisted so if it is trying to pull all of that into my 10GB heap then it will obviously fail. Is someone able to help me understand what might be occurring to cause the heap memory exhaustion that I see? -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Binary type has different fields error
Hello! Yes, they are stored under work/marshaller. Should empty this dir before restarting node. Regards, -- Ilya Kasnacheev 2018-05-16 10:22 GMT+03:00 Raymond Wilson : > Thanks Pavel. > > > > I guess I’m confused that the type in question is not persisted, it is > ephemeral. Is Ignite persisting knowledge about these types behind the > scenes? > > > > Raymond. > > > > *From:* Pavel Tupitsyn [mailto:ptupit...@apache.org] > *Sent:* Wednesday, May 16, 2018 7:02 PM > *To:* user@ignite.apache.org > *Subject:* Re: Binary type has different fields error > > > > In general, Ignite is tolerant to changes within your types, adding > fields, removing them. > > But field type change is a breaking change. > > > > You have to use a new field name. > > > > Other Ignite experts may give advice on how to update schema, I'm a bit > out of the loop on this. > > > > Thanks, > > Pavel > > > > On Wed, May 16, 2018 at 7:06 AM, Raymond Wilson < > raymond_wil...@trimble.com> wrote: > > I just changed a field in a class from a long to a Guid. > > > > The class in question is marked [Serializable] and is passed to Ignite > compute functions as a part of an argument to the compute function and is > not saved to the persistent store. > > > > When I run the modified code against an Ignite grid with a persistent data > store I get the following error. Is this intentional? How should type > evolution ephemeral constructs handed to compute functions in Ignite be > handled? > > > > > > Exception: System.AggregateException: One or more errors occurred. ---> > Apache.Ignite.Core.Binary.BinaryObjectException: Binary type has > different field types [typeName=VSS.TRex.Filters.CellPassAttributeFilter, > fieldName=ElevationRangeDesignID, fieldTypeName1=long, > fieldTypeName2=UUID] ---> Apache.Ignite.Core.Common.JavaException: class > org.apache.ignite.binary.BinaryObjectException: Binary type has different > field types [typeName=VSS.TRex.Filters.CellPassAttributeFilter, > fieldName=ElevationRangeDesignID, > fieldTypeName1=long, fieldTypeName2=UUID] > > at org.apache.ignite.internal.binary.BinaryUtils. > mergeMetadata(BinaryUtils.java:1033) > > at org.apache.ignite.internal.processors.cache.binary. > CacheObjectBinaryProcessorImpl.addMeta(CacheObjectBinaryProcessorImpl > .java:444) > > at org.apache.ignite.internal.processors.cache.binary. > CacheObjectBinaryProcessorImpl$2.addMeta(CacheObjectBinaryProcessorImpl > .java:186) > > at org.apache.ignite.internal.binary.BinaryContext. > updateMetadata(BinaryContext.java:1303) > > at org.apache.ignite.internal.processors.platform. > PlatformContextImpl.processMetadata(PlatformContextImpl.java:336) > > at org.apache.ignite.internal.processors.platform.binary. > PlatformBinaryProcessor.processInStreamOutLong( > PlatformBinaryProcessor.java:70) > > at org.apache.ignite.internal.processors.platform. > PlatformAbstractTarget.processInStreamOutLong(PlatformAbstractTarget.java: > 87) > > at org.apache.ignite.internal.processors.platform. > PlatformTargetProxyImpl.inStreamOutLong(PlatformTargetProxyImpl.java:67) > > > > > >at Apache.Ignite.Core.Impl.Unmanaged.Jni.Env.ExceptionCheck() > >at Apache.Ignite.Core.Impl.Unmanaged.Jni.Env.CallLongMethod(GlobalRef > obj, IntPtr methodId, Int64* argsPtr) > >at Apache.Ignite.Core.Impl.Unmanaged.UnmanagedUtils. > TargetInStreamOutLong(GlobalRef target, Int32 opType, Int64 memPtr) > >at Apache.Ignite.Core.Impl.PlatformJniTarget.InStreamOutLong(Int32 > type, Action`1 writeAction) > >--- End of inner exception stack trace --- > > Etc….. > > > > Thanks, > > Raymond. > > >
RE: Binary type has different fields error
Thanks Pavel. I guess I’m confused that the type in question is not persisted, it is ephemeral. Is Ignite persisting knowledge about these types behind the scenes? Raymond. *From:* Pavel Tupitsyn [mailto:ptupit...@apache.org] *Sent:* Wednesday, May 16, 2018 7:02 PM *To:* user@ignite.apache.org *Subject:* Re: Binary type has different fields error In general, Ignite is tolerant to changes within your types, adding fields, removing them. But field type change is a breaking change. You have to use a new field name. Other Ignite experts may give advice on how to update schema, I'm a bit out of the loop on this. Thanks, Pavel On Wed, May 16, 2018 at 7:06 AM, Raymond Wilson wrote: I just changed a field in a class from a long to a Guid. The class in question is marked [Serializable] and is passed to Ignite compute functions as a part of an argument to the compute function and is not saved to the persistent store. When I run the modified code against an Ignite grid with a persistent data store I get the following error. Is this intentional? How should type evolution ephemeral constructs handed to compute functions in Ignite be handled? Exception: System.AggregateException: One or more errors occurred. ---> Apache.Ignite.Core.Binary.BinaryObjectException: Binary type has different field types [typeName=VSS.TRex.Filters.CellPassAttributeFilter, fieldName=ElevationRangeDesignID, fieldTypeName1=long, fieldTypeName2=UUID] ---> Apache.Ignite.Core.Common.JavaException: class org.apache.ignite.binary.BinaryObjectException: Binary type has different field types [typeName=VSS.TRex.Filters.CellPassAttributeFilter, fieldName=ElevationRangeDesignID, fieldTypeName1=long, fieldTypeName2=UUID] at org.apache.ignite.internal.binary.BinaryUtils.mergeMetadata(BinaryUtils.java:1033) at org.apache.ignite.internal.processors.cache.binary.CacheObjectBinaryProcessorImpl.addMeta(CacheObjectBinaryProcessorImpl.java:444) at org.apache.ignite.internal.processors.cache.binary.CacheObjectBinaryProcessorImpl$2.addMeta(CacheObjectBinaryProcessorImpl.java:186) at org.apache.ignite.internal.binary.BinaryContext.updateMetadata(BinaryContext.java:1303) at org.apache.ignite.internal.processors.platform.PlatformContextImpl.processMetadata(PlatformContextImpl.java:336) at org.apache.ignite.internal.processors.platform.binary.PlatformBinaryProcessor.processInStreamOutLong(PlatformBinaryProcessor.java:70) at org.apache.ignite.internal.processors.platform.PlatformAbstractTarget.processInStreamOutLong(PlatformAbstractTarget.java:87) at org.apache.ignite.internal.processors.platform.PlatformTargetProxyImpl.inStreamOutLong(PlatformTargetProxyImpl.java:67) at Apache.Ignite.Core.Impl.Unmanaged.Jni.Env.ExceptionCheck() at Apache.Ignite.Core.Impl.Unmanaged.Jni.Env.CallLongMethod(GlobalRef obj, IntPtr methodId, Int64* argsPtr) at Apache.Ignite.Core.Impl.Unmanaged.UnmanagedUtils.TargetInStreamOutLong(GlobalRef target, Int32 opType, Int64 memPtr) at Apache.Ignite.Core.Impl.PlatformJniTarget.InStreamOutLong(Int32 type, Action`1 writeAction) --- End of inner exception stack trace --- Etc….. Thanks, Raymond.
Re: Binary type has different fields error
In general, Ignite is tolerant to changes within your types, adding fields, removing them. But field type change is a breaking change. You have to use a new field name. Other Ignite experts may give advice on how to update schema, I'm a bit out of the loop on this. Thanks, Pavel On Wed, May 16, 2018 at 7:06 AM, Raymond Wilson wrote: > I just changed a field in a class from a long to a Guid. > > > > The class in question is marked [Serializable] and is passed to Ignite > compute functions as a part of an argument to the compute function and is > not saved to the persistent store. > > > > When I run the modified code against an Ignite grid with a persistent data > store I get the following error. Is this intentional? How should type > evolution ephemeral constructs handed to compute functions in Ignite be > handled? > > > > > > Exception: System.AggregateException: One or more errors occurred. ---> > Apache.Ignite.Core.Binary.BinaryObjectException: Binary type has > different field types [typeName=VSS.TRex.Filters.CellPassAttributeFilter, > fieldName=ElevationRangeDesignID, fieldTypeName1=long, > fieldTypeName2=UUID] ---> Apache.Ignite.Core.Common.JavaException: class > org.apache.ignite.binary.BinaryObjectException: Binary type has different > field types [typeName=VSS.TRex.Filters.CellPassAttributeFilter, > fieldName=ElevationRangeDesignID, > fieldTypeName1=long, fieldTypeName2=UUID] > > at org.apache.ignite.internal.binary.BinaryUtils. > mergeMetadata(BinaryUtils.java:1033) > > at org.apache.ignite.internal.processors.cache.binary. > CacheObjectBinaryProcessorImpl.addMeta(CacheObjectBinaryProcessorImpl > .java:444) > > at org.apache.ignite.internal.processors.cache.binary. > CacheObjectBinaryProcessorImpl$2.addMeta(CacheObjectBinaryProcessorImpl > .java:186) > > at org.apache.ignite.internal.binary.BinaryContext. > updateMetadata(BinaryContext.java:1303) > > at org.apache.ignite.internal.processors.platform. > PlatformContextImpl.processMetadata(PlatformContextImpl.java:336) > > at org.apache.ignite.internal.processors.platform.binary. > PlatformBinaryProcessor.processInStreamOutLong( > PlatformBinaryProcessor.java:70) > > at org.apache.ignite.internal.processors.platform. > PlatformAbstractTarget.processInStreamOutLong(PlatformAbstractTarget.java: > 87) > > at org.apache.ignite.internal.processors.platform. > PlatformTargetProxyImpl.inStreamOutLong(PlatformTargetProxyImpl.java:67) > > > > > >at Apache.Ignite.Core.Impl.Unmanaged.Jni.Env.ExceptionCheck() > >at Apache.Ignite.Core.Impl.Unmanaged.Jni.Env.CallLongMethod(GlobalRef > obj, IntPtr methodId, Int64* argsPtr) > >at Apache.Ignite.Core.Impl.Unmanaged.UnmanagedUtils. > TargetInStreamOutLong(GlobalRef target, Int32 opType, Int64 memPtr) > >at Apache.Ignite.Core.Impl.PlatformJniTarget.InStreamOutLong(Int32 > type, Action`1 writeAction) > >--- End of inner exception stack trace --- > > Etc….. > > > > Thanks, > > Raymond. >
