Re: Ignite Client Node Stopped With Out Of Memory
On of these, depending on your query type: * new ScanQuery() { PageSize = 5 } * new SqlQuery() { PageSize = 5 } On Wed, Oct 28, 2020 at 5:24 PM Ravi Makwana wrote: > Hi Paval, > > As we are not setting explicitly QueryBase.Pagesize for SqlQuery and > SqlFieldQuery so default will be used as 1024. > > We have not found so far any example by looking the same we can try to > explicitly set the Query base.Pagesize. > > Can we have any reference by checking that we can try to set the lower > value and can able to replicate the scenario? > > > Thanks, > > On Wed, 28 Oct, 2020, 7:53 pm Ravi Makwana, > wrote: > >> >> >> >> >> On Wed, 28 Oct, 2020, 5:55 pm Pavel Tupitsyn, >> wrote: >> >>> I found a bug in Ignite [1] which probably causes the issue on your side. >>> >>> Looks like you are running a query (is it a ScanQuery or SqlQuery?) and >>> the size of one results page exceeds 2GB. >>> Please try using a smaller value for *QueryBase.PageSize*. >>> >>> If you use the default value of 1024, your cache entries are very large. >>> What does the data look like? >>> >>> [1] https://issues.apache.org/jira/browse/IGNITE-13635 >>> >>> On Wed, Oct 28, 2020 at 1:43 PM Ravi Makwana >>> wrote: >>> Hi, Our service is running with 64 bit and we have verified the same in our app server too. Any finding from the logs? Is there any way to replicate it? Thanks, On Wed, 28 Oct 2020 at 15:47, Pavel Tupitsyn wrote: > Looks like the app is running in 32 bit mode, which can't use more > than 2GB of memory. > JVM and memory regions pre-allocate all of it, leaving nothing for > .NET to use. > > Please check the `Platform` column in the Task Manager - does it say > `32 bit`? > If yes, then try disabling `Prefer 32 bit` in the project properties. > > On Wed, Oct 28, 2020 at 1:06 PM Ravi Makwana < > ravi.makw...@hotelhub.com> wrote: > >> Hi Pavel, >> >> Thanks for the information. >> >> We have noticed our service is taking max 2GB memory which is noticed >> in task manager and stopping app pool at same time server memory >> utilization is 80% so we have still memory spare 20%. >> >> Still we are not able to correlate the issue, I would like to share >> some more logs. Could you suggest what is happening at the moment? >> >> Thanks, >> >>
Re: Ignite Client Node Stopped With Out Of Memory
Hi Paval, As we are not setting explicitly QueryBase.Pagesize for SqlQuery and SqlFieldQuery so default will be used as 1024. We have not found so far any example by looking the same we can try to explicitly set the Query base.Pagesize. Can we have any reference by checking that we can try to set the lower value and can able to replicate the scenario? Thanks, On Wed, 28 Oct, 2020, 7:53 pm Ravi Makwana, wrote: > > > > > On Wed, 28 Oct, 2020, 5:55 pm Pavel Tupitsyn, > wrote: > >> I found a bug in Ignite [1] which probably causes the issue on your side. >> >> Looks like you are running a query (is it a ScanQuery or SqlQuery?) and >> the size of one results page exceeds 2GB. >> Please try using a smaller value for *QueryBase.PageSize*. >> >> If you use the default value of 1024, your cache entries are very large. >> What does the data look like? >> >> [1] https://issues.apache.org/jira/browse/IGNITE-13635 >> >> On Wed, Oct 28, 2020 at 1:43 PM Ravi Makwana >> wrote: >> >>> Hi, >>> >>> Our service is running with 64 bit and we have verified the same in our >>> app server too. >>> >>> Any finding from the logs? >>> >>> Is there any way to replicate it? >>> >>> Thanks, >>> >>> >>> On Wed, 28 Oct 2020 at 15:47, Pavel Tupitsyn >>> wrote: >>> Looks like the app is running in 32 bit mode, which can't use more than 2GB of memory. JVM and memory regions pre-allocate all of it, leaving nothing for .NET to use. Please check the `Platform` column in the Task Manager - does it say `32 bit`? If yes, then try disabling `Prefer 32 bit` in the project properties. On Wed, Oct 28, 2020 at 1:06 PM Ravi Makwana wrote: > Hi Pavel, > > Thanks for the information. > > We have noticed our service is taking max 2GB memory which is noticed > in task manager and stopping app pool at same time server memory > utilization is 80% so we have still memory spare 20%. > > Still we are not able to correlate the issue, I would like to share > some more logs. Could you suggest what is happening at the moment? > > Thanks, > >
Re: Ignite Client Node Stopped With Out Of Memory
On Wed, 28 Oct, 2020, 5:55 pm Pavel Tupitsyn, wrote: > I found a bug in Ignite [1] which probably causes the issue on your side. > > Looks like you are running a query (is it a ScanQuery or SqlQuery?) and > the size of one results page exceeds 2GB. > Please try using a smaller value for *QueryBase.PageSize*. > > If you use the default value of 1024, your cache entries are very large. > What does the data look like? > > [1] https://issues.apache.org/jira/browse/IGNITE-13635 > > On Wed, Oct 28, 2020 at 1:43 PM Ravi Makwana > wrote: > >> Hi, >> >> Our service is running with 64 bit and we have verified the same in our >> app server too. >> >> Any finding from the logs? >> >> Is there any way to replicate it? >> >> Thanks, >> >> >> On Wed, 28 Oct 2020 at 15:47, Pavel Tupitsyn >> wrote: >> >>> Looks like the app is running in 32 bit mode, which can't use more than >>> 2GB of memory. >>> JVM and memory regions pre-allocate all of it, leaving nothing for .NET >>> to use. >>> >>> Please check the `Platform` column in the Task Manager - does it say `32 >>> bit`? >>> If yes, then try disabling `Prefer 32 bit` in the project properties. >>> >>> On Wed, Oct 28, 2020 at 1:06 PM Ravi Makwana >>> wrote: >>> Hi Pavel, Thanks for the information. We have noticed our service is taking max 2GB memory which is noticed in task manager and stopping app pool at same time server memory utilization is 80% so we have still memory spare 20%. Still we are not able to correlate the issue, I would like to share some more logs. Could you suggest what is happening at the moment? Thanks,
Re: Ignite Client Node Stopped With Out Of Memory
I found a bug in Ignite [1] which probably causes the issue on your side. Looks like you are running a query (is it a ScanQuery or SqlQuery?) and the size of one results page exceeds 2GB. Please try using a smaller value for *QueryBase.PageSize*. If you use the default value of 1024, your cache entries are very large. What does the data look like? [1] https://issues.apache.org/jira/browse/IGNITE-13635 On Wed, Oct 28, 2020 at 1:43 PM Ravi Makwana wrote: > Hi, > > Our service is running with 64 bit and we have verified the same in our > app server too. > > Any finding from the logs? > > Is there any way to replicate it? > > Thanks, > > > On Wed, 28 Oct 2020 at 15:47, Pavel Tupitsyn wrote: > >> Looks like the app is running in 32 bit mode, which can't use more than >> 2GB of memory. >> JVM and memory regions pre-allocate all of it, leaving nothing for .NET >> to use. >> >> Please check the `Platform` column in the Task Manager - does it say `32 >> bit`? >> If yes, then try disabling `Prefer 32 bit` in the project properties. >> >> On Wed, Oct 28, 2020 at 1:06 PM Ravi Makwana >> wrote: >> >>> Hi Pavel, >>> >>> Thanks for the information. >>> >>> We have noticed our service is taking max 2GB memory which is noticed in >>> task manager and stopping app pool at same time server memory utilization >>> is 80% so we have still memory spare 20%. >>> >>> Still we are not able to correlate the issue, I would like to share some >>> more logs. Could you suggest what is happening at the moment? >>> >>> Thanks, >>> >>>
Re: Ignite Client Node Stopped With Out Of Memory
Hi, Our service is running with 64 bit and we have verified the same in our app server too. Any finding from the logs? Is there any way to replicate it? Thanks, On Wed, 28 Oct 2020 at 15:47, Pavel Tupitsyn wrote: > Looks like the app is running in 32 bit mode, which can't use more than > 2GB of memory. > JVM and memory regions pre-allocate all of it, leaving nothing for .NET to > use. > > Please check the `Platform` column in the Task Manager - does it say `32 > bit`? > If yes, then try disabling `Prefer 32 bit` in the project properties. > > On Wed, Oct 28, 2020 at 1:06 PM Ravi Makwana > wrote: > >> Hi Pavel, >> >> Thanks for the information. >> >> We have noticed our service is taking max 2GB memory which is noticed in >> task manager and stopping app pool at same time server memory utilization >> is 80% so we have still memory spare 20%. >> >> Still we are not able to correlate the issue, I would like to share some >> more logs. Could you suggest what is happening at the moment? >> >> Thanks, >> >>
Re: Ignite Client Node Stopped With Out Of Memory
Looks like the app is running in 32 bit mode, which can't use more than 2GB of memory. JVM and memory regions pre-allocate all of it, leaving nothing for .NET to use. Please check the `Platform` column in the Task Manager - does it say `32 bit`? If yes, then try disabling `Prefer 32 bit` in the project properties. On Wed, Oct 28, 2020 at 1:06 PM Ravi Makwana wrote: > Hi Pavel, > > Thanks for the information. > > We have noticed our service is taking max 2GB memory which is noticed in > task manager and stopping app pool at same time server memory utilization > is 80% so we have still memory spare 20%. > > Still we are not able to correlate the issue, I would like to share some > more logs. Could you suggest what is happening at the moment? > > Thanks, > >
Re: Ignite Client Node Stopped With Out Of Memory
Hi Ravi, The exception indicates that Ignite.NET failed to allocate unmanaged memory on .NET side while trying to pass query data from Java to .NET. This indicates that your system has run out of memory. Possible reasons are: * Memory is consumed by other apps * Memory is consumed by this app * Memory leak in your code * Memory leak in Ignite.NET code I would suggest the following steps: * Find out which app consumes the memory. Are we sure it is the app with Ignite? * If yes, run a memory profiler of your choice to figure out what consumes the memory Thanks, Pavel On Sat, Oct 24, 2020 at 2:04 PM Ravi Makwana wrote: > Hi, > > We are using Apache Ignite 2.7.0 binary and servers are using Linux OS & > app servers are using Windows OS.We are using Apache Ignite .Net APIs. > > Recently we have noticed that our application is stopping due to a client > node throwing Out Of Memory error which we have seen in ignite > client node log file. > > App server has 32 GB RAM & We are specifying JVM Heap = 8 GB & Data Region > Min = 1 GB & Max = 10 GB. > > *Exception:* > > class org.apache.ignite.IgniteException: Platform > error:System.OutOfMemoryException: Insufficient memory to continue the > execution of the program. >at System.Runtime.InteropServices.Marshal.ReAllocHGlobal(IntPtr pv, > IntPtr cb) >at > Apache.Ignite.Core.Impl.Memory.PlatformMemoryUtils.ReallocatePooled(Int64 > memPtr, Int32 cap) >at > Apache.Ignite.Core.Impl.Unmanaged.UnmanagedCallbacks.MemoryReallocate(Int64 > memPtr, Int64 cap, Int64 unused, Void* arg) >at > Apache.Ignite.Core.Impl.Unmanaged.UnmanagedCallbacks.InLongLongLongObjectOutLong(Int32 > type, Int64 val1, Int64 val2, Int64 val3, IntPtr arg) > at > org.apache.ignite.internal.processors.platform.PlatformProcessorImpl.loggerLog(PlatformProcessorImpl.java:404) > at > org.apache.ignite.internal.processors.platform.PlatformProcessorImpl.processInStreamOutLong(PlatformProcessorImpl.java:460) > at > org.apache.ignite.internal.processors.platform.PlatformProcessorImpl.processInStreamOutLong(PlatformProcessorImpl.java:512) > at > org.apache.ignite.internal.processors.platform.PlatformTargetProxyImpl.inStreamOutLong(PlatformTargetProxyImpl.java:67) > at > org.apache.ignite.internal.processors.platform.callback.PlatformCallbackUtils.inLongLongLongObjectOutLong(Native > Method) > at > org.apache.ignite.internal.processors.platform.callback.PlatformCallbackGateway.memoryReallocate(PlatformCallbackGateway.java:840) > at > org.apache.ignite.internal.processors.platform.memory.PlatformExternalMemory.reallocate(PlatformExternalMemory.java:48) > at > org.apache.ignite.internal.processors.platform.memory.PlatformOutputStreamImpl.ensureCapacity(PlatformOutputStreamImpl.java:305) > at > org.apache.ignite.internal.processors.platform.memory.PlatformOutputStreamImpl.unsafeEnsure(PlatformOutputStreamImpl.java:222) > at > org.apache.ignite.internal.binary.BinaryWriterExImpl.doWriteBinaryObject(BinaryWriterExImpl.java:962) > at > org.apache.ignite.internal.binary.BinaryClassDescriptor.write(BinaryClassDescriptor.java:744) > at > org.apache.ignite.internal.binary.BinaryWriterExImpl.marshal0(BinaryWriterExImpl.java:223) > at > org.apache.ignite.internal.binary.BinaryWriterExImpl.marshal(BinaryWriterExImpl.java:164) > at > org.apache.ignite.internal.binary.BinaryWriterExImpl.marshal(BinaryWriterExImpl.java:151) > at > org.apache.ignite.internal.binary.BinaryWriterExImpl.writeObjectDetached(BinaryWriterExImpl.java:1506) > at > org.apache.ignite.internal.processors.platform.cache.query.PlatformQueryCursor.write(PlatformQueryCursor.java:43) > at > org.apache.ignite.internal.processors.platform.cache.query.PlatformQueryCursor.write(PlatformQueryCursor.java:28) > at > org.apache.ignite.internal.processors.platform.cache.query.PlatformAbstractQueryCursor.processOutStream(PlatformAbstractQueryCursor.java:85) > at > org.apache.ignite.internal.processors.platform.PlatformTargetProxyImpl.outStream(PlatformTargetProxyImpl.java:93) > > Can please suggest what is the root cause for the above issue? > > Thanks & Regards, > >
Ignite Client Node Stopped With Out Of Memory
Hi, We are using Apache Ignite 2.7.0 binary and servers are using Linux OS & app servers are using Windows OS.We are using Apache Ignite .Net APIs. Recently we have noticed that our application is stopping due to a client node throwing Out Of Memory error which we have seen in ignite client node log file. App server has 32 GB RAM & We are specifying JVM Heap = 8 GB & Data Region Min = 1 GB & Max = 10 GB. *Exception:* class org.apache.ignite.IgniteException: Platform error:System.OutOfMemoryException: Insufficient memory to continue the execution of the program. at System.Runtime.InteropServices.Marshal.ReAllocHGlobal(IntPtr pv, IntPtr cb) at Apache.Ignite.Core.Impl.Memory.PlatformMemoryUtils.ReallocatePooled(Int64 memPtr, Int32 cap) at Apache.Ignite.Core.Impl.Unmanaged.UnmanagedCallbacks.MemoryReallocate(Int64 memPtr, Int64 cap, Int64 unused, Void* arg) at Apache.Ignite.Core.Impl.Unmanaged.UnmanagedCallbacks.InLongLongLongObjectOutLong(Int32 type, Int64 val1, Int64 val2, Int64 val3, IntPtr arg) at org.apache.ignite.internal.processors.platform.PlatformProcessorImpl.loggerLog(PlatformProcessorImpl.java:404) at org.apache.ignite.internal.processors.platform.PlatformProcessorImpl.processInStreamOutLong(PlatformProcessorImpl.java:460) at org.apache.ignite.internal.processors.platform.PlatformProcessorImpl.processInStreamOutLong(PlatformProcessorImpl.java:512) at org.apache.ignite.internal.processors.platform.PlatformTargetProxyImpl.inStreamOutLong(PlatformTargetProxyImpl.java:67) at org.apache.ignite.internal.processors.platform.callback.PlatformCallbackUtils.inLongLongLongObjectOutLong(Native Method) at org.apache.ignite.internal.processors.platform.callback.PlatformCallbackGateway.memoryReallocate(PlatformCallbackGateway.java:840) at org.apache.ignite.internal.processors.platform.memory.PlatformExternalMemory.reallocate(PlatformExternalMemory.java:48) at org.apache.ignite.internal.processors.platform.memory.PlatformOutputStreamImpl.ensureCapacity(PlatformOutputStreamImpl.java:305) at org.apache.ignite.internal.processors.platform.memory.PlatformOutputStreamImpl.unsafeEnsure(PlatformOutputStreamImpl.java:222) at org.apache.ignite.internal.binary.BinaryWriterExImpl.doWriteBinaryObject(BinaryWriterExImpl.java:962) at org.apache.ignite.internal.binary.BinaryClassDescriptor.write(BinaryClassDescriptor.java:744) at org.apache.ignite.internal.binary.BinaryWriterExImpl.marshal0(BinaryWriterExImpl.java:223) at org.apache.ignite.internal.binary.BinaryWriterExImpl.marshal(BinaryWriterExImpl.java:164) at org.apache.ignite.internal.binary.BinaryWriterExImpl.marshal(BinaryWriterExImpl.java:151) at org.apache.ignite.internal.binary.BinaryWriterExImpl.writeObjectDetached(BinaryWriterExImpl.java:1506) at org.apache.ignite.internal.processors.platform.cache.query.PlatformQueryCursor.write(PlatformQueryCursor.java:43) at org.apache.ignite.internal.processors.platform.cache.query.PlatformQueryCursor.write(PlatformQueryCursor.java:28) at org.apache.ignite.internal.processors.platform.cache.query.PlatformAbstractQueryCursor.processOutStream(PlatformAbstractQueryCursor.java:85) at org.apache.ignite.internal.processors.platform.PlatformTargetProxyImpl.outStream(PlatformTargetProxyImpl.java:93) Can please suggest what is the root cause for the above issue? Thanks & Regards,