Re: removing ControlCenterAgent

[Bastien Durel]

I forget to attach my configuration (I removed the cache config details)I'm 
using the debian package, so I run the cluster with xml
configuration.

Regards,


-- 
Bastien Durel
DATA
Intégration des données de l'entreprise,
Systèmes d'information décisionnels.

bastien.du...@data.fr
tel : +33 (0) 1 57 19 59 28
fax : +33 (0) 1 57 19 59 73
45 avenue Carnot, 94230 CACHAN France
www.data.fr



ignite.xml
Description: XML document

removing ControlCenterAgent

[Bastien Durel]

Hello,

I'm running a 2.9.0 cluster with 2 nodes. I tried to use grid grain's
ControlCenterAgent to investigate a slowdown.

When I removed the agent files from server (I don't like to have to put
it in all clients), the second node cannot join the cluster when I
start it.

If I start node A, then node B, node B fails, but if I start node B,
then node A, node A fails.

If I put the agent files back, then all nodes can start, but clients
fail because they don't have the agent classes themselves.

When a node fails to start, it prints this log :


[17:52:45,265][INFO][tcp-disco-sock-reader-[2f3f6f3a 
192.168.43.29:39675]-#6%ClusterWA%-#50%ClusterWA%][TcpDiscoverySpi] Initialized 
connection with remote server node 
[nodeId=2f3f6f3a-accb-4708-a5cc-26d324a07816, rmtAddr=/192.168.43.29:39675]
[17:52:45,268][SEVERE][main][IgniteKernal%ClusterWA] Failed to start manager: 
GridManagerAdapter [enabled=true, 
name=o.a.i.i.managers.discovery.GridDiscoveryManager]
class org.apache.ignite.IgniteCheckedException: Failed to start SPI: 
TcpDiscoverySpi [addrRslvr=null, sockTimeout=5000, ackTimeout=5000, 
marsh=JdkMarshaller 
[clsFilter=org.apache.ignite.marshaller.MarshallerUtils$1@39a8e2fa], 
reconCnt=10, reconDelay=2000, maxAckTimeout=60, soLinger=5, 
forceSrvMode=false, clientReconnectDisabled=false, internalLsnr=null, 
skipAddrsRandomization=false]
at 
org.apache.ignite.internal.managers.GridManagerAdapter.startSpi(GridManagerAdapter.java:302)
at 
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.start(GridDiscoveryManager.java:967)
at 
org.apache.ignite.internal.IgniteKernal.startManager(IgniteKernal.java:1935)
at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1298)
at 
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:2046)
at 
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1698)
at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1114)
at 
org.apache.ignite.internal.IgnitionEx.startConfigurations(IgnitionEx.java:1032)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:918)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:817)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:687)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:656)
at org.apache.ignite.Ignition.start(Ignition.java:353)
at 
org.apache.ignite.startup.cmdline.CommandLineStartup.main(CommandLineStartup.java:300)
Caused by: class org.apache.ignite.spi.IgniteSpiException: Unable to unmarshal 
key=metastorage.cluster.id.tag
at 
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.checkFailedError(TcpDiscoverySpi.java:2018)
at 
org.apache.ignite.spi.discovery.tcp.ServerImpl.joinTopology(ServerImpl.java:1189)
at 
org.apache.ignite.spi.discovery.tcp.ServerImpl.spiStart(ServerImpl.java:462)
at 
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.spiStart(TcpDiscoverySpi.java:2120)
at 
org.apache.ignite.internal.managers.GridManagerAdapter.startSpi(GridManagerAdapter.java:299)
... 13 more
[17:52:45,271][SEVERE][main][IgniteKernal%ClusterWA] Got exception while 
starting (will rollback startup routine).
class org.apache.ignite.IgniteCheckedException: Failed to start manager: 
GridManagerAdapter [enabled=true, 
name=org.apache.ignite.internal.managers.discovery.GridDiscoveryManager]
at 
org.apache.ignite.internal.IgniteKernal.startManager(IgniteKernal.java:1940)
at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1298)
at 
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:2046)
at 
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1698)
at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1114)
at 
org.apache.ignite.internal.IgnitionEx.startConfigurations(IgnitionEx.java:1032)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:918)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:817)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:687)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:656)
at org.apache.ignite.Ignition.start(Ignition.java:353)
at 
org.apache.ignite.startup.cmdline.CommandLineStartup.main(CommandLineStartup.java:300)
Caused by: class org.apache.ignite.IgniteCheckedException: Failed to start SPI: 
TcpDiscoverySpi [addrRslvr=null, sockTimeout=5000, ackTimeout=5000, 
marsh=JdkMarshaller 
[clsFilter=org.apache.ignite.marshaller.MarshallerUtils$1@39a8e2fa], 
reconCnt=10, reconDelay=2000, maxAckTimeout=60, soLinger=5, 
forceSrvMode=false, clientReconnectDisabled=false, internalLsnr=null, 
skipAddrsRandomization=false]
at 

Large Heap with lots of BinaryMetaDataHolders

[ssansoy]

Hi, could anyone please help understand why the heap of a client app has such
large amounts of data pertaining to binary meta data?

Here it takes up 30mb but in our UAT environment we have approx 50 caches.
The binary meta data that gets added to the client's heap equats to around
220mb (even for a very simple app that doesn't do any subscriptions - it
just calls Igition.start() to connect to the cluster)

It seems meta is kept on the client for every cache even if the client app
needs it or not. Is there any way to tune this at all - e.g. knowing that a
particular client is only interested in a particular cache?

Screenshot:

 

Thanks



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Ignite Client Node Stopped With Out Of Memory

[Pavel Tupitsyn]

On of these, depending on your query type:
* new ScanQuery() { PageSize = 5 }
* new SqlQuery() { PageSize = 5 }



On Wed, Oct 28, 2020 at 5:24 PM Ravi Makwana 
wrote:

> Hi Paval,
>
> As we are not setting explicitly QueryBase.Pagesize for SqlQuery and
> SqlFieldQuery so default will be used as 1024.
>
> We have not found so far any example by looking the same we can try to
> explicitly set the Query base.Pagesize.
>
> Can we have any reference by checking that we can try to set the lower
> value and can able to replicate the scenario?
>
>
> Thanks,
>
> On Wed, 28 Oct, 2020, 7:53 pm Ravi Makwana, 
> wrote:
>
>>
>>
>>
>>
>> On Wed, 28 Oct, 2020, 5:55 pm Pavel Tupitsyn, 
>> wrote:
>>
>>> I found a bug in Ignite [1] which probably causes the issue on your side.
>>>
>>> Looks like you are running a query (is it a ScanQuery or SqlQuery?) and
>>> the size of one results page exceeds 2GB.
>>> Please try using a smaller value for *QueryBase.PageSize*.
>>>
>>> If you use the default value of 1024, your cache entries are very large.
>>> What does the data look like?
>>>
>>> [1] https://issues.apache.org/jira/browse/IGNITE-13635
>>>
>>> On Wed, Oct 28, 2020 at 1:43 PM Ravi Makwana 
>>> wrote:
>>>
 Hi,

 Our service is running with 64 bit and we have verified the same in our
 app server too.

 Any finding from the logs?

 Is there any way to replicate it?

 Thanks,


 On Wed, 28 Oct 2020 at 15:47, Pavel Tupitsyn 
 wrote:

> Looks like the app is running in 32 bit mode, which can't use more
> than 2GB of memory.
> JVM and memory regions pre-allocate all of it, leaving nothing for
> .NET to use.
>
> Please check the `Platform` column in the Task Manager - does it say
> `32 bit`?
> If yes, then try disabling `Prefer 32 bit` in the project properties.
>
> On Wed, Oct 28, 2020 at 1:06 PM Ravi Makwana <
> ravi.makw...@hotelhub.com> wrote:
>
>> Hi Pavel,
>>
>> Thanks for the information.
>>
>> We have noticed our service is taking max 2GB memory which is noticed
>> in task manager and stopping app pool at same time server memory
>> utilization is 80% so we have still memory spare 20%.
>>
>> Still we are not able to correlate the issue, I would like to share
>> some more logs. Could you suggest what is happening at the moment?
>>
>> Thanks,
>>
>>

Re: Ignite Client Node Stopped With Out Of Memory

[Ravi Makwana]

Hi Paval,

As we are not setting explicitly QueryBase.Pagesize for SqlQuery and
SqlFieldQuery so default will be used as 1024.

We have not found so far any example by looking the same we can try to
explicitly set the Query base.Pagesize.

Can we have any reference by checking that we can try to set the lower
value and can able to replicate the scenario?


Thanks,

On Wed, 28 Oct, 2020, 7:53 pm Ravi Makwana, 
wrote:

>
>
>
>
> On Wed, 28 Oct, 2020, 5:55 pm Pavel Tupitsyn, 
> wrote:
>
>> I found a bug in Ignite [1] which probably causes the issue on your side.
>>
>> Looks like you are running a query (is it a ScanQuery or SqlQuery?) and
>> the size of one results page exceeds 2GB.
>> Please try using a smaller value for *QueryBase.PageSize*.
>>
>> If you use the default value of 1024, your cache entries are very large.
>> What does the data look like?
>>
>> [1] https://issues.apache.org/jira/browse/IGNITE-13635
>>
>> On Wed, Oct 28, 2020 at 1:43 PM Ravi Makwana 
>> wrote:
>>
>>> Hi,
>>>
>>> Our service is running with 64 bit and we have verified the same in our
>>> app server too.
>>>
>>> Any finding from the logs?
>>>
>>> Is there any way to replicate it?
>>>
>>> Thanks,
>>>
>>>
>>> On Wed, 28 Oct 2020 at 15:47, Pavel Tupitsyn 
>>> wrote:
>>>
 Looks like the app is running in 32 bit mode, which can't use more than
 2GB of memory.
 JVM and memory regions pre-allocate all of it, leaving nothing for .NET
 to use.

 Please check the `Platform` column in the Task Manager - does it say
 `32 bit`?
 If yes, then try disabling `Prefer 32 bit` in the project properties.

 On Wed, Oct 28, 2020 at 1:06 PM Ravi Makwana 
 wrote:

> Hi Pavel,
>
> Thanks for the information.
>
> We have noticed our service is taking max 2GB memory which is noticed
> in task manager and stopping app pool at same time server memory
> utilization is 80% so we have still memory spare 20%.
>
> Still we are not able to correlate the issue, I would like to share
> some more logs. Could you suggest what is happening at the moment?
>
> Thanks,
>
>

Re: Ignite Client Node Stopped With Out Of Memory

[Ravi Makwana]

On Wed, 28 Oct, 2020, 5:55 pm Pavel Tupitsyn,  wrote:

> I found a bug in Ignite [1] which probably causes the issue on your side.
>
> Looks like you are running a query (is it a ScanQuery or SqlQuery?) and
> the size of one results page exceeds 2GB.
> Please try using a smaller value for *QueryBase.PageSize*.
>
> If you use the default value of 1024, your cache entries are very large.
> What does the data look like?
>
> [1] https://issues.apache.org/jira/browse/IGNITE-13635
>
> On Wed, Oct 28, 2020 at 1:43 PM Ravi Makwana 
> wrote:
>
>> Hi,
>>
>> Our service is running with 64 bit and we have verified the same in our
>> app server too.
>>
>> Any finding from the logs?
>>
>> Is there any way to replicate it?
>>
>> Thanks,
>>
>>
>> On Wed, 28 Oct 2020 at 15:47, Pavel Tupitsyn 
>> wrote:
>>
>>> Looks like the app is running in 32 bit mode, which can't use more than
>>> 2GB of memory.
>>> JVM and memory regions pre-allocate all of it, leaving nothing for .NET
>>> to use.
>>>
>>> Please check the `Platform` column in the Task Manager - does it say `32
>>> bit`?
>>> If yes, then try disabling `Prefer 32 bit` in the project properties.
>>>
>>> On Wed, Oct 28, 2020 at 1:06 PM Ravi Makwana 
>>> wrote:
>>>
 Hi Pavel,

 Thanks for the information.

 We have noticed our service is taking max 2GB memory which is noticed
 in task manager and stopping app pool at same time server memory
 utilization is 80% so we have still memory spare 20%.

 Still we are not able to correlate the issue, I would like to share
 some more logs. Could you suggest what is happening at the moment?

 Thanks,

Re: Ignite Client Node Stopped With Out Of Memory

[Pavel Tupitsyn]

I found a bug in Ignite [1] which probably causes the issue on your side.

Looks like you are running a query (is it a ScanQuery or SqlQuery?) and the
size of one results page exceeds 2GB.
Please try using a smaller value for *QueryBase.PageSize*.

If you use the default value of 1024, your cache entries are very large.
What does the data look like?

[1] https://issues.apache.org/jira/browse/IGNITE-13635

On Wed, Oct 28, 2020 at 1:43 PM Ravi Makwana 
wrote:

> Hi,
>
> Our service is running with 64 bit and we have verified the same in our
> app server too.
>
> Any finding from the logs?
>
> Is there any way to replicate it?
>
> Thanks,
>
>
> On Wed, 28 Oct 2020 at 15:47, Pavel Tupitsyn  wrote:
>
>> Looks like the app is running in 32 bit mode, which can't use more than
>> 2GB of memory.
>> JVM and memory regions pre-allocate all of it, leaving nothing for .NET
>> to use.
>>
>> Please check the `Platform` column in the Task Manager - does it say `32
>> bit`?
>> If yes, then try disabling `Prefer 32 bit` in the project properties.
>>
>> On Wed, Oct 28, 2020 at 1:06 PM Ravi Makwana 
>> wrote:
>>
>>> Hi Pavel,
>>>
>>> Thanks for the information.
>>>
>>> We have noticed our service is taking max 2GB memory which is noticed in
>>> task manager and stopping app pool at same time server memory utilization
>>> is 80% so we have still memory spare 20%.
>>>
>>> Still we are not able to correlate the issue, I would like to share some
>>> more logs. Could you suggest what is happening at the moment?
>>>
>>> Thanks,
>>>
>>>

Re: IgniteAtomicSequence

[narges saleh]

Thanks Pavel for the replies.

I did a quick/basic benchmark of IgniteAtomicLong, and IgniteAtomicLong,
with only two ignite nodes, with 1,000,000 iterations. Unless, I am doing
something wrong, the result was very disappointing.
With IgniteAtomicSequence, the iterations took about 30 seconds.
With IgniteAtomicLong, it took over 3 minutes. I repeated the test 5 times.

On Wed, Oct 28, 2020 at 5:33 AM Pavel Vinokurov 
wrote:

> The atomic sequence/long could be persisted if the default data region is
> persistent[1].
> One of the use cases for IgniteAtomicSequence is to generate unique IDs.
> You could estimate the volume implicitly by getting metrics from the
> default data region[2].
>
> [1]
> https://ignite.apache.org/docs/latest/persistence/native-persistence#enabling-persistent-storage
> [2]
> https://apacheignite.readme.io/v2.8.0/docs/memory-metrics#getting-metrics
>
> вт, 27 окт. 2020 г. в 21:50, narges saleh :
>
>> Questions:
>> I know I should take into account the cluster size and the data volume,
>> but how bad is the overhead of IgniteAtomicLong? Any benchmarking?
>> What are the use cases for IgniteAtomicSequence if it does not
>> guarantee ordering?
>>
>> thanks.
>>
>> On Tue, Oct 27, 2020 at 10:27 AM narges saleh 
>> wrote:
>>
>>> Thank you Pavel, for the information.
>>>
>>> What happens to the atomic sequence/long value in case of a system
>>> restart? Do I need to persist the value somewhere and pass it as the
>>> initial value, across cluster restarts?
>>>
>>> On Tue, Oct 27, 2020 at 1:13 AM Pavel Vinokurov <
>>> vinokurov.pa...@gmail.com> wrote:
>>>
 Hi,

 In case of rebalance the reserves aren't recalculated. So the next
 values will be in the range  [1001,1000+reserveSize].
 After the local reserve exceeds, the node reserves a new range
 starting from the global last sequence number.
 The main idea IgniteAtomicSequence is to generate unique numbers but it
 doesn't guarantee ordering.
 If you need ordering you could set the reserve size to 1 or use
 IgniteAtomicLong.

 Thanks,
 Pavel

 вс, 25 окт. 2020 г. в 05:00, narges saleh :

> Hi All,
> What is the consequence of data rebalancing across nodes as far as
> IgniteAtomicSequence and the reserve on each node is concerned? For
> example, if the last sequence number is 6000 in one node and the record
> moves to a node whose last sequence number is 1000? Do the reserves on 
> both
> nodes get recalculated?
>
> Are there best practices for generating and using these sequences?
>
> Is IgniteAtomicSequence the right approach if I want to keep track of
> the records on each node for a partitioned cache?
>
> thanks.
>


 --

 Regards

 Pavel Vinokurov

>>>
>
> --
>
> Regards
>
> Pavel Vinokurov
>

Re: Ignite Client Node Stopped With Out Of Memory

[Ravi Makwana]

Hi,

Our service is running with 64 bit and we have verified the same in our app
server too.

Any finding from the logs?

Is there any way to replicate it?

Thanks,


On Wed, 28 Oct 2020 at 15:47, Pavel Tupitsyn  wrote:

> Looks like the app is running in 32 bit mode, which can't use more than
> 2GB of memory.
> JVM and memory regions pre-allocate all of it, leaving nothing for .NET to
> use.
>
> Please check the `Platform` column in the Task Manager - does it say `32
> bit`?
> If yes, then try disabling `Prefer 32 bit` in the project properties.
>
> On Wed, Oct 28, 2020 at 1:06 PM Ravi Makwana 
> wrote:
>
>> Hi Pavel,
>>
>> Thanks for the information.
>>
>> We have noticed our service is taking max 2GB memory which is noticed in
>> task manager and stopping app pool at same time server memory utilization
>> is 80% so we have still memory spare 20%.
>>
>> Still we are not able to correlate the issue, I would like to share some
>> more logs. Could you suggest what is happening at the moment?
>>
>> Thanks,
>>
>>

Re: IgniteAtomicSequence

[Pavel Vinokurov]

The atomic sequence/long could be persisted if the default data region is
persistent[1].
One of the use cases for IgniteAtomicSequence is to generate unique IDs.
You could estimate the volume implicitly by getting metrics from the
default data region[2].

[1]
https://ignite.apache.org/docs/latest/persistence/native-persistence#enabling-persistent-storage
[2]
https://apacheignite.readme.io/v2.8.0/docs/memory-metrics#getting-metrics

вт, 27 окт. 2020 г. в 21:50, narges saleh :

> Questions:
> I know I should take into account the cluster size and the data volume,
> but how bad is the overhead of IgniteAtomicLong? Any benchmarking?
> What are the use cases for IgniteAtomicSequence if it does not
> guarantee ordering?
>
> thanks.
>
> On Tue, Oct 27, 2020 at 10:27 AM narges saleh 
> wrote:
>
>> Thank you Pavel, for the information.
>>
>> What happens to the atomic sequence/long value in case of a system
>> restart? Do I need to persist the value somewhere and pass it as the
>> initial value, across cluster restarts?
>>
>> On Tue, Oct 27, 2020 at 1:13 AM Pavel Vinokurov <
>> vinokurov.pa...@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> In case of rebalance the reserves aren't recalculated. So the next
>>> values will be in the range  [1001,1000+reserveSize].
>>> After the local reserve exceeds, the node reserves a new range
>>> starting from the global last sequence number.
>>> The main idea IgniteAtomicSequence is to generate unique numbers but it
>>> doesn't guarantee ordering.
>>> If you need ordering you could set the reserve size to 1 or use
>>> IgniteAtomicLong.
>>>
>>> Thanks,
>>> Pavel
>>>
>>> вс, 25 окт. 2020 г. в 05:00, narges saleh :
>>>
 Hi All,
 What is the consequence of data rebalancing across nodes as far as
 IgniteAtomicSequence and the reserve on each node is concerned? For
 example, if the last sequence number is 6000 in one node and the record
 moves to a node whose last sequence number is 1000? Do the reserves on both
 nodes get recalculated?

 Are there best practices for generating and using these sequences?

 Is IgniteAtomicSequence the right approach if I want to keep track of
 the records on each node for a partitioned cache?

 thanks.

>>>
>>>
>>> --
>>>
>>> Regards
>>>
>>> Pavel Vinokurov
>>>
>>

-- 

Regards

Pavel Vinokurov

Re: Ignite Client Node Stopped With Out Of Memory

[Pavel Tupitsyn]

Looks like the app is running in 32 bit mode, which can't use more than 2GB
of memory.
JVM and memory regions pre-allocate all of it, leaving nothing for .NET to
use.

Please check the `Platform` column in the Task Manager - does it say `32
bit`?
If yes, then try disabling `Prefer 32 bit` in the project properties.

On Wed, Oct 28, 2020 at 1:06 PM Ravi Makwana 
wrote:

> Hi Pavel,
>
> Thanks for the information.
>
> We have noticed our service is taking max 2GB memory which is noticed in
> task manager and stopping app pool at same time server memory utilization
> is 80% so we have still memory spare 20%.
>
> Still we are not able to correlate the issue, I would like to share some
> more logs. Could you suggest what is happening at the moment?
>
> Thanks,
>
>