Re: Is there any way to make sure data and its backups in different data center

[c c]

Thank you very much!

Alex Plehanov  于2019年10月28日周一 下午8:59写道：

> Hello,
>
> You can set custom affinityBackupFilter in RendezvousAffinityFunction. See
> [1] for example.
>
> [1] :
> https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/cache/affinity/rendezvous/ClusterNodeAttributeAffinityBackupFilter.html
>
>
> пн, 28 окт. 2019 г. в 11:31, c c :
>
>> HI, we are working on ignite v2.7.0 . We plan to deploy 5 server nodes
>> across two data center (three in one data center and two in the other). We
>> setup 2 backups for each entry, so we have three copies data for each
>> entry. How can we make sure there is at least one copy in both data center?
>>
>> Regards
>>
>>

Re: Node stopped.

[John Smith]

Ok cool. Thanks. I'm just trying to figure scenarios where it may happen.

When I connect with ignite visor and run the "top" topology command. The
client node is bound to multiple addresses about 12...

Ignite is running on plain regular VMs while the client node is running in
a containerized env but using host network (which is regular VM).

So wondering if that would also screw it up something, by trying to connect
to one of those multiple addresses?

On Mon, 28 Oct 2019 at 12:08, Ilya Kasnacheev 
wrote:

> Hello!
>
> Sure, if nodes can't reach each other, eventually they may segment and
> stop.
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> пт, 25 окт. 2019 г. в 00:08, John Smith :
>
>> Is it possible this is somehow causing the issue of the node stopping?
>>
>> On Thu, 24 Oct 2019 at 11:24, Ilya Kasnacheev 
>> wrote:
>>
>>> Hello!
>>>
>>> This likely means that you have reachability problems in your cluster,
>>> such as, xxx.xxx.xxx.68 can connect to xxx.xxx.xxx.82 (on range
>>> 47100-47200) but not the other way around.
>>>
>>> Regards,
>>> --
>>> Ilya Kasnacheev
>>>
>>>
>>> пн, 21 окт. 2019 г. в 19:36, John Smith :
>>>
 I also see this printing every few seconds on my client application...
 org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi Accepted
 incoming communication connection [locAddr=/xxx.xxx.xxx.68:47101,
 rmtAddr=/xxx.xxx.xxx.82:49816

 On Mon, 21 Oct 2019 at 12:04, John Smith 
 wrote:

> Hi, thanks. I already made sure that each ignite VM runs on a separate
> host within our cloud. I'm not doing any of that migration stuff.
>
> Also recently disabled metrics igniteConfig.setMetricsLogFrequency(0);
> Just to make sure it doesn't get too chatty. But i doubt this would affect
> it...
>
> Should I maybe set the node timeout a bit higher then 30 seconds maybe
> put it to 60 seconds? I remember people suggesting this but not sure...
>
> On Fri, 18 Oct 2019 at 06:09, Denis Mekhanikov 
> wrote:
>
>> The following documentation page has some useful points on deployment
>> in a virtualised environment:
>> https://apacheignite.readme.io/docs/vmware-deployment
>>
>> Denis
>> On 17 Oct 2019, 17:41 +0300, John Smith ,
>> wrote:
>>
>> Ok I have metribeat running on the VM hopefully I will see
>> something...
>>
>> On Thu, 17 Oct 2019 at 05:09, Denis Mekhanikov 
>> wrote:
>>
>>> There are no long pauses in the GC logs, so it must be the whole VM
>>> pause.
>>>
>>> Denis
>>> On 16 Oct 2019, 23:07 +0300, John Smith ,
>>> wrote:
>>>
>>> Sorry here is the gc logs for all 3 machines:
>>> https://www.dropbox.com/s/chbbxigahd4v9di/gc-logs.zip?dl=0
>>>
>>> On Wed, 16 Oct 2019 at 15:49, John Smith 
>>> wrote:
>>>
 Hi, so it happened again here is my latest gc.log stats:
 https://gceasy.io/diamondgc-report.jsp?oTxnId_value=a215d573-d1cf-4d53-acf1-9001432bb28e

 Everything seems ok to me. I also have Elasticsearch Metricbeat
 running, the CPU usage looked normal at the time.

 On Thu, 10 Oct 2019 at 13:05, Denis Mekhanikov <
 dmekhani...@gmail.com> wrote:

> Unfortunately, I don’t.
> You can ask the VM vendor or the cloud provider (if you use any)
> for a proper tooling or logs.
> Make sure, that there is no such step in the VM’s lifecycle that
> makes it freeze for a minute.
> Also make sure that the physical CPU is not overutilized and no
> VMs that run on it are starving.
>
> Denis
> On 10 Oct 2019, 19:03 +0300, John Smith ,
> wrote:
>
> Do you know of any good tools I can use to check the VM?
>
> On Thu, 10 Oct 2019 at 11:38, Denis Mekhanikov <
> dmekhani...@gmail.com> wrote:
>
>> > Hi Dennis, so are you saying I should enable GC logs + the safe
>> point logs as well?
>>
>> Having safepoint statistics in your GC logs may be useful, so I
>> recommend enabling them for troubleshooting purposes.
>> Check the lifecycle of your virtual machines. There is a high
>> chance that the whole machine is frozen, not just the Ignite node.
>>
>> Denis
>> On 10 Oct 2019, 18:25 +0300, John Smith ,
>> wrote:
>>
>> Hi Dennis, so are you saying I should enable GC logs + the safe
>> point logs as well?
>>
>> On Thu, 10 Oct 2019 at 11:22, John Smith 
>> wrote:
>>
>>> You are correct, it is running in a VM.
>>>
>>> On Thu, 10 Oct 2019 at 10:11, Denis Mekhanikov <
>>> dmekhani...@gmail.com> wrote:
>>>
 Hi!

 There are the following messages in the logs:

 

Re: Throttling getAll

[Abhishek Gupta (BLOOMBERG/ 919 3RD A)]

Ack. I've create a JIRA to track this.

https://issues.apache.org/jira/browse/IGNITE-12334


From: user@ignite.apache.org At: 10/28/19 09:08:10To:  user@ignite.apache.org
Subject: Re: Throttling getAll

You might want to open a ticket. Of course, Ignite is open source and I’m sure 
the community would welcome a pull request.

Regards,
Stephen


On 28 Oct 2019, at 12:14, Abhishek Gupta (BLOOMBERG/ 919 3RD A) 
 wrote:



Thanks Ilya for your response.

Even if my value objects were not large, nothing stops clients from doing a 
getAll with say 100,000 keys. Having some kind of throttling would still be 
useful.

-Abhishek


- Original Message -
From: Ilya Kasnacheev 
To: ABHISHEK GUPTA
CC: user@ignite.apache.org
At: 28-Oct-2019 07:20:24


Hello!

Having very large objects is not a priority use case of Apache Ignite. Thus, it 
is your concern to make sure you don't run out of heap when doing operations on 
Ignite caches.

Regards,
--
Ilya Kasnacheev


сб, 26 окт. 2019 г. в 18:51, Abhishek Gupta (BLOOMBERG/ 919 3RD A) 
:

Hello,
  I've benchmarked my grid for users (clients) to do getAll with upto 100 
keys at a time. My value objects tend to be quite large and my worry is if 
there are errant clients might at times do a getAll with a larger number of 
keys - say 1000. If that happens I worry about GC issues/humongous objects/OOM 
on the grid. Is there a way to configure the grid to auto-split these requests 
into smaller batches (smaller number of keys per batch) or rejecting them?   


Thanks,
Abhishek

RE: Ignite node failure after network issue

[alexanderkor12]

Yes, putting the jar files with the necessary classes into the classpath of all 
nodes should solve the serialization issue.

You should see the following in your logs:
[2019-10-28 14:52:29,893][INFO 
][sys-stripe-7-#8%node1%][GridDeploymentLocalStore] Class locally deployed: 
class org.joda.time.chrono.ISOChronology$Stub
[2019-10-28 14:52:29,896][INFO 
][sys-stripe-7-#8%node1%][GridDeploymentLocalStore] Class locally deployed: 
class org.joda.time.DateTimeZone$Stub


-Original Message-
From: ihalilaltun  
Sent: Monday, October 28, 2019 6:03 AM
To: user@ignite.apache.org
Subject: RE: Ignite node failure after network issue

Hi Alex,

I've been removed the IP addreses for the sake of security reasons thats why it 
seems non-standart.

I'll try to adjust all thread-pool sizes, I am not sure if we need them or not, 
since the configurations are made by our previous software architect.

I'll look further on the serialization and marsheller problems, thanks. Will it 
be enough if I add the jar files under the lib directories on server nodes in 
order not to get these serialization problems?

thanks.



-
İbrahim Halil Altun
Senior Software Engineer @ Segmentify
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Intermittent "Partition states validation has failed for group" issues

[Abhishek Gupta (BLOOMBERG/ 919 3RD A)]

Thanks Ilya. The thing is, I've seen these exceptions without any errors 
occurring before them. Also I'm not using persistence. Also, I've seen this 
happen on multiple nodes at the same time. If I bounce multiple nodes, I would 
loose data (since I have only 1 backup). Anything else I could do?


-Abhishek


From: user@ignite.apache.org At: 10/28/19 12:47:23Cc:  user@ignite.apache.org
Subject: Re: Intermittent "Partition states validation has failed for group" 
issues

Hello!

I think this means that backup/primary contents are inconsistent.

The implications is that in case of node failure there will be data 
inconsistency (or maybe it's already there).

The recommendation is to a) check logs for any oddities/exceptions, and b) 
maybe remove problematic partitions' files from persistence and/or restart 
problematic nodes.

Regards,
-- 
Ilya Kasnacheev


пн, 21 окт. 2019 г. в 23:17, Abhishek Gupta (BLOOMBERG/ 731 LEX) 
:

In my otherwise stably running grid (on 2.7.5) I sometimes see intermittent  
GridDhtPartitionsExchangeFuture  warning. This warning the occurs periodically 
and then goes away after some time. I couldn't find any documentation or other 
threads about this warning and its implications. 
* What is the trigger for this warning? 
* What are the implications?
* Is there any recommendation around fixing this issue?


2019-10-21 16:09:44.378 [WARN ] [sys-#26240] GridDhtPartitionsExchangeFuture - 
Partition states validation has failed for group: mainCache. Partitions cache 
sizes are inconsistent for Part 0: [id-dgcasp-ob-398-csp-drp-ny-1=43417 
id-dgcasp-ob-080-csp-drp-ny-1=43416 ] Part 1: 
[id-dgcasp-ob-080-csp-drp-ny-1=43720 id-dgcasp-ob-471-csp-drp-ny-1=43724 ] Part 
2: [id-dgcasp-ob-762-csp-drp-ny-1=43388 id-dgcasp-ob-471-csp-drp-ny-1=43376 ] 
Part 3: [id-dgcasp-ob-775-csp-drp-ny-1=43488 
id-dgcasp-ob-403-csp-drp-ny-1=43484 ] Part 4: 
[id-dgcasp-ob-080-csp-drp-ny-1=43338 id-dgcasp-ob-471-csp-drp-ny-1=43339 ] Part 
5: [id-dgcasp-ob-398-csp-drp-ny-1=43105 id-dgcasp-ob-471-csp-drp-ny-1=43106 ] 
Part 7: [id-dgcasp-ob-775-csp-drp-ny-1=43151 
id-dgcasp-ob-762-csp-drp-ny-1=43157 ] Part 8: 
[id-dgcasp-ob-398-csp-drp-ny-1=42975 id-dgcasp-ob-471-csp-drp-ny-1=42976 ] Part 
10: [id-dgcasp-ob-775-csp-drp-ny-1=43033 id-dgcasp-ob-471-csp-drp-ny-1=43036 ] 
Part 11: [id-dgcasp-ob-762-csp-drp-ny-1=43303 
id-dgcasp-ob-471-csp-drp-ny-1=43299 ] Part 12: 
[id-dgcasp-ob-398-csp-drp-ny-1=43262 id-dgcasp-ob-471-csp-drp-ny-1=43265 ] Part 
13: [id-dgcasp-ob-762-csp-drp-ny-1=43123 id-dgcasp-ob-471-csp-drp-ny-1=43120 ] 
Part 15: [id-dgcasp-ob-775-csp-drp-ny-1=43412 
id-dgcasp-ob-398-csp-drp-ny-1=43413 ] Part 16: 
[id-dgcasp-ob-471-csp-drp-ny-1=43934 id-dgcasp-ob-403-csp-drp-ny-1=43933 ] Part 
20: [id-dgcasp-ob-080-csp-drp-ny-1=43146 id-dgcasp-ob-471-csp-drp-ny-1=43148 ] 
Part 21: [id-dgcasp-ob-762-csp-drp-ny-1=43196 
id-dgcasp-ob-080-csp-drp-ny-1=43197 ] Part 22: 
[id-dgcasp-ob-398-csp-drp-ny-1=43233 id-dgcasp-ob-762-csp-drp-ny-1=43234 ] Part 
23: [id-dgcasp-ob-398-csp-drp-ny-1=43127 id-dgcasp-ob-471-csp-drp-ny-1=43128 ] 
Part 24: [id-dgcasp-ob-775-csp-drp-ny-1=43144 
id-dgcasp-ob-398-csp-drp-ny-1=43142 ]  ... TRUNCATED


Thanks,
Abhishek

RE: Query execution in ignite

[alexanderkor12]

In PARTITIONED mode SQL queries are executed over each node’s primary 
partitions only.

 

Here is more information about distributed joins: 
https://apacheignite-sql.readme.io/docs/distributed-joins

 

 

 

From: Prasad Bhalerao  
Sent: Saturday, October 26, 2019 12:25 AM
To: user@ignite.apache.org
Subject: Re: Query execution in ignite

 

 

Question is specifically about primary and secondary partitions.

 

So in case of replicated cache ignite scans primary and secondary partitions of 
any one of the node of the cluster to to fetch the data.

 

But is it the case with partitioned cache.

I mean in case of partitioned cache when SQL is executed, does ignite scan 
primary as well as secondary partitions of each node in the cluster or it just 
scans primary partitions of all the nodes in the cluster as the query is being 
executed on all nodes?

 

On Fri 25 Oct, 2019, 10:46 PM mailto:alexanderko...@gmail.com>  wrote:

Hi,

 

   If a query is executed against a fully REPLICATED data then Ignite will send 
it to a single cluster node and run it over the local data there.

 

 

 

 if a query is executed over a PARTITIONED data, then the execution flow will 
be the following:

The query will be parsed and split into multiple map queries and a single 
reduce query.

* All the map queries are executed on all the nodes where required data 
resides.

* All the nodes provide result sets of local execution to the query 
initiator (reducer) that, in turn, will accomplish the reduce phase by properly 
merging provided result sets.

 

 

 

 More information here: 
https://apacheignite-sql.readme.io/docs/how-ignite-sql-works

Thanks, Alex

 

From: Prasad Bhalerao mailto:prasadbhalerao1...@gmail.com> > 
Sent: Friday, October 25, 2019 1:31 AM
To: user@ignite.apache.org  
Subject: Query execution in ignite

 

Hi,

 

When SQL is executed, does ignite always scan only primary partitions of all 
available nodes in cluster irrespective of cache mode partitioned or replicated?

 

 

 

Thanks ,

Prasad

Re: "Failed to deserialize object" in Client Mode

[Ilya Kasnacheev]

Hello!

This is likely related to https://issues.apache.org/jira/browse/IGNITE-11738

This class is buggy in latest released versions, consider turning metrics
off if you see this problem. Especially problematic if you use Zk or wrong
kind of marshaller.

Regards,
-- 
Ilya Kasnacheev


ср, 23 окт. 2019 г. в 11:36, j_recuerda :

> Yes.
>
> The thing is that when I run the node (same configuration and same version)
> just switching to server mode it works properly.
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Intermittent "Partition states validation has failed for group" issues

[Ilya Kasnacheev]

Hello!

I think this means that backup/primary contents are inconsistent.

The implications is that in case of node failure there will be data
inconsistency (or maybe it's already there).

The recommendation is to a) check logs for any oddities/exceptions, and b)
maybe remove problematic partitions' files from persistence and/or restart
problematic nodes.

Regards,
-- 
Ilya Kasnacheev


пн, 21 окт. 2019 г. в 23:17, Abhishek Gupta (BLOOMBERG/ 731 LEX) <
agupta...@bloomberg.net>:

> In my otherwise stably running grid (on 2.7.5) I sometimes see
> intermittent GridDhtPartitionsExchangeFuture warning. This warning the
> occurs periodically and then goes away after some time. I couldn't find any
> documentation or other threads about this warning and its implications.
> * What is the trigger for this warning?
> * What are the implications?
> * Is there any recommendation around fixing this issue?
>
>
>
>
> 2019-10-21 16:09:44.378 [WARN ] [sys-#26240]
> GridDhtPartitionsExchangeFuture - Partition states validation has failed
> for group: mainCache. Partitions cache sizes are inconsistent for Part 0:
> [id-dgcasp-ob-398-csp-drp-ny-1=43417 id-dgcasp-ob-080-csp-drp-ny-1=43416 ]
> Part 1: [id-dgcasp-ob-080-csp-drp-ny-1=43720
> id-dgcasp-ob-471-csp-drp-ny-1=43724 ] Part 2:
> [id-dgcasp-ob-762-csp-drp-ny-1=43388 id-dgcasp-ob-471-csp-drp-ny-1=43376 ]
> Part 3: [id-dgcasp-ob-775-csp-drp-ny-1=43488
> id-dgcasp-ob-403-csp-drp-ny-1=43484 ] Part 4:
> [id-dgcasp-ob-080-csp-drp-ny-1=43338 id-dgcasp-ob-471-csp-drp-ny-1=43339 ]
> Part 5: [id-dgcasp-ob-398-csp-drp-ny-1=43105
> id-dgcasp-ob-471-csp-drp-ny-1=43106 ] Part 7:
> [id-dgcasp-ob-775-csp-drp-ny-1=43151 id-dgcasp-ob-762-csp-drp-ny-1=43157 ]
> Part 8: [id-dgcasp-ob-398-csp-drp-ny-1=42975
> id-dgcasp-ob-471-csp-drp-ny-1=42976 ] Part 10:
> [id-dgcasp-ob-775-csp-drp-ny-1=43033 id-dgcasp-ob-471-csp-drp-ny-1=43036 ]
> Part 11: [id-dgcasp-ob-762-csp-drp-ny-1=43303
> id-dgcasp-ob-471-csp-drp-ny-1=43299 ] Part 12:
> [id-dgcasp-ob-398-csp-drp-ny-1=43262 id-dgcasp-ob-471-csp-drp-ny-1=43265 ]
> Part 13: [id-dgcasp-ob-762-csp-drp-ny-1=43123
> id-dgcasp-ob-471-csp-drp-ny-1=43120 ] Part 15:
> [id-dgcasp-ob-775-csp-drp-ny-1=43412 id-dgcasp-ob-398-csp-drp-ny-1=43413 ]
> Part 16: [id-dgcasp-ob-471-csp-drp-ny-1=43934
> id-dgcasp-ob-403-csp-drp-ny-1=43933 ] Part 20:
> [id-dgcasp-ob-080-csp-drp-ny-1=43146 id-dgcasp-ob-471-csp-drp-ny-1=43148 ]
> Part 21: [id-dgcasp-ob-762-csp-drp-ny-1=43196
> id-dgcasp-ob-080-csp-drp-ny-1=43197 ] Part 22:
> [id-dgcasp-ob-398-csp-drp-ny-1=43233 id-dgcasp-ob-762-csp-drp-ny-1=43234 ]
> Part 23: [id-dgcasp-ob-398-csp-drp-ny-1=43127
> id-dgcasp-ob-471-csp-drp-ny-1=43128 ] Part 24:
> [id-dgcasp-ob-775-csp-drp-ny-1=43144 id-dgcasp-ob-398-csp-drp-ny-1=43142 ]
> ... TRUNCATED
>
>
> Thanks,
> Abhishek
>
>

Re: Ingite node in SpringBoot App

[Ilya Kasnacheev]

Hello!

I don't think that IgniteClientStarter is called at all, since I have added
debug logging to it, but it doesn't print anything.  Are you sure that it
runs?

Regards,
-- 
Ilya Kasnacheev


пт, 25 окт. 2019 г. в 16:19, niamin :

> The is no error rather failed use case. I expected the following statement
> to
> be a cache hit:
>
>  ARInvoice arInvoice = clientCache.get(arInvoiceId);
>
> And subsequently, the following statement to be printed on console:
>
>  System.out.println("retrieved deserialized successfully from cache " +
> arInvoice.getArInvoiceId().getInvoiceSiteNo());
>
> The above statement is not printed out which is states it is a cache miss.
> I
> also confirmed it by running the client on debug mode on an IDE.
>
> The same use case works successfully on Ignite server i.e. the
> IgniteCacheLoader project. So the question is why is there a cache miss
> from
> IgniteClient? It seems to work with Ignite server.
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Unresponsive cluster after "Checkpoint read lock acquisition has been timed out" Error

[Ilya Kasnacheev]

Hello!

2-4 kilobytes is not big. Still you may want to check your logs for
long-running transactions, etc.

Regards,
-- 
Ilya Kasnacheev


пт, 25 окт. 2019 г. в 18:21, ihalilaltun :

> Hi Ilya,
>
> It is almost impossible for us to get thread dumps since this is production
> environment we cannot use profiler :(
>
> Our biggest object range from 2 to 4 kilobytes. We are planning to shrink
> the sizes but time for this is not decided yet.
>
> regards.
>
>
>
> -
> İbrahim Halil Altun
> Senior Software Engineer @ Segmentify
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Node stopped.

[Ilya Kasnacheev]

Hello!

Sure, if nodes can't reach each other, eventually they may segment and stop.

Regards,
-- 
Ilya Kasnacheev


пт, 25 окт. 2019 г. в 00:08, John Smith :

> Is it possible this is somehow causing the issue of the node stopping?
>
> On Thu, 24 Oct 2019 at 11:24, Ilya Kasnacheev 
> wrote:
>
>> Hello!
>>
>> This likely means that you have reachability problems in your cluster,
>> such as, xxx.xxx.xxx.68 can connect to xxx.xxx.xxx.82 (on range
>> 47100-47200) but not the other way around.
>>
>> Regards,
>> --
>> Ilya Kasnacheev
>>
>>
>> пн, 21 окт. 2019 г. в 19:36, John Smith :
>>
>>> I also see this printing every few seconds on my client application...
>>> org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi Accepted
>>> incoming communication connection [locAddr=/xxx.xxx.xxx.68:47101,
>>> rmtAddr=/xxx.xxx.xxx.82:49816
>>>
>>> On Mon, 21 Oct 2019 at 12:04, John Smith  wrote:
>>>
 Hi, thanks. I already made sure that each ignite VM runs on a separate
 host within our cloud. I'm not doing any of that migration stuff.

 Also recently disabled metrics igniteConfig.setMetricsLogFrequency(0);
 Just to make sure it doesn't get too chatty. But i doubt this would affect
 it...

 Should I maybe set the node timeout a bit higher then 30 seconds maybe
 put it to 60 seconds? I remember people suggesting this but not sure...

 On Fri, 18 Oct 2019 at 06:09, Denis Mekhanikov 
 wrote:

> The following documentation page has some useful points on deployment
> in a virtualised environment:
> https://apacheignite.readme.io/docs/vmware-deployment
>
> Denis
> On 17 Oct 2019, 17:41 +0300, John Smith ,
> wrote:
>
> Ok I have metribeat running on the VM hopefully I will see something...
>
> On Thu, 17 Oct 2019 at 05:09, Denis Mekhanikov 
> wrote:
>
>> There are no long pauses in the GC logs, so it must be the whole VM
>> pause.
>>
>> Denis
>> On 16 Oct 2019, 23:07 +0300, John Smith ,
>> wrote:
>>
>> Sorry here is the gc logs for all 3 machines:
>> https://www.dropbox.com/s/chbbxigahd4v9di/gc-logs.zip?dl=0
>>
>> On Wed, 16 Oct 2019 at 15:49, John Smith 
>> wrote:
>>
>>> Hi, so it happened again here is my latest gc.log stats:
>>> https://gceasy.io/diamondgc-report.jsp?oTxnId_value=a215d573-d1cf-4d53-acf1-9001432bb28e
>>>
>>> Everything seems ok to me. I also have Elasticsearch Metricbeat
>>> running, the CPU usage looked normal at the time.
>>>
>>> On Thu, 10 Oct 2019 at 13:05, Denis Mekhanikov <
>>> dmekhani...@gmail.com> wrote:
>>>
 Unfortunately, I don’t.
 You can ask the VM vendor or the cloud provider (if you use any)
 for a proper tooling or logs.
 Make sure, that there is no such step in the VM’s lifecycle that
 makes it freeze for a minute.
 Also make sure that the physical CPU is not overutilized and no VMs
 that run on it are starving.

 Denis
 On 10 Oct 2019, 19:03 +0300, John Smith ,
 wrote:

 Do you know of any good tools I can use to check the VM?

 On Thu, 10 Oct 2019 at 11:38, Denis Mekhanikov <
 dmekhani...@gmail.com> wrote:

> > Hi Dennis, so are you saying I should enable GC logs + the safe
> point logs as well?
>
> Having safepoint statistics in your GC logs may be useful, so I
> recommend enabling them for troubleshooting purposes.
> Check the lifecycle of your virtual machines. There is a high
> chance that the whole machine is frozen, not just the Ignite node.
>
> Denis
> On 10 Oct 2019, 18:25 +0300, John Smith ,
> wrote:
>
> Hi Dennis, so are you saying I should enable GC logs + the safe
> point logs as well?
>
> On Thu, 10 Oct 2019 at 11:22, John Smith 
> wrote:
>
>> You are correct, it is running in a VM.
>>
>> On Thu, 10 Oct 2019 at 10:11, Denis Mekhanikov <
>> dmekhani...@gmail.com> wrote:
>>
>>> Hi!
>>>
>>> There are the following messages in the logs:
>>>
>>> [22:26:21,816][WARNING][jvm-pause-detector-worker][IgniteKernal%xx]
>>> Possible too long JVM pause: *55705 milliseconds*.
>>> ...
>>> [22:26:21,847][SEVERE][ttl-cleanup-worker-#48%xx%][G]
>>> Blocked system-critical thread has been detected. This can lead to
>>> cluster-wide undefined behaviour [threadName=partition-exchanger,
>>> blockedFor=*57s*]
>>>
>>> Looks like the JVM was paused for almost a minute. It doesn’t
>>> seem to be caused by a garbage collection, since there is no 
>>> evidence of GC
>>> pressure in the GC log. Usually such big pauses happen in 
>>> virtualised
>>> 

Re: Throttling getAll

[Stephen Darlington]

You might want to open a ticket. Of course, Ignite is open source and I’m sure 
the community would welcome a pull request.

Regards,
Stephen

> On 28 Oct 2019, at 12:14, Abhishek Gupta (BLOOMBERG/ 919 3RD A) 
>  wrote:
> 
> 
> Thanks Ilya for your response.
> 
> Even if my value objects were not large, nothing stops clients from doing a 
> getAll with say 100,000 keys. Having some kind of throttling would still be 
> useful.
> 
> -Abhishek
> 
> 
> 
> - Original Message -
> From: Ilya Kasnacheev 
> To: ABHISHEK GUPTA
> CC: user@ignite.apache.org
> At: 28-Oct-2019 07:20:24
> 
> Hello!
> 
> Having very large objects is not a priority use case of Apache Ignite. Thus, 
> it is your concern to make sure you don't run out of heap when doing 
> operations on Ignite caches.
> 
> Regards,
> --
> Ilya Kasnacheev
> 
> 
> сб, 26 окт. 2019 г. в 18:51, Abhishek Gupta (BLOOMBERG/ 919 3RD A) 
> :
>> Hello,
>>   I've benchmarked my grid for users (clients) to do getAll with upto 
>> 100 keys at a time. My value objects tend to be quite large and my worry is 
>> if there are errant clients might at times do a getAll with a larger number 
>> of keys - say 1000. If that happens I worry about GC issues/humongous 
>> objects/OOM on the grid. Is there a way to configure the grid to auto-split 
>> these requests into smaller batches (smaller number of keys per batch) or 
>> rejecting them?   
>> 
>> 
>> Thanks,
>> Abhishek
>> 

Re: Is there any way to make sure data and its backups in different data center

[Alex Plehanov]

Hello,

You can set custom affinityBackupFilter in RendezvousAffinityFunction. See
[1] for example.

[1] :
https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/cache/affinity/rendezvous/ClusterNodeAttributeAffinityBackupFilter.html


пн, 28 окт. 2019 г. в 11:31, c c :

> HI, we are working on ignite v2.7.0 . We plan to deploy 5 server nodes
> across two data center (three in one data center and two in the other). We
> setup 2 backups for each entry, so we have three copies data for each
> entry. How can we make sure there is at least one copy in both data center?
>
> Regards
>
>

Re: Throttling getAll

[Abhishek Gupta (BLOOMBERG/ 919 3RD A)]

Thanks Ilya for your response.

Even if my value objects were not large, nothing stops clients from doing a 
getAll with say 100,000 keys. Having some kind of throttling would still be 
useful.

-Abhishek


- Original Message -
From: Ilya Kasnacheev 
To: ABHISHEK GUPTA
CC: user@ignite.apache.org
At: 28-Oct-2019 07:20:24


Hello!

Having very large objects is not a priority use case of Apache Ignite. Thus, it 
is your concern to make sure you don't run out of heap when doing operations on 
Ignite caches.

Regards,
--
Ilya Kasnacheev


сб, 26 окт. 2019 г. в 18:51, Abhishek Gupta (BLOOMBERG/ 919 3RD A) 
:

Hello,
  I've benchmarked my grid for users (clients) to do getAll with upto 100 
keys at a time. My value objects tend to be quite large and my worry is if 
there are errant clients might at times do a getAll with a larger number of 
keys - say 1000. If that happens I worry about GC issues/humongous objects/OOM 
on the grid. Is there a way to configure the grid to auto-split these requests 
into smaller batches (smaller number of keys per batch) or rejecting them?   


Thanks,
Abhishek

Re: Issue with adding nested index dynamically

[Hemambara]

Sure. Have created one https://github.com/apache/ignite/pull/7016

Can you please check this. It has all the changes at one place. please let
me know for any issues. Thank you.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Write python dataframe to ignite table.

[Stephen Darlington]

What have you tried? As long as your command-line includes the right JAR files 
it seems to more-or-less just work for me:

https://medium.com/@sdarlington/the-trick-to-successfully-integrating-apache-ignite-and-pyspark-890e436d09ba
 


Regards,
Stephen

> On 22 Oct 2019, at 11:41, Muhammed Favas  
> wrote:
> 
> Hi,
>  
> Is there a way to bulk load python dataframe values to ignite table?
>  
> Regards,
> Favas 

Re: Throttling getAll

[Ilya Kasnacheev]

Hello!

Having very large objects is not a priority use case of Apache Ignite.
Thus, it is your concern to make sure you don't run out of heap when doing
operations on Ignite caches.

Regards,
-- 
Ilya Kasnacheev


сб, 26 окт. 2019 г. в 18:51, Abhishek Gupta (BLOOMBERG/ 919 3RD A) <
agupta...@bloomberg.net>:

> Hello,
> I've benchmarked my grid for users (clients) to do getAll with upto 100
> keys at a time. My value objects tend to be quite large and my worry is if
> there are errant clients might at times do a getAll with a larger number of
> keys - say 1000. If that happens I worry about GC issues/humongous
> objects/OOM on the grid. Is there a way to configure the grid to auto-split
> these requests into smaller batches (smaller number of keys per batch) or
> rejecting them?
>
>
> Thanks,
> Abhishek
>
>

RE: Ignite node failure after network issue

[ihalilaltun]

Hi Alex,

I've been removed the IP addreses for the sake of security reasons thats why
it seems non-standart.

I'll try to adjust all thread-pool sizes, I am not sure if we need them or
not, since the configurations are made by our previous software architect.

I'll look further on the serialization and marsheller problems, thanks. Will
it be enough if I add the jar files under the lib directories on server
nodes in order not to get these serialization problems?

thanks.



-
İbrahim Halil Altun
Senior Software Engineer @ Segmentify
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Is there any way to make sure data and its backups in different data center

[c c]

HI, we are working on ignite v2.7.0 . We plan to deploy 5 server nodes
across two data center (three in one data center and two in the other). We
setup 2 backups for each entry, so we have three copies data for each
entry. How can we make sure there is at least one copy in both data center?

Regards

Re: Issue with adding nested index dynamically

[Ivan Pavlukhin]

Hi Hemambara,

It really good that we have common understanding now.
> I could not merge pull request as it is saying that only users with "Write 
> access" can merge pull requests

By merge I meant merging branches in your fork repository. It is
really inconvenient to work with bunch of PRs simultaneously. It will
be great if there is only one PR.

пт, 25 окт. 2019 г. в 14:43, Hemambara :
>
> Ohh yes you are right. Without the fix, it is failing to query cache. With
> the other approach that I have attached earlier it failed to join node,
> there I have CacheConfiguration in xml file for both server and client node.
> Not sure if that does matters. Either way it is failing
>
> Also this is my bad, my apologies, that I forget to check in one change in
> GridQueryProcessor.java where it is using QueryBinaryProperty constructor,
> instead I am using QUeryUtils.buildBinaryProperty to build it. Now I have
> checked in all the changes along with test cases.
>
> Can you please review pull requests:
> https://github.com/apache/ignite/pull/7008 - Change in QueryUtils
> https://github.com/apache/ignite/pull/7009 - Test case for QueryUtilsTest
> https://github.com/apache/ignite/pull/7010 - Change in
> GridQueryProcessor.java
> https://github.com/apache/ignite/pull/7011 - Test case to make sure it is
> adding nested field
>
> I could not merge pull request as it is saying that only users with "Write
> access" can merge pull requests. Can you please merge these pull requests
> and review. Please let me know if you see any issues.
>
> Appreciate your help on this.
>
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/



-- 
Best regards,
Ivan Pavlukhin