(Ignite ver: 2.6)Failing client node due to not receiving metrics updates from client node

[Prasad Bhalerao]

I am using ignite 2.6 version.

Ignite client node is failing in prod env giving following warning.
So I changed the  clientFailureDetectionTimeout to 9 millis. But this
did not solve the problem. I am still facing this issue.

1) Can someone please explain the reason for this issue?  Is it really
happening because of network issue or there could be some other reason for
this?

Also I see one issue logged for this
https://issues.apache.org/jira/browse/IGNITE-10354.

2) This is fixed in ignite 2.8 version. *Will upgrading to Ignite 2.9.1
solve this issue*?

2021-02-16 07:56:10,874 70815632 [tcp-disco-msg-worker-#2%springDataNode%]
WARN  o.a.i.s.d.tcp.TcpDiscoverySpi - Failing client node due to not
receiving metrics updates from client node within
'IgniteConfiguration.clientFailureDetectionTimeout' (consider increasing
configuration property) [timeout=6, node=TcpDiscoveryNode
[id=7b0d1b78-004b-465a-8dcd-de706cbd4e8e, addrs=[0:0:0:0:0:0:0:1%lo,
127.0.0.1, 1.1.17.224], sockAddrs=[host03.com, /0:0:0:0:0:0:0:1%lo:0, /
127.0.0.1:0], discPort=0, order=34, intOrder=22,
lastExchangeTime=1613461171898, loc=false,
ver=2.6.0#20180710-sha1:669feacc, isClient=true]]

Re: Call for presentations for ApacheCon North America 2020 now open

[Prasad Bhalerao]

Hi Saikat,
Can you please share the slides for both presentations, streaming as well
as ignite internals?
Thanks,
Prasad

On Wed 5 Aug, 2020, 7:10 AM Saikat Maitra  Hi,
>
> I learned that my proposal for talk about Data Streaming using Apache Ignite
> and Apache Flink has been accepted.
>
> I have not yet received the schedule yet. I will share as soon as I have
> it.
>
> Regards,
> Saikat
>
> On Wed, Jan 22, 2020 at 8:09 PM Saikat Maitra 
> wrote:
>
>> Hi Denis,
>>
>> Thank you for your email. I have submitted a talk about Data Streaming
>> using Apache Ignite and Apache Flink.
>>
>> I would also like to co-present on Ignite internals and usecases deep
>> dive.
>>
>> Regards,
>> Saikat
>>
>> On Wed, Jan 22, 2020 at 6:22 PM Denis Magda  wrote:
>>
>>> Igniters,
>>>
>>> I was submitting an Ignite talk to the conference and found out that
>>> this time ASF created a separate category for Ignite-specific proposals.
>>> You can see an abstract submitted by me:
>>> https://drive.google.com/file/d/1woaEOWaIFxN8UIJ7nvFbYoc53mYsUsSN/view?usp=sharing
>>>
>>> Who else is ready to submit? It can be a deep-dive about Ignite
>>> internals or Ignite use case overview. We can co-present. Also, GridGain is
>>> ready to sponsor your trip if the talk is accepted.
>>>
>>>
>>>
>>> -
>>> Denis
>>>
>>>
>>> On Tue, Jan 21, 2020 at 5:22 PM Denis Magda  wrote:
>>>
 Ignite folks, just bringing to your attention and encourage anybody
 interested to submit an Ignite-related session. Both Alex Zinoviev and I
 presented at ApacheCons the previous year -- the events are worth attending
 and speaking at.

 -
 Denis


 -- Forwarded message -
 From: Rich Bowen 
 Date: Tue, Jan 21, 2020 at 7:06 AM
 Subject: Call for presentations for ApacheCon North America 2020 now
 open
 To: plann...@apachecon.com 


 Dear Apache enthusiast,

 (You’re receiving this message because you are subscribed to one or
 more
 project mailing lists at the Apache Software Foundation.)

 The call for presentations for ApacheCon North America 2020 is now open
 at https://apachecon.com/acna2020/cfp

 ApacheCon will be held at the Sheraton, New Orleans, September 28th
 through October 2nd, 2020.

 As in past years, ApacheCon will feature tracks focusing on the various
 technologies within the Apache ecosystem, and so the call for
 presentations will ask you to select one of those tracks, or “General”
 if the content falls outside of one of our already-organized tracks.
 These tracks are:

 Karaf
 Internet of Things
 Fineract
 Community
 Content Delivery
 Solr/Lucene (Search)
 Gobblin/Big Data Integration
 Ignite
 Observability
 Cloudstack
 Geospatial
 Graph
 Camel/Integration
 Flagon
 Tomcat
 Cassandra
 Groovy
 Web/httpd
 General/Other

 The CFP will close Friday, May 1, 2020 8:00 AM (America/New_York time).

 Submit early, submit often, at https://apachecon.com/acna2020/cfp

 Rich, for the ApacheCon Planners

>>>

Default order of SQL result when Order By is not present

[Prasad Bhalerao]

> I am executing following SQL in ignite. As you can see there is no order
> by clause in this SQL. Index is present on field name "id".
>
> SqlFieldsQuery sql = new SqlFieldsQuery("select p.id. p.name from Person
> p join table(id bigint = ?) i on p.id = i.id.") .setArgs(new
> Object[]{5,4,1,7,3});
>
>
> What I have observed is ignite always returns query result in an order as
> it is provided in input.
> E.g. I always get resultant rows in following order. This behaviour is
> consistent.
> p.id.p.name
> 5test5
> 4   Test4
> 1Test1
> 7Test7
> 3 Test3
>
> If I change input "id" sequence, the output changes as per the id sequence
> provided.
>
> Can someone please confirm this behaviour?
>
> Also, will it remain same with calcite integration?
>
>
>

[no subject]

[Prasad Bhalerao]

I am executing following SQL in ignite. As you can see there is no order by
clause in this SQL. Index is present on field name "id".

SqlFieldsQuery sql = new SqlFieldsQuery("select p.id. p.name from Person p
join table(id bigint = ?) i on p.id = i.id.") .setArgs(new
Object[]{5,4,1,7,3});


What I have observed is ignite always returns query result in an order as
it is provided in input.
E.g. I always get resultant rows in following order. This behaviour is
consistent.
p.id.p.name
5test5
4   Test4
1Test1
7Test7
3 Test3

If I change input "id" sequence, the output changes as per the id sequence
provided.

Can someone please confirm this behaviour?

Also, will it remain same with calcite integration?

Re: CountDownLatch issue in Ignite 2.6 version

[Prasad Bhalerao]

I just checked the ignite doc for atomic configuration.
But it doesn't say that it is applicable to distributed data structures.

Is it really applicable to distributed data structures like count down latch

On Tue 9 Jun, 2020, 7:26 AM Prasad Bhalerao  Hi,
> I was under the impression that countdown latch is implemented in
> replicated cache. So when any number of nodes go down it does not loose
> it's state.
>
> Can you please explain why atmoc data structures are using 1 back when its
> state is very important?
>
> Can we enforce  atomic data structures to use replicated cache?
>
> Which cache does ignite use to store atomic data structures?
>
> Thanks
> Prasad
>
> On Mon 8 Jun, 2020, 11:58 PM Evgenii Zhuravlev  wrote:
>
>> Hi,
>>
>> By default, cache, that stores all atomic structures has only 1 backup,
>> so, after losing all data for this certain latch, it recreates it. To
>> change the default atomic configuration use
>> IgniteConfiguration.setAtomicConfiguration.
>>
>> Evgenii
>>
>> сб, 6 июн. 2020 г. в 06:20, Akash Shinde :
>>
>>> *Issue:* Countdown latched gets reinitialize to original value(4) when
>>> one or more (but not all) node goes down. *(Partition loss happened)*
>>>
>>> We are using ignite's distributed countdownlatch to make sure that cache
>>> loading is completed on all server nodes. We do this to make sure that our
>>> kafka consumers starts only after cache loading is complete on all server
>>> nodes. This is the basic criteria which needs to be fulfilled before starts
>>> actual processing
>>>
>>>
>>>  We have 4 server nodes and countdownlatch is initialized to 4. We use
>>> cache.loadCache method to start the cache loading. When each server
>>> completes cache loading it reduces the count by 1 using countDown method.
>>> So when all the nodes completes cache loading, the count reaches to zero.
>>> When this count  reaches to zero we start kafka consumers on all server
>>> nodes.
>>>
>>>  But we saw weird behavior in prod env. The 3 server nodes were shut
>>> down at the same time. But 1 node is still alive. When this happened the
>>> count down was reinitialized to original value i.e. 4. But I am not able to
>>> reproduce this in dev env.
>>>
>>>  Is this a bug, when one or more (but not all) nodes goes down then
>>> count re initializes back to original value?
>>>
>>> Thanks,
>>> Akash
>>>
>>

Ignite visor use

[Prasad Bhalerao]

Hi,
I am starting ignite inside my java spring boot app. I do not use ignite
shell script to start it.
Can I use ignite visor in such case to connect to my ignite cluster?

What all the minimum required scrips do I need to use ignite visor?


Thanks,
Prasad

Re: CountDownLatch issue in Ignite 2.6 version

[Prasad Bhalerao]

Hi,
I was under the impression that countdown latch is implemented in
replicated cache. So when any number of nodes go down it does not loose
it's state.

Can you please explain why atmoc data structures are using 1 back when its
state is very important?

Can we enforce  atomic data structures to use replicated cache?

Which cache does ignite use to store atomic data structures?

Thanks
Prasad

On Mon 8 Jun, 2020, 11:58 PM Evgenii Zhuravlev  Hi,
>
> By default, cache, that stores all atomic structures has only 1 backup,
> so, after losing all data for this certain latch, it recreates it. To
> change the default atomic configuration use
> IgniteConfiguration.setAtomicConfiguration.
>
> Evgenii
>
> сб, 6 июн. 2020 г. в 06:20, Akash Shinde :
>
>> *Issue:* Countdown latched gets reinitialize to original value(4) when
>> one or more (but not all) node goes down. *(Partition loss happened)*
>>
>> We are using ignite's distributed countdownlatch to make sure that cache
>> loading is completed on all server nodes. We do this to make sure that our
>> kafka consumers starts only after cache loading is complete on all server
>> nodes. This is the basic criteria which needs to be fulfilled before starts
>> actual processing
>>
>>
>>  We have 4 server nodes and countdownlatch is initialized to 4. We use
>> cache.loadCache method to start the cache loading. When each server
>> completes cache loading it reduces the count by 1 using countDown method.
>> So when all the nodes completes cache loading, the count reaches to zero.
>> When this count  reaches to zero we start kafka consumers on all server
>> nodes.
>>
>>  But we saw weird behavior in prod env. The 3 server nodes were shut down
>> at the same time. But 1 node is still alive. When this happened the count
>> down was reinitialized to original value i.e. 4. But I am not able to
>> reproduce this in dev env.
>>
>>  Is this a bug, when one or more (but not all) nodes goes down then count
>> re initializes back to original value?
>>
>> Thanks,
>> Akash
>>
>

Re: Read through not working as expected in case of Replicated cache

[Prasad Bhalerao]

Hi Ivan,

Thank you for the clarification.

So the behavior is same for REPLICATED as well as PARTITIONED cache.

1) Can we please have this behavior documented on Ignite web page? This
will just help users to avoid confusion and design their cache effectively.

2)  You said "You can check it using IgniteCache.localPeek method (ask if
more details how to do it are needed)".  Can you please explain this in
detail?


Regard,
Prasad

On Mon, Mar 2, 2020 at 2:45 PM Ivan Pavlukhin  wrote:

> Hi Prasad,
>
> AFAIK, when value is read through it is not sent to backup nodes. You
> can check it using IgniteCache.localPeek method (ask if more details
> how to do it are needed).
>
> I usually think about read-through cache for a following case. There
> is an underlying storage with "real" data, cache is used to speedup an
> access. Some kind of invalidation mechanism might be used but it is
> assumed fine to read values from cache which are not consistent with
> the backing storage at some point.
>
> Consequently it seems there is no need to distribute values from an
> underlying storage over all replicas because if a value is absent a
> reader will receive an actual value from the underlying storage.
>
> Best regards,
> Ivan Pavlukhin
>
> пн, 2 мар. 2020 г. в 10:41, Prasad Bhalerao  >:
> >
> > Hi Ivan/Denis,
> >
> > Are you saying that when a value is loaded to cache from an underlying
> > storage using read-through approach, value is loaded only on primary node
> > and does not get replicated on its back nodes?
> >
> > I am under the impression that when a value is loaded in a cache using
> > read-through approach, this key/value pair gets replicated on all back-up
> > nodes as well, irrespective of REPLICATED OR PARTITIONED cache.
> > Please correct me if I am wrong.
> >
> > I think the key/value must get replicated on all backup nodes when it is
> > read through underlying storage otherwise user will have to add the same
> > key/value explicitly using cache.put(key,value) operation so that it will
> > get replicated on all of its backup nodes.  This is what I am doing right
> > now as a workaround to solve this issue.
> >
> > I will try to explain my use case again.
> >
> > I have few replicated caches for which read-through is enabled but
> > write-through is disabled. The underlying tables for these caches are
> > updated by different systems. Whenever these tables are updated by 3rd
> > party system I want to reload the "cache entries".
> >
> > I achieve this using below given steps:
> > 1) 3rd party systems sends an update message (which contains the key) to
> > our service by invoking our REST api.
> > 2) Delete an entry from cache using cache().remove(key) method. (Entry is
> > just removed from cache but present in DB as write-through is false)
> > 3) Invoke cache().get(key) method for the same key in step 2 to reload an
> > entry.
> >
> > Thanks,
> > Prasad
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > Prasad
> >
> > On Sat, Feb 29, 2020 at 4:49 AM Denis Magda  wrote:
> >
> > > Ivan, thanks for stepping in.
> > >
> > > Prasad, is Ivan's assumption correct that you query the data with SQL
> under
> > > the observed circumstances? My guess is that you were referring to the
> > > key-value APIs as long as the issue is gone when the write-through is
> > > enabled.
> > >
> > > -
> > > Denis
> > >
> > >
> > > On Fri, Feb 28, 2020 at 2:30 PM Ivan Pavlukhin 
> > > wrote:
> > >
> > > > As I understand the thing here is in combination of read-through and
> > > > SQL. SQL queries do not read from underlying storage when
> read-through
> > > > is configured. And an observed result happens because query from a
> > > > client node over REPLICATED cache picks random server node (kind of
> > > > load-balancing) to retrieve data. Following happens in the described
> > > > case:
> > > > 1. Value is loaded to a cache from an underlying storage on a primary
> > > > node when cache.get is called.
> > > > 2. Query is executed multiple times and when the chose node is the
> > > > primary node then the value is observed. On other nodes the value is
> > > > absent.
> > > >
> > > > Actually, behavior for PARTITIONED cache is similar, but an
> > >

Re: cache.containsKey returns false

[Prasad Bhalerao]

Hi,

We have not set any expiration/ eviction policy.
We were getting false for key "key1", so we checked the key/value by
querying cache using web console and the value is present in cache. But
still getting false for key "key1" on subsequent containskey execution.

But for key "key2" we were getting true.

Thanks,
Prasad

On Fri 28 Feb, 2020, 10:58 PM Denis Magda  Hi Akash,
>
> Do you execute the cache.contains() method after reading-through the
> record with cache.get()? Do you have any expiration/eviction policies set
> that may purge the record from memory after being loaded from disk?
>
> -
> Denis
>
>
> On Fri, Feb 28, 2020 at 9:11 AM Akash Shinde 
> wrote:
>
>> Hi,
>> I am using Ignite 2.6 version.
>>
>> I have partitioned cache, read-through and write-through is enabled.
>> Back-up count is 1 and total number of server nodes in cluster are  3.
>>
>> When I try to get the data from a cache for a key using cache.get(key)
>> method, ignite reads the value from database using provided cache loader
>> and returns the value by read-through approach.
>>
>> But when I execute cache().containsKey(key) on client node, I get false.
>>
>> But the strange this is this behavior is not same for all keys of the
>> same cache.
>> For key1 I get false but for key2 I get true. But both the keys are
>> present in cache.
>>
>> I executed the SQL on every node (one node at a time) using web-console,
>> I got data present only on one node out of three. This seems to be primary
>> for this particular key.
>>
>> Can someone please advise why this is happening? Is it a bug in ignite?
>> This seems to be a very basic case.
>>
>>
>>
>> Thanks,
>> Akash
>>
>

Re: NodeOrder in GridCacheVersion

[Prasad Bhalerao]

Hi,

My all tasks are affinity tasks, so they are always submitted to primary
node. How setting DIGNITE_READ_LOAD_BALANCING to false help in this case ?
Even if it is true it will always read the values from primary node as the
task is landed on primary node.

Thanks,
Prasad

On Fri, Feb 28, 2020 at 4:45 PM Alexey Goncharuk 
wrote:

> Prasad,
>
> The current version in the entry is checked agains the version which was
> read from the very same entry, so with absence of concurrent updates the
> version will be the same.
>
> From your description, I think there might be a concurrent read for the
> key that you clear which loads the value on primary node with a different
> version. Then, the read happening inside transaction always reads the value
> from backup which leads to a permanent version mismatch; I reproduced this
> scenario locally. The -DIGNITE_READ_LOAD_BALANCING=false fixes the issue at
> a price of disabling read load balancing.
>
> I will create a ticket for the issue shortly, perhaps someone from the
> community will pick it up.
>

Re: NodeOrder in GridCacheVersion

[Prasad Bhalerao]

Hi,

 * How do you ensure that there are no concurrent updates on the keys?
[Prasad]: The cache for which it is failing is kind of bootstrap cache
which changes very rarely. I made sure that I was the only one working on
this system while debugging the issue.
The cache for which it is failing is REPLICATED cache. Read through is
enabled and Write through is disabled. Whenever I get an update message for
these caches from different system, I update an entry in my cache using
following steps.
1. First Remove an entry from the cache using cache.remove() method.
2. Read the entry from cache using cache().get method, which reads the data
from oracle db using read through approach.

 * How many retry attempts did you run?
[Prasad] I retried the transaction 8-10 times.

 * Are your caches in-memory with eviction policy?
[Prasad] Yes, caches are in-memory but without eviction policy. I am using
Oracle DB as a third party persistence. Ignite native persistence is
disabled.

 * Do you have TTL enabled for either of the caches?
[Prasad]: No, I have not set TTL.

 * Do you have a 3rd-party persistence and read-through and write-through
enabled for either of the caches?
[Prasad]: Yes, I have 3rd Party persistence. I have read-through caches but
for all read-through caches write through is disabled. The write-through is
disabled for some caches as I am not the owner of these tables. I also have
Write through caches but for all such caches read-through is disabled. At
this moment I do not have any cache where read-through and write-through
both are enabled. I reload the all my caches using cache loaders.

 * Can you check if the issue reproduces if you set
-DIGNITE_READ_LOAD_BALANCING=false system property?
[Prasad]: Sure will try to reproduce this using this parameter. But the
problem is this happens intermittently.

As per the following code, serReadVer is the GridCache version of
transaction co-ordinator node which it compares with grid cache version of
other nodes.
So as per your explanation, nodeOrder is unique number assigned to each
node joins the grid. So each node in cluster will have a different
nodeOrder. If this is the case then "serReadVer.equals(ver)" will always
return false.
 Please correct me if I am wrong. I just trying to understand the code.
This will help me to identify the issue.






*public boolean checkSerializableReadVersion(GridCacheVersion
serReadVer)throws GridCacheEntryRemovedException {
lockEntry();try {checkObsolete();*

*if (!serReadVer.equals(ver)) {boolean empty =
isStartVersion() || deletedUnlocked();*











*if
(serReadVer.equals(IgniteTxEntry.SER_READ_EMPTY_ENTRY_VER))
return empty;else if
(serReadVer.equals(IgniteTxEntry.SER_READ_NOT_EMPTY_VER))
return !empty;return false;}return
true;}finally {unlockEntry();}}*

Thanks,
Prasad



On Fri, Feb 28, 2020 at 2:24 PM Alexey Goncharuk 
wrote:

> Prasad,
>
>
>> Can you please answer following questions?
>> 1) The significance of the nodeOrder w.r.t Grid and cache?
>>
> Node order is a unique integer assigned to a node when the node joins
> grid. The node order is included into GridCacheVersion to disambiguate
> versions generated on different nodes that happen to have the same local
> version order.
>
>> 2) When does it change?
>>
> Node order does not change during the node lifetime. If two versions have
> different node order, it means that the versions were generated on
> different nodes.
>
>> 3) How it is important w.r.t. transaction?
>>
> GridCacheVersion is used to detect concurrent read-write conflicts as I
> described in the previous message, as well as for data rebalance.
>
>> 4) Inside transaction I am reading and modifying Replicated as well as
>> Partitioned cache. What I observed is this fails for Replicated cache. As
>> workaround, I have moved the code which reads Replicated cache out of
>> transaction block. Is it allowed to read and modify both replicated and
>> Partitioned cache i.e. use both Replicated and Partitioned?
>>
> Yes, this is perfectly fine to update replicated and transactional caches
> inside one transaction.
>
> From the debug output that you provided we can infer that the version of
> both entries have changed for both caches before transaction prepare phase.
> I would back up Alexei here:
>  * How do you ensure that there are no concurrent updates on the keys?
>  * How many retry attempts did you run?
>  * Are your caches in-memory with eviction policy?
>  * Do you have TTL enabled for either of the caches?
>  * Do you have a 3rd-party persistence and read-through and write-through
> enabled for either of the caches?
>  * Can you check if the issue reproduces if you set
> -DIGNITE_READ_LOAD_BALANCING=false system property?
>
> --AG
>

Re: NodeOrder in GridCacheVersion

[Prasad Bhalerao]

Hi Alexey,

Key value is not getting changed concurrently,  I am sure about it. The
cache for which I am getting the exception is kind of bootstrap data and it
changes very rarely. I have added retry logic in my code and it failed
every time giving the same error .

Every time if fails in GridDhtTxPrepareFuture.checkReadConflict ->
GridCacheEntryEx.checkSerializableReadVersion method and I think it fails
due to the change in value of nodeOrder. This is what I observed while
debugging the method.
This happens intermittently.

I got following values while inspecting GridCacheVersion object on
different nodes.

Cache : Addons (Node 2)
serReadVer of entry read inside Transaction: GridCacheVersion
[topVer=194120123, order=4, nodeOrder=2]
version on node3: GridCacheVersion [topVer=194120123, order=4, nodeOrder=1]

Cache : Subscription  (Node 3)
serReadVer of entry read inside Transaction:  GridCacheVersion
[topVer=194120123, order=1, nodeOrder=2]
version on node2:  GridCacheVersion [topVer=194120123, order=1, nodeOrder
=10]

Can you please answer following questions?
1) The significance of the nodeOrder w.r.t Grid and cache?
2) When does it change?
3) How it is important w.r.t. transaction?
4) Inside transaction I am reading and modifying Replicated as well as
Partitioned cache. What I observed is this fails for Replicated cache. As
workaround, I have moved the code which reads Replicated cache out of
transaction block. Is it allowed to read and modify both replicated and
Partitioned cache i.e. use both Replicated and Partitioned?

Thanks,
Prasad

On Thu, Feb 27, 2020 at 6:01 PM Alexey Goncharuk 
wrote:

> Prasad,
>
> Since optimistic transactions do not acquire key locks until prepare
> phase, it is possible that the key value is concurrently changed before the
> prepare commences. Optimistic exceptions is thrown exactly in this case and
> suggest a user that they should retry the transaction.
>
> Consider the following example:
> Thread 1: Start tx 1, Read (key1) -> val1
> Thread 2: Start tx 2, Read (key2) -> val1
>
> Thread 1: Write (key1, val2)
> Thread 1: Commit
>
> Thread 2: Write (key1, val3)
> Thread 2: Commit *Optimistic exception is thrown here since current value
> of key1 is not val1 anymore*
>
> When optimistic transactions are used, a user is expected to have a retry
> logic. Alternatively, a pessimistic repeatable_read transaction can be used
> (one should remember that in pessimistic mode locks are acquired on first
> key access and released only on transaction commit).
>
> Hope this helps,
> --AG
>

Fwd: NodeOrder in GridCacheVersion

[Prasad Bhalerao]

Hi Ilya, didn't get what are you trying to say.

The problem I am facing is, my transaction is failing giving
TransactionOptimisticException. I do not have a reproducer for this project
and this does not happen frequently.
Transaction is failing during prepare phase. I had to open a debug port on
all grid nodes to do remote  debugging in order to debug this issue.
What I observed is transaction fails because check in
GridCacheMapEntry.checkSerializableReadVersion fails as the nodeOrder in
GridCacheVersion in serialized version is different from the actual
noderOrder in GridCacheVersion of respective node. This method returns
false on 2 nodes out 4 nodes and this is happening for Replicated cache.

This is the reason I asked What is nodeOrder in GridCacheVersion and why it
is important while checking read entries in Transaction context?

I tried to debug nodeOrder in ignite code but could not understand it.

Inside transaction I am reading and modifying Replicated as well as
Partitioned cache. What I observed is this fails for Replicated cache. As
workaround, I have moved the code which reads Replicated cache out of
transaction block.
Is it allowed to read and modify both replicated and Partitioned cache i.e.
use both Replicated and Partition?

Complete exception can be found here
<https://gist.github.com/61979329224e23dbaef2f63976a87a14.git>.

Thanks,
Prasad

On Thu, Feb 27, 2020 at 1:00 AM Ilya Kasnacheev 
wrote:

> Hello!
>
> I don't think this is userlist discussion, this logging is not aimed at
> end-user and you are not supposed to act on it.
>
> Do you have any context for us, such as reproducer project or complete
> logs?
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> ср, 26 февр. 2020 г. в 19:13, Prasad Bhalerao <
> prasadbhalerao1...@gmail.com>:
>
>> Can someone please advise?
>>
>> On Wed 26 Feb, 2020, 12:23 AM Prasad Bhalerao <
>> prasadbhalerao1...@gmail.com wrote:
>>
>>> Hi,
>>>
>>>> Ignite Version: 2.6
>>>> No of nodes: 4
>>>>
>>>> I am getting following exception while committing transaction.
>>>>
>>>> Although I just reading the value from this cache inside transaction
>>>> and I am sure that the  cache and "cache entry" read is not being modified
>>>> out this transaction on any other node.
>>>>
>>>> So I debugged the code and found out that it fails in following code on
>>>> 2 nodes out of 4 nodes.
>>>>
>>>> GridDhtTxPrepareFuture#checkReadConflict -
>>>> GridCacheEntryEx#checkSerializableReadVersion
>>>>
>>>> GridCacheVersion version failing for equals check are given below for 2
>>>> different caches. I can see that it failing because of change in nodeOrder
>>>> of cache.
>>>>
>>>> 1) Can some please explain the significance of the nodeOrder w.r.t Grid
>>>> and cache? When does it change?
>>>> 2) How to solve this problem?
>>>>
>>>> Cache : Addons (Node 2)
>>>> serReadVer of entry read inside Transaction: GridCacheVersion
>>>> [topVer=194120123, order=4, nodeOrder=2]
>>>> version on node3: GridCacheVersion [topVer=194120123, order=4,
>>>> nodeOrder=1]
>>>>
>>>> Cache : Subscription  (Node 3)
>>>> serReadVer of entry read inside Transaction:  GridCacheVersion
>>>> [topVer=194120123, order=1, nodeOrder=2]
>>>> version on node2:  GridCacheVersion [topVer=194120123, order=1,
>>>> nodeOrder=10]
>>>>
>>>>
>>>> *EXCEPTION:*
>>>>
>>>> Caused by:
>>>> org.apache.ignite.internal.transactions.IgniteTxOptimisticCheckedException:
>>>> Failed to prepare transaction, read/write conflict
>>>>
>>>
>>>
>>>>
>>>> Thanks,
>>>> Prasad
>>>>
>>>

Message are not going through

[Prasad Bhalerao]

Hi,
I am trying to send an email regarding transaction failure
Whenever I type a text message with logs and exception, my mails are not
showing up in ignite user list as well as in dev list? I must have resent
an email at least 10 time by now but no luck. Is there anyway to check
what's going on?



Thanks,
Prasad

Re: NodeOrder in GridCacheVersion

[Prasad Bhalerao]

Can someone please advise?

On Wed 26 Feb, 2020, 12:23 AM Prasad Bhalerao  Hi,
>
>> Ignite Version: 2.6
>> No of nodes: 4
>>
>> I am getting following exception while committing transaction.
>>
>> Although I just reading the value from this cache inside transaction and
>> I am sure that the  cache and "cache entry" read is not being modified out
>> this transaction on any other node.
>>
>> So I debugged the code and found out that it fails in following code on 2
>> nodes out of 4 nodes.
>>
>> GridDhtTxPrepareFuture#checkReadConflict -
>> GridCacheEntryEx#checkSerializableReadVersion
>>
>> GridCacheVersion version failing for equals check are given below for 2
>> different caches. I can see that it failing because of change in nodeOrder
>> of cache.
>>
>> 1) Can some please explain the significance of the nodeOrder w.r.t Grid
>> and cache? When does it change?
>> 2) How to solve this problem?
>>
>> Cache : Addons (Node 2)
>> serReadVer of entry read inside Transaction: GridCacheVersion
>> [topVer=194120123, order=4, nodeOrder=2]
>> version on node3: GridCacheVersion [topVer=194120123, order=4,
>> nodeOrder=1]
>>
>> Cache : Subscription  (Node 3)
>> serReadVer of entry read inside Transaction:  GridCacheVersion
>> [topVer=194120123, order=1, nodeOrder=2]
>> version on node2:  GridCacheVersion [topVer=194120123, order=1,
>> nodeOrder=10]
>>
>>
>> *EXCEPTION:*
>>
>> Caused by:
>> org.apache.ignite.internal.transactions.IgniteTxOptimisticCheckedException:
>> Failed to prepare transaction, read/write conflict
>>
>
>
>>
>> Thanks,
>> Prasad
>>
>

Re: NodeOrder in GridCacheVersion

[Prasad Bhalerao]

Hi,

> Ignite Version: 2.6
> No of nodes: 4
>
> I am getting following exception while committing transaction.
>
> Although I just reading the value from this cache inside transaction and I
> am sure that the  cache and "cache entry" read is not being modified out
> this transaction on any other node.
>
> So I debugged the code and found out that it fails in following code on 2
> nodes out of 4 nodes.
>
> GridDhtTxPrepareFuture#checkReadConflict -
> GridCacheEntryEx#checkSerializableReadVersion
>
> GridCacheVersion version failing for equals check are given below for 2
> different caches. I can see that it failing because of change in nodeOrder
> of cache.
>
> 1) Can some please explain the significance of the nodeOrder w.r.t Grid
> and cache? When does it change?
> 2) How to solve this problem?
>
> Cache : Addons (Node 2)
> serReadVer of entry read inside Transaction: GridCacheVersion
> [topVer=194120123, order=4, nodeOrder=2]
> version on node3: GridCacheVersion [topVer=194120123, order=4, nodeOrder=1]
>
> Cache : Subscription  (Node 3)
> serReadVer of entry read inside Transaction:  GridCacheVersion
> [topVer=194120123, order=1, nodeOrder=2]
> version on node2:  GridCacheVersion [topVer=194120123, order=1,
> nodeOrder=10]
>
>
> *EXCEPTION:*
>
> Caused by:
> org.apache.ignite.internal.transactions.IgniteTxOptimisticCheckedException:
> Failed to prepare transaction, read/write conflict
>


>
> Thanks,
> Prasad
>

NodeOrder in GridCacheVersion

[Prasad Bhalerao]

Hi,

1) What is nodeOrder in grid cache version?
2) When does it change?
3) How does it affect transaction?

I think my transaction is failing due to this change in nodeorder.

GridCacheVersion [topVer=194120123, order=4, nodeOrder=2]

Thanks,
Prasad

Re: TEST mail

[Prasad Bhalerao]

my mails with body contents are not showing up in ignite user list. Sent
the same mail multiple times but no luck.
Anyone knows the issue?

On Tue, Feb 25, 2020 at 10:45 PM Rick Alexander 
wrote:

> Test MAIL reply
>
> On Tue, Feb 25, 2020, 12:00 PM Prasad Bhalerao <
> prasadbhalerao1...@gmail.com> wrote:
>
>>
>>

TEST mail

[Prasad Bhalerao]

Re: Read through not working as expected in case of Replicated cache

[Prasad Bhalerao]

Hi,

Is this a bug or the cache is designed to work this way?

If it is as-designed, can this behavior be updated in ignite documentation?

Thanks,
Prasad

On Wed, Oct 30, 2019 at 7:19 PM Ilya Kasnacheev 
wrote:

> Hello!
>
> I have discussed this with fellow Ignite developers, and they say read
> through for replicated cache would work where there is either:
>
> - writeThrough enabled and all changes do through it.
> - database contents do not change for already read keys.
>
> I can see that neither is met in your case, so you can expect the behavior
> that you are seeing.
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> вт, 29 окт. 2019 г. в 18:18, Akash Shinde :
>
>> I am using Ignite 2.6 version.
>>
>> I am starting 3 server nodes with a replicated cache and 1 client node.
>> Cache configuration is as follows.
>> Read-through true on but write-through is false. Load data by key is
>> implemented as given below in cache-loader.
>>
>> Steps to reproduce issue:
>> 1) Delete an entry from cache using IgniteCache.remove() method. (Entry
>> is just removed from cache but present in DB as write-through is false)
>> 2) Invoke IgniteCache.get() method for the same key in step 1.
>> 3) Now query the cache from client node. Every invocation returns
>> different results.
>> Sometimes it returns reloaded entry, sometime returns the results
>> without reloaded entry.
>>
>> Looks like read-through is not replicating the reloaded entry on all
>> nodes in case of REPLICATED cache.
>>
>> So to investigate further I changed the cache mode to PARTITIONED and set
>> the backup count to 3 i.e. total number of nodes present in cluster (to
>> mimic REPLICATED behavior).
>> This time it worked as expected.
>> Every invocation returned the same result with reloaded entry.
>>
>> *  private CacheConfiguration networkCacheCfg() {*
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> *CacheConfiguration networkCacheCfg = new
>> CacheConfiguration<>(CacheName.NETWORK_CACHE.name
>> ());
>> networkCacheCfg.setAtomicityMode(CacheAtomicityMode.TRANSACTIONAL);
>> networkCacheCfg.setWriteThrough(false);
>> networkCacheCfg.setReadThrough(true);
>> networkCacheCfg.setRebalanceMode(CacheRebalanceMode.ASYNC);
>> networkCacheCfg.setWriteSynchronizationMode(CacheWriteSynchronizationMode.FULL_SYNC);
>>   //networkCacheCfg.setBackups(3);
>> networkCacheCfg.setCacheMode(CacheMode.REPLICATED);
>> Factory storeFactory =
>> FactoryBuilder.factoryOf(NetworkDataCacheLoader.class);
>> networkCacheCfg.setCacheStoreFactory(storeFactory);
>> networkCacheCfg.setIndexedTypes(DefaultDataAffinityKey.class,
>> NetworkData.class);networkCacheCfg.setSqlIndexMaxInlineSize(65);
>> RendezvousAffinityFunction affinityFunction = new
>> RendezvousAffinityFunction();
>> affinityFunction.setExcludeNeighbors(false);
>> networkCacheCfg.setAffinity(affinityFunction);
>> networkCacheCfg.setStatisticsEnabled(true);   //
>> networkCacheCfg.setNearConfiguration(nearCacheConfiguration());return
>> networkCacheCfg;  }*
>>
>> @Override
>> public V load(K k) throws CacheLoaderException {
>> V value = null;
>> DataSource dataSource = springCtx.getBean(DataSource.class);
>> try (Connection connection = dataSource.getConnection();
>>  PreparedStatement statement = 
>> connection.prepareStatement(loadByKeySql)) {
>> //statement.setObject(1, k.getId());
>> setPreparedStatement(statement,k);
>> try (ResultSet rs = statement.executeQuery()) {
>> if (rs.next()) {
>> value = rowMapper.mapRow(rs, 0);
>> }
>> }
>> } catch (SQLException e) {
>>
>> throw new CacheLoaderException(e.getMessage(), e);
>> }
>>
>> return value;
>> }
>>
>>
>> Thanks,
>>
>> Akash
>>
>>

Re: Loading cache from Oracle Table

[Prasad Bhalerao]

Hi,
I am using partition aware loading to load big tables using 4-8 threads on
each nodes.
I have around 16 caches. 2 caches have 38-40 million entries. It takes
around approx 21 minutes to load the caches on 6 nodes.
I have kept the backup count to 2.

Will share the code tomorrow.

Regards,
Prasad


On Tue 18 Feb, 2020, 6:08 PM nithin91 <
nithinbharadwaj.govindar...@franklintempleton.com wrote:

> Hi Prasad,
>
> Is there a improvement in performance. If so can you please share the
> sample
> code as i am also looking for a similar solution.
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: GridGain Web Console is available free of charge for Apache Ignite

[Prasad Bhalerao]

Hi Denis/Alexey,

We have found few more vulnerabilities in Gridgain Web console and due to
which we can't deploy it in production as it does not comply with FedRAMP
certification.

Can you please provide us the contact where we can send the detailed
vulnerability report and help your team to find and fix the bugs?

Due to some issues we cannot just publish this report on user community.
Can you please advise?


Thanks,
Prasad

On Thu, Dec 12, 2019 at 5:54 PM Alexey Kuznetsov 
wrote:

> Hi,  Prasad
>
> Thanks for reporting this issue.
> Could you describe how I can reproduce these issues locally?
> What tooling I could use?
>
> We need this to check that issues were fixed before next release.
>
> Thanks!
>
> On Tue, Dec 10, 2019 at 3:10 PM Prasad Bhalerao <
> prasadbhalerao1...@gmail.com> wrote:
>
>> Hi,
>>
>> We found 3 vulnerabilities while scanning Grid Gain Web console
>> application.
>>
>> We are using HTTP and not HTTPS due to some issues on our side. Although
>> vulnerabilities are of lower severity, but thought of reporting it here.
>>
>> 1) HTTP TRACE / TRACK Methods Enabled. (CVE-2004-2320
>> <https://nvd.nist.gov/vuln/detail/CVE-2004-2320>, CVE-2010-0386
>> <https://nvd.nist.gov/vuln/detail/CVE-2010-0386>, CVE-2003-1567
>> <https://nvd.nist.gov/vuln/detail/CVE-2003-1567>)
>> 2) Session Cookie Does Not Contain the "Secure" Attribute.
>> 3) Web Server HTTP Trace/Track Method Support Cross-Site Tracing
>> Vulnerability. (CVE-2004-2320
>> <https://nvd.nist.gov/vuln/detail/CVE-2004-2320>, CVE-2007-3008
>> <https://nvd.nist.gov/vuln/detail/CVE-2007-3008>)
>>
>> Can these be fixed?
>>
>> Thanks,
>> Prasad
>>
>>
>> On Tue, Dec 10, 2019 at 4:39 PM Denis Magda  wrote:
>>
>>> It's free software without limitations. Just download and use it.
>>>
>>> -
>>> Denis
>>>
>>>
>>> On Tue, Dec 10, 2019 at 1:21 PM Prasad Bhalerao <
>>> prasadbhalerao1...@gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> Can apache ignite users use it for free in their production
>>>> environments?
>>>> What license does it fall under?
>>>>
>>>> Thanks,
>>>> Prasad
>>>>
>>>> On Fri, Oct 4, 2019 at 5:33 AM Denis Magda  wrote:
>>>>
>>>>> Igniters,
>>>>>
>>>>> There is good news. GridGain made its distribution of Web Console
>>>>> completely free. It goes with advanced monitoring and management
>>>>> dashboard
>>>>> and other handy screens. More details are here:
>>>>>
>>>>> https://www.gridgain.com/resources/blog/gridgain-road-simplicity-new-docs-and-free-tools-apache-ignite
>>>>>
>>>>> -
>>>>> Denis
>>>>>
>>>>
>
> --
> Alexey Kuznetsov
>

Distinct Query too slow

[Prasad Bhalerao]

Hi,

I am using Ignite 2.6 version. I have around total 6 million entries in my
cache.

Following sql is taking too much time to execute. Some times it takes more
than 180 seconds.

This SQL returns 4.5 million entries for given subscriptionId. I tried to
add query parallelism (4-16 threads) on cache configuration. But it did not
help.

If I remove DISTINCT keyword from sql then it executes quickly. But I need
distinct in this particular case.

Can some please advise?

*SQL:*
select distinct ipStart,ipEnd from IpContainerIpV4Data where subscriptionId
= ?

*2020-01-23 06:49:38,249 264738612 [query-#30600%springDataNode%] WARN
o.a.i.i.p.query.h2.IgniteH2Indexing - Query execution is too long
[time=83159 ms, sql='SELECT DISTINCT __Z0.IPSTART __C0_0, __Z0.IPEND __C0_1
FROM IP_CONTAINER_IPV4_CACHE.IPCONTAINERIPV4DATA __Z0 WHERE
__Z0.SUBSCRIPTIONID = ?1', plan= SELECT DISTINCT __Z0.IPSTART AS __C0_0,
__Z0.IPEND AS __C0_1 FROM IP_CONTAINER_IPV4_CACHE.IPCONTAINERIPV4DATA __Z0
/* IP_CONTAINER_IPV4_CACHE.IP_CONTAINER_IPV4_IDX2: SUBSCRIPTIONID = ?1 */
WHERE __Z0.SUBSCRIPTIONID = ?1 , parameters=[1234]]*


*Index is as follows:*

public class IpContainerIpV4Data implements Data,
UpdatableData {

@QuerySqlField
private long id;

@QuerySqlField(orderedGroups = {@QuerySqlField.Group(name =
"ip_container_ipv4_idx1", order = 2)})
private int moduleId;
@QuerySqlField(orderedGroups = {@QuerySqlField.Group(name =
"ip_container_ipv4_idx1", order = 1),
@QuerySqlField.Group(name = "ip_container_ipv4_idx2", order = 0)})
private long subscriptionId;
@QuerySqlField(orderedGroups = {@QuerySqlField.Group(name =
"ip_container_ipv4_idx1", order = 4, descending = true),
@QuerySqlField.Group(name = "ip_container_ipv4_idx2", order = 2, descending
= true)})
private int ipEnd;
@QuerySqlField(orderedGroups = {@QuerySqlField.Group(name =
"ip_container_ipv4_idx1", order = 3),
@QuerySqlField.Group(name = "ip_container_ipv4_idx2", order = 1)})
private int ipStart;
@QuerySqlField
private int partitionId;
@QuerySqlField
private long updatedDate;

}


*Cache Configuration:*

private CacheConfiguration ipContainerIPV4CacheCfg() {

  CacheConfiguration ipContainerIpV4CacheCfg = new
CacheConfiguration<>(CacheName.IP_CONTAINER_IPV4_CACHE.name());
  ipContainerIpV4CacheCfg.setAtomicityMode(CacheAtomicityMode.TRANSACTIONAL);
  ipContainerIpV4CacheCfg.setWriteThrough(ENABLE_WRITE_THROUGH);
  ipContainerIpV4CacheCfg.setReadThrough(false);
  ipContainerIpV4CacheCfg.setRebalanceMode(CacheRebalanceMode.ASYNC);
  
ipContainerIpV4CacheCfg.setWriteSynchronizationMode(CacheWriteSynchronizationMode.FULL_SYNC);
  ipContainerIpV4CacheCfg.setBackups(this.backupCount);
  Factory storeFactory =
FactoryBuilder.factoryOf(IpContainerIpV4CacheStore.class);
  ipContainerIpV4CacheCfg.setCacheStoreFactory(storeFactory);
  ipContainerIpV4CacheCfg.setIndexedTypes(DefaultDataAffinityKey.class,
IpContainerIpV4Data.class);
  
ipContainerIpV4CacheCfg.setCacheStoreSessionListenerFactories(cacheStoreSessionListenerFactory());
  ipContainerIpV4CacheCfg.*setSqlIndexMaxInlineSize(84);*
  RendezvousAffinityFunction affinityFunction = new
RendezvousAffinityFunction();
  affinityFunction.setExcludeNeighbors(true);
  ipContainerIpV4CacheCfg.setAffinity(affinityFunction);
  ipContainerIpV4CacheCfg.setStatisticsEnabled(true);
  
ipContainerIpV4CacheCfg.setPartitionLossPolicy(PartitionLossPolicy.READ_WRITE_SAFE);

   ipContainerIpV4CacheCfg.setQueryParallelism(4);

  return ipContainerIpV4CacheCfg;
}


Thanks,
Prasad

Re: Issue with replicated cache

[Prasad Bhalerao]

I used cache.remove(key) method to delete an entry from cache.

Basically I  was not getting the consistent result on subsequent  API calls
with the same input.

So I used grid gain console to query the cache. I executed the SQL on
single node at a time.
While doing this I found data only on node n1. But same entry was not
present on nodes n2,n3,n4.

Thanks,
Prasad




On Thu 26 Dec, 2019, 11:09 PM Denis Magda  Hello Prasad,
>
> What APIs did you use to remove the entry from the cache and what method
> did you use to confirm that the entry still exists on some of the nodes?
>
> -
> Denis
>
>
> On Thu, Dec 26, 2019 at 8:54 AM Prasad Bhalerao <
> prasadbhalerao1...@gmail.com> wrote:
>
>> Hi,
>>
>> I am using ignite 2.6.0 version and the time out settings are as follows.
>>
>> IgniteConfiguration cfg = new IgniteConfiguration();
>> cfg.setFailureDetectionTimeout(12);
>> cfg.setNetworkTimeout(1);
>> cfg.setClientFailureDetectionTimeout(12);
>>
>> I have 4 server nodes (n1,n2,n3,n4) and 6 client nodes. I am using a
>> replicated cache and cache configuration is as shown below.
>> As you can see write-through is false, read through is true and write
>> synchronization mode is FULL_SYNC.
>>
>> I got an issue, a network entry was removed from network cache but some
>> how it was removed from only 3 server nodes (n2,n3,n4). I was able to see
>> the network entry on node n1 consistently for a day(when it was removed).
>> So I checked the logs for any errors/warnings but I could not find any.
>> I did not see any segmentation issue in logs, looked like cluster was in
>> healthy state.
>> When I checked the cache after 2 days, I could not find that entry. Cache
>> was in a state as it was supposed to be.  Servers were  not stopped and
>> restarted during this whole time.
>>
>> Some how I am not able to reproduce this issue on dev env.
>>
>> Is there any way to investigate/debug this issue? Can someone please
>> advise?
>>
>> private CacheConfiguration networkCacheCfg() {
>>   CacheConfiguration networkCacheCfg = new 
>> CacheConfiguration<>(CacheName.NETWORK_CACHE.name());
>>   networkCacheCfg.setAtomicityMode(CacheAtomicityMode.ATOMIC);
>>   networkCacheCfg.setWriteThrough(false);
>>   networkCacheCfg.setReadThrough(true);
>>   networkCacheCfg.setRebalanceMode(CacheRebalanceMode.ASYNC);
>>   
>> networkCacheCfg.setWriteSynchronizationMode(CacheWriteSynchronizationMode.FULL_SYNC);
>>   networkCacheCfg.setBackups(this.backupCount);
>>   networkCacheCfg.setCacheMode(CacheMode.REPLICATED);
>>   Factory storeFactory = 
>> FactoryBuilder.factoryOf(NetworkDataCacheLoader.class);
>>   networkCacheCfg.setCacheStoreFactory(storeFactory);
>>   networkCacheCfg.setIndexedTypes(DefaultDataAffinityKey.class, 
>> NetworkData.class);
>>   networkCacheCfg.setSqlIndexMaxInlineSize(65);
>>   RendezvousAffinityFunction affinityFunction = new 
>> RendezvousAffinityFunction();
>>   affinityFunction.setExcludeNeighbors(true);
>>   networkCacheCfg.setAffinity(affinityFunction);
>>   networkCacheCfg.setStatisticsEnabled(true);
>>
>>   return networkCacheCfg;
>> }
>>
>>
>>
>> Thanks,
>> PRasad
>>
>>

Issue with replicated cache

[Prasad Bhalerao]

Hi,

I am using ignite 2.6.0 version and the time out settings are as follows.

IgniteConfiguration cfg = new IgniteConfiguration();
cfg.setFailureDetectionTimeout(12);
cfg.setNetworkTimeout(1);
cfg.setClientFailureDetectionTimeout(12);

I have 4 server nodes (n1,n2,n3,n4) and 6 client nodes. I am using a
replicated cache and cache configuration is as shown below.
As you can see write-through is false, read through is true and write
synchronization mode is FULL_SYNC.

I got an issue, a network entry was removed from network cache but some how
it was removed from only 3 server nodes (n2,n3,n4). I was able to see the
network entry on node n1 consistently for a day(when it was removed). So I
checked the logs for any errors/warnings but I could not find any.
I did not see any segmentation issue in logs, looked like cluster was in
healthy state.
When I checked the cache after 2 days, I could not find that entry. Cache
was in a state as it was supposed to be.  Servers were  not stopped and
restarted during this whole time.

Some how I am not able to reproduce this issue on dev env.

Is there any way to investigate/debug this issue? Can someone please advise?

private CacheConfiguration networkCacheCfg() {
  CacheConfiguration networkCacheCfg = new
CacheConfiguration<>(CacheName.NETWORK_CACHE.name());
  networkCacheCfg.setAtomicityMode(CacheAtomicityMode.ATOMIC);
  networkCacheCfg.setWriteThrough(false);
  networkCacheCfg.setReadThrough(true);
  networkCacheCfg.setRebalanceMode(CacheRebalanceMode.ASYNC);
  
networkCacheCfg.setWriteSynchronizationMode(CacheWriteSynchronizationMode.FULL_SYNC);
  networkCacheCfg.setBackups(this.backupCount);
  networkCacheCfg.setCacheMode(CacheMode.REPLICATED);
  Factory storeFactory =
FactoryBuilder.factoryOf(NetworkDataCacheLoader.class);
  networkCacheCfg.setCacheStoreFactory(storeFactory);
  networkCacheCfg.setIndexedTypes(DefaultDataAffinityKey.class,
NetworkData.class);
  networkCacheCfg.setSqlIndexMaxInlineSize(65);
  RendezvousAffinityFunction affinityFunction = new
RendezvousAffinityFunction();
  affinityFunction.setExcludeNeighbors(true);
  networkCacheCfg.setAffinity(affinityFunction);
  networkCacheCfg.setStatisticsEnabled(true);

  return networkCacheCfg;
}



Thanks,
PRasad

Re: Are data in NearCache BinaryObject?

[Prasad Bhalerao]

I had debugged the near cache once and I found that near cache simply
stores the entries concurrent hashmap in deserialized format.

But be careful with near cache usage. I faced many issues and finally
removed it.
I had reported an issue on this forum but I couldn't not create reproducer
for it but it used give give me exceptions in running application. Check
near cache issue on user list.


On Thu 12 Dec, 2019, 9:09 PM Cong Guo  Hi,
>
> My application needs to read all entries in the cache frequently. The
> entries may be updated by others. I'm thinking about two solutions to avoid
> a lot deserialization. First, I can maintain my own local hash map and
> relies on continuous queries to get the update events. Second, I can use a
> NearCache, but if the data in NearCache are still serialized, this method
> does not work for my application.
>
> Thanks,
> Nap
>
> On Thu, Dec 12, 2019 at 5:37 AM Ilya Kasnacheev 
> wrote:
>
>> Hello!
>>
>> It is actually hard to say without debugging. I expect that it is
>> BinaryObject or primitive type or byte[].
>>
>> It is possible to enable onheap caching, in this case objects will be
>> held as is, and also sed copyOnRead=false, in this case objects will not
>> even be copied.
>> However, I'm not sure if Near Cache will interact with onheap caching.
>>
>> Why does it matter for your use case?
>>
>> Regards,
>> --
>> Ilya Kasnacheev
>>
>>
>> ср, 11 дек. 2019 г. в 22:54, Cong Guo :
>>
>>> Hi,
>>>
>>> Are the entries stored in local NearCache on my client node in the
>>> format of deserialized java objects or BinaryObject? Will the entry in
>>> local on-heap NearCache be deserialized from BinaryObject when I call the
>>> get function?
>>>
>>> Thanks,
>>> Nap
>>>
>>

Re: Topology version changing very frequently

[Prasad Bhalerao]

Can you please try to answer question 1, 2  and 3?

In my logs I could see that number of server are also changing. Topology
version changes when server node becomes 3 and again becomes 4. Total
number of server nodes are 4.

Thanks,
Prasad


On Tue 10 Dec, 2019, 11:45 PM akurbanov  Hello,
>
> Are you able to share full logs from the server and client instances?
>
> In short: by default clients can reconnect to the cluster after the network
> connection was disrupted by anything (network issues, gc pauses etc.).
>
>
> https://apacheignite.readme.io/docs/clients-vs-servers#section-reconnecting-a-client
>
> Server drops client node from topology once failure detection timeout is
> reached, if you want a client to be stopped and segmented in this case, use
> property clientReconnectDisabled and set it to true, the sample is on the
> documentation page.
>
> Network timeout defines the timeout for different network operations, among
> the usages are client reconnect attempt timeout, connection establishing
> timeout on join attempt and sending messages.
>
> Best regards,
> Anton
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: GridGain Web Console is available free of charge for Apache Ignite

[Prasad Bhalerao]

Hi,

We found 3 vulnerabilities while scanning Grid Gain Web console application.

We are using HTTP and not HTTPS due to some issues on our side. Although
vulnerabilities are of lower severity, but thought of reporting it here.

1) HTTP TRACE / TRACK Methods Enabled. (CVE-2004-2320
<https://nvd.nist.gov/vuln/detail/CVE-2004-2320>, CVE-2010-0386
<https://nvd.nist.gov/vuln/detail/CVE-2010-0386>, CVE-2003-1567
<https://nvd.nist.gov/vuln/detail/CVE-2003-1567>)
2) Session Cookie Does Not Contain the "Secure" Attribute.
3) Web Server HTTP Trace/Track Method Support Cross-Site Tracing
Vulnerability. (CVE-2004-2320
<https://nvd.nist.gov/vuln/detail/CVE-2004-2320>, CVE-2007-3008
<https://nvd.nist.gov/vuln/detail/CVE-2007-3008>)

Can these be fixed?

Thanks,
Prasad


On Tue, Dec 10, 2019 at 4:39 PM Denis Magda  wrote:

> It's free software without limitations. Just download and use it.
>
> -
> Denis
>
>
> On Tue, Dec 10, 2019 at 1:21 PM Prasad Bhalerao <
> prasadbhalerao1...@gmail.com> wrote:
>
>> Hi,
>>
>> Can apache ignite users use it for free in their production environments?
>> What license does it fall under?
>>
>> Thanks,
>> Prasad
>>
>> On Fri, Oct 4, 2019 at 5:33 AM Denis Magda  wrote:
>>
>>> Igniters,
>>>
>>> There is good news. GridGain made its distribution of Web Console
>>> completely free. It goes with advanced monitoring and management
>>> dashboard
>>> and other handy screens. More details are here:
>>>
>>> https://www.gridgain.com/resources/blog/gridgain-road-simplicity-new-docs-and-free-tools-apache-ignite
>>>
>>> -
>>> Denis
>>>
>>

Re: GridGain Web Console is available free of charge for Apache Ignite

[Prasad Bhalerao]

Hi,

Can apache ignite users use it for free in their production environments?
What license does it fall under?

Thanks,
Prasad

On Fri, Oct 4, 2019 at 5:33 AM Denis Magda  wrote:

> Igniters,
>
> There is good news. GridGain made its distribution of Web Console
> completely free. It goes with advanced monitoring and management dashboard
> and other handy screens. More details are here:
>
> https://www.gridgain.com/resources/blog/gridgain-road-simplicity-new-docs-and-free-tools-apache-ignite
>
> -
> Denis
>

Topology version changing very frequently

[Prasad Bhalerao]

My application nodes are running on VMs and observing that topology version
is changing very frequently but I do not see any error/exceptions/warnings
in log files.

For ignite I have kept the log level to WARN.

Timeout settings in millis are as follows:

FailureDetectionTimeout = 12
NetworkTimeout = 1
ClientFailureDetectionTimeout = 12

1) Does the topology version change when node becomes unreachable and
becomes reachable again within the failure detection time out?

2) Which time out setting does ignite use  while deciding that topology
version is changed?

3) Why the node is not getting segmented in this case? Is it because the
node is becoming reachable within failure detection timeout?

4) What is the purpose of network timeout ?


*App Logs::*



*Topology snapshot [ver=18, servers=4, clients=6, CPUs=172, offheap=40.0GB,
heap=160.0GB]Topology snapshot [ver=19, servers=4, clients=5, CPUs=152,
offheap=40.0GB, heap=140.0GB]Topology snapshot [ver=20, servers=4,
clients=4, CPUs=132, offheap=40.0GB, heap=130.0GB] *
*...*
*..*




*.Topology snapshot [ver=56, servers=4, clients=4, CPUs=132,
offheap=40.0GB, heap=130.0GB]Topology snapshot [ver=57, servers=4,
clients=5, CPUs=152, offheap=40.0GB, heap=140.0GB]Topology snapshot
[ver=58, servers=4, clients=4, CPUs=132, offheap=40.0GB,
heap=130.0GB]Topology snapshot [ver=59, servers=4, clients=5, CPUs=152,
offheap=40.0GB, heap=140.0GB] *

Re: Local node terminated after segmentation

[Prasad Bhalerao]

> Can someone please advise on this?
>
> -- Forwarded message -
> From: Prasad Bhalerao 
> Date: Fri, Nov 29, 2019 at 7:53 AM
> Subject: Re: Local node terminated after segmentation
> To: 
>
>
> I had checked the resource you mentioned, but I was confused with
> grid-gain doc  describing it as protection against split-brain. Because if
> the node is segmented the only thing one can do is stop/restart/noop.
> I was just wondering how it provides protection against split-brain.
> Now I think by protection it means kill the segmented node/nodes or
> restart it and bring it back in the cluster .
>
> Ignite uses TcpDiscoverSpi to send a heartbeat the next node in the ring
> right to check if the node is reachable or not.
> So the question in what situation one needs one more ways to check if the
> node is reachable or not using different resolvers?
>
> Please let me know if my understanding is correct.
>
> The article you mentioned, I had checked that code. It requires a node to
> be configured in advance so that resolver can check if that node is
> reachable from local host. It doesn't not check if all the nodes are
> reachable from local host.
>
> Eg: node1 will check for node2 and node2 will check for node 3 and node 3
> will check for node1 to complete the ring
> Just wondering how to configure this plugin in prod env with large cluster.
> I tried to check grid-gain doc to see if they have provided any sample
> code to configure their plugins just to get an idea but did not find any.
>
> Can you please advise?
>
>
> Thanks,
> Prasad
>
> On Thu 28 Nov, 2019, 11:41 PM akurbanov 
>> Hello,
>>
>> Basically this is a mechanism to implement custom logical/network
>> split-brain protection. Segmentation resolvers allow you to implement a
>> way
>> to determine if node has to be segmented/stopped/etc in method
>> isValidSegment() and possibly use different combinations of resolvers
>> within
>> processor.
>>
>> If you want to check out how it could be done, some articles/source
>> samples
>> that might give you a good insight may be easily found on the web, like:
>>
>> https://medium.com/@aamargajbhiye/how-to-handle-network-segmentation-in-apache-ignite-35dc5fa6f239
>>
>> http://apache-ignite-users.70518.x6.nabble.com/Segmentation-Plugin-blog-or-article-td27955.html
>>
>> 2-3 are described in the documentation, copying the link just to point out
>> which one: https://apacheignite.readme.io/docs/critical-failures-handling
>>
>> By default answer to 2 is: Ignite doesn't ignote node FailureType
>> SEGMENTATION and calls the failure handler in this case. Actions that are
>> taken are defined in failure handler.
>>
>> AbstractFailureHandler class has only SYSTEM_WORKER_BLOCKED and
>> SYSTEM_CRITICAL_OPERATION_TIMEOUT ignored by default. However, you might
>> override the failure handler and call .setIgnoredFailureTypes().
>>
>> Links:
>> Extend this class:
>>
>> https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/failure/AbstractFailureHandler.java
>> — check for custom implementations used in Ignite tests and how they are
>> used.
>>
>> Sample from tests:
>>
>> https://github.com/apache/ignite/blob/master/modules/core/src/test/java/org/apache/ignite/failure/SystemWorkersBlockingTest.java
>>
>> Failure processor:
>>
>> https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/internal/processors/failure/FailureProcessor.java
>>
>> Best regards,
>> Anton
>>
>>
>>
>>
>>
>> --
>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>
>

Re: Topology version in ignite

[Prasad Bhalerao]

My bad... please ignore this mail... I did not see that clients are
fluctuating

On Mon, Dec 9, 2019 at 5:38 PM Prasad Bhalerao 
wrote:

> Hi,
>
> My application nodes are running on VMs and observing that topology
> version is changing very frequently even though the number of server and
> client nodes are same.
>
> I am observing that the topology version is increasing with change in heap
> size and number of CPUs.
>
> I was under the impression that topology version changes only if the
> number of servers or clients in the given cluster changes. But looks like
> this is not correct.
>
> Can someone please explain what all the things causes topology version to
> change?
>
> *App Logs::*
>
>
>
> *Topology snapshot [ver=18, servers=4, clients=6, CPUs=172,
> offheap=40.0GB, heap=160.0GB]Topology snapshot [ver=19, servers=4,
> clients=5, CPUs=152, offheap=40.0GB, heap=140.0GB]Topology snapshot
> [ver=20, servers=4, clients=4, CPUs=132, offheap=40.0GB, heap=130.0GB] *
> *...*
> *..*
>
>
>
>
> *.Topology snapshot [ver=56, servers=4, clients=4, CPUs=132,
> offheap=40.0GB, heap=130.0GB]Topology snapshot [ver=57, servers=4,
> clients=5, CPUs=152, offheap=40.0GB, heap=140.0GB]Topology snapshot
> [ver=58, servers=4, clients=4, CPUs=132, offheap=40.0GB,
> heap=130.0GB]Topology snapshot [ver=59, servers=4, clients=5, CPUs=152,
> offheap=40.0GB, heap=140.0GB] *
>
> *Thanks,*
> *Prasad*
>

Topology version in ignite

[Prasad Bhalerao]

Hi,

My application nodes are running on VMs and observing that topology version
is changing very frequently even though the number of server and client
nodes are same.

I am observing that the topology version is increasing with change in heap
size and number of CPUs.

I was under the impression that topology version changes only if the number
of servers or clients in the given cluster changes. But looks like this is
not correct.

Can someone please explain what all the things causes topology version to
change?

*App Logs::*

Topology snapshot [ver=18, servers=4, clients=6, CPUs=172, offheap=40.0GB,
heap=160.0GB]
Topology snapshot [ver=19, servers=4, clients=5, CPUs=152, offheap=40.0GB,
heap=140.0GB]
Topology snapshot [ver=20, servers=4, clients=4, CPUs=132, offheap=40.0GB,
heap=130.0GB]
...
..
.
Topology snapshot [ver=56, servers=4, clients=4, CPUs=132, offheap=40.0GB,
heap=130.0GB]
Topology snapshot [ver=57, servers=4, clients=5, CPUs=152, offheap=40.0GB,
heap=140.0GB]
Topology snapshot [ver=58, servers=4, clients=4, CPUs=132, offheap=40.0GB,
heap=130.0GB]
Topology snapshot [ver=59, servers=4, clients=5, CPUs=152, offheap=40.0GB,
heap=140.0GB]

Thanks,
Prasad

Fwd: Local node terminated after segmentation

[Prasad Bhalerao]

Can someone please advise on this?

-- Forwarded message -
From: Prasad Bhalerao 
Date: Fri, Nov 29, 2019 at 7:53 AM
Subject: Re: Local node terminated after segmentation
To: 


I had checked the resource you mentioned, but I was confused with grid-gain
doc  describing it as protection against split-brain. Because if the node
is segmented the only thing one can do is stop/restart/noop.
I was just wondering how it provides protection against split-brain.
Now I think by protection it means kill the segmented node/nodes or restart
it and bring it back in the cluster .

Ignite uses TcpDiscoverSpi to send a heartbeat the next node in the ring
right to check if the node is reachable or not.
So the question in what situation one needs one more ways to check if the
node is reachable or not using different resolvers?

Please let me know if my understanding is correct.

The article you mentioned, I had checked that code. It requires a node to
be configured in advance so that resolver can check if that node is
reachable from local host. It doesn't not check if all the nodes are
reachable from local host.

Eg: node1 will check for node2 and node2 will check for node 3 and node 3
will check for node1 to complete the ring
Just wondering how to configure this plugin in prod env with large cluster.
I tried to check grid-gain doc to see if they have provided any sample code
to configure their plugins just to get an idea but did not find any.

Can you please advise?


Thanks,
Prasad

On Thu 28 Nov, 2019, 11:41 PM akurbanov  Hello,
>
> Basically this is a mechanism to implement custom logical/network
> split-brain protection. Segmentation resolvers allow you to implement a way
> to determine if node has to be segmented/stopped/etc in method
> isValidSegment() and possibly use different combinations of resolvers
> within
> processor.
>
> If you want to check out how it could be done, some articles/source samples
> that might give you a good insight may be easily found on the web, like:
>
> https://medium.com/@aamargajbhiye/how-to-handle-network-segmentation-in-apache-ignite-35dc5fa6f239
>
> http://apache-ignite-users.70518.x6.nabble.com/Segmentation-Plugin-blog-or-article-td27955.html
>
> 2-3 are described in the documentation, copying the link just to point out
> which one: https://apacheignite.readme.io/docs/critical-failures-handling
>
> By default answer to 2 is: Ignite doesn't ignote node FailureType
> SEGMENTATION and calls the failure handler in this case. Actions that are
> taken are defined in failure handler.
>
> AbstractFailureHandler class has only SYSTEM_WORKER_BLOCKED and
> SYSTEM_CRITICAL_OPERATION_TIMEOUT ignored by default. However, you might
> override the failure handler and call .setIgnoredFailureTypes().
>
> Links:
> Extend this class:
>
> https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/failure/AbstractFailureHandler.java
> — check for custom implementations used in Ignite tests and how they are
> used.
>
> Sample from tests:
>
> https://github.com/apache/ignite/blob/master/modules/core/src/test/java/org/apache/ignite/failure/SystemWorkersBlockingTest.java
>
> Failure processor:
>
> https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/internal/processors/failure/FailureProcessor.java
>
> Best regards,
> Anton
>
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Active tasks in cluster

[Prasad Bhalerao]

> Does cancelling an active task cancels the threads being used internally
> for query execution (threads in query pool) or cache put/delete operation?
>
> Thanks,
> Prasad
>
> On Sat, Nov 23, 2019 at 8:26 PM Mikael  wrote:
>
>> The whole idea with a future is that it is a small lightweight compact
>> object, and you still have Igor's suggestion:
>>
>> Collection>> result =
>> ignite.compute().broadcast(() -> ignite.compute().activeTaskFutures());
>>
>> If you would have to implement a cluster wide listening mechanism in the
>> futures you would add a terrible amount of overhead to it, and you would
>> cause a lot of problems, what if you try to deserialize a future on a
>> computer that is in another cluster, it may not even be an Ignite
>> application, what if you deserialize a future that was created 2 years ago
>> and the "id" of the future is now being reused for another future that has
>> nothing to do with the original one, what if you deserialize it in a
>> different cluster where that id is something different and not the same you
>> submitted on the other cluster, yes all these things can be handled but
>> once again you would turn a small nice simple object into a complex beast.
>> Den 2019-11-23 kl. 15:00, skrev Prasad Bhalerao:
>>
>> By member you meant the output of the thread right?
>>
>> If yes, can we keep the member at centralised location like an internal
>> cache?
>> (May be we can provide the flag if turned on then the member can be
>> broadcasted to whoever is listening to it or centralised cache location)
>> I am considering future as a handle to the task which can be used to
>> cancel the task even if the submitter node goes down.
>>
>>
>>
>> On Sat 23 Nov, 2019, 7:21 PM Mikael >
>>> Well, the thread has to set a member in the future when it is finished,
>>> if you serialize the future and send it somewhere else, how is the thread
>>> going to be able to tell the future it had finished ?
>>> Den 2019-11-23 kl. 14:31, skrev Prasad Bhalerao:
>>>
>>> Can someone please explain why Active task futures can't be serialized?
>>>
>>> If we loose the future then we don't have the way to cancel the active
>>> task if it's taking too long. I think this is important feature.
>>>
>>>
>>>
>>> Thanks,
>>> Prasad
>>>
>>> On Thu 21 Nov, 2019, 5:16 AM Denis Magda >>
>>>> I think that you should broadcast another task that will simply ask
>>>> every node if taskA is already running or not every time the topology
>>>> changes. If the response from all the nodes is empty then you need to
>>>> reschedule taskA, otherwise, you will skip this procedure.
>>>>
>>>> -
>>>> Denis
>>>>
>>>>
>>>> On Wed, Nov 20, 2019 at 9:28 AM Prasad Bhalerao <
>>>> prasadbhalerao1...@gmail.com> wrote:
>>>>
>>>>> That means I can't do this..
>>>>>
>>>>> Collection>> result =
>>>>> ignite.compute().broadcast(() -> ignite.compute().activeTaskFutures());
>>>>>
>>>>> Is there any way to get list futures of all active tasks running on
>>>>> all nodes of the cluster?
>>>>>
>>>>> Thanks,
>>>>> Prasad
>>>>>
>>>>>
>>>>> On Wed 20 Nov, 2019, 10:51 PM Mikael >>>>
>>>>>> Hi!
>>>>>>
>>>>>> No you cannot serialize any future object.
>>>>>>
>>>>>> Mikael
>>>>>>
>>>>>>
>>>>>> Den 2019-11-20 kl. 17:59, skrev Prasad Bhalerao:
>>>>>>
>>>>>> Thank you for the suggestion. I will try this.
>>>>>>
>>>>>> I am thinking to store the task future object in a (replicated)cache
>>>>>> against a jobId. If the node goes down as described in case (b), I will 
>>>>>> get
>>>>>> the task's future object from this  cache using a jobId and will invoke 
>>>>>> the
>>>>>> get method on it.
>>>>>>
>>>>>> But I am not sure about this approach, whether a future object can be
>>>>>> serialized and send it over the wire to another node and deserialize it 
>>>>>> and
>>>>>> then invoke the get API on it.
>>>>>>
>>>>>> 

Re: Local node terminated after segmentation

[Prasad Bhalerao]

I had checked the resource you mentioned, but I was confused with grid-gain
doc  describing it as protection against split-brain. Because if the node
is segmented the only thing one can do is stop/restart/noop.
I was just wondering how it provides protection against split-brain.
Now I think by protection it means kill the segmented node/nodes or restart
it and bring it back in the cluster .

Ignite uses TcpDiscoverSpi to send a heartbeat the next node in the ring
right to check if the node is reachable or not.
So the question in what situation one needs one more ways to check if the
node is reachable or not using different resolvers?

Please let me know if my understanding is correct.

The article you mentioned, I had checked that code. It requires a node to
be configured in advance so that resolver can check if that node is
reachable from local host. It doesn't not check if all the nodes are
reachable from local host.

Eg: node1 will check for node2 and node2 will check for node 3 and node 3
will check for node1 to complete the ring
Just wondering how to configure this plugin in prod env with large cluster.
I tried to check grid-gain doc to see if they have provided any sample code
to configure their plugins just to get an idea but did not find any.

Can you please advise?


Thanks,
Prasad

On Thu 28 Nov, 2019, 11:41 PM akurbanov  Hello,
>
> Basically this is a mechanism to implement custom logical/network
> split-brain protection. Segmentation resolvers allow you to implement a way
> to determine if node has to be segmented/stopped/etc in method
> isValidSegment() and possibly use different combinations of resolvers
> within
> processor.
>
> If you want to check out how it could be done, some articles/source samples
> that might give you a good insight may be easily found on the web, like:
>
> https://medium.com/@aamargajbhiye/how-to-handle-network-segmentation-in-apache-ignite-35dc5fa6f239
>
> http://apache-ignite-users.70518.x6.nabble.com/Segmentation-Plugin-blog-or-article-td27955.html
>
> 2-3 are described in the documentation, copying the link just to point out
> which one: https://apacheignite.readme.io/docs/critical-failures-handling
>
> By default answer to 2 is: Ignite doesn't ignote node FailureType
> SEGMENTATION and calls the failure handler in this case. Actions that are
> taken are defined in failure handler.
>
> AbstractFailureHandler class has only SYSTEM_WORKER_BLOCKED and
> SYSTEM_CRITICAL_OPERATION_TIMEOUT ignored by default. However, you might
> override the failure handler and call .setIgnoredFailureTypes().
>
> Links:
> Extend this class:
>
> https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/failure/AbstractFailureHandler.java
> — check for custom implementations used in Ignite tests and how they are
> used.
>
> Sample from tests:
>
> https://github.com/apache/ignite/blob/master/modules/core/src/test/java/org/apache/ignite/failure/SystemWorkersBlockingTest.java
>
> Failure processor:
>
> https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/internal/processors/failure/FailureProcessor.java
>
> Best regards,
> Anton
>
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Local node terminated after segmentation

[Prasad Bhalerao]

Hi,

Can someone please help me out with following questions.

1) If the ignite is capable of detecting nodes segmentation and taking
STOP,RESTART_JVM or NOOP action based on configured failure handlers then
why do we need explicit SegmentationResolvers?

2) Does ignite always treat node segmentation as "Critical system error"
and use "StopNodeOrHaltFailureHandler" to take required action which
"Teminate Node"?

3) Are there any other reasons for   "Critical system error detected"
message?

Thanks,
Prasad




On Wed, Nov 27, 2019 at 11:01 PM akurbanov  wrote:

> Hello,
>
> Please refer to documentation on failure handler:
> https://apacheignite.readme.io/docs/critical-failures-handling.
>
> As it is correctly stated, we cannot restart the JVM without external
> tooling, by default we are doing this for nodes that were started with
> ignite.sh/bat so that Ignite start goes through
>
> https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/startup/cmdline/CommandLineStartup.java
>
> As for the segmentation, subscribe to
>
> https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/events/EventType.html#EVT_NODE_SEGMENTED
>
> Event listeners doc: https://apacheignite.readme.io/docs/events
>
> You will receive this event in the listener and after this you might do
> anything that you want with the JVM, easiest way is to exit JVM with some
> code and handle it outside of the application.
>
> Best regards,
> Anton
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Local node terminated after segmentation

[Prasad Bhalerao]

Why did you have to extend stop handler?
Why couldn't you use the existing one provided in ignite?

Btw the question is about how to restart the JVM? JVM can't restart itself
without taking outside tool/scripts help.

Thanks,
Prasad


On Wed 27 Nov, 2019, 9:43 PM Surinder Mehra  You can extend the stopnode handler and stop Java process when
> stopnodehandler is called by ignite.
> We did similar thing in our project
>
> On Wed, Nov 27, 2019, 17:00 Akash Shinde  wrote:
>
>> Hi ,
>>
>> I have started four server nodes. One of the node got terminated
>> unexpectedly giving following error. Before terminating the JVM the node
>> was segmented.
>>
>> 1) Does ignite always treat node segmentation as "Critical system error"
>> and use "StopNodeOrHaltFailureHandler" to take required action which
>> "Teminate Node" in this case?
>>
>> 2) Are there any other reasons for   "Critical system error detected"
>> message?
>>
>> I have not set the SegmentationPolicy  explicitly.  AFAIK ignite does not
>> provide SegmentationResolver and SegmentationPolicy out of box.
>>
>> 3) Do I need to implement SegmentationResolver and set the
>> SegmenetationPolicy to "STOP" if I want to stop the JVM if the node is
>> segmented?
>>
>> 4) I am starting Ignite in embedded mode. When a node is segmented  I
>> want restart the JVM. I
>> Is there any way to do this? (I am not using ignite.sh/ignite.bat) to
>> start the ignite.
>>
>> Please find attached logs.
>>
>> Exception:
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> *2019-11-27 08:30:46,992 9321188 [disco-event-worker-#61%springDataNode%]
>> WARN  o.a.i.i.m.d.GridDiscoveryManager - Local node SEGMENTED:
>> TcpDiscoveryNode [id=b4fce076-cc7a-47ee-98fd-31e1d610b5de,
>> addrs=[10.45.65.97, 127.0.0.1], sockAddrs=[/10.45.65.97:47500
>> , /127.0.0.1:47500 ],
>> discPort=47500, order=1, intOrder=1, lastExchangeTime=1574843446983,
>> loc=true, ver=2.6.0#20180710-sha1:669feacc, isClient=false]2019-11-27
>> 08:30:46,992 9321188 [disco-event-worker-#61%springDataNode%] WARN
>>  o.a.i.i.m.d.GridDiscoveryManager - Local node SEGMENTED: TcpDiscoveryNode
>> [id=b4fce076-cc7a-47ee-98fd-31e1d610b5de, addrs=[10.45.65.97, 127.0.0.1],
>> sockAddrs=[/10.45.65.97:47500 , /127.0.0.1:47500
>> ], discPort=47500, order=1, intOrder=1,
>> lastExchangeTime=1574843446983, loc=true, ver=2.6.0#20180710-sha1:669feacc,
>> isClient=false]2019-11-27 08:30:46,994 9321190
>> [tcp-disco-srvr-#3%springDataNode%] ERROR  - Critical system error
>> detected. Will be handled accordingly to configured handler [hnd=class
>> o.a.i.failure.StopNodeOrHaltFailureHandler, failureCtx=FailureContext
>> [type=SYSTEM_WORKER_TERMINATION, err=java.lang.IllegalStateException:
>> Thread tcp-disco-srvr-#3%springDataNode% is terminated
>> unexpectedly.]]java.lang.IllegalStateException: Thread
>> tcp-disco-srvr-#3%springDataNode% is terminated unexpectedly.at
>> org.apache.ignite.spi.discovery.tcp.ServerImpl$TcpServer.body(ServerImpl.java:5686)
>>   at
>> org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)2019-11-27
>> 08:30:46,994 9321190 [tcp-disco-srvr-#3%springDataNode%] ERROR  - Critical
>> system error detected. Will be handled accordingly to configured handler
>> [hnd=class o.a.i.failure.StopNodeOrHaltFailureHandler,
>> failureCtx=FailureContext [type=SYSTEM_WORKER_TERMINATION,
>> err=java.lang.IllegalStateException: Thread
>> tcp-disco-srvr-#3%springDataNode% is terminated
>> unexpectedly.]]java.lang.IllegalStateException: Thread
>> tcp-disco-srvr-#3%springDataNode% is terminated unexpectedly.at
>> org.apache.ignite.spi.discovery.tcp.ServerImpl$TcpServer.body(ServerImpl.java:5686)
>>   at
>> org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)2019-11-27
>> 08:30:46,995 9321191 [tcp-disco-srvr-#3%springDataNode%] ERROR  - JVM will
>> be halted immediately due to the failure: [failureCtx=FailureContext
>> [type=SYSTEM_WORKER_TERMINATION, err=java.lang.IllegalStateException:
>> Thread tcp-disco-srvr-#3%springDataNode% is terminated
>> unexpectedly.]]2019-11-27 08:30:46,995 9321191
>> [tcp-disco-srvr-#3%springDataNode%] ERROR  - JVM will be halted immediately
>> due to the failure: [failureCtx=FailureContext
>> [type=SYSTEM_WORKER_TERMINATION, err=java.lang.IllegalStateException:
>> Thread tcp-disco-srvr-#3%springDataNode% is terminated unexpectedly.]]*
>>
>>
>>
>>
>>
>>
>>
>>
>>

Re: Active tasks in cluster

[Prasad Bhalerao]

Does cancelling an active task cancels the threads being used internally
for query execution (threads in query pool) or cache put/delete operation?

Thanks,
Prasad

On Sat, Nov 23, 2019 at 8:26 PM Mikael  wrote:

> The whole idea with a future is that it is a small lightweight compact
> object, and you still have Igor's suggestion:
>
> Collection>> result =
> ignite.compute().broadcast(() -> ignite.compute().activeTaskFutures());
>
> If you would have to implement a cluster wide listening mechanism in the
> futures you would add a terrible amount of overhead to it, and you would
> cause a lot of problems, what if you try to deserialize a future on a
> computer that is in another cluster, it may not even be an Ignite
> application, what if you deserialize a future that was created 2 years ago
> and the "id" of the future is now being reused for another future that has
> nothing to do with the original one, what if you deserialize it in a
> different cluster where that id is something different and not the same you
> submitted on the other cluster, yes all these things can be handled but
> once again you would turn a small nice simple object into a complex beast.
> Den 2019-11-23 kl. 15:00, skrev Prasad Bhalerao:
>
> By member you meant the output of the thread right?
>
> If yes, can we keep the member at centralised location like an internal
> cache?
> (May be we can provide the flag if turned on then the member can be
> broadcasted to whoever is listening to it or centralised cache location)
> I am considering future as a handle to the task which can be used to
> cancel the task even if the submitter node goes down.
>
>
>
> On Sat 23 Nov, 2019, 7:21 PM Mikael 
>> Well, the thread has to set a member in the future when it is finished,
>> if you serialize the future and send it somewhere else, how is the thread
>> going to be able to tell the future it had finished ?
>> Den 2019-11-23 kl. 14:31, skrev Prasad Bhalerao:
>>
>> Can someone please explain why Active task futures can't be serialized?
>>
>> If we loose the future then we don't have the way to cancel the active
>> task if it's taking too long. I think this is important feature.
>>
>>
>>
>> Thanks,
>> Prasad
>>
>> On Thu 21 Nov, 2019, 5:16 AM Denis Magda >
>>> I think that you should broadcast another task that will simply ask
>>> every node if taskA is already running or not every time the topology
>>> changes. If the response from all the nodes is empty then you need to
>>> reschedule taskA, otherwise, you will skip this procedure.
>>>
>>> -
>>> Denis
>>>
>>>
>>> On Wed, Nov 20, 2019 at 9:28 AM Prasad Bhalerao <
>>> prasadbhalerao1...@gmail.com> wrote:
>>>
>>>> That means I can't do this..
>>>>
>>>> Collection>> result =
>>>> ignite.compute().broadcast(() -> ignite.compute().activeTaskFutures());
>>>>
>>>> Is there any way to get list futures of all active tasks running on all
>>>> nodes of the cluster?
>>>>
>>>> Thanks,
>>>> Prasad
>>>>
>>>>
>>>> On Wed 20 Nov, 2019, 10:51 PM Mikael >>>
>>>>> Hi!
>>>>>
>>>>> No you cannot serialize any future object.
>>>>>
>>>>> Mikael
>>>>>
>>>>>
>>>>> Den 2019-11-20 kl. 17:59, skrev Prasad Bhalerao:
>>>>>
>>>>> Thank you for the suggestion. I will try this.
>>>>>
>>>>> I am thinking to store the task future object in a (replicated)cache
>>>>> against a jobId. If the node goes down as described in case (b), I will 
>>>>> get
>>>>> the task's future object from this  cache using a jobId and will invoke 
>>>>> the
>>>>> get method on it.
>>>>>
>>>>> But I am not sure about this approach, whether a future object can be
>>>>> serialized and send it over the wire to another node and deserialize it 
>>>>> and
>>>>> then invoke the get API on it.
>>>>>
>>>>> I will try to implement it tomorrow.
>>>>>
>>>>> Thanks,
>>>>> Prasad
>>>>>
>>>>>
>>>>> On Wed 20 Nov, 2019, 9:06 PM Igor Belyakov >>>> wrote:
>>>>>
>>>>>> Hi Prasad,
>>>>>>
>>>>>> I think that you can use compute().broadcast() for collecting results
>>

Re: Active tasks in cluster

[Prasad Bhalerao]

By member you meant the output of the thread right?

If yes, can we keep the member at centralised location like an internal
cache?
(May be we can provide the flag if turned on then the member can be
broadcasted to whoever is listening to it or centralised cache location)
I am considering future as a handle to the task which can be used to cancel
the task even if the submitter node goes down.



On Sat 23 Nov, 2019, 7:21 PM Mikael  Well, the thread has to set a member in the future when it is finished, if
> you serialize the future and send it somewhere else, how is the thread
> going to be able to tell the future it had finished ?
> Den 2019-11-23 kl. 14:31, skrev Prasad Bhalerao:
>
> Can someone please explain why Active task futures can't be serialized?
>
> If we loose the future then we don't have the way to cancel the active
> task if it's taking too long. I think this is important feature.
>
>
>
> Thanks,
> Prasad
>
> On Thu 21 Nov, 2019, 5:16 AM Denis Magda 
>> I think that you should broadcast another task that will simply ask every
>> node if taskA is already running or not every time the topology changes. If
>> the response from all the nodes is empty then you need to reschedule taskA,
>> otherwise, you will skip this procedure.
>>
>> -
>> Denis
>>
>>
>> On Wed, Nov 20, 2019 at 9:28 AM Prasad Bhalerao <
>> prasadbhalerao1...@gmail.com> wrote:
>>
>>> That means I can't do this..
>>>
>>> Collection>> result =
>>> ignite.compute().broadcast(() -> ignite.compute().activeTaskFutures());
>>>
>>> Is there any way to get list futures of all active tasks running on all
>>> nodes of the cluster?
>>>
>>> Thanks,
>>> Prasad
>>>
>>>
>>> On Wed 20 Nov, 2019, 10:51 PM Mikael >>
>>>> Hi!
>>>>
>>>> No you cannot serialize any future object.
>>>>
>>>> Mikael
>>>>
>>>>
>>>> Den 2019-11-20 kl. 17:59, skrev Prasad Bhalerao:
>>>>
>>>> Thank you for the suggestion. I will try this.
>>>>
>>>> I am thinking to store the task future object in a (replicated)cache
>>>> against a jobId. If the node goes down as described in case (b), I will get
>>>> the task's future object from this  cache using a jobId and will invoke the
>>>> get method on it.
>>>>
>>>> But I am not sure about this approach, whether a future object can be
>>>> serialized and send it over the wire to another node and deserialize it and
>>>> then invoke the get API on it.
>>>>
>>>> I will try to implement it tomorrow.
>>>>
>>>> Thanks,
>>>> Prasad
>>>>
>>>>
>>>> On Wed 20 Nov, 2019, 9:06 PM Igor Belyakov >>> wrote:
>>>>
>>>>> Hi Prasad,
>>>>>
>>>>> I think that you can use compute().broadcast() for collecting results
>>>>> of activeTaskFutures() from all the nodes:
>>>>> Collection>> result =
>>>>> ignite.compute().broadcast(() -> ignite.compute().activeTaskFutures());
>>>>>
>>>>> Regards,
>>>>> Igor Belyakov
>>>>>
>>>>> On Wed, Nov 20, 2019 at 5:27 PM Prasad Bhalerao <
>>>>> prasadbhalerao1...@gmail.com> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I want to get the active tasks running in cluster (tasks running on
>>>>>> all nodes in cluster)
>>>>>>
>>>>>> IgniteCompute interface has method "activeTaskFutures" which returns
>>>>>> tasks future for active tasks started on local node.
>>>>>>
>>>>>> Is there anyway to get the task futures for all active tasks of whole
>>>>>> cluster?
>>>>>>
>>>>>> My use case is as follows.
>>>>>>
>>>>>> a) The node submits the affinity task and task runs on some other
>>>>>> node in the cluster and the node which submitted the task dies.
>>>>>>
>>>>>> b) The node submits the affinity task and the task runs on the same
>>>>>> node and the same node dies.
>>>>>>
>>>>>> The task consumers running on all ignite grid nodes consumes tasks
>>>>>> from kafka topic. If the node which submitted the affinity task dies, 
>>>>>> kafka
>>>>>> re-assigns the partitions to another consumer (running on different
>>>>>> node) as part of its partition rebalance process. In this case my job 
>>>>>> gets
>>>>>> consumed one more time,
>>>>>>
>>>>>> But in this scenario that job might be already running on one of the
>>>>>> node case (a) or already died as mentioned case (b).
>>>>>>
>>>>>> So I want to check if the job is still running on one of the node or
>>>>>> it is already died. For this I need the active job list running on all
>>>>>> nodes.
>>>>>>
>>>>>> Can someone please advise?
>>>>>>
>>>>>> Thanks,
>>>>>> Prasad
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Thanks,
>>>>>> Prasad
>>>>>>
>>>>>>
>>>>>>

Re: Active tasks in cluster

[Prasad Bhalerao]

Can someone please explain why Active task futures can't be serialized?

If we loose the future then we don't have the way to cancel the active task
if it's taking too long. I think this is important feature.



Thanks,
Prasad

On Thu 21 Nov, 2019, 5:16 AM Denis Magda  I think that you should broadcast another task that will simply ask every
> node if taskA is already running or not every time the topology changes. If
> the response from all the nodes is empty then you need to reschedule taskA,
> otherwise, you will skip this procedure.
>
> -
> Denis
>
>
> On Wed, Nov 20, 2019 at 9:28 AM Prasad Bhalerao <
> prasadbhalerao1...@gmail.com> wrote:
>
>> That means I can't do this..
>>
>> Collection>> result =
>> ignite.compute().broadcast(() -> ignite.compute().activeTaskFutures());
>>
>> Is there any way to get list futures of all active tasks running on all
>> nodes of the cluster?
>>
>> Thanks,
>> Prasad
>>
>>
>> On Wed 20 Nov, 2019, 10:51 PM Mikael >
>>> Hi!
>>>
>>> No you cannot serialize any future object.
>>>
>>> Mikael
>>>
>>>
>>> Den 2019-11-20 kl. 17:59, skrev Prasad Bhalerao:
>>>
>>> Thank you for the suggestion. I will try this.
>>>
>>> I am thinking to store the task future object in a (replicated)cache
>>> against a jobId. If the node goes down as described in case (b), I will get
>>> the task's future object from this  cache using a jobId and will invoke the
>>> get method on it.
>>>
>>> But I am not sure about this approach, whether a future object can be
>>> serialized and send it over the wire to another node and deserialize it and
>>> then invoke the get API on it.
>>>
>>> I will try to implement it tomorrow.
>>>
>>> Thanks,
>>> Prasad
>>>
>>>
>>> On Wed 20 Nov, 2019, 9:06 PM Igor Belyakov >> wrote:
>>>
>>>> Hi Prasad,
>>>>
>>>> I think that you can use compute().broadcast() for collecting results
>>>> of activeTaskFutures() from all the nodes:
>>>> Collection>> result =
>>>> ignite.compute().broadcast(() -> ignite.compute().activeTaskFutures());
>>>>
>>>> Regards,
>>>> Igor Belyakov
>>>>
>>>> On Wed, Nov 20, 2019 at 5:27 PM Prasad Bhalerao <
>>>> prasadbhalerao1...@gmail.com> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I want to get the active tasks running in cluster (tasks running on
>>>>> all nodes in cluster)
>>>>>
>>>>> IgniteCompute interface has method "activeTaskFutures" which returns
>>>>> tasks future for active tasks started on local node.
>>>>>
>>>>> Is there anyway to get the task futures for all active tasks of whole
>>>>> cluster?
>>>>>
>>>>> My use case is as follows.
>>>>>
>>>>> a) The node submits the affinity task and task runs on some other node
>>>>> in the cluster and the node which submitted the task dies.
>>>>>
>>>>> b) The node submits the affinity task and the task runs on the same
>>>>> node and the same node dies.
>>>>>
>>>>> The task consumers running on all ignite grid nodes consumes tasks
>>>>> from kafka topic. If the node which submitted the affinity task dies, 
>>>>> kafka
>>>>> re-assigns the partitions to another consumer (running on different
>>>>> node) as part of its partition rebalance process. In this case my job gets
>>>>> consumed one more time,
>>>>>
>>>>> But in this scenario that job might be already running on one of the
>>>>> node case (a) or already died as mentioned case (b).
>>>>>
>>>>> So I want to check if the job is still running on one of the node or
>>>>> it is already died. For this I need the active job list running on all
>>>>> nodes.
>>>>>
>>>>> Can someone please advise?
>>>>>
>>>>> Thanks,
>>>>> Prasad
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Thanks,
>>>>> Prasad
>>>>>
>>>>>
>>>>>

Re: Actual size of Ignite caches

[Prasad Bhalerao]

You can also get the approx size by enabling persistence. With this you can
get the apporx size of each cache in bytes by checking the folder size of
each cache.

On Fri 22 Nov, 2019, 7:23 PM Mikael  Don't think there is any direct way to do it, but you can read out the
> metric for your data regions and see how much memory is used and how
> much is free, that would give you a good indication of memory usage
>
> Den 2019-11-22 kl. 13:01, skrev ashishb888:
> > Yes I am talking about physical size in bytes. How to get it?
> >
> >
> >
> > --
> > Sent from: http://apache-ignite-users.70518.x6.nabble.com/
> >
>

Re: Active tasks in cluster

[Prasad Bhalerao]

That means I can't do this..

Collection>> result =
ignite.compute().broadcast(() -> ignite.compute().activeTaskFutures());

Is there any way to get list futures of all active tasks running on all
nodes of the cluster?

Thanks,
Prasad


On Wed 20 Nov, 2019, 10:51 PM Mikael  Hi!
>
> No you cannot serialize any future object.
>
> Mikael
>
>
> Den 2019-11-20 kl. 17:59, skrev Prasad Bhalerao:
>
> Thank you for the suggestion. I will try this.
>
> I am thinking to store the task future object in a (replicated)cache
> against a jobId. If the node goes down as described in case (b), I will get
> the task's future object from this  cache using a jobId and will invoke the
> get method on it.
>
> But I am not sure about this approach, whether a future object can be
> serialized and send it over the wire to another node and deserialize it and
> then invoke the get API on it.
>
> I will try to implement it tomorrow.
>
> Thanks,
> Prasad
>
>
> On Wed 20 Nov, 2019, 9:06 PM Igor Belyakov  wrote:
>
>> Hi Prasad,
>>
>> I think that you can use compute().broadcast() for collecting results
>> of activeTaskFutures() from all the nodes:
>> Collection>> result =
>> ignite.compute().broadcast(() -> ignite.compute().activeTaskFutures());
>>
>> Regards,
>> Igor Belyakov
>>
>> On Wed, Nov 20, 2019 at 5:27 PM Prasad Bhalerao <
>> prasadbhalerao1...@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> I want to get the active tasks running in cluster (tasks running on all
>>> nodes in cluster)
>>>
>>> IgniteCompute interface has method "activeTaskFutures" which returns
>>> tasks future for active tasks started on local node.
>>>
>>> Is there anyway to get the task futures for all active tasks of whole
>>> cluster?
>>>
>>> My use case is as follows.
>>>
>>> a) The node submits the affinity task and task runs on some other node
>>> in the cluster and the node which submitted the task dies.
>>>
>>> b) The node submits the affinity task and the task runs on the same node
>>> and the same node dies.
>>>
>>> The task consumers running on all ignite grid nodes consumes tasks from
>>> kafka topic. If the node which submitted the affinity task dies, kafka
>>> re-assigns the partitions to another consumer (running on different
>>> node) as part of its partition rebalance process. In this case my job gets
>>> consumed one more time,
>>>
>>> But in this scenario that job might be already running on one of the
>>> node case (a) or already died as mentioned case (b).
>>>
>>> So I want to check if the job is still running on one of the node or it
>>> is already died. For this I need the active job list running on all nodes.
>>>
>>> Can someone please advise?
>>>
>>> Thanks,
>>> Prasad
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> Thanks,
>>> Prasad
>>>
>>>
>>>

Re: Active tasks in cluster

[Prasad Bhalerao]

Thank you for the suggestion. I will try this.

I am thinking to store the task future object in a (replicated)cache
against a jobId. If the node goes down as described in case (b), I will get
the task's future object from this  cache using a jobId and will invoke the
get method on it.

But I am not sure about this approach, whether a future object can be
serialized and send it over the wire to another node and deserialize it and
then invoke the get API on it.

I will try to implement it tomorrow.

Thanks,
Prasad


On Wed 20 Nov, 2019, 9:06 PM Igor Belyakov  Hi Prasad,
>
> I think that you can use compute().broadcast() for collecting results
> of activeTaskFutures() from all the nodes:
> Collection>> result =
> ignite.compute().broadcast(() -> ignite.compute().activeTaskFutures());
>
> Regards,
> Igor Belyakov
>
> On Wed, Nov 20, 2019 at 5:27 PM Prasad Bhalerao <
> prasadbhalerao1...@gmail.com> wrote:
>
>> Hi,
>>
>> I want to get the active tasks running in cluster (tasks running on all
>> nodes in cluster)
>>
>> IgniteCompute interface has method "activeTaskFutures" which returns
>> tasks future for active tasks started on local node.
>>
>> Is there anyway to get the task futures for all active tasks of whole
>> cluster?
>>
>> My use case is as follows.
>>
>> a) The node submits the affinity task and task runs on some other node in
>> the cluster and the node which submitted the task dies.
>>
>> b) The node submits the affinity task and the task runs on the same node
>> and the same node dies.
>>
>> The task consumers running on all ignite grid nodes consumes tasks from
>> kafka topic. If the node which submitted the affinity task dies, kafka
>> re-assigns the partitions to another consumer (running on different
>> node) as part of its partition rebalance process. In this case my job gets
>> consumed one more time,
>>
>> But in this scenario that job might be already running on one of the node
>> case (a) or already died as mentioned case (b).
>>
>> So I want to check if the job is still running on one of the node or it
>> is already died. For this I need the active job list running on all nodes.
>>
>> Can someone please advise?
>>
>> Thanks,
>> Prasad
>>
>>
>>
>>
>>
>>
>>
>> Thanks,
>> Prasad
>>
>>
>>

Active tasks in cluster

[Prasad Bhalerao]

Hi,

I want to get the active tasks running in cluster (tasks running on all
nodes in cluster)

IgniteCompute interface has method "activeTaskFutures" which returns tasks
future for active tasks started on local node.

Is there anyway to get the task futures for all active tasks of whole
cluster?

My use case is as follows.

a) The node submits the affinity task and task runs on some other node in
the cluster and the node which submitted the task dies.

b) The node submits the affinity task and the task runs on the same node
and the same node dies.

The task consumers running on all ignite grid nodes consumes tasks from
kafka topic. If the node which submitted the affinity task dies, kafka
re-assigns the partitions to another consumer (running on different
node) as part of its partition rebalance process. In this case my job gets
consumed one more time,

But in this scenario that job might be already running on one of the node
case (a) or already died as mentioned case (b).

So I want to check if the job is still running on one of the node or it is
already died. For this I need the active job list running on all nodes.

Can someone please advise?

Thanks,
Prasad







Thanks,
Prasad

Re: Service grid webinar

[Prasad Bhalerao]

Can you please share the recording of webinar  in case  if you have it?

On Wed 6 Nov, 2019, 3:30 AM Denis Mekhanikov  Hi Igniters!
>
> I’ve been working on the Service Grid functionality in Apache Ignite for a
> while, and at some point I've decided to make a webinar with a high-level
> overview of this part of the project.
>
> If you want to learn more about services, look at some use-cases or just
> ask a few questions somebody who takes part in the development, please feel
> free to join the presentation on November 6th, at 10 AM PST.
>
> You can sign up by the following link:
> https://www.gridgain.com/resources/webinars/best-practices-microservices-architecture-apache-ignite
>
> Denis
>

Re: resetLostPartitions is blocked inside event listener

[Prasad Bhalerao]

Do you mean to say, spawn a different thread from event listener and reset
the lost partition in that thread?

I tried this and it works.

But wanted to understand the reason, why this call get blocked in event
listener?

Thanks,
Prasad

On Thu 7 Nov, 2019, 9:28 PM Ilya Kasnacheev  Hello!
>
> It is not advisable to call any blocking methods from event listeners.
> Just fire resetLostPartitions from another thread.
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> чт, 7 нояб. 2019 г. в 15:17, Akash Shinde :
>
>> Hi,
>> I am trying to handle lost partition scenario.
>> I have written event listener listening  to
>> EventType.EVT_CACHE_REBALANCE_PART_DATA_LOST event.
>> I want to reset lost partition state of cache after cache loading  is
>> done.
>> *Issue:* ignite.resetLostPartitions(caheName) is getting blocked and not
>> completing.
>>
>> Please find the code for Event Listener. Someone can help on this. *Why
>> this resetLostPartitions getting blocked.*
>>
>> public class IgniteEventListner implements 
>> IgnitePredicate {
>>private static final Logger LOGGER = 
>> LoggerFactory.getLogger(IgniteEventListner.class);
>>
>>   private final Ignite ignite;
>>
>>   public IgniteEventListner(Ignite ignite) {
>> this.ignite = ignite;
>>   }
>>
>>   @Override
>>   public boolean apply(CacheRebalancingEvent evt) {
>>
>> IgniteCache cache = 
>> ignite.getOrCreateCache(CacheName.ASSET_GROUP_CACHE.name());
>> Collection lostPartitions = cache.lostPartitions();
>> reloadCache(lostPartitions); //perform partition based cache loading
>>
>>* 
>> ignite.resetLostPartitions(Arrays.asList(CacheName.ASSET_GROUP_CACHE.name()));
>>   //Reset partitions*
>>
>> System.out.println("Check-1, Partition lost event processed");
>>
>> return true;
>>   }
>> }
>>
>> *Cache Configuration*
>>
>> private CacheConfiguration assetGroupCacheCfg() {
>> CacheConfiguration assetGroupCacheCfg = new 
>> CacheConfiguration<>(CacheName.ASSET_GROUP_CACHE.name());
>> assetGroupCacheCfg.setAtomicityMode(CacheAtomicityMode.TRANSACTIONAL);
>> assetGroupCacheCfg.setWriteThrough(false);
>> assetGroupCacheCfg.setReadThrough(false);
>> assetGroupCacheCfg.setRebalanceMode(CacheRebalanceMode.ASYNC);
>> assetGroupCacheCfg.setBackups(0);
>> assetGroupCacheCfg.setCacheMode(CacheMode.PARTITIONED);
>> assetGroupCacheCfg.setIndexedTypes(DefaultDataAffinityKey.class, 
>> AssetGroupData.class);
>> assetGroupCacheCfg.setSqlIndexMaxInlineSize(100);
>> RendezvousAffinityFunction affinityFunction = new 
>> RendezvousAffinityFunction();
>> assetGroupCacheCfg.setAffinity(affinityFunction);
>> assetGroupCacheCfg.setStatisticsEnabled(true);
>>
>> assetGroupCacheCfg.setPartitionLossPolicy(PartitionLossPolicy.READ_WRITE_SAFE);
>> return assetGroupCacheCfg;
>>   }
>>
>> *Ignite Configuration*
>>
>> private IgniteConfiguration getIgniteConfiguration() {
>>
>>   TcpDiscoveryVmIpFinder ipFinder = new TcpDiscoveryVmIpFinder();
>>   String[] hosts = {"127.0.0.1:47500..47509"};
>>   ipFinder.setAddresses(Arrays.asList(hosts));
>>
>>   TcpDiscoverySpi discoSpi = new TcpDiscoverySpi();
>>   discoSpi.setIpFinder(ipFinder);
>>
>>   IgniteConfiguration cfg = new IgniteConfiguration();
>>   cfg.setDiscoverySpi(discoSpi);
>>   cfg.setIgniteInstanceName("springDataNode");
>>   cfg.setPeerClassLoadingEnabled(false);
>>   cfg.setRebalanceThreadPoolSize(4);
>>   DataStorageConfiguration storageCfg = new DataStorageConfiguration();
>>   DataRegionConfiguration regionConfiguration = new 
>> DataRegionConfiguration();
>>   regionConfiguration.setInitialSize(3L * 1024 * 1024 * 1024);
>>   regionConfiguration.setMaxSize(3L * 1024 * 1024 * 1024);
>>   regionConfiguration.setMetricsEnabled(true);
>>
>>   storageCfg.setDefaultDataRegionConfiguration(regionConfiguration);
>>   storageCfg.setStoragePath("c:/ignite-storage/storage");
>>   storageCfg.setWalPath("c:/ignite-storage/storage/wal");
>>   storageCfg.setWalArchivePath("c:/ignite-storage/storage/wal-archive");
>>   storageCfg.setMetricsEnabled(true);
>>   cfg.setDataStorageConfiguration(storageCfg);
>>   
>> cfg.setIncludeEventTypes(EventType.EVT_CACHE_REBALANCE_PART_DATA_LOST,EventType.EVT_NODE_FAILED);
>>   cfg.setCacheConfiguration(getCacheConfigurations());
>>   return cfg;
>> }
>>
>>
>> Thanks,
>>
>> Akash
>>
>>

Re: TransactionOptimisticException: Failed to prepare transaction, read/write conflict

[Prasad Bhalerao]

Can someone from community advise?

On Wed, Oct 30, 2019 at 7:28 PM Prasad Bhalerao <
prasadbhalerao1...@gmail.com> wrote:

> Hi,
> Ignite version: 2.6.0
>
> I am getting following exception while committing a transaction inside
> IgniteRunnable task.
> Transaction is
>
> try (Transaction tx = transactions.txStart(TransactionConcurrency.OPTIMISTIC, 
> TransactionIsolation.SERIALIZABLE)) {
>
> 
>
> ..
>
> }
>
>
> As per the ignite doc
> <https://apacheignite.readme.io/docs/concurrency-modes-and-isolation-levels#section-optimistic-transactions>
>  if
> the transaction isolation level is serializable,
> Ignite will fail a transaction at the commit stage if the Ignite engine
> detects that at least one of the entries used as part of the initiated
> transaction has been modified.
>
> But inside transaction I am just reading SubscriptionData from
> Sunsbscription cache, I am not modifying this data inside transaction. Also
> this SubscriptionData is not being modified in some other request
> outside the mentioned transaction.
>
> Then what could be the reason for this transaction failure?
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> *Caused by: org.apache.ignite.transactions.TransactionOptimisticException:
> Failed to prepare transaction, read/write conflict
> [key=com.xyz.grid.data.key.DefaultDataKey@fe220,
> keyCls=com.xyz.grid.data.key.DefaultDataKey,
> val=SubscriptionData{subscriptionId=1040928, companyName='test-cmp',
> expirationDate=1592092799000, activatedModules={100=159200640,
> 101=159200640, 102=159200640, 107=159200640, 112=159200640,
> 145=159200640, 114=159200640, 117=159200640,
> 1206=159200640, 119=159200640, 1207=159200640,
> 121=159200640, 1210=159200640, 9211=159200640}, enforced=false,
> ipv6Enabled=false}, valCls=com.xyz.grid.data.SubscriptionData,
> cache=SUBSCRIPTION_CACHE, thread=IgniteThread [compositeRwLockIdx=22,
> stripe=10, plc=-1, name=sys-stripe-10-#11%springDataNode%]]at
> org.apache.ignite.internal.util.IgniteUtils$14.apply(IgniteUtils.java:905)
> at
> org.apache.ignite.internal.util.IgniteUtils$14.apply(IgniteUtils.java:903)
> at
> org.apache.ignite.internal.util.IgniteUtils.convertException(IgniteUtils.java:985)
> at
> org.apache.ignite.internal.processors.cache.transactions.TransactionProxyImpl.commit(TransactionProxyImpl.java:305)
> at
> com.xyz.task.ignite.AbstractModuleIpAdderIgniteTask.addIps(AbstractModuleIpAdderIgniteTask.java:148)
> at
> com.xyz.task.ignite.AbstractModuleIpAdderIgniteTask.run(AbstractModuleIpAdderIgniteTask.java:93)
> at
> org.apache.ignite.internal.processors.closure.GridClosureProcessor$C4.execute(GridClosureProcessor.java:1944)
> at
> org.apache.ignite.internal.processors.job.GridJobWorker$2.call(GridJobWorker.java:568)
> at
> org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6695)
> at
> org.apache.ignite.internal.processors.job.GridJobWorker.execute0(GridJobWorker.java:562)
> at
> org.apache.ignite.internal.processors.job.GridJobWorker.body(GridJobWorker.java:491)
> at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
> ... 3 common frames omittedCaused by:
> org.apache.ignite.internal.transactions.IgniteTxOptimisticCheckedException:
> Failed to prepare transaction, read/write conflict
> [key=com.xyz.grid.data.key.DefaultDataKey@fe220,
> keyCls=com.xyz.grid.data.key.DefaultDataKey,
> val=SubscriptionData{subscriptionId=1040928, companyName='test-cmp',
> expirationDate=1592092799000, activatedModules={100=159200640,
> 101=159200640, 102=159200640, 107=159200640, 112=159200640,
> 145=159200640, 114=159200640, 117=159200640,
> 1206=159200640, 119=159200640, 1207=159200640,
> 121=159200640, 1210=159200640, 9211=159200640}, enforced=false,
> ipv6Enabled=false}, valCls=com.xyz.grid.data.SubscriptionData,
> cache=SUBSCRIPTION_CACHE, thread=IgniteThread [compositeRwLockIdx=22,
> stripe=10, plc=-1, name=sys-stripe-10-#11%springDataNode%]]at
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxPrepareFuture.versionCheckError(GridDhtTxPrepareFuture.java:1190)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxPrepareFuture.checkReadConflict(GridDhtTxPrepareFuture.java:1138)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxPrepareFuture.prepare0(GridDhtTxPrepareFuture.java:1205)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxPrepareF

TransactionOptimisticException: Failed to prepare transaction, read/write conflict

[Prasad Bhalerao]

Hi,
Ignite version: 2.6.0

I am getting following exception while committing a transaction inside
IgniteRunnable task.
Transaction is

try (Transaction tx =
transactions.txStart(TransactionConcurrency.OPTIMISTIC,
TransactionIsolation.SERIALIZABLE)) {



..

}


As per the ignite doc

if
the transaction isolation level is serializable,
Ignite will fail a transaction at the commit stage if the Ignite engine
detects that at least one of the entries used as part of the initiated
transaction has been modified.

But inside transaction I am just reading SubscriptionData from
Sunsbscription cache, I am not modifying this data inside transaction. Also
this SubscriptionData is not being modified in some other request
outside the mentioned transaction.

Then what could be the reason for this transaction failure?








































*Caused by: org.apache.ignite.transactions.TransactionOptimisticException:
Failed to prepare transaction, read/write conflict
[key=com.xyz.grid.data.key.DefaultDataKey@fe220,
keyCls=com.xyz.grid.data.key.DefaultDataKey,
val=SubscriptionData{subscriptionId=1040928, companyName='test-cmp',
expirationDate=1592092799000, activatedModules={100=159200640,
101=159200640, 102=159200640, 107=159200640, 112=159200640,
145=159200640, 114=159200640, 117=159200640,
1206=159200640, 119=159200640, 1207=159200640,
121=159200640, 1210=159200640, 9211=159200640}, enforced=false,
ipv6Enabled=false}, valCls=com.xyz.grid.data.SubscriptionData,
cache=SUBSCRIPTION_CACHE, thread=IgniteThread [compositeRwLockIdx=22,
stripe=10, plc=-1, name=sys-stripe-10-#11%springDataNode%]]at
org.apache.ignite.internal.util.IgniteUtils$14.apply(IgniteUtils.java:905)
at
org.apache.ignite.internal.util.IgniteUtils$14.apply(IgniteUtils.java:903)
at
org.apache.ignite.internal.util.IgniteUtils.convertException(IgniteUtils.java:985)
at
org.apache.ignite.internal.processors.cache.transactions.TransactionProxyImpl.commit(TransactionProxyImpl.java:305)
at
com.xyz.task.ignite.AbstractModuleIpAdderIgniteTask.addIps(AbstractModuleIpAdderIgniteTask.java:148)
at
com.xyz.task.ignite.AbstractModuleIpAdderIgniteTask.run(AbstractModuleIpAdderIgniteTask.java:93)
at
org.apache.ignite.internal.processors.closure.GridClosureProcessor$C4.execute(GridClosureProcessor.java:1944)
at
org.apache.ignite.internal.processors.job.GridJobWorker$2.call(GridJobWorker.java:568)
at
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6695)
at
org.apache.ignite.internal.processors.job.GridJobWorker.execute0(GridJobWorker.java:562)
at
org.apache.ignite.internal.processors.job.GridJobWorker.body(GridJobWorker.java:491)
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
... 3 common frames omittedCaused by:
org.apache.ignite.internal.transactions.IgniteTxOptimisticCheckedException:
Failed to prepare transaction, read/write conflict
[key=com.xyz.grid.data.key.DefaultDataKey@fe220,
keyCls=com.xyz.grid.data.key.DefaultDataKey,
val=SubscriptionData{subscriptionId=1040928, companyName='test-cmp',
expirationDate=1592092799000, activatedModules={100=159200640,
101=159200640, 102=159200640, 107=159200640, 112=159200640,
145=159200640, 114=159200640, 117=159200640,
1206=159200640, 119=159200640, 1207=159200640,
121=159200640, 1210=159200640, 9211=159200640}, enforced=false,
ipv6Enabled=false}, valCls=com.xyz.grid.data.SubscriptionData,
cache=SUBSCRIPTION_CACHE, thread=IgniteThread [compositeRwLockIdx=22,
stripe=10, plc=-1, name=sys-stripe-10-#11%springDataNode%]]at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxPrepareFuture.versionCheckError(GridDhtTxPrepareFuture.java:1190)
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxPrepareFuture.checkReadConflict(GridDhtTxPrepareFuture.java:1138)
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxPrepareFuture.prepare0(GridDhtTxPrepareFuture.java:1205)
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxPrepareFuture.mapIfLocked(GridDhtTxPrepareFuture.java:671)
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxPrepareFuture.prepare(GridDhtTxPrepareFuture.java:1048)
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxLocal.prepareAsync(GridDhtTxLocal.java:397)
at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.prepareNearTx(IgniteTxHandler.java:516)
at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.processNearTxPrepareRequest0(IgniteTxHandler.java:157)
at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.processNearTxPrepareRequest(IgniteTxHandler.java:135)
at

Re: Ignite cache partition size

[Prasad Bhalerao]

I have heard people creating 4096 but not more that.
What's you reasoning behind creating such huge partitions.
I have a table which contains 40+ million rows, but it has only 5 columns
and I have kept the default partitions and getting good results out of it.
You have 1450 columns, an interesting use case.

1) Is your app already in production?
2) Did you calculate the average size of single row.
3) How do you manage and monitor your big cluster.
4) Do you have any scripts or tools to stop,start, restart or rolling
reastart the cluster or node of your cluster?

I have cluster of 7 nodes, each node has a data region size of 40-50 Gb but
whenever a segmentation happens i.e when Node goes of out cluster I have
found it difficult to debug or investigate such issues. Most of the times I
could not figure out the  exact reason for sengmentation and always ended
up with blaming network issue for the segmentation.

If you have already figured out all these things then well and good, please
share your ideas to tools with usergroup if possible.

If you are doing POC and if you want 24/7 availability then I would suggest
to figure out these things as well.

Thanks,
Prasad


On Sun 27 Oct, 2019, 6:53 AM Yong Zhao 
> Cool, thanks .
> I have a quite big cluster with 100 nodes, so I guess 25600 will be good
> partition?
> I am using ignite as database.
> I am creating a table with 20 million rows,each row has 1450 columns.
> Is this a good idea?
>
> Thank you!
>
> On Sat, Oct 26, 2019 at 9:10 AM Prasad Bhalerao <
> prasadbhalerao1...@gmail.com> wrote:
>
>> Please check this link...
>>
>>
>> https://apacheignite.readme.io/docs/affinity-collocation#section-affinity-function
>>
>>
>> Example from the ignite doc.
>>
>>
>> // Preparing Apache Ignite node configuration.IgniteConfiguration cfg = new 
>> IgniteConfiguration();
>> // Creating a cache configuration.CacheConfiguration cacheCfg = new 
>> CacheConfiguration("myCache");
>> // Creating the affinity function with custom 
>> setting.RendezvousAffinityFunction affFunc = new 
>> RendezvousAffinityFunction();
>> affFunc.setExcludeNeighbors(true);
>> affFunc.setPartitions(2048);
>> // Applying the affinity function 
>> configuration.cacheCfg.setAffinity(affFunc);
>> // Setting the cache 
>> configuration.cfg.setCacheConfiguration(cacheCfg);
>>
>>
>>
>> On Sat 26 Oct, 2019, 7:42 PM Andrey Dolmatov >
>>> Try to implement you're own affinity function. If it map any key to
>>> numbers from 1 to n, you have n partitions
>>>
>>> On Sat, Oct 26, 2019, 09:46 codeboyyong  wrote:
>>>
>>>> Hi can you please tell me how to config this ?
>>>> "number of cache partitions"
>>>>
>>>> Thank You
>>>> Yong
>>>>
>>>>
>>>>
>>>> --
>>>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>>>
>>>

Re: Ignite cache partition size

[Prasad Bhalerao]

Please check this link...


https://apacheignite.readme.io/docs/affinity-collocation#section-affinity-function


Example from the ignite doc.


// Preparing Apache Ignite node configuration.IgniteConfiguration cfg
= new IgniteConfiguration();
// Creating a cache configuration.CacheConfiguration cacheCfg
= new CacheConfiguration("myCache");
// Creating the affinity function with custom
setting.RendezvousAffinityFunction affFunc = new
RendezvousAffinityFunction();
affFunc.setExcludeNeighbors(true);
affFunc.setPartitions(2048);
// Applying the affinity function configuration.cacheCfg.setAffinity(affFunc);
// Setting the cache configuration.cfg.setCacheConfiguration(cacheCfg);



On Sat 26 Oct, 2019, 7:42 PM Andrey Dolmatov  Try to implement you're own affinity function. If it map any key to
> numbers from 1 to n, you have n partitions
>
> On Sat, Oct 26, 2019, 09:46 codeboyyong  wrote:
>
>> Hi can you please tell me how to config this ?
>> "number of cache partitions"
>>
>> Thank You
>> Yong
>>
>>
>>
>> --
>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>
>

Re: Query execution in ignite

[Prasad Bhalerao]

Question is specifically about primary and secondary partitions.

So in case of replicated cache ignite scans primary and secondary
partitions of any one of the node of the cluster to to fetch the data.

But is it the case with partitioned cache.
I mean in case of partitioned cache when SQL is executed, does ignite scan
primary as well as secondary partitions of each node in the cluster or it
just scans primary partitions of all the nodes in the cluster as the query
is being executed on all nodes?

On Fri 25 Oct, 2019, 10:46 PM  Hi,
>
>
>
>If a query is executed against a fully *REPLICATED* data then Ignite
> will send it to a single cluster node and run it over the local data there.
>
>
>
>
>
>
>
>  if a query is executed over a *PARTITIONED* data, then the execution
> flow will be the following:
>
> The query will be parsed and split into multiple map queries and a single
> reduce query.
>
> · All the map queries are executed on all the nodes where
> required data resides.
>
> · All the nodes provide result sets of local execution to the
> query initiator (reducer) that, in turn, will accomplish the reduce phase
> by properly merging provided result sets.
>
>
>
>
>
>
>
>  More information here:
> https://apacheignite-sql.readme.io/docs/how-ignite-sql-works
>
> Thanks, Alex
>
>
>
> *From:* Prasad Bhalerao 
> *Sent:* Friday, October 25, 2019 1:31 AM
> *To:* user@ignite.apache.org
> *Subject:* Query execution in ignite
>
>
>
> Hi,
>
>
>
> When SQL is executed, does ignite always scan only primary partitions of
> all available nodes in cluster irrespective of cache mode partitioned or
> replicated?
>
>
>
>
>
>
>
> Thanks ,
>
> Prasad
>

Re: How to control the distribution??

[Prasad Bhalerao]

Please check this thread.

http://apache-ignite-users.70518.x6.nabble.com/Customized-affinity-function-td21126.html

Thanks,
Prasad

On Fri 25 Oct, 2019, 12:06 PM Mikael  Hi!
>
> You can always use node filter to control where the cache is, I am not
> sure if there is a way to control distribution though, I guess it might
> be possible to use affinity keys and some kind of custom affinity
> function that is able to distribute the keys depending on some custom
> criteria, attributes for example.
>
> Mikael
>
>
> Den 2019-10-25 kl. 04:47, skrev BorisBelozerov:
> > Normally, Apache Ignite distribute data equally to nodes
> > How can I control this distribution?
> > I want to distribute more data to some nodes, and less data to other
> nodes
> > I also want to choose node to store cache
> > Thank you!!
> >
> >
> >
> > --
> > Sent from: http://apache-ignite-users.70518.x6.nabble.com/
> >
>

Query execution in ignite

[Prasad Bhalerao]

Hi,

When SQL is executed, does ignite always scan only primary partitions of
all available nodes in cluster irrespective of cache mode partitioned or
replicated?



Thanks ,
Prasad

Invalidate specific cache partitions

[Prasad Bhalerao]

Hi,
1)Is there any way to invalidate specific  cache partitions?

2) is there any way to invalidate the complete cache?

Thanks,
Prasad

Re: Fault tolerance with IgniteCallable and IgniteRunnable on user errors

[Prasad Bhalerao]

Yes I had checked the doc you mentioned.

This doc explains custom failover behavior using ComputeTaskSplitAdapter
which internally implements ComputTask.
But I am submitting IgniteRunnable/IgniteCallable tasks using
affinityRun/affinityCall method. In this case I am not able to define
custom failover behavior by overriding ComputeTask.result method.

Thanks,
Prasad


On Mon 14 Oct, 2019, 4:10 PM Ilya Kasnacheev  Hello!
>
> Have you checked the following doc:
>
> https://apacheignite.readme.io/docs/fault-tolerance
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> пт, 11 окт. 2019 г. в 18:37, Prasad Bhalerao  >:
>
>> Ignite error gives message to implement ComputeTask and override result
>> method to when the task is failed.
>> So I thought of implementing ComputTask to override this behaviour.
>> Ignite documentation describes the same thing but for the task
>> implementing ComputTask and executed with execute method.
>>
>> Thanks,
>> Prasad
>>
>> On Fri 11 Oct, 2019, 8:45 PM Denis Mekhanikov > wrote:
>>
>>> Prasad,
>>>
>>> User errors are propagated to the calling site, so you can implement
>>> failover outside of the task’s logic.
>>>
>>> Denis
>>> On 11 Oct 2019, 17:33 +0300, Prasad Bhalerao <
>>> prasadbhalerao1...@gmail.com>, wrote:
>>>
>>> Hi,
>>>
>>> How to define custom failover behavior when executing IgniteRunnable or
>>> IgniteCallable tasks executed with ignite.compute().affinityRun or
>>> ignite.compute().affinityCall ?
>>>
>>> I want to retry the task on few user errors.
>>>
>>> Implementing additional interface ComputeTask along with IgniteRunnable
>>> is not helping in this case.
>>>
>>> Thanks,
>>> Prasad
>>>
>>>

Re: Fault tolerance with IgniteCallable and IgniteRunnable on user errors

[Prasad Bhalerao]

Ignite error gives message to implement ComputeTask and override result
method to when the task is failed.
So I thought of implementing ComputTask to override this behaviour.
Ignite documentation describes the same thing but for the task implementing
ComputTask and executed with execute method.

Thanks,
Prasad

On Fri 11 Oct, 2019, 8:45 PM Denis Mekhanikov  Prasad,
>
> User errors are propagated to the calling site, so you can implement
> failover outside of the task’s logic.
>
> Denis
> On 11 Oct 2019, 17:33 +0300, Prasad Bhalerao ,
> wrote:
>
> Hi,
>
> How to define custom failover behavior when executing IgniteRunnable or
> IgniteCallable tasks executed with ignite.compute().affinityRun or
> ignite.compute().affinityCall ?
>
> I want to retry the task on few user errors.
>
> Implementing additional interface ComputeTask along with IgniteRunnable is
> not helping in this case.
>
> Thanks,
> Prasad
>
>

Fault tolerance with IgniteCallable and IgniteRunnable on user errors

[Prasad Bhalerao]

Hi,

How to define custom failover behavior when executing IgniteRunnable or
IgniteCallable tasks executed with ignite.compute().affinityRun or
ignite.compute().affinityCall ?

I want to retry the task on few user errors.

Implementing additional interface ComputeTask along with IgniteRunnable is
not helping in this case.

Thanks,
Prasad

Gracefully shutting down the cluster nodes

[Prasad Bhalerao]

Hi,
I want shutdown the cluster nodes one by one (kind of rolling restart) for
upgrade purpose.
Is there anyway to shutdown the ignite node safely as follows
1) it will wait for already executing task to complete.
2) it won't accept any new task submitted, instead the new tasks will be
executed on its secondary node which will become primary once the previous
primary is down.
 I am not using ignite.sh to start node. I am starting ignite as java
embedded mode.

Thanks,
Prasad

Re: Getting javax.cache.CacheException after upgrading to Ignite 2.7 (IGNITE-10884 )

[Prasad Bhalerao]

Hi,

this issue still exists in ignite 2.7.6 version but the ticket is marked as
Fixed. I am still able to reproduce the issue using following reproducer.

Because this issue I am not able to upgrade to latest version from 2.6.0
version.

GitHub project: https://github.com/prasadbhalerao1983/IgniteTestPrj.git

Please run IgniteTransactionTester class to check the issue.

Thanks,
Prasad

On Thu, Jan 10, 2019 at 7:22 PM Ilya Kasnacheev 
wrote:

> Hello!
>
> I have filed a blocker ticket about it:
> https://issues.apache.org/jira/browse/IGNITE-10884
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> ср, 9 янв. 2019 г. в 17:43, Prasad Bhalerao  >:
>
>>
>> Hi Ilya,
>>
>> I have created a reproducer for this issue and uploaded it to GitHub.
>>
>> GitHub project: https://github.com/prasadbhalerao1983/IgniteTestPrj.git
>>
>> Please run IgniteTransactionTester class to check the issue.
>>
>>
>> Exception:
>>
>> Exception in thread "main" javax.cache.CacheException: Only pessimistic
>> repeatable read transactions are supported at the moment.
>>  at
>> org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.query(IgniteCacheProxyImpl.java:697)
>>  at
>> org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.query(IgniteCacheProxyImpl.java:636)
>>  at
>> org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.query(GatewayProtectedCacheProxy.java:388)
>>  at
>> IgniteTransactionTester.testTransactionException(IgniteTransactionTester.java:53)
>>  at IgniteTransactionTester.main(IgniteTransactionTester.java:38)
>> Caused by: class
>> org.apache.ignite.internal.processors.query.IgniteSQLException: Only
>> pessimistic repeatable read transactions are supported at the moment.
>>  at
>> org.apache.ignite.internal.processors.cache.mvcc.MvccUtils.tx(MvccUtils.java:690)
>>  at
>> org.apache.ignite.internal.processors.cache.mvcc.MvccUtils.tx(MvccUtils.java:671)
>>  at
>> org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.runQueryTwoStep(IgniteH2Indexing.java:1793)
>>  at
>> org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.doRunDistributedQuery(IgniteH2Indexing.java:2610)
>>  at
>> org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.doRunPrepared(IgniteH2Indexing.java:2315)
>>  at
>> org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.querySqlFields(IgniteH2Indexing.java:2209)
>>  at
>> org.apache.ignite.internal.processors.query.GridQueryProcessor$4.applyx(GridQueryProcessor.java:2135)
>>  at
>> org.apache.ignite.internal.processors.query.GridQueryProcessor$4.applyx(GridQueryProcessor.java:2130)
>>  at
>> org.apache.ignite.internal.util.lang.IgniteOutClosureX.apply(IgniteOutClosureX.java:36)
>>  at
>> org.apache.ignite.internal.processors.query.GridQueryProcessor.executeQuery(GridQueryProcessor.java:2707)
>>  at
>> org.apache.ignite.internal.processors.query.GridQueryProcessor.querySqlFields(GridQueryProcessor.java:2144)
>>  at
>> org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.query(IgniteCacheProxyImpl.java:685)
>>
>> Thanks,
>>
>> Prasad
>>
>>
>>
>> On Wed, Jan 9, 2019 at 6:22 PM Ilya Kasnacheev 
>> wrote:
>>
>>> Hello!
>>>
>>> It was discussed recently:
>>> http://apache-ignite-users.70518.x6.nabble.com/Migrate-from-2-6-to-2-7-td25738.html
>>>
>>> I don't think you will be able to use SQL from transactions in Ignite
>>> 2.7. While this looks like a regression, you will have to work around it
>>> for now.
>>>
>>> Do you have a small reproducer for this issue? I could file a ticket if
>>> you had. You can try to do it yourself, too.
>>>
>>> Regards,
>>> --
>>> Ilya Kasnacheev
>>>
>>>
>>> ср, 9 янв. 2019 г. в 15:33, Prasad Bhalerao <
>>> prasadbhalerao1...@gmail.com>:
>>>
>>>> Hi,
>>>>
>>>> My cache configuration is as follows. I am using TRANSACTIONAL and not
>>>> TRANSACTIONAL_SNAPSHOT.
>>>>
>>>>
>>>>
>>>> private CacheConfiguration ipContainerIPV4CacheCfg() {
>>>>
>>>>   CacheConfiguration ipContainerIpV4CacheCfg = new 
>>>> CacheConfiguration<>(CacheName.IP_CONTAINER_IPV4_CACHE.name());
>>>>   
>>>> ipContainerIpV4CacheCfg.setAtomicityMode(CacheAtomicityMode.TRANSACTIONAL);
>>>>   ipContainerIpV4CacheCfg.setWriteThrough(ENABLE_WRITE_THROUGH);
>>>>

OR condition in where clause

[Prasad Bhalerao]

As per this doc
,
if the query contains OR operator indexes may not be used as expected. As a
work around the query can be rewritten using UNION ALL condition.

But if I use "USE INDEX" hint to force query optimizer to use given index,
will it solve the problem with OR operator?

Is it guaranteed that query optimizer will always use the index provided in
"USE INDEX" hint?

Thanks,
Prasad

Re: Issue in starting Near cache only on client node

[Prasad Bhalerao]

Actually I wanted to start near cache only on client node using
getOrCreateNearCache method. But somehow it didn't work.

So now. I have configured near cache using cache.setNearCacheConfig method
in cache configuration. But this starts near cache on client as well as on
server node.


On Mon 1 Jul, 2019, 8:53 PM Ilya Kasnacheev  Hello!
>
> I think the cache would start if listed in node's configuration (via XML
> or Java). Make sure you also have near caches there.
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> чт, 27 июн. 2019 г. в 10:32, Prasad Bhalerao  >:
>
>> Hi,
>>
>> I am getting following error on ignite client nodes.
>>
>> *class org.apache.ignite.IgniteCheckedException: Failed to start near
>> cache (a cache with the same name without near cache is already started*
>>
>> To solve this issue I added following code in ApplicationRunner's run()
>> method to explicitly start the cache. Application runner Interface is
>> provided by spring boot to execute required code just before server startup.
>>
>> But even after doing this I am still getting error "a cache with the same
>> name without near cache is already started".
>>
>> I see following info messages in my log file *multiple times*. First
>> time it is logged when ApplicationRunner.run() method is invoked during
>> server startup.
>>
>> *o.a.i.i.p.cache.GridCacheProcessor - Started cache [name=DOMAIN_CACHE,
>> id=-2139013081, memoryPolicyName=null, mode=PARTITIONED,
>> atomicity=TRANSACTIONAL, backups=1]*
>>
>> On which events GridCacheProcessor starts the cache?
>>
>> Can this event be intercepted or handled to start near cache?
>>
>> What is the reason behind getting "GridCacheProcessor - Started cache"
>> info messages *multiple* *times *in log files?
>>
>> Is near cache is destroyed/invalidate when client node is reconnected to
>> the cluster?
>>
>> Can some please advise?
>>
>> *CODE TO START caches explicitly:*
>>
>> @Override
>> public void run(ApplicationArguments args) throws Exception {
>>   for (CacheName cacheName : CacheName.values()) {
>> //We are configuring few caches only on client node for fast lookup 
>> during validations.
>> if (ignite.configuration().isClientMode() && 
>> CacheName.isNearCache(cacheName)) {
>>   NearCacheConfiguration nearCfg = new NearCacheConfiguration<>();
>>   nearCfg.setNearEvictionPolicyFactory(new 
>> LruEvictionPolicyFactory<>(nearCacheSize));
>>   ignite.getOrCreateNearCache(cacheName, nearCfg);
>> } else {
>>   ignite.cache(cacheName);
>> }
>>   }
>>   LOGGER.info("Caches Started");
>> }
>>
>>
>> Thanks,
>> Prasad
>>
>> On Mon, Apr 29, 2019 at 5:34 PM Prasad Bhalerao <
>> prasadbhalerao1...@gmail.com> wrote:
>>
>>> Yes, I found the issue. Actually it was in my Junit. I had accidently
>>> added following line in my junit case. This was starting the cache and I
>>> was trying to get  it using my service call.
>>>
>>> //final IgniteCache cache = 
>>> ignite.cache(CacheName.ASSET_GROUP_CACHE.name());
>>>
>>> Thanks,
>>> Prasad
>>>
>>>
>>>
>>> On Mon, Apr 29, 2019 at 5:10 PM Ilya Kasnacheev <
>>> ilya.kasnach...@gmail.com> wrote:
>>>
>>>> Hello!
>>>>
>>>> As this error points out, you can't have near cache with same name as
>>>> already existing regular cache.
>>>>
>>>> Regards,
>>>> --
>>>> Ilya Kasnacheev
>>>>
>>>>
>>>> пн, 29 апр. 2019 г. в 13:52, Prasad Bhalerao <
>>>> prasadbhalerao1...@gmail.com>:
>>>>
>>>>> Hi,
>>>>>
>>>>> I am getting "*Failed to start near cache* exception" while starting
>>>>> near cache only on client node. Near cache is not configured on server
>>>>> nodes.
>>>>>
>>>>> Cache configuration on server is node is as follows:
>>>>>
>>>>> private CacheConfiguration agCacheCfg() {
>>>>>
>>>>>   CacheConfiguration agCacheCfg = new 
>>>>> CacheConfiguration<>(CacheName.AG_CACHE.name());
>>>>>   agCacheCfg.setAtomicityMode(CacheAtomicityMode.TRANSACTIONAL);
>>>>>   agCacheCfg.setWriteThrough(ENABLE_WRITE_THROUGH);
>>>>>   agCacheCfg.setReadThrough(false);
>&

Re: Query where clause on byte array

[Prasad Bhalerao]

The problem with BigInteger is it takes around 80 bytes. I will be having 2
BigInteger fields in my class which will take around 176 bytes per record.
I will be having around 40 to 60 million such entries in my cache.

So I am thinking to squeeze some bytes using two long numbers or using two
byte arrays.

How can I create index on byte array field and create index on it and use
it in where clause. Can you give simple example?

Also if I use Int128 data type, how can I create index on it and use it in
where clause?
Simple example would be helpful.

Thanks,
Prasad

On Fri 28 Jun, 2019, 10:24 PM Ilya Lantukh  Hi,
>
> Theoretically, you can create an index and use >= and <= comparisons for
> any data type. In your particular case, I think, using BigInteger is the
> most straightforward approach.
>
> Hope this helps.
>
> On Fri, Jun 28, 2019 at 9:39 AM Prasad Bhalerao <
> prasadbhalerao1...@gmail.com> wrote:
>
>> Hi,
>>
>> I want to store 128 bit number as a one the field in value object in my
>> cache.
>>
>> I can do that using multiple ways.
>> e.g.
>> 1) I can store 128 bit number using java.math.BigInteger.
>>   a) But If store it using BigInteger, can I create index on this
>> field?
>>   b) How can I use this field in where clause to filter the data?
>>
>> 2) I can store 128 bit number using byte array (byte[]).
>> a) Can I create an index on byte array?
>> b) Can I use this byte array field in where clause to filter the data.
>> e.g.  mostly the where clause would be bytearr >=? and   bytearr
>> <=
>>
>> 3) I can implement my own Number datatype e.g. Int128 using two long
>> numbers and and I implement compareTo method which is a bit tricky.
>>a) I can create index on nested objects but how I can use this Int128
>> data type field in where clause and make use of overridden compareTo method
>> to filter the data?
>>
>> Can someone please advise?
>>
>>
>>
>> Thanks,
>> Prasad
>>
>>
>>

Re: Issue in starting Near cache only on client node

[Prasad Bhalerao]

Can some please advise?

On Thu, Jun 27, 2019 at 1:01 PM Prasad Bhalerao <
prasadbhalerao1...@gmail.com> wrote:

> Hi,
>
> I am getting following error on ignite client nodes.
>
> *class org.apache.ignite.IgniteCheckedException: Failed to start near
> cache (a cache with the same name without near cache is already started*
>
> To solve this issue I added following code in ApplicationRunner's run()
> method to explicitly start the cache. Application runner Interface is
> provided by spring boot to execute required code just before server startup.
>
> But even after doing this I am still getting error "a cache with the same
> name without near cache is already started".
>
> I see following info messages in my log file *multiple times*. First time
> it is logged when ApplicationRunner.run() method is invoked during server
> startup.
>
> *o.a.i.i.p.cache.GridCacheProcessor - Started cache [name=DOMAIN_CACHE,
> id=-2139013081, memoryPolicyName=null, mode=PARTITIONED,
> atomicity=TRANSACTIONAL, backups=1]*
>
> On which events GridCacheProcessor starts the cache?
>
> Can this event be intercepted or handled to start near cache?
>
> What is the reason behind getting "GridCacheProcessor - Started cache"
> info messages *multiple* *times *in log files?
>
> Is near cache is destroyed/invalidate when client node is reconnected to
> the cluster?
>
> Can some please advise?
>
> *CODE TO START caches explicitly:*
>
> @Override
> public void run(ApplicationArguments args) throws Exception {
>   for (CacheName cacheName : CacheName.values()) {
> //We are configuring few caches only on client node for fast lookup 
> during validations.
> if (ignite.configuration().isClientMode() && 
> CacheName.isNearCache(cacheName)) {
>   NearCacheConfiguration nearCfg = new NearCacheConfiguration<>();
>   nearCfg.setNearEvictionPolicyFactory(new 
> LruEvictionPolicyFactory<>(nearCacheSize));
>   ignite.getOrCreateNearCache(cacheName, nearCfg);
> } else {
>   ignite.cache(cacheName);
> }
>   }
>   LOGGER.info("Caches Started");
> }
>
>
> Thanks,
> Prasad
>
> On Mon, Apr 29, 2019 at 5:34 PM Prasad Bhalerao <
> prasadbhalerao1...@gmail.com> wrote:
>
>> Yes, I found the issue. Actually it was in my Junit. I had accidently
>> added following line in my junit case. This was starting the cache and I
>> was trying to get  it using my service call.
>>
>> //final IgniteCache cache = 
>> ignite.cache(CacheName.ASSET_GROUP_CACHE.name());
>>
>> Thanks,
>> Prasad
>>
>>
>>
>> On Mon, Apr 29, 2019 at 5:10 PM Ilya Kasnacheev <
>> ilya.kasnach...@gmail.com> wrote:
>>
>>> Hello!
>>>
>>> As this error points out, you can't have near cache with same name as
>>> already existing regular cache.
>>>
>>> Regards,
>>> --
>>> Ilya Kasnacheev
>>>
>>>
>>> пн, 29 апр. 2019 г. в 13:52, Prasad Bhalerao <
>>> prasadbhalerao1...@gmail.com>:
>>>
>>>> Hi,
>>>>
>>>> I am getting "*Failed to start near cache* exception" while starting
>>>> near cache only on client node. Near cache is not configured on server
>>>> nodes.
>>>>
>>>> Cache configuration on server is node is as follows:
>>>>
>>>> private CacheConfiguration agCacheCfg() {
>>>>
>>>>   CacheConfiguration agCacheCfg = new 
>>>> CacheConfiguration<>(CacheName.AG_CACHE.name());
>>>>   agCacheCfg.setAtomicityMode(CacheAtomicityMode.TRANSACTIONAL);
>>>>   agCacheCfg.setWriteThrough(ENABLE_WRITE_THROUGH);
>>>>   agCacheCfg.setReadThrough(false);
>>>>   agCacheCfg.setRebalanceMode(CacheRebalanceMode.ASYNC);
>>>>   
>>>> agCacheCfg.setWriteSynchronizationMode(CacheWriteSynchronizationMode.FULL_SYNC);
>>>>   agCacheCfg.setBackups(1);
>>>>   agCacheCfg.setIndexedTypes(DefaultDataAffinityKey.class, AgData.class);
>>>>   Factory storeFactory = 
>>>> FactoryBuilder.factoryOf(AgDataCacheStore.class);
>>>>   agCacheCfg.setCacheStoreFactory(storeFactory);
>>>>   
>>>> agCacheCfg.setCacheStoreSessionListenerFactories(cacheStoreSessionListenerFactory());
>>>>   agCacheCfg.setSqlIndexMaxInlineSize(100);
>>>>   RendezvousAffinityFunction affinityFunction = new 
>>>> RendezvousAffinityFunction();
>>>>   affinityFunction.setExcludeNeighbors(true);
>>>>   agCacheCf

Query where clause on byte array

[Prasad Bhalerao]

Hi,

I want to store 128 bit number as a one the field in value object in my
cache.

I can do that using multiple ways.
e.g.
1) I can store 128 bit number using java.math.BigInteger.
  a) But If store it using BigInteger, can I create index on this field?
  b) How can I use this field in where clause to filter the data?

2) I can store 128 bit number using byte array (byte[]).
a) Can I create an index on byte array?
b) Can I use this byte array field in where clause to filter the data.
e.g.  mostly the where clause would be bytearr >=? and   bytearr <=

3) I can implement my own Number datatype e.g. Int128 using two long
numbers and and I implement compareTo method which is a bit tricky.
   a) I can create index on nested objects but how I can use this Int128
data type field in where clause and make use of overridden compareTo method
to filter the data?

Can someone please advise?



Thanks,
Prasad

Re: Issue in starting Near cache only on client node

[Prasad Bhalerao]

Hi,

I am getting following error on ignite client nodes.

*class org.apache.ignite.IgniteCheckedException: Failed to start near cache
(a cache with the same name without near cache is already started*

To solve this issue I added following code in ApplicationRunner's run()
method to explicitly start the cache. Application runner Interface is
provided by spring boot to execute required code just before server startup.

But even after doing this I am still getting error "a cache with the same
name without near cache is already started".

I see following info messages in my log file *multiple times*. First time
it is logged when ApplicationRunner.run() method is invoked during server
startup.

*o.a.i.i.p.cache.GridCacheProcessor - Started cache [name=DOMAIN_CACHE,
id=-2139013081, memoryPolicyName=null, mode=PARTITIONED,
atomicity=TRANSACTIONAL, backups=1]*

On which events GridCacheProcessor starts the cache?

Can this event be intercepted or handled to start near cache?

What is the reason behind getting "GridCacheProcessor - Started cache" info
messages *multiple* *times *in log files?

Is near cache is destroyed/invalidate when client node is reconnected to
the cluster?

Can some please advise?

*CODE TO START caches explicitly:*

@Override
public void run(ApplicationArguments args) throws Exception {
  for (CacheName cacheName : CacheName.values()) {
//We are configuring few caches only on client node for fast
lookup during validations.
if (ignite.configuration().isClientMode() &&
CacheName.isNearCache(cacheName)) {
  NearCacheConfiguration nearCfg = new NearCacheConfiguration<>();
  nearCfg.setNearEvictionPolicyFactory(new
LruEvictionPolicyFactory<>(nearCacheSize));
  ignite.getOrCreateNearCache(cacheName, nearCfg);
} else {
  ignite.cache(cacheName);
}
  }
  LOGGER.info("Caches Started");
}


Thanks,
Prasad

On Mon, Apr 29, 2019 at 5:34 PM Prasad Bhalerao <
prasadbhalerao1...@gmail.com> wrote:

> Yes, I found the issue. Actually it was in my Junit. I had accidently
> added following line in my junit case. This was starting the cache and I
> was trying to get  it using my service call.
>
> //final IgniteCache cache = 
> ignite.cache(CacheName.ASSET_GROUP_CACHE.name());
>
> Thanks,
> Prasad
>
>
>
> On Mon, Apr 29, 2019 at 5:10 PM Ilya Kasnacheev 
> wrote:
>
>> Hello!
>>
>> As this error points out, you can't have near cache with same name as
>> already existing regular cache.
>>
>> Regards,
>> --
>> Ilya Kasnacheev
>>
>>
>> пн, 29 апр. 2019 г. в 13:52, Prasad Bhalerao <
>> prasadbhalerao1...@gmail.com>:
>>
>>> Hi,
>>>
>>> I am getting "*Failed to start near cache* exception" while starting
>>> near cache only on client node. Near cache is not configured on server
>>> nodes.
>>>
>>> Cache configuration on server is node is as follows:
>>>
>>> private CacheConfiguration agCacheCfg() {
>>>
>>>   CacheConfiguration agCacheCfg = new 
>>> CacheConfiguration<>(CacheName.AG_CACHE.name());
>>>   agCacheCfg.setAtomicityMode(CacheAtomicityMode.TRANSACTIONAL);
>>>   agCacheCfg.setWriteThrough(ENABLE_WRITE_THROUGH);
>>>   agCacheCfg.setReadThrough(false);
>>>   agCacheCfg.setRebalanceMode(CacheRebalanceMode.ASYNC);
>>>   
>>> agCacheCfg.setWriteSynchronizationMode(CacheWriteSynchronizationMode.FULL_SYNC);
>>>   agCacheCfg.setBackups(1);
>>>   agCacheCfg.setIndexedTypes(DefaultDataAffinityKey.class, AgData.class);
>>>   Factory storeFactory = 
>>> FactoryBuilder.factoryOf(AgDataCacheStore.class);
>>>   agCacheCfg.setCacheStoreFactory(storeFactory);
>>>   
>>> agCacheCfg.setCacheStoreSessionListenerFactories(cacheStoreSessionListenerFactory());
>>>   agCacheCfg.setSqlIndexMaxInlineSize(100);
>>>   RendezvousAffinityFunction affinityFunction = new 
>>> RendezvousAffinityFunction();
>>>   affinityFunction.setExcludeNeighbors(true);
>>>   agCacheCfg.setAffinity(affinityFunction);
>>>   agCacheCfg.setStatisticsEnabled(true);
>>>
>>>
>>>   return agCacheCfg;
>>> }
>>>
>>>
>>> I am using following code to get the Near cache on client node only for
>>> configured cache names.
>>> Can some please help me with this?
>>>
>>> protected IgniteCache cache() {
>>>   String cacheName = getCacheName();
>>>   if (this.cache == null) {
>>> //We are configuring few caches only on client node for fast lookup 
>>> during validations.
>>> if (ignite.configuration().

Re: Near cache only on client node

[Prasad Bhalerao]

Yes, I found the issue. Actually it was in my Junit. I had accidently added
following line in my junit case. This was starting the cache and I was
trying to get  it using my service call.

//final IgniteCache cache =
ignite.cache(CacheName.ASSET_GROUP_CACHE.name());

Thanks,
Prasad



On Mon, Apr 29, 2019 at 5:10 PM Ilya Kasnacheev 
wrote:

> Hello!
>
> As this error points out, you can't have near cache with same name as
> already existing regular cache.
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> пн, 29 апр. 2019 г. в 13:52, Prasad Bhalerao  >:
>
>> Hi,
>>
>> I am getting "*Failed to start near cache* exception" while starting
>> near cache only on client node. Near cache is not configured on server
>> nodes.
>>
>> Cache configuration on server is node is as follows:
>>
>> private CacheConfiguration agCacheCfg() {
>>
>>   CacheConfiguration agCacheCfg = new 
>> CacheConfiguration<>(CacheName.AG_CACHE.name());
>>   agCacheCfg.setAtomicityMode(CacheAtomicityMode.TRANSACTIONAL);
>>   agCacheCfg.setWriteThrough(ENABLE_WRITE_THROUGH);
>>   agCacheCfg.setReadThrough(false);
>>   agCacheCfg.setRebalanceMode(CacheRebalanceMode.ASYNC);
>>   
>> agCacheCfg.setWriteSynchronizationMode(CacheWriteSynchronizationMode.FULL_SYNC);
>>   agCacheCfg.setBackups(1);
>>   agCacheCfg.setIndexedTypes(DefaultDataAffinityKey.class, AgData.class);
>>   Factory storeFactory = 
>> FactoryBuilder.factoryOf(AgDataCacheStore.class);
>>   agCacheCfg.setCacheStoreFactory(storeFactory);
>>   
>> agCacheCfg.setCacheStoreSessionListenerFactories(cacheStoreSessionListenerFactory());
>>   agCacheCfg.setSqlIndexMaxInlineSize(100);
>>   RendezvousAffinityFunction affinityFunction = new 
>> RendezvousAffinityFunction();
>>   affinityFunction.setExcludeNeighbors(true);
>>   agCacheCfg.setAffinity(affinityFunction);
>>   agCacheCfg.setStatisticsEnabled(true);
>>
>>
>>   return agCacheCfg;
>> }
>>
>>
>> I am using following code to get the Near cache on client node only for
>> configured cache names.
>> Can some please help me with this?
>>
>> protected IgniteCache cache() {
>>   String cacheName = getCacheName();
>>   if (this.cache == null) {
>> //We are configuring few caches only on client node for fast lookup 
>> during validations.
>> if (ignite.configuration().isClientMode() && 
>> NearCacheName.isNearCache(cacheName)) {
>>   NearCacheConfiguration nearCfg = new NearCacheConfiguration<>();
>>   nearCfg.setNearEvictionPolicyFactory(new 
>> LruEvictionPolicyFactory<>(10_000));
>>   this.cache = ignite.createNearCache(cacheName, nearCfg);
>> } else {
>>   this.cache = ignite.cache(cacheName);
>> }
>>   }
>>   return this.cache;
>> }
>>
>>
>> *EXCEPTION:*
>>
>> *javax.cache.CacheException: class org.apache.ignite.IgniteCheckedException: 
>> Failed to start near cache (a cache with the same name without near cache is 
>> already started)*
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> * at 
>> org.apache.ignite.internal.processors.cache.GridCacheUtils.convertToCacheException(GridCacheUtils.java:1303)
>>  at 
>> org.apache.ignite.internal.IgniteKernal.createNearCache(IgniteKernal.java:2960)
>>  at 
>> org.apache.ignite.IgniteSpringBean.createNearCache(IgniteSpringBean.java:382)
>>  at 
>> com.qualys.agms.grid.dao.AbstractDataGridDAO.cache(AbstractDataGridDAO.java:112)
>>  at 
>> com.qualys.agms.grid.dao.AbstractDataGridDAO.putData(AbstractDataGridDAO.java:47)
>>  at 
>> com.qualys.agms.grid.dao.AbstractDataGridDAO.putData(AbstractDataGridDAO.java:31)
>>  at 
>> com.qualys.agms.grid.dataservice.DefaultDataGridService.putData(DefaultDataGridService.java:27)
>>  at 
>> com.qualys.agms.grid.dataservice.DefaultDataGridService.putData(DefaultDataGridService.java:10)
>>  at com.qualys.agms.IgniteClientTester.testGet(IgniteClientTester.java:382) 
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at 
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>  at 
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>

Re: Near cache only on client node

[Prasad Bhalerao]

RunBeforeTestClassCallbacks.evaluate(RunBeforeTestClassCallbacks.java:61)
at 
org.springframework.test.context.junit4.statements.RunAfterTestClassCallbacks.evaluate(RunAfterTestClassCallbacks.java:70)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at
org.springframework.test.context.junit4.SpringJUnit4ClassRunner.run(SpringJUnit4ClassRunner.java:190)
at org.junit.runner.JUnitCore.run(JUnitCore.java:137) at
com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
at 
com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47)
at 
com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)Caused
by: class org.apache.ignite.IgniteCheckedException: Failed to start
near cache (a cache with the same name without near cache is already
started) at 
org.apache.ignite.internal.IgniteKernal.checkNearCacheStarted(IgniteKernal.java:3019)
at 
org.apache.ignite.internal.IgniteKernal.createNearCache(IgniteKernal.java:2955)
... 37 more*


Thanks,
Prasad

On Sat, Apr 20, 2019 at 9:05 AM Denis Magda  wrote:

> Hi Prasad,
>
> It's required to define a NearCacheConfiguration on a node you'd like to
> have it one. Use one of the approaches defined in the docs:
> https://apacheignite.readme.io/docs/near-caches
>
> Near caches can be created directly on *client* nodes by passing
> NearCacheConfiguration to the
> Ignite.createNearCache(NearCacheConfiguration) or
> Ignite.getOrCreateNearCache(NearCacheConfiguration) methods. Use the 
> Ignite.getOrCreateCache(CacheConfiguration,
> NearCacheConfiguration) method if you need to both start a distributed
> cache dynamically and create a near cache for it.
>
> -
> Denis
>
>
> On Fri, Apr 19, 2019 at 5:53 AM Prasad Bhalerao <
> prasadbhalerao1...@gmail.com> wrote:
>
>> Hi,
>>
>> Is there any to enable near cache only on client nodes?
>>
>> I am using partitioned cache and using colocation to submit the ignite
>> jobs. But in my code I do few lookups on client node to validate the data.
>> To improve this lookup performance I want to enable near cache.
>>
>> I  saw the example on GitHub and in ignite docs. But I am still
>> struggling to configure near cache on client nodes only.
>>
>> This my Ignite configuration on client node. All the cache configurations
>> are defined only on ignite server nodes.
>>
>> Can some please tell me where to include near cache configuration in
>> following config?
>>
>> private IgniteConfiguration getIgniteConfiguration(IgniteSpringBean ignite) {
>>
>>   String[] hosts = appConfigProperties.getHosts().split(",");
>>   TcpDiscoveryVmIpFinder ipFinder = new TcpDiscoveryVmIpFinder();
>>   ipFinder.setAddresses(Arrays.asList(hosts));
>>
>>   TcpDiscoverySpi discoSpi = new TcpDiscoverySpi();
>>   discoSpi.setIpFinder(ipFinder);
>>
>>   IgniteConfiguration cfg = new IgniteConfiguration();
>>   cfg.setDiscoverySpi(discoSpi);
>>   cfg.setIgniteInstanceName("springDataNode");
>>   cfg.setPeerClassLoadingEnabled(false);
>>   cfg.setRebalanceThreadPoolSize(4);
>>   cfg.setClientFailureDetectionTimeout(failureDetectionTimeout);
>>   //cfg.setLifecycleBeans(new IgniteLifeCycleBean());
>>   cfg.setClientMode(true);
>>   HashMap attributeMap = new HashMap<>();
>>   attributeMap.put(NODE_TYPE,NODE_TYPE_WEB);
>>   cfg.setUserAttributes(attributeMap);
>>
>>   return cfg;
>> }
>>
>>
>>
>> Thanks,
>> Prasad
>>
>

Near cache only on client node

[Prasad Bhalerao]

Hi,

Is there any to enable near cache only on client nodes?

I am using partitioned cache and using colocation to submit the ignite
jobs. But in my code I do few lookups on client node to validate the data.
To improve this lookup performance I want to enable near cache.

I  saw the example on GitHub and in ignite docs. But I am still struggling
to configure near cache on client nodes only.

This my Ignite configuration on client node. All the cache configurations
are defined only on ignite server nodes.

Can some please tell me where to include near cache configuration in
following config?

private IgniteConfiguration getIgniteConfiguration(IgniteSpringBean ignite) {

  String[] hosts = appConfigProperties.getHosts().split(",");
  TcpDiscoveryVmIpFinder ipFinder = new TcpDiscoveryVmIpFinder();
  ipFinder.setAddresses(Arrays.asList(hosts));

  TcpDiscoverySpi discoSpi = new TcpDiscoverySpi();
  discoSpi.setIpFinder(ipFinder);

  IgniteConfiguration cfg = new IgniteConfiguration();
  cfg.setDiscoverySpi(discoSpi);
  cfg.setIgniteInstanceName("springDataNode");
  cfg.setPeerClassLoadingEnabled(false);
  cfg.setRebalanceThreadPoolSize(4);
  cfg.setClientFailureDetectionTimeout(failureDetectionTimeout);
  //cfg.setLifecycleBeans(new IgniteLifeCycleBean());
  cfg.setClientMode(true);
  HashMap attributeMap = new HashMap<>();
  attributeMap.put(NODE_TYPE,NODE_TYPE_WEB);
  cfg.setUserAttributes(attributeMap);

  return cfg;
}



Thanks,
Prasad

Re: queries, we are evaluating to use Apache Ignite as caching layer on top of underlying Cassandra database.

[Prasad Bhalerao]

Just out of curiosity how are you planning to load 1 TB of data in cache,
using datastreamer or cachestore?
What's the expected time to load cache?
Since you are not keeping back up, how are you going to handle the
situation when any of the node crashes? This can happen in prod env, so
what is the expected down time for you.

 To load 15-20 GB of data(from Oracle tables) in different caches,  with
backup count 1 it is around 40 Gb, my application is taking 28-30 minutes.


Please share your results if possible.

Thanks,
Prasad

On Wed 6 Mar, 2019, 2:58 PM Navneet Kumar  Ilya,
> Thanks for your quick response. I have gone through the capacity planning
> link shared by you.
> 1,000,000,000 Total objects(Records)
> 1,024 bytes per object (1 KB)
> 0 backup
> 4 nodes
>
> Total number of objects X object size (only primary copy since back up is
> set 0. Better remove the back up property from XML):
> 1,000,000,000 x 1,024 = 10240 bytes (1024000 MB)
>
> No Indexes used. I know if it is used it will take 30% of overhead more.
>
> Approximate additional memory required by the platform:
> 300 MB x 1(No of node in the cluster) = 300 MB
>
> Total size:
> 1024000 + 300 = 1024300 MB
>
> Hence the anticipated total memory consumption would be just over ~ 1024.3
> GB
>
>
> So what I want to know that my use case is I want to load full 1 billion
> subscriber records on datagrid(Apache Ignite) and will read from there. No
> disk swapping once my data is loaded in memory.
> Let me know my calculation is correct or do I need to add some more memory.
> I have a single node cluster as of now and I am not using any index and no
> back up.
>
> Regards
> Navneet
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Native memory tracking of an Ignite Java 1.8 Application

[Prasad Bhalerao]

Hi,

Thank you for the hint. I just wrote a small program to allocate 1 GB
memory and to free the same memory using UNSAFE api. I took the native
memory tracking report 3 times, before allocating 1 GB memory, after
allocating 1 GB memory and after freeing 1 GB memory.

Here is the snippet of reports. From the result I can conclude that
allocated offheap memory using UNSAFE api is listed under "Internal"
category of the report.

NOTE:  DirectByteBuffer internally uses unsafe apis.

1) Before allocating 1 GB memory (off heap).

-  Internal (reserved=9782KB, committed=9782KB)
(malloc=9718KB #3360)
(mmap: reserved=64KB, committed=64KB)

2) After allocating 1 GB memory (off heap).

-  Internal (reserved=1058386KB, committed=1058386KB)
(malloc=1058322KB #3415)
(mmap: reserved=64KB, committed=64KB)

3) After freeing up 1GB memory

-  Internal (reserved=9798KB, committed=9798KB)
(malloc=9734KB #3299)
(mmap: reserved=64KB, committed=64KB)

Sample program:

public static void main(String[] args) throws NoSuchFieldException,
IllegalAccessException, InterruptedException {

  Field f = Unsafe.class.getDeclaredField("theUnsafe");
  f.setAccessible(true);
  final Unsafe unsafe = (Unsafe) f.get(null);

  System.out.println("1)Now going to sleep");
  Thread.sleep(6);
  System.out.println("Now allocating 1GB off-heap.");
  long address = unsafe.allocateMemory(1024 * 1024 * 1024);
  System.out.println("Allocated 1GB off-heap.");
  System.out.println("2)Now going to sleep");
  Thread.sleep(6);
  unsafe.freeMemory(address);
  System.out.println("3)Now going to sleep");
  Thread.sleep(6);
  System.out.println("Exited.");
}



Thanks.,
Prasad

On Mon, Feb 18, 2019 at 7:56 PM Павлухин Иван  wrote:

> Prasad,
>
> Someone has already posted a snippet [1].
>
> [1] https://gist.github.com/prasanthj/48e7063cac88eb396bc9961fb3149b58
>
> пн, 18 февр. 2019 г. в 17:23, Павлухин Иван :
> >
> > Hi Prasad,
> >
> > As far as I remember offheap memory allocated with use of Unsafe is
> > not reflected in Native Memory Tracking report. You are right that
> > documentation is not verbose about reported categories [1]. It might
> > be the case that memory allocated by ByteBuffer.allocateDirect falls
> > into "internal" category. You can check it out by writing an example
> > application using ByteBuffer.allocateDirect.
> >
> > [1]
> https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/tooldescr022.html#BABHIFJC
> >
> > пн, 18 февр. 2019 г. в 16:04, Dmitriy Pavlov :
> > >
> > > Hi.
> > >
> > > Please don't use posting to both dev/user lists simultaneously.
> > >
> > > If your question is not related to any contribution you are planning
> to do, then the user list is a better place to ask, because a possible
> answer may be interesting to all Ignite users.
> > >
> > > If you are going to fix any issue and would like to discuss a
> proposal, please use dev list.
> > >
> > > Sincerely,
> > > Dmitriy Pavlov
> > >
> > > пн, 18 февр. 2019 г. в 16:00, Prasad Bhalerao <
> prasadbhalerao1...@gmail.com>:
> > >>
> > >> Hi,
> > >>
> > >> I have set the off heap size to 500 MB and max heap size to 512 MB.
> > >>
> > >> My process is taking around 1.7 GB on Windows 10 as per the task
> manager. So I decided to track the memory distribution using jcmd to find
> out if there are any memory leaks in non-heap space.
> > >>
> > >> After pushing the data to cache I took the native memory summary
> using jcmd tool.
> > >>
> > >> I am trying to understand in which of the following section,
> allocated off heap memory goes?
> > >>
> > >> Does off heap come under "Internal" category?  Can any ignite memory
> expert help me with this?
> > >>
> > >> Oracle documentation  does not clearly talks about it. I am also
> attaching the native memory detail file.
> > >>
> > >> C:\Java64\jdk1.8.0_144\bin>jcmd.exe 16956 VM.native_memory summary
> > >>
> > >> 16956:
> > >>
> > >>  Total: reserved=3513712KB, committed=2249108KB
> > >>
> > >> - Java Heap (reserved=524288KB, committed=524288KB)
> > >>
> > >> (mmap: rese

Native memory tracking of an Ignite Java 1.8 Application

[Prasad Bhalerao]

Hi,

I have set the off heap size to 500 MB and max heap size to 512 MB.

My process is taking around 1.7 GB on Windows 10 as per the task manager.
So I decided to track the memory distribution using jcmd to find out if
there are any memory leaks in non-heap space.

After pushing the data to cache I took the native memory summary using jcmd
tool.

I am trying to understand in which of the following section, allocated off
heap memory goes?

Does off heap come under "*Internal*" category?  Can any ignite memory
expert help me with this?

Oracle documentation

does
not clearly talks about it. I am also attaching the native memory detail
file.

C:\Java64\jdk1.8.0_144\bin>jcmd.exe 16956 VM.native_memory summary

16956:

 Total: reserved=3513712KB, committed=2249108KB

- Java Heap (reserved=524288KB, committed=524288KB)

(mmap: reserved=524288KB, committed=524288KB)



- Class (reserved=1127107KB, committed=86507KB)

(classes #13259)

(malloc=10947KB #17120)

(mmap: reserved=1116160KB, committed=75560KB)



-Thread (reserved=89748KB, committed=89748KB)

(thread #88)

(stack: reserved=89088KB, committed=89088KB)

(malloc=270KB #454)

(arena=391KB #175)



-  Code (reserved=254854KB, committed=30930KB)

(malloc=5254KB #8013)

(mmap: reserved=249600KB, committed=25676KB)



-GC (reserved=29656KB, committed=29576KB)

(malloc=10392KB #385)

(mmap: reserved=19264KB, committed=19184KB)



-  Compiler (reserved=188KB, committed=188KB)

(malloc=57KB #243)

(arena=131KB #3)



-  Internal (reserved=1464736KB, committed=1464736KB)

(malloc=1464672KB #40848)

(mmap: reserved=64KB, committed=64KB)



-Symbol (reserved=18973KB, committed=18973KB)

(malloc=15353KB #152350)

(arena=3620KB #1)



-Native Memory Tracking (reserved=3450KB, committed=3450KB)

(malloc=14KB #167)

(tracking overhead=3436KB)



-   Arena Chunk (reserved=712KB, committed=712KB)

(malloc=712KB)


mem_detail
Description: Binary data

Re: cache.removeAll cache operation is slow

[Prasad Bhalerao]

The data I am removing from cache is colocated on the same node. So I think
exchange will happen only for updating the backup copies. Please correct me
if I am wrong.

I believe that remove all operation first acquires the lock on all input
keys and then removes the data. If I go for single remove, will it not
acquire lock per key per remove operation. Will it not slow down?

Will try your suggestions.



On Tue 22 Jan, 2019, 7:22 PM Denis Mekhanikov  Prasad,
>
> When you run a transaction, that involves many entries, the whole key set
> is sent between nodes multiple times.
> It also generates a lot of garbage, which cannot be released until the
> transaction is committed.
> It's better to remove values in small batches.
> So, try changing the batch size. It's possible, that singular removes will
> work faster.
>
> Denis
>
> вт, 22 янв. 2019 г. в 16:37, Prasad Bhalerao  >:
>
>> Hi,
>>
>> I am removing around 2 million entries from cache using cache.removeAll
>> api.
>> I removing entries in a batch of 100k.  To remove 2 million
>> entries ignite takes around 218 seconds.
>> This cache is transactiona and write synchronization mode is full sync.
>>
>> Is there any way to improve the performance of removeAll operation?
>>
>> Thanks,
>> Prasad
>>
>>
>>

cache.removeAll cache operation is slow

[Prasad Bhalerao]

Hi,

I am removing around 2 million entries from cache using cache.removeAll api.
I removing entries in a batch of 100k.  To remove 2 million entries ignite
takes around 218 seconds.
This cache is transactiona and write synchronization mode is full sync.

Is there any way to improve the performance of removeAll operation?

Thanks,
Prasad

Re: [IGNITE-10925] After upgrading 2.7 getting Unexpected error occurred during unmarshalling

[Prasad Bhalerao]

Hi,

I am able to create a reproducer for this issue. I have also created a JIRA
IGNITE-10925 <https://issues.apache.org/jira/browse/IGNITE-10925>  for this
issue.

Reproducer: https://github.com/prasadbhalerao1983/IgniteIssueReproducer.git

Step to Reproduce:

1) First Run com.example.demo.Server class as a java program

2) Then run com.example.demo.Client as java program.

Thanks,
Prasad

On Sat, Jan 12, 2019 at 11:17 AM Prasad Bhalerao <
prasadbhalerao1...@gmail.com> wrote:

> No I am not using zookeeper discovery.
> Using TcpDiscoveryVmIpFinder.
>
> Can someone please explain on what event cacheMetrics in TcpDiscoveryNode
> gets populated. It is not getting populated in standalone program.
>
> If it gets populated then I might be able to reproduce this case.
>
> On Fri 11 Jan, 2019, 8:28 PM Ilya Kasnacheev  wrote:
>
>> Hello!
>>
>> Have you tried enabling Zookeeper in your reproducer? I have a hunch that
>> they are linked: this behavior is affected by zookeeper discovery.
>>
>> Regards,
>> --
>> Ilya Kasnacheev
>>
>>
>> пт, 11 янв. 2019 г. в 17:44, Prasad Bhalerao <
>> prasadbhalerao1...@gmail.com>:
>>
>>> I tried to reproduce this in standalone program. But the thing is cache
>>> metrics map in TcpDiscoveryNode is empty even after setting
>>> statisticEnabled to true on all caches.
>>> So the flow does not enter into serializr/deserialize cacheMetrics block.
>>>
>>> Any idea how the cacheMetrics gets populated. On which event?
>>>
>>>
>>> Thanks,
>>> Prasad
>>>
>>> On Fri 11 Jan, 2019, 7:55 PM ilya.kasnacheev >> wrote:
>>>
>>>> Hello!
>>>>
>>>> I think the problem was introduced by
>>>> https://issues.apache.org/jira/browse/IGNITE-6846 which does look very
>>>> suspicious, however it is strange that it does not reproduce right away.
>>>>
>>>> I could try and devise a fix but I could not reproduce this behavior in
>>>> any
>>>> of tests. If you could do a reproducer project that would be awesome.
>>>>
>>>> Regards,
>>>>
>>>>
>>>>
>>>> --
>>>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>>>
>>>

Re: After upgrading 2.7 getting Unexpected error occurred during unmarshalling

[Prasad Bhalerao]

No I am not using zookeeper discovery.
Using TcpDiscoveryVmIpFinder.

Can someone please explain on what event cacheMetrics in TcpDiscoveryNode
gets populated. It is not getting populated in standalone program.

If it gets populated then I might be able to reproduce this case.

On Fri 11 Jan, 2019, 8:28 PM Ilya Kasnacheev  Hello!
>
> Have you tried enabling Zookeeper in your reproducer? I have a hunch that
> they are linked: this behavior is affected by zookeeper discovery.
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> пт, 11 янв. 2019 г. в 17:44, Prasad Bhalerao  >:
>
>> I tried to reproduce this in standalone program. But the thing is cache
>> metrics map in TcpDiscoveryNode is empty even after setting
>> statisticEnabled to true on all caches.
>> So the flow does not enter into serializr/deserialize cacheMetrics block.
>>
>> Any idea how the cacheMetrics gets populated. On which event?
>>
>>
>> Thanks,
>> Prasad
>>
>> On Fri 11 Jan, 2019, 7:55 PM ilya.kasnacheev > wrote:
>>
>>> Hello!
>>>
>>> I think the problem was introduced by
>>> https://issues.apache.org/jira/browse/IGNITE-6846 which does look very
>>> suspicious, however it is strange that it does not reproduce right away.
>>>
>>> I could try and devise a fix but I could not reproduce this behavior in
>>> any
>>> of tests. If you could do a reproducer project that would be awesome.
>>>
>>> Regards,
>>>
>>>
>>>
>>> --
>>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>>
>>

Re: After upgrading 2.7 getting error during unmarshalling (Works fine on 2.6)

[Prasad Bhalerao]

Hi Ilya,

I am not able reproduce this issue in small reproducer project but this is
consistently happening in my project. So I debugged this issue and attached
the screenshot in this mail.



*NOTE:  *This issue occurs if the statistics are enabled on cache
configuration level [cacheCfg.setStatisticsEnabled(true)].



As shown in screenshots there 22 cache metrics in cacheMetrics hashmap. All
these cache metrics gets serialized successfully on server node. But at the
time of deserialization on client node only first metrics get de-serialized
successfully but all other metrics till iteration count 13 are
de-serialized as null value and on iteration 14 “ref” byte value in
“OptimizedObjectInputStream.readObject0()” method is read as 81 and code
throws exception.



I think this where it is going wrong. The object copy at the time of
serialization and de-serialization should be same but that’s not happening
in Ignite 2.7 version.

So I debugged this on Ignite 2.6 version. On 2.6 version all 22
cacheMetrics are being de-serialized successfully.


This Looks like a bug to me in serializer/deseriliazer code.



*AffinityJob result being serialized on server:*







*AffinityJob result being de-serialized on client:*




Thanks,
Prasad
On Wed, Jan 9, 2019 at 6:41 PM Ilya Kasnacheev 
wrote:

> Hello!
>
> Do you have a reproducer project to reliably confirm this issue?
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> ср, 9 янв. 2019 г. в 12:39, Akash Shinde :
>
>> Added  d...@ignite.apache.org.
>>
>> Should I log Jira for this issue?
>>
>> Thanks,
>> Akash
>>
>>
>>
>> On Tue, Jan 8, 2019 at 6:16 PM Akash Shinde 
>> wrote:
>>
>> > Hi,
>> >
>> > No both nodes, client and server are running on Ignite 2.7 version. I am
>> > starting both server and client from Intellij IDE.
>> >
>> > Version printed in Server node log:
>> > Ignite ver. 2.7.0#20181201-sha1:256ae4012cb143b4855b598b740a6f3499ead4db
>> >
>> > Version in client node log:
>> > Ignite ver. 2.7.0#20181201-sha1:256ae4012cb143b4855b598b740a6f3499ead4db
>> >
>> > Thanks,
>> > Akash
>> >
>> > On Tue, Jan 8, 2019 at 5:18 PM Mikael 
>> wrote:
>> >
>> >> Hi!
>> >>
>> >> Any chance you might have one node running 2.6 or something like that ?
>> >>
>> >> It looks like it get a different object that does not match the one
>> >> expected in 2.7
>> >>
>> >> Mikael
>> >> Den 2019-01-08 kl. 12:21, skrev Akash Shinde:
>> >>
>> >> Before submitting the affinity task ignite first gets the affinity
>> cached
>> >> function (AffinityInfo) by submitting the cluster wide task
>> "AffinityJob".
>> >> But while in the process of retrieving the output of this AffinityJob,
>> >> ignite deserializes this output. I am getting exception while
>> deserailizing
>> >> this output.
>> >> In TcpDiscoveryNode.readExternal() method while deserailizing the
>> >> CacheMetrics object from input stream on 14th iteration I am getting
>> >> following exception. Complete stack trace is given in this mail chain.
>> >>
>> >> Caused by: java.io.IOException: Unexpected error occurred during
>> >> unmarshalling of an instance of the class:
>> >> org.apache.ignite.internal.processors.cache.CacheMetricsSnapshot.
>> >>
>> >> This is working fine on Ignite 2.6 version but giving problem on 2.7.
>> >>
>> >> Is this a bug or am I doing something wrong?
>> >>
>> >> Can someone please help?
>> >>
>> >> On Mon, Jan 7, 2019 at 9:41 PM Akash Shinde 
>> >> wrote:
>> >>
>> >>> Hi,
>> >>>
>> >>> When execute affinity.partition(key), I am getting following exception
>> >>> on Ignite  2.7.
>> >>>
>> >>> Stacktrace:
>> >>>
>> >>> 2019-01-07 21:23:03,093 6699878 [mgmt-#67%springDataNode%] ERROR
>> >>> o.a.i.i.p.task.GridTaskWorker - Error deserializing job response:
>> >>> GridJobExecuteResponse [nodeId=c0c832cb-33b0-4139-b11d-5cafab2fd046,
>> >>> sesId=4778e982861-31445139-523d-4d44-b071-9ca1eb2d73df,
>> >>> jobId=5778e982861-31445139-523d-4d44-b071-9ca1eb2d73df, gridEx=null,
>> >>> isCancelled=false, retry=null]
>> >>> org.apache.ignite.IgniteCheckedException: Failed to unmarshal object
>> >>> with optimized marshaller
>> >>>  at
>> >>>
>> org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10146)
>> >>>  at
>> >>>
>> org.apache.ignite.internal.processors.task.GridTaskWorker.onResponse(GridTaskWorker.java:831)
>> >>>  at
>> >>>
>> org.apache.ignite.internal.processors.task.GridTaskProcessor.processJobExecuteResponse(GridTaskProcessor.java:1081)
>> >>>  at
>> >>>
>> org.apache.ignite.internal.processors.task.GridTaskProcessor$JobMessageListener.onMessage(GridTaskProcessor.java:1316)
>> >>>  at
>> >>>
>> org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
>> >>>  at
>> >>>
>> org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
>> >>>  at
>> >>>
>> org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
>> >>>  at
>> >>>
>> 

Re: Getting javax.cache.CacheException after upgrading to Ignite 2.7

[Prasad Bhalerao]

Hi Ilya,

I have created a reproducer for this issue and uploaded it to GitHub.

GitHub project: https://github.com/prasadbhalerao1983/IgniteTestPrj.git

Please run IgniteTransactionTester class to check the issue.


Exception:

Exception in thread "main" javax.cache.CacheException: Only pessimistic
repeatable read transactions are supported at the moment.
 at
org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.query(IgniteCacheProxyImpl.java:697)
 at
org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.query(IgniteCacheProxyImpl.java:636)
 at
org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.query(GatewayProtectedCacheProxy.java:388)
 at
IgniteTransactionTester.testTransactionException(IgniteTransactionTester.java:53)
 at IgniteTransactionTester.main(IgniteTransactionTester.java:38)
Caused by: class
org.apache.ignite.internal.processors.query.IgniteSQLException: Only
pessimistic repeatable read transactions are supported at the moment.
 at
org.apache.ignite.internal.processors.cache.mvcc.MvccUtils.tx(MvccUtils.java:690)
 at
org.apache.ignite.internal.processors.cache.mvcc.MvccUtils.tx(MvccUtils.java:671)
 at
org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.runQueryTwoStep(IgniteH2Indexing.java:1793)
 at
org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.doRunDistributedQuery(IgniteH2Indexing.java:2610)
 at
org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.doRunPrepared(IgniteH2Indexing.java:2315)
 at
org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.querySqlFields(IgniteH2Indexing.java:2209)
 at
org.apache.ignite.internal.processors.query.GridQueryProcessor$4.applyx(GridQueryProcessor.java:2135)
 at
org.apache.ignite.internal.processors.query.GridQueryProcessor$4.applyx(GridQueryProcessor.java:2130)
 at
org.apache.ignite.internal.util.lang.IgniteOutClosureX.apply(IgniteOutClosureX.java:36)
 at
org.apache.ignite.internal.processors.query.GridQueryProcessor.executeQuery(GridQueryProcessor.java:2707)
 at
org.apache.ignite.internal.processors.query.GridQueryProcessor.querySqlFields(GridQueryProcessor.java:2144)
 at
org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.query(IgniteCacheProxyImpl.java:685)

Thanks,

Prasad



On Wed, Jan 9, 2019 at 6:22 PM Ilya Kasnacheev 
wrote:

> Hello!
>
> It was discussed recently:
> http://apache-ignite-users.70518.x6.nabble.com/Migrate-from-2-6-to-2-7-td25738.html
>
> I don't think you will be able to use SQL from transactions in Ignite 2.7.
> While this looks like a regression, you will have to work around it for now.
>
> Do you have a small reproducer for this issue? I could file a ticket if
> you had. You can try to do it yourself, too.
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> ср, 9 янв. 2019 г. в 15:33, Prasad Bhalerao  >:
>
>> Hi,
>>
>> My cache configuration is as follows. I am using TRANSACTIONAL and not
>> TRANSACTIONAL_SNAPSHOT.
>>
>>
>>
>> private CacheConfiguration ipContainerIPV4CacheCfg() {
>>
>>   CacheConfiguration ipContainerIpV4CacheCfg = new 
>> CacheConfiguration<>(CacheName.IP_CONTAINER_IPV4_CACHE.name());
>>   ipContainerIpV4CacheCfg.setAtomicityMode(CacheAtomicityMode.TRANSACTIONAL);
>>   ipContainerIpV4CacheCfg.setWriteThrough(ENABLE_WRITE_THROUGH);
>>   ipContainerIpV4CacheCfg.setReadThrough(false);
>>   ipContainerIpV4CacheCfg.setRebalanceMode(CacheRebalanceMode.ASYNC);
>>   
>> ipContainerIpV4CacheCfg.setWriteSynchronizationMode(CacheWriteSynchronizationMode.FULL_SYNC);
>>   ipContainerIpV4CacheCfg.setBackups(1);
>>   Factory storeFactory = 
>> FactoryBuilder.factoryOf(IpContainerIpV4CacheStore.class);
>>   ipContainerIpV4CacheCfg.setCacheStoreFactory(storeFactory);
>>   ipContainerIpV4CacheCfg.setIndexedTypes(DefaultDataAffinityKey.class, 
>> IpContainerIpV4Data.class);
>>   
>> ipContainerIpV4CacheCfg.setCacheStoreSessionListenerFactories(cacheStoreSessionListenerFactory());
>>   ipContainerIpV4CacheCfg.setSqlIndexMaxInlineSize(84);
>>   RendezvousAffinityFunction affinityFunction = new 
>> RendezvousAffinityFunction();
>>   affinityFunction.setExcludeNeighbors(true);
>>   ipContainerIpV4CacheCfg.setAffinity(affinityFunction);
>>   ipContainerIpV4CacheCfg.setStatisticsEnabled(true);
>>
>>   return ipContainerIpV4CacheCfg;
>> }
>>
>>
>> Thanks,
>> Prasad
>>
>> On Wed, Jan 9, 2019 at 5:45 PM Павлухин Иван  wrote:
>>
>>> Hi Prasad,
>>>
>>> > javax.cache.CacheException: Only pessimistic repeatable read
>>> transactions are supported at the moment.
>>> Exception mentioned by you should happen only for cache with
>>> TRANSACTIONAL_SNAPSHOT atomicity mode configured

Re: Getting javax.cache.CacheException after upgrading to Ignite 2.7

[Prasad Bhalerao]

Hi,

My cache configuration is as follows. I am using TRANSACTIONAL and not
TRANSACTIONAL_SNAPSHOT.



private CacheConfiguration ipContainerIPV4CacheCfg() {

  CacheConfiguration ipContainerIpV4CacheCfg = new
CacheConfiguration<>(CacheName.IP_CONTAINER_IPV4_CACHE.name());
  ipContainerIpV4CacheCfg.setAtomicityMode(CacheAtomicityMode.TRANSACTIONAL);
  ipContainerIpV4CacheCfg.setWriteThrough(ENABLE_WRITE_THROUGH);
  ipContainerIpV4CacheCfg.setReadThrough(false);
  ipContainerIpV4CacheCfg.setRebalanceMode(CacheRebalanceMode.ASYNC);
  
ipContainerIpV4CacheCfg.setWriteSynchronizationMode(CacheWriteSynchronizationMode.FULL_SYNC);
  ipContainerIpV4CacheCfg.setBackups(1);
  Factory storeFactory =
FactoryBuilder.factoryOf(IpContainerIpV4CacheStore.class);
  ipContainerIpV4CacheCfg.setCacheStoreFactory(storeFactory);
  ipContainerIpV4CacheCfg.setIndexedTypes(DefaultDataAffinityKey.class,
IpContainerIpV4Data.class);
  
ipContainerIpV4CacheCfg.setCacheStoreSessionListenerFactories(cacheStoreSessionListenerFactory());
  ipContainerIpV4CacheCfg.setSqlIndexMaxInlineSize(84);
  RendezvousAffinityFunction affinityFunction = new
RendezvousAffinityFunction();
  affinityFunction.setExcludeNeighbors(true);
  ipContainerIpV4CacheCfg.setAffinity(affinityFunction);
  ipContainerIpV4CacheCfg.setStatisticsEnabled(true);

  return ipContainerIpV4CacheCfg;
}


Thanks,
Prasad

On Wed, Jan 9, 2019 at 5:45 PM Павлухин Иван  wrote:

> Hi Prasad,
>
> > javax.cache.CacheException: Only pessimistic repeatable read
> transactions are supported at the moment.
> Exception mentioned by you should happen only for cache with
> TRANSACTIONAL_SNAPSHOT atomicity mode configured. Have you configured
> TRANSACTIONAL_SNAPSHOT atomicity for any cache? As Denis mentioned
> there are number of bugs related to TRANSACTIONAL_SNAPSHOT, e.g. [1].
>
> [1] https://issues.apache.org/jira/browse/IGNITE-10520
>
> вс, 6 янв. 2019 г. в 20:03, Denis Magda :
> >
> > Hello,
> >
> > Ignite versions prior to 2.7 never supported transactions for SQL
> queries. You were enlisting SQL in transactions for your own risk. Ignite
> version 2.7 introduced true transactional support for SQL based on MVCC.
> Presently it's in beta with GA to be available around Q2-Q3 this year. The
> community is working on optimizations.
> >
> > Please refer to this docs for more details:
> > https://apacheignite.readme.io/docs/multiversion-concurrency-control
> > https://apacheignite-sql.readme.io/docs/transactions
> > https://apacheignite-sql.readme.io/docs/multiversion-concurrency-control
> >
> > --
> > Denis
> >
> > On Sat, Jan 5, 2019 at 7:48 PM Prasad Bhalerao <
> prasadbhalerao1...@gmail.com> wrote:
> >>
> >> Can someone please explain if anything has changed in ignite 2.7.
> >>
> >> Started getting this exception after upgrading to 2.7.
> >>
> >>
> >> -- Forwarded message -
> >> From: Prasad Bhalerao 
> >> Date: Fri 4 Jan, 2019, 8:41 PM
> >> Subject: Re: Getting javax.cache.CacheException after upgrading to
> Ignite
> >> 2.7
> >> To: 
> >>
> >>
> >> Can someone please help me with this?
> >>
> >> On Thu 3 Jan, 2019, 7:15 PM Prasad Bhalerao <
> prasadbhalerao1...@gmail.com
> >> wrote:
> >>
> >> > Hi
> >> >
> >> > After upgrading to 2.7 version I am getting following exception. I am
> >> > executing a SELECT sql inside optimistic transaction with
> serialization
> >> > isolation level.
> >> >
> >> > 1) Has anything changed from 2.6 to 2.7 version?  This work fine
> prior to
> >> > 2.7 version.
> >> >
> >> > After changing it to Pessimistic and isolation level to
> REPEATABLE_READ it
> >> > works fine.
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> > *javax.cache.CacheException: Only pessimistic repeatable read
> transactions
> >> > are supported at the moment.at
> >> >
> org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.query(IgniteCacheProxyImpl.java:697)at
> >> >
> org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.query(IgniteCacheProxyImpl.java:636)at
> >> >
> org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.query(GatewayProtectedCacheProxy.java:388)at
> >> >
> com.qualys.agms.grid.dao.AbstractDataGridDAO.getFieldResultsByCriteria(AbstractDataGridDAO.java:85)*
> >> >
> >> > Thanks,
> >> > Prasad
> >> >
>
>
>
> --
> Best regards,
> Ivan Pavlukhin
>

Out of sync clocks on all ignite nodes/VMs

[Prasad Bhalerao]

Can the nodes go out of cluster If the clocks on all nodes are out of sync?

Can it create issue in cluster or cluster formation?

Can it trigger rebalancing process or partition exchange process
unnecessarily on nodes?

Thanks,
Prasad

How to debug network issues in cluster

[Prasad Bhalerao]

Hi,

I am consistently getting "Node is out of topology" message in logs on
node-1 and in other node, node-2 getting message "Timed out waiting for
message delivery receipt (most probably, the reason is in long GC pauses on
remote node; consider tuning GC and increasing '"

I have checked the network bandwidth using iperf and it is 470 Mbit per
sec. I have also checked the gc logs and max pause time is 140 ms.

If it is really happening because of network issues, it there any way to
debug it?

If it is happening because of gc, I would have seen it in gc logs.

Can someone please help me out with this?

Log messages on node-1:
2019-01-06 13:48:19,036 125016 [tcp-disco-srvr-#3%springDataNode%] INFO
o.a.i.s.d.tcp.TcpDiscoverySpi - TCP discovery accepted incoming connection
[rmtAddr=/10.114.113.65, rmtPort=35651]
2019-01-06 13:48:19,037 125017 [tcp-disco-srvr-#3%springDataNode%] INFO
o.a.i.s.d.tcp.TcpDiscoverySpi - TCP discovery spawning a new thread for
connection [rmtAddr=/10.114.113.65, rmtPort=35651]
2019-01-06 13:48:19,037 125017 [tcp-disco-sock-reader-#5%springDataNode%]
INFO  o.a.i.s.d.tcp.TcpDiscoverySpi - Started serving remote node
connection [rmtAddr=/10.114.113.65:35651, rmtPort=35651]
*2019-01-06 13:48:19,040 125020 [tcp-disco-msg-worker-#2%springDataNode%]
WARN  o.a.i.s.d.tcp.TcpDiscoverySpi - Node is out of topology (probably,
due to short-time network problems).*
2019-01-06 13:48:19,041 125021 [disco-event-worker-#62%springDataNode%]
WARN  o.a.i.i.m.d.GridDiscoveryManager - Local node SEGMENTED:
TcpDiscoveryNode [id=a5827f51-096a-4c98-af4f-564d2d3e769d,
addrs=[10.114.113.53, 127.0.0.1], sockAddrs=[/127.0.0.1:47500,
qagmscore02.p13.eng.in03.qualys.com/10.114.113.53:47500], discPort=47500,
order=2, intOrder=2, lastExchangeTime=1546782499034, loc=true,
ver=2.7.0#20181130-sha1:256ae401, isClient=false]
2019-01-06 13:48:19,041 125021 [tcp-disco-sock-reader-#5%springDataNode%]
INFO  o.a.i.s.d.tcp.TcpDiscoverySpi - Finished serving remote node
connection [rmtAddr=/10.114.113.65:35651, rmtPort=35651
2019-01-06 13:48:19,866 125846 [tcp-comm-worker-#1%springDataNode%] INFO
o.a.i.s.d.tcp.TcpDiscoverySpi - Pinging node:
cd9803ac-b810-447e-818e-ab51dada59d8

Re: Getting javax.cache.CacheException after upgrading to Ignite 2.7

[Prasad Bhalerao]

Can someone please help me with this?

On Thu 3 Jan, 2019, 7:15 PM Prasad Bhalerao  Hi
>
> After upgrading to 2.7 version I am getting following exception. I am
> executing a SELECT sql inside optimistic transaction with serialization
> isolation level.
>
> 1) Has anything changed from 2.6 to 2.7 version?  This work fine prior to
> 2.7 version.
>
> After changing it to Pessimistic and isolation level to REPEATABLE_READ it
> works fine.
>
>
>
>
>
>
> *javax.cache.CacheException: Only pessimistic repeatable read transactions
> are supported at the moment.at
> org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.query(IgniteCacheProxyImpl.java:697)at
> org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.query(IgniteCacheProxyImpl.java:636)at
> org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.query(GatewayProtectedCacheProxy.java:388)at
> com.qualys.agms.grid.dao.AbstractDataGridDAO.getFieldResultsByCriteria(AbstractDataGridDAO.java:85)*
>
> Thanks,
> Prasad
>

Getting javax.cache.CacheException after upgrading to Ignite 2.7

[Prasad Bhalerao]

Hi

After upgrading to 2.7 version I am getting following exception. I am
executing a SELECT sql inside optimistic transaction with serialization
isolation level.

1) Has anything changed from 2.6 to 2.7 version?  This work fine prior to
2.7 version.

After changing it to Pessimistic and isolation level to REPEATABLE_READ it
works fine.






*javax.cache.CacheException: Only pessimistic repeatable read transactions
are supported at the moment.at
org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.query(IgniteCacheProxyImpl.java:697)at
org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.query(IgniteCacheProxyImpl.java:636)at
org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.query(GatewayProtectedCacheProxy.java:388)at
com.qualys.agms.grid.dao.AbstractDataGridDAO.getFieldResultsByCriteria(AbstractDataGridDAO.java:85)*

Thanks,
Prasad

Re: Query Execution very slow

[Prasad Bhalerao]

Hi Ilya,

Thank you for the help, I tried your suggestion and it worked fine.  For
Second query h2 was selecting idx1 which was slowing it down.

I have couple of questions.
1) Is there any limit for JOIN temp table clause? In some cases I have
17000 ids and it was causing sql execution to hang but it was before
implementing you suggestion. So I had to split the 17K list into batch of
50 ids.
But now I have to execute this sql 17000/50 times in a loop which is again
slowing down the process. One thing is I can increase the batch size.
Is there any way to handle it in better way?

2) Is it necessary to push the ids first to the temp table and then join it
with necessary table like we did in this case?

3) When I executed the same sql with 900 ids in temp table, it took 30
seconds and fetched 385332 rows. Does ignite suppose to take this much time
when number ids in temp table join case increases?


Regards,
Prasad

On Sat, Dec 29, 2018 at 4:58 PM Ilya Kasnacheev 
wrote:

> Hello!
>
> I have updated my Pull Request with USE INDEX:
> getAffectedIPRange_2 :: SQL_2=SELECT ipv4agd.id, ipv4agd.assetGroupId,
> ipv4agd.ipStart, ipv4agd.ipEnd FROM TABLE (assetGroupId bigint = ? ) temp
> JOIN IpV4AssetGroupData ipv4agd USE INDEX (ipv4_asset_group_data_idx2) ON
> ipv4agd.assetGroupId = temp.assetGroupId WHERE subscriptionId = ? AND
> (ipStart <= ? AND ipEnd >= ?) ORDER BY ipv4agd.assetGroupId
> getAffectedIPRange_2 :: TimeTakenToComplete=9 :: Size=2
>
> getAffectedIPRange_2 :: SQL_2=SELECT ipv4agd.id, ipv4agd.assetGroupId,
> ipv4agd.ipStart, ipv4agd.ipEnd FROM TABLE (assetGroupId bigint = ? ) temp
> JOIN IpV4AssetGroupData ipv4agd USE INDEX (ipv4_asset_group_data_idx2) ON
> ipv4agd.assetGroupId = temp.assetGroupId WHERE subscriptionId = ? AND
> (ipStart <= ? AND ipEnd >= ?) ORDER BY ipv4agd.assetGroupId
> getAffectedIPRange_2 :: TimeTakenToComplete=25 :: Size=1260
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> пт, 28 дек. 2018 г. в 18:59, Prasad Bhalerao  >:
>
>> Hi,
>>
>> After setting enforceJoinOrder to true it worked. Can you please explain
>> how did it work... pushing the data to temp table first and then
>> enforeOrder?
>>
>> Is it documented in ignite docs?
>>
>> getAffectedIPRange_3 :: SQL_3=SELECT ipv4agd.id, ipv4agd.assetGroupId,
>> ipv4agd.ipStart, ipv4agd.ipEnd FROM TABLE (assetGroupId bigint = ? ) temp
>> JOIN IpV4AssetGroupData ipv4agd ON ipv4agd.assetGroupId = temp.assetGroupId
>> WHERE subscriptionId = ? ORDER BY ipv4agd.assetGroupId
>> getAffectedIPRange_3 :: *TimeTakenToComplete=10 ::* Size=1295
>>
>>
>> *But second sql is still taking time. The only difference is it has
>> ipStart and ipEnd filter in where clause.*
>>
>> getAffectedIPRange_2 :: SQL_2=SELECT ipv4agd.id, ipv4agd.assetGroupId,
>> ipv4agd.ipStart, ipv4agd.ipEnd FROM TABLE (assetGroupId bigint = ? ) temp
>> JOIN IpV4AssetGroupData ipv4agd ON ipv4agd.assetGroupId = temp.assetGroupId
>> WHERE subscriptionId = ? AND (ipStart <= ? AND ipEnd >= ?) ORDER BY
>> ipv4agd.assetGroupId
>> getAffectedIPRange_2 :: *TimeTakenToComplete=25436* :: Size=1260
>>
>> Thanks,
>> Prasad
>>
>> On Fri, Dec 28, 2018 at 9:02 PM Ilya Kasnacheev <
>> ilya.kasnach...@gmail.com> wrote:
>>
>>> Hello!
>>>
>>> Did you set enforceJoinOrder to true?
>>>
>>> Regards,
>>> --
>>> Ilya Kasnacheev
>>>
>>>
>>> пт, 28 дек. 2018 г. в 18:19, Prasad Bhalerao <
>>> prasadbhalerao1...@gmail.com>:
>>>
>>>> Hi,
>>>>
>>>> I tried your suggestion but it did not work. It is taking 22.8 seconds.
>>>>
>>>> getAffectedIPRange_3 :: SQL_3=SELECT ipv4agd.id, ipv4agd.assetGroupId,
>>>> ipv4agd.ipStart, ipv4agd.ipEnd FROM TABLE (assetGroupId bigint = ? ) temp
>>>> JOIN IpV4AssetGroupData ipv4agd ON ipv4agd.assetGroupId = temp.assetGroupId
>>>> WHERE subscriptionId = ? ORDER BY ipv4agd.assetGroupId
>>>>
>>>> getAffectedIPRange_3 :: TimeTakenToComplete=22891 :: Size=1295
>>>> Thanks,
>>>> Prasad
>>>>
>>>> On Fri, Dec 28, 2018 at 8:32 PM Ilya Kasnacheev <
>>>> ilya.kasnach...@gmail.com> wrote:
>>>>
>>>>> Hello!
>>>>>
>>>>> I have created a PR for you:
>>>>> https://github.com/prasadbhalerao1983/IgniteTestPrj/pull/1
>>>>> With it, I can see:
>>>>>
>>>>> getAffectedIPRange_3 :: SQL_3=SELECT ipv4agd.id,
>>>>> ipv4agd.assetGroupId, ipv4agd.ipStart, ipv4agd.ipEnd FROM TABLE
>>>>> (assetGr

Re: Query Execution very slow

[Prasad Bhalerao]

Hi,

After setting enforceJoinOrder to true it worked. Can you please explain
how did it work... pushing the data to temp table first and then
enforeOrder?

Is it documented in ignite docs?

getAffectedIPRange_3 :: SQL_3=SELECT ipv4agd.id, ipv4agd.assetGroupId,
ipv4agd.ipStart, ipv4agd.ipEnd FROM TABLE (assetGroupId bigint = ? ) temp
JOIN IpV4AssetGroupData ipv4agd ON ipv4agd.assetGroupId = temp.assetGroupId
WHERE subscriptionId = ? ORDER BY ipv4agd.assetGroupId
getAffectedIPRange_3 :: *TimeTakenToComplete=10 ::* Size=1295


*But second sql is still taking time. The only difference is it has ipStart
and ipEnd filter in where clause.*

getAffectedIPRange_2 :: SQL_2=SELECT ipv4agd.id, ipv4agd.assetGroupId,
ipv4agd.ipStart, ipv4agd.ipEnd FROM TABLE (assetGroupId bigint = ? ) temp
JOIN IpV4AssetGroupData ipv4agd ON ipv4agd.assetGroupId = temp.assetGroupId
WHERE subscriptionId = ? AND (ipStart <= ? AND ipEnd >= ?) ORDER BY
ipv4agd.assetGroupId
getAffectedIPRange_2 :: *TimeTakenToComplete=25436* :: Size=1260

Thanks,
Prasad

On Fri, Dec 28, 2018 at 9:02 PM Ilya Kasnacheev 
wrote:

> Hello!
>
> Did you set enforceJoinOrder to true?
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> пт, 28 дек. 2018 г. в 18:19, Prasad Bhalerao  >:
>
>> Hi,
>>
>> I tried your suggestion but it did not work. It is taking 22.8 seconds.
>>
>> getAffectedIPRange_3 :: SQL_3=SELECT ipv4agd.id, ipv4agd.assetGroupId,
>> ipv4agd.ipStart, ipv4agd.ipEnd FROM TABLE (assetGroupId bigint = ? ) temp
>> JOIN IpV4AssetGroupData ipv4agd ON ipv4agd.assetGroupId = temp.assetGroupId
>> WHERE subscriptionId = ? ORDER BY ipv4agd.assetGroupId
>>
>> getAffectedIPRange_3 :: TimeTakenToComplete=22891 :: Size=1295
>> Thanks,
>> Prasad
>>
>> On Fri, Dec 28, 2018 at 8:32 PM Ilya Kasnacheev <
>> ilya.kasnach...@gmail.com> wrote:
>>
>>> Hello!
>>>
>>> I have created a PR for you:
>>> https://github.com/prasadbhalerao1983/IgniteTestPrj/pull/1
>>> With it, I can see:
>>>
>>> getAffectedIPRange_3 :: SQL_3=SELECT ipv4agd.id, ipv4agd.assetGroupId,
>>> ipv4agd.ipStart, ipv4agd.ipEnd FROM TABLE (assetGroupId bigint = ? ) temp
>>> JOIN IpV4AssetGroupData ipv4agd ON ipv4agd.assetGroupId = temp.assetGroupId
>>> WHERE subscriptionId = ? ORDER BY ipv4agd.assetGroupId
>>> getAffectedIPRange_3 :: TimeTakenToComplete=11 :: Size=1295
>>>
>>> Regards,
>>> --
>>> Ilya Kasnacheev
>>>
>>>
>>> пт, 28 дек. 2018 г. в 17:46, Prasad Bhalerao <
>>> prasadbhalerao1...@gmail.com>:
>>>
>>>> Can someone from community help me with t
>>>>
>>>> I have created a reproducer and uploaded it to GitHub. I have created
>>>> 3 cases to test the sql execution time.
>>>>
>>>> Please run *IgniteQueryTester_4* class to check the issue.
>>>> GitHub project: https://github.com/prasadbhalerao1983/IgniteTestPrj.git
>>>>
>>>> Thanks,
>>>> Prasad
>>>>
>>>>
>>>> On Wed, Dec 26, 2018 at 11:18 PM Prasad Bhalerao <
>>>> prasadbhalerao1...@gmail.com> wrote:
>>>>
>>>>> How to push the ids to temp table, can you please give any example?
>>>>> Is it a in memory temp table created by ignite?
>>>>>
>>>>> Can you please explain how enforceJoinOrder will help in this case?
>>>>> Thanks ,
>>>>> Prasad
>>>>>
>>>>> On Wed 26 Dec, 2018, 9:37 PM Ilya Kasnacheev <
>>>>> ilya.kasnach...@gmail.com wrote:
>>>>>
>>>>>> Hello!
>>>>>>
>>>>>> Can you try pushing temp table to 1st position and setting
>>>>>> enforceJoinOrder=true?
>>>>>>
>>>>>> SELECT ipv4agd.id,
>>>>>>   ipv4agd.assetGroupId,
>>>>>>   ipv4agd.ipStart,
>>>>>>   ipv4agd.ipEnd
>>>>>> FROM TABLE (assetGroupId bigint = ? ) temp
>>>>>> JOIN IpV4AssetGroupData ipv4agd
>>>>>> ON ipv4agd.assetGroupId = temp.assetGroupId
>>>>>> WHERE subscriptionId= ?
>>>>>> AND (ipStart <= ? AND ipEnd >= ?)
>>>>>> ORDER BY ipv4agd.assetGroupId
>>>>>>
>>>>>> See https://apacheignite.readme.io/docs/configuration-parameters for
>>>>>> enforceJoinOrder.
>>>>>>
>>>>>> Regards,
>>>>>> --
>>>>>> Ilya Kasnacheev
>>>>>&g

Re: Query Execution very slow

[Prasad Bhalerao]

Hi,

I tried your suggestion but it did not work. It is taking 22.8 seconds.

getAffectedIPRange_3 :: SQL_3=SELECT ipv4agd.id, ipv4agd.assetGroupId,
ipv4agd.ipStart, ipv4agd.ipEnd FROM TABLE (assetGroupId bigint = ? ) temp
JOIN IpV4AssetGroupData ipv4agd ON ipv4agd.assetGroupId = temp.assetGroupId
WHERE subscriptionId = ? ORDER BY ipv4agd.assetGroupId

getAffectedIPRange_3 :: TimeTakenToComplete=22891 :: Size=1295
Thanks,
Prasad

On Fri, Dec 28, 2018 at 8:32 PM Ilya Kasnacheev 
wrote:

> Hello!
>
> I have created a PR for you:
> https://github.com/prasadbhalerao1983/IgniteTestPrj/pull/1
> With it, I can see:
>
> getAffectedIPRange_3 :: SQL_3=SELECT ipv4agd.id, ipv4agd.assetGroupId,
> ipv4agd.ipStart, ipv4agd.ipEnd FROM TABLE (assetGroupId bigint = ? ) temp
> JOIN IpV4AssetGroupData ipv4agd ON ipv4agd.assetGroupId = temp.assetGroupId
> WHERE subscriptionId = ? ORDER BY ipv4agd.assetGroupId
> getAffectedIPRange_3 :: TimeTakenToComplete=11 :: Size=1295
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> пт, 28 дек. 2018 г. в 17:46, Prasad Bhalerao  >:
>
>> Can someone from community help me with t
>>
>> I have created a reproducer and uploaded it to GitHub. I have created 3
>> cases to test the sql execution time.
>>
>> Please run *IgniteQueryTester_4* class to check the issue.
>> GitHub project: https://github.com/prasadbhalerao1983/IgniteTestPrj.git
>>
>> Thanks,
>> Prasad
>>
>>
>> On Wed, Dec 26, 2018 at 11:18 PM Prasad Bhalerao <
>> prasadbhalerao1...@gmail.com> wrote:
>>
>>> How to push the ids to temp table, can you please give any example?
>>> Is it a in memory temp table created by ignite?
>>>
>>> Can you please explain how enforceJoinOrder will help in this case?
>>> Thanks ,
>>> Prasad
>>>
>>> On Wed 26 Dec, 2018, 9:37 PM Ilya Kasnacheev >> wrote:
>>>
>>>> Hello!
>>>>
>>>> Can you try pushing temp table to 1st position and setting
>>>> enforceJoinOrder=true?
>>>>
>>>> SELECT ipv4agd.id,
>>>>   ipv4agd.assetGroupId,
>>>>   ipv4agd.ipStart,
>>>>   ipv4agd.ipEnd
>>>> FROM TABLE (assetGroupId bigint = ? ) temp
>>>> JOIN IpV4AssetGroupData ipv4agd
>>>> ON ipv4agd.assetGroupId = temp.assetGroupId
>>>> WHERE subscriptionId= ?
>>>> AND (ipStart <= ? AND ipEnd >= ?)
>>>> ORDER BY ipv4agd.assetGroupId
>>>>
>>>> See https://apacheignite.readme.io/docs/configuration-parameters for
>>>> enforceJoinOrder.
>>>>
>>>> Regards,
>>>> --
>>>> Ilya Kasnacheev
>>>>
>>>>
>>>> ср, 26 дек. 2018 г. в 19:01, Prasad Bhalerao <
>>>> prasadbhalerao1...@gmail.com>:
>>>>
>>>>> I am executing following SQL on ignite cache. This cache has 37
>>>>> million records and this data is distributed across 4 nodes.
>>>>> *SQL:*
>>>>> SELECT ipv4agd.id,
>>>>>   ipv4agd.assetGroupId,
>>>>>   ipv4agd.ipStart,
>>>>>   ipv4agd.ipEnd
>>>>> FROM IpV4AssetGroupData ipv4agd
>>>>> JOIN TABLE (assetGroupId bigint = ? ) temp
>>>>> ON ipv4agd.assetGroupId = temp.assetGroupId
>>>>> WHERE subscriptionId= ?
>>>>> AND (ipStart <= ? AND ipEnd >= ?)
>>>>> ORDER BY ipv4agd.assetGroupId
>>>>>
>>>>>
>>>>> As per the execution plan show below, ignite is using index
>>>>> "IPV4_ASSET_GROUP_DATA_IDX2 " and execution plan attached below.
>>>>>
>>>>> This sql is taking around 23 seconds. I have set the max index inline
>>>>> size as 65 bytes.
>>>>> The number of assetGroupIds set in join clause are 50.
>>>>>
>>>>> *Is there anything I can do to improve the performance of this SQL?*
>>>>>
>>>>> I have checked the memory and cpu utilization and it it is very low. I
>>>>> also tried to profile it using jprofiler to find out the issue, but could
>>>>> not find solution. I have also attached profiler snapshot at the end.
>>>>> Please check.
>>>>>
>>>>>
>>>>> .
>>>>>
>>>>> *Indexes:*
>>>>>
>>>>> public class IpV4AssetGroupData implements 
>>>>> UpdatableData {
>>>>>
>>>>>

Re: Query Execution very slow

[Prasad Bhalerao]

Can someone from community help me with t

I have created a reproducer and uploaded it to GitHub. I have created 3
cases to test the sql execution time.

Please run *IgniteQueryTester_4* class to check the issue.
GitHub project: https://github.com/prasadbhalerao1983/IgniteTestPrj.git

Thanks,
Prasad


On Wed, Dec 26, 2018 at 11:18 PM Prasad Bhalerao <
prasadbhalerao1...@gmail.com> wrote:

> How to push the ids to temp table, can you please give any example?
> Is it a in memory temp table created by ignite?
>
> Can you please explain how enforceJoinOrder will help in this case?
> Thanks ,
> Prasad
>
> On Wed 26 Dec, 2018, 9:37 PM Ilya Kasnacheev  wrote:
>
>> Hello!
>>
>> Can you try pushing temp table to 1st position and setting
>> enforceJoinOrder=true?
>>
>> SELECT ipv4agd.id,
>>   ipv4agd.assetGroupId,
>>   ipv4agd.ipStart,
>>   ipv4agd.ipEnd
>> FROM TABLE (assetGroupId bigint = ? ) temp
>> JOIN IpV4AssetGroupData ipv4agd
>> ON ipv4agd.assetGroupId = temp.assetGroupId
>> WHERE subscriptionId= ?
>> AND (ipStart <= ? AND ipEnd >= ?)
>> ORDER BY ipv4agd.assetGroupId
>>
>> See https://apacheignite.readme.io/docs/configuration-parameters for
>> enforceJoinOrder.
>>
>> Regards,
>> --
>> Ilya Kasnacheev
>>
>>
>> ср, 26 дек. 2018 г. в 19:01, Prasad Bhalerao <
>> prasadbhalerao1...@gmail.com>:
>>
>>> I am executing following SQL on ignite cache. This cache has 37 million
>>> records and this data is distributed across 4 nodes.
>>> *SQL:*
>>> SELECT ipv4agd.id,
>>>   ipv4agd.assetGroupId,
>>>   ipv4agd.ipStart,
>>>   ipv4agd.ipEnd
>>> FROM IpV4AssetGroupData ipv4agd
>>> JOIN TABLE (assetGroupId bigint = ? ) temp
>>> ON ipv4agd.assetGroupId = temp.assetGroupId
>>> WHERE subscriptionId= ?
>>> AND (ipStart <= ? AND ipEnd >= ?)
>>> ORDER BY ipv4agd.assetGroupId
>>>
>>>
>>> As per the execution plan show below, ignite is using index
>>> "IPV4_ASSET_GROUP_DATA_IDX2 " and execution plan attached below.
>>>
>>> This sql is taking around 23 seconds. I have set the max index inline
>>> size as 65 bytes.
>>> The number of assetGroupIds set in join clause are 50.
>>>
>>> *Is there anything I can do to improve the performance of this SQL?*
>>>
>>> I have checked the memory and cpu utilization and it it is very low. I
>>> also tried to profile it using jprofiler to find out the issue, but could
>>> not find solution. I have also attached profiler snapshot at the end.
>>> Please check.
>>>
>>>
>>> .
>>>
>>> *Indexes:*
>>>
>>> public class IpV4AssetGroupData implements 
>>> UpdatableData {
>>>
>>>   @QuerySqlField
>>>   private long id;
>>>   @QuerySqlField(orderedGroups = {@QuerySqlField.Group(name = 
>>> "ipv4_asset_group_data_idx2", order = 2)})
>>>   private long assetGroupId;
>>>   @QuerySqlField(orderedGroups = {@QuerySqlField.Group(name = 
>>> "ipv4_asset_group_data_idx1", order = 1),
>>>   @QuerySqlField.Group(name = "ipv4_asset_group_data_idx2", order = 1)})
>>>   private long subscriptionId;
>>>   @QuerySqlField(orderedGroups = {@QuerySqlField.Group(name = 
>>> "ipv4_asset_group_data_idx1", order = 2),
>>>   @QuerySqlField.Group(name = "ipv4_asset_group_data_idx2", order = 3)})
>>>   private int ipStart;
>>>   @QuerySqlField(orderedGroups = {@QuerySqlField.Group(name = 
>>> "ipv4_asset_group_data_idx1", order = 3),
>>>   @QuerySqlField.Group(name = "ipv4_asset_group_data_idx2", order = 4)})
>>>   private int ipEnd;
>>>
>>> }
>>>
>>> *Execution plan:*
>>>
>>> Query execution is too long [time=15788 ms, sql='SELECT
>>> IPV4AGD__Z0.ID __C0_0,
>>> IPV4AGD__Z0.ASSETGROUPID __C0_1,
>>> IPV4AGD__Z0.IPSTART __C0_2,
>>> IPV4AGD__Z0.IPEND __C0_3
>>> FROM IPV4_ASSET_GROUP_DETAIL_CACHE.IPV4ASSETGROUPDATA IPV4AGD__Z0
>>>  INNER JOIN TABLE(ASSETGROUPID BIGINT=?1) TEMP__Z1
>>>  ON TRUE
>>> WHERE (IPV4AGD__Z0.ASSETGROUPID = TEMP__Z1.ASSETGROUPID) AND
>>> ((IPV4AGD__Z0.SUBSCRIPTIONID = ?2) AND ((IPV4AGD__Z0.IPSTART <= ?3) AND
>>> (IPV4AGD__Z0.IPEND >= ?4)))
>>> ORDER BY 2',
>>>
>>> plan=
>>>
>>> SELECT
>>> IP

Re: Query Execution very slow

[Prasad Bhalerao]

How to push the ids to temp table, can you please give any example?
Is it a in memory temp table created by ignite?

Can you please explain how enforceJoinOrder will help in this case?
Thanks ,
Prasad

On Wed 26 Dec, 2018, 9:37 PM Ilya Kasnacheev  Hello!
>
> Can you try pushing temp table to 1st position and setting
> enforceJoinOrder=true?
>
> SELECT ipv4agd.id,
>   ipv4agd.assetGroupId,
>   ipv4agd.ipStart,
>   ipv4agd.ipEnd
> FROM TABLE (assetGroupId bigint = ? ) temp
> JOIN IpV4AssetGroupData ipv4agd
> ON ipv4agd.assetGroupId = temp.assetGroupId
> WHERE subscriptionId= ?
> AND (ipStart <= ? AND ipEnd >= ?)
> ORDER BY ipv4agd.assetGroupId
>
> See https://apacheignite.readme.io/docs/configuration-parameters for
> enforceJoinOrder.
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> ср, 26 дек. 2018 г. в 19:01, Prasad Bhalerao  >:
>
>> I am executing following SQL on ignite cache. This cache has 37 million
>> records and this data is distributed across 4 nodes.
>> *SQL:*
>> SELECT ipv4agd.id,
>>   ipv4agd.assetGroupId,
>>   ipv4agd.ipStart,
>>   ipv4agd.ipEnd
>> FROM IpV4AssetGroupData ipv4agd
>> JOIN TABLE (assetGroupId bigint = ? ) temp
>> ON ipv4agd.assetGroupId = temp.assetGroupId
>> WHERE subscriptionId= ?
>> AND (ipStart <= ? AND ipEnd >= ?)
>> ORDER BY ipv4agd.assetGroupId
>>
>>
>> As per the execution plan show below, ignite is using index
>> "IPV4_ASSET_GROUP_DATA_IDX2 " and execution plan attached below.
>>
>> This sql is taking around 23 seconds. I have set the max index inline
>> size as 65 bytes.
>> The number of assetGroupIds set in join clause are 50.
>>
>> *Is there anything I can do to improve the performance of this SQL?*
>>
>> I have checked the memory and cpu utilization and it it is very low. I
>> also tried to profile it using jprofiler to find out the issue, but could
>> not find solution. I have also attached profiler snapshot at the end.
>> Please check.
>>
>>
>> .
>>
>> *Indexes:*
>>
>> public class IpV4AssetGroupData implements 
>> UpdatableData {
>>
>>   @QuerySqlField
>>   private long id;
>>   @QuerySqlField(orderedGroups = {@QuerySqlField.Group(name = 
>> "ipv4_asset_group_data_idx2", order = 2)})
>>   private long assetGroupId;
>>   @QuerySqlField(orderedGroups = {@QuerySqlField.Group(name = 
>> "ipv4_asset_group_data_idx1", order = 1),
>>   @QuerySqlField.Group(name = "ipv4_asset_group_data_idx2", order = 1)})
>>   private long subscriptionId;
>>   @QuerySqlField(orderedGroups = {@QuerySqlField.Group(name = 
>> "ipv4_asset_group_data_idx1", order = 2),
>>   @QuerySqlField.Group(name = "ipv4_asset_group_data_idx2", order = 3)})
>>   private int ipStart;
>>   @QuerySqlField(orderedGroups = {@QuerySqlField.Group(name = 
>> "ipv4_asset_group_data_idx1", order = 3),
>>   @QuerySqlField.Group(name = "ipv4_asset_group_data_idx2", order = 4)})
>>   private int ipEnd;
>>
>> }
>>
>> *Execution plan:*
>>
>> Query execution is too long [time=15788 ms, sql='SELECT
>> IPV4AGD__Z0.ID __C0_0,
>> IPV4AGD__Z0.ASSETGROUPID __C0_1,
>> IPV4AGD__Z0.IPSTART __C0_2,
>> IPV4AGD__Z0.IPEND __C0_3
>> FROM IPV4_ASSET_GROUP_DETAIL_CACHE.IPV4ASSETGROUPDATA IPV4AGD__Z0
>>  INNER JOIN TABLE(ASSETGROUPID BIGINT=?1) TEMP__Z1
>>  ON TRUE
>> WHERE (IPV4AGD__Z0.ASSETGROUPID = TEMP__Z1.ASSETGROUPID) AND
>> ((IPV4AGD__Z0.SUBSCRIPTIONID = ?2) AND ((IPV4AGD__Z0.IPSTART <= ?3) AND
>> (IPV4AGD__Z0.IPEND >= ?4)))
>> ORDER BY 2',
>>
>> plan=
>>
>> SELECT
>> IPV4AGD__Z0.ID AS __C0_0,
>> IPV4AGD__Z0.ASSETGROUPID AS __C0_1,
>> IPV4AGD__Z0.IPSTART AS __C0_2,
>> IPV4AGD__Z0.IPEND AS __C0_3
>> FROM IPV4_ASSET_GROUP_DETAIL_CACHE.IPV4ASSETGROUPDATA IPV4AGD__Z0
>> /* IPV4_ASSET_GROUP_DETAIL_CACHE.IPV4_ASSET_GROUP_DATA_IDX2:
>> SUBSCRIPTIONID = ?2
>> AND IPSTART <= ?3
>> AND IPEND >= ?4
>>  */
>> /* WHERE (IPV4AGD__Z0.IPEND >= ?4)
>> AND ((IPV4AGD__Z0.SUBSCRIPTIONID = ?2)
>> AND (IPV4AGD__Z0.IPSTART <= ?3))
>> */
>> INNER JOIN TABLE(ASSETGROUPID BIGINT=?1) TEMP__Z1
>> /* function: ASSETGROUPID = IPV4AGD__Z0.ASSETGROUPID */
>> ON 1=1
>> WHERE (IPV4AGD__Z0.ASSETGROUPID = TEMP__Z1.ASSETGROUPID)
>> AND ((IPV4AGD__Z0.SUBSCRIPTIONID = ?2)
>> AND ((IPV4AGD__Z0.IPSTART <= ?3)
>> AND (IPV4AGD__Z0.IPEND >= ?4)))
>> ORDER BY 2
>> , parameters=[[3483555, 3180458, 3250090, 3483563, 3182509, 3213230,
>> 3245998, 3487661, 3215281, 3444657, 3182515, 3372974, 3483573, 3372981,
>> 3200951, 3485624, 3295161, 3485626, 3379125, 3211196, 3213242, 3381181,
>> 3194805, 3213247, 3258299, 3379123, 3377070, 3315637, 3352502, 3295174,
>> 3485618, 3438530, 3483592, 3352516, 3155914, 3424204, 3192775, 3485643,
>> 3317711, 3246026, 3209159, 3485584, 3485645, 3483594, 3248085, 3321799,
>> 3248086, 3190744, 3211222, 3379162], 164307, 1084754675, -2094919442]]
>>
>>
>> Profiler snapshot:
>>
>> [image: image.png]
>>
>

Index inline size

[Prasad Bhalerao]

Hi,

I have set sqlIndexMaxInline size in cache configuration level as follows.

cacheCfg.setSqlIndexMaxInlineSize(100);

But still  I am getting following warning message in log.

WARN  o.a.i.i.p.q.h2.database.H2TreeIndex -  Indexed
columns of a row cannot be fully inlined into index what may lead to
slowdown due to additional data page reads, increase index inline size if
needed (use INLINE_SIZE option for CREATE INDEX command,
QuerySqlField.inlineSize for annotated classes, or QueryIndex.inlineSize
for explicit QueryEntity configuration) [cacheName=USERACCOUNTDATA,
tableName=USER_ACCOUNT_CACHE, idxName=USER_ACCOUNT_IDX4,
idxCols=(SUBSCRIPTIONID, UNITID, _KEY, AFFINITYID), idxType=SECONDARY,
curSize=10, recommendedInlineSize=83]

1) Is it necessary to set inline size using @QuerySqlField annotation?

2) How do I set inline size in case of group index in following case?
Do I need to set inline property inside each @QuerySqlField annotation?

public class Person implements Serializable {
  /** Indexed in a group index with "salary". */
  @QuerySqlField(orderedGroups={@QuerySqlField.Group(
name = "age_salary_idx", order = 0, descending = true)})
  private int age;

  /** Indexed separately and in a group index with "age". */
  @QuerySqlField(index = true, orderedGroups={@QuerySqlField.Group(
name = "age_salary_idx", order = 3)})
  private double salary;
}







Thanks,
Prasad

PublicThreadPoolSize vs FifoQueueCollisionSpi.ParallelJobsNumber

[Prasad Bhalerao]

Hi,

What will happen in following case:

publilcThreadPoolSize is to 16.

But FifoQueueCollisionSpi is used instead of NoopCollisionSpi.
FifoQueueCollisionSpi.setParallelJobsNumber(32);

1) Will ignite execute 32 jobs in parallel even though the publicThreadPool
size is set to 16?

2) Is there any configuration to set fifo queue size as well so that number
of jobs that be submitted to this queue can be limited?

Thanks,
Prasad

Re: Error in write-through

[Prasad Bhalerao]

How does striped pool work exactly? I read the doc
 but
still have some confusion.

Does ignite break putAll cache operation into small chunks/tasks and then
submit it to threads in striped pool to do it concurrently?
Is this the only purpose of striped pool?

Does ignite use IgniteStripedThreadPoolExecutor for this purpose?

Thanks,
Prasad

On Tue, Nov 13, 2018 at 6:49 PM Ilya Kasnacheev 
wrote:

> Hello!
>
> As you can see from this thread dump, Oracle driver is waiting on a
> socket, probably for a query response.
> You should probably take a look at hanging queries from Oracle side.
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> вт, 13 нояб. 2018 г. в 9:21, Akash Shinde :
>
>> Hi,
>> I have started four ignite nodes and configured cache in distributed
>> mode. When I initiated thousands of requests to write the data on this
>> cache(write through enabled) , facing below error.
>> From logs we can see this error is occurring  while witting to oracle
>> database.(using cache write-through).
>> This error is not consistent.Node does stop for while after this error
>> and continues to pick up the next ignite tasks.
>> Please someone advise what does following log means.
>>
>>
>> 2018-11-13 05:52:05,577 2377545 [core-1] INFO
>> c.q.a.a.s.AssetManagementService - Add asset request processing started,
>> requestId ADD_Ip_483, subscriptionId =262604, userId=547159
>> 2018-11-13 05:52:06,647 2378615 [grid-timeout-worker-#39%springDataNode%]
>> WARN  o.a.ignite.internal.util.typedef.G - >>> Possible starvation in
>> striped pool.
>> Thread name: sys-stripe-11-#12%springDataNode%
>> Queue: [Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE,
>> topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false,
>> msg=GridNearSingleGetResponse [futId=1542085977929, res=BinaryObjectImpl
>> [arr= true, ctx=false, start=0], topVer=null, err=null, flags=0]]], Message
>> closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8,
>> ordered=false, timeout=0, skipOnTimeout=false, msg=GridDhtTxPrepareResponse
>> [nearEvicted=null, futId=b67ac7b0761-93ebea72-bf4e-40d8-8a19-d3258be94ce9,
>> miniId=1, super=GridDistributedTxPrepareResponse [txState=null, part=-1,
>> err=null, super=GridDistributedBaseMessage [ver=GridCacheVersion
>> [topVer=153565953, order=1542089536997, nodeOrder=3], committedVers=null,
>> rolledbackVers=null, cnt=0, super=GridCacheIdMessage [cacheId=0]],
>> Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8,
>> ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest
>> [futId=1542085976178, key=BinaryObjectImpl [arr= true, ctx=false, start=0],
>> flags=1, topVer=AffinityTopologyVersion [topVer=7, minorTopVer=0],
>> subjId=9e8db7e7-48ba-4161-881b-ad4fcfc175a0, taskNameHash=0, createTtl=-1,
>> accessTtl=-1
>> Deadlock: false
>> Completed: 703
>> Thread [name="sys-stripe-11-#12%springDataNode%", id=41, state=RUNNABLE,
>> blockCnt=37, waitCnt=729]
>> at java.net.SocketInputStream.socketRead0(Native Method)
>> at
>> java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
>> at java.net.SocketInputStream.read(SocketInputStream.java:171)
>> at java.net.SocketInputStream.read(SocketInputStream.java:141)
>> at oracle.net.ns.Packet.receive(Packet.java:311)
>> at oracle.net.ns.DataPacket.receive(DataPacket.java:105)
>> at
>> oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:305)
>> at oracle.net.ns.NetInputStream.read(NetInputStream.java:249)
>> at oracle.net.ns.NetInputStream.read(NetInputStream.java:171)
>> at oracle.net.ns.NetInputStream.read(NetInputStream.java:89)
>> at
>> oracle.jdbc.driver.T4CSocketInputStreamWrapper.readNextPacket(T4CSocketInputStreamWrapper.java:123)
>> at
>> oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:79)
>> at
>> oracle.jdbc.driver.T4CMAREngineStream.unmarshalUB1(T4CMAREngineStream.java:429)
>> at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:397)
>> at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:257)
>> at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:587)
>> at
>> oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:225)
>> at
>> oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:53)
>> at
>> oracle.jdbc.driver.T4CPreparedStatement.executeForRows(T4CPreparedStatement.java:943)
>> at
>> oracle.jdbc.driver.OraclePreparedStatement.executeForRowsWithTimeout(OraclePreparedStatement.java:12029)
>> at
>> oracle.jdbc.driver.OraclePreparedStatement.executeBatch(OraclePreparedStatement.java:12140)
>> at
>> oracle.jdbc.driver.OracleStatementWrapper.executeBatch(OracleStatementWrapper.java:246)
>> at
>> 

Re: Query execution too long even after providing index

[Prasad Bhalerao]

Hi Evgenii,

Thank you for suggesting the query optimization. It worked perfectly fine.
I unnecessarily complicated the sql.
I really appreciate the efforts you guys are taking to help out the users.

About the test data: Yes in production I will be having more than 100K
records for single subscription and moduleId.
The test data generated has 5 million entries against subscriptionId and
moduleId. This is a worst case scenario but we do have such cases in
production.

Thanks,
Prasad

On Mon, Oct 22, 2018 at 6:22 PM Evgenii Zhuravlev 
wrote:

> Well, looks like you trying to find the segments that intersects defined
> segment, but you're complicating it. You don't need 3 conditions here - it
> will be enough to have only one - ipStart <= MAX and ipEnd >= MIN. I've
> checked it for your case and got absolutely the same results as you have
> with a too complex query.
>
> Aditionally, you have the same subscriptionId AND moduleId  in your test
> data, which means that you will have a bad selectivity when you will filter
> by these fields at first. Do you really have such data in your production?
>
> Also, when you will measure something again - do operation a lot of times
> - that how benchmarks work. Ignite initialize internal structures at first
> executions, so, it will not give an idea of overall latency if you will
> measure it just once.
>
> Best Regards,
> Evgenii
>
> пн, 22 окт. 2018 г. в 6:56, Prasad Bhalerao  >:
>
>> Hi Evgenii,
>>
>> Did you get time to check the reproducer?
>>
>> Can you please suggest solution for this?
>>
>>
>> Thanks,
>> Prasad
>>
>>
>> On Fri, Oct 19, 2018, 4:46 PM Prasad Bhalerao <
>> prasadbhalerao1...@gmail.com> wrote:
>>
>>> Hi Evgenii,
>>>
>>> I have created a reproducer and uploaded it to GitHub. I have created 5
>>> cases to test the sql execution time.
>>>
>>> GitHub project: https://github.com/prasadbhalerao1983/IgniteTestPrj.git
>>>
>>> Please run IgniteQueryTester class.
>>>
>>> Thanks,
>>> Prasad
>>>
>>> On Wed, Oct 17, 2018 at 7:46 PM ezhuravlev 
>>> wrote:
>>>
>>>> How much data do you have?  What is the amount of heap and offheap
>>>> memory?
>>>> Can you share the reproducer with the community?
>>>>
>>>> Evgenii
>>>>
>>>>
>>>>
>>>> --
>>>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>>>
>>>

Off heap limit

[Prasad Bhalerao]

Hi,

Is there any limit for off heap memory?

Thanks,
Prasad

Re: Query execution too long even after providing index

[Prasad Bhalerao]

Hi Evgenii,

Did you get time to check the reproducer?

Can you please suggest solution for this?


Thanks,
Prasad


On Fri, Oct 19, 2018, 4:46 PM Prasad Bhalerao 
wrote:

> Hi Evgenii,
>
> I have created a reproducer and uploaded it to GitHub. I have created 5
> cases to test the sql execution time.
>
> GitHub project: https://github.com/prasadbhalerao1983/IgniteTestPrj.git
>
> Please run IgniteQueryTester class.
>
> Thanks,
> Prasad
>
> On Wed, Oct 17, 2018 at 7:46 PM ezhuravlev 
> wrote:
>
>> How much data do you have?  What is the amount of heap and offheap memory?
>> Can you share the reproducer with the community?
>>
>> Evgenii
>>
>>
>>
>> --
>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>
>

GridMergeTable

[Prasad Bhalerao]

Hi,

I was debugging ignite sql execution code for following query.

SELECT id, moduleId,ipEnd, ipStart
FROM IpContainerIpV4Data USE INDEX(ip_container_ipv4_idx1)
WHERE subscriptionId = ?  AND moduleId = ? AND (ipStart<= ?
AND ipEnd   >= ?)
UNION ALL
SELECT id, moduleId,ipEnd, ipStart
FROM IpContainerIpV4Data USE INDEX(ip_container_ipv4_idx1)
WHERE subscriptionId = ? AND moduleId = ? AND (ipStart<= ?
AND ipEnd   >= ?)
UNION ALL
SELECT id, moduleId,ipEnd, ipStart
FROM IpContainerIpV4Data USE INDEX(ip_container_ipv4_idx1)
WHERE subscriptionId = ? AND moduleId = ? AND (ipStart>= ?
AND ipEnd   <= ?)

I found that ignite splits this union query into 3 different map queries
and then creates GridMergeTable. [code::
GridReduceQueryExecutor.createMergeTable]. But could not spend more time on
debugging.

Is this a in-memory temp table or real table created in H2 DB in file
system.

Can some please explain this?

If it is In memory table then how does ignite handle and merge/sort/group
large results sets which does not fit into application heap ?

Thanks,
Prasad

Sql execution with partitionId

[Prasad Bhalerao]

Hi,

I am thinking to get partitionId using affinity function and then set it
in SqlFieldsQuery  object.

By doing this will I be able to execute the query only a node which is
primary for this partition?

As per the ignite doc it should work. But I just want to confirm there are
no caveats in it.

Can this create any issue if the primary node goes down during query
execution?

public class SqlFieldsQuery extends Query> {
/**
 * Sets partitions for a query.
 * The query will be executed only on nodes which are primary for
specified partitions.
 * 
 * Note what passed array'll be sorted in place for performance
reasons, if it wasn't sorted yet.
 *
 * @param parts Partitions.
 * @return {@code this} for chaining.
 */
public SqlFieldsQuery setPartitions(@Nullable int... parts) {
this.parts = prepare(parts);
return this;
}

Thanks,
Prasad

Re: Query execution too long even after providing index

[Prasad Bhalerao]

Hi Evgenii,

I have created a reproducer and uploaded it to GitHub. I have created 5
cases to test the sql execution time.

GitHub project: https://github.com/prasadbhalerao1983/IgniteTestPrj.git

Please run IgniteQueryTester class.

Thanks,
Prasad

On Wed, Oct 17, 2018 at 7:46 PM ezhuravlev  wrote:

> How much data do you have?  What is the amount of heap and offheap memory?
> Can you share the reproducer with the community?
>
> Evgenii
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

On-heap cache

[Prasad Bhalerao]

What is the advantage of using on-heap cache?

I compared the sql execution time of on-heap cache and off-heap cache and
found that there is not much difference in execution time.

Thanks,
Prasad

Re: Query execution too long even after providing index

[Prasad Bhalerao]

Hi, Evgenii,

I tried to execute sql without UNION ALL operator. I mean I just executed
the first part of the sql as shown below and it is taking 700-800 ms to
complete. I have around 3 million records in my cache. and will be having
around 30-40 million records in real scenario.

SELECT id, moduleId,ipEnd, ipStart
FROM IpContainerIpV4Data USE INDEX(ip_container_ipv4_idx1)
WHERE subscriptionId = ?  AND moduleId = ? AND (ipStart<= ?
AND ipEnd   >= ?)

Does ignite executes 3 sqls independently and then does the union
operation? Is this the reason it is taking time?

Is there any solution to solve this problem? The sql shown below is the
very basic sql which is needed everywhere in my application. Since ignite
does not use index with OR clause I had to rewrite the same sql with UNION
ALL operator.
As mentioned in this email thread, this sql is using indexes. Now the
question is does ignite brings all filtered data in heap space and then
does union all operation?


SELECT id, moduleId,ipEnd, ipStart
 FROM IpContainerIpV4Data USE INDEX(ip_container_ipv4_idx1)
 WHERE subscriptionId = ?  AND moduleId = ? AND (ipStart<=
? AND ipEnd   >= ?)
UNION ALL
SELECT id, moduleId,ipEnd, ipStart
 FROM IpContainerIpV4Data USE INDEX(ip_container_ipv4_idx1)
 WHERE subscriptionId = ? AND moduleId = ? AND (ipStart<= ?
AND ipEnd   >= ?)
UNION ALL
SELECT id, moduleId,ipEnd, ipStart
 FROM IpContainerIpV4Data USE INDEX(ip_container_ipv4_idx1)
 WHERE subscriptionId = ? AND moduleId = ? AND (ipStart>= ?
AND ipEnd   <= ?)



On Thu, Sep 13, 2018 at 9:44 PM ezhuravlev  wrote:

> Hi,
>
> What is the execution time of just a single query without UNION?
>
> Evgenii
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Heap size

[Prasad Bhalerao]

Hi,

Is anyone using on heap cache with more than 30 gb jvm heap size in
production?

Which gc algorithm are you using?

If yes have you faced any issues relates to long gc pauses?


Thanks,
Prasad