Re: Apache Ignite Clientside NearCache and Serverside Eviction blocking cluster

[Вячеслав Коптилин]

Hello Prasad,

Sorry for the late reply.

I checked your reproducer and it looks good to me.
And yes, this behavior looks like a bug. I filed a Jira ticket in order to
address this issue: https://issues.apache.org/jira/browse/IGNITE-13858
You can enable ignite native persistence in order to overcome this issue.

Thanks,
Slava.

пт, 4 дек. 2020 г. в 08:58, pvprsd :

> Hi Slava,
>
> Did you get a chance to look into the issue? Please let us know if there
> are
> configuration problems. We need to have both the features in our current
> implementation for better performance.
>
> Thanks,
> Prasad
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Apache Ignite Clientside NearCache and Serverside Eviction blocking cluster

[pvprsd]

Hi Slava,

Did you get a chance to look into the issue? Please let us know if there are
configuration problems. We need to have both the features in our current
implementation for better performance.

Thanks,
Prasad




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Apache Ignite Clientside NearCache and Serverside Eviction blocking cluster

[Вячеслав Коптилин]

Hello Prasad,

I will take a look and file a ticket if needed.

Thanks,
S.

пн, 23 нояб. 2020 г. в 19:23, pvprsd :

> Hi,
>
> Did anyone get a chance to look into this issue? Is this supported
> configuration for ignite? As these 2 (clientside NearCache and serverside
> eviction) are very common features, I am wondering many projects should be
> using this combination.
>
> Can this be reported as a defect for ignite, if there is no configuration
> fix for this problem?
>
> Many thanks in advance.
>
> Thanks,
> Prasad
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Apache Ignite Clientside NearCache and Serverside Eviction blocking cluster

[pvprsd]

Hi,

Did anyone get a chance to look into this issue? Is this supported
configuration for ignite? As these 2 (clientside NearCache and serverside
eviction) are very common features, I am wondering many projects should be
using this combination.

Can this be reported as a defect for ignite, if there is no configuration
fix for this problem?

Many thanks in advance.

Thanks,
Prasad




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Apache Ignite Clientside NearCache and Serverside Eviction blocking cluster

[pvprsd]

Hi,

I have created a sample reproducer using examples files. I am using ignite
2.8.0 version.

Here I have updated server configurations in ExampleNodeStartup.java, and
client configurations in CacheClientBinaryPutGetExample.java files.

Server is configured with eviction-enabled, and client is configured with
near-cache. Client keeps adding more entries to the cache. Server stores all
the entries, and when memory utilization reached 90%, it should evict the
LRU entries from server cache in this use-case. 

In 7-8 minutes, the server is not able to evict the data and showing the
warning "Too many failed attempts to evict page: 30".

I am attaching the updated files for your reference. Can you please review
and provide your recommendations.

Ignite_Blocker_Issue.zip

  

Thanks,
Prasad




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Apache Ignite Clientside NearCache and Serverside Eviction blocking cluster

[pvprsd]

Hi All,

We have recently enabled Eviction on Ignite server nodes. We were using
ignite nearcache for ignite client nodes.

Due to nearcache, the server side cache-entries are not getting evicted.
Eventually this leads to off-cache full, and "Too many attempts to evict
data page" error and ignite cluster stopped responding. We have given enough
memory to ignite server nodes i.e. 4+4gb for heap and off-heap.

>From debugging the server code, it looks like the near-cache enforces the
entries to maintain the client-node id; and due to this those entries are
not getting evicted from server.

When the clientside the entries got evicted (with clientside LRU eviction),
it seems the client-node reference in serverside entry is not getting
cleared. Hence after sometime it leads to ignite getting blocked. This is
consistently happening.

Either I am doing some thing wrong with the ignite configurations, OR there
is a bug in the ignite with this combination.

Can you please guide me, on how to find resolution to this problem.

Thanks,
Prasad





--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: NearCache for SQL tables

[Denis Magda]

Manesh,

How many server nodes do you have in total and have you tried to perform
SQL optimizations listed below?

https://www.gridgain.com/docs/latest/perf-troubleshooting-guide/sql-tuning


Denis

On Thursday, December 26, 2019, Mahesh Renduchintala <
mahesh.renduchint...@aline-consulting.com> wrote:

> Dennis,
>
> We use ignite for supply chain management usecases and almost all are SQL
> schemas
>
> In replicated mode, for large tables and for a single query, the data
> reads are happening from ram and disk of a single node.
> We measured a 4-time increase in query latencies (reads) in replicated
> mode compared to partitioned mode. This is possible due to the disk coming
> into play.
>
> I was thinking a near cache at client node might improve read latencies.
>
> We find replicated mode much more suited than partitioned mode due to high
> availability needs. We also reported several issues with JOINS in
> partitioned mode and this is a show stopper for us in production
> deployments. Replicated mode does not have these issues with join and is a
> simple solution for backup and restore.
>
> regards
> Mahesh
>
>

-- 
-
Denis

Re: NearCache for SQL tables

[Mahesh Renduchintala]

Dennis,

We use ignite for supply chain management usecases and almost all are SQL 
schemas

In replicated mode, for large tables and for a single query, the data reads are 
happening from ram and disk of a single node.
We measured a 4-time increase in query latencies (reads) in replicated mode 
compared to partitioned mode. This is possible due to the disk coming into play.

I was thinking a near cache at client node might improve read latencies.

We find replicated mode much more suited than partitioned mode due to high 
availability needs. We also reported several issues with JOINS in partitioned 
mode and this is a show stopper for us in production deployments. Replicated 
mode does not have these issues with join and is a simple solution for backup 
and restore.

regards
Mahesh

Re: NearCache for SQL tables

[Denis Magda]

Hello Mahesh,

The near cache works only for key-value APIs. Why would you have for SQL,
what's your usage scenario or issue?

-
Denis


On Thu, Dec 26, 2019 at 4:29 AM Mahesh Renduchintala <
mahesh.renduchint...@aline-consulting.com> wrote:

> Hi,
>
> is there a way to create near cache on CLIENT nodes for SQL tables?
>
>
> regards
> Mahesh
>
>

NearCache for SQL tables

[Mahesh Renduchintala]

Hi,

is there a way to create near cache on CLIENT nodes for SQL tables?


regards
Mahesh

Re: Are data in NearCache BinaryObject?

[Prasad Bhalerao]

I had debugged the near cache once and I found that near cache simply
stores the entries concurrent hashmap in deserialized format.

But be careful with near cache usage. I faced many issues and finally
removed it.
I had reported an issue on this forum but I couldn't not create reproducer
for it but it used give give me exceptions in running application. Check
near cache issue on user list.


On Thu 12 Dec, 2019, 9:09 PM Cong Guo  Hi,
>
> My application needs to read all entries in the cache frequently. The
> entries may be updated by others. I'm thinking about two solutions to avoid
> a lot deserialization. First, I can maintain my own local hash map and
> relies on continuous queries to get the update events. Second, I can use a
> NearCache, but if the data in NearCache are still serialized, this method
> does not work for my application.
>
> Thanks,
> Nap
>
> On Thu, Dec 12, 2019 at 5:37 AM Ilya Kasnacheev 
> wrote:
>
>> Hello!
>>
>> It is actually hard to say without debugging. I expect that it is
>> BinaryObject or primitive type or byte[].
>>
>> It is possible to enable onheap caching, in this case objects will be
>> held as is, and also sed copyOnRead=false, in this case objects will not
>> even be copied.
>> However, I'm not sure if Near Cache will interact with onheap caching.
>>
>> Why does it matter for your use case?
>>
>> Regards,
>> --
>> Ilya Kasnacheev
>>
>>
>> ср, 11 дек. 2019 г. в 22:54, Cong Guo :
>>
>>> Hi,
>>>
>>> Are the entries stored in local NearCache on my client node in the
>>> format of deserialized java objects or BinaryObject? Will the entry in
>>> local on-heap NearCache be deserialized from BinaryObject when I call the
>>> get function?
>>>
>>> Thanks,
>>> Nap
>>>
>>

Re: Are data in NearCache BinaryObject?

[Cong Guo]

Hi,

My application needs to read all entries in the cache frequently. The
entries may be updated by others. I'm thinking about two solutions to avoid
a lot deserialization. First, I can maintain my own local hash map and
relies on continuous queries to get the update events. Second, I can use a
NearCache, but if the data in NearCache are still serialized, this method
does not work for my application.

Thanks,
Nap

On Thu, Dec 12, 2019 at 5:37 AM Ilya Kasnacheev 
wrote:

> Hello!
>
> It is actually hard to say without debugging. I expect that it is
> BinaryObject or primitive type or byte[].
>
> It is possible to enable onheap caching, in this case objects will be held
> as is, and also sed copyOnRead=false, in this case objects will not even be
> copied.
> However, I'm not sure if Near Cache will interact with onheap caching.
>
> Why does it matter for your use case?
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> ср, 11 дек. 2019 г. в 22:54, Cong Guo :
>
>> Hi,
>>
>> Are the entries stored in local NearCache on my client node in the format
>> of deserialized java objects or BinaryObject? Will the entry in local
>> on-heap NearCache be deserialized from BinaryObject when I call the get
>> function?
>>
>> Thanks,
>> Nap
>>
>

Re: Are data in NearCache BinaryObject?

[Ilya Kasnacheev]

Hello!

It is actually hard to say without debugging. I expect that it is
BinaryObject or primitive type or byte[].

It is possible to enable onheap caching, in this case objects will be held
as is, and also sed copyOnRead=false, in this case objects will not even be
copied.
However, I'm not sure if Near Cache will interact with onheap caching.

Why does it matter for your use case?

Regards,
-- 
Ilya Kasnacheev


ср, 11 дек. 2019 г. в 22:54, Cong Guo :

> Hi,
>
> Are the entries stored in local NearCache on my client node in the format
> of deserialized java objects or BinaryObject? Will the entry in local
> on-heap NearCache be deserialized from BinaryObject when I call the get
> function?
>
> Thanks,
> Nap
>

Are data in NearCache BinaryObject?

[Cong Guo]

Hi,

Are the entries stored in local NearCache on my client node in the format
of deserialized java objects or BinaryObject? Will the entry in local
on-heap NearCache be deserialized from BinaryObject when I call the get
function?

Thanks,
Nap

Re: NearCache

[Mikhail]

Hi Greg,

you can try to use OptimizedMarshaller:

igniteConfig.setMarshaller(new OptimizedMarshaller())

however, this will lead to some limitations, which you can find in java doc
of setMarshaller method.

Thanks,
Mike.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: NearCache

[Ilya Kasnacheev]

Hello!

You can have onHeap cache
https://apacheignite.readme.io/docs/memory-configuration#section-on-heap-caching

setting copyOnRead to false will let you keep unmarshalled values in it. Be
sure to avoid modifying them in your code since that's not tracked.

Regards,
-- 
Ilya Kasnacheev


ср, 16 янв. 2019 г. в 18:03, Grégory Jevardat de Fombelle <
gregory.jevar...@unige.ch>:

> Hello
>
> Is there any option to have  a nearCache in Ignite that store
> unMarschalled values instead of serialized ones. I ask this for performance
> reasons.
> I noticed that for big cached objects, default Java unserialization is
> quite expensive, like ~ 2 secs for a complex > 100MB object.  So in the end
> caching this kind of objects in a near cache is not really interesting
> given the penalty of the serialisation.
> On the opposite caching a reference of this object in a custom applicative
> cache like ThreadLocal map of even just a Hashmap, the retrieval is almost
> instantaneous (order of 10-100 nano secs)
>
> Note that I did not tested binary serialisation or storing Json serialized
> objects or other custom serialisation libraries like protobuf or kryo. Also
> I'am not sure if it is possible to use custom libraries for serialisation
> of huge and complex graph of objects.
> Are there any options or architectural recommandations and some benchmarks
> on this topic ?

NearCache

[Grégory Jevardat de Fombelle]

Hello

Is there any option to have  a nearCache in Ignite that store unMarschalled 
values instead of serialized ones. I ask this for performance reasons.
I noticed that for big cached objects, default Java unserialization is quite 
expensive, like ~ 2 secs for a complex > 100MB object.  So in the end caching 
this kind of objects in a near cache is not really interesting given the 
penalty of the serialisation.
On the opposite caching a reference of this object in a custom applicative 
cache like ThreadLocal map of even just a Hashmap, the retrieval is almost 
instantaneous (order of 10-100 nano secs)

Note that I did not tested binary serialisation or storing Json serialized 
objects or other custom serialisation libraries like protobuf or kryo. Also 
I'am not sure if it is possible to use custom libraries for serialisation of 
huge and complex graph of objects.
Are there any options or architectural recommandations and some benchmarks on 
this topic ? 

Re: NearCache entries on heap ?

[Maxim.Pudov]

Near cache is always on-heap. Its not an independent cache, its just a local
copy of a cache stored on another node. In some cases it improves
performance.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: NearCache entries on heap ?

[userx]

Hi,

Can anyone help me on the same ?





--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

NearCache entries on heap ?

[userx]

Hi All,

I have created a NearCache using the following code.

Ignition.setClientMode(true);
Ignite ignite = Ignition.start("example-persistent-store.xml");
ignite.cluster().active(true);
NearCacheConfiguration nCfg = new
NearCacheConfiguration<>();
IgniteCache cache = 
ignite.getOrCreateCache(new
CacheConfiguration("nCfg"),nCfg);
for (int i = 0; i < Integer.MAX_VALUE; ++i) {
cache.put(i, i);
}

When I connected to the cluster during the time client was writing, it
displayed the Total Entries(OnHeap/OffHeap) stats as 

 Name(@)|Mode | Nodes |Entries (Heap / Off-heap)
 Hits|  Misses   |   Reads   |  Writes   |

nCfg(@c1)| PARTITIONED | 2 | min: 1280928 (1280928 / 0)
min: 0| min: 0| min: 0| min: 0|
 | |   | avg: 1922360.00 (1281896.00 /
640464.00)
avg: 0.00 | avg: 0.00 | avg: 0.00 | avg: 0.00 |
 | |   | max: 2563792 (1282864 / 1280928)
max: 0| max: 0| max: 0| max: 0|

Why should there be any entries "on heap" for the client node ? My
understanding was that since 2.0 if we do not enable on heap parameter
explicitly, everything is written off-heap.





--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Near Cache Topoolgy change causes NearCache to always miss.

[slava.koptilin]

Hi Tim,

I was able to reproduce the issue.
I've created the following jira ticket in order to track this:
https://issues.apache.org/jira/browse/IGNITE-6767

Thanks,
S.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Near Cache Topoolgy change causes NearCache to always miss.

[Timay]

Hey Slave, 

Just some more details, it looks like the GridNearCacheEntry.primaryNode is
the suspect. That is what updates the topVer. If you create 2 nodes, then X
clients. Using one of the clients to create a cache with a near cache config
it seems to work as expected. However, if you create a cache on the node
instance then populate that cache, but try to get from a near cache created
through the client the change of a topology will cause the setting of the
topVer to none, then the primaryNode never get to reset the topVer causing
the 
GridNearCacheEntry.valid to return false. 

Hope that makes some sense, but attached is a test i used. It's crude but
should at least show what i am trying to convey. The good test  to what is
expected with a miss count being the initial call and the one after the
invalidation of the topology change, the bad has almost all read marked as
misses which causes the hard hit to the cluster. 

GridCacheNearClientMissTest.java

  

Thanks
Tim(ay)



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Near Cache Topoolgy change causes NearCache to always miss.

[slava.koptilin]

Hi Timay,

I will try to reproduce the issue on my side.

Thanks!



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Near Cache Topoolgy change causes NearCache to always miss.

[Timay]

Hey Slava,

Here they are. 

  @Bean(name ="cache" )
public IgniteCache getCache() throws IOException {
   
CacheConfiguration config = new CacheConfiguration<>();
config.setAtomicityMode(CacheAtomicityMode.ATOMIC);
config.setCacheMode(CacheMode.REPLICATED);
config.setCopyOnRead(false);
config.setName("cache");
  
NearCacheConfiguration nearCfg = new
NearCacheConfiguration<>();
nearCfg.setNearEvictionPolicy(new LruEvictionPolicy<>(25000));

   return _igniteSpringBean.getOrCreateCache(config,nearCfg);
}



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Near Cache Topoolgy change causes NearCache to always miss.

[slava.koptilin]

Hi Timay,

Could you please share the configuration of cache and near cache?

Thanks!.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Near Cache Topoolgy change causes NearCache to always miss.

[Timay]

Hey all,


Versions: 2.1 & 2.3-SNAPSHOT
Setup: 1 client, 1+ servers


I found an issue with Near Caching. When a topology change occurs the
GridNearCacheEntry.valid method will update the GridNearCacheEntry.topVer to
None. This version then never gets "reset". Causing the state of the
GridNearCacheEntry.valid to always be false because the None check is done
before the update of the version. This will cause all future request to miss
the near cache and make a remote call to the cluster

>From what i can see, the the AffinityTopologyVersion get set in a few spots
so i was unsure where to patch this. For testing i added a compare check to
the GridNearCacheEntry.loadedValue which set the topVer to the latest.


This is fairly large issue for us, as the performance goes from <1
millisecond to +40 milliseconds.

Let me know if you have questions.






--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: NearCache can be used through ODBC interface

[Nikolai Tikhonov]

SQL query distributed on whole cluster, near cache not needed for it.
Records will be read on data node (usually cache, not near) in any case for
queries and usage data from near cache will not provide performance benefit.

On Wed, Jan 4, 2017 at 2:43 PM, Navneet Kumar 
wrote:

> In that case I cannot read the records from near cache using the Near
> cache.
> Is it true?
>
>
>
> --
> View this message in context: http://apache-ignite-users.705
> 18.x6.nabble.com/NearCache-can-be-used-through-ODBC-interfac
> e-tp9859p9865.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>

Re: NearCache can be used through ODBC interface

[Navneet Kumar]

In that case I cannot read the records from near cache using the Near cache.
Is it true?



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/NearCache-can-be-used-through-ODBC-interface-tp9859p9865.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: NearCache can be used through ODBC interface

[Nikolai Tikhonov]

Hi Kumar!

You can use ODBC interface for querying to any cache. But keep in your mind
that Ignite SQL engine sends request on all nodes and data will be got from
usual cache (not near).

Thanks,
Nikolay.

On Wed, Jan 4, 2017 at 11:31 AM, Navneet Kumar  wrote:

> Hi Val,
> LocalCache works well with ODBC interface. But Is there any issue using the
> NearCache through ODBC interface? Please give me descriptive idea about how
> to use it via ODBC interface.
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/NearCache-can-be-used-through-ODBC-
> interface-tp9859.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>

NearCache can be used through ODBC interface

[Navneet Kumar]

Hi Val,
LocalCache works well with ODBC interface. But Is there any issue using the
NearCache through ODBC interface? Please give me descriptive idea about how
to use it via ODBC interface.



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/NearCache-can-be-used-through-ODBC-interface-tp9859.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.