get api takes more time when adding more nodes to cluster

[Biren Shah]

Hi,
On adding more nodes to the cluster, the get api slows down. I have a cache 
with 2 million entries. The cache gets loaded at the start of the cluster and 
does not change frequently. Here is the cache configuration. I had start the 
cluster with 2 nodes and added 2 more nodes later on.

CacheConfiguration<String, Entry> cacheConfig = new CacheConfiguration<>();
cacheConfig.setOnheapCacheEnabled(true);
cacheConfig.setCacheMode(CacheMode.PARTITIONED);
cacheConfig.setBackups(1);
cacheConfig.setCopyOnRead(false);
cacheConfig.setWriteSynchronizationMode(CacheWriteSynchronizationMode.FULL_SYNC);
cacheConfig.setStatisticsEnabled(true);

I have few questions.

  1.  Get operations on cache are not based on affinity. So, in that case which 
is better performant get or localpeek?
  2.  Cache is not initialized with keep binary option. We read most of the 
entries of the cache on regular basis. Shall I change it to keep the data in 
binary format? Will that help in bringing the get time down?
  3.  What configuration nobs I can change to investigate the issue further?
  4.  What is recommended cache configuration for such use case?

Each node is setup with 8GB on memory and can use upto 8 cores.

Thanks,
Biren Shah