Re: Off-Heap and Eviction policy questions

[Yasser]

Thank you! I resolved by moving the data load operation outside the server
cluster.

Thanks,
Yasser

On Thu, Aug 3, 2017 at 10:01 AM, slava.koptilin <slava.kopti...@gmail.com>
wrote:

> Hi Yasser,
>
> Well, you are using IgniteDataStreamer in order to load data into cache.
> The IgniteDataStreamer actually buffers entries before sending data to
> remote node, and, obviously, it requires java heap.
> You can try increase Xmx options and adjust the following properties [1]:
>  - perNodeBufferSize
>This setting controls the size of internal per-node buffer before
> buffered data is sent to remote node.
>  - perNodeParallelOperations
>Automatic flush frequency in milliseconds. this is the time after which
> the streamer will make an attempt
>to submit all data added so far to remote nodes (default value is 0).
>
> [1] https://ignite.apache.org/releases/2.1.0/javadoc/index.html
>
> Thanks.
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/Off-Heap-and-Eviction-policy-questions-
> tp15869p15948.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>

Off-Heap and Eviction policy questions

[Yasser]

I am developing a datagrid using Ignite 2.1 to store a large size table
(~25G).
I will be using 8 nodes with 12G RAM.
I am currently developing in my local machine and trying to understand the
cache load mechanism, printed metrics and eviction policy.

Based on the below metrics, the heap size keeps increasing. I thought the
default is off-heap and expected the Non heap memory to increase.
I haven't used setOnheapCacheEnabled in code, so that shouldnt be enabled.
*Metrics for local node (to disable set 'metricsLogFrequency' to 0)*
*^-- Node [id=782e9494, name=somegrid, uptime=00:09:30:317]*
*^-- H/N/C [hosts=1, nodes=1, CPUs=4]*
*^-- CPU [cur=1.7%, avg=1.43%, GC=0%]*
*^-- PageMemory [pages=2250]*
*^-- Heap [used=4921MB, free=19.9%, comm=6144MB]*
*^-- Non heap [used=76MB, free=-1%, comm=78MB]*
*^-- Public thread pool [active=0, idle=0, qSize=0]*
*^-- System thread pool [active=0, idle=6, qSize=0]*
*^-- Outbound messages queue [size=0]*

1) Am I missing something about off-heap vs on-heap or is my interpretation
on the printed metrics incorrect?
2) Another question is about Eviction policy during initial data load -
Since I am running in my local machine(to test code before pushing to
multiple servers), I dont have RAM capacity to load all of the data.
I ran the application in local to see if eviction policy would kick-in
during initial data load, but I received java.lang.OutOfMemoryError: Java
heap space error.
Shouldn't eviction process remove data from page when 80% is reached?
Is eviction policy monitor not started at loadCache or is there an issue
with my configuration (given below)?

Code snippet-- The undisclosed section is just trivial part of the settings

CacheConfiguration<String, String> cacheConfiguration = new
CacheConfiguration<>();
cacheConfiguration.setName(CACHE_NAME)
.setCacheMode(CacheMode.PARTITIONED)
.setMemoryPolicyName(DEFAULT_MEMORY_POLICY)

.setCacheStoreFactory(FactoryBuilder.factoryOf(dataLoader));

MemoryPolicyConfiguration memoryPolicyConfiguration = new
MemoryPolicyConfiguration()
.setName(DEFAULT_MEMORY_POLICY)
.setInitialSize(STARTUP_OFFHEAP_MEMORY_SIZE) //3G
.setPageEvictionMode(DataPageEvictionMode.RANDOM_2_LRU);

MemoryConfiguration memCfg = new MemoryConfiguration()
.setDefaultMemoryPolicyName(DEFAULT_MEMORY_POLICY)
.setMemoryPolicies(memoryPolicyConfiguration);
public class DataLoader extends CacheStoreAdapter<String, String>
implements Serializable {

@Autowired
private transient JdbcTemplate jdbcTemplate;

@Value("${load.all.query}") //select key, vaue from table
private String loadAllQuery;

@Value("${store.one.query}")
private String storeOneQuery;

 .

@Override
public void loadCache(IgniteBiInClosure<String, String> closure,
Object... args) {
log.info("In loadcache");
jdbcTemplate.query(loadAllQuery, (rs, rownNum) -> {
closure.apply(rs.getString(1), rs.getString(2));
    return null;
});
}
}

Thanks,
Yasser