Re: Cache Problems
Hi, In a nutshell: Ignite writes to one of these files while another is being packed by Swap compactor. When packing finished files change places. On Wed, Sep 14, 2016 at 5:35 AM, Level D <724172...@qq.com> wrote: > > I use lucense index. > > By the way, > what do these two files mean? > > -- > View this message in context: http://apache-ignite-users. > 70518.x6.nabble.com/Cache-Problems-tp7477p7697.html > Sent from the Apache Ignite Users mailing list archive at Nabbl >
Re: Cache Problems
I use lucense index. By the way, what do these two files mean? -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Cache-Problems-tp7477p7697.html Sent from the Apache Ignite Users mailing list archive at Nabbl
Re: Cache Problems
It looks like known issue https://issues.apache.org/jira/browse/IGNITE-2714 On Tue, Sep 13, 2016 at 11:13 AM, kromulan wrote: > I've had same problem in the past but it was a memory leak in indexing > code. > Do you use indexing on your entities ? > > > > -- > View this message in context: http://apache-ignite-users. > 70518.x6.nabble.com/Cache-Problems-tp7477p7697.html > Sent from the Apache Ignite Users mailing list archive at Nabble.com. >
Re: Cache Problems
I've had same problem in the past but it was a memory leak in indexing code. Do you use indexing on your entities ? -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Cache-Problems-tp7477p7697.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Cache Problems
Hi, I have about 17GB data needs to be loaded into ignite. Every time I destory the cache and create the cache with the same name, the memory increases 17GB. It seems the off-heap memory not released. I find ignite can release off-heap memory when grid stopped. I'd like to know if ignite can release off-heap memory during grid is running? -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Cache-Problems-tp7477p7592.html
Re: Cache Problems
>
>
>
>
>
> com.ffcs.oss.ignite.rlam.domain.cable.AlarmCableOptKey
> com.ffcs.oss.ignite.rlam.domain.cable.AlarmCableOpt
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> com.ffcs.oss.ignite.rlam.domain.cable.AlarmCableSecStatKey
> com.ffcs.oss.ignite.rlam.domain.cable.AlarmCableSecStat
>
>
>
>
>
>
>
>
>
>
>
> java.lang.Integer
> com.ffcs.oss.ignite.rlam.vo.AlarmCableSecTmp
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> 192.168.14.81:47500..47509
> 192.168.14.85:47500..47509
> 192.168.14.86:47500..47509
>
>
>
>
>
>
>
>
>
> That's my full configuration including cache configuration.
>
> Following is my code about bucking up and destorying caches.
>
> DataLoaderRunnable.java
> public class DataLoaderRunnable implements IgniteRunnable{
>
> private IgnitePublicService ignitePublicService;
>
> private CacheNameVO cacheNameVO;
> private String loadKey;
>
> private List lstSql;
>
> public DataLoaderRunnable(IgnitePublicService
> ignitePublicService,CacheNameVO
> vo, String loadKey,List lstSql){
> this.cacheNameVO = vo;
> this.loadKey = loadKey;
> this.lstSql = lstSql;
> this.ignitePublicService = ignitePublicService;
> }
>
> @Override
> public void run() {
> try {
> DataLoaderTaskAdapter task = new DataLoaderTaskAdapter(
> ignitePublicService,cacheNameVO.value,loadKey,lstSql);
> int task_num = ignitePublicService.getIgnite(
> ).compute().execute(task,null);
> if(task_num == lstSql.size()){
> LogPrint.LOADER_BASE_INFO_LOGGER.info("DataLoaderRunnable is End,
> CacheName:" + cacheNameVO.value);
> cacheNameVO.isLoadedData = true;
> IgniteCache cacheForName =
> ignitePublicService.getIgnite().cache(CacheName.CACHE_NAME);
> cacheForName.put(cacheNameVO.id,cacheNameVO);
> Thread.sleep(1000L);
> //destory former cache
> IgniteCache cacheRemove = ignitePublicService.getCache(
> cacheNameVO.valueTemp);
> cacheRemove.destroy();
> LogPrint.LOADER_BASE_INFO_LOGGER.info("DataLoaderRunnable destroy
> Temp Cache. CacheName:" + cacheNameVO.valueTemp);
> //create cache
> CacheConfiguration[] cacheCfgs = ignitePublicService.getIgnite(
> ).configuration().getCacheConfiguration();
> for (CacheConfiguration cacheCfg : cacheCfgs) {
> if(cacheCfg.getName().equals(cacheNameVO.valueTemp)){
> CacheConfiguration copy = new CacheConfiguration<>(cacheCfg);
> copy.setName(cacheNameVO.valueTemp);
> ignitePublicService.getIgnite().createCache(copy);
> }
> }
> LogPrint.LOADER_BASE_INFO_LOGGER.info("DataLoaderRunnable Create Temp
> Cache. CacheName:" + cacheNameVO.valueTemp);
> }
> } catch (Exception e) {
> LogPrint.ERROR_INFO_LOGGER.info("load entries:" + lstSql,e);
> }
> }
> }
>
> DataLoaderTaskAdapter.java
> public class DataLoaderTaskAdapter extends ComputeTaskAdapter Integer>{
>
> private List lstSql;
> private String cacheName;
> private String loadKey;
> private IgnitePublicService ignitePublicService;
> public DataLoaderTaskAdapter(IgnitePublicService
> ignitePublicService,String cacheName,String loadKey,List lstSql) {
> this.ignitePublicService = ignitePublicService;
> this.cacheName = cacheName;
> this.loadKey = loadKey;
> this.lstSql = lstSql;
> }
> @Override
> public Map map(List
> subgrid, String arg1) throws IgniteException {
> Map map = new HashMap<>(lstSql.size());
> Iterator it = subgrid.iterator();
> for (final String sql : lstSql) {
> if (!it.hasNext())
> it = subgrid.iterator();
> ClusterNode node = it.next();
> map.put(new ComputeJobAdapter() {
> @Override public Object execute() {
> doLoadData(sql);
> return 1;
> }
> }, node);
> }
> return map;
> }
> @Override
> public Integer reduce(List results) throws
> IgniteException {
> int sum = 0;
> for (ComputeJobResult res : results)
> sum += res.getData();
> return sum;
> }
> private void doLoadData(String sql){
> LogPrint.LOADER_BASE_INFO_LOGGER.info("DataLoaderTaskAdapter
> started ,cacheName:" + cacheName + ",sql:" + sql);
> IgniteCache cache = ignitePublicService.getCache(cacheName);
> long startTime = new Date().getTime();
> cache.loadCache(null, loadKey, sql);
> LogPrint.LOADER_BASE_INFO_LOGGER.info("load cache use time:" +
> (new Date().getTime() - startTime) + "ms, size=" +
> cache.size()+",cacheName="+cacheName + ",sql:" + sql);
> }
> }
>
> By the way, does ignite proide an API to release off-heap memory? Is there
> an offHeapMaxMemory property which can rule all caches within an ignite
> instance instead of one cache in an ignite instance?
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/Cache-Problems-tp7477p7592.html
>
>
--
Vladislav Pyatkov
Re: Cache Problems
n();
for (CacheConfiguration cacheCfg : cacheCfgs)
{
if(cacheCfg.getName().equals(cacheNameVO.valueTemp)){
CacheConfiguration copy
= new CacheConfiguration<>(cacheCfg);
copy.setName(cacheNameVO.valueTemp);
ignitePublicService.getIgnite().createCache(copy);
}
}
LogPrint.LOADER_BASE_INFO_LOGGER.info("DataLoaderRunnable Create Temp Cache.
CacheName:" + cacheNameVO.valueTemp);
}
} catch (Exception e) {
LogPrint.ERROR_INFO_LOGGER.info("load entries:" +
lstSql,e);
}
}
}
DataLoaderTaskAdapter.java
public class DataLoaderTaskAdapter extends ComputeTaskAdapter{
private List lstSql;
private String cacheName;
private String loadKey;
private IgnitePublicService ignitePublicService;
public DataLoaderTaskAdapter(IgnitePublicService
ignitePublicService,String cacheName,String loadKey,List lstSql) {
this.ignitePublicService = ignitePublicService;
this.cacheName = cacheName;
this.loadKey = loadKey;
this.lstSql = lstSql;
}
@Override
public Map map(List
subgrid, String arg1) throws IgniteException {
Map map = new HashMap<>(lstSql.size());
Iterator it = subgrid.iterator();
for (final String sql : lstSql) {
if (!it.hasNext())
it = subgrid.iterator();
ClusterNode node = it.next();
map.put(new ComputeJobAdapter() {
@Override public Object execute() {
doLoadData(sql);
return 1;
}
}, node);
}
return map;
}
@Override
public Integer reduce(List results) throws
IgniteException {
int sum = 0;
for (ComputeJobResult res : results)
sum += res.getData();
return sum;
}
private void doLoadData(String sql){
LogPrint.LOADER_BASE_INFO_LOGGER.info("DataLoaderTaskAdapter started
,cacheName:" + cacheName + ",sql:" + sql);
IgniteCache cache = ignitePublicService.getCache(cacheName);
long startTime = new Date().getTime();
cache.loadCache(null, loadKey, sql);
LogPrint.LOADER_BASE_INFO_LOGGER.info("load cache use time:" + (new
Date().getTime() - startTime) + "ms, size=" +
cache.size()+",cacheName="+cacheName + ",sql:" + sql);
}
}
By the way, does ignite proide an API to release off-heap memory? Is there an
offHeapMaxMemory property which can rule all caches within an ignite instance
instead of one cache in an ignite instance?
--
View this message in context:
http://apache-ignite-users.70518.x6.nabble.com/Cache-Problems-tp7477p7592.html
Re: Cache Problems
Hi Zhou, Most likely this ticket is not connected with what you're seeing. Please show full cache configuration and the code sample that reproduces the behavior. -Val -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Cache-Problems-tp7477p7592.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Cache Problems
Hi Val, Thanks a lot for your reply. For the third question, I have a case. I buckup a cache in non-heap memory with some updates eveyday. When it's finished, I destory the former cache. But system memory is used more and more. Here's a url. https://issues.apache.org/jira/browse/IGNITE-2833 Is there a kind of bug like that? For the forth question, here's a printscreen. And yes, it's swap. When I use "cache -a", it shows this. Entries in hard disk are not counted. For the fifth question, here's another printscreen. What do *.left and *.right mean? What are the diffrences between them? Regards, Zhou. E7D76A68@1D09C02F.E2D8CC57.png Description: Binary data 1411B268@1D09C02F.E2D8CC57.png Description: Binary data B9809A08@1D09C02F.E2D8CC57.png Description: Binary data A8F}DE_RP22W}3IXG}[VT0U.png Description: Binary data
Re: Cache Problems
Hi Zhou, 1. Off-heap memory is managed manually by Ignite and it's released whenever it's not needed anymore (e.g., you remove entry from cache). System.gc() has access only to heap memory. 2. I'm not sure I understand this question. Can you clarify? 3. I'm not aware of such bugs. Are you experiencing any unexpected behavior? 4. What do you mean by 'exchanged to hard disk'? Are you using swap? 5. Can you clarify what do you mean by that? -Val -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Cache-Problems-tp7477p7499.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Cache Problems
Hi all, I have several questions about ignite 1.6.0's cache. 1. When does ignite free the non-heap memory? I found a system.gc in source code, is it? 2. How can I check the data in non-heap memory with just ignite itself? 3. Are there any bugs about ignite cannot release non-heap memory correctly? 4. How can I check some data exchanged to hard disk by ignite-visor-console? 5. What does "left right" exactly mean when I exchange some data to hard disk? Regards, Zhou.
