Re: loadAll and removeAll from cache with custom store

[slava.koptilin]

Hi Matt,

I've tried the following code based on your configuration and it works.

public static void main(String[] args) throws Exception {
Ignite ignite = Ignition.start(createIgniteConfiguration());
IgniteCache cache1 =
ignite.getOrCreateCache(createULConfiguration("test-cache-1"));

// upload 10_000 keys
for (int i = 0; i < 10_000; ++i)
cache1.put(i, i);

// remove all keys
cache1.removeAll();

Thread.sleep(10_000);

ignite.close();
}

private static CacheConfiguration createULConfiguration(String name) {
CacheConfiguration cacheConfig = new
CacheConfiguration<>();
cacheConfig.setName(name);
cacheConfig.setCacheMode(CacheMode.PARTITIONED);
cacheConfig.setBackups(2);

   
cacheConfig.setWriteSynchronizationMode(CacheWriteSynchronizationMode.FULL_SYNC);
cacheConfig.setAtomicityMode(CacheAtomicityMode.ATOMIC);

cacheConfig.setWriteBehindEnabled(true);
cacheConfig.setWriteBehindBatchSize(512);
cacheConfig.setWriteBehindFlushSize(10240);
cacheConfig.setWriteBehindFlushFrequency(5_000);

   
cacheConfig.setCacheStoreFactory(FactoryBuilder.factoryOf(CacheStoreExample.class));

cacheConfig.setReadThrough(true);
cacheConfig.setWriteThrough(true);

return cacheConfig;
}

public static class CacheStoreExample extends CacheStoreAdapter {
private Map store = new HashMap();

@Override public Object load(Object key) throws CacheLoaderException
{
return store.get(key);
}
@Override public void write(Cache.Entry entry) throws
CacheWriterException {
store.put(entry.getKey(), entry.getValue());
}
@Override public void delete(Object key) throws CacheWriterException
{
store.remove(key);
}
@Override public void deleteAll(Collection keys) {
System.out.println("CacheStoreExample::deleteAll(), keys.size="
+ keys.size() + " Thread=" + Thread.currentThread().getName());
for (Object k : keys)
store.remove(k);
}
}

The output is following
---
CacheStoreExample::deleteAll(), keys.size=512
Thread=flusher-0-#44%test-grid%
CacheStoreExample::deleteAll(), keys.size=512
Thread=flusher-0-#44%test-grid%
CacheStoreExample::deleteAll(), keys.size=512
Thread=flusher-0-#44%test-grid%
CacheStoreExample::deleteAll(), keys.size=512
Thread=flusher-0-#44%test-grid%
CacheStoreExample::deleteAll(), keys.size=512
Thread=flusher-0-#44%test-grid%
CacheStoreExample::deleteAll(), keys.size=512
Thread=flusher-0-#44%test-grid%
CacheStoreExample::deleteAll(), keys.size=512
Thread=flusher-0-#44%test-grid%
CacheStoreExample::deleteAll(), keys.size=512
Thread=flusher-0-#44%test-grid%
CacheStoreExample::deleteAll(), keys.size=512
Thread=flusher-0-#44%test-grid%
CacheStoreExample::deleteAll(), keys.size=512
Thread=flusher-0-#44%test-grid%
CacheStoreExample::deleteAll(), keys.size=512
Thread=flusher-0-#44%test-grid%
CacheStoreExample::deleteAll(), keys.size=512
Thread=flusher-0-#44%test-grid%
CacheStoreExample::deleteAll(), keys.size=512
Thread=flusher-0-#44%test-grid%
CacheStoreExample::deleteAll(), keys.size=512
Thread=flusher-0-#44%test-grid%
CacheStoreExample::deleteAll(), keys.size=512
Thread=flusher-0-#44%test-grid%
CacheStoreExample::deleteAll(), keys.size=512
Thread=flusher-0-#44%test-grid%
CacheStoreExample::deleteAll(), keys.size=512
Thread=flusher-0-#44%test-grid%
CacheStoreExample::deleteAll(), keys.size=512
Thread=flusher-0-#44%test-grid%
CacheStoreExample::deleteAll(), keys.size=512
Thread=flusher-0-#44%test-grid%
CacheStoreExample::deleteAll(), keys.size=272
Thread=flusher-0-#44%test-grid%
[18:16:53] Ignite node stopped OK [name=test-grid, uptime=00:00:05:083]


So, all 10_000 keys were removed.

Thanks!



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: loadAll and removeAll from cache with custom store

[matt]

Hi,

The size in my store is about 9k. I do have loadAll() working now, so once
that completes, the cache has 9k items as well.

For deleteAll(), I had it work once where it called my adapter multiple
times with all of the expected keys. But then restarting my app and trying
again (after loading new data) it didn't work at all. Do I need to loadAll()
into my Ignite cache before I can delete everything from the backend store?

Here's my configuration:

String name = "foo";

CacheConfiguration cacheConfig = new
CacheConfiguration<>();
cacheConfig.setName(name);
cacheConfig.setCacheMode(CacheMode.PARTITIONED);
cacheConfig.setBackups(2);
   
cacheConfig.setWriteSynchronizationMode(CacheWriteSynchronizationMode.FULL_SYNC);
cacheConfig.setAtomicityMode(CacheAtomicityMode.ATOMIC);

cacheConfig.setWriteBehindEnabled(true);
cacheConfig.setWriteBehindBatchSize(512);
cacheConfig.setWriteBehindFlushSize(10240);
cacheConfig.setWriteBehindFlushFrequency(5_000);

cacheConfig.setCacheStoreFactory(new
MyCacheStoreAdapter.MyCacheStoreFactory(name));
cacheConfig.setReadThrough(true);
cacheConfig.setWriteThrough(true);



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: loadAll and removeAll from cache with custom store

[slava.koptilin]

Hi Matt,

I think you can use IgniteCache.loadCache() [1] in order to load all
key-value pairs into the cache before running any job.

> For removeAll, I'm seeing that my cache store adapter only ever gets
> called once (deleteAll) but the keys only ever have 1000 keys
I've tried this use case and it works as expected.
Could you please share cache configuration and clarify the size of your
cache before IgniteCache.size(..)?

[1] https://apacheignite.readme.io/docs/data-loading#ignitecacheloadcache

Thanks!



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: loadAll and removeAll from cache with custom store

[matt]

For the removeAll() call, I do see that after I do a loadCache((k,v) -> true)
I can delete items from the store, but my store's deleteAll() only ever gets
1k items once, deletes them from the store, and then is never called again
to delete the rest of the items in the store.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/