Failed to execute query because cache partition has been lostParts

[Surinder Mehra]

Hi Guys,
I observed below behavior with persistence enabled. Could you please check
and confirm if it's a bug or I missed some configuration.

Steps:
1. Start ignite node1 and 2 with persistence enabled
2. Step 1 will write few entries to ignite cache
3. All entries go to one node
4. Stop node containing the data
5. Run a query to get data, below exception will be thrown. I guess this is
expected because the node containing the data is down
6. Restart killed node, exception shouldn’t be thrown because data is back
in caches
7. Run a query to get data, below exception will be thrown again. This
looks weird.
8. Kill node2 and try to query again, still the same exception
9. Restart only the left node(data owning node). After restart, the query
starts working again.
10. I checked the cache directory under node1, it didn't have part0.bin
ever. files start from part1 and so on.
11. From the exception, it looks like it is trying to read non existent file
12. Tried setting consistent Id for each node, still same exception.


Another observation : When backup is enabled, shouldn't the backup node
serve the query. Why it throws same below exception when node1(primary) is
down

This exception is only thrown if data containing node goes down

Caused by: class
org.apache.ignite.internal.processors.cache.CacheInvalidStateException:
Failed to execute query because cache partition has been lostParts
[cacheName=deptCache, part=0]

Node2{

DataStorageConfiguration storageCfg = new DataStorageConfiguration();
storageCfg.getDefaultDataRegionConfiguration().setPersistenceEnabled(true);
IgniteConfiguration cfg = new IgniteConfiguration();
cfg.setDataStorageConfiguration(storageCfg);

try (Ignite ignite = Ignition.start(cfg)) {
            ignite.cluster().state(ClusterState.ACTIVE);
            CacheConfiguration<Integer, Department> deptCacheConfig = new
CacheConfiguration<>(DEPT_CACHE);
            deptCacheConfig.setCacheMode(CacheMode.PARTITIONED);
IgniteCache<Integer, Department> deptCache =
ignite.getOrCreateCache(deptCacheConfig);

Department d1 = new Department(1, "CS");
Department d2 = new Department(2, "ECE");
Department d3 = new Department(3, "CIVIL");

if(deptCache.size(CachePeekMode.ALL) == 0){
System.out.println("Adding dept data to cache");
deptCache.put(d1.getDeptId(), d1);
deptCache.put(d2.getDeptId(), d2);
deptCache.put(d3.getDeptId(), d3);
}

}

}

Node1{

DataStorageConfiguration storageCfg = new DataStorageConfiguration();
storageCfg.getDefaultDataRegionConfiguration().setPersistenceEnabled(true);
IgniteConfiguration cfg = new IgniteConfiguration();
cfg.setDataStorageConfiguration(storageCfg);

try (Ignite ignite = Ignition.start(cfg)) {
            ignite.cluster().state(ClusterState.ACTIVE);
            CacheConfiguration<Integer, Department> deptCacheConfig = new
CacheConfiguration<>(DEPT_CACHE);
            deptCacheConfig.setCacheMode(CacheMode.PARTITIONED);
IgniteCache<Integer, Department> deptCache =
ignite.getOrCreateCache(deptCacheConfig);

Department d1 = new Department(1, "CS");
Department d2 = new Department(2, "ECE");
Department d3 = new Department(3, "CIVIL");

if(deptCache.size(CachePeekMode.ALL) == 0){
System.out.println("Adding dept data to cache");
deptCache.put(d1.getDeptId(), d1);
deptCache.put(d2.getDeptId(), d2);
deptCache.put(d3.getDeptId(), d3);
}

}
}

Client{

IgniteConfiguration cfg = new IgniteConfiguration();
cfg.setDataStorageConfiguration(storageCfg);

try (Ignite ignite = Ignition.start(cfg)) {
        IgniteCache<Integer, Department> deptCache =
ignite.getOrCreateCache(DEPT_CACHE);
        List<Cache.Entry<Object, Object>> depts = deptCache.query(new
ScanQuery<>()).getAll();
        depts.stream().forEach(entry ->
System.out.println(entry.getValue()));
    }
}