New setup : Node configuration :
<bean class="org.apache.ignite.configuration.IgniteConfiguration"> <property name="peerClassLoadingEnabled" value="true"/> <property name="deploymentMode" value="CONTINUOUS"/> <property name="dataStorageConfiguration"> <bean class="org.apache.ignite.configuration.DataStorageConfiguration"> <property name="defaultDataRegionConfiguration"> <bean class="org.apache.ignite.configuration.DataRegionConfiguration"> <property name="persistenceEnabled" value="true"/> </bean> </property> </bean> </property> </bean> </beans> Started 3 different nodes with this config : ./ignite.sh ../config/custom-config.xml Client : It adds data once and queries the cache cfg.setClientMode(true); try (Ignite ignite = Ignition.start(cfg)) { ignite.cluster().state(ClusterState.ACTIVE); CacheConfiguration<Integer, Department> deptCacheConfig = new CacheConfiguration<>(DEPT_CACHE); deptCacheConfig.setCacheMode(CacheMode.PARTITIONED); deptCacheConfig.setBackups(1); IgniteCache<Integer, Department> deptCache = ignite.getOrCreateCache(deptCacheConfig); Department d1 = new Department(1, "CS"); Department d2 = new Department(2, "ECE"); Department d3 = new Department(3, "CIVIL"); if(deptCache.size(CachePeekMode.ALL) == 0){ System.out.println("Adding dept data to cache"); deptCache.put(d1.getDeptId(), d1); deptCache.put(d2.getDeptId(), d2); deptCache.put(d3.getDeptId(), d3); } List<Cache.Entry<Object, Object>> depts = deptCache.query(new ScanQuery<>()).getAll(); depts.stream().forEach(entry -> System.out.println(entry.getValue())); } After inserting data, this is how cache data looks like on nodes : [image: image.png] Test : 1. first time it prints data 2. Kill node 3, query still works. I believe because node 1 and 2 are primary nodes for 3 records we inserted. 3. Restart node3, kill node 2. query gets below error. Caused by: class org.apache.ignite.internal.processors.cache.CacheInvalidStateException: Failed to execute query because cache partition has been lostParts [cacheName=deptCache, part=513] Base line log : [17:40:44] Topology snapshot [ver=11, locNode=340bad33, servers=2, clients=1, state=ACTIVE, CPUs=8, offheap=19.0GB, heap=24.0GB] [17:40:44] ^-- Baseline [id=0, size=3, online=2, offline=1] Does it mean for query to run, baseline must have all nodes up ? On Mon, Jan 31, 2022 at 4:46 PM Surinder Mehra <redni...@gmail.com> wrote: > Hi, thanks for pointing out the problem of why data is being stored on the > same machine. > The 2nd point: > As I mentioned in steps, we are getting the same exception even with > backups. > Could you please point out other problems with configuration as you > mentioned > > > On Mon, Jan 31, 2022 at 4:35 PM Stephen Darlington < > stephen.darling...@gridgain.com> wrote: > >> There’s a lot to unpack here, but there are (at least) two problems with >> your configuration. >> >> First, you should not be activating your cluster in your startup script. >> That’s an operation that needs to be completed only once, not every time >> the cluster (or node) starts. This is probably why all the data is being >> stored on a single machine. >> >> Second, you create a cache but without any backups. That means that when >> a node dies, you lose access to all the data. >> >> > On 31 Jan 2022, at 10:57, Surinder Mehra <redni...@gmail.com> wrote: >> > >> > >> > Hi Guys, >> > I observed below behavior with persistence enabled. Could you please >> check and confirm if it's a bug or I missed some configuration. >> > >> > Steps: >> > 1. Start ignite node1 and 2 with persistence enabled >> > 2. Step 1 will write few entries to ignite cache >> > 3. All entries go to one node >> > 4. Stop node containing the data >> > 5. Run a query to get data, below exception will be thrown. I guess >> this is expected because the node containing the data is down >> > 6. Restart killed node, exception shouldn’t be thrown because data is >> back in caches >> > 7. Run a query to get data, below exception will be thrown again. This >> looks weird. >> > 8. Kill node2 and try to query again, still the same exception >> > 9. Restart only the left node(data owning node). After restart, the >> query starts working again. >> > 10. I checked the cache directory under node1, it didn't have part0.bin >> ever. files start from part1 and so on. >> > 11. From the exception, it looks like it is trying to read non existent >> file >> > 12. Tried setting consistent Id for each node, still same exception. >> > >> > >> > Another observation : When backup is enabled, shouldn't the backup node >> serve the query. Why it throws same below exception when node1(primary) is >> down >> > >> > This exception is only thrown if data containing node goes down >> > >> > Caused by: class >> org.apache.ignite.internal.processors.cache.CacheInvalidStateException: >> Failed to execute query because cache partition has been lostParts >> [cacheName=deptCache, part=0] >> > >> > Node2{ >> > >> > DataStorageConfiguration storageCfg = new DataStorageConfiguration(); >> > >> storageCfg.getDefaultDataRegionConfiguration().setPersistenceEnabled(true); >> > IgniteConfiguration cfg = new IgniteConfiguration(); >> > cfg.setDataStorageConfiguration(storageCfg); >> > >> > try (Ignite ignite = Ignition.start(cfg)) { >> > ignite.cluster().state(ClusterState.ACTIVE); >> > CacheConfiguration<Integer, Department> deptCacheConfig = >> new CacheConfiguration<>(DEPT_CACHE); >> > deptCacheConfig.setCacheMode(CacheMode.PARTITIONED); >> > IgniteCache<Integer, Department> deptCache = >> ignite.getOrCreateCache(deptCacheConfig); >> > >> > Department d1 = new Department(1, "CS"); >> > Department d2 = new Department(2, "ECE"); >> > Department d3 = new Department(3, "CIVIL"); >> > >> > if(deptCache.size(CachePeekMode.ALL) == 0){ >> > System.out.println("Adding dept data to cache"); >> > deptCache.put(d1.getDeptId(), d1); >> > deptCache.put(d2.getDeptId(), d2); >> > deptCache.put(d3.getDeptId(), d3); >> > } >> > >> > } >> > >> > } >> > >> > Node1{ >> > >> > DataStorageConfiguration storageCfg = new DataStorageConfiguration(); >> > >> storageCfg.getDefaultDataRegionConfiguration().setPersistenceEnabled(true); >> > IgniteConfiguration cfg = new IgniteConfiguration(); >> > cfg.setDataStorageConfiguration(storageCfg); >> > >> > try (Ignite ignite = Ignition.start(cfg)) { >> > ignite.cluster().state(ClusterState.ACTIVE); >> > CacheConfiguration<Integer, Department> deptCacheConfig = >> new CacheConfiguration<>(DEPT_CACHE); >> > deptCacheConfig.setCacheMode(CacheMode.PARTITIONED); >> > IgniteCache<Integer, Department> deptCache = >> ignite.getOrCreateCache(deptCacheConfig); >> > >> > Department d1 = new Department(1, "CS"); >> > Department d2 = new Department(2, "ECE"); >> > Department d3 = new Department(3, "CIVIL"); >> > >> > if(deptCache.size(CachePeekMode.ALL) == 0){ >> > System.out.println("Adding dept data to cache"); >> > deptCache.put(d1.getDeptId(), d1); >> > deptCache.put(d2.getDeptId(), d2); >> > deptCache.put(d3.getDeptId(), d3); >> > } >> > >> > } >> > } >> > >> > Client{ >> > >> > IgniteConfiguration cfg = new IgniteConfiguration(); >> > cfg.setDataStorageConfiguration(storageCfg); >> > >> > try (Ignite ignite = Ignition.start(cfg)) { >> > IgniteCache<Integer, Department> deptCache = >> ignite.getOrCreateCache(DEPT_CACHE); >> > List<Cache.Entry<Object, Object>> depts = deptCache.query(new >> ScanQuery<>()).getAll(); >> > depts.stream().forEach(entry -> >> System.out.println(entry.getValue())); >> > } >> > } >> >>