Hi Stephen, I tried it again to identify issues. This time I commented out the line which was activating the cluster everytime client started. It looks to work properly now.
Thanks for your help. On Mon, Jan 31, 2022 at 8:58 PM Surinder Mehra <redni...@gmail.com> wrote: > Hi, > Backups are configured, please recheck cacheconfig. Its set to 1, thats > why node 3 has backups from node 1 and node two as shown in pic attached to > my previous reply. > > On Mon, Jan 31, 2022, 20:46 Stephen Darlington < > stephen.darling...@gridgain.com> wrote: > >> According to the configuration you shared, you don’t have backups. >> >> I would expect to see something like: >> >> CacheConfiguration<Integer, Department> deptCacheConfig = new >> CacheConfiguration<>(DEPT_CACHE); >> deptCacheConfig.setCacheMode(CacheMode.PARTITIONED); >> *deptCacheConfig.setBackups(1);* >> IgniteCache<Integer, Department> deptCache = >> ignite.getOrCreateCache(deptCacheConfig); >> >> The default is zero backups, so loss of a single node results in lost >> partitions. >> >> Note, that since you have persistence enabled, you’ll have to drop and >> recreate the table for the change to take effect. >> >> On 31 Jan 2022, at 11:16, Surinder Mehra <redni...@gmail.com> wrote: >> >> Hi, thanks for pointing out the problem of why data is being stored on >> the same machine. >> The 2nd point: >> As I mentioned in steps, we are getting the same exception even with >> backups. >> Could you please point out other problems with configuration as you >> mentioned >> >> >> On Mon, Jan 31, 2022 at 4:35 PM Stephen Darlington < >> stephen.darling...@gridgain.com> wrote: >> >>> There’s a lot to unpack here, but there are (at least) two problems with >>> your configuration. >>> >>> First, you should not be activating your cluster in your startup script. >>> That’s an operation that needs to be completed only once, not every time >>> the cluster (or node) starts. This is probably why all the data is being >>> stored on a single machine. >>> >>> Second, you create a cache but without any backups. That means that when >>> a node dies, you lose access to all the data. >>> >>> > On 31 Jan 2022, at 10:57, Surinder Mehra <redni...@gmail.com> wrote: >>> > >>> > >>> > Hi Guys, >>> > I observed below behavior with persistence enabled. Could you please >>> check and confirm if it's a bug or I missed some configuration. >>> > >>> > Steps: >>> > 1. Start ignite node1 and 2 with persistence enabled >>> > 2. Step 1 will write few entries to ignite cache >>> > 3. All entries go to one node >>> > 4. Stop node containing the data >>> > 5. Run a query to get data, below exception will be thrown. I guess >>> this is expected because the node containing the data is down >>> > 6. Restart killed node, exception shouldn’t be thrown because data is >>> back in caches >>> > 7. Run a query to get data, below exception will be thrown again. This >>> looks weird. >>> > 8. Kill node2 and try to query again, still the same exception >>> > 9. Restart only the left node(data owning node). After restart, the >>> query starts working again. >>> > 10. I checked the cache directory under node1, it didn't have >>> part0.bin ever. files start from part1 and so on. >>> > 11. From the exception, it looks like it is trying to read non >>> existent file >>> > 12. Tried setting consistent Id for each node, still same exception. >>> > >>> > >>> > Another observation : When backup is enabled, shouldn't the backup >>> node serve the query. Why it throws same below exception when >>> node1(primary) is down >>> > >>> > This exception is only thrown if data containing node goes down >>> > >>> > Caused by: class >>> org.apache.ignite.internal.processors.cache.CacheInvalidStateException: >>> Failed to execute query because cache partition has been lostParts >>> [cacheName=deptCache, part=0] >>> > >>> > Node2{ >>> > >>> > DataStorageConfiguration storageCfg = new DataStorageConfiguration(); >>> > >>> storageCfg.getDefaultDataRegionConfiguration().setPersistenceEnabled(true); >>> > IgniteConfiguration cfg = new IgniteConfiguration(); >>> > cfg.setDataStorageConfiguration(storageCfg); >>> > >>> > try (Ignite ignite = Ignition.start(cfg)) { >>> > ignite.cluster().state(ClusterState.ACTIVE); >>> > CacheConfiguration<Integer, Department> deptCacheConfig = >>> new CacheConfiguration<>(DEPT_CACHE); >>> > deptCacheConfig.setCacheMode(CacheMode.PARTITIONED); >>> > IgniteCache<Integer, Department> deptCache = >>> ignite.getOrCreateCache(deptCacheConfig); >>> > >>> > Department d1 = new Department(1, "CS"); >>> > Department d2 = new Department(2, "ECE"); >>> > Department d3 = new Department(3, "CIVIL"); >>> > >>> > if(deptCache.size(CachePeekMode.ALL) == 0){ >>> > System.out.println("Adding dept data to cache"); >>> > deptCache.put(d1.getDeptId(), d1); >>> > deptCache.put(d2.getDeptId(), d2); >>> > deptCache.put(d3.getDeptId(), d3); >>> > } >>> > >>> > } >>> > >>> > } >>> > >>> > Node1{ >>> > >>> > DataStorageConfiguration storageCfg = new DataStorageConfiguration(); >>> > >>> storageCfg.getDefaultDataRegionConfiguration().setPersistenceEnabled(true); >>> > IgniteConfiguration cfg = new IgniteConfiguration(); >>> > cfg.setDataStorageConfiguration(storageCfg); >>> > >>> > try (Ignite ignite = Ignition.start(cfg)) { >>> > ignite.cluster().state(ClusterState.ACTIVE); >>> > CacheConfiguration<Integer, Department> deptCacheConfig = >>> new CacheConfiguration<>(DEPT_CACHE); >>> > deptCacheConfig.setCacheMode(CacheMode.PARTITIONED); >>> > IgniteCache<Integer, Department> deptCache = >>> ignite.getOrCreateCache(deptCacheConfig); >>> > >>> > Department d1 = new Department(1, "CS"); >>> > Department d2 = new Department(2, "ECE"); >>> > Department d3 = new Department(3, "CIVIL"); >>> > >>> > if(deptCache.size(CachePeekMode.ALL) == 0){ >>> > System.out.println("Adding dept data to cache"); >>> > deptCache.put(d1.getDeptId(), d1); >>> > deptCache.put(d2.getDeptId(), d2); >>> > deptCache.put(d3.getDeptId(), d3); >>> > } >>> > >>> > } >>> > } >>> > >>> > Client{ >>> > >>> > IgniteConfiguration cfg = new IgniteConfiguration(); >>> > cfg.setDataStorageConfiguration(storageCfg); >>> > >>> > try (Ignite ignite = Ignition.start(cfg)) { >>> > IgniteCache<Integer, Department> deptCache = >>> ignite.getOrCreateCache(DEPT_CACHE); >>> > List<Cache.Entry<Object, Object>> depts = deptCache.query(new >>> ScanQuery<>()).getAll(); >>> > depts.stream().forEach(entry -> >>> System.out.println(entry.getValue())); >>> > } >>> > } >>> >>> >>