2.7.6 to 2.8.1 migration issue

[Andrey Davydov]

Hello,   We test migration from 2.7.6 to 2.8.1 in our DEV environment and got problem: new code doesn’t start over old data.   Some history: Initially we had following base cache configuration on 2.7.6:       <bean id="cache-template" abstract="true" class="org.apache.ignite.configuration.CacheConfiguration">         <property name="atomicityMode" value="TRANSACTIONAL"/>         <property name="writeSynchronizationMode" value="FULL_SYNC"/>         <property name="rebalanceMode" value="ASYNC"/>         <property name="maxConcurrentAsyncOperations" value="500"/>         <property name="cacheMode" value="PARTITIONED"/>         <property name="backups" value="2"/>         <property name="dataRegionName" value="persistDataRegion"/>         <property name="storeKeepBinary" value="true"/>                <!-- Group the cache belongs to. -->         <property name="groupName" value="applicationGroup"/>         <property name="encryptionEnabled" value="false"/>           <property name="affinity">             <bean class="org.apache.ignite.cache.affinity.rendezvous.RendezvousAffinityFunction">                 <property name="excludeNeighbors" value="true"/>                 <property name="partitions" value="1024"/>             </bean>         </property>     </bean>   And create some caches using it.                   <bean parent="cache-template" class="org.apache.ignite.configuration.CacheConfiguration">                     <property name="name" value="cache1"/>                 </bean>                 <bean parent="cache-template" class="org.apache.ignite.configuration.CacheConfiguration">                     <property name="name" value="cache2"/>                 </bean>                 <bean parent="cache-template" class="org.apache.ignite.configuration.CacheConfiguration">                     <property name="indexedTypes">                         <list>                             <value>java.lang.String</value>                             <value>some.our.WellAnnotatedClass</value>                         </list>                     </property>                        </bean>     Some weeks later we update base cache configuration to use topology validator.         <bean id="base-cache-template" abstract="true" class="org.apache.ignite.configuration.CacheConfiguration">         <property name="topologyValidator" >             <bean class="our.custom.ValidatorClass">                        <property name="minimalValidTopologyNodes" value="2"/>             </bean>         </property>         <property name="sqlIndexMaxInlineSize" value="256"/>     </bean>        <bean id="cache-template" abstract="true" parent="base-cache-template" class="org.apache.ignite.configuration.CacheConfiguration">         <property name="atomicityMode" value="TRANSACTIONAL"/>         <property name="writeSynchronizationMode" value="FULL_SYNC"/>         <property name="rebalanceMode" value="ASYNC"/>         <property name="maxConcurrentAsyncOperations" value="500"/>         <property name="cacheMode" value="PARTITIONED"/>         <property name="backups" value="2"/>         <property name="dataRegionName" value="persistDataRegion"/>         <property name="storeKeepBinary" value="true"/>                <!-- Group the cache belongs to. -->         <property name="groupName" value="applicationGroup"/>         <property name="encryptionEnabled" value="false"/>           <property name="affinity">             <bean class="org.apache.ignite.cache.affinity.rendezvous.RendezvousAffinityFunction">                 <property name="excludeNeighbors" value="true"/>                 <property name="partitions" value="1024"/>             </bean>         </property>     </bean>     We run new version over old data and everything starts ok. After that we update our system and create some more caches.                     <bean parent="cache-template" class="org.apache.ignite.configuration.CacheConfiguration">                     <property name="name" value="cache4"/>                 </bean>                 <bean parent="cache-template" class="org.apache.ignite.configuration.CacheConfiguration">                     <property name="name" value="cache5"/>                 </bean>     And we also start new version of code and configuration over old data directory and everything starts ok. Now we change ignite version from 2.7.6 to 2.8.1 and try start system over old data. One node starts ok and two other fails with following error:   2020-06-18 15:37:12,598 [main] WARN   o.a.i.i.p.c.GridLocalConfigManager - Static configuration for the following caches will be ignored because a persistent cache with the same name already exist (see https://apacheignite.readme.io/docs/cache-configuration for more information): [...] 2020-06-18 15:37:12,602 [main] INFO   o.a.i.i.p.c.p.f.FilePageStoreManager - Cleanup cache stores [total=1, left=0, cleanFiles=false] 2020-06-18 15:37:12,602 [main] ERROR  o.a.i.i.IgniteKernal%dev_app - Exception during start processors, node will be stopped and close connections org.apache.ignite.IgniteCheckedException: Topology validator mismatch for caches related to the same group [groupName=applicationGroup, existingCache=cache1, existingTopologyValidator=null, startingCache=cache4, startingTopologyValidator= our.custom.ValidatorClass]       at org.apache.ignite.internal.processors.cache.GridCacheUtils.validateCacheGroupsAttributesMismatch(GridCacheUtils.java:1032) ~[ignite-core-2.8.1.jar:2.8.1]       at org.apache.ignite.internal.processors.cache.ClusterCachesInfo.validateCacheGroupConfiguration(ClusterCachesInfo.java:2305) ~[ignite-core-2.8.1.jar:2.8.1]       at org.apache.ignite.internal.processors.cache.ClusterCachesInfo.onStart(ClusterCachesInfo.java:286) ~[ignite-core-2.8.1.jar:2.8.1]   In our development environment only the first node makes write operation to cache1 (because we don’t use load balancer in dev and front-end send request to the first node) and only this node starts without problem. Two other nodes crashes. We rollback to 2.7.6 and all three nodes start ok.   If there is way to start 2.8.1 over old data?     Andrey.