[jira] [Commented] (IGNITE-20299) Creating a cache with an unknown data region name causes total unrecoverable failure of the grid
[ https://issues.apache.org/jira/browse/IGNITE-20299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17759823#comment-17759823 ] Pavel Tupitsyn commented on IGNITE-20299: - [~rpwilson] I tried this with default node configuration. While I get the same "Requested DataRegion is not configured" error (which is expected), I don't observe broken grid state. Do you have persistence enabled? Can you please provide the server node configuration? Ideally - a minimal reproducer which demonstrates the broken grid state. > Creating a cache with an unknown data region name causes total unrecoverable > failure of the grid > > > Key: IGNITE-20299 > URL: https://issues.apache.org/jira/browse/IGNITE-20299 > Project: Ignite > Issue Type: Bug > Components: cache >Affects Versions: 2.15 > Environment: Observed in: > C# client and grid running on Linux in a container > C# client and grid running on Windows > >Reporter: Raymond Wilson >Priority: Major > > Using the Ignite C# client. > > Given a running grid, having a client (and perhaps server) node in the grid > attempt to create a cache using a DataRegionName that does not exist in the > grid causes immediate failure in the client node with the following log > output. > > 2023-08-27 17:08:48,520 [44] INF [ImmutableClientServer] Completed > partition exchange [localNode=15122bd7-bf81-44e6-a548-e70dbd9334c0, > exchange=GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion > [topVer=15, minorTopVer=0], evt=NODE_FAILED, evtNode=TcpDiscoveryNode > [id=9d5ed68d-38bb-447d-aed5-189f52660716, > consistentId=9d5ed68d-38bb-447d-aed5-189f52660716, addrs=ArrayList > [127.0.0.1], sockAddrs=null, discPort=0, order=8, intOrder=8, > lastExchangeTime=1693112858024, loc=false, ver=2.15.0#20230425-sha1:f98f7f35, > isClient=true], rebalanced=false, done=true, newCrdFut=null], > topVer=AffinityTopologyVersion [topVer=15, minorTopVer=0]] > 2023-08-27 17:08:48,520 [44] INF [ImmutableClientServer] Exchange timings > [startVer=AffinityTopologyVersion [topVer=15, minorTopVer=0], > resVer=AffinityTopologyVersion [topVer=15, minorTopVer=0], stage="Waiting in > exchange queue" (14850 ms), stage="Exchange parameters initialization" (2 > ms), stage="Determine exchange type" (3 ms), stage="Exchange done" (4 ms), > stage="Total time" (14859 ms)] > 2023-08-27 17:08:48,522 [44] INF [ImmutableClientServer] Exchange longest > local stages [startVer=AffinityTopologyVersion [topVer=15, minorTopVer=0], > resVer=AffinityTopologyVersion [topVer=15, minorTopVer=0]] > 2023-08-27 17:08:48,524 [44] INF [ImmutableClientServer] Finished exchange > init [topVer=AffinityTopologyVersion [topVer=15, minorTopVer=0], crd=false] > 2023-08-27 17:08:48,525 [44] INF [ImmutableClientServer] > AffinityTopologyVersion [topVer=15, minorTopVer=0], evt=NODE_FAILED, > evtNode=9d5ed68d-38bb-447d-aed5-189f52660716, client=true] > Unhandled exception: Apache.Ignite.Core.Cache.CacheException: class > org.apache.ignite.IgniteCheckedException: Failed to complete exchange process. > ---> Apache.Ignite.Core.Common.IgniteException: Failed to complete exchange > process. > ---> Apache.Ignite.Core.Common.JavaException: javax.cache.CacheException: > class org.apache.ignite.IgniteCheckedException: Failed to complete exchange > process. > at > org.apache.ignite.internal.processors.cache.GridCacheUtils.convertToCacheException(GridCacheUtils.java:1272) > at > org.apache.ignite.internal.IgniteKernal.getOrCreateCache0(IgniteKernal.java:2278) > at > org.apache.ignite.internal.IgniteKernal.getOrCreateCache(IgniteKernal.java:2242) > at > org.apache.ignite.internal.processors.platform.PlatformProcessorImpl.processInStreamOutObject(PlatformProcessorImpl.java:643) > at > org.apache.ignite.internal.processors.platform.PlatformTargetProxyImpl.inStreamOutObject(PlatformTargetProxyImpl.java:79) > Caused by: class org.apache.ignite.IgniteCheckedException: Failed to complete > exchange process. > at > org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.createExchangeException(GridDhtPartitionsExchangeFuture.java:3709) > at > org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.sendExchangeFailureMessage(GridDhtPartitionsExchangeFuture.java:3737) > at > org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.finishExchangeOnCoordinator(GridDhtPartitionsExchangeFuture.java:3832) > at > org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.onAllReceived(GridDhtPa
[jira] [Commented] (IGNITE-20299) Creating a cache with an unknown data region name causes total unrecoverable failure of the grid
[
https://issues.apache.org/jira/browse/IGNITE-20299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17759846#comment-17759846
]
Raymond Wilson commented on IGNITE-20299:
-
Yes, we are using persistence.
This is our persistence XML file:
{noformat}
http://www.springframework.org/schema/beans";
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
xmlns:util="http://www.springframework.org/schema/util";
xsi:schemaLocation="http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd
http://www.springframework.org/schema/util
http://www.springframework.org/schema/util/spring-util.xsd";>
{noformat}
Our configuration is mostly in code. Here is the primary configuration for the
server nodes:
{noformat}
public void ConfigureTRexGrid(IgniteConfiguration cfg)
{
cfg.IgniteInstanceName = TRexGrids.ImmutableGridName();
cfg.JvmOptions = CommonJavaJVMOptions();
var configStore = DIContext.Obtain();
// Note: Set the PSN JVM heap size minimum and maximum sizes to be the
maximum defined JVM heap size for the node.
// This is to ensure the JVM always has access to the heap promised to it
so will never act to resize the heap
// This provide better performance and removes chances of surprise if the
OS cannot allocate a larger heap size block
// for other reason.
cfg.JvmMaxMemoryMb =
configStore.GetValueInt(PSNODE_IGNITE_JVM_MAX_HEAP_SIZE_MB,
DEFAULT_IGNITE_JVM_MAX_HEAP_SIZE_MB);
cfg.JvmInitialMemoryMb =
configStore.GetValueInt(PSNODE_IGNITE_JVM_MAX_HEAP_SIZE_MB,
DEFAULT_IGNITE_JVM_MAX_HEAP_SIZE_MB);
cfg.UserAttributes = new Dictionary
{
{ "Owner", TRexGrids.ImmutableGridName() }
};
// Configure the Ignite persistence layer to store our data
cfg.DataStorageConfiguration = new DataStorageConfiguration
{
WalMode = WalMode.Fsync,
PageSize = IgniteDataRegionPageSize(),
StoragePath =
Path.Combine(TRexServerConfig.PersistentCacheStoreLocation, "Immutable",
"Persistence"),
WalPath = Path.Combine(TRexServerConfig.PersistentCacheStoreLocation,
"Immutable", "WalStore"),
WalArchivePath =
Path.Combine(TRexServerConfig.PersistentCacheStoreLocation, "Immutable",
"WalArchive"),
WalSegmentSize = 512 * 1024 * 1024, // Set the WalSegmentSize to 512Mb
to better support high write loads (can be set to max 2Gb)
MaxWalArchiveSize = (long)10 * 512 * 1024 * 1024, // Ensure there are
10 segments in the WAL archive at the defined segment size
CheckpointThreads =
configStore.GetValueInt(IGNITE_NUMBER_OF_CHECKPOINTING_THREADS,
DEFAULT_IGNITE_NUMBER_OF_CHECKPOINTING_THREADS),
CheckpointFrequency =
TimeSpan.FromSeconds(configStore.GetValueInt(IGNITE_CHECKPOINTING_INTERVAL_SECONDS,
DEFAULT_IGNITE_CHECKPOINTING_INTERVAL_SECONDS)),
DefaultDataRegionConfiguration = new DataRegionConfiguration
{
Name = DataRegions.DEFAULT_IMMUTABLE_DATA_REGION_NAME,
InitialSize =
configStore.GetValueLong(IMMUTABLE_DATA_REGION_INITIAL_SIZE_MB,
DEFAULT_IMMUTABLE_DATA_REGION_INITIAL_SIZE_MB) * 1024 * 1024,
MaxSize = configStore.GetValueLong(IMMUTABLE_DATA_REGION_MAX_SIZE_MB,
DEFAULT_IMMUTABLE_DATA_REGION_MAX_SIZE_MB) * 1024 * 1024,
PersistenceEnabled = true
}
};
Log.LogInformation($"cfg.DataStorageConfiguration.StoragePath={cfg.DataStorageConfiguration.StoragePath}");
Log.LogInformation($"cfg.DataStorageConfiguration.WalArchivePath={cfg.DataStorageConfiguration.WalArchivePath}");
Log.LogInformation($"cfg.DataStorageConfiguration.WalPath={cfg.DataStorageConfiguration.WalPath}");
if (!bool.TryParse(Environment.GetEnvironmentVariable("IS_KUBERNETES"),
out var isKubernetes))
{
Log.LogWarning($"Failed to parse the value of the 'IS_KUBERNETES'
environment variable as a bool. Value is
{Environment.GetEnvironmentVariable("IS_KUBERNETES")}. Defaulting to true");
}
cfg = isKubernetes ? SetKubernetesIgniteConfiguration(cfg) :
SetLocalIgniteConfiguration(cfg);
cfg.WorkDirectory =
Path.Combine(TRexServerConfig.PersistentCacheStoreLocation, "Immutable");
cfg.Logger = new TRexIgniteLogger(configStore,
Logger.CreateLogger("ImmutableCacheComputeServer"));
// Set an Ignite metrics heartbeat
cfg.MetricsLogFrequency = new TimeSpan(0, 0, 0,
configStore.GetValueInt(IGNITE_HEARTBEAT_FREQUENCY_SECONDS,
DEFAULT_IGNITE_HEARTBEAT_FREQUENCY_SECONDS));
cfg.PublicThreadPoolSize =
configStore.GetValueInt(IGNITE_PUBLIC_THREAD_POOL_SIZE,
DEFAULT_IGNITE_PUBLIC_THREAD_POOL_SIZE);
cfg.SystemThreadPoolSize =
configStore.GetValueInt(IGNITE_SYSTEM_THREAD_POOL_SIZE,
DEFAULT_IGNIT
[jira] [Commented] (IGNITE-20299) Creating a cache with an unknown data region name causes total unrecoverable failure of the grid
[ https://issues.apache.org/jira/browse/IGNITE-20299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17759859#comment-17759859 ] Raymond Wilson commented on IGNITE-20299: - Just a quick note as to how I reproduce it: I create a new instance of our Ignite grid on my development machine (and as it is new it has no data within it). Running a node which then attempts to create the cache against a data region that does not exist causes the issue. In terms of my local system, there is only a single server node and a small collection of client nodes. > Creating a cache with an unknown data region name causes total unrecoverable > failure of the grid > > > Key: IGNITE-20299 > URL: https://issues.apache.org/jira/browse/IGNITE-20299 > Project: Ignite > Issue Type: Bug > Components: cache >Affects Versions: 2.15 > Environment: Observed in: > C# client and grid running on Linux in a container > C# client and grid running on Windows > >Reporter: Raymond Wilson >Priority: Major > > Using the Ignite C# client. > > Given a running grid, having a client (and perhaps server) node in the grid > attempt to create a cache using a DataRegionName that does not exist in the > grid causes immediate failure in the client node with the following log > output. > > 2023-08-27 17:08:48,520 [44] INF [ImmutableClientServer] Completed > partition exchange [localNode=15122bd7-bf81-44e6-a548-e70dbd9334c0, > exchange=GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion > [topVer=15, minorTopVer=0], evt=NODE_FAILED, evtNode=TcpDiscoveryNode > [id=9d5ed68d-38bb-447d-aed5-189f52660716, > consistentId=9d5ed68d-38bb-447d-aed5-189f52660716, addrs=ArrayList > [127.0.0.1], sockAddrs=null, discPort=0, order=8, intOrder=8, > lastExchangeTime=1693112858024, loc=false, ver=2.15.0#20230425-sha1:f98f7f35, > isClient=true], rebalanced=false, done=true, newCrdFut=null], > topVer=AffinityTopologyVersion [topVer=15, minorTopVer=0]] > 2023-08-27 17:08:48,520 [44] INF [ImmutableClientServer] Exchange timings > [startVer=AffinityTopologyVersion [topVer=15, minorTopVer=0], > resVer=AffinityTopologyVersion [topVer=15, minorTopVer=0], stage="Waiting in > exchange queue" (14850 ms), stage="Exchange parameters initialization" (2 > ms), stage="Determine exchange type" (3 ms), stage="Exchange done" (4 ms), > stage="Total time" (14859 ms)] > 2023-08-27 17:08:48,522 [44] INF [ImmutableClientServer] Exchange longest > local stages [startVer=AffinityTopologyVersion [topVer=15, minorTopVer=0], > resVer=AffinityTopologyVersion [topVer=15, minorTopVer=0]] > 2023-08-27 17:08:48,524 [44] INF [ImmutableClientServer] Finished exchange > init [topVer=AffinityTopologyVersion [topVer=15, minorTopVer=0], crd=false] > 2023-08-27 17:08:48,525 [44] INF [ImmutableClientServer] > AffinityTopologyVersion [topVer=15, minorTopVer=0], evt=NODE_FAILED, > evtNode=9d5ed68d-38bb-447d-aed5-189f52660716, client=true] > Unhandled exception: Apache.Ignite.Core.Cache.CacheException: class > org.apache.ignite.IgniteCheckedException: Failed to complete exchange process. > ---> Apache.Ignite.Core.Common.IgniteException: Failed to complete exchange > process. > ---> Apache.Ignite.Core.Common.JavaException: javax.cache.CacheException: > class org.apache.ignite.IgniteCheckedException: Failed to complete exchange > process. > at > org.apache.ignite.internal.processors.cache.GridCacheUtils.convertToCacheException(GridCacheUtils.java:1272) > at > org.apache.ignite.internal.IgniteKernal.getOrCreateCache0(IgniteKernal.java:2278) > at > org.apache.ignite.internal.IgniteKernal.getOrCreateCache(IgniteKernal.java:2242) > at > org.apache.ignite.internal.processors.platform.PlatformProcessorImpl.processInStreamOutObject(PlatformProcessorImpl.java:643) > at > org.apache.ignite.internal.processors.platform.PlatformTargetProxyImpl.inStreamOutObject(PlatformTargetProxyImpl.java:79) > Caused by: class org.apache.ignite.IgniteCheckedException: Failed to complete > exchange process. > at > org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.createExchangeException(GridDhtPartitionsExchangeFuture.java:3709) > at > org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.sendExchangeFailureMessage(GridDhtPartitionsExchangeFuture.java:3737) > at > org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.finishExchangeOnCoordinator(GridDhtPartitionsExchangeFuture.java:3832) > at > org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsEx
[jira] [Commented] (IGNITE-20299) Creating a cache with an unknown data region name causes total unrecoverable failure of the grid
[
https://issues.apache.org/jira/browse/IGNITE-20299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17759863#comment-17759863
]
Raymond Wilson commented on IGNITE-20299:
-
[~ptupitsyn]
I searched for the error you mentioned "Requested DataRegion is not
configured", and I can see this emitted on each of the server nodes in the dev
environment.
The (long) log line is here:
{noformat}
Exchange future: GridDhtPartitionsExchangeFuture
[firstDiscoEvt=DiscoveryCustomEvent [customMsg=ChangeGlobalStateMessage
[id=e6c78a93a81-145135fd-0ac4-4457-8308-e9798dfa4ee6,
reqId=ff5e4d1b-359d-4cda-94c3-8115da3b1dc1,
initiatingNodeId=4d44108f-cd96-4953-94db-6365f998a91b, state=ACTIVE,
baselineTopology=BaselineTopology [id=0, branchingHash=264269663,
branchingType='Cluster activation',
baselineNodes=[3fb67a8d-b805-4dbd-b400-a02077d827de,
ce2ff0c2-46b7-4082-97ea-847514672d06, b9e51542-6d1c-4535-8a02-ced986e53878,
46f8195d-000f-4d09-8ab9-3bd5246e9109]], forceChangeBaselineTopology=false,
timestamp=1693184516920, forceDeactivation=true],
affTopVer=AffinityTopologyVersion [topVer=12, minorTopVer=1],
super=DiscoveryEvent [evtNode=TcpDiscoveryNode
[id=4d44108f-cd96-4953-94db-6365f998a91b,
consistentId=3fb67a8d-b805-4dbd-b400-a02077d827de, addrs=ArrayList [],
sockAddrs=HashSet [], discPort=47500, order=5, intOrder=5,
lastExchangeTime=1693184505010, loc=false, ver=2.15.0#20230425-sha1:f98f7f35,
isClient=false], topVer=12, msgTemplate=null,
span=org.apache.ignite.internal.processors.tracing.NoopSpan@1b32fd2b,
nodeId8=b1708bc6, msg=null, type=DISCOVERY_CUSTOM_EVT, tstamp=1693184518126]],
crd=TcpDiscoveryNode [id=b1708bc6-8798-4d16-b7ee-7cdf2535944d,
consistentId=b9e51542-6d1c-4535-8a02-ced986e53878, addrs=ArrayList [],
sockAddrs=HashSet [], discPort=47500, order=1, intOrder=1,
lastExchangeTime=1693184529771, loc=true, ver=2.15.0#20230425-sha1:f98f7f35,
isClient=false], exchId=GridDhtPartitionExchangeId
[topVer=AffinityTopologyVersion [topVer=12, minorTopVer=1],
discoEvt=DiscoveryCustomEvent [customMsg=ChangeGlobalStateMessage
[id=e6c78a93a81-145135fd-0ac4-4457-8308-e9798dfa4ee6,
reqId=ff5e4d1b-359d-4cda-94c3-8115da3b1dc1,
initiatingNodeId=4d44108f-cd96-4953-94db-6365f998a91b, state=ACTIVE,
baselineTopology=BaselineTopology [id=0, branchingHash=264269663,
branchingType='Cluster activation',
baselineNodes=[3fb67a8d-b805-4dbd-b400-a02077d827de,
ce2ff0c2-46b7-4082-97ea-847514672d06, b9e51542-6d1c-4535-8a02-ced986e53878,
46f8195d-000f-4d09-8ab9-3bd5246e9109]], forceChangeBaselineTopology=false,
timestamp=1693184516920, forceDeactivation=true],
affTopVer=AffinityTopologyVersion [topVer=12, minorTopVer=1],
super=DiscoveryEvent [evtNode=TcpDiscoveryNode
[id=4d44108f-cd96-4953-94db-6365f998a91b,
consistentId=3fb67a8d-b805-4dbd-b400-a02077d827de, addrs=ArrayList [],
sockAddrs=HashSet [], discPort=47500, order=5, intOrder=5,
lastExchangeTime=1693184505010, loc=false, ver=2.15.0#20230425-sha1:f98f7f35,
isClient=false], topVer=12, msgTemplate=null,
span=org.apache.ignite.internal.processors.tracing.NoopSpan@1b32fd2b,
nodeId8=b1708bc6, msg=null, type=DISCOVERY_CUSTOM_EVT, tstamp=1693184518126]],
nodeId=4d44108f, evt=DISCOVERY_CUSTOM_EVT], added=true, exchangeType=ALL,
initFut=GridFutureAdapter [ignoreInterrupts=false, state=DONE, res=true,
hash=1834334095], init=true, lastVer=GridCacheVersion [topVer=0,
order=1693184474107, nodeOrder=0, dataCenterId=0],
partReleaseFut=PartitionReleaseFuture [topVer=AffinityTopologyVersion
[topVer=12, minorTopVer=1], totalFutures=5, futures=[ExplicitLockReleaseFuture
[topVer=AffinityTopologyVersion [topVer=12, minorTopVer=1], totalFutures=0,
futures=[]], AtomicUpdateReleaseFuture [topVer=AffinityTopologyVersion
[topVer=12, minorTopVer=1], totalFutures=0, futures=[]],
DataStreamerReleaseFuture [topVer=AffinityTopologyVersion [topVer=12,
minorTopVer=1], totalFutures=0, futures=[]], LocalTxReleaseFuture
[topVer=AffinityTopologyVersion [topVer=12, minorTopVer=1], totalFutures=0,
futures=[]], AllTxReleaseFuture [topVer=AffinityTopologyVersion [topVer=12,
minorTopVer=1], totalFutures=1, futures=[RemoteTxReleaseFuture
[topVer=AffinityTopologyVersion [topVer=12, minorTopVer=1], totalFutures=0,
futures=[]], exchActions=ExchangeActions [startCaches=[ignite-sys-cache,
SiteModelMetadata, SiteModelChangeBufferQueue, NonSpatial-Immutable,
SiteModelChangeMaps, Spatial-SubGridSegment-Immutable,
ProductionDataExistenceMap-Immutable, DesignTopologyExistenceMaps,
SiteModels-Immutable, Spatial-SubGridDirectory-Immutable], stopCaches=null,
startGrps=[DesignTopologyExistenceMaps, ignite-sys-cache,
Spatial-SubGridSegment-Immutable, NonSpatial-Immutable, SiteModelChangeMaps,
SiteModelMetadata, ProductionDataExistenceMap-Immutable,
Spatial-SubGridDirectory-Immutable, SiteModelChangeBufferQueue,
SiteModels-Immutable], stopGrps=[], resetParts=null,
fina
[jira] [Commented] (IGNITE-20299) Creating a cache with an unknown data region name causes total unrecoverable failure of the grid
[
https://issues.apache.org/jira/browse/IGNITE-20299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17760119#comment-17760119
]
Raymond Wilson commented on IGNITE-20299:
-
[~ptupitsyn]
As some additional background information, the process that executed the cache
creation command hosts two separate Ignite client node JVMs where each is a
client node of a separate Ignite grid.
The data region in question is a valid data region in one of the grids, but not
in the one that was asked to create the cache.
The code creating the cache is pretty simple:
{noformat}
public SiteModelMetadataManager(StorageMutability mutability)
{
// Obtain the ignite reference for the primary grid orientation of
SiteModels
var ignite = DIContext.Obtain()?.Grid(mutability);
_metaDataCache = ignite?.GetOrCreateCache(ConfigureCache());
if (_metaDataCache == null)
throw new TRexException($"Failed to get or create Ignite cache
{TRexCaches.SiteModelMetadataCacheName()}, ignite reference is {ignite}");
}
{noformat}
We do not observe the message "Failed to get or create Ignite cache" in our
logs, implying GetOrCreate cache threw an exception we are not capturing.
The configuration for the cache in question is set up by this code:
{noformat}
private CacheConfiguration ConfigureCache()
{
return new CacheConfiguration
{
Name = TRexCaches.SiteModelMetadataCacheName(),
KeepBinaryInStore = true,
CacheMode = CacheMode.Replicated,
DataRegionName = DataRegions.MUTABLE_NONSPATIAL_DATA_REGION
};
}
{noformat}
> Creating a cache with an unknown data region name causes total unrecoverable
> failure of the grid
>
>
> Key: IGNITE-20299
> URL: https://issues.apache.org/jira/browse/IGNITE-20299
> Project: Ignite
> Issue Type: Bug
> Components: cache
>Affects Versions: 2.15
> Environment: Observed in:
> C# client and grid running on Linux in a container
> C# client and grid running on Windows
>
>Reporter: Raymond Wilson
>Priority: Major
>
> Using the Ignite C# client.
>
> Given a running grid, having a client (and perhaps server) node in the grid
> attempt to create a cache using a DataRegionName that does not exist in the
> grid causes immediate failure in the client node with the following log
> output.
>
> 2023-08-27 17:08:48,520 [44] INF [ImmutableClientServer] Completed
> partition exchange [localNode=15122bd7-bf81-44e6-a548-e70dbd9334c0,
> exchange=GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion
> [topVer=15, minorTopVer=0], evt=NODE_FAILED, evtNode=TcpDiscoveryNode
> [id=9d5ed68d-38bb-447d-aed5-189f52660716,
> consistentId=9d5ed68d-38bb-447d-aed5-189f52660716, addrs=ArrayList
> [127.0.0.1], sockAddrs=null, discPort=0, order=8, intOrder=8,
> lastExchangeTime=1693112858024, loc=false, ver=2.15.0#20230425-sha1:f98f7f35,
> isClient=true], rebalanced=false, done=true, newCrdFut=null],
> topVer=AffinityTopologyVersion [topVer=15, minorTopVer=0]]
> 2023-08-27 17:08:48,520 [44] INF [ImmutableClientServer] Exchange timings
> [startVer=AffinityTopologyVersion [topVer=15, minorTopVer=0],
> resVer=AffinityTopologyVersion [topVer=15, minorTopVer=0], stage="Waiting in
> exchange queue" (14850 ms), stage="Exchange parameters initialization" (2
> ms), stage="Determine exchange type" (3 ms), stage="Exchange done" (4 ms),
> stage="Total time" (14859 ms)]
> 2023-08-27 17:08:48,522 [44] INF [ImmutableClientServer] Exchange longest
> local stages [startVer=AffinityTopologyVersion [topVer=15, minorTopVer=0],
> resVer=AffinityTopologyVersion [topVer=15, minorTopVer=0]]
> 2023-08-27 17:08:48,524 [44] INF [ImmutableClientServer] Finished exchange
> init [topVer=AffinityTopologyVersion [topVer=15, minorTopVer=0], crd=false]
> 2023-08-27 17:08:48,525 [44] INF [ImmutableClientServer]
> AffinityTopologyVersion [topVer=15, minorTopVer=0], evt=NODE_FAILED,
> evtNode=9d5ed68d-38bb-447d-aed5-189f52660716, client=true]
> Unhandled exception: Apache.Ignite.Core.Cache.CacheException: class
> org.apache.ignite.IgniteCheckedException: Failed to complete exchange process.
> ---> Apache.Ignite.Core.Common.IgniteException: Failed to complete exchange
> process.
> ---> Apache.Ignite.Core.Common.JavaException: javax.cache.CacheException:
> class org.apache.ignite.IgniteCheckedException: Failed to complete exchange
> process.
> at
> org.apache.ignite.internal.processors.cache.GridCacheUtils.convertToCacheException(GridCacheUtils.java:1272)
> at
> org.apache.ignite.internal.IgniteKernal.getOrCreateCache0(IgniteKernal.java:2278)
> at
> org.apache.ignite.internal.IgniteKernal.getOrCreateCache(IgniteKernal.ja
[jira] [Commented] (IGNITE-20299) Creating a cache with an unknown data region name causes total unrecoverable failure of the grid
[ https://issues.apache.org/jira/browse/IGNITE-20299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17760135#comment-17760135 ] Raymond Wilson commented on IGNITE-20299: - Wrapping an exception trapper around GetOrCreateCache() call shows that there is an exception thrown, which is: "Apache.Ignite.Core.Cache.CacheException: class org.apache.ignite.IgniteCheckedException: Failed to complete exchange process. ---> Apache.Ignite.Core.Common.IgniteException: Failed to complete exchange process. However, none of the information in the exception indicates that the "Requested DataRegion is not configured" > Creating a cache with an unknown data region name causes total unrecoverable > failure of the grid > > > Key: IGNITE-20299 > URL: https://issues.apache.org/jira/browse/IGNITE-20299 > Project: Ignite > Issue Type: Bug > Components: cache >Affects Versions: 2.15 > Environment: Observed in: > C# client and grid running on Linux in a container > C# client and grid running on Windows > >Reporter: Raymond Wilson >Priority: Major > > Using the Ignite C# client. > > Given a running grid, having a client (and perhaps server) node in the grid > attempt to create a cache using a DataRegionName that does not exist in the > grid causes immediate failure in the client node with the following log > output. > > 2023-08-27 17:08:48,520 [44] INF [ImmutableClientServer] Completed > partition exchange [localNode=15122bd7-bf81-44e6-a548-e70dbd9334c0, > exchange=GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion > [topVer=15, minorTopVer=0], evt=NODE_FAILED, evtNode=TcpDiscoveryNode > [id=9d5ed68d-38bb-447d-aed5-189f52660716, > consistentId=9d5ed68d-38bb-447d-aed5-189f52660716, addrs=ArrayList > [127.0.0.1], sockAddrs=null, discPort=0, order=8, intOrder=8, > lastExchangeTime=1693112858024, loc=false, ver=2.15.0#20230425-sha1:f98f7f35, > isClient=true], rebalanced=false, done=true, newCrdFut=null], > topVer=AffinityTopologyVersion [topVer=15, minorTopVer=0]] > 2023-08-27 17:08:48,520 [44] INF [ImmutableClientServer] Exchange timings > [startVer=AffinityTopologyVersion [topVer=15, minorTopVer=0], > resVer=AffinityTopologyVersion [topVer=15, minorTopVer=0], stage="Waiting in > exchange queue" (14850 ms), stage="Exchange parameters initialization" (2 > ms), stage="Determine exchange type" (3 ms), stage="Exchange done" (4 ms), > stage="Total time" (14859 ms)] > 2023-08-27 17:08:48,522 [44] INF [ImmutableClientServer] Exchange longest > local stages [startVer=AffinityTopologyVersion [topVer=15, minorTopVer=0], > resVer=AffinityTopologyVersion [topVer=15, minorTopVer=0]] > 2023-08-27 17:08:48,524 [44] INF [ImmutableClientServer] Finished exchange > init [topVer=AffinityTopologyVersion [topVer=15, minorTopVer=0], crd=false] > 2023-08-27 17:08:48,525 [44] INF [ImmutableClientServer] > AffinityTopologyVersion [topVer=15, minorTopVer=0], evt=NODE_FAILED, > evtNode=9d5ed68d-38bb-447d-aed5-189f52660716, client=true] > Unhandled exception: Apache.Ignite.Core.Cache.CacheException: class > org.apache.ignite.IgniteCheckedException: Failed to complete exchange process. > ---> Apache.Ignite.Core.Common.IgniteException: Failed to complete exchange > process. > ---> Apache.Ignite.Core.Common.JavaException: javax.cache.CacheException: > class org.apache.ignite.IgniteCheckedException: Failed to complete exchange > process. > at > org.apache.ignite.internal.processors.cache.GridCacheUtils.convertToCacheException(GridCacheUtils.java:1272) > at > org.apache.ignite.internal.IgniteKernal.getOrCreateCache0(IgniteKernal.java:2278) > at > org.apache.ignite.internal.IgniteKernal.getOrCreateCache(IgniteKernal.java:2242) > at > org.apache.ignite.internal.processors.platform.PlatformProcessorImpl.processInStreamOutObject(PlatformProcessorImpl.java:643) > at > org.apache.ignite.internal.processors.platform.PlatformTargetProxyImpl.inStreamOutObject(PlatformTargetProxyImpl.java:79) > Caused by: class org.apache.ignite.IgniteCheckedException: Failed to complete > exchange process. > at > org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.createExchangeException(GridDhtPartitionsExchangeFuture.java:3709) > at > org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.sendExchangeFailureMessage(GridDhtPartitionsExchangeFuture.java:3737) > at > org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.finishExchangeOnCoordinator(GridDhtPartitionsExchangeFuture.java:3832) > at > org.apache.ignite.internal.pr
[jira] [Commented] (IGNITE-20299) Creating a cache with an unknown data region name causes total unrecoverable failure of the grid
[ https://issues.apache.org/jira/browse/IGNITE-20299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17760136#comment-17760136 ] Raymond Wilson commented on IGNITE-20299: - I think persistence is an important for this issue as it is on restart that the grid complains that it cannot (a) start the incorrectly created cache (which raises the question as to why it is still known about if creation of it was unsuccessful) and (b) fails to initialise the persisted caches. The cache folder for the incorrectly create cache is also constructed which indicates that the grid has somehow accepted the cache as a valid new cache while at the same time throwing the exchange process exception, all of which indicates the validation of the parameters for the new cache is not enforcing the requirement for the data region to be known. > Creating a cache with an unknown data region name causes total unrecoverable > failure of the grid > > > Key: IGNITE-20299 > URL: https://issues.apache.org/jira/browse/IGNITE-20299 > Project: Ignite > Issue Type: Bug > Components: cache >Affects Versions: 2.15 > Environment: Observed in: > C# client and grid running on Linux in a container > C# client and grid running on Windows > >Reporter: Raymond Wilson >Priority: Major > > Using the Ignite C# client. > > Given a running grid, having a client (and perhaps server) node in the grid > attempt to create a cache using a DataRegionName that does not exist in the > grid causes immediate failure in the client node with the following log > output. > > 2023-08-27 17:08:48,520 [44] INF [ImmutableClientServer] Completed > partition exchange [localNode=15122bd7-bf81-44e6-a548-e70dbd9334c0, > exchange=GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion > [topVer=15, minorTopVer=0], evt=NODE_FAILED, evtNode=TcpDiscoveryNode > [id=9d5ed68d-38bb-447d-aed5-189f52660716, > consistentId=9d5ed68d-38bb-447d-aed5-189f52660716, addrs=ArrayList > [127.0.0.1], sockAddrs=null, discPort=0, order=8, intOrder=8, > lastExchangeTime=1693112858024, loc=false, ver=2.15.0#20230425-sha1:f98f7f35, > isClient=true], rebalanced=false, done=true, newCrdFut=null], > topVer=AffinityTopologyVersion [topVer=15, minorTopVer=0]] > 2023-08-27 17:08:48,520 [44] INF [ImmutableClientServer] Exchange timings > [startVer=AffinityTopologyVersion [topVer=15, minorTopVer=0], > resVer=AffinityTopologyVersion [topVer=15, minorTopVer=0], stage="Waiting in > exchange queue" (14850 ms), stage="Exchange parameters initialization" (2 > ms), stage="Determine exchange type" (3 ms), stage="Exchange done" (4 ms), > stage="Total time" (14859 ms)] > 2023-08-27 17:08:48,522 [44] INF [ImmutableClientServer] Exchange longest > local stages [startVer=AffinityTopologyVersion [topVer=15, minorTopVer=0], > resVer=AffinityTopologyVersion [topVer=15, minorTopVer=0]] > 2023-08-27 17:08:48,524 [44] INF [ImmutableClientServer] Finished exchange > init [topVer=AffinityTopologyVersion [topVer=15, minorTopVer=0], crd=false] > 2023-08-27 17:08:48,525 [44] INF [ImmutableClientServer] > AffinityTopologyVersion [topVer=15, minorTopVer=0], evt=NODE_FAILED, > evtNode=9d5ed68d-38bb-447d-aed5-189f52660716, client=true] > Unhandled exception: Apache.Ignite.Core.Cache.CacheException: class > org.apache.ignite.IgniteCheckedException: Failed to complete exchange process. > ---> Apache.Ignite.Core.Common.IgniteException: Failed to complete exchange > process. > ---> Apache.Ignite.Core.Common.JavaException: javax.cache.CacheException: > class org.apache.ignite.IgniteCheckedException: Failed to complete exchange > process. > at > org.apache.ignite.internal.processors.cache.GridCacheUtils.convertToCacheException(GridCacheUtils.java:1272) > at > org.apache.ignite.internal.IgniteKernal.getOrCreateCache0(IgniteKernal.java:2278) > at > org.apache.ignite.internal.IgniteKernal.getOrCreateCache(IgniteKernal.java:2242) > at > org.apache.ignite.internal.processors.platform.PlatformProcessorImpl.processInStreamOutObject(PlatformProcessorImpl.java:643) > at > org.apache.ignite.internal.processors.platform.PlatformTargetProxyImpl.inStreamOutObject(PlatformTargetProxyImpl.java:79) > Caused by: class org.apache.ignite.IgniteCheckedException: Failed to complete > exchange process. > at > org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.createExchangeException(GridDhtPartitionsExchangeFuture.java:3709) > at > org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.sendExchangeFailureMessage(GridDhtPartitionsExchangeFuture.java:3737) > at >
[jira] [Commented] (IGNITE-20299) Creating a cache with an unknown data region name causes total unrecoverable failure of the grid
[
https://issues.apache.org/jira/browse/IGNITE-20299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17760164#comment-17760164
]
Raymond Wilson commented on IGNITE-20299:
-
[~ptupitsyn]
I have managed to create a simple reproducer. Code below.
Running it once will fail with the exchange future exception. Running it a
second time will see it fail to start the bad cache and fails to run.
{noformat}
using Apache.Ignite.Core;
using Apache.Ignite.Core.Cache;
using Apache.Ignite.Core.Cache.Configuration;
using Apache.Ignite.Core.Configuration;
namespace BadCacheCreationReproducer;
static class Program
{
public static ICache cacheServer;
static void Main()
{
// Make the server
var cfgServer = new IgniteConfiguration
{
IgniteInstanceName = "Server",
JvmMaxMemoryMb = 1024,
JvmInitialMemoryMb = 512,
DataStorageConfiguration = new DataStorageConfiguration
{
WalMode = WalMode.Fsync,
PageSize = 4 * 1024,
StoragePath = Path.Combine(@"c:\temp", "BadCacheCreationReproducer",
"Persistence"),
WalArchivePath = Path.Combine(@"c:\temp", "BadCacheCreationReproducer",
"WalArchive"),
WalPath = Path.Combine(@"c:\temp", "BadCacheCreationReproducer",
"WalStore"),
DefaultDataRegionConfiguration = new DataRegionConfiguration
{
Name = "Default",
InitialSize = 64 * 1024 * 1024,
MaxSize = 2L * 1024 * 1024 * 1024,
PersistenceEnabled = true
}
},
JvmOptions = new List() { "-DIGNITE_QUIET=false",
"-Djava.net.preferIPv4Stack=true", "-XX:+UseG1GC" },
WorkDirectory = Path.Combine(@"c:\temp", "BadCacheCreationReproducer")
};
var igniteServer = Ignition.Start(cfgServer);
igniteServer.GetCluster().SetActive(true);
// Attempt to create the bad cache
var cacheCfgServer = new CacheConfiguration { Name = "ABadCache",
KeepBinaryInStore = true, CacheMode = CacheMode.Partitioned, DataRegionName =
"NotDefault" };
try
{
cacheServer = igniteServer.GetOrCreateCache(cacheCfgServer);
}
catch (Exception ex)
{
Console.WriteLine($"Exception!: {ex}");
}
Console.WriteLine("Completed");
}
}
{noformat}
> Creating a cache with an unknown data region name causes total unrecoverable
> failure of the grid
>
>
> Key: IGNITE-20299
> URL: https://issues.apache.org/jira/browse/IGNITE-20299
> Project: Ignite
> Issue Type: Bug
> Components: cache
>Affects Versions: 2.15
> Environment: Observed in:
> C# client and grid running on Linux in a container
> C# client and grid running on Windows
>
>Reporter: Raymond Wilson
>Priority: Major
>
> Using the Ignite C# client.
>
> Given a running grid, having a client (and perhaps server) node in the grid
> attempt to create a cache using a DataRegionName that does not exist in the
> grid causes immediate failure in the client node with the following log
> output.
>
> 2023-08-27 17:08:48,520 [44] INF [ImmutableClientServer] Completed
> partition exchange [localNode=15122bd7-bf81-44e6-a548-e70dbd9334c0,
> exchange=GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion
> [topVer=15, minorTopVer=0], evt=NODE_FAILED, evtNode=TcpDiscoveryNode
> [id=9d5ed68d-38bb-447d-aed5-189f52660716,
> consistentId=9d5ed68d-38bb-447d-aed5-189f52660716, addrs=ArrayList
> [127.0.0.1], sockAddrs=null, discPort=0, order=8, intOrder=8,
> lastExchangeTime=1693112858024, loc=false, ver=2.15.0#20230425-sha1:f98f7f35,
> isClient=true], rebalanced=false, done=true, newCrdFut=null],
> topVer=AffinityTopologyVersion [topVer=15, minorTopVer=0]]
> 2023-08-27 17:08:48,520 [44] INF [ImmutableClientServer] Exchange timings
> [startVer=AffinityTopologyVersion [topVer=15, minorTopVer=0],
> resVer=AffinityTopologyVersion [topVer=15, minorTopVer=0], stage="Waiting in
> exchange queue" (14850 ms), stage="Exchange parameters initialization" (2
> ms), stage="Determine exchange type" (3 ms), stage="Exchange done" (4 ms),
> stage="Total time" (14859 ms)]
> 2023-08-27 17:08:48,522 [44] INF [ImmutableClientServer] Exchange longest
> local stages [startVer=AffinityTopologyVersion [topVer=15, minorTopVer=0],
> resVer=AffinityTopologyVersion [topVer=15, minorTopVer=0]]
> 2023-08-27 17:08:48,524 [44] INF [ImmutableClientServer] Finished exchange
> init [topVer=AffinityTopologyVersion [topVer=15, minorTopVer=0], crd=false]
> 2023-08-27 17:08:48,525 [44] INF [ImmutableClientServer]
> AffinityTopologyVersion [topVer=15, minorTopVer=0], evt=NODE_FAILED,
> evtNode=9d5ed68d-38bb-447d-aed5-189f52660716, client=true]
> Unhandled exception: Apache.Ignite.Core.Cache.CacheException: class
> org.apache.ignite.IgniteCheckedExc
[jira] [Commented] (IGNITE-20299) Creating a cache with an unknown data region name causes total unrecoverable failure of the grid
[ https://issues.apache.org/jira/browse/IGNITE-20299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17760294#comment-17760294 ] Pavel Tupitsyn commented on IGNITE-20299: - With PersistenceEnabled=true it still does not reproduce for me. > Creating a cache with an unknown data region name causes total unrecoverable > failure of the grid > > > Key: IGNITE-20299 > URL: https://issues.apache.org/jira/browse/IGNITE-20299 > Project: Ignite > Issue Type: Bug > Components: cache >Affects Versions: 2.15 > Environment: Observed in: > C# client and grid running on Linux in a container > C# client and grid running on Windows > >Reporter: Raymond Wilson >Priority: Major > > Using the Ignite C# client. > > Given a running grid, having a client (and perhaps server) node in the grid > attempt to create a cache using a DataRegionName that does not exist in the > grid causes immediate failure in the client node with the following log > output. > > 2023-08-27 17:08:48,520 [44] INF [ImmutableClientServer] Completed > partition exchange [localNode=15122bd7-bf81-44e6-a548-e70dbd9334c0, > exchange=GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion > [topVer=15, minorTopVer=0], evt=NODE_FAILED, evtNode=TcpDiscoveryNode > [id=9d5ed68d-38bb-447d-aed5-189f52660716, > consistentId=9d5ed68d-38bb-447d-aed5-189f52660716, addrs=ArrayList > [127.0.0.1], sockAddrs=null, discPort=0, order=8, intOrder=8, > lastExchangeTime=1693112858024, loc=false, ver=2.15.0#20230425-sha1:f98f7f35, > isClient=true], rebalanced=false, done=true, newCrdFut=null], > topVer=AffinityTopologyVersion [topVer=15, minorTopVer=0]] > 2023-08-27 17:08:48,520 [44] INF [ImmutableClientServer] Exchange timings > [startVer=AffinityTopologyVersion [topVer=15, minorTopVer=0], > resVer=AffinityTopologyVersion [topVer=15, minorTopVer=0], stage="Waiting in > exchange queue" (14850 ms), stage="Exchange parameters initialization" (2 > ms), stage="Determine exchange type" (3 ms), stage="Exchange done" (4 ms), > stage="Total time" (14859 ms)] > 2023-08-27 17:08:48,522 [44] INF [ImmutableClientServer] Exchange longest > local stages [startVer=AffinityTopologyVersion [topVer=15, minorTopVer=0], > resVer=AffinityTopologyVersion [topVer=15, minorTopVer=0]] > 2023-08-27 17:08:48,524 [44] INF [ImmutableClientServer] Finished exchange > init [topVer=AffinityTopologyVersion [topVer=15, minorTopVer=0], crd=false] > 2023-08-27 17:08:48,525 [44] INF [ImmutableClientServer] > AffinityTopologyVersion [topVer=15, minorTopVer=0], evt=NODE_FAILED, > evtNode=9d5ed68d-38bb-447d-aed5-189f52660716, client=true] > Unhandled exception: Apache.Ignite.Core.Cache.CacheException: class > org.apache.ignite.IgniteCheckedException: Failed to complete exchange process. > ---> Apache.Ignite.Core.Common.IgniteException: Failed to complete exchange > process. > ---> Apache.Ignite.Core.Common.JavaException: javax.cache.CacheException: > class org.apache.ignite.IgniteCheckedException: Failed to complete exchange > process. > at > org.apache.ignite.internal.processors.cache.GridCacheUtils.convertToCacheException(GridCacheUtils.java:1272) > at > org.apache.ignite.internal.IgniteKernal.getOrCreateCache0(IgniteKernal.java:2278) > at > org.apache.ignite.internal.IgniteKernal.getOrCreateCache(IgniteKernal.java:2242) > at > org.apache.ignite.internal.processors.platform.PlatformProcessorImpl.processInStreamOutObject(PlatformProcessorImpl.java:643) > at > org.apache.ignite.internal.processors.platform.PlatformTargetProxyImpl.inStreamOutObject(PlatformTargetProxyImpl.java:79) > Caused by: class org.apache.ignite.IgniteCheckedException: Failed to complete > exchange process. > at > org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.createExchangeException(GridDhtPartitionsExchangeFuture.java:3709) > at > org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.sendExchangeFailureMessage(GridDhtPartitionsExchangeFuture.java:3737) > at > org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.finishExchangeOnCoordinator(GridDhtPartitionsExchangeFuture.java:3832) > at > org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.onAllReceived(GridDhtPartitionsExchangeFuture.java:3813) > at > org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.distributedExchange(GridDhtPartitionsExchangeFuture.java:1796) > at > org.apache.ignite.internal.processors.cache.distr
[jira] [Commented] (IGNITE-20299) Creating a cache with an unknown data region name causes total unrecoverable failure of the grid
[ https://issues.apache.org/jira/browse/IGNITE-20299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17760296#comment-17760296 ] Pavel Tupitsyn commented on IGNITE-20299: - > none of the information in the exception indicates that the "Requested > DataRegion is not configured" This information is in the inner JavaException. You can also do *ex.ToString()*. This is not great, of course - would be better if the root cause was in the top level exception message. > Creating a cache with an unknown data region name causes total unrecoverable > failure of the grid > > > Key: IGNITE-20299 > URL: https://issues.apache.org/jira/browse/IGNITE-20299 > Project: Ignite > Issue Type: Bug > Components: cache >Affects Versions: 2.15 > Environment: Observed in: > C# client and grid running on Linux in a container > C# client and grid running on Windows > >Reporter: Raymond Wilson >Priority: Major > > Using the Ignite C# client. > > Given a running grid, having a client (and perhaps server) node in the grid > attempt to create a cache using a DataRegionName that does not exist in the > grid causes immediate failure in the client node with the following log > output. > > 2023-08-27 17:08:48,520 [44] INF [ImmutableClientServer] Completed > partition exchange [localNode=15122bd7-bf81-44e6-a548-e70dbd9334c0, > exchange=GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion > [topVer=15, minorTopVer=0], evt=NODE_FAILED, evtNode=TcpDiscoveryNode > [id=9d5ed68d-38bb-447d-aed5-189f52660716, > consistentId=9d5ed68d-38bb-447d-aed5-189f52660716, addrs=ArrayList > [127.0.0.1], sockAddrs=null, discPort=0, order=8, intOrder=8, > lastExchangeTime=1693112858024, loc=false, ver=2.15.0#20230425-sha1:f98f7f35, > isClient=true], rebalanced=false, done=true, newCrdFut=null], > topVer=AffinityTopologyVersion [topVer=15, minorTopVer=0]] > 2023-08-27 17:08:48,520 [44] INF [ImmutableClientServer] Exchange timings > [startVer=AffinityTopologyVersion [topVer=15, minorTopVer=0], > resVer=AffinityTopologyVersion [topVer=15, minorTopVer=0], stage="Waiting in > exchange queue" (14850 ms), stage="Exchange parameters initialization" (2 > ms), stage="Determine exchange type" (3 ms), stage="Exchange done" (4 ms), > stage="Total time" (14859 ms)] > 2023-08-27 17:08:48,522 [44] INF [ImmutableClientServer] Exchange longest > local stages [startVer=AffinityTopologyVersion [topVer=15, minorTopVer=0], > resVer=AffinityTopologyVersion [topVer=15, minorTopVer=0]] > 2023-08-27 17:08:48,524 [44] INF [ImmutableClientServer] Finished exchange > init [topVer=AffinityTopologyVersion [topVer=15, minorTopVer=0], crd=false] > 2023-08-27 17:08:48,525 [44] INF [ImmutableClientServer] > AffinityTopologyVersion [topVer=15, minorTopVer=0], evt=NODE_FAILED, > evtNode=9d5ed68d-38bb-447d-aed5-189f52660716, client=true] > Unhandled exception: Apache.Ignite.Core.Cache.CacheException: class > org.apache.ignite.IgniteCheckedException: Failed to complete exchange process. > ---> Apache.Ignite.Core.Common.IgniteException: Failed to complete exchange > process. > ---> Apache.Ignite.Core.Common.JavaException: javax.cache.CacheException: > class org.apache.ignite.IgniteCheckedException: Failed to complete exchange > process. > at > org.apache.ignite.internal.processors.cache.GridCacheUtils.convertToCacheException(GridCacheUtils.java:1272) > at > org.apache.ignite.internal.IgniteKernal.getOrCreateCache0(IgniteKernal.java:2278) > at > org.apache.ignite.internal.IgniteKernal.getOrCreateCache(IgniteKernal.java:2242) > at > org.apache.ignite.internal.processors.platform.PlatformProcessorImpl.processInStreamOutObject(PlatformProcessorImpl.java:643) > at > org.apache.ignite.internal.processors.platform.PlatformTargetProxyImpl.inStreamOutObject(PlatformTargetProxyImpl.java:79) > Caused by: class org.apache.ignite.IgniteCheckedException: Failed to complete > exchange process. > at > org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.createExchangeException(GridDhtPartitionsExchangeFuture.java:3709) > at > org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.sendExchangeFailureMessage(GridDhtPartitionsExchangeFuture.java:3737) > at > org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.finishExchangeOnCoordinator(GridDhtPartitionsExchangeFuture.java:3832) > at > org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.onAllReceived(GridDhtPartitionsExchangeFuture.java:3813) > at > or
[jira] [Commented] (IGNITE-20299) Creating a cache with an unknown data region name causes total unrecoverable failure of the grid
[ https://issues.apache.org/jira/browse/IGNITE-20299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17760318#comment-17760318 ] Pavel Tupitsyn commented on IGNITE-20299: - [~rpwilson] sorry, did not refresh the page. I will check the reproducer, thank you. > Creating a cache with an unknown data region name causes total unrecoverable > failure of the grid > > > Key: IGNITE-20299 > URL: https://issues.apache.org/jira/browse/IGNITE-20299 > Project: Ignite > Issue Type: Bug > Components: cache >Affects Versions: 2.15 > Environment: Observed in: > C# client and grid running on Linux in a container > C# client and grid running on Windows > >Reporter: Raymond Wilson >Priority: Major > > Using the Ignite C# client. > > Given a running grid, having a client (and perhaps server) node in the grid > attempt to create a cache using a DataRegionName that does not exist in the > grid causes immediate failure in the client node with the following log > output. > > 2023-08-27 17:08:48,520 [44] INF [ImmutableClientServer] Completed > partition exchange [localNode=15122bd7-bf81-44e6-a548-e70dbd9334c0, > exchange=GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion > [topVer=15, minorTopVer=0], evt=NODE_FAILED, evtNode=TcpDiscoveryNode > [id=9d5ed68d-38bb-447d-aed5-189f52660716, > consistentId=9d5ed68d-38bb-447d-aed5-189f52660716, addrs=ArrayList > [127.0.0.1], sockAddrs=null, discPort=0, order=8, intOrder=8, > lastExchangeTime=1693112858024, loc=false, ver=2.15.0#20230425-sha1:f98f7f35, > isClient=true], rebalanced=false, done=true, newCrdFut=null], > topVer=AffinityTopologyVersion [topVer=15, minorTopVer=0]] > 2023-08-27 17:08:48,520 [44] INF [ImmutableClientServer] Exchange timings > [startVer=AffinityTopologyVersion [topVer=15, minorTopVer=0], > resVer=AffinityTopologyVersion [topVer=15, minorTopVer=0], stage="Waiting in > exchange queue" (14850 ms), stage="Exchange parameters initialization" (2 > ms), stage="Determine exchange type" (3 ms), stage="Exchange done" (4 ms), > stage="Total time" (14859 ms)] > 2023-08-27 17:08:48,522 [44] INF [ImmutableClientServer] Exchange longest > local stages [startVer=AffinityTopologyVersion [topVer=15, minorTopVer=0], > resVer=AffinityTopologyVersion [topVer=15, minorTopVer=0]] > 2023-08-27 17:08:48,524 [44] INF [ImmutableClientServer] Finished exchange > init [topVer=AffinityTopologyVersion [topVer=15, minorTopVer=0], crd=false] > 2023-08-27 17:08:48,525 [44] INF [ImmutableClientServer] > AffinityTopologyVersion [topVer=15, minorTopVer=0], evt=NODE_FAILED, > evtNode=9d5ed68d-38bb-447d-aed5-189f52660716, client=true] > Unhandled exception: Apache.Ignite.Core.Cache.CacheException: class > org.apache.ignite.IgniteCheckedException: Failed to complete exchange process. > ---> Apache.Ignite.Core.Common.IgniteException: Failed to complete exchange > process. > ---> Apache.Ignite.Core.Common.JavaException: javax.cache.CacheException: > class org.apache.ignite.IgniteCheckedException: Failed to complete exchange > process. > at > org.apache.ignite.internal.processors.cache.GridCacheUtils.convertToCacheException(GridCacheUtils.java:1272) > at > org.apache.ignite.internal.IgniteKernal.getOrCreateCache0(IgniteKernal.java:2278) > at > org.apache.ignite.internal.IgniteKernal.getOrCreateCache(IgniteKernal.java:2242) > at > org.apache.ignite.internal.processors.platform.PlatformProcessorImpl.processInStreamOutObject(PlatformProcessorImpl.java:643) > at > org.apache.ignite.internal.processors.platform.PlatformTargetProxyImpl.inStreamOutObject(PlatformTargetProxyImpl.java:79) > Caused by: class org.apache.ignite.IgniteCheckedException: Failed to complete > exchange process. > at > org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.createExchangeException(GridDhtPartitionsExchangeFuture.java:3709) > at > org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.sendExchangeFailureMessage(GridDhtPartitionsExchangeFuture.java:3737) > at > org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.finishExchangeOnCoordinator(GridDhtPartitionsExchangeFuture.java:3832) > at > org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.onAllReceived(GridDhtPartitionsExchangeFuture.java:3813) > at > org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.distributedExchange(GridDhtPartitionsExchangeFuture.java:1796) > at > org.apache.ignite.internal.pro
[jira] [Commented] (IGNITE-20299) Creating a cache with an unknown data region name causes total unrecoverable failure of the grid
[
https://issues.apache.org/jira/browse/IGNITE-20299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17761283#comment-17761283
]
Pavel Tupitsyn commented on IGNITE-20299:
-
[~rpwilson] I can reproduce the issue. Some observations:
* To fix the grid, remove
{code}BadCacheCreationReproducer\Persistence\...\cache-ABadCache{code}
directory
* Reproduces on Apache.Ignite 2.15, but not on GridGain.Ignite 8.8.33
> Creating a cache with an unknown data region name causes total unrecoverable
> failure of the grid
>
>
> Key: IGNITE-20299
> URL: https://issues.apache.org/jira/browse/IGNITE-20299
> Project: Ignite
> Issue Type: Bug
> Components: cache
>Affects Versions: 2.15
> Environment: Observed in:
> C# client and grid running on Linux in a container
> C# client and grid running on Windows
>
>Reporter: Raymond Wilson
>Priority: Major
>
> Using the Ignite C# client.
>
> Given a running grid, having a client (and perhaps server) node in the grid
> attempt to create a cache using a DataRegionName that does not exist in the
> grid causes immediate failure in the client node with the following log
> output.
>
> 2023-08-27 17:08:48,520 [44] INF [ImmutableClientServer] Completed
> partition exchange [localNode=15122bd7-bf81-44e6-a548-e70dbd9334c0,
> exchange=GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion
> [topVer=15, minorTopVer=0], evt=NODE_FAILED, evtNode=TcpDiscoveryNode
> [id=9d5ed68d-38bb-447d-aed5-189f52660716,
> consistentId=9d5ed68d-38bb-447d-aed5-189f52660716, addrs=ArrayList
> [127.0.0.1], sockAddrs=null, discPort=0, order=8, intOrder=8,
> lastExchangeTime=1693112858024, loc=false, ver=2.15.0#20230425-sha1:f98f7f35,
> isClient=true], rebalanced=false, done=true, newCrdFut=null],
> topVer=AffinityTopologyVersion [topVer=15, minorTopVer=0]]
> 2023-08-27 17:08:48,520 [44] INF [ImmutableClientServer] Exchange timings
> [startVer=AffinityTopologyVersion [topVer=15, minorTopVer=0],
> resVer=AffinityTopologyVersion [topVer=15, minorTopVer=0], stage="Waiting in
> exchange queue" (14850 ms), stage="Exchange parameters initialization" (2
> ms), stage="Determine exchange type" (3 ms), stage="Exchange done" (4 ms),
> stage="Total time" (14859 ms)]
> 2023-08-27 17:08:48,522 [44] INF [ImmutableClientServer] Exchange longest
> local stages [startVer=AffinityTopologyVersion [topVer=15, minorTopVer=0],
> resVer=AffinityTopologyVersion [topVer=15, minorTopVer=0]]
> 2023-08-27 17:08:48,524 [44] INF [ImmutableClientServer] Finished exchange
> init [topVer=AffinityTopologyVersion [topVer=15, minorTopVer=0], crd=false]
> 2023-08-27 17:08:48,525 [44] INF [ImmutableClientServer]
> AffinityTopologyVersion [topVer=15, minorTopVer=0], evt=NODE_FAILED,
> evtNode=9d5ed68d-38bb-447d-aed5-189f52660716, client=true]
> Unhandled exception: Apache.Ignite.Core.Cache.CacheException: class
> org.apache.ignite.IgniteCheckedException: Failed to complete exchange process.
> ---> Apache.Ignite.Core.Common.IgniteException: Failed to complete exchange
> process.
> ---> Apache.Ignite.Core.Common.JavaException: javax.cache.CacheException:
> class org.apache.ignite.IgniteCheckedException: Failed to complete exchange
> process.
> at
> org.apache.ignite.internal.processors.cache.GridCacheUtils.convertToCacheException(GridCacheUtils.java:1272)
> at
> org.apache.ignite.internal.IgniteKernal.getOrCreateCache0(IgniteKernal.java:2278)
> at
> org.apache.ignite.internal.IgniteKernal.getOrCreateCache(IgniteKernal.java:2242)
> at
> org.apache.ignite.internal.processors.platform.PlatformProcessorImpl.processInStreamOutObject(PlatformProcessorImpl.java:643)
> at
> org.apache.ignite.internal.processors.platform.PlatformTargetProxyImpl.inStreamOutObject(PlatformTargetProxyImpl.java:79)
> Caused by: class org.apache.ignite.IgniteCheckedException: Failed to complete
> exchange process.
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.createExchangeException(GridDhtPartitionsExchangeFuture.java:3709)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.sendExchangeFailureMessage(GridDhtPartitionsExchangeFuture.java:3737)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.finishExchangeOnCoordinator(GridDhtPartitionsExchangeFuture.java:3832)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.onAllReceived(GridDhtPartitionsExchangeFuture.java:3813)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht
[jira] [Commented] (IGNITE-20299) Creating a cache with an unknown data region name causes total unrecoverable failure of the grid
[ https://issues.apache.org/jira/browse/IGNITE-20299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17761417#comment-17761417 ] Raymond Wilson commented on IGNITE-20299: - Thanks for confirming you can reproduce it. In an earlier experiment I tried deleting the cache folder on my local system and this did fix it (locally). Performing that same operation in our dev environment with data in it did not recover. I will see if I can enhance the reproducer further. > Creating a cache with an unknown data region name causes total unrecoverable > failure of the grid > > > Key: IGNITE-20299 > URL: https://issues.apache.org/jira/browse/IGNITE-20299 > Project: Ignite > Issue Type: Bug > Components: cache >Affects Versions: 2.15 > Environment: Observed in: > C# client and grid running on Linux in a container > C# client and grid running on Windows > >Reporter: Raymond Wilson >Priority: Major > > Using the Ignite C# client. > > Given a running grid, having a client (and perhaps server) node in the grid > attempt to create a cache using a DataRegionName that does not exist in the > grid causes immediate failure in the client node with the following log > output. > > 2023-08-27 17:08:48,520 [44] INF [ImmutableClientServer] Completed > partition exchange [localNode=15122bd7-bf81-44e6-a548-e70dbd9334c0, > exchange=GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion > [topVer=15, minorTopVer=0], evt=NODE_FAILED, evtNode=TcpDiscoveryNode > [id=9d5ed68d-38bb-447d-aed5-189f52660716, > consistentId=9d5ed68d-38bb-447d-aed5-189f52660716, addrs=ArrayList > [127.0.0.1], sockAddrs=null, discPort=0, order=8, intOrder=8, > lastExchangeTime=1693112858024, loc=false, ver=2.15.0#20230425-sha1:f98f7f35, > isClient=true], rebalanced=false, done=true, newCrdFut=null], > topVer=AffinityTopologyVersion [topVer=15, minorTopVer=0]] > 2023-08-27 17:08:48,520 [44] INF [ImmutableClientServer] Exchange timings > [startVer=AffinityTopologyVersion [topVer=15, minorTopVer=0], > resVer=AffinityTopologyVersion [topVer=15, minorTopVer=0], stage="Waiting in > exchange queue" (14850 ms), stage="Exchange parameters initialization" (2 > ms), stage="Determine exchange type" (3 ms), stage="Exchange done" (4 ms), > stage="Total time" (14859 ms)] > 2023-08-27 17:08:48,522 [44] INF [ImmutableClientServer] Exchange longest > local stages [startVer=AffinityTopologyVersion [topVer=15, minorTopVer=0], > resVer=AffinityTopologyVersion [topVer=15, minorTopVer=0]] > 2023-08-27 17:08:48,524 [44] INF [ImmutableClientServer] Finished exchange > init [topVer=AffinityTopologyVersion [topVer=15, minorTopVer=0], crd=false] > 2023-08-27 17:08:48,525 [44] INF [ImmutableClientServer] > AffinityTopologyVersion [topVer=15, minorTopVer=0], evt=NODE_FAILED, > evtNode=9d5ed68d-38bb-447d-aed5-189f52660716, client=true] > Unhandled exception: Apache.Ignite.Core.Cache.CacheException: class > org.apache.ignite.IgniteCheckedException: Failed to complete exchange process. > ---> Apache.Ignite.Core.Common.IgniteException: Failed to complete exchange > process. > ---> Apache.Ignite.Core.Common.JavaException: javax.cache.CacheException: > class org.apache.ignite.IgniteCheckedException: Failed to complete exchange > process. > at > org.apache.ignite.internal.processors.cache.GridCacheUtils.convertToCacheException(GridCacheUtils.java:1272) > at > org.apache.ignite.internal.IgniteKernal.getOrCreateCache0(IgniteKernal.java:2278) > at > org.apache.ignite.internal.IgniteKernal.getOrCreateCache(IgniteKernal.java:2242) > at > org.apache.ignite.internal.processors.platform.PlatformProcessorImpl.processInStreamOutObject(PlatformProcessorImpl.java:643) > at > org.apache.ignite.internal.processors.platform.PlatformTargetProxyImpl.inStreamOutObject(PlatformTargetProxyImpl.java:79) > Caused by: class org.apache.ignite.IgniteCheckedException: Failed to complete > exchange process. > at > org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.createExchangeException(GridDhtPartitionsExchangeFuture.java:3709) > at > org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.sendExchangeFailureMessage(GridDhtPartitionsExchangeFuture.java:3737) > at > org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.finishExchangeOnCoordinator(GridDhtPartitionsExchangeFuture.java:3832) > at > org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.onAllReceived(GridDhtPartitionsExchangeFuture.java:3813) > at >
