[jira] [Created] (IGNITE-14469) Adding a list of caches that will not be forced to rebuild indexes to the control.sh --cache indexes_force_rebuild
Kirill Tkalenko created IGNITE-14469: Summary: Adding a list of caches that will not be forced to rebuild indexes to the control.sh --cache indexes_force_rebuild Key: IGNITE-14469 URL: https://issues.apache.org/jira/browse/IGNITE-14469 Project: Ignite Issue Type: Improvement Reporter: Kirill Tkalenko Fix For: 2.11 After the IGNITE-14321 is implemented, it will be necessary to add to the command *control.sh --cache indexes_force_rebuild* the ability to display to the user that the forced rebuilding of the indexes is impossible, since they are already being rebuilt. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IGNITE-14468) Adding a list of caches that will not be forced to rebuild indexes to the control.sh --cache indexes_force_rebuild
Kirill Tkalenko created IGNITE-14468: Summary: Adding a list of caches that will not be forced to rebuild indexes to the control.sh --cache indexes_force_rebuild Key: IGNITE-14468 URL: https://issues.apache.org/jira/browse/IGNITE-14468 Project: Ignite Issue Type: Improvement Components: control.sh, sql Reporter: Kirill Tkalenko Fix For: 2.11 After the IGNITE-14321 is implemented, it will be necessary to add to the command the ability to display to the user that the forced rebuilding of the indexes is impossible, since they are already being rebuilt. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IGNITE-14458) Fix flaky IgniteLocalWalSizeTest
Kirill Tkalenko created IGNITE-14458: Summary: Fix flaky IgniteLocalWalSizeTest Key: IGNITE-14458 URL: https://issues.apache.org/jira/browse/IGNITE-14458 Project: Ignite Issue Type: Improvement Components: persistence Reporter: Kirill Tkalenko Assignee: Kirill Tkalenko Fix For: 2.11 Fix the flaky *IgniteLocalWalSizeTest* tests. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IGNITE-14384) Remove diagnostics at the stop of the node in case of CorruptedTreeException
Kirill Tkalenko created IGNITE-14384: Summary: Remove diagnostics at the stop of the node in case of CorruptedTreeException Key: IGNITE-14384 URL: https://issues.apache.org/jira/browse/IGNITE-14384 Project: Ignite Issue Type: Improvement Components: persistence Reporter: Kirill Tkalenko Assignee: Kirill Tkalenko Fix For: 2.11 When an error *CorruptedTreeException*, diagnostics are launched, which may take a long time and do not allow the node to be stopped until it completes. Instead of waiting for a long time to complete the diagnostics, a command will be output that will allow to do it offline. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IGNITE-14373) There's a race in auto rollover WAL segment and deactivate of WAL
Kirill Tkalenko created IGNITE-14373:
Summary: There's a race in auto rollover WAL segment and
deactivate of WAL
Key: IGNITE-14373
URL: https://issues.apache.org/jira/browse/IGNITE-14373
Project: Ignite
Issue Type: Bug
Components: persistence
Reporter: Kirill Tkalenko
Assignee: Kirill Tkalenko
Fix For: 2.11
A race was detected between auto rollover WAL segments and deactivation of WAL,
which can lead to an error and a fall of the node by FH.
Error:
{noformat}
Error when executing timeout callback:
o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$2@421a963a
java.lang.AssertionError: Concurrent updates on rollover are not allowed
at
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager.rollOver(FileWriteAheadLogManager.java:1325)
at
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager.closeBufAndRollover(FileWriteAheadLogManager.java:928)
at
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager.checkWalRolloverRequiredDuringInactivityPeriod(FileWriteAheadLogManager.java:819)
at
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager.access$900(FileWriteAheadLogManager.java:159)
at
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$2.onTimeout(FileWriteAheadLogManager.java:782)
at
org.apache.ignite.internal.processors.timeout.GridTimeoutProcessor$TimeoutWorker.body(GridTimeoutProcessor.java:234)
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:748)
{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (IGNITE-14321) Forced index rebuilding does not work correctly
Kirill Tkalenko created IGNITE-14321: Summary: Forced index rebuilding does not work correctly Key: IGNITE-14321 URL: https://issues.apache.org/jira/browse/IGNITE-14321 Project: Ignite Issue Type: Bug Components: sql Reporter: Kirill Tkalenko Assignee: Kirill Tkalenko Fix For: 2.11 At the moment, it is not possible to force an index rebuild twice (or more) even if the first run fails, and this also applies to command *control.sh --cache indexes_force_rebuild*. Thus, you need to fix: * *GridCacheDatabaseSharedManager#forceRebuildIndexes* * *CacheIndexesForceRebuild#execute* -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IGNITE-14254) Graceful stop rebuilding indexes when deactivating a cluster
Kirill Tkalenko created IGNITE-14254: Summary: Graceful stop rebuilding indexes when deactivating a cluster Key: IGNITE-14254 URL: https://issues.apache.org/jira/browse/IGNITE-14254 Project: Ignite Issue Type: Bug Components: sql Reporter: Kirill Tkalenko Assignee: Kirill Tkalenko Fix For: 2.11 If in the process of rebuilding the indexes to deactivate the cluster, then a number of errors may appear, including a node fail. It is necessary to stop index rebuilding before stopping the cache. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IGNITE-14246) Add page diagnostics in a WAL to a IgniteWalConverter
Kirill Tkalenko created IGNITE-14246: Summary: Add page diagnostics in a WAL to a IgniteWalConverter Key: IGNITE-14246 URL: https://issues.apache.org/jira/browse/IGNITE-14246 Project: Ignite Issue Type: Improvement Components: persistence Reporter: Kirill Tkalenko Assignee: Kirill Tkalenko Fix For: 2.11 Adding a --pages argument that will accept either a list of pages in the "grpId:pageId,grpId:pageId" format or a file in which each line will have entries in the "grpId:pageId" format. To find all pages that have changed and thereby run diagnostics. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IGNITE-14175) Update metric for loaded keys through rebalance once at the demand message
Kirill Tkalenko created IGNITE-14175:
Summary: Update metric for loaded keys through rebalance once at
the demand message
Key: IGNITE-14175
URL: https://issues.apache.org/jira/browse/IGNITE-14175
Project: Ignite
Issue Type: Improvement
Components: cache
Reporter: Kirill Tkalenko
Assignee: Kirill Tkalenko
Fix For: 2.11
We are consuming more than half of all rebalance time to updating metrics, even
if the statistic does not enable in the cache.
The problem is here:
{code:java}
//TODO: IGNITE-11330: Update metrics for touched cache only.
for (GridCacheContext ctx : grp.caches()) {
if (ctx.statisticsEnabled())
ctx.cache().metrics0().onRebalanceKeyReceived();
}
{code}
It needs to update once for one demand message and only if the statistic
enabled on the cache.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (IGNITE-14170) Adding metrics of using WAL archive
Kirill Tkalenko created IGNITE-14170: Summary: Adding metrics of using WAL archive Key: IGNITE-14170 URL: https://issues.apache.org/jira/browse/IGNITE-14170 Project: Ignite Issue Type: Improvement Components: persistence Reporter: Kirill Tkalenko Assignee: Kirill Tkalenko Fix For: 2.11 At the moment there is no way to estimate how many segments in the archive we may need, for example, per day. It is proposed to add the following metrics: * org.apache.ignite.mxbean.DataStorageMetricsMXBean#getLastArchivedSegmentIndex - Get the index of the last archived segment. * org.apache.ignite.mxbean.DataStorageMetricsMXBean#getMaxSizeComressedArchivedSegment - Getting the size of the maximum compressed segment in the archive. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IGNITE-13912) Incorrect calculation of WAL segments that should be deleted from WAL archive
Kirill Tkalenko created IGNITE-13912: Summary: Incorrect calculation of WAL segments that should be deleted from WAL archive Key: IGNITE-13912 URL: https://issues.apache.org/jira/browse/IGNITE-13912 Project: Ignite Issue Type: Bug Components: persistence Reporter: Kirill Tkalenko Assignee: Kirill Tkalenko Fix For: 2.10 Now there is an incorrect calculation of WAL segments that should be deleted from WAL archive. Since we delete only those segments whose total size should not exceed DataStorageConfiguration#maxWalArchiveSize * IGNITE_THRESHOLD_WAL_ARCHIVE_SIZE_PERCENTAGE, but should DataStorageConfiguration#maxWalArchiveSize - (DataStorageConfiguration#maxWalArchiveSize * IGNITE_THRESHOLD_WAL_ARCHIVE_SIZE_PERCENTAGE). Therefore, an excess of DataStorageConfiguration#maxWalArchiveSize occurs. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IGNITE-13910) Do not release a segment if it is not present in FileWriteAheadLogManager#reserve
Kirill Tkalenko created IGNITE-13910: Summary: Do not release a segment if it is not present in FileWriteAheadLogManager#reserve Key: IGNITE-13910 URL: https://issues.apache.org/jira/browse/IGNITE-13910 Project: Ignite Issue Type: Bug Components: persistence Reporter: Kirill Tkalenko Assignee: Kirill Tkalenko Fix For: 2.10 Error in *FileWriteAheadLogManager#reserve*, the segment is not released in its absence. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IGNITE-13877) Error restarting the node with switching from disabled WAL archiving to enabled
Kirill Tkalenko created IGNITE-13877:
Summary: Error restarting the node with switching from disabled
WAL archiving to enabled
Key: IGNITE-13877
URL: https://issues.apache.org/jira/browse/IGNITE-13877
Project: Ignite
Issue Type: Bug
Components: persistence
Reporter: Kirill Tkalenko
Assignee: Kirill Tkalenko
Fix For: 2.10
If a user starts a node with WAL archiving disabled, and then wants to restart
a node with WAL archiving enabled, they will fail due to the following error:
{noformat}
SEVERE: Critical system error detected. Will be handled accordingly to
configured handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0,
super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet
[SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]],
failureCtx=FailureContext [type=CRITICAL_ERROR, err=class
o.a.i.i.processors.cache.persistence.StorageException: Failed to read
checkpoint record from WAL, persistence consistency cannot be guaranteed. Make
sure configuration points to correct WAL folders and WAL folder is properly
mounted [ptr=FileWALPointer [idx=11, fileOff=15864934, len=21409],
walPath=db/wal, walArchive=db/wal/archive]]]
class org.apache.ignite.internal.processors.cache.persistence.StorageException:
Failed to read checkpoint record from WAL, persistence consistency cannot be
guaranteed. Make sure configuration points to correct WAL folders and WAL
folder is properly mounted [ptr=FileWALPointer [idx=11, fileOff=15864934,
len=21409], walPath=db/wal, walArchive=db/wal/archive]
at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.performBinaryMemoryRestore(GridCacheDatabaseSharedManager.java:2324)
at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.readMetastore(GridCacheDatabaseSharedManager.java:799)
at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.notifyMetaStorageSubscribersOnReadyForRead(GridCacheDatabaseSharedManager.java:3523)
at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1206)
at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:2089)
at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1758)
at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1147)
at
org.apache.ignite.internal.IgnitionEx.startConfigurations(IgnitionEx.java:1065)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:951)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:850)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:720)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:689)
at org.apache.ignite.Ignition.start(Ignition.java:344)
{noformat}
At this point, the user can be offered the following workaround:
Move all segments to WAL archive directory (include consistentId directory) as
they are except the last one. Last one rename as index %
*DataStorageConfiguration#walSegments*.
Described workaround should be done automatically without user intervention.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (IGNITE-13848) Premature update SegmentReservationStorage#minReserveIdx during truncate of segments
Kirill Tkalenko created IGNITE-13848: Summary: Premature update SegmentReservationStorage#minReserveIdx during truncate of segments Key: IGNITE-13848 URL: https://issues.apache.org/jira/browse/IGNITE-13848 Project: Ignite Issue Type: Bug Components: persistence Reporter: Kirill Tkalenko Assignee: Kirill Tkalenko Fix For: 2.10 It was found premature *SegmentReservationStorage#minReserveIdx* update in *FileWriteAheadLogManager#truncate*. Which creates the wrong state of the segments in the archive. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IGNITE-13847) Make GridEncryptionManager#onWalSegmentRemoved async
Kirill Tkalenko created IGNITE-13847: Summary: Make GridEncryptionManager#onWalSegmentRemoved async Key: IGNITE-13847 URL: https://issues.apache.org/jira/browse/IGNITE-13847 Project: Ignite Issue Type: Improvement Components: persistence Reporter: Kirill Tkalenko Fix For: 2.10 Make the *GridEncryptionManager#onWalSegmentRemoved* asynchronous to prevent deadlocks in the *FileWriteAheadLogManager*. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IGNITE-13831) Move WAL archive cleanup from checkpoint to rollover
Kirill Tkalenko created IGNITE-13831: Summary: Move WAL archive cleanup from checkpoint to rollover Key: IGNITE-13831 URL: https://issues.apache.org/jira/browse/IGNITE-13831 Project: Ignite Issue Type: Improvement Components: persistence Reporter: Kirill Tkalenko Assignee: Kirill Tkalenko Fix For: 2.10 At the moment, WAL archive is cleared at the end of the checkpoint, which does not seem correct and needs to be moved to the *FileWriteAheadLogManager*. It is suggested to clear WAL archive in *FileWriteAheadLogManager#rollOver* when the *DataStorageConfiguration#maxWalArchiveSize* is reached. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IGNITE-13815) Remove ability to delete segments from the middle of WAL archive
Kirill Tkalenko created IGNITE-13815: Summary: Remove ability to delete segments from the middle of WAL archive Key: IGNITE-13815 URL: https://issues.apache.org/jira/browse/IGNITE-13815 Project: Ignite Issue Type: Improvement Components: persistence Reporter: Kirill Tkalenko Assignee: Kirill Tkalenko Fix For: 2.10 At the moment we have the option to delete segments from the middle of the archive via the *FileWriteAheadLogManager#truncate*. This creates gaps in the archive and makes it invalid. It should be possible to delete segments sequentially up to the upper boundary. It has also been found that there is no protection against segment deletion, which may be needed for a binary recovery. Also need to get rid of the physical check when reserving segments through the *FileWriteAheadLogManager#reserve*. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IGNITE-13650) Remove FileWriteAheadLogManager#updateCurrentHandle from under assert
Kirill Tkalenko created IGNITE-13650:
Summary: Remove FileWriteAheadLogManager#updateCurrentHandle from
under assert
Key: IGNITE-13650
URL: https://issues.apache.org/jira/browse/IGNITE-13650
Project: Ignite
Issue Type: Bug
Components: persistence
Reporter: Kirill Tkalenko
Assignee: Kirill Tkalenko
Fix For: 2.10
It is necessary to remove FileWriteAheadLogManager#updateCurrentHandle out of
assert:
{code:java}
assert updateCurrentHandle(next, hnd) : "Concurrent updates on rollover are not
allowed";
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (IGNITE-13613) Adding api to get segment sizes
Kirill Tkalenko created IGNITE-13613: Summary: Adding api to get segment sizes Key: IGNITE-13613 URL: https://issues.apache.org/jira/browse/IGNITE-13613 Project: Ignite Issue Type: Improvement Reporter: Kirill Tkalenko Assignee: Kirill Tkalenko At the moment, we have no way to get the sizes of the WAL segments, for example, through a jmx. First we need to add an api for this. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IGNITE-13469) Adaptation control.sh to picocli framework
Kirill Tkalenko created IGNITE-13469: Summary: Adaptation control.sh to picocli framework Key: IGNITE-13469 URL: https://issues.apache.org/jira/browse/IGNITE-13469 Project: Ignite Issue Type: Improvement Components: control.sh Reporter: Kirill Tkalenko Assignee: Kirill Tkalenko Fix For: 2.10 Adding a dependency [1] to module *ignite-control-utility* and adapting code to it. [1] - https://picocli.info/ -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IGNITE-13438) Correction of error behavior in control.sh
Kirill Tkalenko created IGNITE-13438: Summary: Correction of error behavior in control.sh Key: IGNITE-13438 URL: https://issues.apache.org/jira/browse/IGNITE-13438 Project: Ignite Issue Type: Improvement Components: control.sh Reporter: Kirill Tkalenko Fix For: 2.10 # If the command fails, control.sh should not return EXIT_CODE_OK (an example of a violation is testClusterChangeTag); # In case of an error in the normal state (the cluster is inactive, it has already started for a year, etc.) stacktraces should not be printed; # Need to add specific general error codes so that you do not use EXIT_CODE_UNEXPECTED_ERROR in step 2. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IGNITE-13435) Incorrect parsing of arguments of command warm-up control.sh
Kirill Tkalenko created IGNITE-13435: Summary: Incorrect parsing of arguments of command warm-up control.sh Key: IGNITE-13435 URL: https://issues.apache.org/jira/browse/IGNITE-13435 Project: Ignite Issue Type: Bug Components: control.sh Reporter: Kirill Tkalenko Assignee: Kirill Tkalenko Fix For: 2.10 When parsing arguments for the warm-up command, subsequent arguments may be skipped, such as auto-confirmation "--yes". -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IGNITE-13397) NPE in logSupplierDone(UUID nodeId)
Kirill Tkalenko created IGNITE-13397:
Summary: NPE in logSupplierDone(UUID nodeId)
Key: IGNITE-13397
URL: https://issues.apache.org/jira/browse/IGNITE-13397
Project: Ignite
Issue Type: Bug
Reporter: Kirill Tkalenko
Assignee: Kirill Tkalenko
NPE in logSupplierDone(UUID nodeId)
{noformat}
[2020-08-30 18:06:52,360][ERROR][rebalance-#5033%new-version-node%][root]
Critical system error detected. Will be handled accordingly to configured
handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0,
super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet
[SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]],
failureCtx=FailureContext [type=CRITICAL_ERROR,
err=java.lang.NullPointerException]]
java.lang.NullPointerException
at
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionDemander$RebalanceFuture.logSupplierDone(GridDhtPartitionDemander.java:2009)
at
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionDemander$RebalanceFuture.partitionDone(GridDhtPartitionDemander.java:1730)
at
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionDemander$RebalanceFuture.access$1200(GridDhtPartitionDemander.java:1142)
at
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionDemander.ownPartition(GridDhtPartitionDemander.java:751)
at
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionDemander.handleSupplyMessage(GridDhtPartitionDemander.java:649)
at
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPreloader.lambda$handleSupplyMessage$0(GridDhtPreloader.java:356)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (IGNITE-13385) Documentation of cache warm-up strategy
Kirill Tkalenko created IGNITE-13385: Summary: Documentation of cache warm-up strategy Key: IGNITE-13385 URL: https://issues.apache.org/jira/browse/IGNITE-13385 Project: Ignite Issue Type: Test Reporter: Kirill Tkalenko Fix For: 2.10 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IGNITE-13370) Missed README.txt in ignite-control-utility module
Kirill Tkalenko created IGNITE-13370: Summary: Missed README.txt in ignite-control-utility module Key: IGNITE-13370 URL: https://issues.apache.org/jira/browse/IGNITE-13370 Project: Ignite Issue Type: Improvement Components: control.sh Reporter: Kirill Tkalenko Assignee: Kirill Tkalenko Fix For: 2.10 Missed README.txt in ignite-control-utility module -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IGNITE-13362) Stop warm-up via control.sh
Kirill Tkalenko created IGNITE-13362: Summary: Stop warm-up via control.sh Key: IGNITE-13362 URL: https://issues.apache.org/jira/browse/IGNITE-13362 Project: Ignite Issue Type: New Feature Reporter: Kirill Tkalenko Fix For: 2.10 At the moment, stop warm-up via "control.sh" is not possible due to fact that processing messages from "control.sh" occurs after "discovery" and warm-up goes before it. It is necessary to do processing of messages from "control.sh" before warming up and implement command for "control.sh". -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IGNITE-13345) Warming up node
Kirill Tkalenko created IGNITE-13345:
Summary: Warming up node
Key: IGNITE-13345
URL: https://issues.apache.org/jira/browse/IGNITE-13345
Project: Ignite
Issue Type: New Feature
Reporter: Kirill Tkalenko
Assignee: Kirill Tkalenko
Fix For: 2.10
Summary of
[http://apache-ignite-developers.2346864.n4.nabble.com/DISCUSSION-Cache-warmup-td48582.html|Dev-list]:
# Adding a marker interface
*org.apache.ignite.configuration.WarmUpConfiguratio*n;
# Adding a configuration to
*
*org.apache.ignite.configuration.DataRegionConfiguration#setWarmUpConfiguration*
*
*org.apache.ignite.configuration.DataStorageConfiguration#setDefaultWarmUpConfiguration*
# Add an internal warm-up interface that will start in [1] after [2] (after
recovery);
{code:java}
package org.apache.ignite.internal.processors.cache.warmup;
import org.apache.ignite.IgniteCheckedException;
import org.apache.ignite.configuration.WarmUpConfiguration;
import org.apache.ignite.internal.GridKernalContext;
import org.apache.ignite.internal.processors.cache.persistence.DataRegion;
/**
* Interface for warming up.
*/
public interface WarmUpStrategy {
/**
* Returns configuration class for mapping to strategy.
*
* @return Configuration class.
*/
Class configClass();
/**
* Warm up.
*
* @param kernalCtx Kernal context.
* @param cfg Warm-up configuration.
* @param regionData region.
* @throws IgniteCheckedException if faild.
*/
void warmUp(GridKernalContext kernalCtx, T cfg, DataRegion region) throws
IgniteCheckedException;
/**
* Closing warm up.
*
* @throws IgniteCheckedException if faild.
*/
void close() throws IgniteCheckedException;
}
{code}
# Adding an internal plugin extension for add own strategies;
{code:java}
package org.apache.ignite.internal.processors.cache.warmup;
import java.util.Collection;
import org.apache.ignite.plugin.Extension;
/**
* Interface for getting warm-up strategies from plugins.
*/
public interface WarmUpStrategySupplier extends Extension {
/**
* Getting warm-up strategies.
*
* @return Warm-up strategies.
*/
Collection strategies();
}
{code}
# Adding strategies:
* Without implementation, for the possibility of disabling the warm-up:
*org.apache.ignite.internal.processors.cache.warmup.NoOpWarmUp*,
*org.apache.ignite.internal.processors.cache.warmup.NoOpWarmUpConfiguration*
* Loading everything while there is RAM with priority to indexes:
*org.apache.ignite.internal.processors.cache.warmup.LoadAllWarmUp*,
*org.apache.ignite.internal.processors.cache.warmup.LoadAllWarmUpConfiguration*
# Add a command to "control.sh", to stop current warm-up and cancel all others:
--warm-up stop
[1] -
org.apache.ignite.internal.processors.cache.GridCacheProcessor.CacheRecoveryLifecycle#afterLogicalUpdatesApplied
[2] -
org.apache.ignite.internal.processors.cache.GridCacheProcessor.CacheRecoveryLifecycle#restorePartitionStates
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (IGNITE-13327) Add a metric for processed keys when rebuilding indexes.
Kirill Tkalenko created IGNITE-13327: Summary: Add a metric for processed keys when rebuilding indexes. Key: IGNITE-13327 URL: https://issues.apache.org/jira/browse/IGNITE-13327 Project: Ignite Issue Type: Improvement Reporter: Kirill Tkalenko Assignee: Kirill Tkalenko Fix For: 2.10 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IGNITE-13269) Waiting for completion of operations on indexes before cache stop
Kirill Tkalenko created IGNITE-13269:
Summary: Waiting for completion of operations on indexes before
cache stop
Key: IGNITE-13269
URL: https://issues.apache.org/jira/browse/IGNITE-13269
Project: Ignite
Issue Type: Improvement
Reporter: Kirill Tkalenko
Assignee: Kirill Tkalenko
Currently, there is no waiting for completion of operation on indexes when
cache is stopped. Because of this, there may be errors, for example, when
restarting the node:
{code:java}
Suppressed: java.lang.AssertionError: Release pinned page: FullPageId
[pageId=000206bfc352, effectivePageId=06bfc352, grpId=-782612924]
at
org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl$PagePool.releaseFreePage(PageMemoryImpl.java:1902)
at
org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl$PagePool.access$2100(PageMemoryImpl.java:1773)
at
org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl$ClearSegmentRunnable.run(PageMemoryImpl.java:2878)
... 3 common frames omitted
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (IGNITE-13124) Implement autocompletion for control.sh
Kirill Tkalenko created IGNITE-13124: Summary: Implement autocompletion for control.sh Key: IGNITE-13124 URL: https://issues.apache.org/jira/browse/IGNITE-13124 Project: Ignite Issue Type: Improvement Reporter: Kirill Tkalenko Assignee: Kirill Tkalenko Fix For: 2.9 Adding [1] to a new module, implementing autocomplete for commands. [1] - [https://picocli.info/] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IGNITE-13123) Move control.sh to a separate module
Kirill Tkalenko created IGNITE-13123: Summary: Move control.sh to a separate module Key: IGNITE-13123 URL: https://issues.apache.org/jira/browse/IGNITE-13123 Project: Ignite Issue Type: Improvement Reporter: Kirill Tkalenko Assignee: Kirill Tkalenko Move [1] and its associated classes to a separate "ignite-control-utility" module. [1] - org.apache.ignite.internal.commandline.CommandHandler -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IGNITE-13120) Move control.sh to a separate module and add framework for command autocompletion
Kirill Tkalenko created IGNITE-13120: Summary: Move control.sh to a separate module and add framework for command autocompletion Key: IGNITE-13120 URL: https://issues.apache.org/jira/browse/IGNITE-13120 Project: Ignite Issue Type: Improvement Reporter: Kirill Tkalenko Assignee: Kirill Tkalenko Fix For: 2.9 We can't use autocompletion frameworks for control.sh due to it's a part of ignite-core module. Plan: * Put control.sh in a separate module; * Add dependency [1]; * To make the implementation of autocompletion for commands. 1 - [https://picocli.info/] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IGNITE-13119) Data race when getting index rebuild status
Kirill Tkalenko created IGNITE-13119: Summary: Data race when getting index rebuild status Key: IGNITE-13119 URL: https://issues.apache.org/jira/browse/IGNITE-13119 Project: Ignite Issue Type: Improvement Reporter: Kirill Tkalenko Assignee: Kirill Tkalenko Fix For: 2.9 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IGNITE-12906) Add to IgniteWalConverter possibility output only hashes instead real data
Kirill Tkalenko created IGNITE-12906: Summary: Add to IgniteWalConverter possibility output only hashes instead real data Key: IGNITE-12906 URL: https://issues.apache.org/jira/browse/IGNITE-12906 Project: Ignite Issue Type: Improvement Reporter: Kirill Tkalenko Assignee: Kirill Tkalenko Add flag to be able to hide sensitive data in WAL reader output. For example, we can use cryptographic hash to hide data and see some representation of it to know is it unique or not, with high probability. If define binaryMetadata then output field name and hash field value, else output hash of source binary value This change is required for user security reasons -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IGNITE-12902) Concurrent modification in time to iterate by events
Kirill Tkalenko created IGNITE-12902:
Summary: Concurrent modification in time to iterate by events
Key: IGNITE-12902
URL: https://issues.apache.org/jira/browse/IGNITE-12902
Project: Ignite
Issue Type: Improvement
Reporter: Kirill Tkalenko
Assignee: Kirill Tkalenko
{noformat}[10:20:37]W: [org.apache.ignite:ignite-core] [2020-02-20
10:20:37,324][ERROR][main][CacheExchangeMergeTest9] Failed to pre-stop
processor: GridProcessorAdapter []
[10:20:37]W: [org.apache.ignite:ignite-core]
java.util.ConcurrentModificationException
[10:20:37]W: [org.apache.ignite:ignite-core] at
java.util.ArrayList$Itr.checkForComodification(ArrayList.java:909)
[10:20:37]W: [org.apache.ignite:ignite-core] at
java.util.ArrayList$Itr.next(ArrayList.java:859)
[10:20:37]W: [org.apache.ignite:ignite-core] at
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.onDone(GridDhtPartitionsExchangeFuture.java:2302)
[10:20:37]W: [org.apache.ignite:ignite-core] at
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.onDone(GridDhtPartitionsExchangeFuture.java:142)
[10:20:37]W: [org.apache.ignite:ignite-core] at
org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:464)
[10:20:37]W: [org.apache.ignite:ignite-core] at
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager.onKernalStop0(GridCachePartitionExchangeManager.java:821)
[10:20:37]W: [org.apache.ignite:ignite-core] at
org.apache.ignite.internal.processors.cache.GridCacheSharedManagerAdapter.onKernalStop(GridCacheSharedManagerAdapter.java:120)
[10:20:37]W: [org.apache.ignite:ignite-core] at
org.apache.ignite.internal.processors.cache.GridCacheProcessor.onKernalStop(GridCacheProcessor.java:972)\{noformat}
In that place we are going over event’s collection, looking for server leave’s
event:
{code}
for (DiscoveryEvent evt : evts.events()) {
if (serverLeftEvent(evt)) {
for (CacheGroupContext grp : cctx.cache().cacheGroups())
grp.affinityFunction().removeNode(evt.eventNode().id());
}
}
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (IGNITE-12873) Hosts names resolution takes place even if all nodes are configured via ip addresses
Kirill Tkalenko created IGNITE-12873:
Summary: Hosts names resolution takes place even if all nodes are
configured via ip addresses
Key: IGNITE-12873
URL: https://issues.apache.org/jira/browse/IGNITE-12873
Project: Ignite
Issue Type: Improvement
Reporter: Kirill Tkalenko
Assignee: Kirill Tkalenko
Fix For: 2.9
We got a problem: When connection with DNS was lost transactions processing
hanged up. This happened because IgniteUtils.toSocketAddresses resolves
hostName even if all nodes are configured via ip-addresses.
{code:java}
Thread [name=""utility-#432631%GRID%GridNodeName%"", id=992176, state=RUNNABLE,
blockCnt=1, waitCnt=16]
at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method)
at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:929)
at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1324)
at java.net.InetAddress.getAllByName0(InetAddress.java:1277)
at java.net.InetAddress.getAllByName(InetAddress.java:1193)
at java.net.InetAddress.getAllByName(InetAddress.java:1127)
at java.net.InetAddress.getByName(InetAddress.java:1077)
at java.net.InetSocketAddress.(InetSocketAddress.java:220)
at o.a.i.i.util.IgniteUtils.toSocketAddresses(IgniteUtils.java:8982)
at
o.a.i.spi.communication.tcp.TcpCommunicationSpi.nodeAddresses(TcpCommunicationSpi.java:3228)
at
o.a.i.spi.communication.tcp.TcpCommunicationSpi.nodeAddresses(TcpCommunicationSpi.java:3200)
at
o.a.i.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:3291)
at
o.a.i.spi.communication.tcp.TcpCommunicationSpi.createNioClient(TcpCommunicationSpi.java:3027)
at
o.a.i.spi.communication.tcp.TcpCommunicationSpi.reserveClient(TcpCommunicationSpi.java:2907)
at
o.a.i.spi.communication.tcp.TcpCommunicationSpi.sendMessage0(TcpCommunicationSpi.java:2750)
at
o.a.i.spi.communication.tcp.TcpCommunicationSpi.sendMessage(TcpCommunicationSpi.java:2709)
at o.a.i.i.managers.communication.GridIoManager.send(GridIoManager.java:1643)
at
o.a.i.i.managers.communication.GridIoManager.sendOrderedMessage(GridIoManager.java:1863)
at
o.a.i.i.processors.continuous.GridContinuousProcessor.sendWithRetries(GridContinuousProcessor.java:1873)
at
o.a.i.i.processors.continuous.GridContinuousProcessor.sendWithRetries(GridContinuousProcessor.java:1844)
at
o.a.i.i.processors.continuous.GridContinuousProcessor.sendWithRetries(GridContinuousProcessor.java:1826)
at
o.a.i.i.processors.continuous.GridContinuousProcessor.sendNotification(GridContinuousProcessor.java:1244)
at
o.a.i.i.processors.continuous.GridContinuousProcessor.addNotification(GridContinuousProcessor.java:1181)
at
o.a.i.i.processors.cache.query.continuous.CacheContinuousQueryHandler.onEntryUpdate(CacheContinuousQueryHandler.java:882)
at
o.a.i.i.processors.cache.query.continuous.CacheContinuousQueryHandler.access$600(CacheContinuousQueryHandler.java:85)
at
o.a.i.i.processors.cache.query.continuous.CacheContinuousQueryHandler$2.onEntryUpdated(CacheContinuousQueryHandler.java:429)
at
o.a.i.i.processors.cache.query.continuous.CacheContinuousQueryManager.onEntryUpdated(CacheContinuousQueryManager.java:400)
at
o.a.i.i.processors.cache.GridCacheMapEntry.innerSet(GridCacheMapEntry.java:)
at
o.a.i.i.processors.cache.transactions.IgniteTxLocalAdapter.userCommit(IgniteTxLocalAdapter.java:659)
at
o.a.i.i.processors.cache.distributed.dht.GridDhtTxLocalAdapter.localFinish(GridDhtTxLocalAdapter.java:796)
at
o.a.i.i.processors.cache.distributed.dht.GridDhtTxLocal.localFinish(GridDhtTxLocal.java:603)
at
o.a.i.i.processors.cache.distributed.dht.GridDhtTxLocal.finishTx(GridDhtTxLocal.java:475)
at
o.a.i.i.processors.cache.distributed.dht.GridDhtTxLocal.commitDhtLocalAsync(GridDhtTxLocal.java:532)
at
o.a.i.i.processors.cache.transactions.IgniteTxHandler.finishDhtLocal(IgniteTxHandler.java:1017)
at
o.a.i.i.processors.cache.transactions.IgniteTxHandler.finish(IgniteTxHandler.java:896)
at
o.a.i.i.processors.cache.transactions.IgniteTxHandler.processNearTxFinishRequest(IgniteTxHandler.java:852)
at
o.a.i.i.processors.cache.transactions.IgniteTxHandler.access$200(IgniteTxHandler.java:102)
at
o.a.i.i.processors.cache.transactions.IgniteTxHandler$3.apply(IgniteTxHandler.java:196)
at
o.a.i.i.processors.cache.transactions.IgniteTxHandler$3.apply(IgniteTxHandler.java:194)
at
o.a.i.i.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1077)
at
o.a.i.i.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:587)
at
o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:386)
at
o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:312)
at
o.a.i.i.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:102)
at
[jira] [Created] (IGNITE-12836) Writing methods to avoid limit of 65535 bytes for java.io.DataOutput#writeUTF
Kirill Tkalenko created IGNITE-12836: Summary: Writing methods to avoid limit of 65535 bytes for java.io.DataOutput#writeUTF Key: IGNITE-12836 URL: https://issues.apache.org/jira/browse/IGNITE-12836 Project: Ignite Issue Type: Improvement Reporter: Kirill Tkalenko Assignee: Kirill Tkalenko -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IGNITE-12821) control.sh validate_indexes must check that index size and cache size are same
Kirill Tkalenko created IGNITE-12821:
Summary: control.sh validate_indexes must check that index size
and cache size are same
Key: IGNITE-12821
URL: https://issues.apache.org/jira/browse/IGNITE-12821
Project: Ignite
Issue Type: Improvement
Reporter: Kirill Tkalenko
Assignee: Kirill Tkalenko
Validate_indexes should check that index size and cache size are same.
That check must be enabled by option \{{--check-sizes}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (IGNITE-12797) Add command line option to CommandHandler to be able to see full stack trace and cause exception in log in case of error.
Kirill Tkalenko created IGNITE-12797: Summary: Add command line option to CommandHandler to be able to see full stack trace and cause exception in log in case of error. Key: IGNITE-12797 URL: https://issues.apache.org/jira/browse/IGNITE-12797 Project: Ignite Issue Type: Improvement Reporter: Kirill Tkalenko Assignee: Kirill Tkalenko In case of error control.sh can print common error message without any information about root cause of error. Printing full stack trace and cause can ease the analysis. User should be able to turn it on when launching control.sh with specific option. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IGNITE-12750) Correct the setting of the pool to build the index
Kirill Tkalenko created IGNITE-12750:
Summary: Correct the setting of the pool to build the index
Key: IGNITE-12750
URL: https://issues.apache.org/jira/browse/IGNITE-12750
Project: Ignite
Issue Type: Bug
Reporter: Kirill Tkalenko
Assignee: Kirill Tkalenko
Fix For: 2.9
There was a bug in setting up a ThreadPool for building an index, so it turns
out that only 1 thread always works. We need more threads.
{code:java}
org.apache.ignite.internal.IgnitionEx.IgniteNamedInstance#buildIdxExecSvc{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (IGNITE-12684) Optimize count of threads for rebuild indexes
Kirill Tkalenko created IGNITE-12684:
Summary: Optimize count of threads for rebuild indexes
Key: IGNITE-12684
URL: https://issues.apache.org/jira/browse/IGNITE-12684
Project: Ignite
Issue Type: Improvement
Reporter: Kirill Tkalenko
Assignee: Kirill Tkalenko
Fix For: 2.9
We were recently implemented the new property {{INDEX_REBUILDING_PARALLELISM}}
(by issue IGNITE-11979).
That fix do possible to rebuild indexes, after restart without index.bin or
restore from snapshot to another topology, in parallel threads.
But now the process of rebuild can to consume a lot of threads, which limited
on INDEX_REBUILDING_PARALLELISM * AVAILABLE_PROC_CNT (count of thread in public
pool).
For example in case of INDEX_REBUILDING_PARALLELISM = 28 and 64 CPU it will be
more then 1.5К threads.
Look at the method, it will create dedicate thread on each invocation
`org.apache.ignite.internal.processors.query.schema.SchemaIndexCacheVisitorImpl#processPartitionsAsync`
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (IGNITE-12680) Cleanup unneeded explicit usages of SystemPropertiesRule
Kirill Tkalenko created IGNITE-12680:
Summary: Cleanup unneeded explicit usages of SystemPropertiesRule
Key: IGNITE-12680
URL: https://issues.apache.org/jira/browse/IGNITE-12680
Project: Ignite
Issue Type: Improvement
Affects Versions: 2.9
Reporter: Kirill Tkalenko
Assignee: Kirill Tkalenko
In some test classes there are explicit declarations like:
{code}
@ClassRule public static final TestRule classRule = new SystemPropertiesRule();
{code}
Usually there is no need for it as class and instance rules needed for
\{{@WithSystemProperty}} annotation are already present in
\{{GridAbstractTest}}.
Let's cleanup such usages from code.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (IGNITE-12616) At start grid need twice enter psswords for keystore and truststore.
Kirill Tkalenko created IGNITE-12616: Summary: At start grid need twice enter psswords for keystore and truststore. Key: IGNITE-12616 URL: https://issues.apache.org/jira/browse/IGNITE-12616 Project: Ignite Issue Type: Improvement Reporter: Kirill Tkalenko Assignee: Kirill Tkalenko Fix For: 2.9 At start grid with keystore and truststore parameters but without password, need twice enter psswords for keystore and truststore (before and after enter password for user accaunt). sequencing: 1.Start grid: bin/control.sh --deactivate --host test_host --user test_user --keystore /usr/key_store/keystore.jks --truststore /usr/key_store/truststore.jks 2. Enter password for keystore 3. Enter password for truststore 4. Enter password for user accaunt 5. Second time enter password for keystore 6. Second time enter password for trustore 7. Doing start... Expected result : all without 5-6. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IGNITE-12497) PartitionsEvictManager should log all partitions which will be evicted
Kirill Tkalenko created IGNITE-12497: Summary: PartitionsEvictManager should log all partitions which will be evicted Key: IGNITE-12497 URL: https://issues.apache.org/jira/browse/IGNITE-12497 Project: Ignite Issue Type: Improvement Reporter: Kirill Tkalenko Assignee: Kirill Tkalenko Fix For: 2.9 Now we print information about eviction only if it longer then threshold (i.e. progress). And we can't detect in logs, that partition was evicted due to different reasons (rebalance, wrong configuration custom affinity). I think we could print information on info level about each evicted partition before start of eviction. Information about partitions could be aggregated, compacted and printed by timer, but *all evicted partitions must be printed to log anyway.* I would have the following information about each partition: * partitionId * groupId * groupName * reason (eviction, clearing) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IGNITE-12478) Add long running operation timeout to MXBean for manipulate in run-time
Kirill Tkalenko created IGNITE-12478:
Summary: Add long running operation timeout to MXBean for
manipulate in run-time
Key: IGNITE-12478
URL: https://issues.apache.org/jira/browse/IGNITE-12478
Project: Ignite
Issue Type: Improvement
Reporter: Kirill Tkalenko
Assignee: Kirill Tkalenko
Fix For: 2.8
Previously we can change the timeout only through property
({{IGNITE_LONG_OPERATIONS_DUMP_TIMEOUT}} default 60_000), but in a real
environment, we do not want to stop cluster (or node) for modify this property.
We should have a MBean's property, which allow to fluent in run-time behavior
of printing LRT/LRF (like a property
{{SqlQueryMXBeanImpl#setLongQueryWarningTimeout}} which printing long running
queries).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (IGNITE-12466) Monitor query pool starvation
Kirill Tkalenko created IGNITE-12466: Summary: Monitor query pool starvation Key: IGNITE-12466 URL: https://issues.apache.org/jira/browse/IGNITE-12466 Project: Ignite Issue Type: Improvement Reporter: Kirill Tkalenko Assignee: Kirill Tkalenko Fix For: 2.8 We are having a periodical starvation check ones of IGNITE_STARVATION_CHECK_INTERVAL interval. But that had by system and public pool only. I want to see the same monitor by query pool. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IGNITE-12465) Extend test coverage [IGNITE-11995] control.sh if experimental command disabled - don't show help for experemental commands
Kirill Tkalenko created IGNITE-12465: Summary: Extend test coverage [IGNITE-11995] control.sh if experimental command disabled - don't show help for experemental commands Key: IGNITE-12465 URL: https://issues.apache.org/jira/browse/IGNITE-12465 Project: Ignite Issue Type: Improvement Reporter: Kirill Tkalenko Assignee: Kirill Tkalenko Fix For: 2.8 Extend test coverage for IGNITE-11995 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IGNITE-12447) Modification of S#compact method
Kirill Tkalenko created IGNITE-12447: Summary: Modification of S#compact method Key: IGNITE-12447 URL: https://issues.apache.org/jira/browse/IGNITE-12447 Project: Ignite Issue Type: Improvement Reporter: Kirill Tkalenko Assignee: Kirill Tkalenko Fix For: 2.8 Modification of S#compact method so that it is possible to pass collection of Numbers. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IGNITE-12440) Sensetive data leak in partition release future messages
Kirill Tkalenko created IGNITE-12440: Summary: Sensetive data leak in partition release future messages Key: IGNITE-12440 URL: https://issues.apache.org/jira/browse/IGNITE-12440 Project: Ignite Issue Type: Improvement Reporter: Kirill Tkalenko Assignee: Kirill Tkalenko Fix For: 2.8 There is sensitive data lick in partition release future messages. We should hide sensitive data when option -DIGNITE_TO_STRING_INCLUDE_SENSITIVE=false is used 2019-10-04 06:51:40.328[WARN ]exchange-worker-#161%GRID%GridNodeName%[o.a.i.i.p.c.d.d.p.GridDhtPartitionsExchangeFuture] Partition release future: PartitionReleaseFuture -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IGNITE-12392) Faster transaction rolled back when one of backup node failed
Kirill Tkalenko created IGNITE-12392:
Summary: Faster transaction rolled back when one of backup node
failed
Key: IGNITE-12392
URL: https://issues.apache.org/jira/browse/IGNITE-12392
Project: Ignite
Issue Type: Improvement
Reporter: Kirill Tkalenko
Assignee: Kirill Tkalenko
Fix For: 2.8
In case of massive prepared transactions roll back, when node fail, have a
linearizable behavior:
{noformat}2019-09-26
18:48:21.034[ERROR][sys-stripe-16-#17%DPL_GRID%DplGridNodeName%[o.a.i.s.c.tcp.TcpCommunicationSpi]
Failed to send message to remote node [node=TcpDiscoveryNode
[id=1dc0c76a-8e72-48e7-9718-b157eea1b812, addrs=ArrayList [10.124.133.201],
sockAddrs=HashSet [marica63.ca.sbrf.ru/10.124.133.201:47500], discPort=47500,
order=524, intOrder=311, lastExchangeTime=1569430937898, loc=false,
ver=2.5.1#20190327-sha1:6edfea1b, isClient=false], msg=GridIoMessage [plc=2,
topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false,
msg=GridCacheIdMessage [cacheId=0]GridDistributedBaseMessage
[ver=GridCacheVersion [topVer=176921134, order=1634060411645, nodeOrder=1],
committedVers=EmptyList [], rolledbackVers=EmptyList [], cnt=0,
super=]GridDistributedTxFinishRequest [topVer=AffinityTopologyVersion
[topVer=524, minorTopVer=2],
futId=fb44a686e61-9a074a8c-dca4--84fe-e9a93818fbd2, threadId=2098,
commitVer=GridCacheVersion [topVer=176921134, order=1634060411645, nodeOrder=1],
org.apache.ignite.internal.cluster.ClusterTopologyCheckedException: Failed to
send message (node left topology): TcpDiscoveryNode
[id=1dc0c76a-8e72-48e7-9718-b157eea1b812, addrs=ArrayList [10.124.133.201],
sockAddrs=HashSet [marica63.ca.sbrf.ru/10.124.133.201:47500], discPort=47500,
order=524, intOrder=311, lastExchangeTime=1569430937898, loc=false,
ver=2.5.1#20190327-sha1:6edfea1b, isClient=false]
at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:3276)
at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioClient(TcpCommunicationSpi.java:2998)
at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.reserveClient(TcpCommunicationSpi.java:2878)
at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage0(TcpCommunicationSpi.java:2721)
at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage(TcpCommunicationSpi.java:2680)
at
org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:1643)
at
org.apache.ignite.internal.managers.communication.GridIoManager.sendToGridTopic(GridIoManager.java:1715)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.send(GridCacheIoManager.java:1177)
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxFinishFuture.finish(GridDhtTxFinishFuture.java:462)
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxFinishFuture.finish(GridDhtTxFinishFuture.java:291)
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxLocal.finishTx(GridDhtTxLocal.java:495)
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxLocal.rollbackDhtLocalAsync(GridDhtTxLocal.java:571)
at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.finishDhtLocal(IgniteTxHandler.java:1005)
at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.finish(IgniteTxHandler.java:876)
at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.processNearTxFinishRequest(IgniteTxHandler.java:832)
at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.access$200(IgniteTxHandler.java:101)
at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$3.apply(IgniteTxHandler.java:193)
at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$3.apply(IgniteTxHandler.java:191)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1061)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:586)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:385)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:311)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:300)
at
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1556)
at
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1184)
at
[jira] [Created] (IGNITE-12338) Use ignite thread for notifier about long query
Kirill Tkalenko created IGNITE-12338:
Summary: Use ignite thread for notifier about long query
Key: IGNITE-12338
URL: https://issues.apache.org/jira/browse/IGNITE-12338
Project: Ignite
Issue Type: Improvement
Reporter: Kirill Tkalenko
Assignee: Kirill Tkalenko
Fix For: 2.8
If we are using simple Thread that name of it will be random and look as each
any thread.
We have to use IgniteThread for internal threads to platform.
{code}checkWorker = new GridWorker(ctx.igniteInstanceName(), "long-qry", log) {
@Override protected void body() throws InterruptedException,
IgniteInterruptedCheckedException {
while (true) {
checkLongRunning();
U.sleep(CHECK_PERIOD);
}
}
};
timeout = ctx.config().getLongQueryWarningTimeout();
Thread thread = new Thread(checkWorker);
thread.setDaemon(true);
thread.start();\{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (IGNITE-12324) improper message in BinaryObjectException is printed when accessing fieldOrder method binary field with binary object of unregistered type
Kirill Tkalenko created IGNITE-12324:
Summary: improper message in BinaryObjectException is printed when
accessing fieldOrder method binary field with binary object of unregistered type
Key: IGNITE-12324
URL: https://issues.apache.org/jira/browse/IGNITE-12324
Project: Ignite
Issue Type: Improvement
Reporter: Kirill Tkalenko
Assignee: Kirill Tkalenko
Fix For: 2.8
ScenarioScenario1. Create `BinaryObject` instance with `time` field of type
`TimeValue`.2. Extract `BinaryField` `time` from binary object3. Modify binary
object internal array so that it would reference non existing binary type4.
Invoke `fieldOrder` (or `value`) of binary field passing modified binary
object5. Expect that `BinaryObjectException` is thrown5.1 Expect that message
contains information about expected and actual binary type id, expected type
name, field id, field name, expected field type.5.2. Expect that for actual
type name message contains 'null'
Actual: \{noformat}class org.apache.ignite.binary.BinaryObjectException: Failed
to get binary type details [typeId=-1291121110] at
org.apache.ignite.internal.binary.BinaryTypeProxy.target(BinaryTypeProxy.java:116)
at
org.apache.ignite.internal.binary.BinaryTypeProxy.typeName(BinaryTypeProxy.java:75)
at
org.apache.ignite.internal.binary.BinaryFieldImpl.fieldOrder(BinaryFieldImpl.java:291)
at
org.apache.ignite.internal.binary.BinaryFieldImpl.value(BinaryFieldImpl.java:109)\{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (IGNITE-12318) Extend test coverage [IGNITE-11967] control.sh validate_indexes SQL Index issue must contain information about cache group
Kirill Tkalenko created IGNITE-12318:
Summary: Extend test coverage [IGNITE-11967] control.sh
validate_indexes SQL Index issue must contain information about cache group
Key: IGNITE-12318
URL: https://issues.apache.org/jira/browse/IGNITE-12318
Project: Ignite
Issue Type: Improvement
Reporter: Kirill Tkalenko
Assignee: Kirill Tkalenko
Fix For: 2.8
Test:
{quote}org.apache.ignite.util.GridCommandHandlerIndexingClusterByClassTest#testBrokenCacheDataTreeShouldFailValidationWithCacheGroupInfo\{quote}
# Put indexed data.
# Break CacheDataTree.
# Execute “control.sh validate_indexes”.
# Verify that output contains cacheGroup, cacheName.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (IGNITE-12316) Extend test coverage [IGNITE-10761] GridCacheProcessor should add info about cache in exception message, if applicable.
Kirill Tkalenko created IGNITE-12316:
Summary: Extend test coverage [IGNITE-10761] GridCacheProcessor
should add info about cache in exception message, if applicable.
Key: IGNITE-12316
URL: https://issues.apache.org/jira/browse/IGNITE-12316
Project: Ignite
Issue Type: Improvement
Reporter: Kirill Tkalenko
Assignee: Kirill Tkalenko
Fix For: 2.8
Test cache operations that should fail inside tx thread
||*Forbidden operation inside transaction*||
|dynamic single cache start|
|dynamic start of multiple caches|
|dynamic cache destroy|
|dynamic destroy of multiple caches|
|dynamic cache close|
|reset cache state|
{quote}org.apache.ignite.internal.processors.cache.GridCacheProcessorActiveTxTest#testDynamicSingleCacheStart
org.apache.ignite.internal.processors.cache.GridCacheProcessorActiveTxTest#testDynamicStartMultipleCaches
org.apache.ignite.internal.processors.cache.GridCacheProcessorActiveTxTest#testDynamicCacheDestroy
org.apache.ignite.internal.processors.cache.GridCacheProcessorActiveTxTest#testDynamicDestroyMultipleCaches
org.apache.ignite.internal.processors.cache.GridCacheProcessorActiveTxTest#testDynamicCacheClose
org.apache.ignite.internal.processors.cache.GridCacheProcessorActiveTxTest#testResetCacheState\{quote}
# Start cluster, start active transaction in thread
# Assert that in *Forbidden operation inside transaction* scenarios exception
occurs with log information about cache name and reason
# Finish running transaction, assert that there is no exception while
evaluating *Forbidden operation inside transaction*
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (IGNITE-12305) Extend test coverage [IGNITE-11959] NullPointerException If transaction failed and failure handler doesn't configured explicitly
Kirill Tkalenko created IGNITE-12305: Summary: Extend test coverage [IGNITE-11959] NullPointerException If transaction failed and failure handler doesn't configured explicitly Key: IGNITE-12305 URL: https://issues.apache.org/jira/browse/IGNITE-12305 Project: Ignite Issue Type: Improvement Reporter: Kirill Tkalenko Assignee: Kirill Tkalenko Fix For: 2.8 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IGNITE-12267) ClassCastException after change column type (drop, add)
Kirill Tkalenko created IGNITE-12267: Summary: ClassCastException after change column type (drop, add) Key: IGNITE-12267 URL: https://issues.apache.org/jira/browse/IGNITE-12267 Project: Ignite Issue Type: Improvement Reporter: Kirill Tkalenko Assignee: Kirill Tkalenko Fix For: 2.8 SQL column type change is not present, but it is possible to delete and create with a new type. The application of the migration script passes without errors. The error occurs whenever the column is accessed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IGNITE-12226) Improve BinaryObjectException message
Kirill Tkalenko created IGNITE-12226:
Summary: Improve BinaryObjectException message
Key: IGNITE-12226
URL: https://issues.apache.org/jira/browse/IGNITE-12226
Project: Ignite
Issue Type: Improvement
Reporter: Kirill Tkalenko
Assignee: Kirill Tkalenko
Fix For: 2.8
We are getting the following exception in case of different type IDs:
{noformat}
org.apache.ignite.binary.BinaryObjectException: Failed to get field because
type ID of passed object differs from type ID this BinaryField belongs to
[expected=-635374417, actual=1778229603]
at
org.apache.ignite.internal.binary.BinaryFieldImpl.fieldOrder(BinaryFieldImpl.java:287)
at
org.apache.ignite.internal.binary.BinaryFieldImpl.value(BinaryFieldImpl.java:109)
at
org.apache.ignite.internal.processors.query.property.QueryBinaryProperty.fieldValue(QueryBinaryProperty.java:220)
at
org.apache.ignite.internal.processors.query.property.QueryBinaryProperty.value(QueryBinaryProperty.java:116)
at
org.apache.ignite.internal.processors.query.h2.opt.GridH2RowDescriptor.columnValue(GridH2RowDescriptor.java:331)
at
org.apache.ignite.internal.processors.query.h2.opt.GridH2KeyValueRowOnheap.getValue0(GridH2KeyValueRowOnheap.java:122)
at
org.apache.ignite.internal.processors.query.h2.opt.GridH2KeyValueRowOnheap.getValue(GridH2KeyValueRowOnheap.java:106)
at
org.apache.ignite.internal.processors.query.h2.database.H2Tree.compare(H2Tree.java:350)
at
org.apache.ignite.internal.processors.query.h2.database.H2Tree.compare(H2Tree.java:56)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.compare(BPlusTree.java:4614)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findInsertionPoint(BPlusTree.java:4534)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.access$1300(BPlusTree.java:92)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Search.run0(BPlusTree.java:296)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler.run(BPlusTree.java:4967)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Search.run(BPlusTree.java:276)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler.run(BPlusTree.java:4952)
at
org.apache.ignite.internal.processors.cache.persistence.tree.util.PageHandler.readPage(PageHandler.java:161)
at
org.apache.ignite.internal.processors.cache.persistence.DataStructure.read(DataStructure.java:348)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.putDown(BPlusTree.java:2450)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.doPut(BPlusTree.java:2203)
{noformat}
This message isn't informative. We should print more detailes about types, for
example classname, class structure (if applicable), which field in class we
tried to read.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (IGNITE-12162) Add the cluster name to the promt message "--deactivate" the control.sh
Kirill Tkalenko created IGNITE-12162: Summary: Add the cluster name to the promt message "--deactivate" the control.sh Key: IGNITE-12162 URL: https://issues.apache.org/jira/browse/IGNITE-12162 Project: Ignite Issue Type: Improvement Reporter: Kirill Tkalenko Assignee: Kirill Tkalenko Fix For: 2.8 Add the cluster name to the promt message "--deactivate" the control.sh -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (IGNITE-12135) Rework GridCommandHandlerTest
Kirill Tkalenko created IGNITE-12135: Summary: Rework GridCommandHandlerTest Key: IGNITE-12135 URL: https://issues.apache.org/jira/browse/IGNITE-12135 Project: Ignite Issue Type: Improvement Reporter: Kirill Tkalenko Assignee: Kirill Tkalenko Fix For: 2.8 There are 50+ tests. In each test we are start and stop nodes. I think we could split tests at least to two groups: # Tests on normal behaviour. We could start nodes before all tests and stop them after all tests. # Tests required start new cluster before each test. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (IGNITE-12080) Add extended logging for rebalance
Kirill Tkalenko created IGNITE-12080: Summary: Add extended logging for rebalance Key: IGNITE-12080 URL: https://issues.apache.org/jira/browse/IGNITE-12080 Project: Ignite Issue Type: Improvement Reporter: Kirill Tkalenko Assignee: Kirill Tkalenko We should log all information about finished rebalance on demander node. I'd have in log: h3. Total information: # Rebalance duration, rebalance start time/rebalance finish time # How many partitions were processed in each topic (number of paritions, number of entries, number of bytes) # How many nodes were suppliers in rebalance (nodeId, number of supplied paritions, number of supplied entries, number of bytes, duraton of getting and processing partitions from supplier) h3. Information per cache group: # Rebalance duration, rebalance start time/rebalance finish time # How many partitions were processed in each topic (number of paritions, number of entries, number of bytes) # How many nodes were suppliers in rebalance (nodeId, number of supplied paritions, list of partition ids with PRIMARY/BACKUP flag, number of supplied entries, number of bytes, duraton of getting and processing partitions from supplier) # Information about each partition distribution (list of nodeIds with primary/backup flag and marked supplier nodeId) h3. Information per supplier node: # How many paritions were requested: #* Total number #* Primary/backup distribution (number of primary partitions, number of backup partitions) #* Total number of entries #* Total size partitions in bytes # How many paritions were requested per cache group: #* Number of requested partitions #* Number of entries in partitions #* Total size of partitions in bytes #* List of requested partitions with size in bytes, count entries, primary or backup partition flag -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Created] (IGNITE-11995) control.sh if experimental command disabled - don't show help for experemental commands
Kirill Tkalenko created IGNITE-11995: Summary: control.sh if experimental command disabled - don't show help for experemental commands Key: IGNITE-11995 URL: https://issues.apache.org/jira/browse/IGNITE-11995 Project: Ignite Issue Type: Improvement Reporter: Kirill Tkalenko Assignee: Kirill Tkalenko Fix For: 2.8 If experimental command disabled: * don't show WALCommand help * if user ask for help for particular command - print out warning about experimental commands instead of ignoring user request -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Created] (IGNITE-11993) Print warning if awaiting next wal segment it too long
Kirill Tkalenko created IGNITE-11993: Summary: Print warning if awaiting next wal segment it too long Key: IGNITE-11993 URL: https://issues.apache.org/jira/browse/IGNITE-11993 Project: Ignite Issue Type: Improvement Reporter: Kirill Tkalenko Assignee: Kirill Tkalenko Fix For: 2.8 We must print warn to log, if awaiting next WAL segment more then defined threshold. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Created] (IGNITE-11988) control.sh validate_indexes SQL Index issue add information about group and cache id
Kirill Tkalenko created IGNITE-11988: Summary: control.sh validate_indexes SQL Index issue add information about group and cache id Key: IGNITE-11988 URL: https://issues.apache.org/jira/browse/IGNITE-11988 Project: Ignite Issue Type: Improvement Reporter: Kirill Tkalenko Assignee: Kirill Tkalenko Fix For: 2.8 -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Created] (IGNITE-11967) control.sh validate_indexes SQL Index issue must contain information about cache group
Kirill Tkalenko created IGNITE-11967: Summary: control.sh validate_indexes SQL Index issue must contain information about cache group Key: IGNITE-11967 URL: https://issues.apache.org/jira/browse/IGNITE-11967 Project: Ignite Issue Type: Improvement Reporter: Kirill Tkalenko Assignee: Kirill Tkalenko Fix For: 2.8 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (IGNITE-11959) NullPointerException If transaction failed and failure handler doesn't configured explicitly
Kirill Tkalenko created IGNITE-11959:
Summary: NullPointerException If transaction failed and failure
handler doesn't configured explicitly
Key: IGNITE-11959
URL: https://issues.apache.org/jira/browse/IGNITE-11959
Project: Ignite
Issue Type: Bug
Reporter: Kirill Tkalenko
Assignee: Kirill Tkalenko
Fix For: 2.8
In IGNITE-9082 was introduced code in IgniteTxAdapter:
{code:java}
U.error(log, String.format(fmt, commit, CU.txString(this),
cctx.gridConfig().getFailureHandler().getClass().getSimpleName()), e);
{code}
We get NullPointerException, If failureHander doesn't configured explicitly.
Information about configured failureHandler is printed in FailureProcessor
logic. I think we could remove failureHandler classname from such log messages.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
