(ignite) branch IGNITE-13202__jdk11_javadoc updated (59a741f2652 -> f94dfb32e9a)
This is an automated email from the ASF dual-hosted git repository. timoninmaxim pushed a change to branch IGNITE-13202__jdk11_javadoc in repository https://gitbox.apache.org/repos/asf/ignite.git discard 59a741f2652 IGNITE-13202 Fix javadoc exclusion for java11 add f94dfb32e9a IGNITE-13202 Fix javadoc exclusion for java11 This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (59a741f2652) \ N -- N -- N refs/heads/IGNITE-13202__jdk11_javadoc (f94dfb32e9a) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. No new revisions were added by this update. Summary of changes: modules/ducktests/pom.xml | 2 +- parent/pom.xml| 2 -- 2 files changed, 1 insertion(+), 3 deletions(-)
(ignite) branch IGNITE-13202__jdk11_javadoc updated (86f6ae2fdd4 -> 59a741f2652)
This is an automated email from the ASF dual-hosted git repository. timoninmaxim pushed a change to branch IGNITE-13202__jdk11_javadoc in repository https://gitbox.apache.org/repos/asf/ignite.git discard 86f6ae2fdd4 IGNITE-13202 Fix javadoc exclusion for java11 add 59a741f2652 IGNITE-13202 Fix javadoc exclusion for java11 This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (86f6ae2fdd4) \ N -- N -- N refs/heads/IGNITE-13202__jdk11_javadoc (59a741f2652) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. No new revisions were added by this update. Summary of changes: modules/yardstick/pom-standalone.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
(ignite) branch IGNITE-13202__jdk11_javadoc created (now 86f6ae2fdd4)
This is an automated email from the ASF dual-hosted git repository. timoninmaxim pushed a change to branch IGNITE-13202__jdk11_javadoc in repository https://gitbox.apache.org/repos/asf/ignite.git at 86f6ae2fdd4 IGNITE-13202 Fix javadoc exclusion for java11 This branch includes the following new commits: new 86f6ae2fdd4 IGNITE-13202 Fix javadoc exclusion for java11 The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
(ignite) 01/01: IGNITE-13202 Fix javadoc exclusion for java11
This is an automated email from the ASF dual-hosted git repository.
timoninmaxim pushed a commit to branch IGNITE-13202__jdk11_javadoc
in repository https://gitbox.apache.org/repos/asf/ignite.git
commit 86f6ae2fdd424f96ceb0557aeb5641fec38478fb
Author: Maksim Timonin
AuthorDate: Thu Jun 20 16:48:30 2024 +0300
IGNITE-13202 Fix javadoc exclusion for java11
---
modules/ducktests/pom.xml| 10
modules/yardstick/pom-standalone.xml | 2 +-
parent/pom.xml | 4 ++--
pom.xml | 46 +++-
4 files changed, 58 insertions(+), 4 deletions(-)
diff --git a/modules/ducktests/pom.xml b/modules/ducktests/pom.xml
index e5756e74872..2fa27c9f3b6 100644
--- a/modules/ducktests/pom.xml
+++ b/modules/ducktests/pom.xml
@@ -163,6 +163,16 @@
+
+
+org.apache.maven.plugins
+maven-javadoc-plugin
+${maven.javadoc.plugin.version}
+
+true
+
+
+
diff --git a/modules/yardstick/pom-standalone.xml
b/modules/yardstick/pom-standalone.xml
index c667650c325..755ce7bd929 100644
--- a/modules/yardstick/pom-standalone.xml
+++ b/modules/yardstick/pom-standalone.xml
@@ -175,7 +175,7 @@
org.apache.maven.plugins
maven-javadoc-plugin
-2.9.1
+${maven.javadoc.plugin.version}
true
diff --git a/parent/pom.xml b/parent/pom.xml
index 1fe2f1731ff..9dbf3472d50 100644
--- a/parent/pom.xml
+++ b/parent/pom.xml
@@ -915,6 +915,8 @@
--add-exports
java.base/sun.nio.ch=ALL-UNNAMED
--add-exports
+java.base/sun.net.util=ALL-UNNAMED
+--add-exports
java.management/com.sun.jmx.mbeanserver=ALL-UNNAMED
--add-exports
jdk.internal.jvmstat/sun.jvmstat.monitor=ALL-UNNAMED
@@ -945,9 +947,7 @@
org.apache.maven.plugins
maven-javadoc-plugin
-
${javadoc.opts}
--add-exports=java.base/sun.nio.ch=ALL-UNNAMED
--add-exports=java.management/com.sun.jmx.mbeanserver=ALL-UNNAMED
-false
diff --git a/pom.xml b/pom.xml
index de4d928f2dc..d90d3129a62 100644
--- a/pom.xml
+++ b/pom.xml
@@ -160,7 +160,51 @@
${basedir}/target/javadoc
core
-org.apache.ignite -exclude
org.apache.ignite.codegen:org.apache.ignite.examples:org.apache.ignite.internal:org.apache.ignite.schema:org.apache.ignite.tests:org.apache.ignite.tools:org.apache.ignite.util:org.apache.ignite.spi.discovery.tcp.messages:org.apache.ignite.spi.discovery.tcp.internal:org.apache.ignite.spi.communication.tcp.internal:org.apache.ignite.spi.discovery.zk.internal:org.apache.ignite.spi.deployment.uri.scanners:org.apache.
[...]
+
+
+
+
+io
+:io.*
+:org.apache.calcite
+:org.apache.calcite.*
+:org.jsr166
+:org.jsr166.*
+:org.mindrot
+:org.mindrot.*
+
+
+:org.apache.ignite.codegen
+:org.apache.ignite.codegen.*
+:org.apache.ignite.examples
+:org.apache.ignite.examples.*
+:org.apache.ignite.internal
+:org.apache.ignite.internal.*
+:org.apache.ignite.schema
+:org.apache.ignite.schema.*
+:org.apache.ignite.tests
+:org.apache.ignite.tests.*
+:org.apache.ignite.tools
+:org.apache.ignite.tools.*
+:org.apache.ignite.util
+:org.apache.ignite.util
(ignite) branch master updated: IGNITE-21445 Optimize iterator for local IndexQuery (#11317)
This is an automated email from the ASF dual-hosted git repository.
timoninmaxim pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/ignite.git
The following commit(s) were added to refs/heads/master by this push:
new f8c6dc4685a IGNITE-21445 Optimize iterator for local IndexQuery
(#11317)
f8c6dc4685a is described below
commit f8c6dc4685aa5f250cb64542c1e9c3cfa56c12bf
Author: oleg-vlsk <153691984+oleg-v...@users.noreply.github.com>
AuthorDate: Mon Jun 17 17:48:17 2024 +1000
IGNITE-21445 Optimize iterator for local IndexQuery (#11317)
---
.../processors/cache/IgniteCacheProxyImpl.java | 12
.../cache/query/GridCacheQueryManager.java | 76 ++
.../ignite/cache/query/IndexQueryLocalTest.java| 15 -
.../cache/query/IndexQueryPartitionTest.java | 6 +-
4 files changed, 106 insertions(+), 3 deletions(-)
diff --git
a/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/IgniteCacheProxyImpl.java
b/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/IgniteCacheProxyImpl.java
index fe28eb70705..2d2533a29a2 100644
---
a/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/IgniteCacheProxyImpl.java
+++
b/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/IgniteCacheProxyImpl.java
@@ -81,6 +81,7 @@ import org.apache.ignite.internal.IgniteInternalFuture;
import org.apache.ignite.internal.processors.affinity.AffinityTopologyVersion;
import org.apache.ignite.internal.processors.cache.query.CacheQuery;
import org.apache.ignite.internal.processors.cache.query.CacheQueryFuture;
+import org.apache.ignite.internal.processors.cache.query.GridCacheQueryAdapter;
import org.apache.ignite.internal.processors.cache.query.GridCacheQueryType;
import org.apache.ignite.internal.processors.cache.query.QueryCursorEx;
import org.apache.ignite.internal.processors.query.GridQueryFieldMetadata;
@@ -557,6 +558,17 @@ public class IgniteCacheProxyImpl extends
AsyncSupportAdapter 0)
qry.limit(q.getLimit());
+if (query.isLocal()) {
+final GridCloseableIterator iter =
ctx.kernalContext().query().executeQuery(GridCacheQueryType.INDEX,
+cacheName, ctx, new
IgniteOutClosureX() {
+@Override public GridCloseableIterator applyx() throws
IgniteCheckedException {
+return
ctx.queries().indexQueryLocal((GridCacheQueryAdapter)qry);
+}
+}, true);
+
+return new QueryCursorImpl(iter);
+}
+
fut =
ctx.kernalContext().query().executeQuery(GridCacheQueryType.INDEX,
q.getValueType(), ctx,
new IgniteOutClosureX>>() {
@Override public CacheQueryFuture>
applyx() {
diff --git
a/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/query/GridCacheQueryManager.java
b/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/query/GridCacheQueryManager.java
index b08fa52496c..ef6686253f6 100644
---
a/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/query/GridCacheQueryManager.java
+++
b/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/query/GridCacheQueryManager.java
@@ -127,6 +127,7 @@ import org.apache.ignite.lang.IgniteClosure;
import org.apache.ignite.lang.IgniteProductVersion;
import org.apache.ignite.lang.IgniteReducer;
import org.apache.ignite.lang.IgniteUuid;
+import org.apache.ignite.plugin.security.SecurityPermission;
import org.apache.ignite.resources.IgniteInstanceResource;
import org.apache.ignite.spi.IgniteSpiCloseableIterator;
import org.apache.ignite.spi.indexing.IndexingQueryFilter;
@@ -1445,6 +1446,81 @@ public abstract class GridCacheQueryManager
extends GridCacheManagerAdapte
}
}
+/**
+ * Process local index query.
+ *
+ * @param qry Query.
+ * @return GridCloseableIterator.
+ */
+@SuppressWarnings({"unchecked"})
+public GridCloseableIterator indexQueryLocal(final GridCacheQueryAdapter
qry) throws IgniteCheckedException {
+if (!enterBusy())
+throw new IllegalStateException("Failed to process query request
(grid is stopping).");
+
+try {
+assert qry.type() == INDEX : "Wrong processing of query: " +
qry.type();
+
+cctx.checkSecurity(SecurityPermission.CACHE_READ);
+
+if (cctx.localNode().isClient())
+throw new IgniteException("Failed to execute local index query
on a client node.");
+
+final Integer part = qry.partition();
+
+int[] parts = null;
+
+if (part != null) {
+final GridDhtLocalPartition locPart =
cctx.dht().topology().localPartition(part);
+
+if (locPart == null || locP
(ignite) branch master updated: IGNITE-22420 Fix expireTime setting for thin client replication (#11376)
This is an automated email from the ASF dual-hosted git repository.
timoninmaxim pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/ignite.git
The following commit(s) were added to refs/heads/master by this push:
new 42b3285bbbf IGNITE-22420 Fix expireTime setting for thin client
replication (#11376)
42b3285bbbf is described below
commit 42b3285bbbf1b7c423b380c4b30b8c77f7f64ddb
Author: Maksim Timonin
AuthorDate: Thu Jun 13 09:01:19 2024 +0300
IGNITE-22420 Fix expireTime setting for thin client replication (#11376)
---
.../cache/ClientCachePutAllConflictRequest.java| 8 +---
.../client/thin/DataReplicationOperationsTest.java | 51 ++
2 files changed, 23 insertions(+), 36 deletions(-)
diff --git
a/modules/core/src/main/java/org/apache/ignite/internal/processors/platform/client/cache/ClientCachePutAllConflictRequest.java
b/modules/core/src/main/java/org/apache/ignite/internal/processors/platform/client/cache/ClientCachePutAllConflictRequest.java
index ac3182b751b..ba133bbbadb 100644
---
a/modules/core/src/main/java/org/apache/ignite/internal/processors/platform/client/cache/ClientCachePutAllConflictRequest.java
+++
b/modules/core/src/main/java/org/apache/ignite/internal/processors/platform/client/cache/ClientCachePutAllConflictRequest.java
@@ -33,8 +33,6 @@ import
org.apache.ignite.internal.processors.platform.client.tx.ClientTxAwareReq
import org.apache.ignite.internal.util.typedef.internal.CU;
import org.apache.ignite.internal.util.typedef.internal.U;
-import static
org.apache.ignite.internal.processors.cache.GridCacheUtils.EXPIRE_TIME_CALCULATE;
-import static
org.apache.ignite.internal.processors.cache.GridCacheUtils.TTL_NOT_CHANGED;
import static
org.apache.ignite.internal.processors.platform.utils.PlatformUtils.readCacheObject;
/**
@@ -53,8 +51,6 @@ public class ClientCachePutAllConflictRequest extends
ClientCacheDataRequest imp
public ClientCachePutAllConflictRequest(BinaryReaderExImpl reader,
ClientConnectionContext ctx) {
super(reader);
-boolean expPlc = cachex(ctx).configuration().getExpiryPolicyFactory()
!= null;
-
int cnt = reader.readInt();
map = new LinkedHashMap<>(cnt);
@@ -67,9 +63,7 @@ public class ClientCachePutAllConflictRequest extends
ClientCacheDataRequest imp
GridCacheDrInfo info = expireTime != CU.EXPIRE_TIME_ETERNAL ?
new GridCacheDrExpirationInfo(val, ver, CU.TTL_ETERNAL,
expireTime) :
-(expPlc
-? new GridCacheDrExpirationInfo(val, ver, TTL_NOT_CHANGED,
EXPIRE_TIME_CALCULATE)
-: new GridCacheDrInfo(val, ver));
+new GridCacheDrInfo(val, ver);
map.put(key, info);
}
diff --git
a/modules/core/src/test/java/org/apache/ignite/internal/client/thin/DataReplicationOperationsTest.java
b/modules/core/src/test/java/org/apache/ignite/internal/client/thin/DataReplicationOperationsTest.java
index c0c78101d9e..3405d3563be 100644
---
a/modules/core/src/test/java/org/apache/ignite/internal/client/thin/DataReplicationOperationsTest.java
+++
b/modules/core/src/test/java/org/apache/ignite/internal/client/thin/DataReplicationOperationsTest.java
@@ -27,7 +27,6 @@ import org.apache.ignite.client.ClientCacheConfiguration;
import org.apache.ignite.client.IgniteClient;
import org.apache.ignite.client.Person;
import org.apache.ignite.internal.processors.cache.version.GridCacheVersion;
-import
org.apache.ignite.internal.processors.platform.cache.expiry.PlatformExpiryPolicy;
import org.apache.ignite.internal.util.typedef.T3;
import org.apache.ignite.internal.util.typedef.internal.CU;
import org.junit.Test;
@@ -139,33 +138,6 @@ public class DataReplicationOperationsTest extends
AbstractThinClientTest {
map.keySet().forEach(key -> assertFalse(cache.containsKey(key)));
}
-/** @throws Exception If fails. */
-@Test
-public void testWithExpiryPolicy() throws Exception {
-PlatformExpiryPolicy expPlc = new PlatformExpiryPolicy(TTL, TTL, TTL);
-
-ClientCacheConfiguration ccfgWithExpPlc = new
ClientCacheConfiguration()
-.setName("cache-with-expiry-policy")
-.setAtomicityMode(mode)
-.setExpiryPolicy(expPlc);
-
-TcpClientCache cache = (TcpClientCache)client.getOrCreateCache(ccfgWithExpPlc);
-
-TcpClientCache cacheWithExpPlc = binary ?
-(TcpClientCache)cache.withKeepBinary() : cache;
-
-Map> data =
createPutAllData(CU.EXPIRE_TIME_ETERNAL);
-
-cacheWithExpPlc.putAllConflict(data);
-
-assertTrue(cacheWithExpPlc.containsKeys(data.keySet()));
-
-assertTrue(waitForCondition(
-() ->
data.keySet().stream().noneMatch(cacheWithExpPlc::containsKey),
-2 * expPlc.getExpiryForCreation().getDurationAmount()
-));
-}
-
/** @throws Exception If fails. */
(ignite) branch master updated: IGNITE-22412: Unify snapshot validation jobs (#11368)
This is an automated email from the ASF dual-hosted git repository.
timoninmaxim pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/ignite.git
The following commit(s) were added to refs/heads/master by this push:
new 8be2e2cd54f IGNITE-22412: Unify snapshot validation jobs (#11368)
8be2e2cd54f is described below
commit 8be2e2cd54f7a7a8930086f5476c7bca0663d4ed
Author: Vladimir Steshin
AuthorDate: Thu Jun 6 09:11:52 2024 +0300
IGNITE-22412: Unify snapshot validation jobs (#11368)
---
.../snapshot/AbstractSnapshotVerificationTask.java | 86 --
.../IncrementalSnapshotVerificationTask.java | 51 ++---
.../snapshot/SnapshotHandlerContext.java | 6 +-
.../snapshot/SnapshotHandlerRestoreTask.java | 60 ++-
.../snapshot/SnapshotPartitionsVerifyTask.java | 50 ++---
.../snapshot/SnapshotPartitionsVerifyTaskArg.java | 10 +--
6 files changed, 89 insertions(+), 174 deletions(-)
diff --git
a/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/snapshot/AbstractSnapshotVerificationTask.java
b/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/snapshot/AbstractSnapshotVerificationTask.java
index cff0b0e110d..e8d348fc6da 100644
---
a/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/snapshot/AbstractSnapshotVerificationTask.java
+++
b/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/snapshot/AbstractSnapshotVerificationTask.java
@@ -24,14 +24,17 @@ import java.util.List;
import java.util.Map;
import java.util.Set;
import org.apache.ignite.IgniteException;
+import org.apache.ignite.IgniteLogger;
import org.apache.ignite.cluster.ClusterNode;
import org.apache.ignite.compute.ComputeJob;
+import org.apache.ignite.compute.ComputeJobAdapter;
import org.apache.ignite.compute.ComputeJobResult;
import org.apache.ignite.compute.ComputeJobResultPolicy;
import org.apache.ignite.compute.ComputeTaskAdapter;
import org.apache.ignite.internal.IgniteEx;
import org.apache.ignite.internal.util.typedef.F;
import org.apache.ignite.resources.IgniteInstanceResource;
+import org.apache.ignite.resources.LoggerResource;
import org.jetbrains.annotations.Nullable;
/**
@@ -49,6 +52,10 @@ public abstract class AbstractSnapshotVerificationTask
extends
@IgniteInstanceResource
protected IgniteEx ignite;
+/** Injected logger. */
+@LoggerResource
+protected IgniteLogger log;
+
/** {@inheritDoc} */
@Override public Map map(List
subgrid, SnapshotPartitionsVerifyTaskArg arg) {
Map> clusterMetas =
arg.clusterMetadata();
@@ -72,17 +79,7 @@ public abstract class AbstractSnapshotVerificationTask
extends
if (meta == null)
continue;
-jobs.put(
-createJob(
-meta.snapshotName(),
-arg.snapshotPath(),
-arg.incrementIndex(),
-meta.consistentId(),
-arg.cacheGroupNames(),
-arg.check()
-),
-e.getKey()
-);
+jobs.put(createJob(meta.snapshotName(), meta.consistentId(),
arg), e.getKey());
if (allMetas.isEmpty())
break;
@@ -100,19 +97,60 @@ public abstract class AbstractSnapshotVerificationTask
extends
/**
* @param name Snapshot name.
- * @param path Snapshot directory path.
- * @param incIdx Incremental snapshot index.
- * @param constId Snapshot metadata file name.
- * @param groups Cache groups to be restored from the snapshot. May be
empty if all cache groups are being restored.
- * @param check If {@code true} check snapshot before restore.
+ * @param consId Consistent id of the related node.
+ * @param args Check snapshot parameters.
+ *
* @return Compute job.
*/
-protected abstract ComputeJob createJob(
-String name,
-@Nullable String path,
-int incIdx,
-String constId,
-Collection groups,
-boolean check
-);
+protected abstract AbstractSnapshotVerificationJob createJob(String name,
String consId, SnapshotPartitionsVerifyTaskArg args);
+
+/** */
+protected abstract static class AbstractSnapshotVerificationJob extends
ComputeJobAdapter {
+/** Serial version uid. */
+private static final long serialVersionUID = 0L;
+
+/** Ignite instance. */
+@IgniteInstanceResource
+protected IgniteEx ignite;
+
+/** Injected logger. */
+@LoggerResource
+protected IgniteLogger log;
+
+/** Snapshot name. */
+protected final String snpName;
+
+/** Snapshot directory path. */
+@Nullable protected final String snpP
(ignite-extensions) branch master updated: IGNITE-19613 Add log4j-slf4j-impl dependency into ignite-cdc-ext (#268)
This is an automated email from the ASF dual-hosted git repository.
timoninmaxim pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/ignite-extensions.git
The following commit(s) were added to refs/heads/master by this push:
new 03eb0724 IGNITE-19613 Add log4j-slf4j-impl dependency into
ignite-cdc-ext (#268)
03eb0724 is described below
commit 03eb0724d600322e7ee821d650f8694ab7ee7d90
Author: Andrey N
AuthorDate: Wed Jun 5 14:43:22 2024 +0300
IGNITE-19613 Add log4j-slf4j-impl dependency into ignite-cdc-ext (#268)
---
modules/cdc-ext/pom.xml | 6 ++
1 file changed, 6 insertions(+)
diff --git a/modules/cdc-ext/pom.xml b/modules/cdc-ext/pom.xml
index 614a303a..c8052fe2 100644
--- a/modules/cdc-ext/pom.xml
+++ b/modules/cdc-ext/pom.xml
@@ -64,6 +64,12 @@
${snappy.version}
+
+org.apache.logging.log4j
+log4j-slf4j-impl
+${log4j2.version}
+
+
org.apache.ignite
ignite-log4j2
(ignite) branch master updated: IGNITE-22140 Add test for IndexQuery pagination (#11350)
This is an automated email from the ASF dual-hosted git repository.
timoninmaxim pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/ignite.git
The following commit(s) were added to refs/heads/master by this push:
new dfff5d7675e IGNITE-22140 Add test for IndexQuery pagination (#11350)
dfff5d7675e is described below
commit dfff5d7675ec30deba1b09acb388efdb7edc6da3
Author: oleg-vlsk <153691984+oleg-v...@users.noreply.github.com>
AuthorDate: Mon Jun 3 18:08:22 2024 +1000
IGNITE-22140 Add test for IndexQuery pagination (#11350)
---
.../cache/query/IndexQueryPaginationTest.java | 159 +
.../ignite/cache/query/IndexQueryTestSuite.java| 3 +-
.../cache/query/ThinClientIndexQueryTest.java | 2 +-
3 files changed, 162 insertions(+), 2 deletions(-)
diff --git
a/modules/indexing/src/test/java/org/apache/ignite/cache/query/IndexQueryPaginationTest.java
b/modules/indexing/src/test/java/org/apache/ignite/cache/query/IndexQueryPaginationTest.java
new file mode 100644
index 000..1e20a7b5b67
--- /dev/null
+++
b/modules/indexing/src/test/java/org/apache/ignite/cache/query/IndexQueryPaginationTest.java
@@ -0,0 +1,159 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cache.query;
+
+import java.util.ArrayList;
+import java.util.List;
+import javax.cache.Cache;
+import org.apache.ignite.Ignite;
+import org.apache.ignite.IgniteCache;
+import org.apache.ignite.IgniteDataStreamer;
+import org.apache.ignite.cache.CacheMode;
+import org.apache.ignite.cache.query.annotations.QuerySqlField;
+import org.apache.ignite.configuration.CacheConfiguration;
+import org.apache.ignite.configuration.IgniteConfiguration;
+import org.apache.ignite.internal.TestRecordingCommunicationSpi;
+import org.apache.ignite.internal.processors.cache.query.GridCacheQueryRequest;
+import
org.apache.ignite.internal.processors.cache.query.GridCacheQueryResponse;
+import org.apache.ignite.testframework.junits.common.GridCommonAbstractTest;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import static
org.apache.ignite.cache.query.ThinClientIndexQueryTest.getFilteredMessages;
+
+/** */
+@RunWith(Parameterized.class)
+public class IndexQueryPaginationTest extends GridCommonAbstractTest {
+/** */
+private static final int PAGE_SIZE = 512;
+
+/** */
+private static final int NODES = 2;
+
+/** */
+private Ignite grid;
+
+/** */
+private IgniteCache cache;
+
+/** */
+@Parameterized.Parameter
+public int entries;
+
+/** */
+@Parameterized.Parameters(name = "entries={0}")
+public static Object[] params() {
+return new Object[] {100, 1000, 5000, 10_000, 50_000, 100_000};
+}
+
+/** {@inheritDoc} */
+@Override protected void beforeTest() throws Exception {
+grid = startGrids(NODES);
+
+cache = grid.cache("cache");
+
+insertData(grid, cache, entries);
+}
+
+/** {@inheritDoc} */
+@Override protected void afterTest() throws Exception {
+stopAllGrids();
+}
+
+/** {@inheritDoc} */
+@Override protected IgniteConfiguration getConfiguration(String
igniteInstanceName) throws Exception {
+CacheConfiguration ccfg = new
CacheConfiguration("cache")
+.setIndexedTypes(Integer.class, Person.class)
+.setCacheMode(CacheMode.PARTITIONED);
+
+return super.getConfiguration(igniteInstanceName)
+.setCommunicationSpi(new TestRecordingCommunicationSpi())
+.setCacheConfiguration(ccfg);
+}
+
+/**
+ * Insert data into a cache.
+ *
+ * @param ignite Ignite instance.
+ * @param cache Cache.
+ * @param entries Number of entries.
+ */
+private void insertData(Ignite ignite, IgniteCache cache,
int entries) {
+try (IgniteDataStreamer streamer =
ignite.dataStreamer(cache.getName())) {
+for (int i = 0; i < entries; i++)
+streamer.addData(i, new Person(i));
+}
+}
+
+/**
+ * Check if the number and the size of next page request
(ignite) branch master updated: IGNITE-22182 : Move snapshop metadata validation to SnapshotMetadataVerificationTask (#11346)
This is an automated email from the ASF dual-hosted git repository.
timoninmaxim pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/ignite.git
The following commit(s) were added to refs/heads/master by this push:
new 370693a0c0e IGNITE-22182 : Move snapshop metadata validation to
SnapshotMetadataVerificationTask (#11346)
370693a0c0e is described below
commit 370693a0c0e02d1b8f8c144cc8d2e1c7b3721fc1
Author: Vladimir Steshin
AuthorDate: Wed May 29 14:14:18 2024 +0300
IGNITE-22182 : Move snapshop metadata validation to
SnapshotMetadataVerificationTask (#11346)
---
.../snapshot/SnapshotCompressionBasicTest.java | 11 ++-
...CommandHandlerCheckIncrementalSnapshotTest.java | 4 +-
.../apache/ignite/util/GridCommandHandlerTest.java | 6 +-
.../snapshot/AbstractSnapshotVerificationTask.java | 32 ---
.../snapshot/IgniteSnapshotManager.java| 81 ++--
.../snapshot/SnapshotMetadataVerificationTask.java | 103 +++--
.../SnapshotMetadataVerificationTaskArg.java | 23 -
.../snapshot/SnapshotPartitionsVerifyHandler.java | 17 +---
.../snapshot/SnapshotRestoreProcess.java | 34 +--
.../cache/persistence/snapshot/dump/Dump.java | 2 +-
.../snapshot/EncryptedSnapshotTest.java| 36 +++
.../snapshot/IgniteClusterSnapshotCheckTest.java | 14 +--
.../snapshot/dump/IgniteCacheDumpSelf2Test.java| 7 +-
.../IncrementalSnapshotCheckBeforeRestoreTest.java | 24 +++--
.../IncrementalSnapshotRestoreTest.java| 10 +-
15 files changed, 189 insertions(+), 215 deletions(-)
diff --git
a/modules/compress/src/test/java/org/apache/ignite/internal/processors/cache/persistence/snapshot/SnapshotCompressionBasicTest.java
b/modules/compress/src/test/java/org/apache/ignite/internal/processors/cache/persistence/snapshot/SnapshotCompressionBasicTest.java
index 9820303e51a..08a4acbb1cd 100644
---
a/modules/compress/src/test/java/org/apache/ignite/internal/processors/cache/persistence/snapshot/SnapshotCompressionBasicTest.java
+++
b/modules/compress/src/test/java/org/apache/ignite/internal/processors/cache/persistence/snapshot/SnapshotCompressionBasicTest.java
@@ -217,9 +217,14 @@ public class SnapshotCompressionBasicTest extends
AbstractSnapshotSelfTest {
G.allGrids().forEach(this::failCompressionProcessor);
-for (String snpName: Arrays.asList(SNAPSHOT_WITH_HOLES,
SNAPSHOT_WITHOUT_HOLES)) {
-GridTestUtils.assertThrows(log, () ->
ignite.snapshot().restoreSnapshot(snpName, null).get(TIMEOUT),
-IgniteException.class, "Snapshot contains compressed cache
groups");
+for (String snpName : Arrays.asList(SNAPSHOT_WITH_HOLES,
SNAPSHOT_WITHOUT_HOLES)) {
+GridTestUtils.assertThrowsAnyCause(
+log,
+() -> ignite.snapshot().restoreSnapshot(snpName,
null).get(TIMEOUT),
+IllegalStateException.class,
+"from snapshot '" + snpName + "' are compressed while disk
page compression is disabled. To check " +
+"these groups please start Ignite with ignite-compress"
+);
}
}
diff --git
a/modules/control-utility/src/test/java/org/apache/ignite/util/GridCommandHandlerCheckIncrementalSnapshotTest.java
b/modules/control-utility/src/test/java/org/apache/ignite/util/GridCommandHandlerCheckIncrementalSnapshotTest.java
index 6fe71e0fc46..cbee596138c 100644
---
a/modules/control-utility/src/test/java/org/apache/ignite/util/GridCommandHandlerCheckIncrementalSnapshotTest.java
+++
b/modules/control-utility/src/test/java/org/apache/ignite/util/GridCommandHandlerCheckIncrementalSnapshotTest.java
@@ -54,7 +54,6 @@ import org.junit.runners.Parameterized;
import static org.apache.ignite.cluster.ClusterState.ACTIVE;
import static
org.apache.ignite.internal.commandline.CommandHandler.EXIT_CODE_INVALID_ARGUMENTS;
import static
org.apache.ignite.internal.commandline.CommandHandler.EXIT_CODE_OK;
-import static
org.apache.ignite.internal.commandline.CommandHandler.EXIT_CODE_UNEXPECTED_ERROR;
import static org.apache.ignite.testframework.GridTestUtils.assertContains;
import static org.apache.ignite.testframework.GridTestUtils.assertNotContains;
import static
org.apache.ignite.testframework.GridTestUtils.runMultiThreadedAsync;
@@ -183,7 +182,8 @@ public class GridCommandHandlerCheckIncrementalSnapshotTest
extends GridCommandH
assertContains(log, testOut.toString(), "--increment argument
specified twice");
// Non existent increment.
-assertEquals(EXIT_CODE_UNEXPECTED_ERROR, execute(cmd, "--snapshot",
"check", SNP, "--increment", "2"));
+assertEquals(EXIT_CODE_INVALID_ARGUMENTS, execute(cmd, "--snapshot",
"check", SNP, "--increment", "2"))
(ignite) branch master updated: IGNITE-22299 Drop GridCacheVersion#otherClusterVersion for cache resending (#11360)
This is an automated email from the ASF dual-hosted git repository.
timoninmaxim pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/ignite.git
The following commit(s) were added to refs/heads/master by this push:
new 67dd34efe32 IGNITE-22299 Drop GridCacheVersion#otherClusterVersion for
cache resending (#11360)
67dd34efe32 is described below
commit 67dd34efe329ed11cb252f0f9d3a5598287b818d
Author: Maksim Timonin
AuthorDate: Mon May 27 22:07:50 2024 +0300
IGNITE-22299 Drop GridCacheVersion#otherClusterVersion for cache resending
(#11360)
---
.../apache/ignite/util/CdcResendCommandTest.java | 182 +
.../management/cdc/CdcCacheDataResendTask.java | 10 +-
2 files changed, 191 insertions(+), 1 deletion(-)
diff --git
a/modules/control-utility/src/test/java/org/apache/ignite/util/CdcResendCommandTest.java
b/modules/control-utility/src/test/java/org/apache/ignite/util/CdcResendCommandTest.java
index 81cb9864e6c..c333fd3ed0d 100644
---
a/modules/control-utility/src/test/java/org/apache/ignite/util/CdcResendCommandTest.java
+++
b/modules/control-utility/src/test/java/org/apache/ignite/util/CdcResendCommandTest.java
@@ -17,12 +17,41 @@
package org.apache.ignite.util;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import org.apache.ignite.binary.BinaryType;
import org.apache.ignite.cdc.AbstractCdcTest;
+import org.apache.ignite.cdc.CdcCacheEvent;
+import org.apache.ignite.cdc.CdcConfiguration;
+import org.apache.ignite.cdc.CdcConsumer;
+import org.apache.ignite.cdc.CdcEvent;
+import org.apache.ignite.cdc.TypeMapping;
import org.apache.ignite.configuration.CacheConfiguration;
import org.apache.ignite.configuration.DataRegionConfiguration;
import org.apache.ignite.configuration.DataStorageConfiguration;
import org.apache.ignite.configuration.IgniteConfiguration;
import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.cdc.CdcMain;
+import
org.apache.ignite.internal.processors.cache.CacheConflictResolutionManager;
+import org.apache.ignite.internal.processors.cache.CacheObject;
+import org.apache.ignite.internal.processors.cache.CacheObjectImpl;
+import org.apache.ignite.internal.processors.cache.CacheObjectValueContext;
+import org.apache.ignite.internal.processors.cache.GridCacheManagerAdapter;
+import org.apache.ignite.internal.processors.cache.IgniteInternalCache;
+import org.apache.ignite.internal.processors.cache.KeyCacheObject;
+import org.apache.ignite.internal.processors.cache.KeyCacheObjectImpl;
+import org.apache.ignite.internal.processors.cache.dr.GridCacheDrInfo;
+import
org.apache.ignite.internal.processors.cache.version.CacheVersionConflictResolver;
+import org.apache.ignite.internal.processors.cache.version.GridCacheVersion;
+import
org.apache.ignite.internal.processors.cache.version.GridCacheVersionConflictContext;
+import
org.apache.ignite.internal.processors.cache.version.GridCacheVersionedEntryEx;
+import org.apache.ignite.metric.MetricRegistry;
+import org.apache.ignite.plugin.AbstractTestPluginProvider;
+import org.apache.ignite.plugin.PluginContext;
+import org.apache.ignite.testframework.GridTestUtils;
import org.junit.Test;
import static org.apache.ignite.cdc.AbstractCdcTest.KEYS_CNT;
@@ -53,9 +82,27 @@ public class CdcResendCommandTest extends
GridCommandHandlerAbstractTest {
cfg.setCacheConfiguration(new CacheConfiguration<>(DEFAULT_CACHE_NAME)
.setBackups(1));
+cfg.setPluginProviders(new AbstractTestPluginProvider() {
+@Override public String name() {
+return "ConflictResolverProvider";
+}
+
+@Override public T createComponent(PluginContext ctx, Class
cls) {
+if (cls != CacheConflictResolutionManager.class)
+return null;
+
+return (T)new AlwaysNewResolutionManager<>();
+}
+});
+
return cfg;
}
+/** {@inheritDoc} */
+@Override protected void beforeTest() throws Exception {
+cleanPersistenceDir();
+}
+
/** {@inheritDoc} */
@Override protected void afterTest() throws Exception {
super.afterTest();
@@ -94,4 +141,139 @@ public class CdcResendCommandTest extends
GridCommandHandlerAbstractTest {
assertEquals(KEYS_CNT, ign.cache(DEFAULT_CACHE_NAME).size());
}
+
+/** */
+@Test
+public void testResendConflictVersion() throws Exception {
+IgniteEx ign = startGrid(0);
+
+ign.cluster().state(ACTIVE);
+
+enableCheckpoints(ign, false);
+
+IgniteInternalCache cachex =
ign.cachex(DEFAULT_CACHE_NAME);
+
+// Put data.
+cachex.put(0, 0);
+
+// Override data from clusterId=2.
+KeyCacheObject key = new KeyCacheObjectImpl(0, null,
cachex.affinity().partition(0));
+
(ignite) branch master updated: IGNITE-22215 Fix for ThinClientIndexQueryTest#testPageSize (#11349)
This is an automated email from the ASF dual-hosted git repository.
timoninmaxim pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/ignite.git
The following commit(s) were added to refs/heads/master by this push:
new 987d1a1a3ab IGNITE-22215 Fix for ThinClientIndexQueryTest#testPageSize
(#11349)
987d1a1a3ab is described below
commit 987d1a1a3ab7e5a634a5e7ee4357d1004ee2bb78
Author: oleg-vlsk <153691984+oleg-v...@users.noreply.github.com>
AuthorDate: Tue May 28 02:33:55 2024 +1000
IGNITE-22215 Fix for ThinClientIndexQueryTest#testPageSize (#11349)
---
.../cache/query/ThinClientIndexQueryTest.java | 58 --
1 file changed, 54 insertions(+), 4 deletions(-)
diff --git
a/modules/indexing/src/test/java/org/apache/ignite/cache/query/ThinClientIndexQueryTest.java
b/modules/indexing/src/test/java/org/apache/ignite/cache/query/ThinClientIndexQueryTest.java
index 08cc8b76477..0d9ffed37b5 100644
---
a/modules/indexing/src/test/java/org/apache/ignite/cache/query/ThinClientIndexQueryTest.java
+++
b/modules/indexing/src/test/java/org/apache/ignite/cache/query/ThinClientIndexQueryTest.java
@@ -18,12 +18,14 @@
package org.apache.ignite.cache.query;
import java.lang.reflect.Field;
+import java.util.ArrayList;
import java.util.Collections;
import java.util.EnumSet;
import java.util.List;
import java.util.Random;
import java.util.function.Consumer;
import java.util.function.Function;
+import java.util.stream.Collectors;
import javax.cache.Cache;
import org.apache.ignite.Ignite;
import org.apache.ignite.IgniteDataStreamer;
@@ -40,6 +42,8 @@ import org.apache.ignite.configuration.ClientConfiguration;
import org.apache.ignite.configuration.IgniteConfiguration;
import org.apache.ignite.internal.TestRecordingCommunicationSpi;
import org.apache.ignite.internal.client.thin.ProtocolBitmaskFeature;
+import org.apache.ignite.internal.processors.cache.query.GridCacheQueryRequest;
+import
org.apache.ignite.internal.processors.cache.query.GridCacheQueryResponse;
import
org.apache.ignite.internal.processors.query.h2.twostep.messages.GridQueryNextPageRequest;
import org.apache.ignite.internal.util.tostring.GridToStringInclude;
import org.apache.ignite.internal.util.typedef.F;
@@ -224,14 +228,46 @@ public class ThinClientIndexQueryTest extends
GridCommonAbstractTest {
for (int pageSize: F.asList(1, 10, 100, 1000, 10_000)) {
idxQry.setPageSize(pageSize);
-
TestRecordingCommunicationSpi.spi(grid(0)).record(GridQueryNextPageRequest.class);
+for (int i = 0; i < NODES; i++) {
+TestRecordingCommunicationSpi.spi(grid(i)).record(
+GridCacheQueryRequest.class,
+GridCacheQueryResponse.class);
+}
assertClientQuery(cache, NULLS_CNT, CNT, idxQry);
-List reqs =
TestRecordingCommunicationSpi.spi(grid(0)).recordedMessages(true);
+int nodeOneEntries = cache.query(new ScanQuery().setLocal(true)).getAll().size();
+int nodeTwoEntries = (CNT - NULLS_CNT) - nodeOneEntries;
+
+int nodeOneExpectedReqs = (nodeOneEntries + pageSize - 1) /
pageSize;
+int nodeTwoExpectedReqs = (nodeTwoEntries + pageSize - 1) /
pageSize;
+
+int nodeOneLastPageEntries = nodeOneEntries % pageSize;
+int nodeTwoLastPageEntries = nodeTwoEntries % pageSize;
+
+List msgs = new ArrayList<>();
+
+for (int i = 0; i < NODES; i++)
+
msgs.addAll(TestRecordingCommunicationSpi.spi(grid(i)).recordedMessages(true));
+
+List reqs = getFilteredMessages(msgs,
GridCacheQueryRequest.class);
+List resp = getFilteredMessages(msgs,
GridCacheQueryResponse.class);
-for (Object r: reqs)
-assertEquals(pageSize,
((GridQueryNextPageRequest)r).pageSize());
+int reqsSize = reqs.size();
+
+assert (reqsSize == nodeOneExpectedReqs || reqsSize ==
nodeTwoExpectedReqs) && reqsSize == resp.size();
+
+for (int i = 0; i < reqsSize; i++) {
+int reqPage = reqs.get(i).pageSize();
+int respData = resp.get(i).data().size();
+
+assert reqPage == pageSize;
+
+if (i == reqsSize - 1 && (nodeOneLastPageEntries != 0 ||
nodeTwoLastPageEntries != 0))
+assert respData == nodeOneLastPageEntries || respData
== nodeTwoLastPageEntries;
+else
+assert respData == reqPage;
+}
}
for (int pageSize: F.asList(-10, -1, 0)) {
@@ -449,6 +485,20 @@ public class ThinClientIndexQueryTest extends
GridCommonAbstractTest {
}
}
+/**
+ * Filter messag
(ignite) branch master updated: IGNITE-22219 Remove useless SqlQuery code in GridCacheQueryManager (#11347)
This is an automated email from the ASF dual-hosted git repository.
timoninmaxim pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/ignite.git
The following commit(s) were added to refs/heads/master by this push:
new 9836647fa8e IGNITE-22219 Remove useless SqlQuery code in
GridCacheQueryManager (#11347)
9836647fa8e is described below
commit 9836647fa8e7a0dc47f87295dba3c5c1cb720e03
Author: Maksim Timonin
AuthorDate: Fri May 17 09:44:10 2024 +0300
IGNITE-22219 Remove useless SqlQuery code in GridCacheQueryManager (#11347)
---
.../cache/query/GridCacheQueryManager.java | 98 +-
1 file changed, 22 insertions(+), 76 deletions(-)
diff --git
a/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/query/GridCacheQueryManager.java
b/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/query/GridCacheQueryManager.java
index fec2585834a..b08fa52496c 100644
---
a/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/query/GridCacheQueryManager.java
+++
b/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/query/GridCacheQueryManager.java
@@ -114,7 +114,6 @@ import org.apache.ignite.internal.util.typedef.CIX1;
import org.apache.ignite.internal.util.typedef.F;
import org.apache.ignite.internal.util.typedef.P1;
import org.apache.ignite.internal.util.typedef.T2;
-import org.apache.ignite.internal.util.typedef.T3;
import org.apache.ignite.internal.util.typedef.X;
import org.apache.ignite.internal.util.typedef.internal.A;
import org.apache.ignite.internal.util.typedef.internal.CU;
@@ -147,7 +146,6 @@ import static
org.apache.ignite.internal.processors.cache.distributed.dht.topolo
import static
org.apache.ignite.internal.processors.cache.query.GridCacheQueryType.INDEX;
import static
org.apache.ignite.internal.processors.cache.query.GridCacheQueryType.SCAN;
import static
org.apache.ignite.internal.processors.cache.query.GridCacheQueryType.SPI;
-import static
org.apache.ignite.internal.processors.cache.query.GridCacheQueryType.SQL;
import static
org.apache.ignite.internal.processors.cache.query.GridCacheQueryType.SQL_FIELDS;
import static
org.apache.ignite.internal.processors.cache.query.GridCacheQueryType.TEXT;
import static
org.apache.ignite.internal.processors.security.SecurityUtils.securitySubjectId;
@@ -535,7 +533,6 @@ public abstract class GridCacheQueryManager extends
GridCacheManagerAdapte
* Performs query.
*
* @param qry Query.
- * @param args Arguments.
* @param loc Local query or not.
* @param taskName Task name.
* @param rcpt ID of the recipient.
@@ -543,7 +540,7 @@ public abstract class GridCacheQueryManager extends
GridCacheManagerAdapte
* @throws IgniteCheckedException In case of error.
*/
@SuppressWarnings("unchecked")
-private QueryResult executeQuery(GridCacheQueryAdapter qry,
@Nullable Object[] args,
+private QueryResult executeQuery(GridCacheQueryAdapter qry,
IgniteClosure transformer, boolean loc, @Nullable String taskName,
Object rcpt)
throws IgniteCheckedException {
if (qry.type() == null) {
@@ -554,33 +551,12 @@ public abstract class GridCacheQueryManager extends
GridCacheManagerAdapte
"CacheConfiguration.getMaxQueryIteratorsCount() configuration
property).");
}
-QueryResult res;
-
-T3> resKey = null;
-
-if (qry.type() == SQL) {
-resKey = new T3<>(qry.queryClassName(), qry.clause(),
F.asList(args));
-
-res = (QueryResult)qryResCache.get(resKey);
-
-if (res != null && res.addRecipient(rcpt))
-return res;
-
-res = new QueryResult<>(qry.type(), rcpt);
-
-if (qryResCache.putIfAbsent(resKey, res) != null)
-resKey = null;
-}
-else
-res = new QueryResult<>(qry.type(), rcpt);
+QueryResult res = new QueryResult<>(qry.type(), rcpt);
GridCloseableIterator> iter;
try {
switch (qry.type()) {
-case SQL:
-throw new IllegalStateException("Should never be called.");
-
case SCAN:
if (cctx.events().isRecordable(EVT_CACHE_QUERY_EXECUTED)) {
cctx.gridEvents().record(new CacheQueryExecutedEvent<>(
@@ -670,10 +646,6 @@ public abstract class GridCacheQueryManager extends
GridCacheManagerAdapte
catch (Exception e) {
res.onDone(e);
}
-finally {
-if (resKey != null)
-qryResCache.remove(resKey, res);
-}
return res;
}
@@ -1171,7 +1143,7 @@ public abstract class GridCacheQueryManager extends
GridCacheManagerAdapte
GridCacheQueryType type;
res = lo
(ignite) branch master updated: IGNITE-22183 Recording of IndexQuery events added (#11342)
This is an automated email from the ASF dual-hosted git repository.
timoninmaxim pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/ignite.git
The following commit(s) were added to refs/heads/master by this push:
new e1a7f8e3989 IGNITE-22183 Recording of IndexQuery events added (#11342)
e1a7f8e3989 is described below
commit e1a7f8e39894200857408aeafa2a4477058286ec
Author: oleg-vlsk <153691984+oleg-v...@users.noreply.github.com>
AuthorDate: Thu May 16 01:21:13 2024 +1000
IGNITE-22183 Recording of IndexQuery events added (#11342)
---
.../cache/query/index/IndexQueryProcessor.java | 30 +-
.../processors/cache/query/CacheQueryType.java | 5 +-
.../cache/query/GridCacheQueryManager.java | 18 +++-
.../processors/query/GridQueryProcessor.java | 6 +-
.../cache/IgniteCacheAbstractQuerySelfTest.java| 115 +
5 files changed, 169 insertions(+), 5 deletions(-)
diff --git
a/modules/core/src/main/java/org/apache/ignite/internal/cache/query/index/IndexQueryProcessor.java
b/modules/core/src/main/java/org/apache/ignite/internal/cache/query/index/IndexQueryProcessor.java
index 4048971a38a..bef88ec2e41 100644
---
a/modules/core/src/main/java/org/apache/ignite/internal/cache/query/index/IndexQueryProcessor.java
+++
b/modules/core/src/main/java/org/apache/ignite/internal/cache/query/index/IndexQueryProcessor.java
@@ -28,6 +28,7 @@ import java.util.stream.Stream;
import org.apache.ignite.IgniteCheckedException;
import org.apache.ignite.cache.query.IndexQuery;
import org.apache.ignite.cache.query.IndexQueryCriterion;
+import org.apache.ignite.events.CacheQueryReadEvent;
import org.apache.ignite.internal.cache.query.RangeIndexQueryCriterion;
import org.apache.ignite.internal.cache.query.index.sorted.IndexKeyDefinition;
import org.apache.ignite.internal.cache.query.index.sorted.IndexRow;
@@ -41,6 +42,7 @@ import
org.apache.ignite.internal.processors.cache.CacheObjectContext;
import org.apache.ignite.internal.processors.cache.CacheObjectUtils;
import org.apache.ignite.internal.processors.cache.GridCacheContext;
import org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree;
+import org.apache.ignite.internal.processors.cache.query.CacheQueryType;
import org.apache.ignite.internal.processors.cache.query.IndexQueryDesc;
import org.apache.ignite.internal.processors.query.QueryUtils;
import org.apache.ignite.internal.util.GridCloseableIteratorAdapter;
@@ -52,6 +54,9 @@ import org.apache.ignite.lang.IgniteBiTuple;
import org.apache.ignite.spi.indexing.IndexingQueryFilter;
import org.jetbrains.annotations.Nullable;
+import static org.apache.ignite.events.EventType.EVT_CACHE_QUERY_OBJECT_READ;
+import static
org.apache.ignite.internal.processors.security.SecurityUtils.securitySubjectId;
+
/**
* Processor of {@link IndexQuery}.
*/
@@ -74,7 +79,8 @@ public class IndexQueryProcessor {
IndexQueryDesc idxQryDesc,
@Nullable IgniteBiPredicate filter,
IndexingQueryFilter cacheFilter,
-boolean keepBinary
+boolean keepBinary,
+int taskHash
) throws IgniteCheckedException {
InlineIndexImpl idx = (InlineIndexImpl)findSortedIndex(cctx,
idxQryDesc);
@@ -86,6 +92,8 @@ public class IndexQueryProcessor {
IndexQueryResultMeta meta = new IndexQueryResultMeta(def,
qry.critSize());
+boolean isRecordable =
cctx.events().isRecordable(EVT_CACHE_QUERY_OBJECT_READ);
+
// Map IndexRow to Cache Key-Value pair.
return new IndexQueryResult<>(meta, new
GridCloseableIteratorAdapter>() {
private IgniteBiTuple currVal;
@@ -111,6 +119,26 @@ public class IndexQueryProcessor {
continue;
}
+if (isRecordable) {
+cctx.gridEvents().record(new CacheQueryReadEvent<>(
+cctx.localNode(),
+"Index query entry read.",
+EVT_CACHE_QUERY_OBJECT_READ,
+CacheQueryType.INDEX.name(),
+cctx.name(),
+idxQryDesc.valType(),
+null,
+filter,
+null,
+null,
+securitySubjectId(cctx),
+
cctx.kernalContext().task().resolveTaskName(taskHash),
+k,
+v,
+null,
+null));
+}
+
currVal = new IgniteBiTuple<>(k, v);
}
diff --git
a/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/query/CacheQueryType.java
b/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/query/
(ignite) branch master updated: IGNITE-22165: Add dump cancellation test (#11337)
This is an automated email from the ASF dual-hosted git repository.
timoninmaxim pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/ignite.git
The following commit(s) were added to refs/heads/master by this push:
new ad73c62a3e5 IGNITE-22165: Add dump cancellation test (#11337)
ad73c62a3e5 is described below
commit ad73c62a3e5a5b24acb7baaf1bb039fb8665c4da
Author: Vladimir Steshin
AuthorDate: Tue May 7 18:13:11 2024 +0300
IGNITE-22165: Add dump cancellation test (#11337)
---
.../snapshot/AbstractSnapshotSelfTest.java | 20 ++-
.../snapshot/dump/IgniteCacheDumpSelf2Test.java| 40 ++
2 files changed, 59 insertions(+), 1 deletion(-)
diff --git
a/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/persistence/snapshot/AbstractSnapshotSelfTest.java
b/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/persistence/snapshot/AbstractSnapshotSelfTest.java
index bc0d3b1e8c6..23b37cabf37 100644
---
a/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/persistence/snapshot/AbstractSnapshotSelfTest.java
+++
b/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/persistence/snapshot/AbstractSnapshotSelfTest.java
@@ -783,6 +783,23 @@ public abstract class AbstractSnapshotSelfTest extends
GridCommonAbstractTest {
List srvs,
IgniteCache cache,
Consumer snpCanceller
+) {
+doSnapshotCancellationTest(false, startCli, srvs, cache, snpCanceller);
+}
+
+/**
+ * @param dump Dump flag.
+ * @param startCli Client node to start snapshot.
+ * @param srvs Server nodes.
+ * @param cache Persisted cache.
+ * @param snpCanceller Snapshot cancel closure.
+ */
+public static void doSnapshotCancellationTest(
+boolean dump,
+IgniteEx startCli,
+List srvs,
+IgniteCache cache,
+Consumer snpCanceller
) {
IgniteEx srv = srvs.get(0);
@@ -795,7 +812,8 @@ public abstract class AbstractSnapshotSelfTest extends
GridCommonAbstractTest {
List execs = setBlockingSnapshotExecutor(srvs);
-IgniteFuture fut =
startCli.snapshot().createSnapshot(SNAPSHOT_NAME);
+IgniteFuture fut = snp(startCli).createSnapshot(SNAPSHOT_NAME,
null, null, false,
+false, dump, false, false);
for (BlockingExecutor exec : execs)
exec.waitForBlocked(30_000L);
diff --git
a/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/persistence/snapshot/dump/IgniteCacheDumpSelf2Test.java
b/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/persistence/snapshot/dump/IgniteCacheDumpSelf2Test.java
index 67f6a928d71..158f4369a5f 100644
---
a/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/persistence/snapshot/dump/IgniteCacheDumpSelf2Test.java
+++
b/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/persistence/snapshot/dump/IgniteCacheDumpSelf2Test.java
@@ -26,6 +26,7 @@ import java.nio.file.Files;
import java.nio.file.Paths;
import java.util.ArrayList;
import java.util.Arrays;
+import java.util.Collection;
import java.util.Collections;
import java.util.HashMap;
import java.util.Iterator;
@@ -33,6 +34,7 @@ import java.util.List;
import java.util.Map;
import java.util.Objects;
import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.ThreadLocalRandom;
import java.util.concurrent.atomic.AtomicBoolean;
import java.util.function.BiConsumer;
@@ -108,12 +110,16 @@ import static java.nio.file.StandardOpenOption.WRITE;
import static
org.apache.ignite.configuration.IgniteConfiguration.DFLT_SNAPSHOT_DIRECTORY;
import static org.apache.ignite.dump.DumpReaderConfiguration.DFLT_THREAD_CNT;
import static org.apache.ignite.dump.DumpReaderConfiguration.DFLT_TIMEOUT;
+import static org.apache.ignite.events.EventType.EVTS_CLUSTER_SNAPSHOT;
+import static org.apache.ignite.events.EventType.EVT_CLUSTER_SNAPSHOT_FAILED;
+import static org.apache.ignite.events.EventType.EVT_CLUSTER_SNAPSHOT_STARTED;
import static
org.apache.ignite.internal.encryption.AbstractEncryptionTest.MASTER_KEY_NAME_2;
import static
org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.CACHE_DATA_FILENAME;
import static
org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.CACHE_DIR_PREFIX;
import static
org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.PART_FILE_PREFIX;
import static
org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.ZIP_SUFFIX;
import static
org.apache.ignite.internal.processors.cache.persistence.filename.PdsFolderResolver.DB_DEFAULT_FOLDER;
+import static
org.apache.ignite.internal.processors.cache.persistence.snapshot.AbstractSnapshotSelfTest.doSnapshotCancellationTest;
import static
(ignite-extensions) 01/01: IGNITE-22080 Fix ConflictResolver fails on debug logging (#261)
This is an automated email from the ASF dual-hosted git repository. timoninmaxim pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/ignite-extensions.git commit d3692283f8e47d9a5527836ad2f085b4120325c8 Merge: af7fdb09 88236cae Author: Maksim Timonin AuthorDate: Fri Apr 19 17:27:20 2024 +0300 IGNITE-22080 Fix ConflictResolver fails on debug logging (#261) .../CacheVersionConflictResolverImpl.java| 2 +- .../DebugCacheVersionConflictResolverImpl.java | 16 ++-- 2 files changed, 15 insertions(+), 3 deletions(-)
(ignite-extensions) branch master updated (af7fdb09 -> d3692283)
This is an automated email from the ASF dual-hosted git repository. timoninmaxim pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/ignite-extensions.git from af7fdb09 IGNITE-22072 Fixed compilation after custom metrics (#260) add 107b38b6 IGNITE-22080 Fix ConflictResolver fails on debug logging add 5cf71339 Add debug log for error add 88236cae Merge branch 'master' of github.com:apache/ignite-extensions into IGNITE-22080__debug_resolver new d3692283 IGNITE-22080 Fix ConflictResolver fails on debug logging (#261) The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../CacheVersionConflictResolverImpl.java| 2 +- .../DebugCacheVersionConflictResolverImpl.java | 16 ++-- 2 files changed, 15 insertions(+), 3 deletions(-)
(ignite) branch master updated: IGNITE-22026 Fix incorrect retry logic in GridCacheIoManager send method (#11311)
This is an automated email from the ASF dual-hosted git repository.
timoninmaxim pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/ignite.git
The following commit(s) were added to refs/heads/master by this push:
new 1c34e2daca5 IGNITE-22026 Fix incorrect retry logic in
GridCacheIoManager send method (#11311)
1c34e2daca5 is described below
commit 1c34e2daca546f46a9a0c7eb05a7e9d45fa75d59
Author: Ilya Shishkov
AuthorDate: Fri Apr 19 10:51:16 2024 +0300
IGNITE-22026 Fix incorrect retry logic in GridCacheIoManager send method
(#11311)
---
.../processors/cache/GridCacheIoManager.java | 18 +-
.../cache/GridCacheIoManagerRetryTest.java | 208 +
.../ignite/testsuites/IgniteCacheTestSuite13.java | 3 +
3 files changed, 218 insertions(+), 11 deletions(-)
diff --git
a/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/GridCacheIoManager.java
b/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/GridCacheIoManager.java
index b4f291977ff..201bd14f698 100644
---
a/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/GridCacheIoManager.java
+++
b/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/GridCacheIoManager.java
@@ -1204,12 +1204,13 @@ public class GridCacheIoManager extends
GridCacheSharedManagerAdapter {
int cnt = 0;
-while (cnt <= retryCnt) {
+while (true) {
try {
-cnt++;
-
cctx.gridIO().sendToGridTopic(node, TOPIC_CACHE, msg, plc);
+if (log.isDebugEnabled())
+log.debug("Sent cache message [msg=" + msg + ", node=" +
U.toShortString(node) + ']');
+
return;
}
catch (ClusterTopologyCheckedException e) {
@@ -1219,7 +1220,7 @@ public class GridCacheIoManager extends
GridCacheSharedManagerAdapter {
if (!cctx.discovery().alive(node.id()) ||
!cctx.discovery().pingNode(node.id()))
throw new ClusterTopologyCheckedException("Node left grid
while sending message to: " + node.id(), e);
-if (cnt == retryCnt || cctx.kernalContext().isStopping())
+if (cnt++ >= retryCnt || cctx.kernalContext().isStopping())
throw e;
else if (log.isDebugEnabled())
log.debug("Failed to send message to node (will retry): "
+ node.id());
@@ -1227,9 +1228,6 @@ public class GridCacheIoManager extends
GridCacheSharedManagerAdapter {
U.sleep(retryDelay);
}
-
-if (log.isDebugEnabled())
-log.debug("Sent cache message [msg=" + msg + ", node=" +
U.toShortString(node) + ']');
}
/**
@@ -1267,10 +1265,8 @@ public class GridCacheIoManager extends
GridCacheSharedManagerAdapter {
int cnt = 0;
-while (cnt <= retryCnt) {
+while (true) {
try {
-cnt++;
-
cctx.gridIO().sendOrderedMessage(node, topic, msg, plc,
timeout, false);
if (log.isDebugEnabled())
@@ -1286,7 +1282,7 @@ public class GridCacheIoManager extends
GridCacheSharedManagerAdapter {
if (cctx.discovery().node(node.id()) == null)
throw new ClusterTopologyCheckedException("Node left grid
while sending ordered message to: " + node.id(), e);
-if (cnt == retryCnt)
+if (cnt++ >= retryCnt)
throw e;
else if (log.isDebugEnabled())
log.debug("Failed to send message to node (will retry): "
+ node.id());
diff --git
a/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/GridCacheIoManagerRetryTest.java
b/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/GridCacheIoManagerRetryTest.java
new file mode 100644
index 000..0ab5cd53822
--- /dev/null
+++
b/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/GridCacheIoManagerRetryTest.java
@@ -0,0 +1,208 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governin
(ignite) branch master updated: IGNITE-21770 Extend logging for the AbstractFileIO#fully() method (#11294)
This is an automated email from the ASF dual-hosted git repository.
timoninmaxim pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/ignite.git
The following commit(s) were added to refs/heads/master by this push:
new af869206090 IGNITE-21770 Extend logging for the AbstractFileIO#fully()
method (#11294)
af869206090 is described below
commit af869206090a36ff456d6d336fbd88906a2764c4
Author: oleg-vlsk <153691984+oleg-v...@users.noreply.github.com>
AuthorDate: Mon Apr 15 18:04:28 2024 +1000
IGNITE-21770 Extend logging for the AbstractFileIO#fully() method (#11294)
---
.../cache/persistence/file/AbstractFileIO.java | 18 +++---
1 file changed, 15 insertions(+), 3 deletions(-)
diff --git
a/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/file/AbstractFileIO.java
b/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/file/AbstractFileIO.java
index 68bedb04b5d..6b6fb53649e 100644
---
a/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/file/AbstractFileIO.java
+++
b/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/file/AbstractFileIO.java
@@ -44,6 +44,8 @@ public abstract class AbstractFileIO implements FileIO {
* @param operation IO operation.
*
* @param num Number of bytes to operate.
+ *
+ * @return Number of bytes operated.
*/
private int fully(IOOperation operation, long position, int num, boolean
write) throws IOException {
if (num > 0) {
@@ -62,9 +64,19 @@ public abstract class AbstractFileIO implements FileIO {
if (time == 0)
time = System.nanoTime();
-else if ((System.nanoTime() - time) >=
U.millisToNanos(MAX_IO_TIMEOUT_MS))
-throw new IOException(write && (position + i) ==
size() ? "Failed to extend file." :
-"Probably disk is too busy, please check your
device.");
+else if ((System.nanoTime() - time) >=
U.millisToNanos(MAX_IO_TIMEOUT_MS)) {
+String errMsg = " operation unsuccessful, timeout
exceeds the maximum IO timeout ("
++ U.millisToNanos(MAX_IO_TIMEOUT_MS) + " ms). ";
+
+if (write && (position + i) == size())
+errMsg = "Write" + errMsg + "Failed to extend
file.";
+else {
+errMsg = (write ? "Write" : "Read") + errMsg +
+"Probably disk is too busy, please check your
device.";
+}
+
+throw new IOException(errMsg);
+}
}
else
return -1;
(ignite) branch master updated: IGNITE-22020 Add 'keepRaw' parameter to DumpReaderConfiguration (#11309)
This is an automated email from the ASF dual-hosted git repository.
timoninmaxim pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/ignite.git
The following commit(s) were added to refs/heads/master by this push:
new 1fb6ad0a8f8 IGNITE-22020 Add 'keepRaw' parameter to
DumpReaderConfiguration (#11309)
1fb6ad0a8f8 is described below
commit 1fb6ad0a8f8a8db622c956bee727417f0f5d70ec
Author: Vladimir Steshin
AuthorDate: Fri Apr 12 14:24:57 2024 +0300
IGNITE-22020 Add 'keepRaw' parameter to DumpReaderConfiguration (#11309)
---
.../java/org/apache/ignite/dump/DumpReader.java| 2 +-
.../ignite/dump/DumpReaderConfiguration.java | 36 -
.../snapshot/dump/AbstractCacheDumpTest.java | 1 +
.../snapshot/dump/IgniteCacheDumpSelf2Test.java| 88 ++
4 files changed, 122 insertions(+), 5 deletions(-)
diff --git a/modules/core/src/main/java/org/apache/ignite/dump/DumpReader.java
b/modules/core/src/main/java/org/apache/ignite/dump/DumpReader.java
index 6dcfbb7cce7..125d9c959fe 100644
--- a/modules/core/src/main/java/org/apache/ignite/dump/DumpReader.java
+++ b/modules/core/src/main/java/org/apache/ignite/dump/DumpReader.java
@@ -77,7 +77,7 @@ public class DumpReader implements Runnable {
@Override public void run() {
ackAsciiLogo();
-try (Dump dump = new Dump(cfg.dumpRoot(), null, cfg.keepBinary(),
false, encryptionSpi(), log)) {
+try (Dump dump = new Dump(cfg.dumpRoot(), null, cfg.keepBinary(),
cfg.keepRaw(), encryptionSpi(), log)) {
DumpConsumer cnsmr = cfg.consumer();
cnsmr.start();
diff --git
a/modules/core/src/main/java/org/apache/ignite/dump/DumpReaderConfiguration.java
b/modules/core/src/main/java/org/apache/ignite/dump/DumpReaderConfiguration.java
index aeb62fba05b..14c202991b6 100644
---
a/modules/core/src/main/java/org/apache/ignite/dump/DumpReaderConfiguration.java
+++
b/modules/core/src/main/java/org/apache/ignite/dump/DumpReaderConfiguration.java
@@ -19,6 +19,7 @@ package org.apache.ignite.dump;
import java.io.File;
import java.time.Duration;
+import org.apache.ignite.binary.BinaryObject;
import org.apache.ignite.internal.processors.cache.CacheObject;
import org.apache.ignite.internal.processors.cache.KeyCacheObject;
import org.apache.ignite.lang.IgniteExperimental;
@@ -53,9 +54,18 @@ public class DumpReaderConfiguration {
/** Stop processing partitions if consumer fail to process one. */
private final boolean failFast;
-/** If {@code true} then don't deserialize {@link KeyCacheObject} and
{@link CacheObject}. */
+/**
+ * If {@code true} and if {@link #keepRaw} is {@code false} then keeps
{@link DumpEntry#key()} and
+ * {@link DumpEntry#value()} as {@link BinaryObject}.
+ */
private final boolean keepBinary;
+/**
+ * If {@code true}, doesn't deserialize cache data and keeps {@link
DumpEntry#key()} as {@link KeyCacheObject} and
+ * {@link DumpEntry#value()} as {@link CacheObject}. If {@code true},
disables {@link #keepBinary}.
+ */
+private final boolean keepRaw;
+
/** Cache group names. */
private final String[] cacheGrpNames;
@@ -70,7 +80,7 @@ public class DumpReaderConfiguration {
* @param cnsmr Dump consumer.
*/
public DumpReaderConfiguration(File dir, DumpConsumer cnsmr) {
-this(dir, cnsmr, DFLT_THREAD_CNT, DFLT_TIMEOUT, true, true, null,
false, null);
+this(dir, cnsmr, DFLT_THREAD_CNT, DFLT_TIMEOUT, true, true, false,
null, false, null);
}
/**
@@ -79,7 +89,11 @@ public class DumpReaderConfiguration {
* @param thCnt Count of threads to consume dumped partitions.
* @param timeout Timeout of dump reader invocation.
* @param failFast Stop processing partitions if consumer fail to process
one.
- * @param keepBinary If {@code true} then don't deserialize {@link
KeyCacheObject} and {@link CacheObject}.
+ * @param keepBinary If {@code true} and if {@link #keepRaw} is {@code
false} then keeps {@link DumpEntry#key()} and
+ * {@link DumpEntry#value()} as {@link BinaryObject}.
+ * @param keepRaw If {@code true}, doesn't deserialize cache data and
keeps {@link DumpEntry#key()} as
+ *{@link KeyCacheObject} and {@link DumpEntry#value()} as
{@link CacheObject}. If {@code true},
+ *disables {@link #keepBinary}.
* @param cacheGrpNames Cache group names.
* @param skipCopies Skip copies.
* @param encSpi Encryption SPI.
@@ -91,6 +105,7 @@ public class DumpReaderConfiguration {
Duration timeout,
boolean failFast,
boolean keepBinary,
+boolean keepRaw,
String[] cacheGrpNames,
boolean skipCopies,
EncryptionSpi encSpi
@@ -101,6 +116,7 @@ public class DumpReaderConfiguration {
this.timeout = timeout;
this.failFast = failFast;
this.keepBinary
(ignite) branch master updated: IGNITE-21656: Fix cache dump check fails on a cache with a node filter (#11276)
This is an automated email from the ASF dual-hosted git repository.
timoninmaxim pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/ignite.git
The following commit(s) were added to refs/heads/master by this push:
new 5145c80f38e IGNITE-21656: Fix cache dump check fails on a cache with a
node filter (#11276)
5145c80f38e is described below
commit 5145c80f38edd8a62307b2ab14cbbdaa68a76b56
Author: Vladimir Steshin
AuthorDate: Fri Apr 5 15:40:46 2024 +0300
IGNITE-21656: Fix cache dump check fails on a cache with a node filter
(#11276)
---
.../java/org/apache/ignite/dump/DumpReader.java| 2 +-
.../snapshot/IgniteSnapshotManager.java| 34 +--
.../snapshot/dump/IgniteCacheDumpSelfTest.java | 69 ++
3 files changed, 85 insertions(+), 20 deletions(-)
diff --git a/modules/core/src/main/java/org/apache/ignite/dump/DumpReader.java
b/modules/core/src/main/java/org/apache/ignite/dump/DumpReader.java
index 2d8596ee657..6dcfbb7cce7 100644
--- a/modules/core/src/main/java/org/apache/ignite/dump/DumpReader.java
+++ b/modules/core/src/main/java/org/apache/ignite/dump/DumpReader.java
@@ -97,7 +97,7 @@ public class DumpReader implements Runnable {
: null;
for (SnapshotMetadata meta : dump.metadata()) {
-for (Integer grp : meta.cacheGroupIds()) {
+for (Integer grp : meta.partitions().keySet()) {
if (cacheGrpIds == null || cacheGrpIds.contains(grp))
grpToNodes.computeIfAbsent(grp, key -> new
ArrayList<>()).add(meta.folderName());
}
diff --git
a/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/snapshot/IgniteSnapshotManager.java
b/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/snapshot/IgniteSnapshotManager.java
index f0bd25b5e73..7e2b0eb03d7 100644
---
a/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/snapshot/IgniteSnapshotManager.java
+++
b/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/snapshot/IgniteSnapshotManager.java
@@ -1181,27 +1181,23 @@ public class IgniteSnapshotManager extends
GridCacheSharedManagerAdapter
parts.put(grpId, null);
}
-IgniteInternalFuture task0;
+IgniteInternalFuture task0 =
registerSnapshotTask(req.snapshotName(),
+req.snapshotPath(),
+req.operationalNodeId(),
+req.requestId(),
+parts,
+withMetaStorage,
+req.dump(),
+req.compress(),
+req.encrypt(),
+locSndrFactory.apply(req.snapshotName(), req.snapshotPath())
+);
-if (parts.isEmpty() && !withMetaStorage)
-task0 = new GridFinishedFuture<>(Collections.emptySet());
-else {
-task0 = registerSnapshotTask(req.snapshotName(),
-req.snapshotPath(),
-req.operationalNodeId(),
-req.requestId(),
-parts,
-withMetaStorage,
-req.dump(),
-req.compress(),
-req.encrypt(),
-locSndrFactory.apply(req.snapshotName(), req.snapshotPath())
-);
+if (withMetaStorage) {
+assert task0 instanceof SnapshotFutureTask;
-if (withMetaStorage && task0 instanceof SnapshotFutureTask) {
-
((DistributedMetaStorageImpl)cctx.kernalContext().distributedMetastorage())
-.suspend(((SnapshotFutureTask)task0).started());
-}
+
((DistributedMetaStorageImpl)cctx.kernalContext().distributedMetastorage())
+.suspend(((SnapshotFutureTask)task0).started());
}
return task0.chain(() -> {
diff --git
a/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/persistence/snapshot/dump/IgniteCacheDumpSelfTest.java
b/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/persistence/snapshot/dump/IgniteCacheDumpSelfTest.java
index 741e797d904..0a44519c776 100644
---
a/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/persistence/snapshot/dump/IgniteCacheDumpSelfTest.java
+++
b/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/persistence/snapshot/dump/IgniteCacheDumpSelfTest.java
@@ -29,15 +29,19 @@ import java.util.concurrent.CountDownLatch;
import java.util.concurrent.atomic.AtomicBoolean;
import java.util.concurrent.atomic.AtomicInteger;
import java.util.function.Consumer;
+import java.util.stream.Collectors;
import java.util.stream.IntStream;
+import java.util.stream.Stream;
import javax.cache.expiry.Duration;
import javax.cache.expiry.ExpiryPolicy;
import javax.cache.processor.MutableEntry;
import
(ignite) branch master updated: IGNITE-21678 Fix Cdc error with ignite to ignite streamer in case BinaryConfiguration is specified in config (#11275)
This is an automated email from the ASF dual-hosted git repository.
timoninmaxim pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/ignite.git
The following commit(s) were added to refs/heads/master by this push:
new 2ebc8df1aa2 IGNITE-21678 Fix Cdc error with ignite to ignite streamer
in case BinaryConfiguration is specified in config (#11275)
2ebc8df1aa2 is described below
commit 2ebc8df1aa22ac105003360b57d83bbd0e9fc2df
Author: Sergey Korotkov
AuthorDate: Thu Mar 14 15:49:46 2024 +0700
IGNITE-21678 Fix Cdc error with ignite to ignite streamer in case
BinaryConfiguration is specified in config (#11275)
---
.../org/apache/ignite/internal/cdc/CdcMain.java| 9 +-
.../wal/reader/StandaloneSpiContext.java | 250 +
.../ignite/cdc/CdcPushMetricsExporterTest.java | 126 ++-
3 files changed, 376 insertions(+), 9 deletions(-)
diff --git
a/modules/core/src/main/java/org/apache/ignite/internal/cdc/CdcMain.java
b/modules/core/src/main/java/org/apache/ignite/internal/cdc/CdcMain.java
index a85cd9bf858..33d015bfb34 100644
--- a/modules/core/src/main/java/org/apache/ignite/internal/cdc/CdcMain.java
+++ b/modules/core/src/main/java/org/apache/ignite/internal/cdc/CdcMain.java
@@ -62,6 +62,7 @@ import
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAhea
import org.apache.ignite.internal.processors.cache.persistence.wal.WALPointer;
import
org.apache.ignite.internal.processors.cache.persistence.wal.reader.IgniteWalIteratorFactory;
import
org.apache.ignite.internal.processors.cache.persistence.wal.reader.StandaloneGridKernalContext;
+import
org.apache.ignite.internal.processors.cache.persistence.wal.reader.StandaloneSpiContext;
import org.apache.ignite.internal.processors.metric.MetricRegistry;
import org.apache.ignite.internal.processors.metric.impl.AtomicLongMetric;
import org.apache.ignite.internal.processors.metric.impl.HistogramMetricImpl;
@@ -74,11 +75,11 @@ import org.apache.ignite.lang.IgniteBiPredicate;
import org.apache.ignite.lang.IgniteBiTuple;
import org.apache.ignite.marshaller.MarshallerUtils;
import org.apache.ignite.platform.PlatformType;
+import org.apache.ignite.spi.IgniteSpi;
import org.apache.ignite.spi.metric.jmx.JmxMetricExporterSpi;
import org.apache.ignite.spi.metric.noop.NoopMetricExporterSpi;
import org.apache.ignite.startup.cmdline.CdcCommandLineStartup;
-import static
org.apache.ignite.IgniteSystemProperties.IGNITE_SKIP_CONFIGURATION_CONSISTENCY_CHECK;
import static org.apache.ignite.internal.IgniteKernal.NL;
import static org.apache.ignite.internal.IgniteKernal.SITE;
import static org.apache.ignite.internal.IgniteVersionUtils.ACK_VER_STR;
@@ -322,8 +323,6 @@ public class CdcMain implements Runnable {
throw new IllegalArgumentException(ERR_MSG);
}
-System.setProperty(IGNITE_SKIP_CONFIGURATION_CONSISTENCY_CHECK,
"true");
-
try (CdcFileLockHolder lock = lockPds()) {
String consIdDir = cdcDir.getName(cdcDir.getNameCount() -
1).toString();
@@ -419,7 +418,9 @@ public class CdcMain implements Runnable {
startAllComponents(kctx);
-kctx.metric().onKernalStart(true);
+for (IgniteSpi metricSpi : kctx.config().getMetricExporterSpi()) {
+metricSpi.onContextInitialized(new StandaloneSpiContext());
+}
mreg = kctx.metric().registry("cdc");
diff --git
a/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/wal/reader/StandaloneSpiContext.java
b/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/wal/reader/StandaloneSpiContext.java
new file mode 100644
index 000..887d35739b3
--- /dev/null
+++
b/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/wal/reader/StandaloneSpiContext.java
@@ -0,0 +1,250 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.processors.cache.persistence.wal.reader;
+
+import java.io.Serializable;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Map;
+import java.util.U
(ignite) branch master updated: IGNITE-21550 Expose CDC metrics via push metrics exporter spi (#11248)
This is an automated email from the ASF dual-hosted git repository.
timoninmaxim pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/ignite.git
The following commit(s) were added to refs/heads/master by this push:
new b27817afe2b IGNITE-21550 Expose CDC metrics via push metrics exporter
spi (#11248)
b27817afe2b is described below
commit b27817afe2b159b0562f999be6fb5537ce129433
Author: Sergey Korotkov
AuthorDate: Thu Feb 29 21:51:47 2024 +0700
IGNITE-21550 Expose CDC metrics via push metrics exporter spi (#11248)
---
.../org/apache/ignite/internal/cdc/CdcMain.java| 5 ++
.../wal/reader/StandaloneGridKernalContext.java| 24 +-
.../org/apache/ignite/cdc/AbstractCdcTest.java | 1 +
.../ignite/cdc/CdcPushMetricsExporterTest.java | 92 ++
.../ignite/testsuites/IgnitePdsTestSuite2.java | 2 +
.../ignitetest/services/utils/cdc/ignite_cdc.py| 4 +-
.../opencensus/OpenCensusMetricExporterSpi.java| 6 +-
7 files changed, 128 insertions(+), 6 deletions(-)
diff --git
a/modules/core/src/main/java/org/apache/ignite/internal/cdc/CdcMain.java
b/modules/core/src/main/java/org/apache/ignite/internal/cdc/CdcMain.java
index 2f9269b18b1..a85cd9bf858 100644
--- a/modules/core/src/main/java/org/apache/ignite/internal/cdc/CdcMain.java
+++ b/modules/core/src/main/java/org/apache/ignite/internal/cdc/CdcMain.java
@@ -78,6 +78,7 @@ import org.apache.ignite.spi.metric.jmx.JmxMetricExporterSpi;
import org.apache.ignite.spi.metric.noop.NoopMetricExporterSpi;
import org.apache.ignite.startup.cmdline.CdcCommandLineStartup;
+import static
org.apache.ignite.IgniteSystemProperties.IGNITE_SKIP_CONFIGURATION_CONSISTENCY_CHECK;
import static org.apache.ignite.internal.IgniteKernal.NL;
import static org.apache.ignite.internal.IgniteKernal.SITE;
import static org.apache.ignite.internal.IgniteVersionUtils.ACK_VER_STR;
@@ -321,6 +322,8 @@ public class CdcMain implements Runnable {
throw new IllegalArgumentException(ERR_MSG);
}
+System.setProperty(IGNITE_SKIP_CONFIGURATION_CONSISTENCY_CHECK,
"true");
+
try (CdcFileLockHolder lock = lockPds()) {
String consIdDir = cdcDir.getName(cdcDir.getNameCount() -
1).toString();
@@ -416,6 +419,8 @@ public class CdcMain implements Runnable {
startAllComponents(kctx);
+kctx.metric().onKernalStart(true);
+
mreg = kctx.metric().registry("cdc");
return kctx;
diff --git
a/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/wal/reader/StandaloneGridKernalContext.java
b/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/wal/reader/StandaloneGridKernalContext.java
index af8a24f503c..a2da615224c 100644
---
a/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/wal/reader/StandaloneGridKernalContext.java
+++
b/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/wal/reader/StandaloneGridKernalContext.java
@@ -137,6 +137,9 @@ public class StandaloneGridKernalContext implements
GridKernalContext {
/** System view manager. */
private final GridSystemViewManager sysViewMgr;
+/** Timeout processor. */
+private final GridTimeoutProcessor timeoutProc;
+
/** */
@GridToStringExclude
private CacheObjectTransformerProcessor transProc;
@@ -201,6 +204,7 @@ public class StandaloneGridKernalContext implements
GridKernalContext {
rsrcProc = new GridResourceProcessor(this);
metricMgr = new GridMetricManager(this);
sysViewMgr = new GridSystemViewManager(this);
+timeoutProc = new GridTimeoutProcessor(this);
transProc = createComponent(CacheObjectTransformerProcessor.class);
// Fake folder provided to perform processor startup on empty folder.
@@ -212,6 +216,7 @@ public class StandaloneGridKernalContext implements
GridKernalContext {
comps.add(rsrcProc);
comps.add(cacheObjProcessor);
comps.add(metricMgr);
+comps.add(timeoutProc);
if (marshallerMappingFileStoreDir != null) {
marshallerCtx.setMarshallerMappingFileStoreDir(marshallerMappingFileStoreDir);
@@ -306,7 +311,20 @@ public class StandaloneGridKernalContext implements
GridKernalContext {
/** {@inheritDoc} */
@Override public IgniteEx grid() {
-final IgniteEx kernal = new IgniteKernal();
+final IgniteEx kernal = new IgniteKernal() {
+/**
+ * Override to return the non-null context instance to make metric
SPIs happy.
+ *
+ * Say the SqlViewMetricExporterSpi one which may be automatically
added by
+ * the {@link GridMetricManager} if indexing or query engine are
found in classpath
+ * (which is the default behaviour).
+ *
+ * @r
(ignite) branch master updated: IGNITE-21570 Add buffer for reading files while checking cache dumps (#11252)
This is an automated email from the ASF dual-hosted git repository.
timoninmaxim pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/ignite.git
The following commit(s) were added to refs/heads/master by this push:
new 6f6b94793b2 IGNITE-21570 Add buffer for reading files while checking
cache dumps (#11252)
6f6b94793b2 is described below
commit 6f6b94793b2c0fb1de62f7d650188e32612c029d
Author: yurinaryshkin <135707807+yurinarysh...@users.noreply.github.com>
AuthorDate: Wed Feb 21 16:24:26 2024 +0300
IGNITE-21570 Add buffer for reading files while checking cache dumps
(#11252)
---
.../cache/persistence/snapshot/dump/Dump.java | 118 -
.../snapshot/dump/IgniteCacheDumpSelf2Test.java| 4 +
2 files changed, 119 insertions(+), 3 deletions(-)
diff --git
a/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/snapshot/dump/Dump.java
b/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/snapshot/dump/Dump.java
index 215a7e9570f..7c149118a42 100644
---
a/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/snapshot/dump/Dump.java
+++
b/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/snapshot/dump/Dump.java
@@ -33,6 +33,7 @@ import java.util.Objects;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.ConcurrentMap;
import java.util.stream.Collectors;
+import java.util.zip.ZipInputStream;
import org.apache.ignite.IgniteCheckedException;
import org.apache.ignite.IgniteException;
import org.apache.ignite.IgniteLogger;
@@ -47,10 +48,10 @@ import
org.apache.ignite.internal.processors.cache.KeyCacheObject;
import org.apache.ignite.internal.processors.cache.StoredCacheData;
import
org.apache.ignite.internal.processors.cache.binary.CacheObjectBinaryProcessorImpl;
import org.apache.ignite.internal.processors.cache.persistence.file.FileIO;
+import
org.apache.ignite.internal.processors.cache.persistence.file.FileIODecorator;
import
org.apache.ignite.internal.processors.cache.persistence.file.FileIOFactory;
import
org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager;
-import
org.apache.ignite.internal.processors.cache.persistence.file.RandomAccessFileIOFactory;
-import
org.apache.ignite.internal.processors.cache.persistence.file.UnzipFileIOFactory;
+import
org.apache.ignite.internal.processors.cache.persistence.file.RandomAccessFileIO;
import
org.apache.ignite.internal.processors.cache.persistence.snapshot.SnapshotMetadata;
import
org.apache.ignite.internal.processors.cache.persistence.wal.reader.StandaloneGridKernalContext;
import org.apache.ignite.internal.util.typedef.F;
@@ -62,6 +63,7 @@ import org.apache.ignite.marshaller.jdk.JdkMarshaller;
import org.apache.ignite.spi.encryption.EncryptionSpi;
import org.jetbrains.annotations.Nullable;
+import static java.nio.file.StandardOpenOption.READ;
import static
org.apache.ignite.configuration.DataStorageConfiguration.DFLT_BINARY_METADATA_PATH;
import static
org.apache.ignite.configuration.DataStorageConfiguration.DFLT_MARSHALLER_PATH;
import static
org.apache.ignite.internal.processors.cache.GridLocalConfigManager.readCacheData;
@@ -267,7 +269,9 @@ public class Dump implements AutoCloseable {
* @return Dump iterator.
*/
public DumpedPartitionIterator iterator(String node, int group, int part) {
-FileIOFactory ioFactory = comprParts ? new UnzipFileIOFactory() : new
RandomAccessFileIOFactory();
+FileIOFactory ioFactory = comprParts
+? (file, modes) -> new ReadOnlyUnzipFileIO(file)
+: (file, modes) -> new ReadOnlyBufferedFileIO(file);
FileIO dumpFile;
@@ -396,4 +400,112 @@ public class Dump implements AutoCloseable {
public interface DumpedPartitionIterator extends Iterator,
AutoCloseable {
// No-op.
}
+
+/** */
+private static class ReadOnlyBufferedFileIO extends FileIODecorator {
+/** */
+private static final int DEFAULT_BLOCK_SIZE = 4096;
+
+/** */
+private final ByteBuffer buf;
+
+/** */
+private long pos;
+
+/** */
+ReadOnlyBufferedFileIO(File file) throws IOException {
+super(new RandomAccessFileIO(file, READ));
+
+int blockSize = getFileSystemBlockSize();
+
+if (blockSize <= 0)
+blockSize = DEFAULT_BLOCK_SIZE;
+
+buf = ByteBuffer.allocateDirect(blockSize);
+
+buf.position(buf.limit());
+}
+
+/** {@inheritDoc} */
+@Override public int readFully(ByteBuffer dst) throws IOException {
+int totalRead = 0;
+
+while (dst.hasRemaining()) {
+if (!buf.hasRemaining()) {
+// Buf limit will be at its capacity unless partial fill
has happened at
(ignite) branch master updated: IGNITE-21102 Use ClusterState enum for baseline control.sh commands output (#11138)
This is an automated email from the ASF dual-hosted git repository.
timoninmaxim pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/ignite.git
The following commit(s) were added to refs/heads/master by this push:
new ffde1db96f0 IGNITE-21102 Use ClusterState enum for baseline control.sh
commands output (#11138)
ffde1db96f0 is described below
commit ffde1db96f0c9d0d86c773daa4f6baee18ab1b66
Author: oleg-vlsk <153691984+oleg-v...@users.noreply.github.com>
AuthorDate: Tue Feb 13 18:16:03 2024 +1000
IGNITE-21102 Use ClusterState enum for baseline control.sh commands output
(#11138)
---
.../apache/ignite/util/GridCommandHandlerTest.java | 33 --
.../ignite/internal/management/StateCommand.java | 20 +
.../baseline/AbstractBaselineCommand.java | 2 +-
.../internal/management/baseline/BaselineTask.java | 2 +-
.../management/baseline/BaselineTaskResult.java| 25 ++--
.../visor/baseline/VisorBaselineViewTask.java | 2 +-
6 files changed, 50 insertions(+), 34 deletions(-)
diff --git
a/modules/control-utility/src/test/java/org/apache/ignite/util/GridCommandHandlerTest.java
b/modules/control-utility/src/test/java/org/apache/ignite/util/GridCommandHandlerTest.java
index 887448cc1cf..fc193081c41 100644
---
a/modules/control-utility/src/test/java/org/apache/ignite/util/GridCommandHandlerTest.java
+++
b/modules/control-utility/src/test/java/org/apache/ignite/util/GridCommandHandlerTest.java
@@ -903,7 +903,7 @@ public class GridCommandHandlerTest extends
GridCommandHandlerClusterPerMethodAb
assertEquals(EXIT_CODE_OK, execute("--state"));
-assertContains(log, testOut.toString(), "Cluster is inactive");
+assertClusterState(INACTIVE, testOut.toString());
String out = testOut.toString();
@@ -919,7 +919,7 @@ public class GridCommandHandlerTest extends
GridCommandHandlerClusterPerMethodAb
assertEquals(EXIT_CODE_OK, execute("--state"));
-assertContains(log, testOut.toString(), "Cluster is active");
+assertClusterState(ACTIVE, testOut.toString());
ignite.cluster().state(ACTIVE_READ_ONLY);
@@ -929,7 +929,7 @@ public class GridCommandHandlerTest extends
GridCommandHandlerClusterPerMethodAb
assertEquals(EXIT_CODE_OK, execute("--state"));
-assertContains(log, testOut.toString(), "Cluster is active
(read-only)");
+assertClusterState(ACTIVE_READ_ONLY, testOut.toString());
boolean tagUpdated = GridTestUtils.waitForCondition(() -> {
try {
@@ -3972,6 +3972,25 @@ public class GridCommandHandlerTest extends
GridCommandHandlerClusterPerMethodAb
assertContains(log, testOut.toString(), "no conflicts have been
found");
}
+/**
+ * Test to make sure that the '--baseline' command shows correct cluster
state
+ *
+ * @throws Exception if failed.
+ */
+@Test
+public void testClusterStateInBaselineCommand() throws Exception {
+Ignite ignite = startGrids(1);
+
+injectTestSystemOut();
+
+for (ClusterState state : ClusterState.values()) {
+ignite.cluster().state(state);
+assertEquals(EXIT_CODE_OK, execute("--baseline"));
+assertEquals(state, ignite.cluster().state());
+assertClusterState(state, testOut.toString());
+}
+}
+
/**
* @param ignite Ignite to execute task on.
* @param delFoundGarbage If clearing mode should be used.
@@ -4060,4 +4079,12 @@ public class GridCommandHandlerTest extends
GridCommandHandlerClusterPerMethodAb
}
}
}
+
+/**
+ * @param state Current state of the cluster.
+ * @param logOutput Logger output where current cluster state is supposed
to be specified.
+ */
+public static void assertClusterState(ClusterState state, String
logOutput) {
+assertTrue(Pattern.compile("Cluster state: " + state +
"\\s+").matcher(logOutput).find());
+}
}
diff --git
a/modules/core/src/main/java/org/apache/ignite/internal/management/StateCommand.java
b/modules/core/src/main/java/org/apache/ignite/internal/management/StateCommand.java
index 8b7ff823a86..21eca4fd8a1 100644
---
a/modules/core/src/main/java/org/apache/ignite/internal/management/StateCommand.java
+++
b/modules/core/src/main/java/org/apache/ignite/internal/management/StateCommand.java
@@ -73,25 +73,7 @@ public class StateCommand implements LocalCommand printer
) {
-printer.accept("Cluster state: " + (res.isActive() ? "active" :
"inactive"));
+printer.accept("Cluster state: " + res.clusterState());
printer.accept("Current topology version: " +
res.getTopologyVersion());
BaselineAutoAdjustSettings autoAdjustSettings =
res.getA
(ignite) branch master updated: IGNITE-21485 : Refactoring of lost code and codestyle related to GridCacheVersion (#11236)
This is an automated email from the ASF dual-hosted git repository.
timoninmaxim pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/ignite.git
The following commit(s) were added to refs/heads/master by this push:
new b6b8cfcdadb IGNITE-21485 : Refactoring of lost code and codestyle
related to GridCacheVersion (#11236)
b6b8cfcdadb is described below
commit b6b8cfcdadbd46e46de6d4c0d8af33323e4748c6
Author: Vladimir Steshin
AuthorDate: Fri Feb 9 17:13:14 2024 +0300
IGNITE-21485 : Refactoring of lost code and codestyle related to
GridCacheVersion (#11236)
---
.../dht/atomic/DhtAtomicUpdateResult.java| 20
.../preloader/GridDhtPartitionsExchangeFuture.java | 2 +-
.../cache/distributed/near/GridNearGetResponse.java | 3 +--
.../processors/cache/version/GridCacheVersion.java | 5 +
.../cache/version/GridCacheVersionManager.java | 9 -
5 files changed, 7 insertions(+), 32 deletions(-)
diff --git
a/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/distributed/dht/atomic/DhtAtomicUpdateResult.java
b/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/distributed/dht/atomic/DhtAtomicUpdateResult.java
index 15db625dc46..c265bf3edd3 100644
---
a/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/distributed/dht/atomic/DhtAtomicUpdateResult.java
+++
b/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/distributed/dht/atomic/DhtAtomicUpdateResult.java
@@ -50,26 +50,6 @@ class DhtAtomicUpdateResult {
*/
private int processedEntriesCount;
-/**
- *
- */
-DhtAtomicUpdateResult() {
-// No-op.
-}
-
-/**
- * @param retVal Return value.
- * @param deleted Deleted entries.
- * @param dhtFut DHT update future.
- */
-DhtAtomicUpdateResult(GridCacheReturn retVal,
-Collection> deleted,
-GridDhtAtomicAbstractUpdateFuture dhtFut) {
-this.retVal = retVal;
-this.deleted = deleted;
-this.dhtFut = dhtFut;
-}
-
/**
* @param expiry Expiry policy.
*/
diff --git
a/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/distributed/dht/preloader/GridDhtPartitionsExchangeFuture.java
b/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/distributed/dht/preloader/GridDhtPartitionsExchangeFuture.java
index c1e49d21107..ede7f87973d 100644
---
a/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/distributed/dht/preloader/GridDhtPartitionsExchangeFuture.java
+++
b/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/distributed/dht/preloader/GridDhtPartitionsExchangeFuture.java
@@ -2787,7 +2787,7 @@ public class GridDhtPartitionsExchangeFuture extends
GridDhtTopologyFutureAdapte
while (true) {
GridCacheVersion old = lastVer.get();
-if (old == null || Long.compare(old.order(), ver.order()) < 0) {
+if (old == null || old.order() < ver.order()) {
if (lastVer.compareAndSet(old, ver))
break;
}
diff --git
a/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/distributed/near/GridNearGetResponse.java
b/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/distributed/near/GridNearGetResponse.java
index 578c46b6ac3..50e3dcdef4e 100644
---
a/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/distributed/near/GridNearGetResponse.java
+++
b/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/distributed/near/GridNearGetResponse.java
@@ -45,8 +45,7 @@ import org.jetbrains.annotations.NotNull;
/**
* Get response.
*/
-public class GridNearGetResponse extends GridCacheIdMessage implements
GridCacheDeployable,
-GridCacheVersionable {
+public class GridNearGetResponse extends GridCacheIdMessage implements
GridCacheDeployable, GridCacheVersionable {
/** */
private static final long serialVersionUID = 0L;
diff --git
a/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/version/GridCacheVersion.java
b/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/version/GridCacheVersion.java
index a54a9f5a54a..9044a15cfd2 100644
---
a/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/version/GridCacheVersion.java
+++
b/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/version/GridCacheVersion.java
@@ -88,6 +88,11 @@ public class GridCacheVersion implements Message,
Externalizable, CacheEntryVers
* @param order Version order.
*/
public GridCacheVersion(int topVer, int nodeOrderDrId, long order) {
+assert topVer >= 0 : topVer;
+assert order >= 0 : order;
+assert (nodeOrderDrId >>> DR_ID_SHIFT) >= 0 : (nodeOrderDrId >>>
DR_ID_SH
(ignite) branch master updated: IGNITE-21236 Fix dump doesn't contain data inserted before node restart (#11233)
This is an automated email from the ASF dual-hosted git repository.
timoninmaxim pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/ignite.git
The following commit(s) were added to refs/heads/master by this push:
new 3ad3f0f4523 IGNITE-21236 Fix dump doesn't contain data inserted before
node restart (#11233)
3ad3f0f4523 is described below
commit 3ad3f0f45239472745392ae4820f04b24527b5e1
Author: Vladimir Steshin
AuthorDate: Fri Feb 9 16:32:42 2024 +0300
IGNITE-21236 Fix dump doesn't contain data inserted before node restart
(#11233)
---
.../snapshot/dump/CreateDumpFutureTask.java| 57 ++--
.../snapshot/dump/IgniteCacheDumpSelf2Test.java| 103 +
2 files changed, 129 insertions(+), 31 deletions(-)
diff --git
a/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/snapshot/dump/CreateDumpFutureTask.java
b/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/snapshot/dump/CreateDumpFutureTask.java
index 3b6790fc64b..1267e18bf88 100644
---
a/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/snapshot/dump/CreateDumpFutureTask.java
+++
b/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/snapshot/dump/CreateDumpFutureTask.java
@@ -137,6 +137,15 @@ public class CreateDumpFutureTask extends
AbstractCreateSnapshotFutureTask imple
*/
private final @Nullable ConcurrentMap encThLocBufs;
+/**
+ * Cache entry version at the dump start.
+ * Regular updates with {@link IgniteCache#put(Object, Object)} and
similar calls use version generated with
+ * {@link GridCacheVersionManager#next(GridCacheVersion)}. This version
value monotonically grows. It is generated
+ * on a primary node and is propagated to the backups. So,
on a primary we can distinguish updates
+ * that happen before and after the dump start.
+ */
+private GridCacheVersion startVer;
+
/**
* @param cctx Cache context.
* @param srcNodeId Node id which cause snapshot task creation.
@@ -211,6 +220,8 @@ public class CreateDumpFutureTask extends
AbstractCreateSnapshotFutureTask imple
/** Prepares all data structures to dump entries. */
private void prepare() throws IOException, IgniteCheckedException {
+startVer =
cctx.versions().next(cctx.kernalContext().discovery().topologyVersion());
+
for (Map.Entry> e : processed.entrySet()) {
int grp = e.getKey();
@@ -424,47 +435,28 @@ public class CreateDumpFutureTask extends
AbstractCreateSnapshotFutureTask imple
*/
private class PartitionDumpContext implements Closeable {
/** Group id. */
-final int grp;
+private final int grp;
/** Partition id. */
-final int part;
+private final int part;
/**
* Key is cache id, values is set of keys dumped via
* {@link #writeChanged(int, long, KeyCacheObject, CacheObject,
GridCacheVersion)}.
*/
-final Map> changed;
+private final Map> changed;
/** Cache id for {@link #iterLastKey} */
-int iterLastKeyCache;
+private int iterLastKeyCache;
/** Last key dumped via {@link #writeForIterator(int, long,
KeyCacheObject, CacheObject, GridCacheVersion, CacheObjectContext)}. */
-@Nullable KeyCacheObject iterLastKey;
+private @Nullable KeyCacheObject iterLastKey;
/** Count of entries changed during dump creation. */
-LongAdder changedCnt = new LongAdder();
+private final LongAdder changedCnt = new LongAdder();
/** Partition dump file. Lazily initialized to prevent creation files
for empty partitions. */
-final FileIO file;
-
-/**
- * Regular updates with {@link IgniteCache#put(Object, Object)} and
similar calls
- * will use version generated with {@link
GridCacheVersionManager#next(GridCacheVersion)}.
- * Version is monotonically increase.
- * Version generated on primary node and propagated to backups.
- * So on primary we can distinguish updates that happens before and
after dump start comparing versions
- * with the version we read with {@link
GridCacheVersionManager#last()}.
- */
-@Nullable final GridCacheVersion startVer;
-
-/**
- * Unlike regular update, {@link IgniteDataStreamer} updates receive
the same version for all entries.
- * See {@code IsolatedUpdater.receive}.
- * Note, using {@link IgniteDataStreamer} during cache dump creation
can lead to dump inconsistency.
- *
- * @see GridCacheVersionManager#isolatedStreamerVersion()
- */
-final GridCacheVersion isolatedStreamerVer;
+private final FileIO file;
/** Topology Version. */
private final AffinityTopologyVersi
(ignite) branch master updated: IGNITE-21449 Avoid excessive invocation of channel.position() while dump create (#11234)
This is an automated email from the ASF dual-hosted git repository.
timoninmaxim pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/ignite.git
The following commit(s) were added to refs/heads/master by this push:
new 5481a2d8169 IGNITE-21449 Avoid excessive invocation of
channel.position() while dump create (#11234)
5481a2d8169 is described below
commit 5481a2d8169dfbb4188efe9b18384f69224c5657
Author: yurinaryshkin <135707807+yurinarysh...@users.noreply.github.com>
AuthorDate: Wed Feb 7 14:58:23 2024 +0300
IGNITE-21449 Avoid excessive invocation of channel.position() while dump
create (#11234)
---
.../cache/persistence/snapshot/dump/BufferedFileIO.java | 9 -
.../cache/persistence/snapshot/dump/IgniteCacheDumpSelfTest.java | 8
2 files changed, 12 insertions(+), 5 deletions(-)
diff --git
a/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/snapshot/dump/BufferedFileIO.java
b/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/snapshot/dump/BufferedFileIO.java
index 40c97df8f3a..84a2d523f29 100644
---
a/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/snapshot/dump/BufferedFileIO.java
+++
b/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/snapshot/dump/BufferedFileIO.java
@@ -36,6 +36,9 @@ public class BufferedFileIO extends FileIODecorator {
/** */
private ByteBuffer buf;
+/** */
+private long position;
+
/** */
public BufferedFileIO(FileIO fileIO) {
super(fileIO);
@@ -107,9 +110,13 @@ public class BufferedFileIO extends FileIODecorator {
private void flush() throws IOException {
buf.flip();
-if (delegate.writeFully(buf) < 0)
+int len = delegate.writeFully(buf, position);
+
+if (len < 0)
throw new IOException("Couldn't write data");
+position = position + len;
+
buf.clear();
}
diff --git
a/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/persistence/snapshot/dump/IgniteCacheDumpSelfTest.java
b/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/persistence/snapshot/dump/IgniteCacheDumpSelfTest.java
index a5dc92cbb8e..741e797d904 100644
---
a/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/persistence/snapshot/dump/IgniteCacheDumpSelfTest.java
+++
b/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/persistence/snapshot/dump/IgniteCacheDumpSelfTest.java
@@ -514,14 +514,14 @@ public class IgniteCacheDumpSelfTest extends
AbstractCacheDumpTest {
if (file.getName().endsWith(DUMP_FILE_EXT)) {
return new FileIODecorator(delegate.create(file, modes)) {
/** {@inheritDoc} */
-@Override public int writeFully(ByteBuffer srcBuf)
throws IOException {
+@Override public int writeFully(ByteBuffer srcBuf,
long position) throws IOException {
if (findValToFail(srcBuf)) {
keyToFailFound.set(true);
throw new IOException("Val to fail found");
}
-return super.writeFully(srcBuf);
+return super.writeFully(srcBuf, position);
}
private boolean findValToFail(ByteBuffer srcBuf) {
@@ -671,9 +671,9 @@ public class IgniteCacheDumpSelfTest extends
AbstractCacheDumpTest {
if (failOnWrite) {
return new FileIODecorator(delegate.create(file, modes)) {
/** {@inheritDoc} */
-@Override public int writeFully(ByteBuffer srcBuf) throws
IOException {
+@Override public int writeFully(ByteBuffer srcBuf, long
position) throws IOException {
if (errorAfter.decrementAndGet() > 0)
-return super.writeFully(srcBuf);
+return super.writeFully(srcBuf, position);
throw new IOException("Test write error");
}
(ignite) branch master updated: IGNITE-21132 Buffered disk writing when creating dump (#11118)
This is an automated email from the ASF dual-hosted git repository.
timoninmaxim pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/ignite.git
The following commit(s) were added to refs/heads/master by this push:
new 5a068eb0f35 IGNITE-21132 Buffered disk writing when creating dump
(#8)
5a068eb0f35 is described below
commit 5a068eb0f35ae45cf4139f60c494fbfef9462369
Author: yurinaryshkin <135707807+yurinarysh...@users.noreply.github.com>
AuthorDate: Thu Feb 1 20:00:21 2024 +0300
IGNITE-21132 Buffered disk writing when creating dump (#8)
---
.../persistence/snapshot/dump/BufferedFileIO.java | 127 ++
...leIOFactory.java => BufferedFileIOFactory.java} | 16 +-
.../snapshot/dump/CreateDumpFutureTask.java| 12 +-
.../snapshot/dump/WriteOnlyZipFileIO.java | 123 ++---
.../snapshot/dump/WriteOnlyZipFileIOFactory.java | 16 +-
.../snapshot/dump/BufferedFileIOTest.java | 194 +
.../snapshot/dump/IgniteCacheDumpSelf2Test.java| 29 ++-
.../snapshot/dump/IgniteCacheDumpSelfTest.java | 9 +-
.../testsuites/IgniteSnapshotTestSuite3.java | 2 +
9 files changed, 410 insertions(+), 118 deletions(-)
diff --git
a/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/snapshot/dump/BufferedFileIO.java
b/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/snapshot/dump/BufferedFileIO.java
new file mode 100644
index 000..40c97df8f3a
--- /dev/null
+++
b/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/snapshot/dump/BufferedFileIO.java
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.processors.cache.persistence.snapshot.dump;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import org.apache.ignite.internal.processors.cache.persistence.file.FileIO;
+import
org.apache.ignite.internal.processors.cache.persistence.file.FileIODecorator;
+import org.apache.ignite.internal.util.typedef.internal.A;
+
+/**
+ * The class implements a {@link FileIO} that allows to write bytes to the
underlying {@link FileIO} without necessarily
+ * causing a call to the underlying system for each byte written.
+ * This IO writes to underlying storage whole buffer thus reduces number of
system calls.
+ * It doesn't support reading or random access. It is not designed for writing
concurrently from several threads.
+ */
+public class BufferedFileIO extends FileIODecorator {
+/** */
+private static final int DEFAULT_BLOCK_SIZE = 4096;
+
+/** */
+private ByteBuffer buf;
+
+/** */
+public BufferedFileIO(FileIO fileIO) {
+super(fileIO);
+
+A.ensure(fileIO != null, "fileIO must not be null");
+
+int blockSize = getFileSystemBlockSize();
+
+if (blockSize <= 0)
+blockSize = DEFAULT_BLOCK_SIZE;
+
+buf = ByteBuffer.allocateDirect(blockSize);
+}
+
+/** {@inheritDoc} */
+@Override public int writeFully(ByteBuffer srcBuf) throws IOException {
+if (buf == null)
+throw new IOException("FileIO closed");
+
+int len = srcBuf.remaining();
+
+writeBytes(srcBuf.array(), srcBuf.arrayOffset() + srcBuf.position(),
len);
+
+srcBuf.position(srcBuf.position() + len);
+
+return len;
+}
+
+/**
+ * Writes byte array.
+ * @throws IOException If some I/O error occurs.
+ */
+protected void writeBytes(byte[] srcBuf, int off, int len) throws
IOException {
+if (buf == null)
+throw new IOException("FileIO closed");
+
+if ((len | off) < 0 || len > srcBuf.length - off)
+throw new IndexOutOfBoundsException("Range [" + off + ", " + off +
" + " + len + ") out of bounds for length " + srcBuf.length);
+
+int p = off;
+
+while (p < off + len) {
+int bytesCnt = Math.min(buf.remaining(), off + len - p);
+
+buf.put(srcBuf, p, bytesCnt);
+
+if (!buf.hasRemaining(
(ignite) branch master updated: IGNITE-21135 Fix CdcManager collects MemoryRecoveryRecord before restore state (#11120)
This is an automated email from the ASF dual-hosted git repository.
timoninmaxim pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/ignite.git
The following commit(s) were added to refs/heads/master by this push:
new 23fc12f8858 IGNITE-21135 Fix CdcManager collects MemoryRecoveryRecord
before restore state (#11120)
23fc12f8858 is described below
commit 23fc12f88583d78764b350341e4a8e08e7c905fe
Author: Maksim Timonin
AuthorDate: Fri Dec 22 20:45:46 2023 +0300
IGNITE-21135 Fix CdcManager collects MemoryRecoveryRecord before restore
state (#11120)
---
.../org/apache/ignite/internal/cdc/CdcManager.java | 10 +---
.../GridCacheDatabaseSharedManager.java| 3 +
.../java/org/apache/ignite/cdc/CdcManagerTest.java | 69 ++
3 files changed, 74 insertions(+), 8 deletions(-)
diff --git
a/modules/core/src/main/java/org/apache/ignite/internal/cdc/CdcManager.java
b/modules/core/src/main/java/org/apache/ignite/internal/cdc/CdcManager.java
index af9bed790e5..262ced12d42 100644
--- a/modules/core/src/main/java/org/apache/ignite/internal/cdc/CdcManager.java
+++ b/modules/core/src/main/java/org/apache/ignite/internal/cdc/CdcManager.java
@@ -25,13 +25,11 @@ import org.apache.ignite.cdc.CdcConsumer;
import org.apache.ignite.internal.pagemem.wal.record.CdcManagerRecord;
import org.apache.ignite.internal.pagemem.wal.record.CdcManagerStopRecord;
import org.apache.ignite.internal.processors.cache.GridCacheSharedManager;
-import
org.apache.ignite.internal.processors.cache.persistence.DatabaseLifecycleListener;
import
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager;
import
org.apache.ignite.internal.processors.cache.persistence.IgniteCacheDatabaseSharedManager;
import org.apache.ignite.internal.processors.cache.persistence.wal.WALPointer;
import
org.apache.ignite.internal.processors.cache.persistence.wal.io.FileInput;
import
org.apache.ignite.internal.processors.cache.persistence.wal.serializer.RecordSerializer;
-import
org.apache.ignite.internal.processors.subscription.GridInternalSubscriptionProcessor;
import org.apache.ignite.lang.IgniteExperimental;
import org.apache.ignite.plugin.PluginContext;
import org.apache.ignite.plugin.PluginProvider;
@@ -65,7 +63,7 @@ import org.apache.ignite.plugin.PluginProvider;
* @see CdcConsumerState#saveCdcMode(CdcMode)
*/
@IgniteExperimental
-public interface CdcManager extends GridCacheSharedManager,
DatabaseLifecycleListener {
+public interface CdcManager extends GridCacheSharedManager {
/**
* If this manager isn't enabled then Ignite skips notifying the manager
with following methods.
*
@@ -78,16 +76,12 @@ public interface CdcManager extends GridCacheSharedManager,
DatabaseLifecycleLis
* It's executed before the first call of {@link #collect(ByteBuffer)}.
* Implementation suggestions:
*
- *
- * Implementation must subscribe to the database events to get
notified with this callback via
- * {@link
GridInternalSubscriptionProcessor#registerDatabaseListener(DatabaseLifecycleListener)}
- *
* Callback can be used for restoring CDC state on Ignite node
start, collecting missed events from WAL segments.
* Be aware, this method runs in the Ignite system thread and
might get longer the Ignite start procedure.
* Ignite node will fail in case the method throws an
exception.
*
*/
-@Override public default void
afterBinaryMemoryRestore(IgniteCacheDatabaseSharedManager mgr,
+public default void
afterBinaryMemoryRestore(IgniteCacheDatabaseSharedManager mgr,
GridCacheDatabaseSharedManager.RestoreBinaryState restoreState) throws
IgniteCheckedException {
// No-op.
}
diff --git
a/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/GridCacheDatabaseSharedManager.java
b/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/GridCacheDatabaseSharedManager.java
index b6e35b706e7..298dfd6233d 100755
---
a/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/GridCacheDatabaseSharedManager.java
+++
b/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/GridCacheDatabaseSharedManager.java
@@ -1102,6 +1102,9 @@ public class GridCacheDatabaseSharedManager extends
IgniteCacheDatabaseSharedMan
// Wal logging is now available.
cctx.wal().resumeLogging(restored);
+if (cctx.cdc() != null && cctx.cdc().enabled())
+cctx.cdc().afterBinaryMemoryRestore(this, binaryState);
+
// Log MemoryRecoveryRecord to make sure that old physical records
are not replayed during
// next physical recovery.
checkpointManager.memoryRecoveryRecordPtr(cctx.wal().log(new
MemoryRecoveryRecord(U.currentTimeMillis(;
di
(ignite) branch master updated: IGNITE-21123 Add CdcEvent capture metric (#11111)
This is an automated email from the ASF dual-hosted git repository.
timoninmaxim pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/ignite.git
The following commit(s) were added to refs/heads/master by this push:
new a8462443617 IGNITE-21123 Add CdcEvent capture metric (#1)
a8462443617 is described below
commit a8462443617ca14efb11ca9a338aeb58c4c1a6f0
Author: Maksim Timonin
AuthorDate: Thu Dec 21 15:04:08 2023 +0300
IGNITE-21123 Add CdcEvent capture metric (#1)
---
.../java/org/apache/ignite/internal/cdc/CdcMain.java | 19 ++-
.../ignite/internal/cdc/WalRecordsConsumer.java | 15 +--
.../java/org/apache/ignite/cdc/AbstractCdcTest.java | 5 +
3 files changed, 36 insertions(+), 3 deletions(-)
diff --git
a/modules/core/src/main/java/org/apache/ignite/internal/cdc/CdcMain.java
b/modules/core/src/main/java/org/apache/ignite/internal/cdc/CdcMain.java
index 33b23af82a9..2f9269b18b1 100644
--- a/modules/core/src/main/java/org/apache/ignite/internal/cdc/CdcMain.java
+++ b/modules/core/src/main/java/org/apache/ignite/internal/cdc/CdcMain.java
@@ -52,6 +52,7 @@ import org.apache.ignite.internal.binary.BinaryUtils;
import org.apache.ignite.internal.cdc.WalRecordsConsumer.DataEntryIterator;
import org.apache.ignite.internal.pagemem.wal.WALIterator;
import org.apache.ignite.internal.pagemem.wal.record.CdcManagerRecord;
+import org.apache.ignite.internal.pagemem.wal.record.DataRecord;
import org.apache.ignite.internal.pagemem.wal.record.WALRecord;
import org.apache.ignite.internal.processors.cache.GridLocalConfigManager;
import
org.apache.ignite.internal.processors.cache.binary.CacheObjectBinaryProcessorImpl;
@@ -157,6 +158,9 @@ public class CdcMain implements Runnable {
/** Metadata update time. */
public static final String META_UPDATE = "MetadataUpdateTime";
+/** Event capture time. */
+public static final String EVT_CAPTURE_TIME = "EventCaptureTime";
+
/** Binary metadata metric name. */
public static final String BINARY_META_DIR = "BinaryMetaDir";
@@ -201,6 +205,12 @@ public class CdcMain implements Runnable {
/** Metadata update time. */
private HistogramMetricImpl metaUpdate;
+/**
+ * Metric represents time between creating {@link DataRecord}, containing
the data change events, and capturing them
+ * by {@link CdcConsumer}.
+ */
+private HistogramMetricImpl evtCaptureTime;
+
/** Change Data Capture configuration. */
protected final CdcConfiguration cdcCfg;
@@ -423,6 +433,10 @@ public class CdcMain implements Runnable {
lastSegmentConsumptionTs =
mreg.longMetric(LAST_SEG_CONSUMPTION_TIME, "Last time of
consumption of WAL segment");
metaUpdate = mreg.histogram(META_UPDATE, new long[] {100, 500, 1000},
"Metadata update time");
+evtCaptureTime = mreg.histogram(
+EVT_CAPTURE_TIME,
+new long[] {5_000, 10_000, 15_000, 30_000, 60_000},
+"Time between creating an event on Ignite node and capturing it by
CdcConsumer");
mreg.register(CDC_MODE, () -> cdcModeState.name(), String.class, "CDC
mode");
}
@@ -572,7 +586,10 @@ public class CdcMain implements Runnable {
* Consumes CDC events in {@link CdcMode#CDC_UTILITY_ACTIVE} mode.
*/
private void
consumeSegmentActively(IgniteWalIteratorFactory.IteratorParametersBuilder
builder) {
-try (DataEntryIterator iter = new DataEntryIterator(new
IgniteWalIteratorFactory(log).iterator(builder.addFilter(ACTIVE_RECS {
+try (DataEntryIterator iter = new DataEntryIterator(
+new
IgniteWalIteratorFactory(log).iterator(builder.addFilter(ACTIVE_RECS)),
+evtCaptureTime)
+) {
if (walState != null)
iter.init(walState.get2());
diff --git
a/modules/core/src/main/java/org/apache/ignite/internal/cdc/WalRecordsConsumer.java
b/modules/core/src/main/java/org/apache/ignite/internal/cdc/WalRecordsConsumer.java
index 2167e27934a..1cbf69bb3f4 100644
---
a/modules/core/src/main/java/org/apache/ignite/internal/cdc/WalRecordsConsumer.java
+++
b/modules/core/src/main/java/org/apache/ignite/internal/cdc/WalRecordsConsumer.java
@@ -31,12 +31,14 @@ import org.apache.ignite.cdc.TypeMapping;
import org.apache.ignite.internal.pagemem.wal.WALIterator;
import org.apache.ignite.internal.pagemem.wal.record.DataEntry;
import org.apache.ignite.internal.pagemem.wal.record.DataRecord;
+import org.apache.ignite.internal.pagemem.wal.record.TimeStampRecord;
import org.apache.ignite.internal.pagemem.wal.record.UnwrappedDataEntry;
import org.apache.ignite.internal.pagemem.wal.record.WALRecord;
import org.apache.ignite.internal.processors.cache.GridCacheOperation;
import org.apache.ignite.internal.processors.cache.persistence.wal.WALPointer;
im
(ignite) branch master updated: IGNITE-21051 Bulletpoints added for index discovering algorithms in Javadoc in IndexQuery class (#11091)
This is an automated email from the ASF dual-hosted git repository.
timoninmaxim pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/ignite.git
The following commit(s) were added to refs/heads/master by this push:
new 136e9a31587 IGNITE-21051 Bulletpoints added for index discovering
algorithms in Javadoc in IndexQuery class (#11091)
136e9a31587 is described below
commit 136e9a31587b384ae9645fce3999a96c2426f30b
Author: oleg-vlsk <153691984+oleg-v...@users.noreply.github.com>
AuthorDate: Mon Dec 18 22:24:09 2023 +1000
IGNITE-21051 Bulletpoints added for index discovering algorithms in Javadoc
in IndexQuery class (#11091)
---
.../java/org/apache/ignite/cache/query/IndexQuery.java| 15 ++-
1 file changed, 10 insertions(+), 5 deletions(-)
diff --git
a/modules/core/src/main/java/org/apache/ignite/cache/query/IndexQuery.java
b/modules/core/src/main/java/org/apache/ignite/cache/query/IndexQuery.java
index 8c61ccc1936..69874941c80 100644
--- a/modules/core/src/main/java/org/apache/ignite/cache/query/IndexQuery.java
+++ b/modules/core/src/main/java/org/apache/ignite/cache/query/IndexQuery.java
@@ -30,12 +30,17 @@ import org.jetbrains.annotations.Nullable;
/**
* Index queries work over distributed indexes and retrieve cache entries that
match the specified criteria.
+ *
* {@code QueryCursor} delivers sorted cache entries by the order defined for
queried index.
- *
- * {@code IndexQuery} has to be initialized with cache value class or type.
The algorithm of discovering index is as follows:
- * 1. If {@link #idxName} is set, then use it.
- * 2. If {@link #idxName} is not set, then find an index that matches criteria
fields.
- * 3. If neither {@link #idxName}, nor {@link #setCriteria(List)} is used,
then perform index scan over PK index for specified Value type.
+ *
+ * {@code IndexQuery} has to be initialized with cache value class or type.
The algorithm of discovering index is
+ * as follows:
+ *
+ * If {@link #idxName} is set, then use it.
+ * If {@link #idxName} is not set, then find an index that matches
criteria fields.
+ * If neither {@link #idxName}, nor {@link #setCriteria(List)} is
used, then perform index scan over PK index
+ * for specified Value type.
+ *
*
* Conjuction of items in {@link #criteria} has to represent a valid range to
traverse the index tree.
*/
(ignite) 04/05: IGNITE-21041 Make FilteredRecord local for RecordSerializer (#11082)
This is an automated email from the ASF dual-hosted git repository.
timoninmaxim pushed a commit to branch ignite-2.16
in repository https://gitbox.apache.org/repos/asf/ignite.git
commit 369644c989e85aa4c889d89a2a9de69cf8124f05
Author: Maksim Timonin
AuthorDate: Fri Dec 8 15:23:42 2023 +0300
IGNITE-21041 Make FilteredRecord local for RecordSerializer (#11082)
---
.../apache/ignite/internal/pagemem/wal/record/FilteredRecord.java| 3 ---
.../cache/persistence/wal/reader/StandaloneWalRecordsIterator.java | 5 -
.../cache/persistence/wal/serializer/RecordV1Serializer.java | 5 -
.../cache/persistence/wal/serializer/RecordV2Serializer.java | 5 -
4 files changed, 12 insertions(+), 6 deletions(-)
diff --git
a/modules/core/src/main/java/org/apache/ignite/internal/pagemem/wal/record/FilteredRecord.java
b/modules/core/src/main/java/org/apache/ignite/internal/pagemem/wal/record/FilteredRecord.java
index e077e5accc2..519e825953d 100644
---
a/modules/core/src/main/java/org/apache/ignite/internal/pagemem/wal/record/FilteredRecord.java
+++
b/modules/core/src/main/java/org/apache/ignite/internal/pagemem/wal/record/FilteredRecord.java
@@ -24,9 +24,6 @@ import
org.apache.ignite.internal.processors.cache.persistence.wal.AbstractWalRe
* {@link AbstractWalRecordsIterator}.
*/
public class FilteredRecord extends WALRecord {
-/** Instance. */
-public static final FilteredRecord INSTANCE = new FilteredRecord();
-
/** {@inheritDoc} */
@Override public RecordType type() {
return null;
diff --git
a/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/wal/reader/StandaloneWalRecordsIterator.java
b/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/wal/reader/StandaloneWalRecordsIterator.java
index c7582b7cf14..26dd7c3e647 100644
---
a/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/wal/reader/StandaloneWalRecordsIterator.java
+++
b/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/wal/reader/StandaloneWalRecordsIterator.java
@@ -96,6 +96,9 @@ class StandaloneWalRecordsIterator extends
AbstractWalRecordsIterator {
/** Replay from bound include. */
private final WALPointer lowBound;
+/** Singleton instance of {@link FilteredRecord} */
+private final WALRecord filteredRecord = new FilteredRecord();
+
/**
* Creates iterator in file-by-file iteration mode. Directory
*
@@ -286,7 +289,7 @@ class StandaloneWalRecordsIterator extends
AbstractWalRecordsIterator {
return tup;
if (!checkBounds(tup.get1()))
-return new T2<>(tup.get1(), FilteredRecord.INSTANCE); //
FilteredRecord for mark as filtered.
+return new T2<>(tup.get1(), filteredRecord); // FilteredRecord for
mark as filtered.
return tup;
}
diff --git
a/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/wal/serializer/RecordV1Serializer.java
b/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/wal/serializer/RecordV1Serializer.java
index 295e174abf8..52c360d47eb 100644
---
a/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/wal/serializer/RecordV1Serializer.java
+++
b/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/wal/serializer/RecordV1Serializer.java
@@ -98,6 +98,9 @@ public class RecordV1Serializer implements RecordSerializer {
*/
private final boolean marshalledMode;
+/** Singleton instance of {@link FilteredRecord} */
+private final FilteredRecord filteredRecord = new FilteredRecord();
+
/** Thread-local heap byte buffer. */
private final ThreadLocal heapTlb = new
ThreadLocal() {
@Override protected ByteBuffer initialValue() {
@@ -148,7 +151,7 @@ public class RecordV1Serializer implements RecordSerializer
{
if (recType.purpose() != WALRecord.RecordPurpose.INTERNAL
&& recordFilter != null && !recordFilter.apply(rec.type(),
ptr))
-return FilteredRecord.INSTANCE;
+return filteredRecord;
else if (marshalledMode) {
ByteBuffer buf = heapTlb.get();
diff --git
a/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/wal/serializer/RecordV2Serializer.java
b/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/wal/serializer/RecordV2Serializer.java
index 60c68c13f7c..21aa52a91ba 100644
---
a/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/wal/serializer/RecordV2Serializer.java
+++
b/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/wal/serializer/RecordV2Serializer.java
@@ -74,6 +74,9 @@ public class RecordV2Serializer implements RecordSeriali
(ignite) branch ignite-2.16 updated (d18d899ba2e -> 6b2d1773af7)
This is an automated email from the ASF dual-hosted git repository.
timoninmaxim pushed a change to branch ignite-2.16
in repository https://gitbox.apache.org/repos/asf/ignite.git
from d18d899ba2e [MINOR] Fix error message in test
new f46fa294cb2 IGNITE-17700 Introduce CdcManager (#11044)
new 654b467d779 IGNITE-21042 Move CreateDumpFutureTask#toLong to
IgniteUtils (#11048)
new 5ab68b2be79 IGNITE-20822 Filter by higherBound in
AbstractWalRecordsIterator (#11034)
new 369644c989e IGNITE-21041 Make FilteredRecord local for
RecordSerializer (#11082)
new 6b2d1773af7 IGNITE-21046 Move binaryMetadataUpdateListener to basic
interface (#11049)
The 5 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.../ignite/internal/cdc/CdcConsumerState.java | 39 ++
.../org/apache/ignite/internal/cdc/CdcMain.java| 240 +---
.../org/apache/ignite/internal/cdc/CdcManager.java | 123 ++
.../org/apache/ignite/internal/cdc/CdcMode.java} | 19 +-
.../internal/cdc/CdcUtilityActiveCdcManager.java | 73
.../{CdcDataRecord.java => CdcManagerRecord.java} | 30 +-
...dcDataRecord.java => CdcManagerStopRecord.java} | 21 +-
.../pagemem/wal/record/FilteredRecord.java | 3 -
.../internal/pagemem/wal/record/WALRecord.java | 8 +-
.../processors/cache/GridCacheProcessor.java | 27 ++
.../processors/cache/GridCacheSharedContext.java | 37 +-
.../binary/CacheObjectBinaryProcessorImpl.java | 6 +-
.../IgniteCacheDatabaseSharedManager.java | 4 +
.../snapshot/dump/CreateDumpFutureTask.java| 8 +-
.../wal/AbstractWalRecordsIterator.java| 10 +
.../persistence/wal/FileWriteAheadLogManager.java | 19 +-
.../wal/SingleSegmentLogicalRecordsIterator.java | 9 +-
.../wal/filehandle/FileWriteHandleImpl.java| 42 ++-
.../wal/reader/StandaloneWalRecordsIterator.java | 19 +-
.../wal/serializer/RecordDataV1Serializer.java | 34 ++
.../wal/serializer/RecordV1Serializer.java | 5 +-
.../wal/serializer/RecordV2Serializer.java | 5 +-
.../cacheobject/IgniteCacheObjectProcessor.java| 8 +
.../apache/ignite/internal/util/IgniteUtils.java | 11 +
.../ignite/cdc/CdcIgniteNodeActiveModeTest.java| 394 +++
.../java/org/apache/ignite/cdc/CdcManagerTest.java | 415 +
.../java/org/apache/ignite/cdc/WalForCdcTest.java | 4 +
.../snapshot/dump/AbstractCacheDumpTest.java | 2 +-
.../testframework/wal/record/RecordUtils.java | 18 +
.../ignite/testsuites/IgnitePdsTestSuite2.java | 4 +
30 files changed, 1485 insertions(+), 152 deletions(-)
create mode 100644
modules/core/src/main/java/org/apache/ignite/internal/cdc/CdcManager.java
copy
modules/{calcite/src/main/java/org/apache/ignite/internal/processors/query/calcite/exec/exp/agg/AggregateType.java
=> core/src/main/java/org/apache/ignite/internal/cdc/CdcMode.java} (75%)
create mode 100644
modules/core/src/main/java/org/apache/ignite/internal/cdc/CdcUtilityActiveCdcManager.java
copy
modules/core/src/main/java/org/apache/ignite/internal/pagemem/wal/record/{CdcDataRecord.java
=> CdcManagerRecord.java} (53%)
copy
modules/core/src/main/java/org/apache/ignite/internal/pagemem/wal/record/{CdcDataRecord.java
=> CdcManagerStopRecord.java} (66%)
create mode 100644
modules/core/src/test/java/org/apache/ignite/cdc/CdcIgniteNodeActiveModeTest.java
create mode 100644
modules/core/src/test/java/org/apache/ignite/cdc/CdcManagerTest.java
(ignite) 03/05: IGNITE-20822 Filter by higherBound in AbstractWalRecordsIterator (#11034)
This is an automated email from the ASF dual-hosted git repository.
timoninmaxim pushed a commit to branch ignite-2.16
in repository https://gitbox.apache.org/repos/asf/ignite.git
commit 5ab68b2be79f28ed4cb72c95efb223b033412826
Author: Maksim Timonin
AuthorDate: Mon Nov 13 17:39:43 2023 +0300
IGNITE-20822 Filter by higherBound in AbstractWalRecordsIterator (#11034)
---
.../persistence/wal/AbstractWalRecordsIterator.java | 10 ++
.../cache/persistence/wal/FileWriteAheadLogManager.java | 13 +
.../wal/SingleSegmentLogicalRecordsIterator.java | 9 -
.../wal/reader/StandaloneWalRecordsIterator.java | 16 ++--
4 files changed, 25 insertions(+), 23 deletions(-)
diff --git
a/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/wal/AbstractWalRecordsIterator.java
b/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/wal/AbstractWalRecordsIterator.java
index 926bce8fe39..c020a30be29 100644
---
a/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/wal/AbstractWalRecordsIterator.java
+++
b/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/wal/AbstractWalRecordsIterator.java
@@ -97,6 +97,9 @@ public abstract class AbstractWalRecordsIterator
/** Factory to provide I/O interfaces for read primitives with files. */
private final SegmentFileInputFactory segmentFileInputFactory;
+/** Optional inclusive high bound. */
+protected final @Nullable WALPointer highBound;
+
/** Position of last read valid record. */
private WALPointer lastRead;
@@ -106,6 +109,7 @@ public abstract class AbstractWalRecordsIterator
* @param serializerFactory Serializer of current version to read headers.
* @param ioFactory ioFactory for file IO access.
* @param initialReadBufferSize buffer for reading records size.
+ * @param highBound Optional inclusive high bound.
* @param segmentFileInputFactory Factory to provide I/O interfaces for
read primitives with files.
*/
protected AbstractWalRecordsIterator(
@@ -114,12 +118,14 @@ public abstract class AbstractWalRecordsIterator
@NotNull final RecordSerializerFactory serializerFactory,
@NotNull final FileIOFactory ioFactory,
final int initialReadBufferSize,
+@Nullable WALPointer highBound,
SegmentFileInputFactory segmentFileInputFactory) {
this.log = log;
this.sharedCtx = sharedCtx;
this.serializerFactory = serializerFactory;
this.ioFactory = ioFactory;
this.segmentFileInputFactory = segmentFileInputFactory;
+this.highBound = highBound;
buf = new ByteBufferExpander(initialReadBufferSize,
ByteOrder.nativeOrder());
}
@@ -269,6 +275,10 @@ public abstract class AbstractWalRecordsIterator
WALPointer actualFilePtr = new WALPointer(hnd.idx(),
(int)hnd.in().position(), 0);
+// Fast stop condition, after high bound reached.
+if (highBound != null && actualFilePtr.compareTo(highBound) > 0)
+return null;
+
try {
WALRecord rec = hnd.ser().readRecord(hnd.in(), actualFilePtr);
diff --git
a/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/wal/FileWriteAheadLogManager.java
b/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/wal/FileWriteAheadLogManager.java
index 2b74c48d8c4..12c40e81081 100644
---
a/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/wal/FileWriteAheadLogManager.java
+++
b/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/wal/FileWriteAheadLogManager.java
@@ -2976,9 +2976,6 @@ public class FileWriteAheadLogManager extends
GridCacheSharedManagerAdapter impl
/** Optional start pointer. */
@Nullable private final WALPointer start;
-/** Optional end pointer. */
-@Nullable private final WALPointer end;
-
/** Manager of segment location. */
private final SegmentRouter segmentRouter;
@@ -3022,6 +3019,7 @@ public class FileWriteAheadLogManager extends
GridCacheSharedManagerAdapter impl
serializerFactory,
ioFactory,
dsCfg.getWalRecordIteratorBufferSize(),
+end,
segmentFileInputFactory
);
@@ -3029,7 +3027,6 @@ public class FileWriteAheadLogManager extends
GridCacheSharedManagerAdapter impl
this.walWorkDir = walWorkDir;
this.archiver = archiver;
this.start = start;
-this.end = end;
this.dsCfg = dsCfg;
this.decompressor = decompressor;
@@ -3116,7 +3113,7 @@ public class FileWriteAheadLogManager extends
GridCacheSharedManagerAdapter impl
cu
(ignite) 01/05: IGNITE-17700 Introduce CdcManager (#11044)
This is an automated email from the ASF dual-hosted git repository.
timoninmaxim pushed a commit to branch ignite-2.16
in repository https://gitbox.apache.org/repos/asf/ignite.git
commit f46fa294cb21bdf1b979ad2d0866bdc28ed9edc6
Author: Maksim Timonin
AuthorDate: Thu Dec 7 19:33:58 2023 +0300
IGNITE-17700 Introduce CdcManager (#11044)
Co-authored-by: Nikolay Izhikov
---
.../ignite/internal/cdc/CdcConsumerState.java | 39 ++
.../org/apache/ignite/internal/cdc/CdcMain.java| 240 +---
.../org/apache/ignite/internal/cdc/CdcManager.java | 123 ++
.../org/apache/ignite/internal/cdc/CdcMode.java| 27 ++
.../internal/cdc/CdcUtilityActiveCdcManager.java | 73
.../pagemem/wal/record/CdcManagerRecord.java | 50 +++
.../pagemem/wal/record/CdcManagerStopRecord.java | 33 ++
.../internal/pagemem/wal/record/WALRecord.java | 8 +-
.../processors/cache/GridCacheProcessor.java | 27 ++
.../processors/cache/GridCacheSharedContext.java | 37 +-
.../IgniteCacheDatabaseSharedManager.java | 4 +
.../persistence/wal/FileWriteAheadLogManager.java | 6 +-
.../wal/filehandle/FileWriteHandleImpl.java| 42 ++-
.../wal/serializer/RecordDataV1Serializer.java | 34 ++
.../ignite/cdc/CdcIgniteNodeActiveModeTest.java| 394 +++
.../java/org/apache/ignite/cdc/CdcManagerTest.java | 415 +
.../java/org/apache/ignite/cdc/WalForCdcTest.java | 4 +
.../testframework/wal/record/RecordUtils.java | 18 +
.../ignite/testsuites/IgnitePdsTestSuite2.java | 4 +
19 files changed, 1501 insertions(+), 77 deletions(-)
diff --git
a/modules/core/src/main/java/org/apache/ignite/internal/cdc/CdcConsumerState.java
b/modules/core/src/main/java/org/apache/ignite/internal/cdc/CdcConsumerState.java
index 379c2bb6f95..942fd248da6 100644
---
a/modules/core/src/main/java/org/apache/ignite/internal/cdc/CdcConsumerState.java
+++
b/modules/core/src/main/java/org/apache/ignite/internal/cdc/CdcConsumerState.java
@@ -71,6 +71,15 @@ public class CdcConsumerState {
/** */
public static final String CACHES_STATE_FILE_NAME = "cdc-caches-state" +
FILE_SUFFIX;
+/**
+ * The file stores state of CDC mode. Content of the file is a {@link
CdcMode} value:
+ *
+ * {@link CdcMode#CDC_UTILITY_ACTIVE} means that {@link CdcMain}
utility captures data.
+ * {@link CdcMode#IGNITE_NODE_ACTIVE} means that {@link
CdcManager} captures data within Ignite node.
+ *
+ */
+public static final String CDC_MODE_FILE_NAME = "cdc-mode" + FILE_SUFFIX;
+
/** Log. */
private final IgniteLogger log;
@@ -98,6 +107,12 @@ public class CdcConsumerState {
/** Mappings types state file. */
private final Path tmpCaches;
+/** CDC manager mode state file. */
+private final Path cdcMode;
+
+/** Temp CDC manager mode state file. */
+private final Path tmpCdcMode;
+
/**
* @param stateDir State directory.
*/
@@ -111,6 +126,8 @@ public class CdcConsumerState {
tmpMappings = stateDir.resolve(MAPPINGS_STATE_FILE_NAME + TMP_SUFFIX);
caches = stateDir.resolve(CACHES_STATE_FILE_NAME);
tmpCaches = stateDir.resolve(CACHES_STATE_FILE_NAME + TMP_SUFFIX);
+cdcMode = stateDir.resolve(CDC_MODE_FILE_NAME);
+tmpCdcMode = stateDir.resolve(CDC_MODE_FILE_NAME + TMP_SUFFIX);
}
/**
@@ -278,4 +295,26 @@ public class CdcConsumerState {
throw new RuntimeException(e);
}
}
+
+/**
+ * Loads CDC mode state from file.
+ *
+ * @return CDC mode state.
+ */
+public CdcMode loadCdcMode() {
+CdcMode state = load(cdcMode, () -> CdcMode.IGNITE_NODE_ACTIVE);
+
+log.info("CDC mode loaded [" + state + ']');
+
+return state;
+}
+
+/**
+ * Saves CDC mode state to file.
+ *
+ * @param mode CDC mode.
+ */
+public void saveCdcMode(CdcMode mode) throws IOException {
+save(mode, tmpCdcMode, cdcMode);
+}
}
diff --git
a/modules/core/src/main/java/org/apache/ignite/internal/cdc/CdcMain.java
b/modules/core/src/main/java/org/apache/ignite/internal/cdc/CdcMain.java
index 84dc0a02dc9..cbf44b0b11e 100644
--- a/modules/core/src/main/java/org/apache/ignite/internal/cdc/CdcMain.java
+++ b/modules/core/src/main/java/org/apache/ignite/internal/cdc/CdcMain.java
@@ -50,6 +50,9 @@ import
org.apache.ignite.internal.IgniteInterruptedCheckedException;
import org.apache.ignite.internal.MarshallerContextImpl;
import org.apache.ignite.internal.binary.BinaryUtils;
import org.apache.ignite.internal.cdc.WalRecordsConsumer.DataEntryIterator;
+import org.apache.ignite.internal.pagemem.wal.WALIterator;
+import org.apache.ignite.internal.pagemem.wal.record.CdcManagerRecord;
+import org.apache.ignite.internal.pagemem.wal.record.WALRecord;
import org.apache.ignite.internal.processors.cach
(ignite) 05/05: IGNITE-21046 Move binaryMetadataUpdateListener to basic interface (#11049)
This is an automated email from the ASF dual-hosted git repository.
timoninmaxim pushed a commit to branch ignite-2.16
in repository https://gitbox.apache.org/repos/asf/ignite.git
commit 6b2d1773af715ab2eab4afe0138294dbefb3c24f
Author: Maksim Timonin
AuthorDate: Fri Dec 8 15:29:19 2023 +0300
IGNITE-21046 Move binaryMetadataUpdateListener to basic interface (#11049)
---
.../processors/cache/binary/CacheObjectBinaryProcessorImpl.java | 6 ++
.../processors/cacheobject/IgniteCacheObjectProcessor.java| 8
2 files changed, 10 insertions(+), 4 deletions(-)
diff --git
a/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/binary/CacheObjectBinaryProcessorImpl.java
b/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/binary/CacheObjectBinaryProcessorImpl.java
index 7ca52e66449..4e1bc0b59c8 100644
---
a/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/binary/CacheObjectBinaryProcessorImpl.java
+++
b/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/binary/CacheObjectBinaryProcessorImpl.java
@@ -372,10 +372,8 @@ public class CacheObjectBinaryProcessorImpl extends
GridProcessorAdapter impleme
}
}
-/**
- * @param lsnr Listener.
- */
-public void addBinaryMetadataUpdateListener(BinaryMetadataUpdatedListener
lsnr) {
+/** {@inheritDoc} */
+@Override public void
addBinaryMetadataUpdateListener(BinaryMetadataUpdatedListener lsnr) {
if (transport != null)
transport.addBinaryMetadataUpdateListener(lsnr);
}
diff --git
a/modules/core/src/main/java/org/apache/ignite/internal/processors/cacheobject/IgniteCacheObjectProcessor.java
b/modules/core/src/main/java/org/apache/ignite/internal/processors/cacheobject/IgniteCacheObjectProcessor.java
index 68953161430..1b3cd9bcc88 100644
---
a/modules/core/src/main/java/org/apache/ignite/internal/processors/cacheobject/IgniteCacheObjectProcessor.java
+++
b/modules/core/src/main/java/org/apache/ignite/internal/processors/cacheobject/IgniteCacheObjectProcessor.java
@@ -39,6 +39,7 @@ import
org.apache.ignite.internal.processors.cache.CacheObjectValueContext;
import org.apache.ignite.internal.processors.cache.GridCacheContext;
import org.apache.ignite.internal.processors.cache.IncompleteCacheObject;
import org.apache.ignite.internal.processors.cache.KeyCacheObject;
+import
org.apache.ignite.internal.processors.cache.binary.BinaryMetadataUpdatedListener;
import org.jetbrains.annotations.Nullable;
/**
@@ -377,4 +378,11 @@ public interface IgniteCacheObjectProcessor extends
GridProcessor {
* @throws org.apache.ignite.binary.BinaryObjectException In case of error.
*/
public BinaryType registerClass(Class cls) throws BinaryObjectException;
+
+/**
+ * Add binary metadata update listener.
+ *
+ * @param lsnr Listener.
+ */
+public void addBinaryMetadataUpdateListener(BinaryMetadataUpdatedListener
lsnr);
}
(ignite) 02/05: IGNITE-21042 Move CreateDumpFutureTask#toLong to IgniteUtils (#11048)
This is an automated email from the ASF dual-hosted git repository.
timoninmaxim pushed a commit to branch ignite-2.16
in repository https://gitbox.apache.org/repos/asf/ignite.git
commit 654b467d779550444367d5fd844bb8c2680a6a48
Author: Maksim Timonin
AuthorDate: Fri Dec 8 15:16:57 2023 +0300
IGNITE-21042 Move CreateDumpFutureTask#toLong to IgniteUtils (#11048)
---
.../cache/persistence/snapshot/dump/CreateDumpFutureTask.java | 8 ++--
.../java/org/apache/ignite/internal/util/IgniteUtils.java | 11 +++
.../persistence/snapshot/dump/AbstractCacheDumpTest.java | 2 +-
3 files changed, 14 insertions(+), 7 deletions(-)
diff --git
a/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/snapshot/dump/CreateDumpFutureTask.java
b/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/snapshot/dump/CreateDumpFutureTask.java
index d0654e719b0..3f181b71a97 100644
---
a/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/snapshot/dump/CreateDumpFutureTask.java
+++
b/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/snapshot/dump/CreateDumpFutureTask.java
@@ -75,6 +75,7 @@ import static
org.apache.ignite.internal.processors.cache.persistence.file.FileP
import static
org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.CACHE_GRP_DIR_PREFIX;
import static
org.apache.ignite.internal.processors.cache.persistence.snapshot.IgniteSnapshotManager.DUMP_LOCK;
import static
org.apache.ignite.internal.processors.cache.persistence.snapshot.dump.Dump.dumpPartFileName;
+import static org.apache.ignite.internal.util.IgniteUtils.toLong;
/**
* Task creates cache group dump.
@@ -111,7 +112,7 @@ public class CreateDumpFutureTask extends
AbstractCreateSnapshotFutureTask imple
* Dump contextes.
* Key is [group_id, partition_id] combined in single long value.
*
- * @see #toLong(int, int)
+ * @see IgniteUtils#toLong(int, int)
*/
private final Map dumpCtxs = new
ConcurrentHashMap<>();
@@ -631,11 +632,6 @@ public class CreateDumpFutureTask extends
AbstractCreateSnapshotFutureTask imple
}
}
-/** */
-public static long toLong(int high, int low) {
-return (((long)high) << Integer.SIZE) | (low & 0xL);
-}
-
/** */
private File groupDirectory(CacheGroupContext grpCtx) throws
IgniteCheckedException {
return new File(
diff --git
a/modules/core/src/main/java/org/apache/ignite/internal/util/IgniteUtils.java
b/modules/core/src/main/java/org/apache/ignite/internal/util/IgniteUtils.java
index 2f736091ed1..de17a4eea7f 100755
---
a/modules/core/src/main/java/org/apache/ignite/internal/util/IgniteUtils.java
+++
b/modules/core/src/main/java/org/apache/ignite/internal/util/IgniteUtils.java
@@ -7704,6 +7704,17 @@ public abstract class IgniteUtils {
return ret;
}
+/**
+ * Concats two integers to long.
+ *
+ * @param high Highest bits.
+ * @param low Lowest bits.
+ * @return Long.
+ */
+public static long toLong(int high, int low) {
+return (((long)high) << Integer.SIZE) | (low & 0xL);
+}
+
/**
* Copies all elements from collection to array and asserts that
* array is big enough to hold the collection. This method should
diff --git
a/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/persistence/snapshot/dump/AbstractCacheDumpTest.java
b/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/persistence/snapshot/dump/AbstractCacheDumpTest.java
index 75d9bdda06b..bdcfab73f25 100644
---
a/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/persistence/snapshot/dump/AbstractCacheDumpTest.java
+++
b/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/persistence/snapshot/dump/AbstractCacheDumpTest.java
@@ -79,7 +79,7 @@ import static
org.apache.ignite.internal.encryption.AbstractEncryptionTest.KEYST
import static
org.apache.ignite.internal.encryption.AbstractEncryptionTest.KEYSTORE_PATH;
import static
org.apache.ignite.internal.processors.cache.GridCacheUtils.UTILITY_CACHE_NAME;
import static
org.apache.ignite.internal.processors.cache.persistence.snapshot.IgniteSnapshotManager.SNP_RUNNING_DIR_KEY;
-import static
org.apache.ignite.internal.processors.cache.persistence.snapshot.dump.CreateDumpFutureTask.toLong;
+import static org.apache.ignite.internal.util.IgniteUtils.toLong;
import static org.apache.ignite.platform.model.AccessLevel.SUPER;
import static org.apache.ignite.testframework.GridTestUtils.runAsync;
import static org.apache.ignite.testframework.GridTestUtils.waitForCondition;
(ignite) branch master updated: IGNITE-21046 Move binaryMetadataUpdateListener to basic interface (#11049)
This is an automated email from the ASF dual-hosted git repository.
timoninmaxim pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/ignite.git
The following commit(s) were added to refs/heads/master by this push:
new e298242654e IGNITE-21046 Move binaryMetadataUpdateListener to basic
interface (#11049)
e298242654e is described below
commit e298242654e2fa56ea43d2e188f08a2b5634f2f7
Author: Maksim Timonin
AuthorDate: Fri Dec 8 15:29:19 2023 +0300
IGNITE-21046 Move binaryMetadataUpdateListener to basic interface (#11049)
---
.../processors/cache/binary/CacheObjectBinaryProcessorImpl.java | 6 ++
.../processors/cacheobject/IgniteCacheObjectProcessor.java| 8
2 files changed, 10 insertions(+), 4 deletions(-)
diff --git
a/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/binary/CacheObjectBinaryProcessorImpl.java
b/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/binary/CacheObjectBinaryProcessorImpl.java
index 7ca52e66449..4e1bc0b59c8 100644
---
a/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/binary/CacheObjectBinaryProcessorImpl.java
+++
b/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/binary/CacheObjectBinaryProcessorImpl.java
@@ -372,10 +372,8 @@ public class CacheObjectBinaryProcessorImpl extends
GridProcessorAdapter impleme
}
}
-/**
- * @param lsnr Listener.
- */
-public void addBinaryMetadataUpdateListener(BinaryMetadataUpdatedListener
lsnr) {
+/** {@inheritDoc} */
+@Override public void
addBinaryMetadataUpdateListener(BinaryMetadataUpdatedListener lsnr) {
if (transport != null)
transport.addBinaryMetadataUpdateListener(lsnr);
}
diff --git
a/modules/core/src/main/java/org/apache/ignite/internal/processors/cacheobject/IgniteCacheObjectProcessor.java
b/modules/core/src/main/java/org/apache/ignite/internal/processors/cacheobject/IgniteCacheObjectProcessor.java
index 68953161430..1b3cd9bcc88 100644
---
a/modules/core/src/main/java/org/apache/ignite/internal/processors/cacheobject/IgniteCacheObjectProcessor.java
+++
b/modules/core/src/main/java/org/apache/ignite/internal/processors/cacheobject/IgniteCacheObjectProcessor.java
@@ -39,6 +39,7 @@ import
org.apache.ignite.internal.processors.cache.CacheObjectValueContext;
import org.apache.ignite.internal.processors.cache.GridCacheContext;
import org.apache.ignite.internal.processors.cache.IncompleteCacheObject;
import org.apache.ignite.internal.processors.cache.KeyCacheObject;
+import
org.apache.ignite.internal.processors.cache.binary.BinaryMetadataUpdatedListener;
import org.jetbrains.annotations.Nullable;
/**
@@ -377,4 +378,11 @@ public interface IgniteCacheObjectProcessor extends
GridProcessor {
* @throws org.apache.ignite.binary.BinaryObjectException In case of error.
*/
public BinaryType registerClass(Class cls) throws BinaryObjectException;
+
+/**
+ * Add binary metadata update listener.
+ *
+ * @param lsnr Listener.
+ */
+public void addBinaryMetadataUpdateListener(BinaryMetadataUpdatedListener
lsnr);
}
(ignite) branch master updated: IGNITE-21041 Make FilteredRecord local for RecordSerializer (#11082)
This is an automated email from the ASF dual-hosted git repository.
timoninmaxim pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/ignite.git
The following commit(s) were added to refs/heads/master by this push:
new dbe7e2513d3 IGNITE-21041 Make FilteredRecord local for
RecordSerializer (#11082)
dbe7e2513d3 is described below
commit dbe7e2513d30aff3f42df618d88dba70a721e229
Author: Maksim Timonin
AuthorDate: Fri Dec 8 15:23:42 2023 +0300
IGNITE-21041 Make FilteredRecord local for RecordSerializer (#11082)
---
.../apache/ignite/internal/pagemem/wal/record/FilteredRecord.java| 3 ---
.../cache/persistence/wal/reader/StandaloneWalRecordsIterator.java | 5 -
.../cache/persistence/wal/serializer/RecordV1Serializer.java | 5 -
.../cache/persistence/wal/serializer/RecordV2Serializer.java | 5 -
4 files changed, 12 insertions(+), 6 deletions(-)
diff --git
a/modules/core/src/main/java/org/apache/ignite/internal/pagemem/wal/record/FilteredRecord.java
b/modules/core/src/main/java/org/apache/ignite/internal/pagemem/wal/record/FilteredRecord.java
index e077e5accc2..519e825953d 100644
---
a/modules/core/src/main/java/org/apache/ignite/internal/pagemem/wal/record/FilteredRecord.java
+++
b/modules/core/src/main/java/org/apache/ignite/internal/pagemem/wal/record/FilteredRecord.java
@@ -24,9 +24,6 @@ import
org.apache.ignite.internal.processors.cache.persistence.wal.AbstractWalRe
* {@link AbstractWalRecordsIterator}.
*/
public class FilteredRecord extends WALRecord {
-/** Instance. */
-public static final FilteredRecord INSTANCE = new FilteredRecord();
-
/** {@inheritDoc} */
@Override public RecordType type() {
return null;
diff --git
a/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/wal/reader/StandaloneWalRecordsIterator.java
b/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/wal/reader/StandaloneWalRecordsIterator.java
index c7582b7cf14..26dd7c3e647 100644
---
a/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/wal/reader/StandaloneWalRecordsIterator.java
+++
b/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/wal/reader/StandaloneWalRecordsIterator.java
@@ -96,6 +96,9 @@ class StandaloneWalRecordsIterator extends
AbstractWalRecordsIterator {
/** Replay from bound include. */
private final WALPointer lowBound;
+/** Singleton instance of {@link FilteredRecord} */
+private final WALRecord filteredRecord = new FilteredRecord();
+
/**
* Creates iterator in file-by-file iteration mode. Directory
*
@@ -286,7 +289,7 @@ class StandaloneWalRecordsIterator extends
AbstractWalRecordsIterator {
return tup;
if (!checkBounds(tup.get1()))
-return new T2<>(tup.get1(), FilteredRecord.INSTANCE); //
FilteredRecord for mark as filtered.
+return new T2<>(tup.get1(), filteredRecord); // FilteredRecord for
mark as filtered.
return tup;
}
diff --git
a/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/wal/serializer/RecordV1Serializer.java
b/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/wal/serializer/RecordV1Serializer.java
index 295e174abf8..52c360d47eb 100644
---
a/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/wal/serializer/RecordV1Serializer.java
+++
b/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/wal/serializer/RecordV1Serializer.java
@@ -98,6 +98,9 @@ public class RecordV1Serializer implements RecordSerializer {
*/
private final boolean marshalledMode;
+/** Singleton instance of {@link FilteredRecord} */
+private final FilteredRecord filteredRecord = new FilteredRecord();
+
/** Thread-local heap byte buffer. */
private final ThreadLocal heapTlb = new
ThreadLocal() {
@Override protected ByteBuffer initialValue() {
@@ -148,7 +151,7 @@ public class RecordV1Serializer implements RecordSerializer
{
if (recType.purpose() != WALRecord.RecordPurpose.INTERNAL
&& recordFilter != null && !recordFilter.apply(rec.type(),
ptr))
-return FilteredRecord.INSTANCE;
+return filteredRecord;
else if (marshalledMode) {
ByteBuffer buf = heapTlb.get();
diff --git
a/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/wal/serializer/RecordV2Serializer.java
b/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/wal/serializer/RecordV2Serializer.java
index 60c68c13f7c..21aa52a91ba 100644
---
a/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/wal/serializer/RecordV2Serializer.java
+++
b/modules/c
(ignite) branch master updated (dbe7e2513d3 -> ace734e6327)
This is an automated email from the ASF dual-hosted git repository. timoninmaxim pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/ignite.git from dbe7e2513d3 IGNITE-21041 Make FilteredRecord local for RecordSerializer (#11082) add ace734e6327 IGNITE-17700 Refactoring WalRecordsConsumer (#11050) No new revisions were added by this update. Summary of changes: .../java/org/apache/ignite/internal/cdc/CdcMain.java | 2 +- .../ignite/internal/cdc/WalRecordsConsumer.java | 20 2 files changed, 13 insertions(+), 9 deletions(-)
(ignite) branch master updated: IGNITE-21042 Move CreateDumpFutureTask#toLong to IgniteUtils (#11048)
This is an automated email from the ASF dual-hosted git repository.
timoninmaxim pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/ignite.git
The following commit(s) were added to refs/heads/master by this push:
new 27d9298de8f IGNITE-21042 Move CreateDumpFutureTask#toLong to
IgniteUtils (#11048)
27d9298de8f is described below
commit 27d9298de8f2e76a6fdeb7cf5b6d065b1a6c652e
Author: Maksim Timonin
AuthorDate: Fri Dec 8 15:16:57 2023 +0300
IGNITE-21042 Move CreateDumpFutureTask#toLong to IgniteUtils (#11048)
---
.../cache/persistence/snapshot/dump/CreateDumpFutureTask.java | 8 ++--
.../java/org/apache/ignite/internal/util/IgniteUtils.java | 11 +++
.../persistence/snapshot/dump/AbstractCacheDumpTest.java | 2 +-
3 files changed, 14 insertions(+), 7 deletions(-)
diff --git
a/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/snapshot/dump/CreateDumpFutureTask.java
b/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/snapshot/dump/CreateDumpFutureTask.java
index d0654e719b0..3f181b71a97 100644
---
a/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/snapshot/dump/CreateDumpFutureTask.java
+++
b/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/snapshot/dump/CreateDumpFutureTask.java
@@ -75,6 +75,7 @@ import static
org.apache.ignite.internal.processors.cache.persistence.file.FileP
import static
org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.CACHE_GRP_DIR_PREFIX;
import static
org.apache.ignite.internal.processors.cache.persistence.snapshot.IgniteSnapshotManager.DUMP_LOCK;
import static
org.apache.ignite.internal.processors.cache.persistence.snapshot.dump.Dump.dumpPartFileName;
+import static org.apache.ignite.internal.util.IgniteUtils.toLong;
/**
* Task creates cache group dump.
@@ -111,7 +112,7 @@ public class CreateDumpFutureTask extends
AbstractCreateSnapshotFutureTask imple
* Dump contextes.
* Key is [group_id, partition_id] combined in single long value.
*
- * @see #toLong(int, int)
+ * @see IgniteUtils#toLong(int, int)
*/
private final Map dumpCtxs = new
ConcurrentHashMap<>();
@@ -631,11 +632,6 @@ public class CreateDumpFutureTask extends
AbstractCreateSnapshotFutureTask imple
}
}
-/** */
-public static long toLong(int high, int low) {
-return (((long)high) << Integer.SIZE) | (low & 0xL);
-}
-
/** */
private File groupDirectory(CacheGroupContext grpCtx) throws
IgniteCheckedException {
return new File(
diff --git
a/modules/core/src/main/java/org/apache/ignite/internal/util/IgniteUtils.java
b/modules/core/src/main/java/org/apache/ignite/internal/util/IgniteUtils.java
index 2f736091ed1..de17a4eea7f 100755
---
a/modules/core/src/main/java/org/apache/ignite/internal/util/IgniteUtils.java
+++
b/modules/core/src/main/java/org/apache/ignite/internal/util/IgniteUtils.java
@@ -7704,6 +7704,17 @@ public abstract class IgniteUtils {
return ret;
}
+/**
+ * Concats two integers to long.
+ *
+ * @param high Highest bits.
+ * @param low Lowest bits.
+ * @return Long.
+ */
+public static long toLong(int high, int low) {
+return (((long)high) << Integer.SIZE) | (low & 0xL);
+}
+
/**
* Copies all elements from collection to array and asserts that
* array is big enough to hold the collection. This method should
diff --git
a/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/persistence/snapshot/dump/AbstractCacheDumpTest.java
b/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/persistence/snapshot/dump/AbstractCacheDumpTest.java
index 75d9bdda06b..bdcfab73f25 100644
---
a/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/persistence/snapshot/dump/AbstractCacheDumpTest.java
+++
b/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/persistence/snapshot/dump/AbstractCacheDumpTest.java
@@ -79,7 +79,7 @@ import static
org.apache.ignite.internal.encryption.AbstractEncryptionTest.KEYST
import static
org.apache.ignite.internal.encryption.AbstractEncryptionTest.KEYSTORE_PATH;
import static
org.apache.ignite.internal.processors.cache.GridCacheUtils.UTILITY_CACHE_NAME;
import static
org.apache.ignite.internal.processors.cache.persistence.snapshot.IgniteSnapshotManager.SNP_RUNNING_DIR_KEY;
-import static
org.apache.ignite.internal.processors.cache.persistence.snapshot.dump.CreateDumpFutureTask.toLong;
+import static org.apache.ignite.internal.util.IgniteUtils.toLong;
import static org.apache.ignite.platform.model.AccessLevel.SUPER;
import static org.apache.ignite.testframework.GridTestUtils.runAsync;
import static org.apache.ignite.testframework.GridTestUtils.waitForCondition;
(ignite) branch master updated (e70b66c5deb -> d053cb88048)
This is an automated email from the ASF dual-hosted git repository.
timoninmaxim pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/ignite.git
from e70b66c5deb IGNITE-21029 Updated log4j2 dependency to 2.22.0 version
(#11080)
add d053cb88048 IGNITE-17700 Introduce CdcManager (#11044)
No new revisions were added by this update.
Summary of changes:
.../ignite/internal/cdc/CdcConsumerState.java | 39 ++
.../org/apache/ignite/internal/cdc/CdcMain.java| 240 +---
.../org/apache/ignite/internal/cdc/CdcManager.java | 123 ++
.../org/apache/ignite/internal/cdc/CdcMode.java} | 19 +-
.../internal/cdc/CdcUtilityActiveCdcManager.java | 73
.../{CdcDataRecord.java => CdcManagerRecord.java} | 30 +-
...dcDataRecord.java => CdcManagerStopRecord.java} | 21 +-
.../internal/pagemem/wal/record/WALRecord.java | 8 +-
.../processors/cache/GridCacheProcessor.java | 27 ++
.../processors/cache/GridCacheSharedContext.java | 37 +-
.../IgniteCacheDatabaseSharedManager.java | 4 +
.../persistence/wal/FileWriteAheadLogManager.java | 6 +-
.../wal/filehandle/FileWriteHandleImpl.java| 42 ++-
.../wal/serializer/RecordDataV1Serializer.java | 34 ++
.../ignite/cdc/CdcIgniteNodeActiveModeTest.java| 394 +++
.../java/org/apache/ignite/cdc/CdcManagerTest.java | 415 +
.../java/org/apache/ignite/cdc/WalForCdcTest.java | 4 +
.../testframework/wal/record/RecordUtils.java | 18 +
.../ignite/testsuites/IgnitePdsTestSuite2.java | 4 +
19 files changed, 1425 insertions(+), 113 deletions(-)
create mode 100644
modules/core/src/main/java/org/apache/ignite/internal/cdc/CdcManager.java
copy
modules/{calcite/src/main/java/org/apache/ignite/internal/processors/query/calcite/exec/exp/agg/AggregateType.java
=> core/src/main/java/org/apache/ignite/internal/cdc/CdcMode.java} (75%)
create mode 100644
modules/core/src/main/java/org/apache/ignite/internal/cdc/CdcUtilityActiveCdcManager.java
copy
modules/core/src/main/java/org/apache/ignite/internal/pagemem/wal/record/{CdcDataRecord.java
=> CdcManagerRecord.java} (53%)
copy
modules/core/src/main/java/org/apache/ignite/internal/pagemem/wal/record/{CdcDataRecord.java
=> CdcManagerStopRecord.java} (66%)
create mode 100644
modules/core/src/test/java/org/apache/ignite/cdc/CdcIgniteNodeActiveModeTest.java
create mode 100644
modules/core/src/test/java/org/apache/ignite/cdc/CdcManagerTest.java
(ignite) branch master updated (56a33decd41 -> 65aa55af170)
This is an automated email from the ASF dual-hosted git repository. timoninmaxim pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/ignite.git from 56a33decd41 IGNITE-21015 Added the security subject ID to the cluster state change event (#11075) add 65aa55af170 IGNITE-20614 Add rate limiter for dumps (#11076) No new revisions were added by this update. Summary of changes: .../snapshot/IgniteSnapshotManager.java| 3 +- .../snapshot/dump/CreateDumpFutureTask.java| 19 .../snapshot/dump/IgniteCacheDumpSelf2Test.java| 50 ++ 3 files changed, 71 insertions(+), 1 deletion(-)
(ignite) branch master updated (efccc7ca54c -> 36bd6ff35e8)
This is an automated email from the ASF dual-hosted git repository.
timoninmaxim pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/ignite.git
from efccc7ca54c IGNITE-20947 Fixed misprint in add-opens of
sun.util.calendar for java 15+ (#11066)
add 36bd6ff35e8 IGNITE-20836 Support zipping of dump files (#11040)
No new revisions were added by this update.
Summary of changes:
.../ignite/dump/DumpReaderConfiguration.java | 20 ---
...sFileIOFactory.java => UnzipFileIOFactory.java} | 8 +-
.../snapshot/IgniteSnapshotManager.java| 43 +--
.../persistence/snapshot/SnapshotMetadata.java | 17 +++
.../snapshot/SnapshotOperationRequest.java | 13 +-
.../snapshot/SnapshotPartitionsVerifyHandler.java | 5 +-
.../snapshot/SnapshotRestoreProcess.java | 1 +
.../snapshot/dump/CreateDumpFutureTask.java| 14 ++-
.../cache/persistence/snapshot/dump/Dump.java | 25 +++-
.../dump/WriteOnlyZipFileIO.java} | 137 ++---
.../dump/WriteOnlyZipFileIOFactory.java} | 11 +-
.../snapshot/AbstractSnapshotSelfTest.java | 12 +-
.../snapshot/EncryptedSnapshotTest.java| 13 +-
.../snapshot/IgniteSnapshotManagerSelfTest.java| 1 +
.../snapshot/dump/AbstractCacheDumpTest.java | 22 +++-
.../snapshot/dump/IgniteCacheDumpSelf2Test.java| 66 +-
.../snapshot/dump/IgniteCacheDumpSelfTest.java | 30 -
17 files changed, 303 insertions(+), 135 deletions(-)
copy
modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/file/{RandomAccessFileIOFactory.java
=> UnzipFileIOFactory.java} (78%)
copy
modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/{file/RandomAccessFileIO.java
=> snapshot/dump/WriteOnlyZipFileIO.java} (55%)
copy
modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/{file/RandomAccessFileIOFactory.java
=> snapshot/dump/WriteOnlyZipFileIOFactory.java} (72%)
(ignite) branch master updated: IGNITE-20572 BinaryMeta file atomic creation (#11055)
This is an automated email from the ASF dual-hosted git repository.
timoninmaxim pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/ignite.git
The following commit(s) were added to refs/heads/master by this push:
new 78e161ad53a IGNITE-20572 BinaryMeta file atomic creation (#11055)
78e161ad53a is described below
commit 78e161ad53a7e3482ebfbb56571ec1d9a219a022
Author: Maksim Timonin
AuthorDate: Tue Nov 21 13:47:28 2023 +0300
IGNITE-20572 BinaryMeta file atomic creation (#11055)
---
.../processors/cache/binary/BinaryMetadataFileStore.java| 13 +++--
1 file changed, 11 insertions(+), 2 deletions(-)
diff --git
a/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/binary/BinaryMetadataFileStore.java
b/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/binary/BinaryMetadataFileStore.java
index db43a7a0c11..408e6f1ffc8 100644
---
a/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/binary/BinaryMetadataFileStore.java
+++
b/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/binary/BinaryMetadataFileStore.java
@@ -46,6 +46,8 @@ import org.apache.ignite.internal.util.typedef.internal.U;
import org.apache.ignite.internal.util.worker.GridWorker;
import org.apache.ignite.thread.IgniteThread;
+import static java.nio.file.StandardCopyOption.ATOMIC_MOVE;
+import static java.nio.file.StandardCopyOption.REPLACE_EXISTING;
import static
org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.TMP_SUFFIX;
/**
@@ -150,16 +152,23 @@ class BinaryMetadataFileStore {
try {
File file = new File(metadataDir,
BinaryUtils.binaryMetaFileName(binMeta.typeId()));
+File tmpFile = new File(file.getAbsolutePath() + TMP_SUFFIX);
+
+// TODO: delete it on Ignite start.
https://issues.apache.org/jira/browse/IGNITE-20897
+if (tmpFile.exists())
+U.delete(tmpFile);
byte[] marshalled = U.marshal(ctx, binMeta);
-try (final FileIO out = fileIOFactory.create(file)) {
+try (final FileIO out = fileIOFactory.create(tmpFile)) {
int left = marshalled.length;
while ((left -= out.writeFully(marshalled, 0,
Math.min(marshalled.length, left))) > 0)
;
out.force();
}
+
+Files.move(tmpFile.toPath(), file.toPath(), ATOMIC_MOVE,
REPLACE_EXISTING);
}
catch (Exception e) {
final String msg = "Failed to save metadata for typeId: " +
binMeta.typeId() +
@@ -208,7 +217,7 @@ class BinaryMetadataFileStore {
if (!enabled)
return;
-for (File file : metadataDir.listFiles())
+for (File file : metadataDir.listFiles(BinaryUtils::notTmpFile))
restoreMetadata(file);
}
(ignite) branch master updated (aba0aaa5b8d -> fd8757e4950)
This is an automated email from the ASF dual-hosted git repository. timoninmaxim pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/ignite.git from aba0aaa5b8d IGNITE-19807 Deprecated legacy authorization approach via Security Context. (#10800) add fd8757e4950 IGNITE-20822 Filter by higherBound in AbstractWalRecordsIterator (#11034) No new revisions were added by this update. Summary of changes: .../persistence/wal/AbstractWalRecordsIterator.java | 10 ++ .../cache/persistence/wal/FileWriteAheadLogManager.java | 13 + .../wal/SingleSegmentLogicalRecordsIterator.java | 9 - .../wal/reader/StandaloneWalRecordsIterator.java | 16 ++-- 4 files changed, 25 insertions(+), 23 deletions(-)
[ignite] branch master updated (2c930ab4eb8 -> c861ac5ba42)
This is an automated email from the ASF dual-hosted git repository. timoninmaxim pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/ignite.git from 2c930ab4eb8 IGNITE-20730 Fix IgniteLock cannot be acquired after release on JDK 17 (#11012) add c861ac5ba42 IGNITE-20733 Add GridCacheSharedContext builder (#11011) No new revisions were added by this update. Summary of changes: .../processors/cache/GridCacheProcessor.java | 60 ++ .../processors/cache/GridCacheSharedContext.java | 213 - .../snapshot/IgniteSnapshotManager.java| 5 +- .../wal/reader/IgniteWalIteratorFactory.java | 9 +- .../db/wal/IgniteWalIteratorSwitchSegmentTest.java | 52 + .../pagemem/BPlusTreePageMemoryImplTest.java | 27 +-- .../BPlusTreeReuseListPageMemoryImplTest.java | 26 +-- .../pagemem/IndexStoragePageMemoryImplTest.java| 27 +-- .../pagemem/PageMemoryImplNoLoadTest.java | 26 +-- .../persistence/pagemem/PageMemoryImplTest.java| 26 +-- .../loadtests/hashmap/GridCacheTestContext.java| 36 ++-- 11 files changed, 287 insertions(+), 220 deletions(-)
[ignite] branch master updated: IGNITE-20654 Forbid H2 connection "init" setting (#10997)
This is an automated email from the ASF dual-hosted git repository.
timoninmaxim pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/ignite.git
The following commit(s) were added to refs/heads/master by this push:
new ccab242a588 IGNITE-20654 Forbid H2 connection "init" setting (#10997)
ccab242a588 is described below
commit ccab242a58858d433aa8ca70f6866a2ed6af
Author: Maksim Timonin
AuthorDate: Tue Oct 17 10:17:31 2023 +0300
IGNITE-20654 Forbid H2 connection "init" setting (#10997)
---
.../processors/query/h2/ConnectionManager.java | 37 +
.../processors/query/h2/IgniteH2Indexing.java | 10 +++
.../cache/index/H2ConnectionSettingsTest.java | 87 ++
.../IgniteBinaryCacheQueryTestSuite3.java | 2 +
4 files changed, 136 insertions(+)
diff --git
a/modules/indexing/src/main/java/org/apache/ignite/internal/processors/query/h2/ConnectionManager.java
b/modules/indexing/src/main/java/org/apache/ignite/internal/processors/query/h2/ConnectionManager.java
index 098e552d21c..d68fbc26f05 100644
---
a/modules/indexing/src/main/java/org/apache/ignite/internal/processors/query/h2/ConnectionManager.java
+++
b/modules/indexing/src/main/java/org/apache/ignite/internal/processors/query/h2/ConnectionManager.java
@@ -17,11 +17,14 @@
package org.apache.ignite.internal.processors.query.h2;
+import java.lang.reflect.Field;
+import java.lang.reflect.Modifier;
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.SQLException;
import java.sql.Statement;
import java.util.Collections;
+import java.util.HashSet;
import java.util.Set;
import java.util.concurrent.ConcurrentHashMap;
import org.apache.ignite.IgniteCheckedException;
@@ -35,6 +38,7 @@ import
org.apache.ignite.internal.processors.timeout.GridTimeoutProcessor;
import org.apache.ignite.internal.util.GridBusyLock;
import org.apache.ignite.internal.util.typedef.internal.U;
import org.h2.api.JavaObjectSerializer;
+import org.h2.engine.ConnectionInfo;
import org.h2.engine.Database;
import org.h2.jdbc.JdbcConnection;
import org.h2.store.DataHandler;
@@ -76,6 +80,9 @@ public class ConnectionManager {
// H2 DbSettings.
System.setProperty("h2.optimizeTwoEquals", "false"); // Makes splitter
fail on subqueries in WHERE.
System.setProperty("h2.dropRestrict", "false"); // Drop schema with
cascade semantics.
+
+// Forbid vulnerable settings.
+forbidH2DbSettings("INIT");
}
/** The period of clean up the statement cache. */
@@ -351,4 +358,34 @@ public class ConnectionManager {
if (dataNhd != null && dataNhd instanceof Database)
DB_JOBJ_SERIALIZER.set((Database)dataNhd, serializer);
}
+
+/**
+ * Forbids specified settings from KNOWN_SETTINGS for security reasons.
+ *
+ * @param settings Settings to forbid.
+ */
+private static void forbidH2DbSettings(String... settings) {
+try {
+Field knownSettingsField =
ConnectionInfo.class.getDeclaredField("KNOWN_SETTINGS");
+Field modifiers = Field.class.getDeclaredField("modifiers");
+
+modifiers.setAccessible(true);
+modifiers.setInt(knownSettingsField,
knownSettingsField.getModifiers() & ~Modifier.FINAL);
+
+knownSettingsField.setAccessible(true);
+
+HashSet knownSettings =
(HashSet)knownSettingsField.get(null);
+
+for (String s: settings)
+knownSettings.remove(s);
+
+modifiers.setInt(knownSettingsField,
knownSettingsField.getModifiers() & Modifier.FINAL);
+modifiers.setAccessible(false);
+
+knownSettingsField.setAccessible(false);
+}
+catch (Exception e) {
+// No-op.
+}
+}
}
diff --git
a/modules/indexing/src/main/java/org/apache/ignite/internal/processors/query/h2/IgniteH2Indexing.java
b/modules/indexing/src/main/java/org/apache/ignite/internal/processors/query/h2/IgniteH2Indexing.java
index e9ddd2e0434..0ca832360e9 100644
---
a/modules/indexing/src/main/java/org/apache/ignite/internal/processors/query/h2/IgniteH2Indexing.java
+++
b/modules/indexing/src/main/java/org/apache/ignite/internal/processors/query/h2/IgniteH2Indexing.java
@@ -194,6 +194,16 @@ public class IgniteH2Indexing implements GridQueryIndexing
{
private final boolean updateInTxAllowed =
Boolean.getBoolean(IgniteSystemProperties.IGNITE_ALLOW_DML_INSIDE_TRANSACTION);
+static {
+// Required to skip checks of forbidden H2 settings, otherwise Ignite
fails to start.
+//
+// Note, H2 system properties must be overriden here, because the
properties are finalized while the class
+// org.h2.engine.SysProperties is loaded in the
IgniteH2Indexing.start(...) method.
+//
+
[ignite] branch master updated (1c2020212fb -> a6751098758)
This is an automated email from the ASF dual-hosted git repository. timoninmaxim pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/ignite.git from 1c2020212fb IGNITE-20546 Fix transactions expiration set for reinsert (#10972) add a6751098758 IGNITE-20543 Wrap CacheStore creation to sandbox (#10971) No new revisions were added by this update. Summary of changes: .../processors/cache/GridCacheProcessor.java | 17 +++- .../security/sandbox/AbstractSandboxTest.java | 2 +- .../sandbox/CacheStoreFactorySandboxTest.java | 113 + .../ignite/testsuites/SecurityTestSuite.java | 2 + 4 files changed, 130 insertions(+), 4 deletions(-) create mode 100644 modules/core/src/test/java/org/apache/ignite/internal/processors/security/sandbox/CacheStoreFactorySandboxTest.java
[ignite] branch master updated (d2af35b9320 -> 1c2020212fb)
This is an automated email from the ASF dual-hosted git repository. timoninmaxim pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/ignite.git from d2af35b9320 IGNITE-20532 Fix JavaNioAccess handling in JDK 14 and later. (#10969) add 1c2020212fb IGNITE-20546 Fix transactions expiration set for reinsert (#10972) No new revisions were added by this update. Summary of changes: .../distributed/dht/GridDhtTxPrepareFuture.java| 2 +- .../cache/CacheQueryFilterExpiredTest.java | 106 ++--- 2 files changed, 93 insertions(+), 15 deletions(-)
[ignite] branch master updated: IGNITE-20228 Fix connectable nodes list for control.sh (#10898)
This is an automated email from the ASF dual-hosted git repository.
timoninmaxim pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/ignite.git
The following commit(s) were added to refs/heads/master by this push:
new 2ba290e701a IGNITE-20228 Fix connectable nodes list for control.sh
(#10898)
2ba290e701a is described below
commit 2ba290e701a3c12d7979a38bca8294f9aefdbe80
Author: Maksim Timonin
AuthorDate: Thu Aug 17 11:17:41 2023 +0300
IGNITE-20228 Fix connectable nodes list for control.sh (#10898)
---
.../java/org/apache/ignite/internal/management/api/CommandUtils.java| 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git
a/modules/core/src/main/java/org/apache/ignite/internal/management/api/CommandUtils.java
b/modules/core/src/main/java/org/apache/ignite/internal/management/api/CommandUtils.java
index 8611d1bb8fc..f3a8a7e57ca 100644
---
a/modules/core/src/main/java/org/apache/ignite/internal/management/api/CommandUtils.java
+++
b/modules/core/src/main/java/org/apache/ignite/internal/management/api/CommandUtils.java
@@ -707,7 +707,7 @@ public class CommandUtils {
.collect(Collectors.toList());
if (!connectable.isEmpty())
-compute = compute.projection(nodes);
+compute = compute.projection(connectable);
return compute.execute(
taskCls.getName(),
[ignite] branch IGNITE-17700__realtime_cdc updated: IGNITE-19622 Collect to buffer only data records (#10845)
This is an automated email from the ASF dual-hosted git repository.
timoninmaxim pushed a commit to branch IGNITE-17700__realtime_cdc
in repository https://gitbox.apache.org/repos/asf/ignite.git
The following commit(s) were added to refs/heads/IGNITE-17700__realtime_cdc by
this push:
new b609117e168 IGNITE-19622 Collect to buffer only data records (#10845)
b609117e168 is described below
commit b609117e168bb377859982428a1343db36286063
Author: Maksim Timonin
AuthorDate: Wed Jul 19 12:36:53 2023 +0300
IGNITE-19622 Collect to buffer only data records (#10845)
---
.../cache/persistence/cdc/CdcBuffer.java | 9 +-
.../cache/persistence/cdc/CdcBufferConsumer.java | 7 +-
.../cache/persistence/cdc/CdcProcessor.java| 72 +--
.../cache/persistence/cdc/CdcWorker.java | 110 +
.../persistence/wal/ByteBufferWalIterator.java | 55 -
.../persistence/wal/FileWriteAheadLogManager.java | 3 +
.../wal/filehandle/FileHandleManagerImpl.java | 2 +
.../wal/filehandle/FileWriteHandleImpl.java| 1 +
.../wal/filehandle/FsyncFileWriteHandle.java | 1 +
.../persistence/cdc/RealtimeCdcBufferTest.java | 131 ++---
.../persistence/wal/ByteBufferWalIteratorTest.java | 30 ++---
11 files changed, 274 insertions(+), 147 deletions(-)
diff --git
a/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/cdc/CdcBuffer.java
b/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/cdc/CdcBuffer.java
index 3f7b3ef8722..aa7785cf7e0 100644
---
a/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/cdc/CdcBuffer.java
+++
b/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/cdc/CdcBuffer.java
@@ -62,6 +62,9 @@ public class CdcBuffer {
public boolean offer(ByteBuffer data) {
int bufSize = data.limit() - data.position();
+if (bufSize == 0)
+return true;
+
if (size.addAndGet(bufSize) > maxSize) {
overflowed = true;
@@ -72,7 +75,11 @@ public class CdcBuffer {
data.get(cp, 0, bufSize);
-LinkedNode newNode = new LinkedNode(ByteBuffer.wrap(cp));
+ByteBuffer dataBuf = ByteBuffer
+.wrap(cp)
+.order(data.order());
+
+LinkedNode newNode = new LinkedNode(dataBuf);
LinkedNode oldNode = producerNode;
producerNode = newNode;
diff --git
a/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/cdc/CdcBufferConsumer.java
b/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/cdc/CdcBufferConsumer.java
index 5e96fa2c98f..c75ee9b57ae 100644
---
a/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/cdc/CdcBufferConsumer.java
+++
b/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/cdc/CdcBufferConsumer.java
@@ -17,17 +17,18 @@
package org.apache.ignite.internal.processors.cache.persistence.cdc;
-import java.nio.ByteBuffer;
+import java.util.Collection;
+import org.apache.ignite.cdc.CdcEvent;
/** Mock for Realtime CDC buffer consumer. */
public interface CdcBufferConsumer {
/**
* Consumes raw WAL data.
*
- * @param data Raw data to consume.
+ * @param cdcEvents CDC events to consume.
* @return {@code True} if current offset in WAL should be commited.
*/
-public boolean consume(ByteBuffer data);
+public boolean consume(Collection cdcEvents);
/** */
public void close();
diff --git
a/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/cdc/CdcProcessor.java
b/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/cdc/CdcProcessor.java
index 9e5cb0dc4d8..2e8975fb1ba 100644
---
a/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/cdc/CdcProcessor.java
+++
b/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/cdc/CdcProcessor.java
@@ -18,13 +18,31 @@
package org.apache.ignite.internal.processors.cache.persistence.cdc;
import java.nio.ByteBuffer;
+import org.apache.ignite.IgniteCheckedException;
+import org.apache.ignite.IgniteException;
import org.apache.ignite.IgniteLogger;
+import org.apache.ignite.IgniteSystemProperties;
import org.apache.ignite.internal.IgniteInterruptedCheckedException;
+import org.apache.ignite.internal.pagemem.wal.WALIterator;
+import org.apache.ignite.internal.pagemem.wal.record.MarshalledRecord;
+import org.apache.ignite.internal.pagemem.wal.record.WALRecord;
import org.apache.ignite.internal.processors.cache.GridCacheSharedContext;
+import
org.apache.ignite.internal.processors.cache.persistence.wal.ByteBufferWalIterator;
+import org.apache.ignite.internal.processors.cache.persistence.wal.WALPointer;
+imp
[ignite] branch IGNITE-17700__realtime_cdc updated: IGNITE-19675 [IEP-104] Implement WALIterator over ByteBuffer (#10798)
This is an automated email from the ASF dual-hosted git repository.
timoninmaxim pushed a commit to branch IGNITE-17700__realtime_cdc
in repository https://gitbox.apache.org/repos/asf/ignite.git
The following commit(s) were added to refs/heads/IGNITE-17700__realtime_cdc by
this push:
new 9b8cf4f1ec0 IGNITE-19675 [IEP-104] Implement WALIterator over
ByteBuffer (#10798)
9b8cf4f1ec0 is described below
commit 9b8cf4f1ec0fd2764a9e43a6aabe783782e54180
Author: yurinaryshkin <135707807+yurinarysh...@users.noreply.github.com>
AuthorDate: Thu Jul 13 15:19:18 2023 +0300
IGNITE-19675 [IEP-104] Implement WALIterator over ByteBuffer (#10798)
---
.../pagemem/wal/record/FilteredRecord.java | 5 +-
...or.java => AbstractFileWalRecordsIterator.java} | 47 +-
.../wal/AbstractWalRecordsIteratorAdapter.java | 78 +++
.../persistence/wal/ByteBufferBackedDataInput.java | 10 +
.../wal/ByteBufferBackedDataInputImpl.java | 14 +
.../persistence/wal/ByteBufferWalIterator.java | 132 +
.../persistence/wal/Crc32CheckingDataInput.java| 200 +++
.../cache/persistence/wal/FileDescriptor.java | 2 +-
.../persistence/wal/FileWriteAheadLogManager.java | 4 +-
.../wal/SingleSegmentLogicalRecordsIterator.java | 2 +-
.../cache/persistence/wal/io/FileInput.java| 221 +--
.../cache/persistence/wal/io/SimpleFileInput.java | 8 -
.../wal/reader/StandaloneWalRecordsIterator.java | 4 +-
.../wal/serializer/RecordSerializer.java | 4 +-
.../wal/serializer/RecordV1Serializer.java | 11 +-
.../wal/serializer/RecordV2Serializer.java | 3 +-
.../main/resources/META-INF/classnames.properties | 4 +-
.../db/wal/crc/IgniteDataIntegrityTests.java | 4 +-
.../persistence/wal/ByteBufferWalIteratorTest.java | 639 +
.../ignite/testsuites/IgnitePdsTestSuite5.java | 3 +
20 files changed, 1101 insertions(+), 294 deletions(-)
diff --git
a/modules/core/src/main/java/org/apache/ignite/internal/pagemem/wal/record/FilteredRecord.java
b/modules/core/src/main/java/org/apache/ignite/internal/pagemem/wal/record/FilteredRecord.java
index e077e5accc2..915353ebf2a 100644
---
a/modules/core/src/main/java/org/apache/ignite/internal/pagemem/wal/record/FilteredRecord.java
+++
b/modules/core/src/main/java/org/apache/ignite/internal/pagemem/wal/record/FilteredRecord.java
@@ -16,12 +16,9 @@
*/
package org.apache.ignite.internal.pagemem.wal.record;
-import
org.apache.ignite.internal.processors.cache.persistence.wal.AbstractWalRecordsIterator;
-
/**
* Special type of WAL record. Shouldn't be stored in file.
- * Returned by deserializer if next record is not matched by filter.
Automatically handled by
- * {@link AbstractWalRecordsIterator}.
+ * Returned by deserializer if next record is not matched by filter.
*/
public class FilteredRecord extends WALRecord {
/** Instance. */
diff --git
a/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/wal/AbstractWalRecordsIterator.java
b/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/wal/AbstractFileWalRecordsIterator.java
similarity index 91%
rename from
modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/wal/AbstractWalRecordsIterator.java
rename to
modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/wal/AbstractFileWalRecordsIterator.java
index 926bce8fe39..4dcb55ed368 100644
---
a/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/wal/AbstractWalRecordsIterator.java
+++
b/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/wal/AbstractFileWalRecordsIterator.java
@@ -25,7 +25,6 @@ import java.nio.ByteOrder;
import java.util.Optional;
import org.apache.ignite.IgniteCheckedException;
import org.apache.ignite.IgniteLogger;
-import org.apache.ignite.internal.pagemem.wal.WALIterator;
import org.apache.ignite.internal.pagemem.wal.record.WALRecord;
import org.apache.ignite.internal.processors.cache.GridCacheSharedContext;
import org.apache.ignite.internal.processors.cache.persistence.file.FileIO;
@@ -36,7 +35,6 @@ import
org.apache.ignite.internal.processors.cache.persistence.wal.io.SegmentIO;
import
org.apache.ignite.internal.processors.cache.persistence.wal.serializer.RecordSerializer;
import
org.apache.ignite.internal.processors.cache.persistence.wal.serializer.RecordSerializerFactory;
import
org.apache.ignite.internal.processors.cache.persistence.wal.serializer.SegmentHeader;
-import org.apache.ignite.internal.util.GridCloseableIteratorAdapter;
import org.apache.ignite.internal.util.typedef.P2;
import org.apache.ignite.internal.util.typedef.internal.U;
import org.apache.ignite.lang.IgniteBiTuple;
@@ -49,22 +47,10 @@ import static
org.apache.ignite.internal.processors.cache.persistence.wal.serial
* Iterator over WAL segments. This abstr
[ignite] branch IGNITE-17700__realtime_cdc updated: IGNITE-19639 Add realtime CDC records (#10840)
This is an automated email from the ASF dual-hosted git repository.
timoninmaxim pushed a commit to branch IGNITE-17700__realtime_cdc
in repository https://gitbox.apache.org/repos/asf/ignite.git
The following commit(s) were added to refs/heads/IGNITE-17700__realtime_cdc by
this push:
new 213d0951c14 IGNITE-19639 Add realtime CDC records (#10840)
213d0951c14 is described below
commit 213d0951c14344629223b16e6e114067fbfddf78
Author: Maksim Timonin
AuthorDate: Thu Jul 13 14:12:30 2023 +0300
IGNITE-19639 Add realtime CDC records (#10840)
---
.../wal/record/RealtimeCdcRecord.java} | 21 ++---
.../wal/record/RealtimeCdcStopRecord.java} | 21 ++---
.../internal/pagemem/wal/record/WALRecord.java | 8 +-
.../cache/persistence/cdc/CdcBuffer.java | 16 +++-
.../cache/persistence/cdc/CdcBufferConsumer.java | 3 +-
.../cache/persistence/cdc/CdcProcessor.java| 9 +--
.../cache/persistence/cdc/CdcWorker.java | 55 ++---
.../wal/serializer/RecordDataV1Serializer.java | 16
.../persistence/cdc/RealtimeCdcBufferTest.java | 90 +++---
9 files changed, 178 insertions(+), 61 deletions(-)
diff --git
a/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/cdc/CdcBufferConsumer.java
b/modules/core/src/main/java/org/apache/ignite/internal/pagemem/wal/record/RealtimeCdcRecord.java
similarity index 69%
copy from
modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/cdc/CdcBufferConsumer.java
copy to
modules/core/src/main/java/org/apache/ignite/internal/pagemem/wal/record/RealtimeCdcRecord.java
index 67cd422fb0f..5a9e68067ab 100644
---
a/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/cdc/CdcBufferConsumer.java
+++
b/modules/core/src/main/java/org/apache/ignite/internal/pagemem/wal/record/RealtimeCdcRecord.java
@@ -15,19 +15,12 @@
* limitations under the License.
*/
-package org.apache.ignite.internal.processors.cache.persistence.cdc;
+package org.apache.ignite.internal.pagemem.wal.record;
-import java.nio.ByteBuffer;
-
-/** Mock for Realtime CDC buffer consumer. */
-public interface CdcBufferConsumer {
-/**
- * Consumes raw WAL data.
- *
- * @param data Raw data to consume.
- */
-public void consume(ByteBuffer data);
-
-/** */
-public void close();
+/** */
+public class RealtimeCdcRecord extends WALRecord {
+/** {@inheritDoc} */
+@Override public RecordType type() {
+return RecordType.REALTIME_CDC_RECORD;
+}
}
diff --git
a/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/cdc/CdcBufferConsumer.java
b/modules/core/src/main/java/org/apache/ignite/internal/pagemem/wal/record/RealtimeCdcStopRecord.java
similarity index 69%
copy from
modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/cdc/CdcBufferConsumer.java
copy to
modules/core/src/main/java/org/apache/ignite/internal/pagemem/wal/record/RealtimeCdcStopRecord.java
index 67cd422fb0f..5b89bcd564a 100644
---
a/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/cdc/CdcBufferConsumer.java
+++
b/modules/core/src/main/java/org/apache/ignite/internal/pagemem/wal/record/RealtimeCdcStopRecord.java
@@ -15,19 +15,12 @@
* limitations under the License.
*/
-package org.apache.ignite.internal.processors.cache.persistence.cdc;
+package org.apache.ignite.internal.pagemem.wal.record;
-import java.nio.ByteBuffer;
-
-/** Mock for Realtime CDC buffer consumer. */
-public interface CdcBufferConsumer {
-/**
- * Consumes raw WAL data.
- *
- * @param data Raw data to consume.
- */
-public void consume(ByteBuffer data);
-
-/** */
-public void close();
+/** */
+public class RealtimeCdcStopRecord extends WALRecord {
+/** {@inheritDoc} */
+@Override public RecordType type() {
+return RecordType.REALTIME_STOP_CDC_RECORD;
+}
}
diff --git
a/modules/core/src/main/java/org/apache/ignite/internal/pagemem/wal/record/WALRecord.java
b/modules/core/src/main/java/org/apache/ignite/internal/pagemem/wal/record/WALRecord.java
index 3457251e0f4..02f4125215e 100644
---
a/modules/core/src/main/java/org/apache/ignite/internal/pagemem/wal/record/WALRecord.java
+++
b/modules/core/src/main/java/org/apache/ignite/internal/pagemem/wal/record/WALRecord.java
@@ -291,7 +291,13 @@ public abstract class WALRecord {
INCREMENTAL_SNAPSHOT_FINISH_RECORD(77, LOGICAL),
/** CDC data record. */
-CDC_DATA_RECORD(78, CUSTOM);
+CDC_DATA_RECORD(78, CUSTOM),
+
+/** Realtime CDC record. */
+REALTIME_CDC_RECORD(79, CUSTOM),
+
+/** Realtime CDC record. */
+REALTIME_STOP_CDC_RECORD(80, CUSTOM);
/** Index for serialization. Should be consistent throughout all
versions. */
private final int idx;
diff --git
a/modules/core/src/main/java
[ignite] branch IGNITE-17700__realtime_cdc updated: IGNITE-19622 Add realtime CDC buffer (#10778)
This is an automated email from the ASF dual-hosted git repository.
timoninmaxim pushed a commit to branch IGNITE-17700__realtime_cdc
in repository https://gitbox.apache.org/repos/asf/ignite.git
The following commit(s) were added to refs/heads/IGNITE-17700__realtime_cdc by
this push:
new 1c2cf8dd0a4 IGNITE-19622 Add realtime CDC buffer (#10778)
1c2cf8dd0a4 is described below
commit 1c2cf8dd0a426e88d9b70cc7d15db71f9edd694c
Author: Maksim Timonin
AuthorDate: Wed Jul 12 13:47:01 2023 +0300
IGNITE-19622 Add realtime CDC buffer (#10778)
---
.../org/apache/ignite/IgniteSystemProperties.java | 8 +
.../configuration/DataStorageConfiguration.java| 55 +
.../cache/persistence/cdc/CdcBuffer.java | 164 +
.../cache/persistence/cdc/CdcBufferConsumer.java | 33 +++
.../cache/persistence/cdc/CdcProcessor.java| 81 +++
.../cache/persistence/cdc/CdcWorker.java | 93
.../persistence/wal/FileWriteAheadLogManager.java | 19 +-
.../wal/filehandle/FileHandleManagerFactory.java | 6 +
.../wal/filehandle/FileHandleManagerImpl.java | 21 +-
.../wal/filehandle/FileWriteHandleImpl.java| 17 +-
.../wal/filehandle/FsyncFileHandleManagerImpl.java | 11 +-
.../wal/filehandle/FsyncFileWriteHandle.java | 16 +-
.../cdc/RealtimeCdcBufferDedicatedWorkerTest.java | 42
.../persistence/cdc/RealtimeCdcBufferSelfTest.java | 113 +
.../persistence/cdc/RealtimeCdcBufferTest.java | 259 +
.../cdc/RealtimeCdcBufferWalMmapDisabledTest.java | 27 +++
.../ignite/testsuites/IgnitePdsTestSuite2.java | 10 +
17 files changed, 966 insertions(+), 9 deletions(-)
diff --git
a/modules/core/src/main/java/org/apache/ignite/IgniteSystemProperties.java
b/modules/core/src/main/java/org/apache/ignite/IgniteSystemProperties.java
index 5fb101aebce..eaabed6607a 100644
--- a/modules/core/src/main/java/org/apache/ignite/IgniteSystemProperties.java
+++ b/modules/core/src/main/java/org/apache/ignite/IgniteSystemProperties.java
@@ -93,6 +93,7 @@ import static
org.apache.ignite.internal.processors.cache.distributed.dht.topolo
import static
org.apache.ignite.internal.processors.cache.mvcc.MvccCachingManager.DFLT_MVCC_TX_SIZE_CACHING_THRESHOLD;
import static
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.DFLT_DEFRAGMENTATION_REGION_SIZE_PERCENTAGE;
import static
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.DFLT_PDS_WAL_REBALANCE_THRESHOLD;
+import static
org.apache.ignite.internal.processors.cache.persistence.cdc.CdcWorker.DFLT_POLL_CDC_BUF_THROTTLING_TIMEOUT;
import static
org.apache.ignite.internal.processors.cache.persistence.checkpoint.CheckpointHistory.DFLT_PDS_MAX_CHECKPOINT_MEMORY_HISTORY_SIZE;
import static
org.apache.ignite.internal.processors.cache.persistence.checkpoint.CheckpointMarkersStorage.DFLT_IGNITE_CHECKPOINT_MAP_SNAPSHOT_THRESHOLD;
import static
org.apache.ignite.internal.processors.cache.persistence.checkpoint.CheckpointWorkflow.DFLT_CHECKPOINT_PARALLEL_SORT_THRESHOLD;
@@ -1531,6 +1532,13 @@ public final class IgniteSystemProperties {
type = Long.class, defaults = "" + DFLT_UNWIND_THROTTLING_TIMEOUT)
public static final String IGNITE_UNWIND_THROTTLING_TIMEOUT =
"IGNITE_UNWIND_THROTTLING_TIMEOUT";
+/**
+ * Throttling timeout in millis for polling CDC buffer in realtime mode.
Default is 100 ms.
+ */
+@SystemProperty(value = "Throttling timeout in millis for polling CDC
buffer in realtime mode", type = Long.class,
+defaults = "" + DFLT_POLL_CDC_BUF_THROTTLING_TIMEOUT)
+public static final String IGNITE_THROTTLE_POLL_CDC_BUF =
"IGNITE_THROTTLE_POLL_CDC_BUF";
+
/**
* Threshold for throttling operations logging.
*/
diff --git
a/modules/core/src/main/java/org/apache/ignite/configuration/DataStorageConfiguration.java
b/modules/core/src/main/java/org/apache/ignite/configuration/DataStorageConfiguration.java
index 8daa97a1c7b..86ed3c4e3ea 100644
---
a/modules/core/src/main/java/org/apache/ignite/configuration/DataStorageConfiguration.java
+++
b/modules/core/src/main/java/org/apache/ignite/configuration/DataStorageConfiguration.java
@@ -20,6 +20,7 @@ package org.apache.ignite.configuration;
import java.io.Serializable;
import java.util.zip.Deflater;
import org.apache.ignite.IgniteSystemProperties;
+import
org.apache.ignite.internal.processors.cache.persistence.cdc.CdcBufferConsumer;
import
org.apache.ignite.internal.processors.cache.persistence.file.AsyncFileIOFactory;
import
org.apache.ignite.internal.processors.cache.persistence.file.FileIOFactory;
import
org.apache.ignite.internal.processors.cache.persistence.file.RandomAccessFileIOFactory;
@@ -252,6 +253,14 @@ public class DataStorageConfiguration implements
Serializable {
@IgniteExperimental
private long
[ignite] branch master updated: IGNITE-17651 Update Ignite 2.14 release notes (#10826)
This is an automated email from the ASF dual-hosted git repository. timoninmaxim pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/ignite.git The following commit(s) were added to refs/heads/master by this push: new 5f6ab39a399 IGNITE-17651 Update Ignite 2.14 release notes (#10826) 5f6ab39a399 is described below commit 5f6ab39a3993a5b16acb368ebe46f3fab00d457b Author: Maksim Timonin AuthorDate: Thu Jul 6 17:26:17 2023 +0300 IGNITE-17651 Update Ignite 2.14 release notes (#10826) --- RELEASE_NOTES.txt | 3 +++ 1 file changed, 3 insertions(+) diff --git a/RELEASE_NOTES.txt b/RELEASE_NOTES.txt index b7ef164c981..a576a4a07d6 100644 --- a/RELEASE_NOTES.txt +++ b/RELEASE_NOTES.txt @@ -155,6 +155,9 @@ Apache Ignite In-Memory Distributed Database 2.14.0 --- (!) WARNINGS: +* Java thin client retrieves BinaryConfiguration from server and overrides configured and/or default properties. + The behavior is defined by ClientConfiguration#isAutoBinaryConfigurationEnabled. + This is a breaking change for users who use POJO as cache keys and rely on default BinaryConfiguration. * Removed LOCAL caches support. * Removed scalar module.
[ignite] branch IGNITE-17700__realtime_cdc updated (a225705fc5f -> ab168e7b661)
This is an automated email from the ASF dual-hosted git repository.
timoninmaxim pushed a change to branch IGNITE-17700__realtime_cdc
in repository https://gitbox.apache.org/repos/asf/ignite.git
from a225705fc5f IGNITE-16619 IndexQuery should support limit (#10767)
add 3c5cf62061a IGNITE-19742 Disabled Spring expressions in the cache
create command. (#10776)
add c85131866d0 IGNITE-19584 SQL Calcite: Fix SQL metrics
add 3d3862a7272 IGNITE-19767 Update Ignite dependency: Jetty - Fixes
#10784.
add fd74772c3f1 IGNITE-19766 Added topology snapshot info to services
system view (#10783)
add 7b0fab66f72 IGNITE-19676 Command JMX Invoker (#10775)
add c6abcdc3474 IGNITE-19845 IgniteTxLocalAdapter.sndTransformedVals field
removal WIP (#10805)
add 5aaa711b949 IGNITE-19847 IgniteTxAdapter.TxShadow removal WIP (#10807)
add 8d0e5e23cf7 IGNITE-19874 GridDhtTxLocal initial cleanup WIP (#10814)
add 7925f21fe50 IGNITE-19846 IgniteTxAdapter initial cleanup (#10806)
add 2fd73b23ffb IGNITE-19875 GridDhtTxLocalAdapter initial cleanup WIP
(#10815)
add 961fdd020a2 IGNITE-19873 GridNearTxLocal initial cleanup (#10813)
add a0320dcd320 IGNITE-19872 IgniteTxLocalAdapter initial cleanup (#10812)
add fe4d41b0483 IGNITE-19753 Move task clasees to management package -
Fixes #10797.
add ab168e7b661 IGNITE-19878 Restrict possibility to create two caches
with the same schemas and index names - Fixes #10817.
No new revisions were added by this update.
Summary of changes:
config/ignite-log4j.xml| 2 +-
docs/_docs/monitoring-metrics/new-metrics.adoc | 26 +
.../query/calcite/CalciteQueryProcessor.java | 39 +-
.../internal/processors/query/calcite/Query.java | 24 +-
.../processors/query/calcite/RootQuery.java| 30 +-
.../calcite/exec/ExecutionCancelledException.java | 24 -
.../query/calcite/exec/ExecutionServiceImpl.java | 3 +-
.../query/calcite/exec/rel/AbstractNode.java | 6 +-
.../query/calcite/exec/rel/RootNode.java | 13 +-
.../query/calcite/prepare/QueryPlanCacheImpl.java | 16 +-
.../integration/RunningQueriesIntegrationTest.java | 41 +-
.../integration/SqlDiagnosticIntegrationTest.java | 163 +
.../ignite/jdbc/thin/JdbcThinMetadataSelfTest.java | 1 +
.../snapshot/SnapshotCompressionBasicTest.java | 2 +-
.../internal/commandline/ArgumentParser.java | 180 +-
...{CommandInvoker.java => CliCommandInvoker.java} | 200 ++
.../internal/commandline/CommandHandler.java | 117 ++--
.../commandline/ConnectionAndSslParameters.java| 23 +-
.../argument/parser/CLIArgumentParser.java | 2 +-
.../commandline/indexreader/IgniteIndexReader.java | 4 +-
.../apache/ignite/events/BaselineEventsTest.java | 18 +-
.../commandline/CommandHandlerParsingTest.java | 12 +-
.../indexreader/IgniteIndexReaderTest.java | 7 +
.../GridCommandHandlerSslWithSecurityTest.java | 31 +-
.../ignite/util/CacheMetricsCommandTest.java | 2 +-
.../org/apache/ignite/util/CdcCommandTest.java | 41 +-
.../util/GridCommandHandlerAbstractTest.java | 15 +-
...CommandHandlerCheckIncrementalSnapshotTest.java | 27 +-
.../util/GridCommandHandlerClusterByClassTest.java | 73 ++-
...ridCommandHandlerClusterByClassWithSSLTest.java | 9 +
.../GridCommandHandlerConsistencyCountersTest.java | 29 +-
...sistencyRepairCorrectnessTransactionalTest.java | 15 +-
.../util/GridCommandHandlerConsistencyTest.java| 25 +-
.../GridCommandHandlerDefragmentationTest.java | 13 +-
.../GridCommandHandlerFactoryAbstractTest.java | 338 ++
...ridCommandHandlerIncompatibleSslConfigTest.java | 8 +
.../GridCommandHandlerIndexForceRebuildTest.java | 4 +-
.../util/GridCommandHandlerIndexListTest.java | 27 +-
.../GridCommandHandlerIndexRebuildStatusTest.java | 9 +-
.../GridCommandHandlerIndexingCheckSizeTest.java | 9 +-
.../GridCommandHandlerIndexingWithSSLTest.java | 9 +
.../GridCommandHandlerInterruptCommandTest.java| 11 +-
.../util/GridCommandHandlerMetadataTest.java | 14 +-
.../util/GridCommandHandlerPropertiesTest.java | 31 +-
.../ignite/util/GridCommandHandlerSslTest.java | 11 +-
.../apache/ignite/util/GridCommandHandlerTest.java | 150 +++--
...GridCommandHandlerTracingConfigurationTest.java | 35 +-
.../util/GridCommandHandlerWithSslFactoryTest.java | 10 +
.../ignite/util/GridCommandHandlerWithSslTest.java | 9 +
.../ignite/util/KillCommandsControlShTest.java | 33 +-
.../org/apache/ignite/util/MetricCommandTest.java | 32 +-
.../util/PerformanceStatisticsCommandTest.java | 5 +-
.../apache/ignite/util/SystemViewCommandTest.java | 46 +-
...ndHandlerFactoryAbstractTest$TestCommandHandler | 2 +
.../config/cache/cache-create-with-spel.xml| 27 +
.../java/org/apache/ignite/internal/IgniteEx.java | 7 +
.../org/apache/ign
[ignite] branch IGNITE-17700__realtime_cdc updated (6eda7c91b41 -> a225705fc5f)
This is an automated email from the ASF dual-hosted git repository.
timoninmaxim pushed a change to branch IGNITE-17700__realtime_cdc
in repository https://gitbox.apache.org/repos/asf/ignite.git
from 6eda7c91b41 IGNITE-19607 Fixed flaky IgniteClusterSnapshotStreamerTest
test (#10755)
add 9eaae91e0a5 IGNITE-19628 Add IndexQuery to performance statistics
add 6d022194ed3 Revert "IGNITE-19628 Add IndexQuery to performance
statistics"
add 8950aefc115 IGNITE-19620 Provided ability to configure metric exporter
filter from Spring XML (#10757)
add 53dba922826 IGNITE-19590 SQL Calcite: Add sensitive information hiding
when IGNITE_TO_STRING_INCLUDE_SENSITIVE property set to false - Fixes #10760.
add 5c2b42dd4ec IGNITE-19628 Added IndexQuery to performance statistics
(#10758)
add 97176a35396 IGNITE-19650 Fixed broken serialization of communication
messages due to incorrect GridCacheQueryRequest class marshalling. (#10768)
add 0fb4ac567d0 IGNITE-19378 Documentation: Improve docs related to lazy
loading - Fixes #10671.
add 46d865d63dd IGNITE-15629 Management API introduced (#10675)
add 77723f0d95c IGNITE-19588 SQL Calcite: Add long-running queries
warnings - Fixes #10769.
add 2fbd3a3329f IGNITE-19164 Improved message about requested partitions
during snapshot restore (#10638)
add f6e03372967 IGNITE-19164 Removed node consistent ID from snapshot
restore message (#10771)
add e7c9fe7eb1e IGNITE-19715 Thin client socket#open timeout changed to
ClientConfiguration#timeout property. (#10770)
add fc51f0e4327 IGNITE-19652 Adapt to DirectByteBuffer ctor changes in JDK
21 (#10764)
add a225705fc5f IGNITE-16619 IndexQuery should support limit (#10767)
No new revisions were added by this update.
Summary of changes:
docs/_docs/SQL/JDBC/jdbc-client-driver.adoc| 10 -
docs/_docs/SQL/JDBC/jdbc-driver.adoc | 6 -
docs/_docs/SQL/ODBC/connection-string-dsn.adoc | 6 -
.../_docs/perf-and-troubleshooting/sql-tuning.adoc | 17 +-
.../query/calcite/CalciteQueryProcessor.java | 66 +-
.../internal/processors/query/calcite/Query.java | 32 +-
.../processors/query/calcite/QueryRegistry.java| 11 +-
.../query/calcite/QueryRegistryImpl.java | 29 +-
.../processors/query/calcite}/QueryState.java | 2 +-
.../processors/query/calcite/RootQuery.java| 67 +-
.../query/calcite/exec/ExchangeService.java| 5 +
.../query/calcite/exec/ExchangeServiceImpl.java| 16 +-
.../query/calcite/exec/ExecutionServiceImpl.java | 35 +-
.../query/calcite/exec/rel/RootNode.java | 62 +-
.../calcite/prepare/AbstractMultiStepPlan.java | 5 +-
.../{QueryPlan.java => AbstractQueryPlan.java} | 24 +-
.../processors/query/calcite/prepare/DdlPlan.java | 6 +-
.../query/calcite/prepare/ExplainPlan.java | 6 +-
.../query/calcite/prepare/FragmentPlan.java| 6 +-
.../query/calcite/prepare/MultiStepDmlPlan.java| 5 +-
.../query/calcite/prepare/MultiStepQueryPlan.java | 5 +-
.../query/calcite/prepare/PrepareServiceImpl.java | 11 +-
.../query/calcite/prepare/QueryPlan.java | 5 +
.../calcite/util/ConvertingClosableIterator.java | 10 +-
.../processors/query/calcite/CancelTest.java | 6 +-
.../exec/rel/TimeCalculationExecutionTest.java | 170 +
.../KillQueryCommandDdlIntegrationTest.java| 2 +-
.../integration/RunningQueriesIntegrationTest.java | 25 +-
.../integration/SqlDiagnosticIntegrationTest.java | 190 +-
...citeCommunicationMessageSerializationTest.java} | 28 +-
.../query/calcite/planner/PlannerTest.java | 2 +-
.../ignite/testsuites/ExecutionTestSuite.java | 2 +
.../ignite/testsuites/IgniteCalciteTestSuite.java | 2 +
.../thin/JdbcThinPartitionAwarenessSelfTest.java | 2 +-
...ThinPartitionAwarenessTransactionsSelfTest.java | 2 +-
modules/control-utility/pom.xml| 7 +
.../internal/commandline/AbstractCommand.java | 70 --
.../internal/commandline/ActivateCommand.java | 75 ---
.../internal/commandline/ArgumentParser.java | 508 +++
.../internal/commandline/BaselineCommand.java | 316 -
.../commandline/ClusterChangeTagCommand.java | 115
.../commandline/ClusterStateChangeCommand.java | 130
.../ignite/internal/commandline/Command.java | 281
.../internal/commandline/CommandArgIterator.java | 216 ---
.../internal/commandline/CommandHandler.java | 400 +---
.../internal/commandline/CommandInvoker.java | 316 +
.../ignite/internal/commandline/CommandList.java | 167 -
.../ignite/internal/commandline/CommandLogger.java | 99 ---
.../internal/commandline/CommandsProvider.java | 5 +-
.../internal/commandline/CommonArgParser.java | 402
.../commandline/ConnectionAndSslParameters.
[ignite] branch master updated: IGNITE-16619 IndexQuery should support limit (#10767)
This is an automated email from the ASF dual-hosted git repository.
timoninmaxim pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/ignite.git
The following commit(s) were added to refs/heads/master by this push:
new a225705fc5f IGNITE-16619 IndexQuery should support limit (#10767)
a225705fc5f is described below
commit a225705fc5f487c2170dc584913f769f1b80f34b
Author: yurinaryshkin <135707807+yurinarysh...@users.noreply.github.com>
AuthorDate: Thu Jun 15 13:05:20 2023 +0300
IGNITE-16619 IndexQuery should support limit (#10767)
---
.../org/apache/ignite/cache/query/IndexQuery.java | 26 +++
.../client/thin/ProtocolBitmaskFeature.java| 5 +-
.../internal/client/thin/TcpClientCache.java | 7 +
.../processors/cache/IgniteCacheProxyImpl.java | 3 +
.../platform/client/ClientBitmaskFeature.java | 5 +-
.../platform/client/ClientMessageParser.java | 2 +-
.../client/cache/ClientCacheIndexQueryRequest.java | 15 +-
.../ignite/cache/query/IndexQueryLimitTest.java| 233 +
.../ignite/cache/query/IndexQueryTestSuite.java| 3 +-
.../cache/query/ThinClientIndexQueryTest.java | 65 ++
10 files changed, 359 insertions(+), 5 deletions(-)
diff --git
a/modules/core/src/main/java/org/apache/ignite/cache/query/IndexQuery.java
b/modules/core/src/main/java/org/apache/ignite/cache/query/IndexQuery.java
index 0e7f2a1abc9..8c61ccc1936 100644
--- a/modules/core/src/main/java/org/apache/ignite/cache/query/IndexQuery.java
+++ b/modules/core/src/main/java/org/apache/ignite/cache/query/IndexQuery.java
@@ -50,6 +50,9 @@ public final class IndexQuery extends
Query> {
/** Index name. */
private final @Nullable String idxName;
+/** Limit */
+private int limit;
+
/** Index query criteria. */
private @Nullable List criteria;
@@ -152,6 +155,29 @@ public final class IndexQuery extends
Query> {
return idxName;
}
+/**
+ * Gets limit to response records count.
+ *
+ * @return Limit value.
+ */
+public int getLimit() {
+return limit;
+}
+
+/**
+ * Sets limit to response records count.
+ *
+ * @param limit POsitive limit to set.
+ * @return {@code this} For chaining.
+ */
+public IndexQuery setLimit(int limit) {
+A.ensure(limit > 0, "Limit must be positive.");
+
+this.limit = limit;
+
+return this;
+}
+
/**
* Sets remote cache entries filter.
*
diff --git
a/modules/core/src/main/java/org/apache/ignite/internal/client/thin/ProtocolBitmaskFeature.java
b/modules/core/src/main/java/org/apache/ignite/internal/client/thin/ProtocolBitmaskFeature.java
index 3a586ec4952..751f66dbc27 100644
---
a/modules/core/src/main/java/org/apache/ignite/internal/client/thin/ProtocolBitmaskFeature.java
+++
b/modules/core/src/main/java/org/apache/ignite/internal/client/thin/ProtocolBitmaskFeature.java
@@ -71,7 +71,10 @@ public enum ProtocolBitmaskFeature {
ALL_AFFINITY_MAPPINGS(13),
/** IndexQuery. */
-INDEX_QUERY(14);
+INDEX_QUERY(14),
+
+/** IndexQuery limit. */
+INDEX_QUERY_LIMIT(15);
/** */
private static final EnumSet
ALL_FEATURES_AS_ENUM_SET =
diff --git
a/modules/core/src/main/java/org/apache/ignite/internal/client/thin/TcpClientCache.java
b/modules/core/src/main/java/org/apache/ignite/internal/client/thin/TcpClientCache.java
index 5edf586652b..130447a2309 100644
---
a/modules/core/src/main/java/org/apache/ignite/internal/client/thin/TcpClientCache.java
+++
b/modules/core/src/main/java/org/apache/ignite/internal/client/thin/TcpClientCache.java
@@ -1016,6 +1016,13 @@ public class TcpClientCache implements
ClientCache {
w.writeBoolean(qry.isLocal());
w.writeInt(qry.getPartition() == null ? -1 :
qry.getPartition());
+if
(!payloadCh.clientChannel().protocolCtx().isFeatureSupported(ProtocolBitmaskFeature.INDEX_QUERY_LIMIT))
{
+if (qry.getLimit() > 0)
+throw new
ClientFeatureNotSupportedByServerException(ProtocolBitmaskFeature.INDEX_QUERY_LIMIT);
+}
+else
+w.writeInt(qry.getLimit());
+
w.writeString(qry.getValueType());
w.writeString(qry.getIndexName());
diff --git
a/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/IgniteCacheProxyImpl.java
b/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/IgniteCacheProxyImpl.java
index 274f476042d..8d9f23f9a00 100644
---
a/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/IgniteCacheProxyImpl.java
+++
b/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/IgniteCacheProxyImpl.java
@@ -558,6 +558,9 @@ public class IgniteCacheProxyImpl extends
AsyncSupportAdapter 0)
+qry.limit(q.get
[ignite] branch IGNITE-17700__realtime_cdc created (now 6eda7c91b41)
This is an automated email from the ASF dual-hosted git repository. timoninmaxim pushed a change to branch IGNITE-17700__realtime_cdc in repository https://gitbox.apache.org/repos/asf/ignite.git at 6eda7c91b41 IGNITE-19607 Fixed flaky IgniteClusterSnapshotStreamerTest test (#10755) No new revisions were added by this update.
[ignite] branch master updated: IGNITE-19281 Fix flaky IgniteSnapshotMXBeanTest#testStatus (#10646)
This is an automated email from the ASF dual-hosted git repository.
timoninmaxim pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/ignite.git
The following commit(s) were added to refs/heads/master by this push:
new 75efe3d48a3 IGNITE-19281 Fix flaky IgniteSnapshotMXBeanTest#testStatus
(#10646)
75efe3d48a3 is described below
commit 75efe3d48a3c6b0cd1c2c8e285c4f661dd99c3e3
Author: Maksim Timonin
AuthorDate: Mon Apr 17 15:24:04 2023 +0300
IGNITE-19281 Fix flaky IgniteSnapshotMXBeanTest#testStatus (#10646)
---
.../cache/persistence/snapshot/SnapshotRestoreProcess.java | 9 ++---
.../cache/persistence/snapshot/IgniteSnapshotMXBeanTest.java | 4
2 files changed, 6 insertions(+), 7 deletions(-)
diff --git
a/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/snapshot/SnapshotRestoreProcess.java
b/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/snapshot/SnapshotRestoreProcess.java
index 12f9f0c54f2..508ae7e2ba9 100644
---
a/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/snapshot/SnapshotRestoreProcess.java
+++
b/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/snapshot/SnapshotRestoreProcess.java
@@ -290,7 +290,7 @@ public class SnapshotRestoreProcess {
throw new IgniteException(OP_REJECT_MSG + "A cluster snapshot
operation is in progress.");
synchronized (this) {
-if (restoringSnapshotName() != null)
+if (restoringSnapshotName() != null || fut != null)
throw new IgniteException(OP_REJECT_MSG + "The previous
snapshot restore operation was not completed.");
fut = new ClusterSnapshotFuture(UUID.randomUUID(), snpName,
incIdx);
@@ -437,12 +437,7 @@ public class SnapshotRestoreProcess {
public @Nullable String restoringSnapshotName() {
SnapshotRestoreContext opCtx0 = opCtx;
-if (opCtx0 != null)
-return opCtx0.snpName;
-
-ClusterSnapshotFuture fut0 = fut;
-
-return fut0 != null ? fut0.name : null;
+return opCtx0 == null ? null : opCtx0.snpName;
}
/**
diff --git
a/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/persistence/snapshot/IgniteSnapshotMXBeanTest.java
b/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/persistence/snapshot/IgniteSnapshotMXBeanTest.java
index 375ec60ccd2..91b88a45461 100644
---
a/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/persistence/snapshot/IgniteSnapshotMXBeanTest.java
+++
b/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/persistence/snapshot/IgniteSnapshotMXBeanTest.java
@@ -260,6 +260,8 @@ public class IgniteSnapshotMXBeanTest extends
AbstractSnapshotSelfTest {
// Restore full snapshot.
srv.destroyCache(DEFAULT_CACHE_NAME);
+awaitPartitionMapExchange();
+
spi.blockMessages((node, msg) -> msg instanceof SingleNodeMessage);
fut = srv.snapshot().restoreSnapshot(SNAPSHOT_NAME,
F.asList(DEFAULT_CACHE_NAME));
@@ -279,6 +281,8 @@ public class IgniteSnapshotMXBeanTest extends
AbstractSnapshotSelfTest {
if (!encryption) {
srv.destroyCache(DEFAULT_CACHE_NAME);
+awaitPartitionMapExchange();
+
spi.blockMessages((node, msg) -> msg instanceof SingleNodeMessage);
fut = srv.snapshot().restoreSnapshot(SNAPSHOT_NAME,
F.asList(DEFAULT_CACHE_NAME), 1);
[ignite] branch master updated: IGNITE-19279 Fix AssertionError on inactive cluster for thin client cache request (#10645)
This is an automated email from the ASF dual-hosted git repository.
timoninmaxim pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/ignite.git
The following commit(s) were added to refs/heads/master by this push:
new 2b1406e00fd IGNITE-19279 Fix AssertionError on inactive cluster for
thin client cache request (#10645)
2b1406e00fd is described below
commit 2b1406e00fdd9c226b60697fde7e3b3fc176b121
Author: Maksim Timonin
AuthorDate: Thu Apr 13 16:25:51 2023 +0300
IGNITE-19279 Fix AssertionError on inactive cluster for thin client cache
request (#10645)
---
.../client/cache/ClientCachePartitionsRequest.java | 3 +
.../thin/InactiveClusterCacheRequestTest.java | 88 ++
.../org/apache/ignite/client/ClientTestSuite.java | 2 +
3 files changed, 93 insertions(+)
diff --git
a/modules/core/src/main/java/org/apache/ignite/internal/processors/platform/client/cache/ClientCachePartitionsRequest.java
b/modules/core/src/main/java/org/apache/ignite/internal/processors/platform/client/cache/ClientCachePartitionsRequest.java
index 062d0ebf528..35740da13c6 100644
---
a/modules/core/src/main/java/org/apache/ignite/internal/processors/platform/client/cache/ClientCachePartitionsRequest.java
+++
b/modules/core/src/main/java/org/apache/ignite/internal/processors/platform/client/cache/ClientCachePartitionsRequest.java
@@ -82,6 +82,9 @@ public class ClientCachePartitionsRequest extends
ClientRequest {
/** {@inheritDoc} */
@Override public ClientResponse process(ClientConnectionContext ctx) {
+if (!ctx.kernalContext().state().publicApiActiveState(true))
+return new ClientResponse(requestId(), "Cannot perform cache
operation because the cluster is inactive.");
+
Map grps = new HashMap<>(cacheIds.length);
ClientAffinityTopologyVersion affinityVer =
ctx.checkAffinityTopologyVersion();
diff --git
a/modules/core/src/test/java/org/apache/ignite/internal/client/thin/InactiveClusterCacheRequestTest.java
b/modules/core/src/test/java/org/apache/ignite/internal/client/thin/InactiveClusterCacheRequestTest.java
new file mode 100644
index 000..b1c87d18b91
--- /dev/null
+++
b/modules/core/src/test/java/org/apache/ignite/internal/client/thin/InactiveClusterCacheRequestTest.java
@@ -0,0 +1,88 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.client.thin;
+
+import java.util.List;
+import org.apache.ignite.Ignition;
+import org.apache.ignite.client.ClientException;
+import org.apache.ignite.client.IgniteClient;
+import org.apache.ignite.cluster.ClusterState;
+import org.apache.ignite.configuration.CacheConfiguration;
+import org.apache.ignite.configuration.ClientConfiguration;
+import org.apache.ignite.configuration.DataRegionConfiguration;
+import org.apache.ignite.configuration.DataStorageConfiguration;
+import org.apache.ignite.configuration.IgniteConfiguration;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.util.typedef.F;
+import org.apache.ignite.testframework.GridTestUtils;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+/** */
+@RunWith(Parameterized.class)
+public class InactiveClusterCacheRequestTest extends AbstractThinClientTest {
+/** */
+@Parameterized.Parameter
+public boolean partitionAwarenessEnabled;
+
+/** */
+@Parameterized.Parameters(name = "partitionAwareness={0}")
+public static List params() {
+return F.asList(false, true);
+}
+
+/** {@inheritDoc} */
+@Override protected void afterTest() {
+stopAllGrids();
+}
+
+/** {@inheritDoc} */
+@Override protected IgniteConfiguration getConfiguration(String
igniteInstanceName) throws Exception {
+IgniteConfiguration cfg = super.getConfiguration(igniteInstanceName);
+
+cfg.setDataStorageConfiguration(new DataStorageConfiguration()
+.setDefaultDataRegionConfiguration(new
DataRegionConfiguration().setPersistenceEnabled(true)));
+
+cfg.setCacheConfiguration(new
CacheConfiguration<>(DEFAULT_CACHE_NAME));
+
[ignite-extensions] branch master updated (1a2d0087 -> 00c4040d)
This is an automated email from the ASF dual-hosted git repository. timoninmaxim pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/ignite-extensions.git from 1a2d0087 IGNITE-19254 Move ignite-ssh to extensions (#208) add 3959d6af IGNITE-18532 Fix runners join loop new 00c4040d IGNITE-18532 Fix runners join loop waits for the excess thread The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../org/apache/ignite/cdc/kafka/AbstractKafkaToIgniteCdcStreamer.java | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
[ignite-extensions] 01/01: IGNITE-18532 Fix runners join loop waits for the excess thread
This is an automated email from the ASF dual-hosted git repository. timoninmaxim pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/ignite-extensions.git commit 00c4040dabca263da663f730c2f0519d02f72be0 Merge: 1a2d0087 3959d6af Author: Maksim Timonin AuthorDate: Wed Apr 12 19:16:27 2023 +0300 IGNITE-18532 Fix runners join loop waits for the excess thread .../org/apache/ignite/cdc/kafka/AbstractKafkaToIgniteCdcStreamer.java | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
[ignite] branch master updated (ee922ece351 -> d0d3ead6d7b)
This is an automated email from the ASF dual-hosted git repository. timoninmaxim pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/ignite.git from ee922ece351 IGNITE-19254 Move ignite-ssh to extensions (#10635) add d0d3ead6d7b IGNITE-18550 Add idle_verify check of incremental snapshots (#10617) No new revisions were added by this update. Summary of changes: docs/_docs/tools/control-script.adoc | 19 +- .../commandline/snapshot/SnapshotCheckCommand.java | 18 +- .../snapshot/SnapshotCheckCommandOption.java | 6 +- .../testsuites/IgniteControlUtilityTestSuite.java | 2 + ...CommandHandlerCheckIncrementalSnapshotTest.java | 443 .../org/apache/ignite/internal/cdc/CdcMain.java| 49 +-- .../managers/discovery/ConsistentIdMapper.java | 6 +- .../processors/cache/GridLocalConfigManager.java | 35 ++ .../snapshot/AbstractSnapshotVerificationTask.java | 3 + .../snapshot/IgniteSnapshotManager.java| 10 +- .../snapshot/IncrementalSnapshotProcessor.java | 22 +- .../IncrementalSnapshotVerificationTask.java | 464 + .../IncrementalSnapshotVerificationTaskResult.java | 106 + .../snapshot/SnapshotHandlerRestoreTask.java | 1 + .../SnapshotPartitionsQuickVerifyHandler.java | 3 +- .../snapshot/SnapshotPartitionsVerifyTask.java | 1 + .../snapshot/SnapshotPartitionsVerifyTaskArg.java | 15 + .../snapshot/SnapshotRestoreProcess.java | 2 +- .../cache/verify/IdleVerifyResultV2.java | 75 +++- .../cache/verify/PartitionHashRecordV2.java| 4 +- .../cache/verify/TransactionsHashRecord.java | 89 .../visor/snapshot/VisorSnapshotCheckTask.java | 3 +- .../visor/snapshot/VisorSnapshotCheckTaskArg.java | 14 +- .../visor/snapshot/VisorSnapshotCreateTaskArg.java | 32 +- .../snapshot/IgniteClusterSnapshotCheckTest.java | 1 + .../ConcurrentTxsIncrementalSnapshotTest.java | 9 + ...ridCommandHandlerClusterByClassTest_help.output | 7 +- ...andHandlerClusterByClassWithSSLTest_help.output | 7 +- 28 files changed, 1372 insertions(+), 74 deletions(-) create mode 100644 modules/control-utility/src/test/java/org/apache/ignite/util/GridCommandHandlerCheckIncrementalSnapshotTest.java create mode 100644 modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/snapshot/IncrementalSnapshotVerificationTask.java create mode 100644 modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/snapshot/IncrementalSnapshotVerificationTaskResult.java create mode 100644 modules/core/src/main/java/org/apache/ignite/internal/processors/cache/verify/TransactionsHashRecord.java
[ignite] branch master updated: IGNITE-18927 Add incremental snapshot docs (#10609)
This is an automated email from the ASF dual-hosted git repository.
timoninmaxim pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/ignite.git
The following commit(s) were added to refs/heads/master by this push:
new 857fe891c5d IGNITE-18927 Add incremental snapshot docs (#10609)
857fe891c5d is described below
commit 857fe891c5d887820e0076912ce78f84ee8c4733
Author: Maksim Timonin
AuthorDate: Tue Mar 28 15:55:09 2023 +0300
IGNITE-18927 Add incremental snapshot docs (#10609)
---
.../java/org/apache/ignite/snippets/Snapshots.java | 9 ++-
docs/_docs/monitoring-metrics/system-views.adoc| 3 +
docs/_docs/snapshots/snapshots.adoc| 82 ++
.../snapshot/IgniteSnapshotManager.java| 2 +-
4 files changed, 82 insertions(+), 14 deletions(-)
diff --git
a/docs/_docs/code-snippets/java/src/main/java/org/apache/ignite/snippets/Snapshots.java
b/docs/_docs/code-snippets/java/src/main/java/org/apache/ignite/snippets/Snapshots.java
index 1e3f74fdc2a..0aa2fb44414 100644
---
a/docs/_docs/code-snippets/java/src/main/java/org/apache/ignite/snippets/Snapshots.java
+++
b/docs/_docs/code-snippets/java/src/main/java/org/apache/ignite/snippets/Snapshots.java
@@ -41,8 +41,11 @@ public class Snapshots {
try (IgniteCache cache =
ignite.getOrCreateCache(ccfg)) {
cache.put(1, "Maxim");
-// Start snapshot operation.
+// Create snapshot operation.
ignite.snapshot().createSnapshot("snapshot_02092020").get();
+
+// Create incremental snapshot operation.
+
ignite.snapshot().createIncrementalSnapshot("snapshot_02092020").get();
}
finally {
ignite.destroyCache(ccfg.getName());
@@ -52,6 +55,10 @@ public class Snapshots {
//tag::restore[]
// Restore cache named "snapshot-cache" from the snapshot
"snapshot_02092020".
ignite.snapshot().restoreSnapshot("snapshot_02092020",
Collections.singleton("snapshot-cache")).get();
+
+// Restore cache named "snapshot-cache" from the snapshot
"snapshot_02092020" and its increment with index 1.
+ignite.snapshot().restoreSnapshot("snapshot_02092020",
Collections.singleton("snapshot-cache"), 1).get();
+
//end::restore[]
ignite.close();
diff --git a/docs/_docs/monitoring-metrics/system-views.adoc
b/docs/_docs/monitoring-metrics/system-views.adoc
index 55f4dea139e..363f0006d24 100644
--- a/docs/_docs/monitoring-metrics/system-views.adoc
+++ b/docs/_docs/monitoring-metrics/system-views.adoc
@@ -988,4 +988,7 @@ The SNAPSHOT view exposes information about local snapshots.
| CONSISTENT_ID | VARCHAR | Consistent ID of a node to which snapshot data
relates.
| BASELINE_NODES | VARCHAR | Baseline nodes affected by the snapshot.
| CACHE_GROUPS | VARCHAR | Cache group names that were included in the
snapshot.
+| SNAPSHOT_RECORD_SEGMENT | BIGINT | Index of WAL segment containing
snapshot's WAL record.
+| INCREMENT_INDEX | INTEGER | Incremental snapshot index.
+| TYPE | VARCHAR | Type of snapshot - full or incremental.
|===
diff --git a/docs/_docs/snapshots/snapshots.adoc
b/docs/_docs/snapshots/snapshots.adoc
index c1e47857e15..b40b16ce83d 100644
--- a/docs/_docs/snapshots/snapshots.adoc
+++ b/docs/_docs/snapshots/snapshots.adoc
@@ -16,7 +16,7 @@
== Overview
-Ignite provides an ability to create full cluster snapshots for deployments
using
+Ignite provides an ability to create full and incremental cluster snapshots
for deployments using
link:persistence/native-persistence[Ignite Persistence]. An Ignite snapshot
includes a consistent cluster-wide copy of
all data records persisted on disk and some other files needed for a restore
procedure.
@@ -28,15 +28,15 @@ with several exceptions. Let's take this snapshot as an
example to review the st
work
└── snapshots
└── backup23012020
+├── increments
+│└── 0001
└── db
├── binary_meta
│ ├── node1
│ ├── node2
│ └── node3
├── marshaller
-│ ├── node1
-│ ├── node2
-│ └── node3
+│ └── classname0
├── node1
│└── my-sample-cache
│├── cache_data.dat
@@ -62,8 +62,10 @@ the nodes are named as `node1`, `node2`, and `node3`, while
in practice, the nam
link:https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Persistent+Store+-+under+the+hood#IgnitePersistentStoreunderthehood-SubfoldersGeneration[consistent
IDs].
* The snapshot keeps a copy of the `my-sample-cache` cache.
* The `db` folder keeps a copy of data records in `part-N.bin` and
`cache_data.dat` files. Write-ahead and checkpointing
-are no
[ignite] branch master updated: IGNITE-19052 Introduce IncrementalSnapshotProcessor (#10600)
This is an automated email from the ASF dual-hosted git repository.
timoninmaxim pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/ignite.git
The following commit(s) were added to refs/heads/master by this push:
new 307ebcc5d3a IGNITE-19052 Introduce IncrementalSnapshotProcessor
(#10600)
307ebcc5d3a is described below
commit 307ebcc5d3a7744dd9cf1a9cac103a776209c084
Author: Maksim Timonin
AuthorDate: Tue Mar 28 10:07:28 2023 +0300
IGNITE-19052 Introduce IncrementalSnapshotProcessor (#10600)
---
.../snapshot/IncrementalSnapshotProcessor.java | 288 +
.../snapshot/SnapshotRestoreProcess.java | 246 +++---
2 files changed, 317 insertions(+), 217 deletions(-)
diff --git
a/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/snapshot/IncrementalSnapshotProcessor.java
b/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/snapshot/IncrementalSnapshotProcessor.java
new file mode 100644
index 000..2ca529e086a
--- /dev/null
+++
b/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/snapshot/IncrementalSnapshotProcessor.java
@@ -0,0 +1,288 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.processors.cache.persistence.snapshot;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashSet;
+import java.util.Set;
+import java.util.UUID;
+import java.util.concurrent.atomic.LongAdder;
+import java.util.function.Consumer;
+import org.apache.ignite.IgniteCheckedException;
+import org.apache.ignite.IgniteLogger;
+import org.apache.ignite.internal.pagemem.wal.WALIterator;
+import org.apache.ignite.internal.pagemem.wal.record.DataEntry;
+import org.apache.ignite.internal.pagemem.wal.record.DataRecord;
+import
org.apache.ignite.internal.pagemem.wal.record.IncrementalSnapshotFinishRecord;
+import
org.apache.ignite.internal.pagemem.wal.record.IncrementalSnapshotStartRecord;
+import org.apache.ignite.internal.pagemem.wal.record.WALRecord;
+import
org.apache.ignite.internal.pagemem.wal.record.delta.ClusterSnapshotRecord;
+import org.apache.ignite.internal.processors.cache.GridCacheSharedContext;
+import
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager;
+import org.apache.ignite.internal.processors.cache.persistence.wal.WALPointer;
+import
org.apache.ignite.internal.processors.cache.persistence.wal.reader.IgniteWalIteratorFactory;
+import org.apache.ignite.internal.processors.cache.version.GridCacheVersion;
+import org.apache.ignite.internal.util.typedef.F;
+import org.apache.ignite.lang.IgniteBiTuple;
+import org.apache.ignite.lang.IgnitePredicate;
+import org.jetbrains.annotations.Nullable;
+
+import static
org.apache.ignite.internal.pagemem.wal.record.WALRecord.RecordType.CLUSTER_SNAPSHOT;
+import static
org.apache.ignite.internal.pagemem.wal.record.WALRecord.RecordType.DATA_RECORD_V2;
+import static
org.apache.ignite.internal.pagemem.wal.record.WALRecord.RecordType.INCREMENTAL_SNAPSHOT_FINISH_RECORD;
+import static
org.apache.ignite.internal.pagemem.wal.record.WALRecord.RecordType.INCREMENTAL_SNAPSHOT_START_RECORD;
+import static
org.apache.ignite.internal.processors.cache.persistence.snapshot.IgniteSnapshotManager.incrementalSnapshotWalsDir;
+import static
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager.WAL_SEGMENT_COMPACTED_OR_RAW_FILE_FILTER;
+
+/** Processes incremental snapshot: parse WAL segments and handles records. */
+abstract class IncrementalSnapshotProcessor {
+/** Cache shared context. */
+private final GridCacheSharedContext cctx;
+
+/** Ignite logger. */
+private final IgniteLogger log;
+
+/** Snapshot name. */
+private final String snpName;
+
+/** Snapshot path. */
+private final String snpPath;
+
+/** Incremental snapshot index. */
+private final int incIdx;
+
+/** Snapshot cache IDs. */
+private final Set cacheIds;
+
+/** */
+IncrementalSnapshotProcessor(Gr
[ignite] branch master updated: IGNITE-17321 Add docs for java client partition awareness use cases (#10453)
This is an automated email from the ASF dual-hosted git repository.
timoninmaxim pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/ignite.git
The following commit(s) were added to refs/heads/master by this push:
new dc4c60330e0 IGNITE-17321 Add docs for java client partition awareness
use cases (#10453)
dc4c60330e0 is described below
commit dc4c60330e0d3f347be5d37458f63775b84d0c71
Author: Julia Bakulina <96814257+j-bak...@users.noreply.github.com>
AuthorDate: Fri Mar 10 11:11:23 2023 +0300
IGNITE-17321 Add docs for java client partition awareness use cases (#10453)
---
.../org/apache/ignite/snippets/JavaThinClient.java | 3 ++
docs/_docs/thin-clients/java-thin-client.adoc | 6
.../ignite/configuration/ClientConfiguration.java | 38 --
3 files changed, 37 insertions(+), 10 deletions(-)
diff --git
a/docs/_docs/code-snippets/java/src/main/java/org/apache/ignite/snippets/JavaThinClient.java
b/docs/_docs/code-snippets/java/src/main/java/org/apache/ignite/snippets/JavaThinClient.java
index c11241a4ae8..8a9c2fdc642 100644
---
a/docs/_docs/code-snippets/java/src/main/java/org/apache/ignite/snippets/JavaThinClient.java
+++
b/docs/_docs/code-snippets/java/src/main/java/org/apache/ignite/snippets/JavaThinClient.java
@@ -359,6 +359,9 @@ public class JavaThinClient {
try (IgniteClient client = Ignition.startClient(cfg)) {
ClientCache cache = client.cache("myCache");
// Put, get or remove data from the cache...
+cache.put(0, "Hello, world!");
+// The partition number can be specified with
IndexQuery#setPartition(Integer) as well.
+ScanQuery scanQuery = new ScanQuery().setPartition(part);
} catch (ClientException e) {
System.err.println(e.getMessage());
}
diff --git a/docs/_docs/thin-clients/java-thin-client.adoc
b/docs/_docs/thin-clients/java-thin-client.adoc
index 1469e446645..5f955d35615 100644
--- a/docs/_docs/thin-clients/java-thin-client.adoc
+++ b/docs/_docs/thin-clients/java-thin-client.adoc
@@ -79,6 +79,12 @@ Note that the code above provides a failover mechanism in
case of server node fa
include::includes/partition-awareness.adoc[]
+Partition awareness functionality helps to avoid an additional network hop in
the following scenarios:
+
+1. Single-key operations API, like put(), get(), etc. However, the
functionality has no effect on those operations within explicit transactions
(initiated via ClientTransaction#txStart() described in <>
section).
+
+2. ScanQuery and IndexQuery accept a partition number as a parameter with
which the query is routed to a particular server node that stores the requested
data. Refer to <> and
link:key-value-api/using-cache-queries#executing-index-queries[Executing Index
Queries] sections for more information.
+
The following code sample illustrates how to use the partition awareness
feature with the java thin client.
[source, java]
diff --git
a/modules/core/src/main/java/org/apache/ignite/configuration/ClientConfiguration.java
b/modules/core/src/main/java/org/apache/ignite/configuration/ClientConfiguration.java
index a2d1cdbfbf9..59a8b1c6ac7 100644
---
a/modules/core/src/main/java/org/apache/ignite/configuration/ClientConfiguration.java
+++
b/modules/core/src/main/java/org/apache/ignite/configuration/ClientConfiguration.java
@@ -27,11 +27,14 @@ import javax.cache.configuration.Factory;
import javax.net.ssl.SSLContext;
import org.apache.ignite.IgniteLogger;
+import org.apache.ignite.cache.query.IndexQuery;
+import org.apache.ignite.cache.query.ScanQuery;
import org.apache.ignite.client.ClientAddressFinder;
import org.apache.ignite.client.ClientPartitionAwarenessMapper;
import org.apache.ignite.client.ClientPartitionAwarenessMapperFactory;
import org.apache.ignite.client.ClientRetryAllPolicy;
import org.apache.ignite.client.ClientRetryPolicy;
+import org.apache.ignite.client.ClientTransactions;
import org.apache.ignite.client.SslMode;
import org.apache.ignite.client.SslProtocol;
import org.apache.ignite.internal.client.thin.TcpIgniteClient;
@@ -529,12 +532,20 @@ public final class ClientConfiguration implements
Serializable {
}
/**
- * @return A value indicating whether partition awareness should be
enabled.
- *
- * Default is {@code true}: client sends requests directly to the primary
node for the given cache key.
- * To do so, connection is established to every known server node.
- *
+ * Default is {@code true}: client sends requests directly to the
primary node for the given cache key.
+ * To do so, connection is established to every known server node.
* When {@code false}, only one connection is established at a given
moment to a random server node.
+ *
+ * Partition awareness functionality helps to avoid an additional network
hop in the following scenari
[ignite] branch master updated (04304d4e9e5 -> a8e16899514)
This is an automated email from the ASF dual-hosted git repository. timoninmaxim pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/ignite.git omit 04304d4e9e5 IGNITE-17177 Added incremental snapshots (#10539) new a8e16899514 IGNITE-17177 Added incremental snapshots (#10539) This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (04304d4e9e5) \ N -- N -- N refs/heads/master (a8e16899514) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes:
[ignite] branch IGNITE-17177_inc_snapshots updated (1b0e1b6cf12 -> f4ab4091f47)
This is an automated email from the ASF dual-hosted git repository.
timoninmaxim pushed a change to branch IGNITE-17177_inc_snapshots
in repository https://gitbox.apache.org/repos/asf/ignite.git
from 1b0e1b6cf12 Merge branch 'master' into IGNITE-17177_inc_snapshots
add f4ab4091f47 IGNITE-18877 Add yardstick benchmark for incremental
snapshots (#10587)
No new revisions were added by this update.
Summary of changes:
...roperties => benchmark-inc-snapshot.properties} | 29 ++
...xml => ignite-config-incremental-snapshots.xml} | 20 +---
.../cache/IgniteIncrementalSnapshotsBenchmark.java | 101 +
3 files changed, 109 insertions(+), 41 deletions(-)
copy modules/yardstick/config/{benchmark-sample.properties =>
benchmark-inc-snapshot.properties} (73%)
copy modules/yardstick/config/{vendors/ignite-config-persistence.xml =>
ignite-config-incremental-snapshots.xml} (78%)
create mode 100644
modules/yardstick/src/main/java/org/apache/ignite/yardstick/cache/IgniteIncrementalSnapshotsBenchmark.java
[ignite] branch master updated: IGNITE-18546 Add confirmation for clear command (#10589)
This is an automated email from the ASF dual-hosted git repository.
timoninmaxim pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/ignite.git
The following commit(s) were added to refs/heads/master by this push:
new 28048e234c3 IGNITE-18546 Add confirmation for clear command (#10589)
28048e234c3 is described below
commit 28048e234c3bc3006375944827b6a66330f202ac
Author: Maksim Timonin
AuthorDate: Thu Mar 9 11:42:59 2023 +0300
IGNITE-18546 Add confirmation for clear command (#10589)
---
.../apache/ignite/internal/commandline/cache/CacheClear.java | 9 +
.../ignite/util/GridCommandHandlerClusterByClassTest.java | 11 +++
2 files changed, 20 insertions(+)
diff --git
a/modules/control-utility/src/main/java/org/apache/ignite/internal/commandline/cache/CacheClear.java
b/modules/control-utility/src/main/java/org/apache/ignite/internal/commandline/cache/CacheClear.java
index 33acfb0cec5..45fa20647c0 100644
---
a/modules/control-utility/src/main/java/org/apache/ignite/internal/commandline/cache/CacheClear.java
+++
b/modules/control-utility/src/main/java/org/apache/ignite/internal/commandline/cache/CacheClear.java
@@ -43,6 +43,10 @@ public class CacheClear extends
AbstractCommand {
/** Message that contains not-cleared caches (they don't exist). */
public static final String SKIP_CLEAR_MSG = "The following caches don't
exist: %s";
+/** Confirmation message format. */
+public static final String CONFIRM_MSG = "Warning! The command will clear
all data from %d caches: %s.\n" +
+"If you continue, it will be impossible to recover cleared data.";
+
/** Comma-separated list of cache names. */
public static final String CACHES = "--caches";
@@ -64,6 +68,11 @@ public class CacheClear extends
AbstractCommand {
return null;
}
+/** {@inheritDoc} */
+@Override public String confirmationPrompt() {
+return String.format(CONFIRM_MSG, arg.caches().size(), String.join(",
", arg.caches()));
+}
+
/** {@inheritDoc} */
@Override public ClearCachesTaskArg arg() {
return arg;
diff --git
a/modules/control-utility/src/test/java/org/apache/ignite/util/GridCommandHandlerClusterByClassTest.java
b/modules/control-utility/src/test/java/org/apache/ignite/util/GridCommandHandlerClusterByClassTest.java
index b6fa0ee3053..339247850f2 100644
---
a/modules/control-utility/src/test/java/org/apache/ignite/util/GridCommandHandlerClusterByClassTest.java
+++
b/modules/control-utility/src/test/java/org/apache/ignite/util/GridCommandHandlerClusterByClassTest.java
@@ -1306,6 +1306,17 @@ public class GridCommandHandlerClusterByClassTest
extends GridCommandHandlerClus
assertEquals(EXIT_CODE_INVALID_ARGUMENTS, execute("--cache",
CLEAR.text(), CacheClear.CACHES, "X,Y", "Z"));
assertContains(log, testOut.toString(), "Invalid argument \"Z\", no
more arguments are expected.");
+autoConfirmation = false;
+
+String expConfirmation = String.format(CacheClear.CONFIRM_MSG, 2,
"cache1, cache2");
+
+// Ensure we cannot delete a cache groups.
+injectTestSystemIn(CONFIRM_MSG);
+assertEquals(EXIT_CODE_OK, execute("--cache", CLEAR.text(),
CacheClear.CACHES, "cache1,cache2"));
+assertContains(log, testOut.toString(), expConfirmation);
+
+autoConfirmation = true;
+
List caches = F.asList("cache1", "cache2", "cache3");
for (boolean sql: new boolean[] {false, true}) {
[ignite] branch IGNITE-17177_inc_snapshots updated (527bcd538727 -> 1b0e1b6cf12c)
This is an automated email from the ASF dual-hosted git repository. timoninmaxim pushed a change to branch IGNITE-17177_inc_snapshots in repository https://gitbox.apache.org/repos/asf/ignite.git from 527bcd538727 MINOR: fix misprints, make logs more useful add b56f2e54a3d8 IGNITE-18816 Changed example in docs. Fixes #10556 add d139861141b1 IGNITE-18887 .NET: Fix TestContextLogger and ConsoleLogger to include exception details (#10560) add df22bb8d9c77 IGNITE-18834 Index rebuild ducktest (#10561) add bb4e9be54383 IGNITE-18823 Add page read time and page replace time metrics - Fixes #10545. add 1c18dcb9e082 IGNITE-12483 Add default constructor to PlatformDotNetSessionLockResult to avoid issues with ReflectionFactory - Fixes #10550. add b59c3a9dd16e IGNITE-18873 Update Ignite dependency: Netty - Fixes #10557. add 479281e1aa69 IGNITE-16319 [ducktests] Use the only application to generate test data (#10566) add 799673548476 IGNITE-18921 [ducktests] Fix consistent ID directory name (#10568) add 9db91c4348c1 IGNITE-18534 Fix WalWritingRate calculation (#10508) add c569d8682c59 IGNITE-18849 Add LOB and BLOB fields support to CacheJdbcPojoStore (#10553) add 511736b175ed IGNITE-18946 Fix refresh command about the statistic in the ddl doc - Fixes #10578. add 1f238846c8be IGNITE-18936 [ducktests] Fix parsing of the control.sh baseline command output and store JVM error file (#10575) add e220e8006e45 IGNITE-18546 Add cache clear command to control.sh (#10577) add d5af66c56583 IGNITE-18819 Added cluster state change permission (#10558) add 248d9e562ce0 IGNITE-18923 Added authorization of user management operations. (#10567) add 6229b46c2777 IGNITE-18935 Fix late stopping of TTL workers during deactivation leads to corrupted PDS (#10570) add 1b0e1b6cf12c Merge branch 'master' into IGNITE-17177_inc_snapshots No new revisions were added by this update. Summary of changes: docs/_docs/clustering/discovery-in-the-cloud.adoc | 2 +- docs/_docs/sql-reference/ddl.adoc | 6 +- docs/_docs/tools/control-script.adoc | 24 ++ .../common/AbstractEventSecurityContextTest.java | 23 +- ...CacheCreateDestroyEventSecurityContextTest.java | 6 +- .../common/CacheEventSecurityContextTest.java | 6 +- .../ComputeTaskRemoteSecurityContextTest.java | 34 +-- .../apache/ignite/jdbc/JdbcAuthorizationTest.java | 97 +- .../JdbcThinConnectionAdditionalSecurityTest.java | 26 +- modules/compress/pom.xml | 7 + .../internal/commandline/cache/CacheClear.java | 108 +++ .../commandline/cache/CacheCommandList.java| 5 + .../commandline/cache/CacheSubcommands.java| 7 +- .../util/GridCommandHandlerClusterByClassTest.java | 93 ++ .../cache/store/jdbc/CacheAbstractJdbcStore.java | 29 +- .../store/jdbc/JdbcTypesDefaultTransformer.java| 3 + .../internal/binary/BinaryClassDescriptor.java | 5 + .../ignite/internal/binary/BinaryContext.java | 10 + .../managers/indexing/IndexesRebuildTask.java | 4 +- .../internal/processors/cache/ClearCachesTask.java | 81 + .../processors/cache/ClearCachesTaskArg.java | 67 .../processors/cache/ClearCachesTaskResult.java| 70 + .../processors/cache/GridCacheProcessor.java | 58 ++-- .../cache/persistence/DataRegionMetricsImpl.java | 28 +- .../cache/persistence/DataStorageMetricsImpl.java | 8 - .../cache/persistence/pagemem/PageMemoryImpl.java | 19 +- .../wal/filehandle/FileHandleManagerImpl.java | 2 - .../wal/filehandle/FsyncFileWriteHandle.java | 2 - .../cluster/GridClusterStateProcessor.java | 3 + .../PlatformDotNetSessionLockResult.java | 7 + .../processors/rest/GridRestProcessor.java | 10 +- .../security/IgniteSecurityProcessor.java | 7 + .../ignite/plugin/security/SecurityPermission.java | 11 + .../cache/store/jdbc/CacheJdbcPojoStoreTest.java | 138 - .../apache/ignite/cache/store/jdbc/model/Logo.java | 138 + .../ignite/cache/store/jdbc/model/LogoKey.java | 86 ++ .../binary/BinaryContextPredefinedTypesTest.java | 46 +++ .../processors/cache/IoDatastorageMetricsTest.java | 104 +++ .../db/IgnitePdsDataRegionMetricsTest.java | 70 + ...IgnitePdsWithTtlExpirationOnDeactivateTest.java | 222 ++ .../processors/security/AbstractSecurityTest.java | 16 +- .../processors/security/InvalidServerTest.java | 3 +- .../security/client/CommonSecurityCheckTest.java | 3 +- .../cluster/ClusterStatePermissionTest.java| 339 + .../impl/TestCertificateSecurityProcessor.java | 2 +- .../security/impl/TestSecurityContext.java | 2 + .../processors/security/impl/TestSecurityData.java | 81 + .../security/impl/TestSecurityProcessor.java | 77
[ignite] branch IGNITE-17177_inc_snapshots updated (06ee012f2b4 -> 527bcd53872)
This is an automated email from the ASF dual-hosted git repository. timoninmaxim pushed a change to branch IGNITE-17177_inc_snapshots in repository https://gitbox.apache.org/repos/asf/ignite.git omit 06ee012f2b4 MINOR: fix misprints, make logs more useful add 527bcd53872 MINOR: fix misprints, make logs more useful This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (06ee012f2b4) \ N -- N -- N refs/heads/IGNITE-17177_inc_snapshots (527bcd53872) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. No new revisions were added by this update. Summary of changes: .../src/test/java/org/apache/ignite/util/GridCommandHandlerTest.java| 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[ignite] branch IGNITE-17177_inc_snapshots updated (2d8e8250a05 -> 06ee012f2b4)
This is an automated email from the ASF dual-hosted git repository. timoninmaxim pushed a change to branch IGNITE-17177_inc_snapshots in repository https://gitbox.apache.org/repos/asf/ignite.git omit 2d8e8250a05 MINOR: fix misprints, make logs more useful add 06ee012f2b4 MINOR: fix misprints, make logs more useful This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (2d8e8250a05) \ N -- N -- N refs/heads/IGNITE-17177_inc_snapshots (06ee012f2b4) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. No new revisions were added by this update. Summary of changes: .../ignite/testsuites/IncrementalSnapshotsTestSuite.java | 12 ++-- 1 file changed, 6 insertions(+), 6 deletions(-)
[ignite] branch IGNITE-17177_inc_snapshots updated (5af734f57ce -> 2d8e8250a05)
This is an automated email from the ASF dual-hosted git repository. timoninmaxim pushed a change to branch IGNITE-17177_inc_snapshots in repository https://gitbox.apache.org/repos/asf/ignite.git from 5af734f57ce IGNITE-18975 Add incremental snapshot restore option to control.sh (#10582) add 2d8e8250a05 MINOR: fix misprints, make logs more useful No new revisions were added by this update. Summary of changes: .../snapshot/SnapshotStatusCommand.java| 4 +- .../java/org/apache/ignite/IgniteSnapshot.java | 6 +-- .../systemview/walker/SnapshotViewWalker.java | 4 +- .../record/IncrementalSnapshotFinishRecord.java| 8 ++-- .../cache/distributed/near/GridNearTxLocal.java| 2 +- .../snapshot/IgniteSnapshotManager.java| 44 ++ .../snapshot/IncrementalSnapshotAwareMessage.java | 2 +- .../snapshot/IncrementalSnapshotFutureTask.java| 15 +++- .../IncrementalSnapshotFutureTaskResult.java | 22 --- .../snapshot/IncrementalSnapshotMarkWalFuture.java | 17 + .../snapshot/IncrementalSnapshotMetadata.java | 4 +- .../persistence/snapshot/SnapshotMXBeanImpl.java | 6 +-- .../snapshot/SnapshotMetadataVerificationTask.java | 12 +++--- .../SnapshotMetadataVerificationTaskArg.java | 2 +- .../snapshot/SnapshotRestoreProcess.java | 31 --- .../cache/transactions/IgniteInternalTx.java | 4 +- .../cache/transactions/IgniteTxAdapter.java| 8 ++-- .../cache/transactions/IgniteTxHandler.java| 8 ++-- .../visor/snapshot/VisorSnapshotStatusTask.java| 4 +- .../org/apache/ignite/mxbean/SnapshotMXBean.java | 2 +- .../ignite/spi/systemview/view/SnapshotView.java | 6 +-- .../ignite/internal/metric/SystemViewSelfTest.java | 4 +- .../snapshot/IgniteSnapshotMXBeanTest.java | 2 +- .../snapshot/IncrementalSnapshotMetricTest.java| 4 +- .../IncrementalSnapshotCheckBeforeRestoreTest.java | 10 ++--- .../testsuites/IncrementalSnapshotsTestSuite.java | 12 +++--- 26 files changed, 110 insertions(+), 133 deletions(-) delete mode 100644 modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/snapshot/IncrementalSnapshotFutureTaskResult.java
[ignite] branch IGNITE-17177_inc_snapshots updated (4206dec3834 -> 5af734f57ce)
This is an automated email from the ASF dual-hosted git repository. timoninmaxim pushed a change to branch IGNITE-17177_inc_snapshots in repository https://gitbox.apache.org/repos/asf/ignite.git from 4206dec3834 IGNITE-18967 Fix inc snapshot directories structure (#10580) add 5af734f57ce IGNITE-18975 Add incremental snapshot restore option to control.sh (#10582) No new revisions were added by this update. Summary of changes: .../snapshot/SnapshotRestoreCommand.java | 21 +- .../snapshot/SnapshotRestoreCommandOption.java | 4 + .../apache/ignite/util/GridCommandHandlerTest.java | 85 +- .../GridCacheDatabaseSharedManager.java| 2 +- .../snapshot/IgniteSnapshotManager.java| 4 +- .../visor/snapshot/VisorSnapshotRestoreTask.java | 2 +- .../snapshot/VisorSnapshotRestoreTaskArg.java | 13 ...ridCommandHandlerClusterByClassTest_help.output | 11 +-- ...andHandlerClusterByClassWithSSLTest_help.output | 11 +-- 9 files changed, 135 insertions(+), 18 deletions(-)
[ignite] branch IGNITE-17177_inc_snapshots updated (4670f1aaa04 -> 4206dec3834)
This is an automated email from the ASF dual-hosted git repository. timoninmaxim pushed a change to branch IGNITE-17177_inc_snapshots in repository https://gitbox.apache.org/repos/asf/ignite.git from 4670f1aaa04 MINOR: rename restoreIncrementalSnapshot method add 4206dec3834 IGNITE-18967 Fix inc snapshot directories structure (#10580) No new revisions were added by this update. Summary of changes: .../snapshot/IgniteSnapshotManager.java| 39 .../snapshot/IncrementalSnapshotFutureTask.java| 52 +++--- .../snapshot/IncrementalSnapshotMetadata.java | 10 +++-- .../snapshot/SnapshotMetadataVerificationTask.java | 12 ++--- .../snapshot/SnapshotRestoreProcess.java | 42 + .../snapshot/AbstractSnapshotSelfTest.java | 21 + .../snapshot/IncrementalSnapshotTest.java | 10 +++-- .../IncrementalSnapshotCheckBeforeRestoreTest.java | 35 --- .../IncrementalSnapshotRestoreTest.java| 24 +- 9 files changed, 121 insertions(+), 124 deletions(-)
[ignite] branch IGNITE-17177_inc_snapshots updated (196398dff1c -> 4670f1aaa04)
This is an automated email from the ASF dual-hosted git repository. timoninmaxim pushed a change to branch IGNITE-17177_inc_snapshots in repository https://gitbox.apache.org/repos/asf/ignite.git from 196398dff1c IGNITE-17649 Disable incremental snapshots creation after rebalance (#10562) add 4670f1aaa04 MINOR: rename restoreIncrementalSnapshot method No new revisions were added by this update. Summary of changes: .../apache/ignite/util/GridCommandHandlerTest.java | 2 +- .../java/org/apache/ignite/IgniteSnapshot.java | 5 ++- .../snapshot/IgniteSnapshotManager.java| 2 +- .../persistence/snapshot/SnapshotMXBeanImpl.java | 2 +- .../org/apache/ignite/mxbean/SnapshotMXBean.java | 7 ++-- .../snapshot/IgniteSnapshotMXBeanTest.java | 4 +- .../snapshot/IncrementalSnapshotMetricTest.java| 2 +- .../snapshot/IncrementalSnapshotRebalanceTest.java | 4 +- .../IncrementalSnapshotJoiningClientTest.java | 2 +- .../IncrementalSnapshotRestoreTest.java| 44 +++--- .../IncrementalSnapshotWarnAtomicCachesTest.java | 2 +- ...IncrementalSnapshotRestoreWithIndexingTest.java | 2 +- 12 files changed, 40 insertions(+), 38 deletions(-)
[ignite] branch IGNITE-17177_inc_snapshots updated (b509109eb6e -> 196398dff1c)
This is an automated email from the ASF dual-hosted git repository. timoninmaxim pushed a change to branch IGNITE-17177_inc_snapshots in repository https://gitbox.apache.org/repos/asf/ignite.git from b509109eb6e IGNITE-18069 Check incremental snapshot meta (#10364) add 196398dff1c IGNITE-17649 Disable incremental snapshots creation after rebalance (#10562) No new revisions were added by this update. Summary of changes: .../GridCacheDatabaseSharedManager.java| 2 + .../snapshot/IgniteSnapshotManager.java| 63 .../snapshot/SnapshotRestoreProcess.java | 3 + .../snapshot/IncrementalSnapshotRebalanceTest.java | 173 + .../testsuites/IncrementalSnapshotsTestSuite.java | 2 + 5 files changed, 243 insertions(+) create mode 100644 modules/core/src/test/java/org/apache/ignite/internal/processors/cache/persistence/snapshot/IncrementalSnapshotRebalanceTest.java
[ignite] branch master updated (1f238846c8b -> e220e8006e4)
This is an automated email from the ASF dual-hosted git repository.
timoninmaxim pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/ignite.git
from 1f238846c8b IGNITE-18936 [ducktests] Fix parsing of the control.sh
baseline command output and store JVM error file (#10575)
add e220e8006e4 IGNITE-18546 Add cache clear command to control.sh (#10577)
No new revisions were added by this update.
Summary of changes:
docs/_docs/tools/control-script.adoc | 24 +
.../internal/commandline/cache/CacheClear.java | 108 +
.../commandline/cache/CacheCommandList.java| 5 +
.../commandline/cache/CacheSubcommands.java| 7 +-
.../util/GridCommandHandlerClusterByClassTest.java | 93 ++
.../internal/processors/cache/ClearCachesTask.java | 81
.../cache/ClearCachesTaskArg.java} | 44 -
.../cache/ClearCachesTaskResult.java} | 53 +-
...mandHandlerClusterByClassTest_cache_help.output | 6 ++
...dlerClusterByClassWithSSLTest_cache_help.output | 6 ++
10 files changed, 373 insertions(+), 54 deletions(-)
create mode 100644
modules/control-utility/src/main/java/org/apache/ignite/internal/commandline/cache/CacheClear.java
create mode 100644
modules/core/src/main/java/org/apache/ignite/internal/processors/cache/ClearCachesTask.java
copy
modules/core/src/main/java/org/apache/ignite/internal/{visor/tx/VisorTxTaskResult.java
=> processors/cache/ClearCachesTaskArg.java} (70%)
copy
modules/core/src/main/java/org/apache/ignite/internal/{visor/tx/VisorTxTaskResult.java
=> processors/cache/ClearCachesTaskResult.java} (59%)
[ignite] branch IGNITE-17177_inc_snapshots updated (fbb4feb3561 -> b509109eb6e)
This is an automated email from the ASF dual-hosted git repository.
timoninmaxim pushed a change to branch IGNITE-17177_inc_snapshots
in repository https://gitbox.apache.org/repos/asf/ignite.git
from fbb4feb3561 IGNITE-18852 Add incremental snapshot metrics (#10569)
add b509109eb6e IGNITE-18069 Check incremental snapshot meta (#10364)
No new revisions were added by this update.
Summary of changes:
.../ignite/common/ComputeTaskPermissionsTest.java | 8 +-
.../snapshot/IgniteSnapshotManager.java| 42 ++-
.../snapshot/IncrementalSnapshotMetadata.java | 15 +
.../persistence/snapshot/SnapshotMetadata.java | 2 +
.../snapshot/SnapshotMetadataCollectorTask.java| 113 ---
.../snapshot/SnapshotMetadataVerificationTask.java | 228 +
...va => SnapshotMetadataVerificationTaskArg.java} | 23 +-
.../SnapshotMetadataVerificationTaskResult.java} | 58 ++--
.../snapshot/SnapshotRestoreProcess.java | 2 +-
.../main/resources/META-INF/classnames.properties | 6 +-
.../snapshot/IgniteClusterSnapshotCheckTest.java | 2 +-
.../IncrementalSnapshotCheckBeforeRestoreTest.java | 368 +
.../IncrementalSnapshotRestoreTest.java| 13 +-
.../testsuites/IncrementalSnapshotsTestSuite.java | 2 +
14 files changed, 710 insertions(+), 172 deletions(-)
delete mode 100644
modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/snapshot/SnapshotMetadataCollectorTask.java
create mode 100644
modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/snapshot/SnapshotMetadataVerificationTask.java
rename
modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/snapshot/{SnapshotMetadataCollectorTaskArg.java
=> SnapshotMetadataVerificationTaskArg.java} (78%)
copy
modules/core/src/main/java/org/apache/ignite/internal/{visor/diagnostic/availability/VisorConnectivityResult.java
=>
processors/cache/persistence/snapshot/SnapshotMetadataVerificationTaskResult.java}
(53%)
create mode 100644
modules/core/src/test/java/org/apache/ignite/internal/processors/cache/persistence/snapshot/incremental/IncrementalSnapshotCheckBeforeRestoreTest.java
[ignite] branch IGNITE-17177_inc_snapshots updated (ff557cc0233 -> fbb4feb3561)
This is an automated email from the ASF dual-hosted git repository. timoninmaxim pushed a change to branch IGNITE-17177_inc_snapshots in repository https://gitbox.apache.org/repos/asf/ignite.git from ff557cc0233 IGNITE-18934 Restore incremental snapshot with SnapshotMXBean (#10571) add fbb4feb3561 IGNITE-18852 Add incremental snapshot metrics (#10569) No new revisions were added by this update. Summary of changes: .../snapshot/SnapshotStatusCommand.java| 180 -- .../util/GridCommandHandlerAbstractTest.java | 13 ++ .../apache/ignite/util/GridCommandHandlerTest.java | 57 +- .../snapshot/IgniteSnapshotManager.java| 60 +- .../persistence/snapshot/SnapshotMXBeanImpl.java | 7 +- .../snapshot/SnapshotRestoreProcess.java | 54 +- .../visor/snapshot/VisorSnapshotStatusTask.java| 59 -- .../snapshot/IgniteSnapshotMXBeanTest.java | 57 +- .../snapshot/IncrementalSnapshotMetricTest.java| 210 + .../AbstractIncrementalSnapshotTest.java | 18 ++ .../IncrementalSnapshotRestoreTest.java| 17 -- .../testsuites/IncrementalSnapshotsTestSuite.java | 4 +- 12 files changed, 654 insertions(+), 82 deletions(-) create mode 100644 modules/core/src/test/java/org/apache/ignite/internal/processors/cache/persistence/snapshot/IncrementalSnapshotMetricTest.java
[ignite] branch IGNITE-17177_inc_snapshots updated (514f24f534d -> ff557cc0233)
This is an automated email from the ASF dual-hosted git repository. timoninmaxim pushed a change to branch IGNITE-17177_inc_snapshots in repository https://gitbox.apache.org/repos/asf/ignite.git from 514f24f534d IGNITE-18876 Add incremental snapshots to snapshot view (#10564) add ff557cc0233 IGNITE-18934 Restore incremental snapshot with SnapshotMXBean (#10571) No new revisions were added by this update. Summary of changes: .../persistence/snapshot/SnapshotMXBeanImpl.java | 11 +++ .../org/apache/ignite/mxbean/SnapshotMXBean.java | 21 .../snapshot/IgniteSnapshotMXBeanTest.java | 37 ++ 3 files changed, 69 insertions(+)
[ignite] branch IGNITE-17177_inc_snapshots updated (3dd7584ddc9 -> 514f24f534d)
This is an automated email from the ASF dual-hosted git repository. timoninmaxim pushed a change to branch IGNITE-17177_inc_snapshots in repository https://gitbox.apache.org/repos/asf/ignite.git from 3dd7584ddc9 Merge branch 'master' into IGNITE-17177_inc_snapshots add 514f24f534d IGNITE-18876 Add incremental snapshots to snapshot view (#10564) No new revisions were added by this update. Summary of changes: .../systemview/walker/SnapshotViewWalker.java | 6 ++- .../snapshot/IgniteSnapshotManager.java| 56 +- .../snapshot/IncrementalSnapshotMetadata.java | 16 +++ .../ignite/spi/systemview/view/SnapshotView.java | 48 +++ .../ignite/internal/metric/SystemViewSelfTest.java | 45 + 5 files changed, 136 insertions(+), 35 deletions(-)
[ignite] branch IGNITE-17177_inc_snapshots updated (75fc6e4bc42 -> 3dd7584ddc9)
This is an automated email from the ASF dual-hosted git repository. timoninmaxim pushed a change to branch IGNITE-17177_inc_snapshots in repository https://gitbox.apache.org/repos/asf/ignite.git from 75fc6e4bc42 IGNITE-18109 ConsistentCut ignores joining nodes (#10368) add 075a76a144f IGNITE-18826 Java thin client: fixed the pending requests race on close (#10549) add 3dd7584ddc9 Merge branch 'master' into IGNITE-17177_inc_snapshots No new revisions were added by this update. Summary of changes: .../internal/client/thin/TcpClientChannel.java | 53 -- .../GridNioClientConnectionMultiplexer.java| 4 +- ...ientPartitionAwarenessUnstableTopologyTest.java | 117 + 3 files changed, 162 insertions(+), 12 deletions(-)
[ignite] branch IGNITE-17177_inc_snapshots updated (1e067eb058f -> 75fc6e4bc42)
This is an automated email from the ASF dual-hosted git repository. timoninmaxim pushed a change to branch IGNITE-17177_inc_snapshots in repository https://gitbox.apache.org/repos/asf/ignite.git from 1e067eb058f IGNITE-17650 Write a warning message for atomic caches (#10552) add 75fc6e4bc42 IGNITE-18109 ConsistentCut ignores joining nodes (#10368) No new revisions were added by this update. Summary of changes: .../snapshot/IgniteSnapshotManager.java| 10 +- .../snapshot/IncrementalSnapshotAwareMessage.java | 47 +++- .../snapshot/IncrementalSnapshotMarkWalFuture.java | 17 +- .../cache/transactions/IgniteTxHandler.java| 7 +- .../AbstractIncrementalSnapshotBlockingTest.java | 4 +- .../IncrementalSnapshotJoiningClientTest.java | 274 + .../IncrementalSnapshotRestoreTest.java| 4 +- .../testsuites/IncrementalSnapshotsTestSuite.java | 2 + 8 files changed, 345 insertions(+), 20 deletions(-) create mode 100644 modules/core/src/test/java/org/apache/ignite/internal/processors/cache/persistence/snapshot/incremental/IncrementalSnapshotJoiningClientTest.java
[ignite] branch IGNITE-17177_inc_snapshots updated (c4a093e84e4 -> 1e067eb058f)
This is an automated email from the ASF dual-hosted git repository. timoninmaxim pushed a change to branch IGNITE-17177_inc_snapshots in repository https://gitbox.apache.org/repos/asf/ignite.git from c4a093e84e4 IGNITE-18837 CacheStripedExecutor should handle execution error (#10551) add 1e067eb058f IGNITE-17650 Write a warning message for atomic caches (#10552) No new revisions were added by this update. Summary of changes: .../snapshot/IgniteSnapshotManager.java| 32 +++ .../snapshot/SnapshotRestoreProcess.java | 9 + .../IncrementalSnapshotWarnAtomicCachesTest.java | 253 + .../testsuites/IncrementalSnapshotsTestSuite.java | 4 +- 4 files changed, 297 insertions(+), 1 deletion(-) create mode 100644 modules/core/src/test/java/org/apache/ignite/internal/processors/cache/persistence/snapshot/incremental/IncrementalSnapshotWarnAtomicCachesTest.java
[ignite] branch IGNITE-17177_inc_snapshots updated (e7a34f5c1ab -> c4a093e84e4)
This is an automated email from the ASF dual-hosted git repository. timoninmaxim pushed a change to branch IGNITE-17177_inc_snapshots in repository https://gitbox.apache.org/repos/asf/ignite.git from e7a34f5c1ab IGNITE-18720 Fix multiple segments handling during IS restore (#10538) add c4a093e84e4 IGNITE-18837 CacheStripedExecutor should handle execution error (#10551) No new revisions were added by this update. Summary of changes: .../cache/persistence/CacheStripedExecutor.java| 3 +++ .../snapshot/SnapshotRestoreProcess.java | 9 +++- .../AbstractIncrementalSnapshotTest.java | 13 +++ .../IncrementalSnapshotRestoreTest.java| 26 -- ...IncrementalSnapshotRestoreWithIndexingTest.java | 2 +- 5 files changed, 39 insertions(+), 14 deletions(-)
[ignite] branch IGNITE-17177_inc_snapshots updated (d9a4c2857a2 -> e7a34f5c1ab)
This is an automated email from the ASF dual-hosted git repository. timoninmaxim pushed a change to branch IGNITE-17177_inc_snapshots in repository https://gitbox.apache.org/repos/asf/ignite.git from d9a4c2857a2 Merge branch 'master' into IGNITE-17177_inc_snapshots add e7a34f5c1ab IGNITE-18720 Fix multiple segments handling during IS restore (#10538) No new revisions were added by this update. Summary of changes: .../org/apache/ignite/util/CdcCommandTest.java | 2 +- .../org/apache/ignite/internal/cdc/CdcMain.java| 14 +-- .../snapshot/SnapshotRestoreProcess.java | 10 +- .../persistence/wal/FileWriteAheadLogManager.java | 11 ++ .../visor/cdc/VisorCdcDeleteLostSegmentsTask.java | 5 +- .../IncrementalSnapshotRestoreTest.java| 127 ++--- 6 files changed, 137 insertions(+), 32 deletions(-)
[ignite] branch IGNITE-17177_inc_snapshots updated (ea61edf3511 -> d9a4c2857a2)
This is an automated email from the ASF dual-hosted git repository. timoninmaxim pushed a change to branch IGNITE-17177_inc_snapshots in repository https://gitbox.apache.org/repos/asf/ignite.git from ea61edf3511 Merge remote-tracking branch 'upstream/master' into IGNITE-17177_inc_snapshots add 4284a50b39a IGNITE-18825 Fixed OpenCensusSqlNativeTracingTest#testDistributedJoin test. (#10546) add 4496972ef87 IGNITE-17016 Fixed an issue that could lead to skipping 3rd party cache store while executing REST API GET command. Fixes #10525 add 904eeb769b6 IGNITE-18721 Disable WAL for index partition during full rebuild (#10533) add 5d7e5d9d6ba IGNITE-15322 Stop authorization of internal tasks by their class name. (#10226) add d9a4c2857a2 Merge branch 'master' into IGNITE-17177_inc_snapshots No new revisions were added by this update. Summary of changes: .../common/AbstractEventSecurityContextTest.java | 2 +- ...CacheCreateDestroyEventSecurityContextTest.java | 4 +- .../ignite/common/ComputeTaskPermissionsTest.java | 809 + .../client/suite/IgniteClientTestSuite.java| 4 +- .../rest/JettyRestProcessorAbstractSelfTest.java | 126 .../apache/ignite/jdbc/JdbcAuthorizationTest.java | 3 - .../JdbcThinConnectionAdditionalSecurityTest.java | 4 +- .../GridCommandHandlerSslWithSecurityTest.java | 4 +- .../apache/ignite/internal/ComputeMXBeanImpl.java | 15 +- .../ignite/internal/ComputeTaskInternalFuture.java | 12 +- .../ignite/internal/GridTaskSessionImpl.java | 20 +- .../ignite/internal/IgniteComputeHandler.java | 14 + .../apache/ignite/internal/IgniteComputeImpl.java | 6 +- .../org/apache/ignite/internal/IgnitionEx.java | 15 +- .../apache/ignite/internal/QueryMXBeanImpl.java| 26 +- .../ignite/internal/TransactionsMXBeanImpl.java| 29 +- .../sorted/maintenance/RebuildIndexAction.java | 2 +- .../ignite/internal/cluster/IgniteClusterImpl.java | 23 +- .../commandline/meta/tasks/MetadataRemoveTask.java | 28 +- .../commandline/meta/tasks/MetadataUpdateTask.java | 12 +- .../property/tasks/PropertiesListTask.java | 9 +- .../commandline/property/tasks/PropertyTask.java | 15 +- .../internal/executor/GridExecutorService.java | 2 +- .../ignite/internal/jdbc2/JdbcConnection.java | 11 +- .../ignite/internal/jdbc2/JdbcResultSet.java | 31 +- .../ignite/internal/jdbc2/JdbcStatement.java | 37 +- .../managers/indexing/IndexesRebuildTask.java | 43 +- .../pagemem/store/IgnitePageStoreManager.java | 4 +- .../pagemem/wal/IgniteWriteAheadLogManager.java| 3 +- .../IgniteAuthenticationProcessor.java | 4 +- .../processors/cache/CacheGroupContext.java| 15 + .../cache/GridCachePartitionExchangeManager.java | 21 +- .../processors/cache/GridCacheProxyImpl.java | 2 +- .../internal/processors/cache/WalStateManager.java | 8 +- .../persistence/file/FilePageStoreManager.java | 46 +- .../cache/persistence/pagemem/PageMemoryImpl.java | 4 +- .../snapshot/IgniteSnapshotManager.java| 18 +- .../cache/persistence/tree/util/PageHandler.java | 2 +- .../persistence/wal/FileWriteAheadLogManager.java | 4 +- .../cluster/GridClusterStateProcessor.java | 17 +- .../internal/processors/job/GridJobProcessor.java | 2 +- .../platform/client/compute/ClientComputeTask.java | 1 + .../platform/compute/PlatformCompute.java | 2 +- ...latformDotNetEntityFrameworkCacheExtension.java | 17 +- .../processors/query/GridQueryProcessor.java | 21 +- .../query/aware/IndexBuildStatusHolder.java| 18 +- .../query/aware/IndexBuildStatusStorage.java | 38 +- .../query/aware/IndexRebuildCacheInfo.java | 23 +- .../management/SortedIndexDescriptorFactory.java | 5 +- .../processors/rest/GridRestProcessor.java | 2 +- .../rest/handlers/task/GridTaskCommandHandler.java | 2 +- .../processors/security/PublicAccessJob.java} | 19 +- .../processors/security/SecurityUtils.java | 43 +- .../processors/service/GridServiceProxy.java | 1 - .../session/GridTaskSessionProcessor.java | 9 +- .../processors/task/GridTaskProcessor.java | 74 +- .../internal/processors/task/GridTaskWorker.java | 78 +- .../processors/task/TaskExecutionOptions.java | 18 +- .../internal/util/lang/gridfunc/NoOpClosure.java | 11 +- .../org/apache/ignite/internal/visor/VisorJob.java | 11 +- .../visor/snapshot/VisorSnapshotCancelTask.java| 2 +- .../visor/snapshot/VisorSnapshotCheckTask.java | 2 +- .../visor/snapshot/VisorSnapshotCreateTask.java| 2 +- .../snapshot/VisorSnapshotJob.java}| 21 +- .../visor/snapshot/VisorSnapshotRestoreTask.java | 6 +- .../visor/snapshot/VisorSnapshotStatusTask.java| 2 +- .../internal/visor/verify/VisorIdleVerifyJob.java | 30
[ignite] branch IGNITE-17177_inc_snapshots updated (b8b4d802c65 -> ea61edf3511)
This is an automated email from the ASF dual-hosted git repository. timoninmaxim pushed a change to branch IGNITE-17177_inc_snapshots in repository https://gitbox.apache.org/repos/asf/ignite.git from b8b4d802c65 Merge branch 'master' into IGNITE-17177_inc_snapshots add ab278862bd3 IGNITE-18783 Fix DistributedProcess hangs on AssertionError (#10537) add ea61edf3511 Merge remote-tracking branch 'upstream/master' into IGNITE-17177_inc_snapshots No new revisions were added by this update. Summary of changes: .../managers/encryption/GridEncryptionManager.java | 8 +- .../managers/encryption/GroupKeyChangeProcess.java | 6 +- .../snapshot/IgniteSnapshotManager.java| 4 +- .../snapshot/SnapshotRestoreProcess.java | 20 +-- .../util/distributed/DistributedProcess.java | 55 --- .../internal/util/distributed/FullMessage.java | 6 +- .../util/distributed/SingleNodeMessage.java| 6 +- .../util/DistributedProcessErrorHandlingTest.java | 169 + .../ignite/testsuites/IgniteUtilSelfTestSuite.java | 2 + 9 files changed, 231 insertions(+), 45 deletions(-) create mode 100644 modules/core/src/test/java/org/apache/ignite/internal/util/DistributedProcessErrorHandlingTest.java
[ignite] branch master updated: IGNITE-18783 Fix DistributedProcess hangs on AssertionError (#10537)
This is an automated email from the ASF dual-hosted git repository.
timoninmaxim pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/ignite.git
The following commit(s) were added to refs/heads/master by this push:
new ab278862bd3 IGNITE-18783 Fix DistributedProcess hangs on
AssertionError (#10537)
ab278862bd3 is described below
commit ab278862bd322e0d2d76557c263c6c1a7b97375c
Author: Maksim Timonin
AuthorDate: Thu Feb 16 11:43:38 2023 +0300
IGNITE-18783 Fix DistributedProcess hangs on AssertionError (#10537)
---
.../managers/encryption/GridEncryptionManager.java | 8 +-
.../managers/encryption/GroupKeyChangeProcess.java | 6 +-
.../snapshot/IgniteSnapshotManager.java| 4 +-
.../snapshot/SnapshotRestoreProcess.java | 14 +-
.../util/distributed/DistributedProcess.java | 55 ---
.../internal/util/distributed/FullMessage.java | 6 +-
.../util/distributed/SingleNodeMessage.java| 6 +-
.../util/DistributedProcessErrorHandlingTest.java | 169 +
.../ignite/testsuites/IgniteUtilSelfTestSuite.java | 2 +
9 files changed, 228 insertions(+), 42 deletions(-)
diff --git
a/modules/core/src/main/java/org/apache/ignite/internal/managers/encryption/GridEncryptionManager.java
b/modules/core/src/main/java/org/apache/ignite/internal/managers/encryption/GridEncryptionManager.java
index 8e1f5be7b7e..f1f5f11c553 100644
---
a/modules/core/src/main/java/org/apache/ignite/internal/managers/encryption/GridEncryptionManager.java
+++
b/modules/core/src/main/java/org/apache/ignite/internal/managers/encryption/GridEncryptionManager.java
@@ -1568,7 +1568,7 @@ public class GridEncryptionManager extends
GridManagerAdapter imp
* @param res Results.
* @param err Errors.
*/
-private void finishPrepareMasterKeyChange(UUID id, Map
res, Map err) {
+private void finishPrepareMasterKeyChange(UUID id, Map
res, Map err) {
if (!err.isEmpty()) {
if (masterKeyChangeRequest != null &&
masterKeyChangeRequest.requestId().equals(id))
masterKeyChangeRequest = null;
@@ -1613,7 +1613,7 @@ public class GridEncryptionManager extends
GridManagerAdapter imp
* @param res Results.
* @param err Errors.
*/
-private void finishPerformMasterKeyChange(UUID id, Map
res, Map err) {
+private void finishPerformMasterKeyChange(UUID id, Map
res, Map err) {
completeMasterKeyChangeFuture(id, err);
}
@@ -1621,7 +1621,7 @@ public class GridEncryptionManager extends
GridManagerAdapter imp
* @param reqId Request id.
* @param err Exception.
*/
-private void completeMasterKeyChangeFuture(UUID reqId, Map err) {
+private void completeMasterKeyChangeFuture(UUID reqId, Map err) {
synchronized (opsMux) {
boolean isInitiator = masterKeyChangeFut != null &&
masterKeyChangeFut.id().equals(reqId);
@@ -1629,7 +1629,7 @@ public class GridEncryptionManager extends
GridManagerAdapter imp
return;
if (!F.isEmpty(err)) {
-Exception e = err.values().stream().findFirst().get();
+Throwable e = err.values().stream().findFirst().get();
masterKeyChangeFut.onDone(e);
}
diff --git
a/modules/core/src/main/java/org/apache/ignite/internal/managers/encryption/GroupKeyChangeProcess.java
b/modules/core/src/main/java/org/apache/ignite/internal/managers/encryption/GroupKeyChangeProcess.java
index 855d34e22d4..a8b63bce215 100644
---
a/modules/core/src/main/java/org/apache/ignite/internal/managers/encryption/GroupKeyChangeProcess.java
+++
b/modules/core/src/main/java/org/apache/ignite/internal/managers/encryption/GroupKeyChangeProcess.java
@@ -273,7 +273,7 @@ class GroupKeyChangeProcess {
* @param res Results.
* @param err Errors.
*/
-private void finishPrepare(UUID id, Map res, Map err) {
+private void finishPrepare(UUID id, Map res, Map err) {
if (!err.isEmpty()) {
if (req != null && req.requestId().equals(id))
req = null;
@@ -318,7 +318,7 @@ class GroupKeyChangeProcess {
* @param res Results.
* @param err Errors.
*/
-private void finishPerform(UUID id, Map res, Map err) {
+private void finishPerform(UUID id, Map res, Map err) {
completeFuture(id, err, fut);
}
@@ -328,7 +328,7 @@ class GroupKeyChangeProcess {
* @param fut Key change future.
* @return {@code True} if future was completed by this call.
*/
-private boolean completeFuture(UUID reqId, Map err,
GroupKeyChangeFuture fut) {
+private boolean completeFuture(UUID reqId, Map err,
GroupKeyChangeFuture fut) {
boolean isInitiator = fut != null && fut.id().equals(reqId);
if (!isInitiator || fut.isDone())
diff --git
a/modules/core/src/main/java/org/apache/ignite/internal/proce
[ignite] branch IGNITE-17177_inc_snapshots updated (4e15c1c8674 -> b8b4d802c65)
This is an automated email from the ASF dual-hosted git repository. timoninmaxim pushed a change to branch IGNITE-17177_inc_snapshots in repository https://gitbox.apache.org/repos/asf/ignite.git from 4e15c1c8674 IGNITE-18150 Incremental snapshot restores binary and marshaller meta (#10536) add 7ea953999ae IGNITE-18750 .NET: Fix task options not being reset after task execution (#10523) add 9a8241951cb IGNITE-18719 Deprecate SqlFieldsQuery#lazy and set default to true. (#10518) add f22ceb2d336 IGNITE-17368 Remove obsolete GridClient JDBC driver (#10473) add b8b4d802c65 Merge branch 'master' into IGNITE-17177_inc_snapshots No new revisions were added by this update. Summary of changes: .../common/RunningQueryInfoCheckInitiatorTest.java |4 +- .../internal/jdbc2/JdbcConnectionSelfTest.java | 13 +- .../internal/jdbc2/JdbcResultSetSelfTest.java | 63 +- .../ignite/jdbc/JdbcComplexQuerySelfTest.java |2 +- .../apache/ignite/jdbc/JdbcConnectionSelfTest.java | 211 --- .../ignite/jdbc/JdbcDefaultNoOpCacheTest.java | 35 - .../apache/ignite/jdbc/JdbcEmptyCacheSelfTest.java | 125 -- .../apache/ignite/jdbc/JdbcMetadataSelfTest.java | 419 -- .../apache/ignite/jdbc/JdbcNoDefaultCacheTest.java | 154 -- .../ignite/jdbc/JdbcPojoLegacyQuerySelfTest.java | 46 - .../ignite/jdbc/JdbcPreparedStatementSelfTest.java | 779 -- .../apache/ignite/jdbc/JdbcResultSetSelfTest.java | 803 -- .../apache/ignite/jdbc/JdbcStatementSelfTest.java | 277 .../jdbc/suite/IgniteJdbcDriverTestSuite.java | 18 - .../jdbc/thin/JdbcThinConnectionSelfTest.java | 39 +- .../jdbc/thin/JdbcThinDataSourceSelfTest.java | 13 +- .../qa/query/WarningOnBigQueryResultsTest.java |4 +- .../java/org/apache/ignite/IgniteJdbcDriver.java | 277 +--- .../org/apache/ignite/IgniteJdbcThinDriver.java|2 +- .../apache/ignite/cache/query/SqlFieldsQuery.java | 15 +- .../ignite/internal/IgniteComputeHandler.java | 695 + .../apache/ignite/internal/IgniteComputeImpl.java | 668 + .../ignite/internal/jdbc/JdbcConnection.java | 603 .../jdbc/JdbcConnectionValidationTask.java | 54 - .../ignite/internal/jdbc/JdbcDatabaseMetadata.java | 1327 - .../internal/jdbc/JdbcPreparedStatement.java | 426 -- .../apache/ignite/internal/jdbc/JdbcResultSet.java | 1548 .../internal/jdbc/JdbcResultSetMetadata.java | 175 --- .../apache/ignite/internal/jdbc/JdbcStatement.java | 464 -- .../org/apache/ignite/internal/jdbc/JdbcUtils.java | 257 .../jdbc/thin/ConnectionPropertiesImpl.java|3 +- .../ignite/internal/jdbc2/JdbcConnection.java |3 +- .../query/jdbc/GridCacheQueryJdbcMetadataTask.java | 186 --- .../cache/query/jdbc/GridCacheQueryJdbcTask.java | 459 -- .../jdbc/GridCacheQueryJdbcValidationTask.java | 67 - .../processors/closure/GridClosureProcessor.java | 58 + .../platform/compute/PlatformCompute.java | 128 +- .../platform/compute/PlatformFullTask.java | 14 +- .../platform/utils/PlatformFutureUtils.java| 11 + .../main/resources/META-INF/classnames.properties | 23 +- .../IgniteBinaryCacheQueryTestSuite.java |8 + .../IgniteBinaryCacheQueryTestSuite2.java |8 + .../IgniteBinaryCacheQueryTestSuite3.java |8 + .../IgniteBinaryCacheQueryTestSuite4.java |8 + .../Compute/ComputeApiTest.cs | 14 + .../yardstick/config/benchmark-query.properties|1 - 46 files changed, 1078 insertions(+), 9437 deletions(-) delete mode 100644 modules/clients/src/test/java/org/apache/ignite/jdbc/JdbcConnectionSelfTest.java delete mode 100644 modules/clients/src/test/java/org/apache/ignite/jdbc/JdbcDefaultNoOpCacheTest.java delete mode 100644 modules/clients/src/test/java/org/apache/ignite/jdbc/JdbcEmptyCacheSelfTest.java delete mode 100644 modules/clients/src/test/java/org/apache/ignite/jdbc/JdbcMetadataSelfTest.java delete mode 100644 modules/clients/src/test/java/org/apache/ignite/jdbc/JdbcNoDefaultCacheTest.java delete mode 100644 modules/clients/src/test/java/org/apache/ignite/jdbc/JdbcPojoLegacyQuerySelfTest.java delete mode 100644 modules/clients/src/test/java/org/apache/ignite/jdbc/JdbcPreparedStatementSelfTest.java delete mode 100644 modules/clients/src/test/java/org/apache/ignite/jdbc/JdbcResultSetSelfTest.java delete mode 100644 modules/clients/src/test/java/org/apache/ignite/jdbc/JdbcStatementSelfTest.java create mode 100644 modules/core/src/main/java/org/apache/ignite/internal/IgniteComputeHandler.java delete mode 100644 modules/core/src/main/java/org/apache/ignite/internal/jdbc/JdbcConnection.java delete mode 100644 modules/core/src/main/java/org/apache/ignite/internal/jdbc/JdbcConnectionValidationTask.java delete mode 100644 modules/core/src
[ignite] branch IGNITE-17177_inc_snapshots updated (2ebaedc764e -> 4e15c1c8674)
This is an automated email from the ASF dual-hosted git repository. timoninmaxim pushed a change to branch IGNITE-17177_inc_snapshots in repository https://gitbox.apache.org/repos/asf/ignite.git from 2ebaedc764e Merge remote-tracking branch 'upstream/master' into IGNITE-17177_inc_snapshots add 4e15c1c8674 IGNITE-18150 Incremental snapshot restores binary and marshaller meta (#10536) No new revisions were added by this update. Summary of changes: .../snapshot/IncrementalSnapshotFutureTask.java| 16 ++- .../snapshot/SnapshotRestoreProcess.java | 21 +++- .../IncrementalSnapshotRestoreTest.java| 121 + 3 files changed, 150 insertions(+), 8 deletions(-)
[ignite-extensions] branch master updated: IGNITE-18803 Fix compilation error of ignite-hibernate-ext
This is an automated email from the ASF dual-hosted git repository.
timoninmaxim pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/ignite-extensions.git
The following commit(s) were added to refs/heads/master by this push:
new bfe45be5 IGNITE-18803 Fix compilation error of ignite-hibernate-ext
new dd5d4d10 Merge pull request #203 from nao-it/IGNITE-18803
bfe45be5 is described below
commit bfe45be585efbbd12bb398417298279bdbf971e1
Author: Aleksandr Nikolaev
AuthorDate: Tue Feb 14 19:27:02 2023 +0300
IGNITE-18803 Fix compilation error of ignite-hibernate-ext
---
.../apache/ignite/cache/hibernate/HibernateCacheProxy.java| 11 ---
1 file changed, 11 deletions(-)
diff --git
a/modules/hibernate-ext/hibernate/src/main/java/org/apache/ignite/cache/hibernate/HibernateCacheProxy.java
b/modules/hibernate-ext/hibernate/src/main/java/org/apache/ignite/cache/hibernate/HibernateCacheProxy.java
index 15b7fc48..1da411bf 100644
---
a/modules/hibernate-ext/hibernate/src/main/java/org/apache/ignite/cache/hibernate/HibernateCacheProxy.java
+++
b/modules/hibernate-ext/hibernate/src/main/java/org/apache/ignite/cache/hibernate/HibernateCacheProxy.java
@@ -41,7 +41,6 @@ import
org.apache.ignite.internal.processors.cache.GridCacheContext;
import org.apache.ignite.internal.processors.cache.IgniteInternalCache;
import
org.apache.ignite.internal.processors.cache.distributed.near.GridNearTxLocal;
import org.apache.ignite.lang.IgniteBiPredicate;
-import org.apache.ignite.mxbean.CacheMetricsMXBean;
import org.apache.ignite.transactions.Transaction;
import org.apache.ignite.transactions.TransactionConcurrency;
import org.apache.ignite.transactions.TransactionIsolation;
@@ -545,16 +544,6 @@ public class HibernateCacheProxy implements
IgniteInternalCache
return delegate.get().localMetrics();
}
-/** {@inheritDoc} */
-@Override public CacheMetricsMXBean clusterMxBean() {
-return delegate.get().clusterMxBean();
-}
-
-/** {@inheritDoc} */
-@Override public CacheMetricsMXBean localMxBean() {
-return delegate.get().localMxBean();
-}
-
/** {@inheritDoc} */
@Override public long offHeapEntriesCount() {
return delegate.get().offHeapEntriesCount();
[ignite] branch IGNITE-17177_inc_snapshots updated (fa158adcca7 -> 2ebaedc764e)
This is an automated email from the ASF dual-hosted git repository. timoninmaxim pushed a change to branch IGNITE-17177_inc_snapshots in repository https://gitbox.apache.org/repos/asf/ignite.git omit fa158adcca7 Merge remote-tracking branch 'upstream/master' into IGNITE-17177_inc_snapshots add 2ebaedc764e Merge remote-tracking branch 'upstream/master' into IGNITE-17177_inc_snapshots This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (fa158adcca7) \ N -- N -- N refs/heads/IGNITE-17177_inc_snapshots (2ebaedc764e) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. No new revisions were added by this update. Summary of changes: .../incremental/AbstractIncrementalSnapshotTest.java | 18 +- 1 file changed, 1 insertion(+), 17 deletions(-)
[ignite] branch IGNITE-17177_inc_snapshots updated (6cd8e90a919 -> fa158adcca7)
This is an automated email from the ASF dual-hosted git repository. timoninmaxim pushed a change to branch IGNITE-17177_inc_snapshots in repository https://gitbox.apache.org/repos/asf/ignite.git from 6cd8e90a919 Merge branch 'master' into IGNITE-17177_inc_snapshots add ca60ad5abbd IGNITE-18765 [ducktests] Use PEP 440 compatible version number for python module (#10530) add 1b537d444be IGNITE-18615 Implement monitoring events in java thin client (#10501) add 714ae9cc213 IGNITE-18591 Java thin client: Endpoints discovery - Fixes #10514. add 18953681dfd IGNITE-18769 Fix CacheMetricsForClusterGroupSelfTest.testMetricsStatisticsEnabled failure (#10531) add 7d4d6db1470 IGNITE-18690 Fixed assertion on paired connections (#10526) add df5f77e656c IGNITE-18779 Fix typo in .NET doc (#10534) add bbcc563ccee IGNITE-18780 Remove note in docs about Services being unsupported in mixed-platform clusters (#10535) add 601a4cc0458 IGNITE-18764 Restore marshaller data from snapshot (#10527) add ecc34e4e018 IGNITE-18590 Fix unexpected rollback transaction on tx recovery (#10521) add fa158adcca7 Merge remote-tracking branch 'upstream/master' into IGNITE-17177_inc_snapshots No new revisions were added by this update. Summary of changes: .../net-specific/net-java-services-execution.adoc | 2 +- .../net-platform-interoperability.adoc | 1 - .../ComputeTaskRemoteSecurityContextTest.java | 2 +- .../ignite/common/NodeSslConnectionMetricTest.java | 2 + .../snapshot/SnapshotCompressionBasicTest.java | 2 +- .../util/GridCommandHandlerMetadataTest.java | 3 +- .../events/ConnectionClosedEvent.java} | 37 +- .../client/events/ConnectionDescription.java | 82 .../ignite/client/events/ConnectionEvent.java} | 27 +- .../events/ConnectionEventListener.java} | 40 +- .../ignite/client/events/HandshakeFailEvent.java | 64 +++ .../ignite/client/events/HandshakeStartEvent.java} | 25 +- .../events/HandshakeSuccessEvent.java} | 38 +- .../apache/ignite/client/events/RequestEvent.java | 79 .../client/events/RequestEventListener.java} | 38 +- .../ignite/client/events/RequestFailEvent.java | 70 +++ .../ignite/client/events/RequestStartEvent.java} | 26 +- .../ignite/client/events/RequestSuccessEvent.java | 55 +++ .../ignite/configuration/ClientConfiguration.java | 21 + .../monitoring/EventListenerDemultiplexer.java | 222 ++ .../client/thin/ClientCacheAffinityContext.java| 3 +- .../client/thin/ClientChannelConfiguration.java| 22 +- .../client/thin/ClientDiscoveryContext.java| 303 + .../internal/client/thin/ClientOperation.java | 6 + .../client/thin/ProtocolBitmaskFeature.java| 3 + .../internal/client/thin/ProtocolContext.java | 17 + .../internal/client/thin/ProtocolVersion.java | 2 +- .../internal/client/thin/ReliableChannel.java | 474 ++--- .../internal/client/thin/TcpClientChannel.java | 335 ++- .../internal/client/thin/io/ClientConnection.java | 15 + .../io/gridnioserver/GridNioClientConnection.java | 11 + .../thin/io/gridnioserver/GridNioClientParser.java | 2 +- .../snapshot/SnapshotRestoreProcess.java | 34 ++ .../cache/transactions/IgniteTxManager.java| 22 +- .../tcp/internal/ConnectionClientPool.java | 6 +- .../tcp/internal/GridNioServerWrapper.java | 10 +- .../tcp/internal/InboundConnectionHandler.java | 13 +- .../org/apache/ignite/client/ReliabilityTest.java | 104 +++-- .../client/thin/AbstractThinClientTest.java| 42 +- .../client/thin/CacheEntryListenersTest.java | 34 ++ .../internal/client/thin/ComputeTaskTest.java | 7 + .../internal/client/thin/ReliableChannelTest.java | 9 +- .../ThinClientAbstractPartitionAwarenessTest.java | 23 +- .../thin/ThinClientEnpointsDiscoveryTest.java | 110 + ...ClientPartitionAwarenessStableTopologyTest.java | 13 + .../client/thin/events/FakeIgniteServer.java | 261 .../IgniteClientConnectionEventListenerTest.java | 233 ++ .../IgniteClientRequestEventListenerTest.java | 162 +++ .../cache/CacheMetricsForClusterGroupSelfTest.java | 104 ++--- .../dht/TxRecoveryCommitMessagesTest.java | 128 ++ .../IgniteClusterSnapshotRestoreSelfTest.java | 57 +++ .../IgniteSnapshotRestoreFromRemoteTest.java | 50 ++- .../processors/odbc/ClientListenerMetricsTest.java | 8 +- .../PerformanceStatisticsThinClientTest.java | 4 +- .../junits/common/GridCommonAbstractTest.java | 11 +- .../IgniteCacheTxRecoverySelfTestSuite.java| 5 +- .../ducktests/tests/ignitetest/utils/version.py| 4 +- modules/ducktests/tests/setup.py | 2 +- .../org/apache/ignite/client/ClientTestSuite.java | 11 +- .../cache
[ignite] branch master updated: IGNITE-18590 Fix unexpected rollback transaction on tx recovery (#10521)
This is an automated email from the ASF dual-hosted git repository.
timoninmaxim pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/ignite.git
The following commit(s) were added to refs/heads/master by this push:
new ecc34e4e018 IGNITE-18590 Fix unexpected rollback transaction on tx
recovery (#10521)
ecc34e4e018 is described below
commit ecc34e4e0184e82e52c408d8f7a017fa82951d3c
Author: Maksim Timonin
AuthorDate: Mon Feb 13 15:13:35 2023 +0300
IGNITE-18590 Fix unexpected rollback transaction on tx recovery (#10521)
---
.../cache/transactions/IgniteTxManager.java| 22 ++--
.../dht/TxRecoveryCommitMessagesTest.java | 128 +
.../IgniteCacheTxRecoverySelfTestSuite.java| 5 +-
3 files changed, 144 insertions(+), 11 deletions(-)
diff --git
a/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/transactions/IgniteTxManager.java
b/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/transactions/IgniteTxManager.java
index 163945f8e9c..be8636e1768 100644
---
a/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/transactions/IgniteTxManager.java
+++
b/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/transactions/IgniteTxManager.java
@@ -2452,18 +2452,20 @@ public class IgniteTxManager extends
GridCacheSharedManagerAdapter {
if (commit)
tx.commitAsync().listen(new CommitListener(tx));
-else if (!tx.local()) {
-// This tx was rolled back on recovery because of primary node
fail, other backups may be not aware of it.
-TxCounters cnts = tx.txCounters(false);
+else {
+if (!tx.local()) {
+// This tx was rolled back on recovery because of primary node
fail, other backups may be not aware of it.
+TxCounters cnts = tx.txCounters(false);
+
+if (cnts != null)
+// Skipping counters update to keep them the same
everywhere without any sync.
+// Tx counters will be finalized (gaps removed) on local
txs recovery finish.
+// Each node will have counters equals to latest
successful transactions counters.
+cnts.updateCounters().clear();
+}
-if (cnts != null)
-// Skipping counters update to keep them the same everywhere
without any sync.
-// Tx counters will be finalized (gaps removed) on local txs
recovery finish.
-// Each node will have counters equals to latest successful
transactions counters.
-cnts.updateCounters().clear();
+tx.rollbackAsync();
}
-
-tx.rollbackAsync();
}
/**
diff --git
a/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/distributed/dht/TxRecoveryCommitMessagesTest.java
b/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/distributed/dht/TxRecoveryCommitMessagesTest.java
new file mode 100644
index 000..b33d987164c
--- /dev/null
+++
b/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/distributed/dht/TxRecoveryCommitMessagesTest.java
@@ -0,0 +1,128 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.processors.cache.distributed.dht;
+
+import java.util.List;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.stream.Collectors;
+import org.apache.ignite.Ignite;
+import org.apache.ignite.cache.CacheWriteSynchronizationMode;
+import org.apache.ignite.cache.affinity.Affinity;
+import org.apache.ignite.configuration.CacheConfiguration;
+import org.apache.ignite.configuration.IgniteConfiguration;
+import org.apache.ignite.failure.StopNodeFailureHandler;
+import org.apache.ignite.internal.TestRecordingCommunicationSpi;
+import
org.apache.ignite.internal.processors.cache.distributed.near.GridNearTxFinishRequest;
+import
org.apache.ignite.internal.processors.cache.transactions.IgniteInternalTx;
+import org.apache.ignite.internal.util.future.GridCom
[ignite] branch master updated (bbcc563ccee -> 601a4cc0458)
This is an automated email from the ASF dual-hosted git repository. timoninmaxim pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/ignite.git from bbcc563ccee IGNITE-18780 Remove note in docs about Services being unsupported in mixed-platform clusters (#10535) add 601a4cc0458 IGNITE-18764 Restore marshaller data from snapshot (#10527) No new revisions were added by this update. Summary of changes: .../snapshot/SnapshotCompressionBasicTest.java | 2 +- .../snapshot/SnapshotRestoreProcess.java | 34 + .../IgniteClusterSnapshotRestoreSelfTest.java | 57 ++ .../junits/common/GridCommonAbstractTest.java | 11 - 4 files changed, 102 insertions(+), 2 deletions(-)
