[jira] [Work logged] (ARTEMIS-4558) AMQP Mirror ACKS should be idempotent
[ https://issues.apache.org/jira/browse/ARTEMIS-4558?focusedWorklogId=899747=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-899747 ] ASF GitHub Bot logged work on ARTEMIS-4558: --- Author: ASF GitHub Bot Created on: 16/Jan/24 00:13 Start Date: 16/Jan/24 00:13 Worklog Time Spent: 10m Work Description: clebertsuconic merged PR #4734: URL: https://github.com/apache/activemq-artemis/pull/4734 Issue Time Tracking --- Worklog Id: (was: 899747) Time Spent: 2h 20m (was: 2h 10m) > AMQP Mirror ACKS should be idempotent > - > > Key: ARTEMIS-4558 > URL: https://issues.apache.org/jira/browse/ARTEMIS-4558 > Project: ActiveMQ Artemis > Issue Type: Bug >Reporter: Clebert Suconic >Assignee: Clebert Suconic >Priority: Major > Fix For: 2.32.0 > > Time Spent: 2h 20m > Remaining Estimate: 0h > > When I first developed Mirroring, I assumed sending the mirrored ACK on a > aferACK and disconnected from any other transactions would be enough, with > the caveat you could get a duplicate delivery on the target mirror in case of > failures. > I got some complains that this is not safe enough from some users, and I'm > making this now idempotent. > I took an overal mirroring hardening approach and I'm improving test coverage > for this improvement. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (ARTEMIS-4558) AMQP Mirror ACKS should be idempotent
[
https://issues.apache.org/jira/browse/ARTEMIS-4558?focusedWorklogId=899709=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-899709
]
ASF GitHub Bot logged work on ARTEMIS-4558:
---
Author: ASF GitHub Bot
Created on: 15/Jan/24 17:32
Start Date: 15/Jan/24 17:32
Worklog Time Spent: 10m
Work Description: clebertsuconic commented on code in PR #4734:
URL: https://github.com/apache/activemq-artemis/pull/4734#discussion_r1452627220
##
tests/soak-tests/src/test/java/org/apache/activemq/artemis/tests/soak/brokerConnection/mirror/IdempotentACKTest.java:
##
@@ -0,0 +1,303 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.activemq.artemis.tests.soak.brokerConnection.mirror;
+
+import javax.jms.Connection;
+import javax.jms.ConnectionFactory;
+import javax.jms.JMSException;
+import javax.jms.Message;
+import javax.jms.MessageConsumer;
+import javax.jms.MessageProducer;
+import javax.jms.Queue;
+import javax.jms.Session;
+import javax.jms.TextMessage;
+import javax.jms.TransactionRolledBackException;
+import java.io.File;
+import java.io.StringWriter;
+import java.lang.invoke.MethodHandles;
+import java.util.Properties;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import org.apache.activemq.artemis.api.core.management.SimpleManagement;
+import
org.apache.activemq.artemis.core.config.amqpBrokerConnectivity.AMQPBrokerConnectionAddressType;
+import org.apache.activemq.artemis.tests.soak.SoakTestBase;
+import org.apache.activemq.artemis.tests.util.CFUtil;
+import org.apache.activemq.artemis.tests.util.Wait;
+import org.apache.activemq.artemis.util.ServerUtil;
+import org.apache.activemq.artemis.utils.cli.helper.HelperCreate;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class IdempotentACKTest extends SoakTestBase {
+
+ private static final Logger logger =
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ private static String largeBody;
+
+ static {
+ StringWriter writer = new StringWriter();
+ while (writer.getBuffer().length() < 1024 * 1024) {
+ writer.append("This is a large string . ");
+ }
+ largeBody = writer.toString();
+ }
+
+ private static final String QUEUE_NAME = "myQueue";
+
+ public static final String DC1_NODE_A = "idempotentMirror/DC1";
+ public static final String DC2_NODE_A = "idempotentMirror/DC2";
+
+ Process processDC1_node_A;
+ Process processDC2_node_A;
+
+ private static String DC1_NODEA_URI = "tcp://localhost:61616";
+ private static String DC2_NODEA_URI = "tcp://localhost:61618";
+
+ private static void createServer(String serverName,
+String connectionName,
+String mirrorURI,
+int porOffset) throws Exception {
+ File serverLocation = getFileServerLocation(serverName);
+ deleteDirectory(serverLocation);
+
+ HelperCreate cliCreateServer = new HelperCreate();
+
cliCreateServer.setAllowAnonymous(true).setNoWeb(true).setArtemisInstance(serverLocation);
+ cliCreateServer.setMessageLoadBalancing("ON_DEMAND");
+ cliCreateServer.setClustered(false);
+ cliCreateServer.setNoWeb(true);
+ cliCreateServer.setArgs("--no-stomp-acceptor", "--no-hornetq-acceptor",
"--no-mqtt-acceptor", "--no-amqp-acceptor", "--max-hops", "1", "--name",
DC1_NODE_A);
+ cliCreateServer.addArgs("--queues", QUEUE_NAME);
+ cliCreateServer.setPortOffset(porOffset);
+ cliCreateServer.createServer();
+
+ Properties brokerProperties = new Properties();
+ brokerProperties.put("AMQPConnections." + connectionName + ".uri",
mirrorURI);
+ brokerProperties.put("AMQPConnections." + connectionName +
".retryInterval", "1000");
+ brokerProperties.put("AMQPConnections."
[jira] [Work logged] (ARTEMIS-4558) AMQP Mirror ACKS should be idempotent
[
https://issues.apache.org/jira/browse/ARTEMIS-4558?focusedWorklogId=899699=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-899699
]
ASF GitHub Bot logged work on ARTEMIS-4558:
---
Author: ASF GitHub Bot
Created on: 15/Jan/24 17:10
Start Date: 15/Jan/24 17:10
Worklog Time Spent: 10m
Work Description: clebertsuconic commented on code in PR #4734:
URL: https://github.com/apache/activemq-artemis/pull/4734#discussion_r1452611296
##
artemis-protocols/artemis-amqp-protocol/src/main/java/org/apache/activemq/artemis/protocol/amqp/connect/AMQPBrokerConnection.java:
##
@@ -498,7 +498,7 @@ private Queue
installMirrorController(AMQPMirrorBrokerConnectionElement replicaC
Queue mirrorControlQueue =
server.locateQueue(getMirrorSNF(replicaConfig));
if (mirrorControlQueue == null) {
- mirrorControlQueue = server.createQueue(new
QueueConfiguration(getMirrorSNF(replicaConfig)).setAddress(getMirrorSNF(replicaConfig)).setRoutingType(RoutingType.ANYCAST).setDurable(replicaConfig.isDurable()).setInternal(true),
true);
+ mirrorControlQueue = server.createQueue(new
QueueConfiguration(getMirrorSNF(replicaConfig)).setAddress(getMirrorSNF(replicaConfig)).setRoutingType(RoutingType.ANYCAST).setDurable(replicaConfig.isDurable()).setInternal(false),
true);
Review Comment:
I am keeping it as internal, but allowing managemen ton this queue, by
calling the register directly. I also open a new JIRA for this.
Issue Time Tracking
---
Worklog Id: (was: 899699)
Time Spent: 2h (was: 1h 50m)
> AMQP Mirror ACKS should be idempotent
> -
>
> Key: ARTEMIS-4558
> URL: https://issues.apache.org/jira/browse/ARTEMIS-4558
> Project: ActiveMQ Artemis
> Issue Type: Bug
>Reporter: Clebert Suconic
>Assignee: Clebert Suconic
>Priority: Major
> Fix For: 2.32.0
>
> Time Spent: 2h
> Remaining Estimate: 0h
>
> When I first developed Mirroring, I assumed sending the mirrored ACK on a
> aferACK and disconnected from any other transactions would be enough, with
> the caveat you could get a duplicate delivery on the target mirror in case of
> failures.
> I got some complains that this is not safe enough from some users, and I'm
> making this now idempotent.
> I took an overal mirroring hardening approach and I'm improving test coverage
> for this improvement.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (ARTEMIS-4558) AMQP Mirror ACKS should be idempotent
[
https://issues.apache.org/jira/browse/ARTEMIS-4558?focusedWorklogId=898980=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-898980
]
ASF GitHub Bot logged work on ARTEMIS-4558:
---
Author: ASF GitHub Bot
Created on: 10/Jan/24 13:08
Start Date: 10/Jan/24 13:08
Worklog Time Spent: 10m
Work Description: gemmellr commented on code in PR #4734:
URL: https://github.com/apache/activemq-artemis/pull/4734#discussion_r1447347508
##
tests/soak-tests/src/test/java/org/apache/activemq/artemis/tests/soak/brokerConnection/mirror/IdempotentACKTest.java:
##
@@ -0,0 +1,303 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.activemq.artemis.tests.soak.brokerConnection.mirror;
+
+import javax.jms.Connection;
+import javax.jms.ConnectionFactory;
+import javax.jms.JMSException;
+import javax.jms.Message;
+import javax.jms.MessageConsumer;
+import javax.jms.MessageProducer;
+import javax.jms.Queue;
+import javax.jms.Session;
+import javax.jms.TextMessage;
+import javax.jms.TransactionRolledBackException;
+import java.io.File;
+import java.io.StringWriter;
+import java.lang.invoke.MethodHandles;
+import java.util.Properties;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import org.apache.activemq.artemis.api.core.management.SimpleManagement;
+import
org.apache.activemq.artemis.core.config.amqpBrokerConnectivity.AMQPBrokerConnectionAddressType;
+import org.apache.activemq.artemis.tests.soak.SoakTestBase;
+import org.apache.activemq.artemis.tests.util.CFUtil;
+import org.apache.activemq.artemis.tests.util.Wait;
+import org.apache.activemq.artemis.util.ServerUtil;
+import org.apache.activemq.artemis.utils.cli.helper.HelperCreate;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class IdempotentACKTest extends SoakTestBase {
+
+ private static final Logger logger =
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ private static String largeBody;
+
+ static {
+ StringWriter writer = new StringWriter();
+ while (writer.getBuffer().length() < 1024 * 1024) {
+ writer.append("This is a large string . ");
+ }
+ largeBody = writer.toString();
+ }
+
+ private static final String QUEUE_NAME = "myQueue";
+
+ public static final String DC1_NODE_A = "idempotentMirror/DC1";
+ public static final String DC2_NODE_A = "idempotentMirror/DC2";
+
+ Process processDC1_node_A;
+ Process processDC2_node_A;
+
+ private static String DC1_NODEA_URI = "tcp://localhost:61616";
+ private static String DC2_NODEA_URI = "tcp://localhost:61618";
+
+ private static void createServer(String serverName,
+String connectionName,
+String mirrorURI,
+int porOffset) throws Exception {
+ File serverLocation = getFileServerLocation(serverName);
+ deleteDirectory(serverLocation);
+
+ HelperCreate cliCreateServer = new HelperCreate();
+
cliCreateServer.setAllowAnonymous(true).setNoWeb(true).setArtemisInstance(serverLocation);
+ cliCreateServer.setMessageLoadBalancing("ON_DEMAND");
+ cliCreateServer.setClustered(false);
+ cliCreateServer.setNoWeb(true);
+ cliCreateServer.setArgs("--no-stomp-acceptor", "--no-hornetq-acceptor",
"--no-mqtt-acceptor", "--no-amqp-acceptor", "--max-hops", "1", "--name",
DC1_NODE_A);
+ cliCreateServer.addArgs("--queues", QUEUE_NAME);
+ cliCreateServer.setPortOffset(porOffset);
+ cliCreateServer.createServer();
+
+ Properties brokerProperties = new Properties();
+ brokerProperties.put("AMQPConnections." + connectionName + ".uri",
mirrorURI);
+ brokerProperties.put("AMQPConnections." + connectionName +
".retryInterval", "1000");
+ brokerProperties.put("AMQPConnections." +
[jira] [Work logged] (ARTEMIS-4558) AMQP Mirror ACKS should be idempotent
[
https://issues.apache.org/jira/browse/ARTEMIS-4558?focusedWorklogId=898979=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-898979
]
ASF GitHub Bot logged work on ARTEMIS-4558:
---
Author: ASF GitHub Bot
Created on: 10/Jan/24 13:06
Start Date: 10/Jan/24 13:06
Worklog Time Spent: 10m
Work Description: gemmellr commented on code in PR #4734:
URL: https://github.com/apache/activemq-artemis/pull/4734#discussion_r1447347508
##
tests/soak-tests/src/test/java/org/apache/activemq/artemis/tests/soak/brokerConnection/mirror/IdempotentACKTest.java:
##
@@ -0,0 +1,303 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.activemq.artemis.tests.soak.brokerConnection.mirror;
+
+import javax.jms.Connection;
+import javax.jms.ConnectionFactory;
+import javax.jms.JMSException;
+import javax.jms.Message;
+import javax.jms.MessageConsumer;
+import javax.jms.MessageProducer;
+import javax.jms.Queue;
+import javax.jms.Session;
+import javax.jms.TextMessage;
+import javax.jms.TransactionRolledBackException;
+import java.io.File;
+import java.io.StringWriter;
+import java.lang.invoke.MethodHandles;
+import java.util.Properties;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import org.apache.activemq.artemis.api.core.management.SimpleManagement;
+import
org.apache.activemq.artemis.core.config.amqpBrokerConnectivity.AMQPBrokerConnectionAddressType;
+import org.apache.activemq.artemis.tests.soak.SoakTestBase;
+import org.apache.activemq.artemis.tests.util.CFUtil;
+import org.apache.activemq.artemis.tests.util.Wait;
+import org.apache.activemq.artemis.util.ServerUtil;
+import org.apache.activemq.artemis.utils.cli.helper.HelperCreate;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class IdempotentACKTest extends SoakTestBase {
+
+ private static final Logger logger =
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ private static String largeBody;
+
+ static {
+ StringWriter writer = new StringWriter();
+ while (writer.getBuffer().length() < 1024 * 1024) {
+ writer.append("This is a large string . ");
+ }
+ largeBody = writer.toString();
+ }
+
+ private static final String QUEUE_NAME = "myQueue";
+
+ public static final String DC1_NODE_A = "idempotentMirror/DC1";
+ public static final String DC2_NODE_A = "idempotentMirror/DC2";
+
+ Process processDC1_node_A;
+ Process processDC2_node_A;
+
+ private static String DC1_NODEA_URI = "tcp://localhost:61616";
+ private static String DC2_NODEA_URI = "tcp://localhost:61618";
+
+ private static void createServer(String serverName,
+String connectionName,
+String mirrorURI,
+int porOffset) throws Exception {
+ File serverLocation = getFileServerLocation(serverName);
+ deleteDirectory(serverLocation);
+
+ HelperCreate cliCreateServer = new HelperCreate();
+
cliCreateServer.setAllowAnonymous(true).setNoWeb(true).setArtemisInstance(serverLocation);
+ cliCreateServer.setMessageLoadBalancing("ON_DEMAND");
+ cliCreateServer.setClustered(false);
+ cliCreateServer.setNoWeb(true);
+ cliCreateServer.setArgs("--no-stomp-acceptor", "--no-hornetq-acceptor",
"--no-mqtt-acceptor", "--no-amqp-acceptor", "--max-hops", "1", "--name",
DC1_NODE_A);
+ cliCreateServer.addArgs("--queues", QUEUE_NAME);
+ cliCreateServer.setPortOffset(porOffset);
+ cliCreateServer.createServer();
+
+ Properties brokerProperties = new Properties();
+ brokerProperties.put("AMQPConnections." + connectionName + ".uri",
mirrorURI);
+ brokerProperties.put("AMQPConnections." + connectionName +
".retryInterval", "1000");
+ brokerProperties.put("AMQPConnections." +
[jira] [Work logged] (ARTEMIS-4558) AMQP Mirror ACKS should be idempotent
[
https://issues.apache.org/jira/browse/ARTEMIS-4558?focusedWorklogId=898963=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-898963
]
ASF GitHub Bot logged work on ARTEMIS-4558:
---
Author: ASF GitHub Bot
Created on: 10/Jan/24 12:37
Start Date: 10/Jan/24 12:37
Worklog Time Spent: 10m
Work Description: gemmellr commented on code in PR #4734:
URL: https://github.com/apache/activemq-artemis/pull/4734#discussion_r1447330865
##
artemis-protocols/artemis-amqp-protocol/src/main/java/org/apache/activemq/artemis/protocol/amqp/connect/AMQPBrokerConnection.java:
##
@@ -498,7 +498,7 @@ private Queue
installMirrorController(AMQPMirrorBrokerConnectionElement replicaC
Queue mirrorControlQueue =
server.locateQueue(getMirrorSNF(replicaConfig));
if (mirrorControlQueue == null) {
- mirrorControlQueue = server.createQueue(new
QueueConfiguration(getMirrorSNF(replicaConfig)).setAddress(getMirrorSNF(replicaConfig)).setRoutingType(RoutingType.ANYCAST).setDurable(replicaConfig.isDurable()).setInternal(true),
true);
+ mirrorControlQueue = server.createQueue(new
QueueConfiguration(getMirrorSNF(replicaConfig)).setAddress(getMirrorSNF(replicaConfig)).setRoutingType(RoutingType.ANYCAST).setDurable(replicaConfig.isDurable()).setInternal(false),
true);
Review Comment:
I have no concern around whether it can be managed or not. I'm saying that
my impression from the prior change that added the 'dual mirror' stuff and made
it 'internal' was that it was done more to have it treated a specific way (e.g
excluded) by other e.g mirror/cluster/etc related code in the broker and not
just to hide the queue from management. I am raising the question over whether
thats actually the case and if so whether any previously desired treatment
elsewhere wont occur as a result from simply marking it not-internal again.
This didnt occur to me before, but while typing that it did... I believe
Justin (or maybe you?) made a change to persist the 'internal' value at some
point...if so is the broker code actually going to end up dealing with a
mixture now where some are and some arent?
Issue Time Tracking
---
Worklog Id: (was: 898963)
Time Spent: 1.5h (was: 1h 20m)
> AMQP Mirror ACKS should be idempotent
> -
>
> Key: ARTEMIS-4558
> URL: https://issues.apache.org/jira/browse/ARTEMIS-4558
> Project: ActiveMQ Artemis
> Issue Type: Bug
>Reporter: Clebert Suconic
>Assignee: Clebert Suconic
>Priority: Major
> Fix For: 2.32.0
>
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> When I first developed Mirroring, I assumed sending the mirrored ACK on a
> aferACK and disconnected from any other transactions would be enough, with
> the caveat you could get a duplicate delivery on the target mirror in case of
> failures.
> I got some complains that this is not safe enough from some users, and I'm
> making this now idempotent.
> I took an overal mirroring hardening approach and I'm improving test coverage
> for this improvement.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (ARTEMIS-4558) AMQP Mirror ACKS should be idempotent
[
https://issues.apache.org/jira/browse/ARTEMIS-4558?focusedWorklogId=898836=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-898836
]
ASF GitHub Bot logged work on ARTEMIS-4558:
---
Author: ASF GitHub Bot
Created on: 10/Jan/24 02:20
Start Date: 10/Jan/24 02:20
Worklog Time Spent: 10m
Work Description: clebertsuconic commented on code in PR #4734:
URL: https://github.com/apache/activemq-artemis/pull/4734#discussion_r1446811859
##
artemis-protocols/artemis-amqp-protocol/src/main/java/org/apache/activemq/artemis/protocol/amqp/connect/mirror/AMQPMirrorControllerSource.java:
##
@@ -609,10 +600,22 @@ private Message createMessage(SimpleString address,
SimpleString queue, Object e
return
AMQPMirrorMessageFactory.createMessage(snfQueue.getAddress().toString(),
address, queue, event, brokerID, body, ackReason);
}
- public static void route(ActiveMQServer server, Message message) throws
Exception {
+ public static void routeMirrorCommand(ActiveMQServer server, Message
message) throws Exception {
+ message.setMessageID(server.getStorageManager().generateID());
+ RoutingContext ctx = mirrorControlRouting.get();
+ // it is important to use local only at the source to avoid having the
message strictly load balancing
+ // to other nodes if the SNF queue has the same name as the one on this
node.
+
ctx.clear().setMirrorOption(MirrorOption.disabled).setLoadBalancingType(MessageLoadBalancingType.LOCAL_ONLY);
+ server.getPostOffice().route(message, ctx, false);
+ }
Review Comment:
thanks for pointing that out.. During my hacking I had a few versions where
I was using a different route Method. As it stands I missed simplifying at the
end.. thanks
Issue Time Tracking
---
Worklog Id: (was: 898836)
Time Spent: 1h 20m (was: 1h 10m)
> AMQP Mirror ACKS should be idempotent
> -
>
> Key: ARTEMIS-4558
> URL: https://issues.apache.org/jira/browse/ARTEMIS-4558
> Project: ActiveMQ Artemis
> Issue Type: Bug
>Reporter: Clebert Suconic
>Assignee: Clebert Suconic
>Priority: Major
> Fix For: 2.32.0
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> When I first developed Mirroring, I assumed sending the mirrored ACK on a
> aferACK and disconnected from any other transactions would be enough, with
> the caveat you could get a duplicate delivery on the target mirror in case of
> failures.
> I got some complains that this is not safe enough from some users, and I'm
> making this now idempotent.
> I took an overal mirroring hardening approach and I'm improving test coverage
> for this improvement.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (ARTEMIS-4558) AMQP Mirror ACKS should be idempotent
[
https://issues.apache.org/jira/browse/ARTEMIS-4558?focusedWorklogId=898785=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-898785
]
ASF GitHub Bot logged work on ARTEMIS-4558:
---
Author: ASF GitHub Bot
Created on: 09/Jan/24 21:09
Start Date: 09/Jan/24 21:09
Worklog Time Spent: 10m
Work Description: clebertsuconic commented on code in PR #4734:
URL: https://github.com/apache/activemq-artemis/pull/4734#discussion_r1446614999
##
artemis-protocols/artemis-amqp-protocol/src/main/java/org/apache/activemq/artemis/protocol/amqp/connect/mirror/AMQPMirrorControllerSource.java:
##
@@ -213,15 +215,31 @@ public void deleteQueue(SimpleString address,
SimpleString queue) throws Excepti
if (deleteQueues) {
Message message = createMessage(address, queue, DELETE_QUEUE, null,
queue.toString());
- route(server, message);
+ routeMirrorCommand(server, message);
}
}
+ private boolean invalidTarget(MirrorController controller, Message message)
{
+ if (controller == null) {
+ return false;
+ }
+ String remoteID = getRemoteMirrorId();
+ if (remoteID == null) {
+ // This is to avoid a reflection from a small period of time one node
reconnects but not the opposite direction
+ remoteID =
String.valueOf(message.getAnnotation(BROKER_ID_SIMPLE_STRING));
+ logger.debug("Remote link is not initialized yet, setting remoteID
from message as {}", remoteID);
Review Comment:
In the short period span the remoteID could be null, and if the message has
no annotation, it would receive "null", and the check if the same ID would fail
as expected.
I could make it more intentional by checking though.
Issue Time Tracking
---
Worklog Id: (was: 898785)
Time Spent: 1h 10m (was: 1h)
> AMQP Mirror ACKS should be idempotent
> -
>
> Key: ARTEMIS-4558
> URL: https://issues.apache.org/jira/browse/ARTEMIS-4558
> Project: ActiveMQ Artemis
> Issue Type: Bug
>Reporter: Clebert Suconic
>Assignee: Clebert Suconic
>Priority: Major
> Fix For: 2.32.0
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> When I first developed Mirroring, I assumed sending the mirrored ACK on a
> aferACK and disconnected from any other transactions would be enough, with
> the caveat you could get a duplicate delivery on the target mirror in case of
> failures.
> I got some complains that this is not safe enough from some users, and I'm
> making this now idempotent.
> I took an overal mirroring hardening approach and I'm improving test coverage
> for this improvement.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (ARTEMIS-4558) AMQP Mirror ACKS should be idempotent
[
https://issues.apache.org/jira/browse/ARTEMIS-4558?focusedWorklogId=898784=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-898784
]
ASF GitHub Bot logged work on ARTEMIS-4558:
---
Author: ASF GitHub Bot
Created on: 09/Jan/24 21:04
Start Date: 09/Jan/24 21:04
Worklog Time Spent: 10m
Work Description: clebertsuconic commented on code in PR #4734:
URL: https://github.com/apache/activemq-artemis/pull/4734#discussion_r1446610828
##
tests/soak-tests/src/test/java/org/apache/activemq/artemis/tests/soak/brokerConnection/mirror/IdemPotentACKTest.java:
##
@@ -0,0 +1,303 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.activemq.artemis.tests.soak.brokerConnection.mirror;
+
+import javax.jms.Connection;
+import javax.jms.ConnectionFactory;
+import javax.jms.JMSException;
+import javax.jms.Message;
+import javax.jms.MessageConsumer;
+import javax.jms.MessageProducer;
+import javax.jms.Queue;
+import javax.jms.Session;
+import javax.jms.TextMessage;
+import javax.jms.TransactionRolledBackException;
+import java.io.File;
+import java.io.StringWriter;
+import java.lang.invoke.MethodHandles;
+import java.util.Properties;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import org.apache.activemq.artemis.api.core.management.SimpleManagement;
+import
org.apache.activemq.artemis.core.config.amqpBrokerConnectivity.AMQPBrokerConnectionAddressType;
+import org.apache.activemq.artemis.tests.soak.SoakTestBase;
+import org.apache.activemq.artemis.tests.util.CFUtil;
+import org.apache.activemq.artemis.tests.util.Wait;
+import org.apache.activemq.artemis.util.ServerUtil;
+import org.apache.activemq.artemis.utils.cli.helper.HelperCreate;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class IdemPotentACKTest extends SoakTestBase {
+
+ private static final Logger logger =
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ private static String largeBody;
+
+ static {
+ StringWriter writer = new StringWriter();
+ while (writer.getBuffer().length() < 1024 * 1024) {
+ writer.append("This is a large string . ");
+ }
+ largeBody = writer.toString();
+ }
+
+ private static final String QUEUE_NAME = "myQueue";
+
+ public static final String DC1_NODE_A = "idemPotentMirror/DC1";
+ public static final String DC2_NODE_A = "idemPotentMirror/DC2";
+
+ Process processDC1_node_A;
+ Process processDC2_node_A;
+
+ private static String DC1_NODEA_URI = "tcp://localhost:61616";
+ private static String DC2_NODEA_URI = "tcp://localhost:61618";
+
+ private static void createServer(String serverName,
+String connectionName,
+String mirrorURI,
+int porOffset) throws Exception {
+ File serverLocation = getFileServerLocation(serverName);
+ deleteDirectory(serverLocation);
+
+ HelperCreate cliCreateServer = new HelperCreate();
+
cliCreateServer.setAllowAnonymous(true).setNoWeb(true).setArtemisInstance(serverLocation);
+ cliCreateServer.setMessageLoadBalancing("ON_DEMAND");
+ cliCreateServer.setClustered(false);
+ cliCreateServer.setNoWeb(true);
+ cliCreateServer.setArgs("--no-stomp-acceptor", "--no-hornetq-acceptor",
"--no-mqtt-acceptor", "--no-amqp-acceptor", "--max-hops", "1", "--name",
DC1_NODE_A);
+ cliCreateServer.addArgs("--queues", QUEUE_NAME);
+ cliCreateServer.setPortOffset(porOffset);
+ cliCreateServer.createServer();
+
+ Properties brokerProperties = new Properties();
+ brokerProperties.put("AMQPConnections." + connectionName + ".uri",
mirrorURI);
+ brokerProperties.put("AMQPConnections." + connectionName +
".retryInterval",
[jira] [Work logged] (ARTEMIS-4558) AMQP Mirror ACKS should be idempotent
[
https://issues.apache.org/jira/browse/ARTEMIS-4558?focusedWorklogId=898781=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-898781
]
ASF GitHub Bot logged work on ARTEMIS-4558:
---
Author: ASF GitHub Bot
Created on: 09/Jan/24 20:52
Start Date: 09/Jan/24 20:52
Worklog Time Spent: 10m
Work Description: clebertsuconic commented on code in PR #4734:
URL: https://github.com/apache/activemq-artemis/pull/4734#discussion_r1446601359
##
artemis-protocols/artemis-amqp-protocol/src/main/java/org/apache/activemq/artemis/protocol/amqp/connect/mirror/AMQPMirrorControllerSource.java:
##
@@ -213,15 +215,31 @@ public void deleteQueue(SimpleString address,
SimpleString queue) throws Excepti
if (deleteQueues) {
Message message = createMessage(address, queue, DELETE_QUEUE, null,
queue.toString());
- route(server, message);
+ routeMirrorCommand(server, message);
}
}
+ private boolean invalidTarget(MirrorController controller, Message message)
{
+ if (controller == null) {
+ return false;
+ }
+ String remoteID = getRemoteMirrorId();
+ if (remoteID == null) {
+ // This is to avoid a reflection from a small period of time one node
reconnects but not the opposite direction
+ remoteID =
String.valueOf(message.getAnnotation(BROKER_ID_SIMPLE_STRING));
+ logger.debug("Remote link is not initialized yet, setting remoteID
from message as {}", remoteID);
+ }
+ return sameNode(remoteID, controller.getRemoteMirrorId());
+ }
+
private boolean invalidTarget(MirrorController controller) {
return controller != null && sameNode(getRemoteMirrorId(),
controller.getRemoteMirrorId());
}
private boolean ignoreAddress(SimpleString address) {
+ if
(address.startsWith(server.getConfiguration().getManagementAddress())) {
+ return true;
+ }
Review Comment:
I would prefer if we keep this one together. It's part of the same tests.
some of the messages were leaking to other nodes in cluster.
Issue Time Tracking
---
Worklog Id: (was: 898781)
Time Spent: 50m (was: 40m)
> AMQP Mirror ACKS should be idempotent
> -
>
> Key: ARTEMIS-4558
> URL: https://issues.apache.org/jira/browse/ARTEMIS-4558
> Project: ActiveMQ Artemis
> Issue Type: Bug
>Reporter: Clebert Suconic
>Assignee: Clebert Suconic
>Priority: Major
> Fix For: 2.32.0
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> When I first developed Mirroring, I assumed sending the mirrored ACK on a
> aferACK and disconnected from any other transactions would be enough, with
> the caveat you could get a duplicate delivery on the target mirror in case of
> failures.
> I got some complains that this is not safe enough from some users, and I'm
> making this now idempotent.
> I took an overal mirroring hardening approach and I'm improving test coverage
> for this improvement.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (ARTEMIS-4558) AMQP Mirror ACKS should be idempotent
[
https://issues.apache.org/jira/browse/ARTEMIS-4558?focusedWorklogId=898779=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-898779
]
ASF GitHub Bot logged work on ARTEMIS-4558:
---
Author: ASF GitHub Bot
Created on: 09/Jan/24 20:48
Start Date: 09/Jan/24 20:48
Worklog Time Spent: 10m
Work Description: clebertsuconic commented on code in PR #4734:
URL: https://github.com/apache/activemq-artemis/pull/4734#discussion_r1446598128
##
artemis-protocols/artemis-amqp-protocol/src/main/java/org/apache/activemq/artemis/protocol/amqp/connect/AMQPBrokerConnection.java:
##
@@ -498,7 +498,7 @@ private Queue
installMirrorController(AMQPMirrorBrokerConnectionElement replicaC
Queue mirrorControlQueue =
server.locateQueue(getMirrorSNF(replicaConfig));
if (mirrorControlQueue == null) {
- mirrorControlQueue = server.createQueue(new
QueueConfiguration(getMirrorSNF(replicaConfig)).setAddress(getMirrorSNF(replicaConfig)).setRoutingType(RoutingType.ANYCAST).setDurable(replicaConfig.isDurable()).setInternal(true),
true);
+ mirrorControlQueue = server.createQueue(new
QueueConfiguration(getMirrorSNF(replicaConfig)).setAddress(getMirrorSNF(replicaConfig)).setRoutingType(RoutingType.ANYCAST).setDurable(replicaConfig.isDurable()).setInternal(false),
true);
Review Comment:
@gemmellr I think it was a mistake to hide management. There is some
movement to make internal queues managed. I'm not sure we merged that one...
Like the test I wrote for example, I need to manage from the queue.
So is the user I'm dealing with now.. they need to manage it.. check sizes,
do operations.. etc...
Issue Time Tracking
---
Worklog Id: (was: 898779)
Time Spent: 40m (was: 0.5h)
> AMQP Mirror ACKS should be idempotent
> -
>
> Key: ARTEMIS-4558
> URL: https://issues.apache.org/jira/browse/ARTEMIS-4558
> Project: ActiveMQ Artemis
> Issue Type: Bug
>Reporter: Clebert Suconic
>Assignee: Clebert Suconic
>Priority: Major
> Fix For: 2.32.0
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> When I first developed Mirroring, I assumed sending the mirrored ACK on a
> aferACK and disconnected from any other transactions would be enough, with
> the caveat you could get a duplicate delivery on the target mirror in case of
> failures.
> I got some complains that this is not safe enough from some users, and I'm
> making this now idempotent.
> I took an overal mirroring hardening approach and I'm improving test coverage
> for this improvement.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (ARTEMIS-4558) AMQP Mirror ACKS should be idempotent
[
https://issues.apache.org/jira/browse/ARTEMIS-4558?focusedWorklogId=898711=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-898711
]
ASF GitHub Bot logged work on ARTEMIS-4558:
---
Author: ASF GitHub Bot
Created on: 09/Jan/24 15:05
Start Date: 09/Jan/24 15:05
Worklog Time Spent: 10m
Work Description: gemmellr commented on code in PR #4734:
URL: https://github.com/apache/activemq-artemis/pull/4734#discussion_r1445899444
##
tests/integration-tests/src/test/java/org/apache/activemq/artemis/tests/integration/amqp/connect/MirrorControllerBasicTest.java:
##
@@ -94,7 +94,7 @@ public void testDirectSend() throws Exception {
server.createQueue(new
QueueConfiguration("test").setAddress("test").setRoutingType(RoutingType.ANYCAST));
Message message = AMQPMirrorMessageFactory.createMessage("test",
SimpleString.toSimpleString("ad1"), SimpleString.toSimpleString("qu1"), "test",
"someUID", "body-test", AckReason.KILLED);
- AMQPMirrorControllerSource.route(server, message);
+ AMQPMirrorControllerSource.routeMirrorCommand(server, message);
Review Comment:
This change doesnt appear to be in the same commit as the later change that
actually renamed the method, so it seems like one or more of these commits
might not compile.
##
tests/soak-tests/src/test/java/org/apache/activemq/artemis/tests/soak/brokerConnection/mirror/IdemPotentACKTest.java:
##
@@ -0,0 +1,303 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.activemq.artemis.tests.soak.brokerConnection.mirror;
+
+import javax.jms.Connection;
+import javax.jms.ConnectionFactory;
+import javax.jms.JMSException;
+import javax.jms.Message;
+import javax.jms.MessageConsumer;
+import javax.jms.MessageProducer;
+import javax.jms.Queue;
+import javax.jms.Session;
+import javax.jms.TextMessage;
+import javax.jms.TransactionRolledBackException;
+import java.io.File;
+import java.io.StringWriter;
+import java.lang.invoke.MethodHandles;
+import java.util.Properties;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import org.apache.activemq.artemis.api.core.management.SimpleManagement;
+import
org.apache.activemq.artemis.core.config.amqpBrokerConnectivity.AMQPBrokerConnectionAddressType;
+import org.apache.activemq.artemis.tests.soak.SoakTestBase;
+import org.apache.activemq.artemis.tests.util.CFUtil;
+import org.apache.activemq.artemis.tests.util.Wait;
+import org.apache.activemq.artemis.util.ServerUtil;
+import org.apache.activemq.artemis.utils.cli.helper.HelperCreate;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class IdemPotentACKTest extends SoakTestBase {
+
+ private static final Logger logger =
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ private static String largeBody;
+
+ static {
+ StringWriter writer = new StringWriter();
+ while (writer.getBuffer().length() < 1024 * 1024) {
+ writer.append("This is a large string . ");
+ }
+ largeBody = writer.toString();
+ }
+
+ private static final String QUEUE_NAME = "myQueue";
+
+ public static final String DC1_NODE_A = "idemPotentMirror/DC1";
+ public static final String DC2_NODE_A = "idemPotentMirror/DC2";
Review Comment:
ditto
##
tests/soak-tests/src/test/java/org/apache/activemq/artemis/tests/soak/brokerConnection/mirror/IdemPotentACKTest.java:
##
@@ -0,0 +1,303 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in
[jira] [Work logged] (ARTEMIS-4558) AMQP Mirror ACKS should be idempotent
[
https://issues.apache.org/jira/browse/ARTEMIS-4558?focusedWorklogId=898552=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-898552
]
ASF GitHub Bot logged work on ARTEMIS-4558:
---
Author: ASF GitHub Bot
Created on: 09/Jan/24 01:49
Start Date: 09/Jan/24 01:49
Worklog Time Spent: 10m
Work Description: clebertsuconic commented on code in PR #4734:
URL: https://github.com/apache/activemq-artemis/pull/4734#discussion_r1445545266
##
artemis-protocols/artemis-amqp-protocol/src/main/java/org/apache/activemq/artemis/protocol/amqp/connect/AMQPBrokerConnection.java:
##
@@ -498,7 +498,7 @@ private Queue
installMirrorController(AMQPMirrorBrokerConnectionElement replicaC
Queue mirrorControlQueue =
server.locateQueue(getMirrorSNF(replicaConfig));
if (mirrorControlQueue == null) {
- mirrorControlQueue = server.createQueue(new
QueueConfiguration(getMirrorSNF(replicaConfig)).setAddress(getMirrorSNF(replicaConfig)).setRoutingType(RoutingType.ANYCAST).setDurable(replicaConfig.isDurable()).setInternal(true),
true);
+ mirrorControlQueue = server.createQueue(new
QueueConfiguration(getMirrorSNF(replicaConfig)).setAddress(getMirrorSNF(replicaConfig)).setRoutingType(RoutingType.ANYCAST).setDurable(replicaConfig.isDurable()).setInternal(false),
true);
Review Comment:
I think it's better to remove the internal clause on this queue. as it
allows management.
Issue Time Tracking
---
Worklog Id: (was: 898552)
Time Spent: 20m (was: 10m)
> AMQP Mirror ACKS should be idempotent
> -
>
> Key: ARTEMIS-4558
> URL: https://issues.apache.org/jira/browse/ARTEMIS-4558
> Project: ActiveMQ Artemis
> Issue Type: Bug
>Reporter: Clebert Suconic
>Assignee: Clebert Suconic
>Priority: Major
> Fix For: 2.32.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> When I first developed Mirroring, I assumed sending the mirrored ACK on a
> aferACK and disconnected from any other transactions would be enough, with
> the caveat you could get a duplicate delivery on the target mirror in case of
> failures.
> I got some complains that this is not safe enough from some users, and I'm
> making this now idempotent.
> I took an overal mirroring hardening approach and I'm improving test coverage
> for this improvement.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (ARTEMIS-4558) AMQP Mirror ACKS should be idempotent
[ https://issues.apache.org/jira/browse/ARTEMIS-4558?focusedWorklogId=898544=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-898544 ] ASF GitHub Bot logged work on ARTEMIS-4558: --- Author: ASF GitHub Bot Created on: 08/Jan/24 23:29 Start Date: 08/Jan/24 23:29 Worklog Time Spent: 10m Work Description: clebertsuconic opened a new pull request, #4734: URL: https://github.com/apache/activemq-artemis/pull/4734 Mirror acks should be performed atomically with the storage of the source ACK. Both the send of the ack and the recording of the ack should be part of the same transaction (in case of transactional). We are also adding support on transactions for an afterWired callback for the proper plug of OperationContext sync. Issue Time Tracking --- Worklog Id: (was: 898544) Remaining Estimate: 0h Time Spent: 10m > AMQP Mirror ACKS should be idempotent > - > > Key: ARTEMIS-4558 > URL: https://issues.apache.org/jira/browse/ARTEMIS-4558 > Project: ActiveMQ Artemis > Issue Type: Bug >Reporter: Clebert Suconic >Assignee: Clebert Suconic >Priority: Major > Fix For: 2.32.0 > > Time Spent: 10m > Remaining Estimate: 0h > > When I first developed Mirroring, I assumed sending the mirrored ACK on a > aferACK and disconnected from any other transactions would be enough, with > the caveat you could get a duplicate delivery on the target mirror in case of > failures. > I got some complains that this is not safe enough from some users, and I'm > making this now idempotent. > I took an overal mirroring hardening approach and I'm improving test coverage > for this improvement. -- This message was sent by Atlassian Jira (v8.20.10#820010)
