[jira] [Work logged] (ARTEMIS-2336) Use zero copy to replicate journal/page/large message file
[
https://issues.apache.org/jira/browse/ARTEMIS-2336?focusedWorklogId=243859&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-243859
]
ASF GitHub Bot logged work on ARTEMIS-2336:
---
Author: ASF GitHub Bot
Created on: 17/May/19 07:01
Start Date: 17/May/19 07:01
Worklog Time Spent: 10m
Work Description: franz1981 commented on pull request #2666: ARTEMIS-2336
Use zero copy to replicate journal/page/large message file
URL: https://github.com/apache/activemq-artemis/pull/2666#discussion_r284583148
##
File path:
artemis-server/src/main/java/org/apache/activemq/artemis/core/remoting/impl/invm/InVMConnection.java
##
@@ -237,6 +240,22 @@ public void run() {
}
+ @Override
+ public void write(ActiveMQBuffer buffer,
+ RandomAccessFile raf,
+ FileChannel fileChannel,
+ long offset,
+ int dataSize,
+ boolean closeChannel) {
+ try {
+ if (closeChannel)
Review comment:
This method is correct that's empty?
Anyway please use a proper logger here
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 243859)
Time Spent: 5.5h (was: 5h 20m)
> Use zero copy to replicate journal/page/large message file
> --
>
> Key: ARTEMIS-2336
> URL: https://issues.apache.org/jira/browse/ARTEMIS-2336
> Project: ActiveMQ Artemis
> Issue Type: Improvement
> Components: Broker
>Reporter: yangwei
>Priority: Major
> Time Spent: 5.5h
> Remaining Estimate: 0h
>
> Use sendfile to replicate files during backup sever bootstrap.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (ARTEMIS-2336) Use zero copy to replicate journal/page/large message file
[ https://issues.apache.org/jira/browse/ARTEMIS-2336?focusedWorklogId=243858&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-243858 ] ASF GitHub Bot logged work on ARTEMIS-2336: --- Author: ASF GitHub Bot Created on: 17/May/19 06:59 Start Date: 17/May/19 06:59 Worklog Time Spent: 10m Work Description: franz1981 commented on pull request #2666: ARTEMIS-2336 Use zero copy to replicate journal/page/large message file URL: https://github.com/apache/activemq-artemis/pull/2666#discussion_r284581196 ## File path: artemis-core-client/src/main/java/org/apache/activemq/artemis/core/remoting/impl/netty/NettyConnection.java ## @@ -385,6 +402,34 @@ public final void write(ActiveMQBuffer buffer, } } + @Override + public void write(ActiveMQBuffer buffer, Review comment: You don't need the buffer parameter here: we already have an API to write just buffers: just create a specific API for the file case This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 243858) Time Spent: 5h 20m (was: 5h 10m) > Use zero copy to replicate journal/page/large message file > -- > > Key: ARTEMIS-2336 > URL: https://issues.apache.org/jira/browse/ARTEMIS-2336 > Project: ActiveMQ Artemis > Issue Type: Improvement > Components: Broker >Reporter: yangwei >Priority: Major > Time Spent: 5h 20m > Remaining Estimate: 0h > > Use sendfile to replicate files during backup sever bootstrap. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (ARTEMIS-2336) Use zero copy to replicate journal/page/large message file
[ https://issues.apache.org/jira/browse/ARTEMIS-2336?focusedWorklogId=243962&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-243962 ] ASF GitHub Bot logged work on ARTEMIS-2336: --- Author: ASF GitHub Bot Created on: 17/May/19 11:57 Start Date: 17/May/19 11:57 Worklog Time Spent: 10m Work Description: wy96f commented on pull request #2666: ARTEMIS-2336 Use zero copy to replicate journal/page/large message file URL: https://github.com/apache/activemq-artemis/pull/2666#discussion_r285094410 ## File path: artemis-core-client/src/main/java/org/apache/activemq/artemis/core/remoting/impl/netty/NettyConnection.java ## @@ -385,6 +402,34 @@ public final void write(ActiveMQBuffer buffer, } } + @Override + public void write(ActiveMQBuffer buffer, Review comment: Do you mean we call write twice, first write buffer api, second new write file api in channel::send()? This way we will call netty channel::writeAndFlush twice, right? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 243962) Time Spent: 5h 40m (was: 5.5h) > Use zero copy to replicate journal/page/large message file > -- > > Key: ARTEMIS-2336 > URL: https://issues.apache.org/jira/browse/ARTEMIS-2336 > Project: ActiveMQ Artemis > Issue Type: Improvement > Components: Broker >Reporter: yangwei >Priority: Major > Time Spent: 5h 40m > Remaining Estimate: 0h > > Use sendfile to replicate files during backup sever bootstrap. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (ARTEMIS-2336) Use zero copy to replicate journal/page/large message file
[
https://issues.apache.org/jira/browse/ARTEMIS-2336?focusedWorklogId=243964&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-243964
]
ASF GitHub Bot logged work on ARTEMIS-2336:
---
Author: ASF GitHub Bot
Created on: 17/May/19 12:00
Start Date: 17/May/19 12:00
Worklog Time Spent: 10m
Work Description: wy96f commented on pull request #2666: ARTEMIS-2336 Use
zero copy to replicate journal/page/large message file
URL: https://github.com/apache/activemq-artemis/pull/2666#discussion_r285095264
##
File path:
artemis-server/src/main/java/org/apache/activemq/artemis/core/remoting/impl/invm/InVMConnection.java
##
@@ -237,6 +240,22 @@ public void run() {
}
+ @Override
+ public void write(ActiveMQBuffer buffer,
+ RandomAccessFile raf,
+ FileChannel fileChannel,
+ long offset,
+ int dataSize,
+ boolean closeChannel) {
+ try {
+ if (closeChannel)
Review comment:
In ReplicationSyncFileMessage::encode, the file is read into buffer for non
netty connection, so we just need to send buffer.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 243964)
Time Spent: 5h 50m (was: 5h 40m)
> Use zero copy to replicate journal/page/large message file
> --
>
> Key: ARTEMIS-2336
> URL: https://issues.apache.org/jira/browse/ARTEMIS-2336
> Project: ActiveMQ Artemis
> Issue Type: Improvement
> Components: Broker
>Reporter: yangwei
>Priority: Major
> Time Spent: 5h 50m
> Remaining Estimate: 0h
>
> Use sendfile to replicate files during backup sever bootstrap.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (ARTEMIS-2336) Use zero copy to replicate journal/page/large message file
[ https://issues.apache.org/jira/browse/ARTEMIS-2336?focusedWorklogId=243968&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-243968 ] ASF GitHub Bot logged work on ARTEMIS-2336: --- Author: ASF GitHub Bot Created on: 17/May/19 12:03 Start Date: 17/May/19 12:03 Worklog Time Spent: 10m Work Description: franz1981 commented on pull request #2666: ARTEMIS-2336 Use zero copy to replicate journal/page/large message file URL: https://github.com/apache/activemq-artemis/pull/2666#discussion_r285096033 ## File path: artemis-core-client/src/main/java/org/apache/activemq/artemis/core/remoting/impl/netty/NettyConnection.java ## @@ -385,6 +402,34 @@ public final void write(ActiveMQBuffer buffer, } } + @Override + public void write(ActiveMQBuffer buffer, Review comment: Depends if we are using batching or not: it depends on the writeable bytes on the channel This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 243968) Time Spent: 6h (was: 5h 50m) > Use zero copy to replicate journal/page/large message file > -- > > Key: ARTEMIS-2336 > URL: https://issues.apache.org/jira/browse/ARTEMIS-2336 > Project: ActiveMQ Artemis > Issue Type: Improvement > Components: Broker >Reporter: yangwei >Priority: Major > Time Spent: 6h > Remaining Estimate: 0h > > Use sendfile to replicate files during backup sever bootstrap. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (ARTEMIS-2336) Use zero copy to replicate journal/page/large message file
[
https://issues.apache.org/jira/browse/ARTEMIS-2336?focusedWorklogId=243993&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-243993
]
ASF GitHub Bot logged work on ARTEMIS-2336:
---
Author: ASF GitHub Bot
Created on: 17/May/19 12:21
Start Date: 17/May/19 12:21
Worklog Time Spent: 10m
Work Description: wy96f commented on pull request #2666: ARTEMIS-2336 Use
zero copy to replicate journal/page/large message file
URL: https://github.com/apache/activemq-artemis/pull/2666#discussion_r285101658
##
File path:
artemis-core-client/src/main/java/org/apache/activemq/artemis/core/protocol/core/impl/ChannelImpl.java
##
@@ -274,67 +276,103 @@ private void waitForFailOver(String timeoutMsg) {
}
}
- // This must never called by more than one thread concurrently
- private boolean send(final Packet packet, final int reconnectID, final
boolean flush, final boolean batch) {
- if (invokeInterceptors(packet, interceptors, connection) != null) {
- return false;
+ private ActiveMQBuffer beforeSend(final Packet packet, final int
reconnectID) {
+ packet.setChannelID(id);
+
+ if (responseAsyncCache != null && packet.isRequiresResponse() &&
packet.isResponseAsync()) {
+ packet.setCorrelationID(responseAsyncCache.nextCorrelationID());
}
- synchronized (sendLock) {
- packet.setChannelID(id);
+ if (logger.isTraceEnabled()) {
+ logger.trace("RemotingConnectionID=" + (connection == null ? "NULL" :
connection.getID()) + " Sending packet nonblocking " + packet + " on
channelID=" + id);
+ }
- if (responseAsyncCache != null && packet.isRequiresResponse() &&
packet.isResponseAsync()) {
-packet.setCorrelationID(responseAsyncCache.nextCorrelationID());
+ ActiveMQBuffer buffer = packet.encode(connection);
+
+ lock.lock();
+
+ try {
+ if (failingOver) {
+waitForFailOver("RemotingConnectionID=" + (connection == null ?
"NULL" : connection.getID()) + " timed-out waiting for fail-over condition on
non-blocking send");
}
- if (logger.isTraceEnabled()) {
-logger.trace("RemotingConnectionID=" + (connection == null ?
"NULL" : connection.getID()) + " Sending packet nonblocking " + packet + " on
channelID=" + id);
+ // Sanity check
+ if (transferring) {
+throw
ActiveMQClientMessageBundle.BUNDLE.cannotSendPacketDuringFailover();
}
- ActiveMQBuffer buffer = packet.encode(connection);
+ if (resendCache != null && packet.isRequiresConfirmations()) {
+addResendPacket(packet);
+ }
- lock.lock();
+ } finally {
+ lock.unlock();
+ }
- try {
-if (failingOver) {
- waitForFailOver("RemotingConnectionID=" + (connection == null ?
"NULL" : connection.getID()) + " timed-out waiting for fail-over condition on
non-blocking send");
-}
+ if (logger.isTraceEnabled()) {
+ logger.trace("RemotingConnectionID=" + (connection == null ? "NULL" :
connection.getID()) + " Writing buffer for channelID=" + id);
+ }
-// Sanity check
-if (transferring) {
- throw
ActiveMQClientMessageBundle.BUNDLE.cannotSendPacketDuringFailover();
-}
+ checkReconnectID(reconnectID);
-if (resendCache != null && packet.isRequiresConfirmations()) {
- addResendPacket(packet);
+ //We do this outside the lock as ResponseCache is threadsafe and allows
responses to come in,
+ //As the send could block if the response cache cannot add, preventing
responses to be handled.
+ if (responseAsyncCache != null && packet.isRequiresResponse() &&
packet.isResponseAsync()) {
+ while (!responseAsyncCache.add(packet)) {
+try {
+ Thread.sleep(1);
+} catch (Exception e) {
+ // Ignore
Review comment:
It seems responseAsyncCache::add always returns true, so the sleep code
doesn't make sense?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 243993)
Time Spent: 6h 20m (was: 6h 10m)
> Use zero copy to replicate journal/page/large message file
> --
>
> Key: ARTEMIS-2336
> URL: https://issues.apache.org/jira/browse/ARTEMIS-2336
> Project: ActiveMQ Artemis
> Issue Type: Improvement
> Components: Broker
>
[jira] [Work logged] (ARTEMIS-2336) Use zero copy to replicate journal/page/large message file
[
https://issues.apache.org/jira/browse/ARTEMIS-2336?focusedWorklogId=243992&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-243992
]
ASF GitHub Bot logged work on ARTEMIS-2336:
---
Author: ASF GitHub Bot
Created on: 17/May/19 12:21
Start Date: 17/May/19 12:21
Worklog Time Spent: 10m
Work Description: wy96f commented on pull request #2666: ARTEMIS-2336 Use
zero copy to replicate journal/page/large message file
URL: https://github.com/apache/activemq-artemis/pull/2666#discussion_r285101553
##
File path:
artemis-server/src/main/java/org/apache/activemq/artemis/core/replication/ReplicationManager.java
##
@@ -551,49 +580,52 @@ private void
sendLargeFile(AbstractJournalStorageManager.JournalContent content,
if (!file.isOpen()) {
file.open();
}
- int size = 32 * 1024;
+ final int size = 1024 * 1024;
+ long fileSize = file.size();
int flowControlSize = 10;
int packetsSent = 0;
FlushAction action = new FlushAction();
+ long offset = 0;
+ RandomAccessFile raf;
+ FileChannel fileChannel = null;
try {
- try (FileInputStream fis = new FileInputStream(file.getJavaFile());
FileChannel channel = fis.getChannel()) {
-
-// We can afford having a single buffer here for this entire loop
-// because sendReplicatePacket will encode the packet as a
NettyBuffer
-// through ActiveMQBuffer class leaving this buffer free to be
reused on the next copy
-while (true) {
- final ByteBuf buffer =
PooledByteBufAllocator.DEFAULT.directBuffer(size, size);
- buffer.clear();
- ByteBuffer byteBuffer =
buffer.writerIndex(size).readerIndex(0).nioBuffer();
- final int bytesRead = channel.read(byteBuffer);
- int toSend = bytesRead;
- if (bytesRead > 0) {
- if (bytesRead >= maxBytesToSend) {
- toSend = (int) maxBytesToSend;
- maxBytesToSend = 0;
- } else {
- maxBytesToSend = maxBytesToSend - bytesRead;
- }
- }
- logger.debug("sending " + buffer.writerIndex() + " bytes on
file " + file.getFileName());
- // sending -1 or 0 bytes will close the file at the backup
- // We cannot simply send everything of a file through the
executor,
- // otherwise we would run out of memory.
- // so we don't use the executor here
- sendReplicatePacket(new ReplicationSyncFileMessage(content,
pageStore, id, toSend, buffer), true);
- packetsSent++;
-
- if (packetsSent % flowControlSize == 0) {
- flushReplicationStream(action);
+ raf = new RandomAccessFile(file.getJavaFile(), "r");
+ fileChannel = raf.getChannel();
+ while (true) {
+long chunkSize = Math.min(size, fileSize - offset);
+int toSend = (int) chunkSize;
+if (chunkSize > 0) {
+ if (chunkSize >= maxBytesToSend) {
+ toSend = (int) maxBytesToSend;
+ maxBytesToSend = 0;
+ } else {
+ maxBytesToSend = maxBytesToSend - chunkSize;
}
- if (bytesRead == -1 || bytesRead == 0 || maxBytesToSend == 0)
- break;
}
+logger.debug("sending " + toSend + " bytes on file " +
file.getFileName());
+// sending -1 or 0 bytes will close the file at the backup
+// We cannot simply send everything of a file through the executor,
+// otherwise we would run out of memory.
+// so we don't use the executor here
+sendSyncFileMessage(new ReplicationSyncFileMessage(content,
pageStore, id, raf, fileChannel, offset, toSend), offset + toSend == fileSize);
+packetsSent++;
+offset += toSend;
+
+if (packetsSent % flowControlSize == 0) {
+ flushReplicationStream(action);
+}
+if (toSend == 0 || maxBytesToSend == 0)
+ break;
}
flushReplicationStream(action);
+
+ } catch (Exception e) {
+ if (fileChannel != null)
+fileChannel.close();
Review comment:
You mean on caller side we add a callback where is called in netty
channelFutureListener after write?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Workl
[jira] [Work logged] (ARTEMIS-2336) Use zero copy to replicate journal/page/large message file
[
https://issues.apache.org/jira/browse/ARTEMIS-2336?focusedWorklogId=243994&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-243994
]
ASF GitHub Bot logged work on ARTEMIS-2336:
---
Author: ASF GitHub Bot
Created on: 17/May/19 12:26
Start Date: 17/May/19 12:26
Worklog Time Spent: 10m
Work Description: franz1981 commented on pull request #2666: ARTEMIS-2336
Use zero copy to replicate journal/page/large message file
URL: https://github.com/apache/activemq-artemis/pull/2666#discussion_r285103297
##
File path:
artemis-server/src/main/java/org/apache/activemq/artemis/core/replication/ReplicationManager.java
##
@@ -551,49 +580,52 @@ private void
sendLargeFile(AbstractJournalStorageManager.JournalContent content,
if (!file.isOpen()) {
file.open();
}
- int size = 32 * 1024;
+ final int size = 1024 * 1024;
+ long fileSize = file.size();
int flowControlSize = 10;
int packetsSent = 0;
FlushAction action = new FlushAction();
+ long offset = 0;
+ RandomAccessFile raf;
+ FileChannel fileChannel = null;
try {
- try (FileInputStream fis = new FileInputStream(file.getJavaFile());
FileChannel channel = fis.getChannel()) {
-
-// We can afford having a single buffer here for this entire loop
-// because sendReplicatePacket will encode the packet as a
NettyBuffer
-// through ActiveMQBuffer class leaving this buffer free to be
reused on the next copy
-while (true) {
- final ByteBuf buffer =
PooledByteBufAllocator.DEFAULT.directBuffer(size, size);
- buffer.clear();
- ByteBuffer byteBuffer =
buffer.writerIndex(size).readerIndex(0).nioBuffer();
- final int bytesRead = channel.read(byteBuffer);
- int toSend = bytesRead;
- if (bytesRead > 0) {
- if (bytesRead >= maxBytesToSend) {
- toSend = (int) maxBytesToSend;
- maxBytesToSend = 0;
- } else {
- maxBytesToSend = maxBytesToSend - bytesRead;
- }
- }
- logger.debug("sending " + buffer.writerIndex() + " bytes on
file " + file.getFileName());
- // sending -1 or 0 bytes will close the file at the backup
- // We cannot simply send everything of a file through the
executor,
- // otherwise we would run out of memory.
- // so we don't use the executor here
- sendReplicatePacket(new ReplicationSyncFileMessage(content,
pageStore, id, toSend, buffer), true);
- packetsSent++;
-
- if (packetsSent % flowControlSize == 0) {
- flushReplicationStream(action);
+ raf = new RandomAccessFile(file.getJavaFile(), "r");
+ fileChannel = raf.getChannel();
+ while (true) {
+long chunkSize = Math.min(size, fileSize - offset);
+int toSend = (int) chunkSize;
+if (chunkSize > 0) {
+ if (chunkSize >= maxBytesToSend) {
+ toSend = (int) maxBytesToSend;
+ maxBytesToSend = 0;
+ } else {
+ maxBytesToSend = maxBytesToSend - chunkSize;
}
- if (bytesRead == -1 || bytesRead == 0 || maxBytesToSend == 0)
- break;
}
+logger.debug("sending " + toSend + " bytes on file " +
file.getFileName());
+// sending -1 or 0 bytes will close the file at the backup
+// We cannot simply send everything of a file through the executor,
+// otherwise we would run out of memory.
+// so we don't use the executor here
+sendSyncFileMessage(new ReplicationSyncFileMessage(content,
pageStore, id, raf, fileChannel, offset, toSend), offset + toSend == fileSize);
+packetsSent++;
+offset += toSend;
+
+if (packetsSent % flowControlSize == 0) {
+ flushReplicationStream(action);
+}
+if (toSend == 0 || maxBytesToSend == 0)
+ break;
}
flushReplicationStream(action);
+
+ } catch (Exception e) {
+ if (fileChannel != null)
+fileChannel.close();
Review comment:
Yes, but just the supposed last packet need that listener, while the other
writes can just use a void promise instead
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
--
[jira] [Updated] (ARTEMIS-2344) [AMQP] Broker does not send security errors for unauthorized anonymous sasl
[
https://issues.apache.org/jira/browse/ARTEMIS-2344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Timothy Bish updated ARTEMIS-2344:
--
Affects Version/s: (was: 2.9.0)
2.8.0
2.8.1
> [AMQP] Broker does not send security errors for unauthorized anonymous sasl
> ---
>
> Key: ARTEMIS-2344
> URL: https://issues.apache.org/jira/browse/ARTEMIS-2344
> Project: ActiveMQ Artemis
> Issue Type: Bug
> Components: AMQP
>Affects Versions: 2.8.0, 2.8.1
>Reporter: Domenico Bruscino
>Priority: Major
>
> When user attempts unauthorized anonymous sasl, using AMQP protocol, the
> broker can return an internal error ("amqp:internal-error") instead of the
> security error ("amqp:unauthorized-access") that is expected in these cases.
>
> Source code to reproduce the issue using test
> testNoUserOrPasswordWithoutSaslRestrictions:
> https://github.com/brusdev/activemq-artemis/blob/amqp_security_error/tests/integration-tests/src/test/java/org/apache/activemq/artemis/tests/integration/amqp/JMSConnectionWithSecurityTest.java
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (AMQ-7209) [STOMP] SecurityError stack trace should no be in error frames
Gary Tully created AMQ-7209: --- Summary: [STOMP] SecurityError stack trace should no be in error frames Key: AMQ-7209 URL: https://issues.apache.org/jira/browse/AMQ-7209 Project: ActiveMQ Issue Type: Bug Components: Broker, stomp Affects Versions: 5.15.0 Reporter: Gary Tully Assignee: Gary Tully Fix For: 5.16.0 The stack trace of all exceptions is returned in the body of error frames. For security exceptions this can leak some information about the implementation and configured plugins which is not sensible. the stack trace should not be present for any SecurityException, just the exception message. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AMQ-7209) [STOMP] SecurityException stack trace should no be in error frames
[ https://issues.apache.org/jira/browse/AMQ-7209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Tully updated AMQ-7209: Summary: [STOMP] SecurityException stack trace should no be in error frames (was: [STOMP] SecurityError stack trace should no be in error frames) > [STOMP] SecurityException stack trace should no be in error frames > -- > > Key: AMQ-7209 > URL: https://issues.apache.org/jira/browse/AMQ-7209 > Project: ActiveMQ > Issue Type: Bug > Components: Broker, stomp >Affects Versions: 5.15.0 >Reporter: Gary Tully >Assignee: Gary Tully >Priority: Major > Fix For: 5.16.0 > > > The stack trace of all exceptions is returned in the body of error frames. > For security exceptions this can leak some information about the > implementation and configured plugins which is not sensible. > the stack trace should not be present for any SecurityException, just the > exception message. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AMQ-7209) [STOMP] SecurityException stack trace should no be in error frames
[ https://issues.apache.org/jira/browse/AMQ-7209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16842293#comment-16842293 ] ASF subversion and git services commented on AMQ-7209: -- Commit 4129c1f6592616ad8731e29c95092634699f96be in activemq's branch refs/heads/master from gtully [ https://gitbox.apache.org/repos/asf?p=activemq.git;h=4129c1f ] AMQ-7209 suppress stack trace in stomp error frame for SecurityExceptions to avoid any implementation detail leakage, fix and test > [STOMP] SecurityException stack trace should no be in error frames > -- > > Key: AMQ-7209 > URL: https://issues.apache.org/jira/browse/AMQ-7209 > Project: ActiveMQ > Issue Type: Bug > Components: Broker, stomp >Affects Versions: 5.15.0 >Reporter: Gary Tully >Assignee: Gary Tully >Priority: Major > Fix For: 5.16.0 > > > The stack trace of all exceptions is returned in the body of error frames. > For security exceptions this can leak some information about the > implementation and configured plugins which is not sensible. > the stack trace should not be present for any SecurityException, just the > exception message. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AMQ-7209) [STOMP] SecurityException stack trace should no be in error frames
[ https://issues.apache.org/jira/browse/AMQ-7209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Tully resolved AMQ-7209. - Resolution: Fixed > [STOMP] SecurityException stack trace should no be in error frames > -- > > Key: AMQ-7209 > URL: https://issues.apache.org/jira/browse/AMQ-7209 > Project: ActiveMQ > Issue Type: Bug > Components: Broker, stomp >Affects Versions: 5.15.0 >Reporter: Gary Tully >Assignee: Gary Tully >Priority: Major > Fix For: 5.16.0 > > > The stack trace of all exceptions is returned in the body of error frames. > For security exceptions this can leak some information about the > implementation and configured plugins which is not sensible. > the stack trace should not be present for any SecurityException, just the > exception message. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AMQ-7209) [STOMP] SecurityException stack trace should not be in error frames
[ https://issues.apache.org/jira/browse/AMQ-7209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Tully updated AMQ-7209: Summary: [STOMP] SecurityException stack trace should not be in error frames (was: [STOMP] SecurityException stack trace should no be in error frames) > [STOMP] SecurityException stack trace should not be in error frames > --- > > Key: AMQ-7209 > URL: https://issues.apache.org/jira/browse/AMQ-7209 > Project: ActiveMQ > Issue Type: Bug > Components: Broker, stomp >Affects Versions: 5.15.0 >Reporter: Gary Tully >Assignee: Gary Tully >Priority: Major > Fix For: 5.16.0 > > > The stack trace of all exceptions is returned in the body of error frames. > For security exceptions this can leak some information about the > implementation and configured plugins which is not sensible. > the stack trace should not be present for any SecurityException, just the > exception message. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AMQ-7210) AMQ slows down due to high number of stale ActiveMQ Connection Executor threads
Sahil Patayar created AMQ-7210: -- Summary: AMQ slows down due to high number of stale ActiveMQ Connection Executor threads Key: AMQ-7210 URL: https://issues.apache.org/jira/browse/AMQ-7210 Project: ActiveMQ Issue Type: Bug Components: Broker Affects Versions: 5.15.9 Reporter: Sahil Patayar Attachments: ActiveMQ_Connector_Executor_threads.JPG, executor_threads_comparison_from_different_thread_dumps.txt We have an enterprise application, and we are using embedded ActiveMQ. We are currently using 5.15.2 version. We tried the latest(5.15.9) version, but the issue still persists. ActiveMQ Connector Executor threads are being created in ActiveMQConnection class. ActiveMQ Connection Executor threads are not going away even after restarting the broker programmatically. Threads keep increasing and after bouncing the Broker, they don't go away. Steps to reproduce: 1. Run load test on ActiveMQ. 2. Take thread dumps 3. Stop the Broker multiple times 4. Take thread dumps again. Result: Compare the thread dumps, executor threads from the previous dump will be present in all next dumps. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AMQ-7210) AMQ slows down due to high number of stale ActiveMQ Connection Executor threads
[ https://issues.apache.org/jira/browse/AMQ-7210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sahil Patayar updated AMQ-7210: --- Attachment: executor_threads_comparison_from_different_thread_dumps.txt ActiveMQ_Connector_Executor_threads.JPG > AMQ slows down due to high number of stale ActiveMQ Connection Executor > threads > --- > > Key: AMQ-7210 > URL: https://issues.apache.org/jira/browse/AMQ-7210 > Project: ActiveMQ > Issue Type: Bug > Components: Broker >Affects Versions: 5.15.9 >Reporter: Sahil Patayar >Priority: Major > Attachments: ActiveMQ_Connector_Executor_threads.JPG, > executor_threads_comparison_from_different_thread_dumps.txt > > > We have an enterprise application, and we are using embedded ActiveMQ. > We are currently using 5.15.2 version. We tried the latest(5.15.9) version, > but the issue still persists. > ActiveMQ Connector Executor threads are being created in ActiveMQConnection > class. > > ActiveMQ Connection Executor threads are not going away even after restarting > the broker programmatically. Threads keep increasing and after bouncing the > Broker, they don't go away. > > Steps to reproduce: > 1. Run load test on ActiveMQ. > 2. Take thread dumps > 3. Stop the Broker multiple times > 4. Take thread dumps again. > Result: Compare the thread dumps, executor threads from the previous dump > will be present in all next dumps. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AMQ-7210) AMQ slows down due to high number of stale ActiveMQ Connection Executor threads
[ https://issues.apache.org/jira/browse/AMQ-7210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sahil Patayar updated AMQ-7210: --- Priority: Critical (was: Major) > AMQ slows down due to high number of stale ActiveMQ Connection Executor > threads > --- > > Key: AMQ-7210 > URL: https://issues.apache.org/jira/browse/AMQ-7210 > Project: ActiveMQ > Issue Type: Bug > Components: Broker >Affects Versions: 5.15.9 >Reporter: Sahil Patayar >Priority: Critical > Attachments: ActiveMQ_Connector_Executor_threads.JPG, > executor_threads_comparison_from_different_thread_dumps.txt > > > We have an enterprise application, and we are using embedded ActiveMQ. > We are currently using 5.15.2 version. We tried the latest(5.15.9) version, > but the issue still persists. > ActiveMQ Connector Executor threads are being created in ActiveMQConnection > class. > > ActiveMQ Connection Executor threads are not going away even after restarting > the broker programmatically. Threads keep increasing and after bouncing the > Broker, they don't go away. > > Steps to reproduce: > 1. Run load test on ActiveMQ. > 2. Take thread dumps > 3. Stop the Broker multiple times > 4. Take thread dumps again. > Result: Compare the thread dumps, executor threads from the previous dump > will be present in all next dumps. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AMQ-7210) AMQ slows down due to high number of stale ActiveMQ Connection Executor threads
[ https://issues.apache.org/jira/browse/AMQ-7210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16842425#comment-16842425 ] Timothy Bish commented on AMQ-7210: --- Have you closed the connections you created to communicate with that broker you keep restarting? > AMQ slows down due to high number of stale ActiveMQ Connection Executor > threads > --- > > Key: AMQ-7210 > URL: https://issues.apache.org/jira/browse/AMQ-7210 > Project: ActiveMQ > Issue Type: Bug > Components: Broker >Affects Versions: 5.15.9 >Reporter: Sahil Patayar >Priority: Critical > Attachments: ActiveMQ_Connector_Executor_threads.JPG, > executor_threads_comparison_from_different_thread_dumps.txt > > > We have an enterprise application, and we are using embedded ActiveMQ. > We are currently using 5.15.2 version. We tried the latest(5.15.9) version, > but the issue still persists. > ActiveMQ Connector Executor threads are being created in ActiveMQConnection > class. > > ActiveMQ Connection Executor threads are not going away even after restarting > the broker programmatically. Threads keep increasing and after bouncing the > Broker, they don't go away. > > Steps to reproduce: > 1. Run load test on ActiveMQ. > 2. Take thread dumps > 3. Stop the Broker multiple times > 4. Take thread dumps again. > Result: Compare the thread dumps, executor threads from the previous dump > will be present in all next dumps. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (ARTEMIS-2344) [AMQP] Broker does not send security errors for unauthorized anonymous sasl
[
https://issues.apache.org/jira/browse/ARTEMIS-2344?focusedWorklogId=244252&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-244252
]
ASF GitHub Bot logged work on ARTEMIS-2344:
---
Author: ASF GitHub Bot
Created on: 17/May/19 19:52
Start Date: 17/May/19 19:52
Worklog Time Spent: 10m
Work Description: asfgit commented on pull request #2671: ARTEMIS-2344
return security errors for unauthorized anonymous sasl
URL: https://github.com/apache/activemq-artemis/pull/2671
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 244252)
Time Spent: 10m
Remaining Estimate: 0h
> [AMQP] Broker does not send security errors for unauthorized anonymous sasl
> ---
>
> Key: ARTEMIS-2344
> URL: https://issues.apache.org/jira/browse/ARTEMIS-2344
> Project: ActiveMQ Artemis
> Issue Type: Bug
> Components: AMQP
>Affects Versions: 2.8.0, 2.8.1
>Reporter: Domenico Bruscino
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> When user attempts unauthorized anonymous sasl, using AMQP protocol, the
> broker can return an internal error ("amqp:internal-error") instead of the
> security error ("amqp:unauthorized-access") that is expected in these cases.
>
> Source code to reproduce the issue using test
> testNoUserOrPasswordWithoutSaslRestrictions:
> https://github.com/brusdev/activemq-artemis/blob/amqp_security_error/tests/integration-tests/src/test/java/org/apache/activemq/artemis/tests/integration/amqp/JMSConnectionWithSecurityTest.java
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (ARTEMIS-2344) [AMQP] Broker does not send security errors for unauthorized anonymous sasl
[
https://issues.apache.org/jira/browse/ARTEMIS-2344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16842429#comment-16842429
]
ASF subversion and git services commented on ARTEMIS-2344:
--
Commit e533bf876e43059eb5f52ed81117a6012c76addb in activemq-artemis's branch
refs/heads/master from brusdev
[ https://gitbox.apache.org/repos/asf?p=activemq-artemis.git;h=e533bf8 ]
ARTEMIS-2344 return security errors for unauthorized anonymous sasl
When user attempts unauthorized anonymous sasl the broker can return an
error of 'failed' instead of the security error that is expected in
these cases.
> [AMQP] Broker does not send security errors for unauthorized anonymous sasl
> ---
>
> Key: ARTEMIS-2344
> URL: https://issues.apache.org/jira/browse/ARTEMIS-2344
> Project: ActiveMQ Artemis
> Issue Type: Bug
> Components: AMQP
>Affects Versions: 2.8.0, 2.8.1
>Reporter: Domenico Bruscino
>Priority: Major
>
> When user attempts unauthorized anonymous sasl, using AMQP protocol, the
> broker can return an internal error ("amqp:internal-error") instead of the
> security error ("amqp:unauthorized-access") that is expected in these cases.
>
> Source code to reproduce the issue using test
> testNoUserOrPasswordWithoutSaslRestrictions:
> https://github.com/brusdev/activemq-artemis/blob/amqp_security_error/tests/integration-tests/src/test/java/org/apache/activemq/artemis/tests/integration/amqp/JMSConnectionWithSecurityTest.java
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AMQ-7210) AMQ slows down due to high number of stale ActiveMQ Connection Executor threads
[ https://issues.apache.org/jira/browse/AMQ-7210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16842469#comment-16842469 ] Sahil Patayar commented on AMQ-7210: Yes, we are closing the connections before attempting to restart the broker. > AMQ slows down due to high number of stale ActiveMQ Connection Executor > threads > --- > > Key: AMQ-7210 > URL: https://issues.apache.org/jira/browse/AMQ-7210 > Project: ActiveMQ > Issue Type: Bug > Components: Broker >Affects Versions: 5.15.9 >Reporter: Sahil Patayar >Priority: Critical > Attachments: ActiveMQ_Connector_Executor_threads.JPG, > executor_threads_comparison_from_different_thread_dumps.txt > > > We have an enterprise application, and we are using embedded ActiveMQ. > We are currently using 5.15.2 version. We tried the latest(5.15.9) version, > but the issue still persists. > ActiveMQ Connector Executor threads are being created in ActiveMQConnection > class. > > ActiveMQ Connection Executor threads are not going away even after restarting > the broker programmatically. Threads keep increasing and after bouncing the > Broker, they don't go away. > > Steps to reproduce: > 1. Run load test on ActiveMQ. > 2. Take thread dumps > 3. Stop the Broker multiple times > 4. Take thread dumps again. > Result: Compare the thread dumps, executor threads from the previous dump > will be present in all next dumps. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AMQ-7211) Pending ACKs errantly being cleared on STOMP transaction COMMIT/ABORT
Borgstrom created AMQ-7211:
--
Summary: Pending ACKs errantly being cleared on STOMP transaction
COMMIT/ABORT
Key: AMQ-7211
URL: https://issues.apache.org/jira/browse/AMQ-7211
Project: ActiveMQ
Issue Type: Bug
Components: stomp
Affects Versions: 5.15.6
Reporter: Borgstrom
Given a queue with many messages in it, and a client that is creating
transactions for each message it is processing will lead to the following error:
org.apache.activemq.transport.stomp.ProtocolException: Unexpected ACK received
for message-id [null]\n\tat
org.apache.activemq.transport.stomp.ProtocolConverter.onStompAck(ProtocolConverter.java:476)\n\tat
org.apache.activemq.transport.stomp.ProtocolConverter.onStompCommand(ProtocolConverter.java:251)\n\tat
org.apache.activemq.transport.stomp.StompTransportFilter.onCommand(StompTransportFilter.java:85)\n\tat
org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)\n\tat
org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:233)\n\tat
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:215)\n\tat
java.lang.Thread.run(Thread.java:748)\n
The client has a prefetch value of 1.
Here is the debug logs from my client showing frames sent & received. We end
up processing a number of messages, but eventually this happens:
{noformat}
DEBU[0119] Read frame body="[123 34 101 110
118 101 108 111 112 101 34 58 123 34 101 120 99 104 97 110 103 101 73 100 34 58
34 89 65 75 95 72 69 65 82 84 66 69 65 84 34 44 34 117 110 116 114 117 115 116
101 100 95 115 111 117 114 99 101 34 58 102 97 108 115 101 44 34 105 100 34 58
34 56 50 49 53 52 100 98 55 45 102 102 53 99 45 52 98 57 49 45 98 52 98 52 45
101 57 49 52 54 54 99 52 100 50 54 102 34 44 34 109 101 115 115 97 103 101 95
116 105 109 101 115 116 97 109 112 34 58 49 53 53 56 49 50 52 57 57 54 57 49 51
125 44 34 112 97 121 108 111 97 100 34 58 123 34 120 34 58 51 55 44 34 98 121
116 101 115 34 58 34 120 108 52 61 34 125 44 34 97 99 116 105 111 110 34 58 34
119 111 114 107 34 125]" command=MESSAGE
header="map[ack:ID\\c6e56db7da800-41539-1558117094401-130\\c12
content-type:application/json destination:/queue/YAK_HEARTBEAT-YAK_HEARTBEAT
expires:0
message-id:ID\\c6e56db7da800-41539-1558117094401-5\\c68\\c-1\\c1\\c538
persistent:true priority:4 subscription:1E81E7A4F3D6C40314373C0F20B9F1AF
timestamp:1558124997013]"
DEBU[0119] Writing frame body="[]"
command=BEGIN header="map[transaction:55CE4B41F0CF0C86C0D95EA9C7182A26]"
DEBU[0119] Writing frame body="[]" command=ACK
header="map[id:ID\\c6e56db7da800-41539-1558117094401-130\\c12
transaction:55CE4B41F0CF0C86C0D95EA9C7182A26]"
DEBU[0119] Writing frame body="[]"
command=COMMIT header="map[transaction:55CE4B41F0CF0C86C0D95EA9C7182A26]"
DEBU[0119] Read frame body="[123 34 101 110
118 101 108 111 112 101 34 58 123 34 101 120 99 104 97 110 103 101 73 100 34 58
34 89 65 75 95 72 69 65 82 84 66 69 65 84 34 44 34 117 110 116 114 117 115 116
101 100 95 115 111 117 114 99 101 34 58 102 97 108 115 101 44 34 105 100 34 58
34 101 100 49 51 56 51 53 101 45 56 98 50 98 45 52 98 100 101 45 97 52 99 50 45
102 51 50 49 98 49 54 99 50 51 57 57 34 44 34 109 101 115 115 97 103 101 95 116
105 109 101 115 116 97 109 112 34 58 49 53 53 56 49 50 52 57 57 54 57 49 51 125
44 34 112 97 121 108 111 97 100 34 58 123 34 120 34 58 51 56 44 34 98 121 116
101 115 34 58 34 52 103 61 61 34 125 44 34 97 99 116 105 111 110 34 58 34 119
111 114 107 34 125]" command=MESSAGE
header="map[ack:ID\\c6e56db7da800-41539-1558117094401-130\\c13
content-type:application/json destination:/queue/YAK_HEARTBEAT-YAK_HEARTBEAT
expires:0
message-id:ID\\c6e56db7da800-41539-1558117094401-5\\c68\\c-1\\c1\\c539
persistent:true priority:4 subscription:1E81E7A4F3D6C40314373C0F20B9F1AF
timestamp:1558124997017]"
DEBU[0119] Writing frame body="[]"
command=BEGIN header="map[transaction:0082C668C3243EDBA05FE48D25728979]"
DEBU[0119] Writing frame body="[]" command=ACK
header="map[id:ID\\c6e56db7da800-41539-1558117094401-130\\c13
transaction:0082C668C3243EDBA05FE48D25728979]"
DEBU[0119] Writing frame body="[]"
command=COMMIT header="map[transaction:0082C668C3243EDBA05FE48D25728979]"
DEBU[0119] Read frame body="[123 34 101 110
118 101 108 111 112 101 34 58 123 34 101 120 99 104 97 110 103 101 73 100 34 58
34 89 65 75 95 72 69 65 82 84 66 69 65 84 34 44 34 117 110 116 114 117 115 116
101 100 95 115 111 117 114 99 101 34 58 102 97 108 115 101 44 34 105 100 34 58
34 49 54 102 98 99 50 97 101 45 53 52 99 54 45 52 53 49 101 45 97 98 97 50 45
50 49 98
[jira] [Commented] (AMQ-7211) Pending ACKs errantly being cleared on STOMP transaction COMMIT/ABORT
[
https://issues.apache.org/jira/browse/AMQ-7211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16842787#comment-16842787
]
Borgstrom commented on AMQ-7211:
For more context, when the following block is processed for "c15":
[https://github.com/apache/activemq/blob/activemq-5.15.6/activemq-stomp/src/main/java/org/apache/activemq/transport/stomp/ProtocolConverter.java#L385-L394]
Then pendingAck is null because the previous COMMIT cleared this.pedingAck:
AckEntry pendingAck = this.pedingAcks.remove(ackId);
As such, messageId remains null as can be seen in the error message my consumer
receives:
throw new ProtocolException("Unexpected NACK received for message-id [" +
messageId + "]");
> Pending ACKs errantly being cleared on STOMP transaction COMMIT/ABORT
> -
>
> Key: AMQ-7211
> URL: https://issues.apache.org/jira/browse/AMQ-7211
> Project: ActiveMQ
> Issue Type: Bug
> Components: stomp
>Affects Versions: 5.15.6
>Reporter: Borgstrom
>Priority: Major
>
> Given a queue with many messages in it, and a client that is creating
> transactions for each message it is processing will lead to the following
> error:
> org.apache.activemq.transport.stomp.ProtocolException: Unexpected ACK
> received for message-id [null]\n\tat
> org.apache.activemq.transport.stomp.ProtocolConverter.onStompAck(ProtocolConverter.java:476)\n\tat
>
> org.apache.activemq.transport.stomp.ProtocolConverter.onStompCommand(ProtocolConverter.java:251)\n\tat
>
> org.apache.activemq.transport.stomp.StompTransportFilter.onCommand(StompTransportFilter.java:85)\n\tat
>
> org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)\n\tat
>
> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:233)\n\tat
>
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:215)\n\tat
> java.lang.Thread.run(Thread.java:748)\n
> The client has a prefetch value of 1.
> Here is the debug logs from my client showing frames sent & received. We end
> up processing a number of messages, but eventually this happens:
> {noformat}
> DEBU[0119] Read frame body="[123 34 101
> 110 118 101 108 111 112 101 34 58 123 34 101 120 99 104 97 110 103 101 73 100
> 34 58 34 89 65 75 95 72 69 65 82 84 66 69 65 84 34 44 34 117 110 116 114 117
> 115 116 101 100 95 115 111 117 114 99 101 34 58 102 97 108 115 101 44 34 105
> 100 34 58 34 56 50 49 53 52 100 98 55 45 102 102 53 99 45 52 98 57 49 45 98
> 52 98 52 45 101 57 49 52 54 54 99 52 100 50 54 102 34 44 34 109 101 115 115
> 97 103 101 95 116 105 109 101 115 116 97 109 112 34 58 49 53 53 56 49 50 52
> 57 57 54 57 49 51 125 44 34 112 97 121 108 111 97 100 34 58 123 34 120 34 58
> 51 55 44 34 98 121 116 101 115 34 58 34 120 108 52 61 34 125 44 34 97 99 116
> 105 111 110 34 58 34 119 111 114 107 34 125]" command=MESSAGE
> header="map[ack:ID\\c6e56db7da800-41539-1558117094401-130\\c12
> content-type:application/json destination:/queue/YAK_HEARTBEAT-YAK_HEARTBEAT
> expires:0
> message-id:ID\\c6e56db7da800-41539-1558117094401-5\\c68\\c-1\\c1\\c538
> persistent:true priority:4 subscription:1E81E7A4F3D6C40314373C0F20B9F1AF
> timestamp:1558124997013]"
> DEBU[0119] Writing frame body="[]"
> command=BEGIN header="map[transaction:55CE4B41F0CF0C86C0D95EA9C7182A26]"
> DEBU[0119] Writing frame body="[]"
> command=ACK header="map[id:ID\\c6e56db7da800-41539-1558117094401-130\\c12
> transaction:55CE4B41F0CF0C86C0D95EA9C7182A26]"
> DEBU[0119] Writing frame body="[]"
> command=COMMIT header="map[transaction:55CE4B41F0CF0C86C0D95EA9C7182A26]"
> DEBU[0119] Read frame body="[123 34 101
> 110 118 101 108 111 112 101 34 58 123 34 101 120 99 104 97 110 103 101 73 100
> 34 58 34 89 65 75 95 72 69 65 82 84 66 69 65 84 34 44 34 117 110 116 114 117
> 115 116 101 100 95 115 111 117 114 99 101 34 58 102 97 108 115 101 44 34 105
> 100 34 58 34 101 100 49 51 56 51 53 101 45 56 98 50 98 45 52 98 100 101 45 97
> 52 99 50 45 102 51 50 49 98 49 54 99 50 51 57 57 34 44 34 109 101 115 115 97
> 103 101 95 116 105 109 101 115 116 97 109 112 34 58 49 53 53 56 49 50 52 57
> 57 54 57 49 51 125 44 34 112 97 121 108 111 97 100 34 58 123 34 120 34 58 51
> 56 44 34 98 121 116 101 115 34 58 34 52 103 61 61 34 125 44 34 97 99 116 105
> 111 110 34 58 34 119 111 114 107 34 125]" command=MESSAGE
> header="map[ack:ID\\c6e56db7da800-41539-1558117094401-130\\c13
> content-type:application/json destination:/queue/YAK_HEARTBEAT-YAK_HEARTBEAT
> expires:0
> message-id:ID\\c6e56db7da800-41539-1558117094401-5\\c68\\c-1\\c1\\c539
> persistent:true priority:4 subscription:1E81E7A4F3D6C403143
[jira] [Commented] (AMQ-7211) Pending ACKs errantly being cleared on STOMP transaction COMMIT/ABORT
[
https://issues.apache.org/jira/browse/AMQ-7211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16842798#comment-16842798
]
Borgstrom commented on AMQ-7211:
The reason we're using transactions like this is that if our consumer fails to
process the message because of some transient error, then we want to send a new
copy of the message back into the queue that leverages the AMQ_SCHEDULED_DELAY
property of the scheduler to delay the retry, and we want the ACK of the
original message and the SEND of the new message with the delay property to
happen as a single, atomic operation on the broker.
CC [~tabish121] - I'm not sure what the policy for triaging unassigned issues
is, but I've seen you comment and work on other STOMP issues and I wanted to
make sure this didn't get lost. Hope you don't mind the direct mention.
Thanks!
> Pending ACKs errantly being cleared on STOMP transaction COMMIT/ABORT
> -
>
> Key: AMQ-7211
> URL: https://issues.apache.org/jira/browse/AMQ-7211
> Project: ActiveMQ
> Issue Type: Bug
> Components: stomp
>Affects Versions: 5.15.6
>Reporter: Borgstrom
>Priority: Major
>
> Given a queue with many messages in it, and a client that is creating
> transactions for each message it is processing will lead to the following
> error:
> org.apache.activemq.transport.stomp.ProtocolException: Unexpected ACK
> received for message-id [null]\n\tat
> org.apache.activemq.transport.stomp.ProtocolConverter.onStompAck(ProtocolConverter.java:476)\n\tat
>
> org.apache.activemq.transport.stomp.ProtocolConverter.onStompCommand(ProtocolConverter.java:251)\n\tat
>
> org.apache.activemq.transport.stomp.StompTransportFilter.onCommand(StompTransportFilter.java:85)\n\tat
>
> org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)\n\tat
>
> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:233)\n\tat
>
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:215)\n\tat
> java.lang.Thread.run(Thread.java:748)\n
> The client has a prefetch value of 1.
> Here is the debug logs from my client showing frames sent & received. We end
> up processing a number of messages, but eventually this happens:
> {noformat}
> DEBU[0119] Read frame body="[123 34 101
> 110 118 101 108 111 112 101 34 58 123 34 101 120 99 104 97 110 103 101 73 100
> 34 58 34 89 65 75 95 72 69 65 82 84 66 69 65 84 34 44 34 117 110 116 114 117
> 115 116 101 100 95 115 111 117 114 99 101 34 58 102 97 108 115 101 44 34 105
> 100 34 58 34 56 50 49 53 52 100 98 55 45 102 102 53 99 45 52 98 57 49 45 98
> 52 98 52 45 101 57 49 52 54 54 99 52 100 50 54 102 34 44 34 109 101 115 115
> 97 103 101 95 116 105 109 101 115 116 97 109 112 34 58 49 53 53 56 49 50 52
> 57 57 54 57 49 51 125 44 34 112 97 121 108 111 97 100 34 58 123 34 120 34 58
> 51 55 44 34 98 121 116 101 115 34 58 34 120 108 52 61 34 125 44 34 97 99 116
> 105 111 110 34 58 34 119 111 114 107 34 125]" command=MESSAGE
> header="map[ack:ID\\c6e56db7da800-41539-1558117094401-130\\c12
> content-type:application/json destination:/queue/YAK_HEARTBEAT-YAK_HEARTBEAT
> expires:0
> message-id:ID\\c6e56db7da800-41539-1558117094401-5\\c68\\c-1\\c1\\c538
> persistent:true priority:4 subscription:1E81E7A4F3D6C40314373C0F20B9F1AF
> timestamp:1558124997013]"
> DEBU[0119] Writing frame body="[]"
> command=BEGIN header="map[transaction:55CE4B41F0CF0C86C0D95EA9C7182A26]"
> DEBU[0119] Writing frame body="[]"
> command=ACK header="map[id:ID\\c6e56db7da800-41539-1558117094401-130\\c12
> transaction:55CE4B41F0CF0C86C0D95EA9C7182A26]"
> DEBU[0119] Writing frame body="[]"
> command=COMMIT header="map[transaction:55CE4B41F0CF0C86C0D95EA9C7182A26]"
> DEBU[0119] Read frame body="[123 34 101
> 110 118 101 108 111 112 101 34 58 123 34 101 120 99 104 97 110 103 101 73 100
> 34 58 34 89 65 75 95 72 69 65 82 84 66 69 65 84 34 44 34 117 110 116 114 117
> 115 116 101 100 95 115 111 117 114 99 101 34 58 102 97 108 115 101 44 34 105
> 100 34 58 34 101 100 49 51 56 51 53 101 45 56 98 50 98 45 52 98 100 101 45 97
> 52 99 50 45 102 51 50 49 98 49 54 99 50 51 57 57 34 44 34 109 101 115 115 97
> 103 101 95 116 105 109 101 115 116 97 109 112 34 58 49 53 53 56 49 50 52 57
> 57 54 57 49 51 125 44 34 112 97 121 108 111 97 100 34 58 123 34 120 34 58 51
> 56 44 34 98 121 116 101 115 34 58 34 52 103 61 61 34 125 44 34 97 99 116 105
> 111 110 34 58 34 119 111 114 107 34 125]" command=MESSAGE
> header="map[ack:ID\\c6e56db7da800-41539-1558117094401-130\\c13
> content-type:application/json destination:/queue/YAK_HEARTBEAT-YAK_HEARTBEAT
> ex
[jira] [Commented] (AMQ-7211) Pending ACKs errantly being cleared on STOMP transaction COMMIT/ABORT
[
https://issues.apache.org/jira/browse/AMQ-7211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16842842#comment-16842842
]
Borgstrom commented on AMQ-7211:
FWIW, if I simply comment out the two "pedingAcks.clear();" lines all of the
tests still pass:
{noformat}
Results :
Tests run: 968, Failures: 0, Errors: 0, Skipped: 40
[INFO]
[INFO] BUILD SUCCESS
[INFO]
[INFO] Total time: 26:06 min
[INFO] Finished at: 2019-05-17T17:22:19-07:00
[INFO]
[WARNING] The requested profile "tests" could not be activated because it does
not exist.
eborgstrom@eborgstrom-NW3081 ~/Projects/activemq/activemq-stomp — u:7 j:2
git:activemq-5.15.x● (17:22:19 05.17)
#558 ❯❯❯ git diff
diff --git
a/activemq-stomp/src/main/java/org/apache/activemq/transport/stomp/ProtocolConverter.java
b/activemq-stomp/src/main/java/org/apache/activemq/transport/stomp/ProtocolConverter.java
index 2c4d4023b..24873bffc 100644
---
a/activemq-stomp/src/main/java/org/apache/activemq/transport/stomp/ProtocolConverter.java
+++
b/activemq-stomp/src/main/java/org/apache/activemq/transport/stomp/ProtocolConverter.java
@@ -522,7 +522,7 @@ public class ProtocolConverter {
sub.onStompCommit(activemqTx);
}
- pedingAcks.clear();
+ // pedingAcks.clear();
TransactionInfo tx = new TransactionInfo();
tx.setConnectionId(connectionId);
@@ -553,7 +553,7 @@ public class ProtocolConverter {
}
}
- pedingAcks.clear();
+ // pedingAcks.clear();
TransactionInfo tx = new TransactionInfo();
tx.setConnectionId(connectionId);
{noformat}
> Pending ACKs errantly being cleared on STOMP transaction COMMIT/ABORT
> -
>
> Key: AMQ-7211
> URL: https://issues.apache.org/jira/browse/AMQ-7211
> Project: ActiveMQ
> Issue Type: Bug
> Components: stomp
>Affects Versions: 5.15.6
>Reporter: Borgstrom
>Priority: Major
>
> Given a queue with many messages in it, and a client that is creating
> transactions for each message it is processing will lead to the following
> error:
> org.apache.activemq.transport.stomp.ProtocolException: Unexpected ACK
> received for message-id [null]\n\tat
> org.apache.activemq.transport.stomp.ProtocolConverter.onStompAck(ProtocolConverter.java:476)\n\tat
>
> org.apache.activemq.transport.stomp.ProtocolConverter.onStompCommand(ProtocolConverter.java:251)\n\tat
>
> org.apache.activemq.transport.stomp.StompTransportFilter.onCommand(StompTransportFilter.java:85)\n\tat
>
> org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)\n\tat
>
> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:233)\n\tat
>
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:215)\n\tat
> java.lang.Thread.run(Thread.java:748)\n
> The client has a prefetch value of 1.
> Here is the debug logs from my client showing frames sent & received. We end
> up processing a number of messages, but eventually this happens:
> {noformat}
> DEBU[0119] Read frame body="[123 34 101
> 110 118 101 108 111 112 101 34 58 123 34 101 120 99 104 97 110 103 101 73 100
> 34 58 34 89 65 75 95 72 69 65 82 84 66 69 65 84 34 44 34 117 110 116 114 117
> 115 116 101 100 95 115 111 117 114 99 101 34 58 102 97 108 115 101 44 34 105
> 100 34 58 34 56 50 49 53 52 100 98 55 45 102 102 53 99 45 52 98 57 49 45 98
> 52 98 52 45 101 57 49 52 54 54 99 52 100 50 54 102 34 44 34 109 101 115 115
> 97 103 101 95 116 105 109 101 115 116 97 109 112 34 58 49 53 53 56 49 50 52
> 57 57 54 57 49 51 125 44 34 112 97 121 108 111 97 100 34 58 123 34 120 34 58
> 51 55 44 34 98 121 116 101 115 34 58 34 120 108 52 61 34 125 44 34 97 99 116
> 105 111 110 34 58 34 119 111 114 107 34 125]" command=MESSAGE
> header="map[ack:ID\\c6e56db7da800-41539-1558117094401-130\\c12
> content-type:application/json destination:/queue/YAK_HEARTBEAT-YAK_HEARTBEAT
> expires:0
> message-id:ID\\c6e56db7da800-41539-1558117094401-5\\c68\\c-1\\c1\\c538
> persistent:true priority:4 subscription:1E81E7A4F3D6C40314373C0F20B9F1AF
> timestamp:1558124997013]"
> DEBU[0119] Writing frame body="[]"
> command=BEGIN header="map[transaction:55CE4B41F0CF0C86C0D95EA9C7182A26]"
> DEBU[0119] Writing frame body="[]"
> command=ACK header="map[id:ID\\c6e56db7da800-41539-1558117094401-130\\c12
> transaction:55CE4B41F0CF0C86C0D95EA9C7182A26]"
> DEBU[0119] Writing frame body="[]"
[jira] [Created] (AMQ-7212) Topic memory percent can't reduce after message expired
xuxuanwan created AMQ-7212:
--
Summary: Topic memory percent can't reduce after message expired
Key: AMQ-7212
URL: https://issues.apache.org/jira/browse/AMQ-7212
Project: ActiveMQ
Issue Type: Bug
Components: Broker
Affects Versions: 5.15.9, 5.15.4
Environment: openjdk version "1.8.0_192"
OpenJDK Runtime Environment (AdoptOpenJDK)(build 1.8.0_192-b12)
OpenJDK 64-Bit Server VM (AdoptOpenJDK)(build 25.192-b12, mixed mode)
Reporter: xuxuanwan
Attachments: ActiveMqTest.java
Given:
* The policyEntry of Topic set memoryLimit ="128kb"
* let message expir in 10s
When：
* there have an inactive durable subscription subscribes a Topic
* messages are send to that topic with DeliveryMode.PERSISTENT
Then：
* messages are kept in brokers memory(visible in memoryPercentUsage)
When:
* messages expiration time or TTL is elapsed
Then:
* messages are still kept in brokers memory (visible in
{{memoryPercentUsage}}, on heap etc)
* only after unsubscribeDurableSubscription the memory will reduce.
This is similar to AMQ-6387 except that I use broker with persistence.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
