[jira] [Commented] (AMQ-4050) Broker unable to distribute messages to consumers with NegativeArraySizeException loading message from JDBC Persistence
[ https://issues.apache.org/jira/browse/AMQ-4050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14352805#comment-14352805 ] Felix Ehm commented on AMQ-4050: We experience the same problem with 5.11.1 and the KahaDB storage. But here, we have to delete the full kahadb (and loose the messages) . > Broker unable to distribute messages to consumers with > NegativeArraySizeException loading message from JDBC Persistence > --- > > Key: AMQ-4050 > URL: https://issues.apache.org/jira/browse/AMQ-4050 > Project: ActiveMQ > Issue Type: Bug > Components: Broker >Affects Versions: 5.5.1 > Environment: Broker is running in JVM via java code initialization. > JDBC Persistence Adapter to MySQL database. Production is CentOS. > Replicable on MacOS 10.7, Java 1.6.0_35, 5.1.63 MySQL Community Server >Reporter: Ernest Wallace > Attachments: ACTIVEMQ_MSGS.sql > > > When trying to distribute messages to consumers a corrupted message causes > the Broker to stall and stop distributing messages with a > NegativeArraySizeException. While the broker can no longer distribute > messages it does in fact accept them and properly persists them to the queue. > Attached is a sql dump of ACTIVEMQ_MSGS with the offending message to > reproduce. > StackTrace while running: > [07:33:46:882]|[09-13-2012]|[SYSOUT]|[INFO]|[122]|: > java.lang.NegativeArraySizeException| > [07:33:46:882]|[09-13-2012]|[SYSOUT]|[INFO]|[122]|: at > org.apache.activemq.openwire.v6.BaseDataStreamMarshaller.looseUnmarshalByteSequence(BaseDataStreamMarshaller.java:639)| > [07:33:46:882]|[09-13-2012]|[SYSOUT]|[INFO]|[122]|: at > org.apache.activemq.openwire.v6.MessageMarshaller.looseUnmarshal(MessageMarshaller.java:229)| > [07:33:46:882]|[09-13-2012]|[SYSOUT]|[INFO]|[122]|: at > org.apache.activemq.openwire.v6.ActiveMQMessageMarshaller.looseUnmarshal(ActiveMQMessageMarshaller.java:101)| > [07:33:46:882]|[09-13-2012]|[SYSOUT]|[INFO]|[122]|: at > org.apache.activemq.openwire.v6.ActiveMQTextMessageMarshaller.looseUnmarshal(ActiveMQTextMessageMarshaller.java:101)| > [07:33:46:882]|[09-13-2012]|[SYSOUT]|[INFO]|[122]|: at > org.apache.activemq.openwire.OpenWireFormat.doUnmarshal(OpenWireFormat.java:364)| > [07:33:46:882]|[09-13-2012]|[SYSOUT]|[INFO]|[122]|: at > org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:204)| > [07:33:46:883]|[09-13-2012]|[SYSOUT]|[INFO]|[122]|: at > org.apache.activemq.store.jdbc.JDBCMessageStore$2.recoverMessage(JDBCMessageStore.java:256)| > [07:33:46:883]|[09-13-2012]|[SYSOUT]|[INFO]|[122]|: at > org.apache.activemq.store.jdbc.adapter.DefaultJDBCAdapter.doRecoverNextMessages(DefaultJDBCAdapter.java:938)| > [07:33:46:883]|[09-13-2012]|[SYSOUT]|[INFO]|[122]|: at > org.apache.activemq.store.jdbc.JDBCMessageStore.recoverNextMessages(JDBCMessageStore.java:251)| > [07:33:46:883]|[09-13-2012]|[SYSOUT]|[INFO]|[122]|: at > org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)| > [07:33:46:883]|[09-13-2012]|[SYSOUT]|[INFO]|[122]|: at > org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)| > [07:33:46:883]|[09-13-2012]|[SYSOUT]|[INFO]|[122]|: at > org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:260)| > [07:33:46:883]|[09-13-2012]|[SYSOUT]|[INFO]|[122]|: at > org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:108)| > [07:33:46:883]|[09-13-2012]|[SYSOUT]|[INFO]|[122]|: at > org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)| > [07:33:46:883]|[09-13-2012]|[SYSOUT]|[INFO]|[122]|: at > org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1712)| > [07:33:46:883]|[09-13-2012]|[SYSOUT]|[INFO]|[122]|: at > org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1932)| > [07:33:46:883]|[09-13-2012]|[SYSOUT]|[INFO]|[122]|: at > org.apache.activemq.broker.region.Queue.iterate(Queue.java:1440)| > [07:33:46:883]|[09-13-2012]|[SYSOUT]|[INFO]|[122]|: at > org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:127)| > [07:33:46:883]|[09-13-2012]|[SYSOUT]|[INFO]|[122]|: at > org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:48)| > [07:33:46:883]|[09-13-2012]|[SYSOUT]|[INFO]|[122]|: at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)| > [07:33:46:883]|[09-13-2012]|[SYSOUT]|[INFO]|[122]|: at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)| > [07:33:46:883]|[09-13-2012]|[SYSOUT]|[INFO]|[122]|: at
[jira] [Commented] (AMQ-4986) tightEncoding=false + failover triggers Broker memory exhaust
[
https://issues.apache.org/jira/browse/AMQ-4986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14223146#comment-14223146
]
Felix Ehm commented on AMQ-4986:
In case this is of interest for you: I have removed this option from the URL
and was able to upgrade slowly the brokers.
I haven't really found a solution for this problem.
> tightEncoding=false + failover triggers Broker memory exhaust
> -
>
> Key: AMQ-4986
> URL: https://issues.apache.org/jira/browse/AMQ-4986
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
>Affects Versions: 5.6.0, 5.7.0, 5.8.0, 5.9.0
> Environment: java 1.7
> Client library 5.5.1
>Reporter: Felix Ehm
> Fix For: NEEDS_REVIEW
>
>
> We experience this problem in combination with 5.5.1 client and the
> wireformat.tightEncodingEnabled=false+ failover:
> Scenario:
> 1. start standard broker
> 2. start Client (with e.g. a MessageListener) with failover protocol: e.g.
> failover:(tcp://123.123.123.123:61616?wireformat.tightEncodingEnabled=false)
> 3. wait around 30sec (default for inactivity check)
> Result:
> The client closes the connection and re-tries to the broker which in turn
> throws the following exception:
> {code}
> 2014-01-21 20:12:49,568 [ActiveMQ Transport:
> tcp:///123.123.123.123:60156@61616] DEBUG Transport Transport Connection to:
> tcp://124.124.124.124:60156 failed: java.io.IOException: Unexpected error
> occured: java.lang.OutOfMemoryError: Java heap space
> java.io.IOException: Unexpected error occured: java.lang.OutOfMemoryError:
> Java heap space
> at
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:203)
> at java.lang.Thread.run(Thread.java:722)
> Caused by: java.lang.OutOfMemoryError: Java heap space
> at
> org.apache.activemq.openwire.v8.BaseDataStreamMarshaller.looseUnmarshalByteSequence(BaseDataStreamMarshaller.java:638)
> at
> org.apache.activemq.openwire.v8.WireFormatInfoMarshaller.looseUnmarshal(WireFormatInfoMarshaller.java:132)
> at
> org.apache.activemq.openwire.OpenWireFormat.doUnmarshal(OpenWireFormat.java:373)
> at
> org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:285)
> at
> org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:221)
> at
> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:213)
> at
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:196)
> ... 1 more
> {code}
> The problem here is that the BaseDataStreamMarshaller reads an int from the
> buffer and re-uses it immediately to allocate a byte array:
> {code}
> protected byte[] looseUnmarshalByteArray(DataInput dataIn) throws IOException
> {
> byte rc[] = null;
> if (dataIn.readBoolean()) {
> int size = dataIn.readInt();
> rc = new byte[size]; // PROBLEM! What happens if size has been
> read and interpreted wrongly ?
> dataIn.readFully(rc);
> }
> return rc;
> }
> {code}
> In our case the dadaIn.readInt() read an int number of 785.477.224 which
> triggers the broker to allocate blindly this amount of mem.
> We do not know yet what triggers the wrong byte sequence from the client, but
> on the brokers side, there should be a protection against this case.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (AMQ-4986) tightEncoding=false + failover triggers Broker memory exhaust
[
https://issues.apache.org/jira/browse/AMQ-4986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13878723#comment-13878723
]
Felix Ehm commented on AMQ-4986:
Yes. We have a very large environment with > 200 applications where we are not
in the situation to be able to update client and broker setups always together
(all openwire).
According to Hiram this mixing should be ok and from my understanding the
openwire protocol should at least make sure the communication is fine. But
exactly here is the point where it breaks.
As I wrote, we're still trying to find out why the broker reads something
buggy from the wire from the client.
Did I mention that the client and broker come into a CON/DISCON loop ? The
client-side failover receives a invalid magic back and closes the connection,
and reconnects, and so on
The workaround of course is to remove this option, which I strongly consider.
Independently from this, the presented code should be looked at to prevent this
allocation. What do you think?
> tightEncoding=false + failover triggers Broker memory exhaust
> -
>
> Key: AMQ-4986
> URL: https://issues.apache.org/jira/browse/AMQ-4986
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
>Affects Versions: 5.6.0, 5.7.0, 5.8.0, 5.9.0
> Environment: java 1.7
> Client library 5.5.1
>Reporter: Felix Ehm
>Priority: Critical
> Fix For: 5.10.0
>
>
> We experience this problem in combination with 5.5.1 client and the
> wireformat.tightEncodingEnabled=false+ failover:
> Scenario:
> 1. start standard broker
> 2. start Client (with e.g. a MessageListener) with failover protocol: e.g.
> failover:(tcp://123.123.123.123:61616?wireformat.tightEncodingEnabled=false)
> 3. wait around 30sec (default for inactivity check)
> Result:
> The client closes the connection and re-tries to the broker which in turn
> throws the following exception:
> {code}
> 2014-01-21 20:12:49,568 [ActiveMQ Transport:
> tcp:///123.123.123.123:60156@61616] DEBUG Transport Transport Connection to:
> tcp://124.124.124.124:60156 failed: java.io.IOException: Unexpected error
> occured: java.lang.OutOfMemoryError: Java heap space
> java.io.IOException: Unexpected error occured: java.lang.OutOfMemoryError:
> Java heap space
> at
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:203)
> at java.lang.Thread.run(Thread.java:722)
> Caused by: java.lang.OutOfMemoryError: Java heap space
> at
> org.apache.activemq.openwire.v8.BaseDataStreamMarshaller.looseUnmarshalByteSequence(BaseDataStreamMarshaller.java:638)
> at
> org.apache.activemq.openwire.v8.WireFormatInfoMarshaller.looseUnmarshal(WireFormatInfoMarshaller.java:132)
> at
> org.apache.activemq.openwire.OpenWireFormat.doUnmarshal(OpenWireFormat.java:373)
> at
> org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:285)
> at
> org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:221)
> at
> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:213)
> at
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:196)
> ... 1 more
> {code}
> The problem here is that the BaseDataStreamMarshaller reads an int from the
> buffer and re-uses it immediately to allocate a byte array:
> {code}
> protected byte[] looseUnmarshalByteArray(DataInput dataIn) throws IOException
> {
> byte rc[] = null;
> if (dataIn.readBoolean()) {
> int size = dataIn.readInt();
> rc = new byte[size]; // PROBLEM! What happens if size has been
> read and interpreted wrongly ?
> dataIn.readFully(rc);
> }
> return rc;
> }
> {code}
> In our case the dadaIn.readInt() read an int number of 785.477.224 which
> triggers the broker to allocate blindly this amount of mem.
> We do not know yet what triggers the wrong byte sequence from the client, but
> on the brokers side, there should be a protection against this case.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Updated] (AMQ-4986) tightEncoding=false + failover triggers Broker memory exhaust
[
https://issues.apache.org/jira/browse/AMQ-4986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Felix Ehm updated AMQ-4986:
---
Description:
We experience this problem in combination with 5.5.1 client and the
wireformat.tightEncodingEnabled=false+ failover:
Scenario:
1. start standard broker
2. start Client (with e.g. a MessageListener) with failover protocol: e.g.
failover:(tcp://123.123.123.123:61616?wireformat.tightEncodingEnabled=false)
3. wait around 30sec (default for inactivity check)
Result:
The client closes the connection and re-tries to the broker which in turn
throws the following exception:
{code}
2014-01-21 20:12:49,568 [ActiveMQ Transport:
tcp:///123.123.123.123:60156@61616] DEBUG Transport Transport Connection to:
tcp://124.124.124.124:60156 failed: java.io.IOException: Unexpected error
occured: java.lang.OutOfMemoryError: Java heap space
java.io.IOException: Unexpected error occured: java.lang.OutOfMemoryError: Java
heap space
at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:203)
at java.lang.Thread.run(Thread.java:722)
Caused by: java.lang.OutOfMemoryError: Java heap space
at
org.apache.activemq.openwire.v8.BaseDataStreamMarshaller.looseUnmarshalByteSequence(BaseDataStreamMarshaller.java:638)
at
org.apache.activemq.openwire.v8.WireFormatInfoMarshaller.looseUnmarshal(WireFormatInfoMarshaller.java:132)
at
org.apache.activemq.openwire.OpenWireFormat.doUnmarshal(OpenWireFormat.java:373)
at
org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:285)
at
org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:221)
at
org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:213)
at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:196)
... 1 more
{code}
The problem here is that the BaseDataStreamMarshaller reads an int from the
buffer and re-uses it immediately to allocate a byte array:
{code}
protected byte[] looseUnmarshalByteArray(DataInput dataIn) throws IOException {
byte rc[] = null;
if (dataIn.readBoolean()) {
int size = dataIn.readInt();
rc = new byte[size]; // PROBLEM! What happens if size has been
read and interpreted wrongly ?
dataIn.readFully(rc);
}
return rc;
}
{code}
In our case the dadaIn.readInt() read an int number of 785.477.224 which
triggers the broker to allocate blindly this amount of mem.
We do not know yet what triggers the wrong byte sequence from the client, but
on the brokers side, there should be a protection against this case.
was:
We experience this problem in combination with 5.5.1 client and the
wireformat.tightEncodingEnabled=false+ failover:
Scenario:
1. start standard broker
2. start Client (with e.g. a MessageListener) with failover protocol: e.g.
failover:(tcp://123.123.123.123:61616?wireformat.tightEncodingEnabled=false)
3. wait around 30sec (default for inactivity check)
Result:
The client closes the connection and re-tries to the broker which in turn
throws the following exception:
{code}
2014-01-21 20:12:49,568 [ActiveMQ Transport: tcp:///10.18.3.97:60156@61616]
DEBUG Transport Transport Connection to: tcp://10.18.3.97:60156 failed:
java.io.IOException: Unexpected error occured: java.lang.OutOfMemoryError: Java
heap space
java.io.IOException: Unexpected error occured: java.lang.OutOfMemoryError: Java
heap space
at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:203)
at java.lang.Thread.run(Thread.java:722)
Caused by: java.lang.OutOfMemoryError: Java heap space
at
org.apache.activemq.openwire.v8.BaseDataStreamMarshaller.looseUnmarshalByteSequence(BaseDataStreamMarshaller.java:638)
at
org.apache.activemq.openwire.v8.WireFormatInfoMarshaller.looseUnmarshal(WireFormatInfoMarshaller.java:132)
at
org.apache.activemq.openwire.OpenWireFormat.doUnmarshal(OpenWireFormat.java:373)
at
org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:285)
at
org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:221)
at
org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:213)
at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:196)
... 1 more
{code}
The problem here is that the BaseDataStreamMarshaller reads an int from the
buffer and re-uses it immediately to allocate a byte array:
{code}
protected byte[] looseUnmarshalByteArray(DataInput dataIn) throws IOException {
byte rc[] = null;
if (dataIn.readBoolean()) {
int size = dataIn.readInt();
rc = new byte[size]; // PROBLEM! What happens if size has been
read and interpreted wrongly ?
dataIn.readFully(rc);
}
return rc;
}
{c
[jira] [Updated] (AMQ-4986) tightEncoding=false + failover triggers Broker memory exhaust
[
https://issues.apache.org/jira/browse/AMQ-4986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Felix Ehm updated AMQ-4986:
---
Description:
We experience this problem in combination with 5.5.1 client and the
wireformat.tightEncodingEnabled=false+ failover:
Scenario:
1. start standard broker
2. start Client (with e.g. a MessageListener) with failover protocol: e.g.
failover:(tcp://123.123.123.123:61616?wireformat.tightEncodingEnabled=false)
3. wait around 30sec (default for inactivity check)
Result:
The client closes the connection and re-tries to the broker which in turn
throws the following exception:
{code}
2014-01-21 20:12:49,568 [ActiveMQ Transport: tcp:///10.18.3.97:60156@61616]
DEBUG Transport Transport Connection to: tcp://10.18.3.97:60156 failed:
java.io.IOException: Unexpected error occured: java.lang.OutOfMemoryError: Java
heap space
java.io.IOException: Unexpected error occured: java.lang.OutOfMemoryError: Java
heap space
at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:203)
at java.lang.Thread.run(Thread.java:722)
Caused by: java.lang.OutOfMemoryError: Java heap space
at
org.apache.activemq.openwire.v8.BaseDataStreamMarshaller.looseUnmarshalByteSequence(BaseDataStreamMarshaller.java:638)
at
org.apache.activemq.openwire.v8.WireFormatInfoMarshaller.looseUnmarshal(WireFormatInfoMarshaller.java:132)
at
org.apache.activemq.openwire.OpenWireFormat.doUnmarshal(OpenWireFormat.java:373)
at
org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:285)
at
org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:221)
at
org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:213)
at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:196)
... 1 more
{code}
The problem here is that the BaseDataStreamMarshaller reads an int from the
buffer and re-uses it immediately to allocate a byte array:
{code}
protected byte[] looseUnmarshalByteArray(DataInput dataIn) throws IOException {
byte rc[] = null;
if (dataIn.readBoolean()) {
int size = dataIn.readInt();
rc = new byte[size]; // PROBLEM! What happens if size has been
read and interpreted wrongly ?
dataIn.readFully(rc);
}
return rc;
}
{code}
In our case the dadaIn.readInt() read an int number of 785.477.224 which
triggers the broker to allocate blindly this amount of mem.
We do not know yet what triggers the wrong byte sequence from the client, but
on the brokers side, there should be a protection against this case.
was:
We experience this problem in combination with 5.5.1 client and the
wireformat.tightEncodingEnabled=false+ failover:
Scenario:
1. start standard broker
2. start Client (with e.g. a MessageListener) with failover protocol: e.g.
failover:(tcp://123.123.123.123:61616?wireformat.tightEncodingEnabled=true)
3. wait around 30sec (default for inactivity check)
Result:
The client closes the connection and re-tries to the broker which in turn
throws the following exception:
{code}
2014-01-21 20:12:49,568 [ActiveMQ Transport: tcp:///10.18.3.97:60156@61616]
DEBUG Transport Transport Connection to: tcp://10.18.3.97:60156 failed:
java.io.IOException: Unexpected error occured: java.lang.OutOfMemoryError: Java
heap space
java.io.IOException: Unexpected error occured: java.lang.OutOfMemoryError: Java
heap space
at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:203)
at java.lang.Thread.run(Thread.java:722)
Caused by: java.lang.OutOfMemoryError: Java heap space
at
org.apache.activemq.openwire.v8.BaseDataStreamMarshaller.looseUnmarshalByteSequence(BaseDataStreamMarshaller.java:638)
at
org.apache.activemq.openwire.v8.WireFormatInfoMarshaller.looseUnmarshal(WireFormatInfoMarshaller.java:132)
at
org.apache.activemq.openwire.OpenWireFormat.doUnmarshal(OpenWireFormat.java:373)
at
org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:285)
at
org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:221)
at
org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:213)
at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:196)
... 1 more
{code}
The problem here is that the BaseDataStreamMarshaller reads an int from the
buffer and re-uses it immediately to allocate a byte array:
{code}
protected byte[] looseUnmarshalByteArray(DataInput dataIn) throws IOException {
byte rc[] = null;
if (dataIn.readBoolean()) {
int size = dataIn.readInt();
rc = new byte[size]; // PROBLEM! What happens if size has been
read and interpreted wrongly ?
dataIn.readFully(rc);
}
return rc;
}
{code}
In our
[jira] [Updated] (AMQ-4986) tightEncoding=false + failover triggers Broker memory exhaust
[
https://issues.apache.org/jira/browse/AMQ-4986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Felix Ehm updated AMQ-4986:
---
Description:
We experience this problem in combination with 5.5.1 client and the
wireformat.tightEncodingEnabled=false+ failover:
Scenario:
1. start standard broker
2. start Client (with e.g. a MessageListener) with failover protocol: e.g.
failover:(tcp://123.123.123.123:61616?wireformat.tightEncodingEnabled=true)
3. wait around 30sec (default for inactivity check)
Result:
The client closes the connection and re-tries to the broker which in turn
throws the following exception:
{code}
2014-01-21 20:12:49,568 [ActiveMQ Transport: tcp:///10.18.3.97:60156@61616]
DEBUG Transport Transport Connection to: tcp://10.18.3.97:60156 failed:
java.io.IOException: Unexpected error occured: java.lang.OutOfMemoryError: Java
heap space
java.io.IOException: Unexpected error occured: java.lang.OutOfMemoryError: Java
heap space
at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:203)
at java.lang.Thread.run(Thread.java:722)
Caused by: java.lang.OutOfMemoryError: Java heap space
at
org.apache.activemq.openwire.v8.BaseDataStreamMarshaller.looseUnmarshalByteSequence(BaseDataStreamMarshaller.java:638)
at
org.apache.activemq.openwire.v8.WireFormatInfoMarshaller.looseUnmarshal(WireFormatInfoMarshaller.java:132)
at
org.apache.activemq.openwire.OpenWireFormat.doUnmarshal(OpenWireFormat.java:373)
at
org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:285)
at
org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:221)
at
org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:213)
at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:196)
... 1 more
{code}
The problem here is that the BaseDataStreamMarshaller reads an int from the
buffer and re-uses it immediately to allocate a byte array:
{code}
protected byte[] looseUnmarshalByteArray(DataInput dataIn) throws IOException {
byte rc[] = null;
if (dataIn.readBoolean()) {
int size = dataIn.readInt();
rc = new byte[size]; // PROBLEM! What happens if size has been
read and interpreted wrongly ?
dataIn.readFully(rc);
}
return rc;
}
{code}
In our case the dadaIn.readInt() read an int number of 785.477.224 which
triggers the broker to allocate blindly this amount of mem.
We do not know yet what triggers the wrong byte sequence from the client, but
on the brokers side, there should be a protection against this case.
was:
We experience this problem in combination with 5.5.1 client and the
wireformat.tightEncodingEnabled=true + failover:
Scenario:
1. start standard broker
2. start Client (with e.g. a MessageListener) with failover protocol: e.g.
failover:(tcp://123.123.123.123:61616?wireformat.tightEncodingEnabled=true)
3. wait around 30sec (default for inactivity check)
Result:
The client closes the connection and re-tries to the broker which in turn
throws the following exception:
{code}
2014-01-21 20:12:49,568 [ActiveMQ Transport: tcp:///10.18.3.97:60156@61616]
DEBUG Transport Transport Connection to: tcp://10.18.3.97:60156 failed:
java.io.IOException: Unexpected error occured: java.lang.OutOfMemoryError: Java
heap space
java.io.IOException: Unexpected error occured: java.lang.OutOfMemoryError: Java
heap space
at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:203)
at java.lang.Thread.run(Thread.java:722)
Caused by: java.lang.OutOfMemoryError: Java heap space
at
org.apache.activemq.openwire.v8.BaseDataStreamMarshaller.looseUnmarshalByteSequence(BaseDataStreamMarshaller.java:638)
at
org.apache.activemq.openwire.v8.WireFormatInfoMarshaller.looseUnmarshal(WireFormatInfoMarshaller.java:132)
at
org.apache.activemq.openwire.OpenWireFormat.doUnmarshal(OpenWireFormat.java:373)
at
org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:285)
at
org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:221)
at
org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:213)
at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:196)
... 1 more
{code}
The problem here is that the BaseDataStreamMarshaller reads an int from the
buffer and re-uses it immediately to allocate a byte array:
{code}
protected byte[] looseUnmarshalByteArray(DataInput dataIn) throws IOException {
byte rc[] = null;
if (dataIn.readBoolean()) {
int size = dataIn.readInt();
rc = new byte[size]; // PROBLEM! What happens if size has been
read and interpreted wrongly ?
dataIn.readFully(rc);
}
return rc;
}
{code}
In our
[jira] [Updated] (AMQ-4986) tightEncoding=false + failover triggers Broker memory exhaust
[
https://issues.apache.org/jira/browse/AMQ-4986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Felix Ehm updated AMQ-4986:
---
Description:
We experience this problem in combination with 5.5.1 client and the
wireformat.tightEncodingEnabled=true + failover:
Scenario:
1. start standard broker
2. start Client (with e.g. a MessageListener) with failover protocol: e.g.
failover:(tcp://123.123.123.123:61616?wireformat.tightEncodingEnabled=true)
3. wait around 30sec (default for inactivity check)
Result:
The client closes the connection and re-tries to the broker which in turn
throws the following exception:
{code}
2014-01-21 20:12:49,568 [ActiveMQ Transport: tcp:///10.18.3.97:60156@61616]
DEBUG Transport Transport Connection to: tcp://10.18.3.97:60156 failed:
java.io.IOException: Unexpected error occured: java.lang.OutOfMemoryError: Java
heap space
java.io.IOException: Unexpected error occured: java.lang.OutOfMemoryError: Java
heap space
at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:203)
at java.lang.Thread.run(Thread.java:722)
Caused by: java.lang.OutOfMemoryError: Java heap space
at
org.apache.activemq.openwire.v8.BaseDataStreamMarshaller.looseUnmarshalByteSequence(BaseDataStreamMarshaller.java:638)
at
org.apache.activemq.openwire.v8.WireFormatInfoMarshaller.looseUnmarshal(WireFormatInfoMarshaller.java:132)
at
org.apache.activemq.openwire.OpenWireFormat.doUnmarshal(OpenWireFormat.java:373)
at
org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:285)
at
org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:221)
at
org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:213)
at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:196)
... 1 more
{code}
The problem here is that the BaseDataStreamMarshaller reads an int from the
buffer and re-uses it immediately to allocate a byte array:
{code}
protected byte[] looseUnmarshalByteArray(DataInput dataIn) throws IOException {
byte rc[] = null;
if (dataIn.readBoolean()) {
int size = dataIn.readInt();
rc = new byte[size]; // PROBLEM! What happens if size has been
read and interpreted wrongly ?
dataIn.readFully(rc);
}
return rc;
}
{code}
In our case the dadaIn.readInt() read an int number of 785.477.224 which
triggers the broker to allocate blindly this amount of mem.
We do not know yet what triggers the wrong byte sequence from the client, but
on the brokers side, there should be a protection against this case.
was:
We experience this problem in combination with 5.5.1 client and the
tightEncodingEnabled=true.
Scenario:
1. start standard broker
2. start Client (with e.g. a MessageListener)
3. wait around 30sec (default for inactivity check)
Result:
The client closes the connection and re-tries to the broker which in turn
throws the following exception:
{code}
2014-01-21 20:12:49,568 [ActiveMQ Transport: tcp:///10.18.3.97:60156@61616]
DEBUG Transport Transport Connection to: tcp://10.18.3.97:60156 failed:
java.io.IOException: Unexpected error occured: java.lang.OutOfMemoryError: Java
heap space
java.io.IOException: Unexpected error occured: java.lang.OutOfMemoryError: Java
heap space
at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:203)
at java.lang.Thread.run(Thread.java:722)
Caused by: java.lang.OutOfMemoryError: Java heap space
at
org.apache.activemq.openwire.v8.BaseDataStreamMarshaller.looseUnmarshalByteSequence(BaseDataStreamMarshaller.java:638)
at
org.apache.activemq.openwire.v8.WireFormatInfoMarshaller.looseUnmarshal(WireFormatInfoMarshaller.java:132)
at
org.apache.activemq.openwire.OpenWireFormat.doUnmarshal(OpenWireFormat.java:373)
at
org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:285)
at
org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:221)
at
org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:213)
at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:196)
... 1 more
{code}
The problem here is that the BaseDataStreamMarshaller reads an int from the
buffer and re-uses it immediately to allocate a byte array:
{code}
protected byte[] looseUnmarshalByteArray(DataInput dataIn) throws IOException {
byte rc[] = null;
if (dataIn.readBoolean()) {
int size = dataIn.readInt();
rc = new byte[size]; // PROBLEM! What happens if size has been
read and interpreted wrongly ?
dataIn.readFully(rc);
}
return rc;
}
{code}
In our case the dadaIn.readInt() read an int number of 785.477.224 which
triggers the broker to allocate blindly this amount of mem.
[jira] [Updated] (AMQ-4986) tightEncoding=false + failover triggers Broker memory exhaust
[
https://issues.apache.org/jira/browse/AMQ-4986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Felix Ehm updated AMQ-4986:
---
Priority: Critical (was: Blocker)
> tightEncoding=false + failover triggers Broker memory exhaust
> -
>
> Key: AMQ-4986
> URL: https://issues.apache.org/jira/browse/AMQ-4986
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
>Affects Versions: 5.6.0, 5.7.0, 5.8.0, 5.9.0
> Environment: java 1.7
> Client library 5.5.1
>Reporter: Felix Ehm
>Priority: Critical
> Fix For: 5.10.0
>
>
> We experience this problem in combination with 5.5.1 client and the
> tightEncodingEnabled=true.
> Scenario:
> 1. start standard broker
> 2. start Client (with e.g. a MessageListener)
> 3. wait around 30sec (default for inactivity check)
> Result:
> The client closes the connection and re-tries to the broker which in turn
> throws the following exception:
> {code}
> 2014-01-21 20:12:49,568 [ActiveMQ Transport: tcp:///10.18.3.97:60156@61616]
> DEBUG Transport Transport Connection to: tcp://10.18.3.97:60156 failed:
> java.io.IOException: Unexpected error occured: java.lang.OutOfMemoryError:
> Java heap space
> java.io.IOException: Unexpected error occured: java.lang.OutOfMemoryError:
> Java heap space
> at
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:203)
> at java.lang.Thread.run(Thread.java:722)
> Caused by: java.lang.OutOfMemoryError: Java heap space
> at
> org.apache.activemq.openwire.v8.BaseDataStreamMarshaller.looseUnmarshalByteSequence(BaseDataStreamMarshaller.java:638)
> at
> org.apache.activemq.openwire.v8.WireFormatInfoMarshaller.looseUnmarshal(WireFormatInfoMarshaller.java:132)
> at
> org.apache.activemq.openwire.OpenWireFormat.doUnmarshal(OpenWireFormat.java:373)
> at
> org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:285)
> at
> org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:221)
> at
> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:213)
> at
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:196)
> ... 1 more
> {code}
> The problem here is that the BaseDataStreamMarshaller reads an int from the
> buffer and re-uses it immediately to allocate a byte array:
> {code}
> protected byte[] looseUnmarshalByteArray(DataInput dataIn) throws IOException
> {
> byte rc[] = null;
> if (dataIn.readBoolean()) {
> int size = dataIn.readInt();
> rc = new byte[size]; // PROBLEM! What happens if size has been
> read and interpreted wrongly ?
> dataIn.readFully(rc);
> }
> return rc;
> }
> {code}
> In our case the dadaIn.readInt() read an int number of 785.477.224 which
> triggers the broker to allocate blindly this amount of mem.
> We do not know yet what triggers the wrong byte sequence from the client, but
> on the brokers side, there should be a protection against this case.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Created] (AMQ-4986) tightEncoding=false + failover triggers Broker memory exhaust
Felix Ehm created AMQ-4986:
--
Summary: tightEncoding=false + failover triggers Broker memory
exhaust
Key: AMQ-4986
URL: https://issues.apache.org/jira/browse/AMQ-4986
Project: ActiveMQ
Issue Type: Bug
Components: Broker
Affects Versions: 5.9.0, 5.8.0, 5.7.0, 5.6.0
Environment: java 1.7
Client library 5.5.1
Reporter: Felix Ehm
Priority: Blocker
Fix For: 5.10.0
We experience this problem in combination with 5.5.1 client and the
tightEncodingEnabled=true.
Scenario:
1. start standard broker
2. start Client (with e.g. a MessageListener)
3. wait around 30sec (default for inactivity check)
Result:
The client closes the connection and re-tries to the broker which in turn
throws the following exception:
{code}
2014-01-21 20:12:49,568 [ActiveMQ Transport: tcp:///10.18.3.97:60156@61616]
DEBUG Transport Transport Connection to: tcp://10.18.3.97:60156 failed:
java.io.IOException: Unexpected error occured: java.lang.OutOfMemoryError: Java
heap space
java.io.IOException: Unexpected error occured: java.lang.OutOfMemoryError: Java
heap space
at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:203)
at java.lang.Thread.run(Thread.java:722)
Caused by: java.lang.OutOfMemoryError: Java heap space
at
org.apache.activemq.openwire.v8.BaseDataStreamMarshaller.looseUnmarshalByteSequence(BaseDataStreamMarshaller.java:638)
at
org.apache.activemq.openwire.v8.WireFormatInfoMarshaller.looseUnmarshal(WireFormatInfoMarshaller.java:132)
at
org.apache.activemq.openwire.OpenWireFormat.doUnmarshal(OpenWireFormat.java:373)
at
org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:285)
at
org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:221)
at
org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:213)
at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:196)
... 1 more
{code}
The problem here is that the BaseDataStreamMarshaller reads an int from the
buffer and re-uses it immediately to allocate a byte array:
{code}
protected byte[] looseUnmarshalByteArray(DataInput dataIn) throws IOException {
byte rc[] = null;
if (dataIn.readBoolean()) {
int size = dataIn.readInt();
rc = new byte[size]; // PROBLEM! What happens if size has been
read and interpreted wrongly ?
dataIn.readFully(rc);
}
return rc;
}
{code}
In our case the dadaIn.readInt() read an int number of 785.477.224 which
triggers the broker to allocate blindly this amount of mem.
We do not know yet what triggers the wrong byte sequence from the client, but
on the brokers side, there should be a protection against this case.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] Updated: (AMQ-2632) Update client connections with information about a cluster of networked brokers
[
https://issues.apache.org/jira/browse/AMQ-2632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Felix Ehm updated AMQ-2632:
---
Comment: was deleted
(was: Thanks for this very useful feature!
However, it is not clear to me why I have the switch on the transport
configuration (updateClusterClients="true|false") which is happily ignored?
I've tested it with the two example dynamic broker configuration files shipped
in 5.4.1 package. The clients are always updated about a cluster member even
when setting updateClusterClients=false or using the broker filter:
{code:xml}
dynamic-broker-1:
updateClusterClients="false"
updateClusterFilter="dynamic-broker-1"/>
{code}
{code:xml}
dynamic-broker-2:
{code}
btw: changing the connection url on the client side is unfortunately not an
option for us.
Is this mechanism going to be changed/fixed in the next version ?
)
> Update client connections with information about a cluster of networked
> brokers
> ---
>
> Key: AMQ-2632
> URL: https://issues.apache.org/jira/browse/AMQ-2632
> Project: ActiveMQ
> Issue Type: Improvement
> Components: JMS client
>Affects Versions: 5.0.0, 5.1.0, 5.2.0, 5.3.0
>Reporter: Rob Davies
>Assignee: Rob Davies
> Fix For: 5.4.0
>
>
> Currently it is up to the client to decide which broker(s) it should connect
> to. It would be beneficial to allow clients to be informed of brokers
> joining/leaving a cluster of networked brokers, and optionally load balance
> across them.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AMQ-2632) Update client connections with information about a cluster of networked brokers
[
https://issues.apache.org/jira/browse/AMQ-2632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12979591#action_12979591
]
Felix Ehm commented on AMQ-2632:
Thanks for this very useful feature!
However, it is not clear to me why I have the switch on the transport
configuration (updateClusterClients="true|false") which is happily ignored?
I've tested it with the two example dynamic broker configuration files shipped
in 5.4.1 package. The clients are always updated about a cluster member even
when setting updateClusterClients=false or using the broker filter:
{code:xml}
dynamic-broker-1:
updateClusterClients="false"
updateClusterFilter="dynamic-broker-1"/>
{code}
{code:xml}
dynamic-broker-2:
{code}
btw: changing the connection url on the client side is unfortunately not an
option for us.
Is this mechanism going to be changed/fixed in the next version ?
> Update client connections with information about a cluster of networked
> brokers
> ---
>
> Key: AMQ-2632
> URL: https://issues.apache.org/jira/browse/AMQ-2632
> Project: ActiveMQ
> Issue Type: Improvement
> Components: JMS client
>Affects Versions: 5.0.0, 5.1.0, 5.2.0, 5.3.0
>Reporter: Rob Davies
>Assignee: Rob Davies
> Fix For: 5.4.0
>
>
> Currently it is up to the client to decide which broker(s) it should connect
> to. It would be beneficial to allow clients to be informed of brokers
> joining/leaving a cluster of networked brokers, and optionally load balance
> across them.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AMQ-2853) Inflight count increases when subscribing to topic using '>' on destiantion name.
[ https://issues.apache.org/activemq/browse/AMQ-2853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=63440#action_63440 ] Felix Ehm commented on AMQ-2853: It seems that 5.4.1 has solved this issue. However, I can't really reproduce it, because the InflightCount has been removed in the latest version. > Inflight count increases when subscribing to topic using '>' on destiantion > name. > - > > Key: AMQ-2853 > URL: https://issues.apache.org/activemq/browse/AMQ-2853 > Project: ActiveMQ > Issue Type: Bug >Affects Versions: 5.3.2 > Environment: Java(TM) SE Runtime Environment (build 1.6.0_17-b04) > 2.6.18-194.el5 #1 SMP > x86_64 >Reporter: Felix Ehm >Priority: Minor > > When subscribing to a topic using the '>' selector, the inflight counter for > the affected (sub)topics increases with every message sent. > However, the listeners receive the message correctly. > You can easily reproduce this by starting two producers on A.B and A.C. > If you subscribe using a MessageListener to 'A.>' you will see the increasing > Inflight counter. > This does not happen when subscribing directly to the topic. I.e. using two > MessageListeners. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (AMQ-2853) Inflight count increases when subscribing to topic using '>' on destiantion name.
[ https://issues.apache.org/activemq/browse/AMQ-2853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Ehm updated AMQ-2853: --- Description: When subscribing to a topic using the '>' selector, the inflight counter for the affected (sub)topics increases with every message sent. However, the listeners receive the message correctly. You can easily reproduce this by starting two producers on A.B and A.C. If you subscribe using a MessageListener to 'A.>' you will see the increasing Inflight counter. This does not happen when subscribing directly to the topic. I.e. using two MessageListeners. was: When subscribing to a topic using the '>' selector, the inflight counter for the affected (sub)topics are affected. You can easily reproduce this by starting two producers on A.B and A.C. If you subscribe using a MessageListener to 'A.>' you will see the increasing Inflight counter. This does not happen when subscribing directly to the topic. I.e. using two MessageListeners. Priority: Minor (was: Major) > Inflight count increases when subscribing to topic using '>' on destiantion > name. > - > > Key: AMQ-2853 > URL: https://issues.apache.org/activemq/browse/AMQ-2853 > Project: ActiveMQ > Issue Type: Bug >Affects Versions: 5.3.2 > Environment: Java(TM) SE Runtime Environment (build 1.6.0_17-b04) > 2.6.18-194.el5 #1 SMP > x86_64 >Reporter: Felix Ehm >Priority: Minor > > When subscribing to a topic using the '>' selector, the inflight counter for > the affected (sub)topics increases with every message sent. > However, the listeners receive the message correctly. > You can easily reproduce this by starting two producers on A.B and A.C. > If you subscribe using a MessageListener to 'A.>' you will see the increasing > Inflight counter. > This does not happen when subscribing directly to the topic. I.e. using two > MessageListeners. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (AMQ-2853) Inflight count increases when subscribing to topic using '>' on destiantion name.
Inflight count increases when subscribing to topic using '>' on destiantion name. - Key: AMQ-2853 URL: https://issues.apache.org/activemq/browse/AMQ-2853 Project: ActiveMQ Issue Type: Bug Affects Versions: 5.3.2 Environment: Java(TM) SE Runtime Environment (build 1.6.0_17-b04) 2.6.18-194.el5 #1 SMP x86_64 Reporter: Felix Ehm When subscribing to a topic using the '>' selector, the inflight counter for the affected (sub)topics are affected. You can easily reproduce this by starting two producers on A.B and A.C. If you subscribe using a MessageListener to 'A.>' you will see the increasing Inflight counter. This does not happen when subscribing directly to the topic. I.e. using two MessageListeners. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (AMQ-2760) Dequeue Counter is not decrementing for the messages sent to "topic"
[ https://issues.apache.org/activemq/browse/AMQ-2760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=60130#action_60130 ] Felix Ehm edited comment on AMQ-2760 at 6/17/10 10:46 AM: -- I could reproduce this issue with ActiveMQ 5.3.1 (default config) 1 TopicPublisher 1 MessageListener indeed the TotalQueueCounter does not increase, although the messages are delivered to the client. *However, it does work if you don't use wildcat subscriptions (i.e. FOO.>)* was (Author: felixehm): I could reproduce this issue with ActiveMQ 5.3.1 (default config) 1 TopicPublisher 1 MessageListener indeed the TotalQueueCounter does not increase, although the messages are delivered to the client. *However, it does work if you don't use wildcat subscriptions (i.e. FOO.>)* I've attached tow screenshots showing the behaviour. > Dequeue Counter is not decrementing for the messages sent to "topic" > > > Key: AMQ-2760 > URL: https://issues.apache.org/activemq/browse/AMQ-2760 > Project: ActiveMQ > Issue Type: Bug > Components: JMX >Reporter: Pavithra > Fix For: 5.4.1 > > Attachments: totalQueue_Issue_1.PNG, totalQueue_Issue_2.PNG > > > Dequeue Counter is not decrementing for the messages sent to "topic". Will > this create memory issues?? > I used a Java client to publish 40 messages to a topic using single > publisher. Topic en-queued all the 40 messages but the dequeue counter did > not dequeue completely. > Please confirm, why these messages is not dequeuing correctly? > I also see a similar issue that was created 2 years ago, but still in "open" > status. > https://issues.apache.org/activemq/browse/AMQ-1991 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (AMQ-2760) Dequeue Counter is not decrementing for the messages sent to "topic"
[ https://issues.apache.org/activemq/browse/AMQ-2760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Ehm updated AMQ-2760: --- Attachment: totalQueue_Issue_1.PNG totalQueue_Issue_2.PNG JConsole screenshots showing Total DequeueCounter not increasing although the subscription counter does. > Dequeue Counter is not decrementing for the messages sent to "topic" > > > Key: AMQ-2760 > URL: https://issues.apache.org/activemq/browse/AMQ-2760 > Project: ActiveMQ > Issue Type: Bug > Components: JMX >Reporter: Pavithra > Fix For: 5.4.1 > > Attachments: totalQueue_Issue_1.PNG, totalQueue_Issue_2.PNG > > > Dequeue Counter is not decrementing for the messages sent to "topic". Will > this create memory issues?? > I used a Java client to publish 40 messages to a topic using single > publisher. Topic en-queued all the 40 messages but the dequeue counter did > not dequeue completely. > Please confirm, why these messages is not dequeuing correctly? > I also see a similar issue that was created 2 years ago, but still in "open" > status. > https://issues.apache.org/activemq/browse/AMQ-1991 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (AMQ-2760) Dequeue Counter is not decrementing for the messages sent to "topic"
[ https://issues.apache.org/activemq/browse/AMQ-2760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=60130#action_60130 ] Felix Ehm commented on AMQ-2760: I could reproduce this issue with ActiveMQ 5.3.1 (default config) 1 TopicPublisher 1 MessageListener indeed the TotalQueueCounter does not increase, although the messages are delivered to the client. *However, it does work if you don't use wildcat subscriptions (i.e. FOO.>)* I've attached tow screenshots showing the behaviour. > Dequeue Counter is not decrementing for the messages sent to "topic" > > > Key: AMQ-2760 > URL: https://issues.apache.org/activemq/browse/AMQ-2760 > Project: ActiveMQ > Issue Type: Bug > Components: JMX >Reporter: Pavithra > Fix For: 5.4.1 > > > Dequeue Counter is not decrementing for the messages sent to "topic". Will > this create memory issues?? > I used a Java client to publish 40 messages to a topic using single > publisher. Topic en-queued all the 40 messages but the dequeue counter did > not dequeue completely. > Please confirm, why these messages is not dequeuing correctly? > I also see a similar issue that was created 2 years ago, but still in "open" > status. > https://issues.apache.org/activemq/browse/AMQ-1991 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (AMQ-2773) javax.jms.TopicPublisher is not announced via Advisory Topic
[ https://issues.apache.org/activemq/browse/AMQ-2773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Ehm updated AMQ-2773: --- Attachment: ProducerAnnouncementTest.java You can use the attached (rudimentary) Test class to test this case. > javax.jms.TopicPublisher is not announced via Advisory Topic > > > Key: AMQ-2773 > URL: https://issues.apache.org/activemq/browse/AMQ-2773 > Project: ActiveMQ > Issue Type: Bug > Components: Broker >Affects Versions: 5.3.2 >Reporter: Felix Ehm > Attachments: ProducerAnnouncementTest.java > > > When using a javax.jms.TopicPublisher object over a TopicSession, no > ActiveMQ.Advisory.Producer.Topic is created. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (AMQ-2773) javax.jms.TopicPublisher is not announced via Advisory Topic
javax.jms.TopicPublisher is not announced via Advisory Topic Key: AMQ-2773 URL: https://issues.apache.org/activemq/browse/AMQ-2773 Project: ActiveMQ Issue Type: Bug Components: Broker Affects Versions: 5.3.2 Reporter: Felix Ehm When using a javax.jms.TopicPublisher object over a TopicSession, no ActiveMQ.Advisory.Producer.Topic is created. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Resolved: (AMQ-2339) Enforcing persistent mode for incoming messages
[ https://issues.apache.org/activemq/browse/AMQ-2339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Ehm resolved AMQ-2339. Resolution: Fixed The new Broker plugin is attached to the issue. To install the plugin using spring you can add these lines to the conf/activemq.xml : http://www.springframework.org/schema/beans"; id="forcePersistencyModeBroker" class="org.apache.activemq.plugin.ForcePersistencyModeBrokerPlugin"> > Enforcing persistent mode for incoming messages > --- > > Key: AMQ-2339 > URL: https://issues.apache.org/activemq/browse/AMQ-2339 > Project: ActiveMQ > Issue Type: Improvement > Components: Broker >Affects Versions: 5.2.0 >Reporter: Felix Ehm >Priority: Minor > Fix For: 5.4.0 > > Attachments: ForcePersistencyModeBroker.java, > ForcePersistencyModeBrokerPlugin.java > > Original Estimate: 2 hours > Remaining Estimate: 2 hours > > It would be valuable to have a possibility to enforce the persistence mode > for incoming messages from clients. > In our case, one client connected and sent persistent messages with TTL=0 to > a queue without having a consuming client. This caused the broker to pile up > the messages until it died due to missing space (disk & mem). -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (AMQ-2339) Enforcing persistent mode for incoming messages
[ https://issues.apache.org/activemq/browse/AMQ-2339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Ehm updated AMQ-2339: --- Attachment: ForcePersistencyModeBrokerPlugin.java ForcePersistencyModeBroker.java A new Broker plugin which realizes the requested feature of enforcing persistence mode for incoming messages. > Enforcing persistent mode for incoming messages > --- > > Key: AMQ-2339 > URL: https://issues.apache.org/activemq/browse/AMQ-2339 > Project: ActiveMQ > Issue Type: Improvement > Components: Broker >Affects Versions: 5.2.0 >Reporter: Felix Ehm >Priority: Minor > Fix For: 5.4.0 > > Attachments: ForcePersistencyModeBroker.java, > ForcePersistencyModeBrokerPlugin.java > > Original Estimate: 2 hours > Remaining Estimate: 2 hours > > It would be valuable to have a possibility to enforce the persistence mode > for incoming messages from clients. > In our case, one client connected and sent persistent messages with TTL=0 to > a queue without having a consuming client. This caused the broker to pile up > the messages until it died due to missing space (disk & mem). -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (AMQ-2339) Enforcing persistent mode for incoming messages
Enforcing persistent mode for incoming messages --- Key: AMQ-2339 URL: https://issues.apache.org/activemq/browse/AMQ-2339 Project: ActiveMQ Issue Type: Improvement Components: Broker Affects Versions: 5.2.0 Reporter: Felix Ehm Priority: Minor Fix For: 5.4.0 It would be valuable to have a possibility to enforce the persistence mode for incoming messages from clients. In our case, one client connected and sent persistent messages with TTL=0 to a queue without having a consuming client. This caused the broker to pile up the messages until it died due to missing space (disk & mem). -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (AMQ-1993) Systems hang due to inability to timeout socket write operation
[ https://issues.apache.org/activemq/browse/AMQ-1993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=52385#action_52385 ] Felix Ehm commented on AMQ-1993: Hi, thanks for the hint. Haven't tested it, but it appears to be reasonable. Is there a reason for not having a timeout configured by default ? Cheers, Felix > Systems hang due to inability to timeout socket write operation > --- > > Key: AMQ-1993 > URL: https://issues.apache.org/activemq/browse/AMQ-1993 > Project: ActiveMQ > Issue Type: Bug > Components: Broker >Affects Versions: 5.1.0, 5.2.0 > Environment: Unix (Solaris and Linux tested) >Reporter: Filip Hanik >Assignee: Gary Tully >Priority: Critical > Fix For: 5.2.0 > > Attachments: jstack.blockedSession, patch-1-threadname-filter.patch, > patch-3-tcp-writetimeout.patch > > > the blocking Java Socket API doesn't have a timeout on socketWrite > invocations. > This means, if a TCP session is dropped or terminated without RST or FIN > packets, the operating system it left to eventually time out the session. On > the linux kernel this timeout usually takes 15 to 30minutes. > For this entire period, the AMQ server hangs, and producers and consumers are > unable to use a topic. > I have created two patches for this at the page: > http://www.hanik.com/covalent/amq/index.html > Let me show a bit more > - > "ActiveMQ Transport: tcp:///X.YYY.XXX.:2011" daemon prio=10 > tid=0x55d39000 nid=0xc78 runnable > [0x447c9000..0x447cac10] >java.lang.Thread.State: RUNNABLE > at java.net.SocketOutputStream.socketWrite0(Native Method) > at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92) > This is a thread stuck in blocking IO, and can be stuck for 30 minutes during > the kernel TCP retransmission attempts. > Unfortunately the thread dump is very misleading since the name of the > thread, is not the destination or even remotely related to the socket it is > operating on. > To mend this, a very simple (and configurable) ThreadNameFilter has been > suggested to the patch, that appends the destination and helps the system > administrator correctly identify the client that is about to receive data. > --- > at org.apache.activemq.broker.region.Topic.dispatch(Topic.java:581) > at org.apache.activemq.broker.region.Topic.doMessageSend(Topic.java:421) > - locked <0x2aaaec155818> (a > org.apache.activemq.broker.region.Topic) > at org.apache.activemq.broker.region.Topic.send(Topic.java:363) > The lock being held at this issue unfortunately makes the entire Topic single > threaded. > When this lock is being held, no other clients (producers and consumers) can > publish to/receive from this topic. > And this lock can hold up to 30 minutes. > I consider solving this single threaded behavior a 'feature enhancement' that > should be handled separately from this bug. Because even if it is solved, > threads still risk being stuck in socketWrite0 for dropped connections that > still appear to be established. > For this, I have implemented a socket timeout filter, based on a > TransportFilter, this filter only times out connections that are actually > writing data. > The two patches are at: > http://www.hanik.com/covalent/amq/patch-1-threadname-filter.patch > http://www.hanik.com/covalent/amq/patch-3-tcp-writetimeout.patch > the binary .jar applies to both 5.1 and trunk and can be used today in > existing environments. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (AMQ-1993) Systems hang due to inability to timeout socket write operation
[ https://issues.apache.org/activemq/browse/AMQ-1993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Ehm updated AMQ-1993: --- Attachment: jstack.blockedSession jstack thread dump > Systems hang due to inability to timeout socket write operation > --- > > Key: AMQ-1993 > URL: https://issues.apache.org/activemq/browse/AMQ-1993 > Project: ActiveMQ > Issue Type: Bug > Components: Broker >Affects Versions: 5.1.0, 5.2.0 > Environment: Unix (Solaris and Linux tested) >Reporter: Filip Hanik >Assignee: Gary Tully >Priority: Critical > Fix For: 5.2.0 > > Attachments: jstack.blockedSession, patch-1-threadname-filter.patch, > patch-3-tcp-writetimeout.patch > > > the blocking Java Socket API doesn't have a timeout on socketWrite > invocations. > This means, if a TCP session is dropped or terminated without RST or FIN > packets, the operating system it left to eventually time out the session. On > the linux kernel this timeout usually takes 15 to 30minutes. > For this entire period, the AMQ server hangs, and producers and consumers are > unable to use a topic. > I have created two patches for this at the page: > http://www.hanik.com/covalent/amq/index.html > Let me show a bit more > - > "ActiveMQ Transport: tcp:///X.YYY.XXX.:2011" daemon prio=10 > tid=0x55d39000 nid=0xc78 runnable > [0x447c9000..0x447cac10] >java.lang.Thread.State: RUNNABLE > at java.net.SocketOutputStream.socketWrite0(Native Method) > at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92) > This is a thread stuck in blocking IO, and can be stuck for 30 minutes during > the kernel TCP retransmission attempts. > Unfortunately the thread dump is very misleading since the name of the > thread, is not the destination or even remotely related to the socket it is > operating on. > To mend this, a very simple (and configurable) ThreadNameFilter has been > suggested to the patch, that appends the destination and helps the system > administrator correctly identify the client that is about to receive data. > --- > at org.apache.activemq.broker.region.Topic.dispatch(Topic.java:581) > at org.apache.activemq.broker.region.Topic.doMessageSend(Topic.java:421) > - locked <0x2aaaec155818> (a > org.apache.activemq.broker.region.Topic) > at org.apache.activemq.broker.region.Topic.send(Topic.java:363) > The lock being held at this issue unfortunately makes the entire Topic single > threaded. > When this lock is being held, no other clients (producers and consumers) can > publish to/receive from this topic. > And this lock can hold up to 30 minutes. > I consider solving this single threaded behavior a 'feature enhancement' that > should be handled separately from this bug. Because even if it is solved, > threads still risk being stuck in socketWrite0 for dropped connections that > still appear to be established. > For this, I have implemented a socket timeout filter, based on a > TransportFilter, this filter only times out connections that are actually > writing data. > The two patches are at: > http://www.hanik.com/covalent/amq/patch-1-threadname-filter.patch > http://www.hanik.com/covalent/amq/patch-3-tcp-writetimeout.patch > the binary .jar applies to both 5.1 and trunk and can be used today in > existing environments. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (AMQ-1993) Systems hang due to inability to timeout socket write operation
[ https://issues.apache.org/activemq/browse/AMQ-1993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Ehm updated AMQ-1993: --- Attachment: (was: jstack.blockedSession) > Systems hang due to inability to timeout socket write operation > --- > > Key: AMQ-1993 > URL: https://issues.apache.org/activemq/browse/AMQ-1993 > Project: ActiveMQ > Issue Type: Bug > Components: Broker >Affects Versions: 5.1.0, 5.2.0 > Environment: Unix (Solaris and Linux tested) >Reporter: Filip Hanik >Assignee: Gary Tully >Priority: Critical > Fix For: 5.2.0 > > Attachments: jstack.blockedSession, patch-1-threadname-filter.patch, > patch-3-tcp-writetimeout.patch > > > the blocking Java Socket API doesn't have a timeout on socketWrite > invocations. > This means, if a TCP session is dropped or terminated without RST or FIN > packets, the operating system it left to eventually time out the session. On > the linux kernel this timeout usually takes 15 to 30minutes. > For this entire period, the AMQ server hangs, and producers and consumers are > unable to use a topic. > I have created two patches for this at the page: > http://www.hanik.com/covalent/amq/index.html > Let me show a bit more > - > "ActiveMQ Transport: tcp:///X.YYY.XXX.:2011" daemon prio=10 > tid=0x55d39000 nid=0xc78 runnable > [0x447c9000..0x447cac10] >java.lang.Thread.State: RUNNABLE > at java.net.SocketOutputStream.socketWrite0(Native Method) > at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92) > This is a thread stuck in blocking IO, and can be stuck for 30 minutes during > the kernel TCP retransmission attempts. > Unfortunately the thread dump is very misleading since the name of the > thread, is not the destination or even remotely related to the socket it is > operating on. > To mend this, a very simple (and configurable) ThreadNameFilter has been > suggested to the patch, that appends the destination and helps the system > administrator correctly identify the client that is about to receive data. > --- > at org.apache.activemq.broker.region.Topic.dispatch(Topic.java:581) > at org.apache.activemq.broker.region.Topic.doMessageSend(Topic.java:421) > - locked <0x2aaaec155818> (a > org.apache.activemq.broker.region.Topic) > at org.apache.activemq.broker.region.Topic.send(Topic.java:363) > The lock being held at this issue unfortunately makes the entire Topic single > threaded. > When this lock is being held, no other clients (producers and consumers) can > publish to/receive from this topic. > And this lock can hold up to 30 minutes. > I consider solving this single threaded behavior a 'feature enhancement' that > should be handled separately from this bug. Because even if it is solved, > threads still risk being stuck in socketWrite0 for dropped connections that > still appear to be established. > For this, I have implemented a socket timeout filter, based on a > TransportFilter, this filter only times out connections that are actually > writing data. > The two patches are at: > http://www.hanik.com/covalent/amq/patch-1-threadname-filter.patch > http://www.hanik.com/covalent/amq/patch-3-tcp-writetimeout.patch > the binary .jar applies to both 5.1 and trunk and can be used today in > existing environments. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (AMQ-1993) Systems hang due to inability to timeout socket write operation
[ https://issues.apache.org/activemq/browse/AMQ-1993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Ehm updated AMQ-1993: --- Attachment: jstack.blockedSession jstack thread dump > Systems hang due to inability to timeout socket write operation > --- > > Key: AMQ-1993 > URL: https://issues.apache.org/activemq/browse/AMQ-1993 > Project: ActiveMQ > Issue Type: Bug > Components: Broker >Affects Versions: 5.1.0, 5.2.0 > Environment: Unix (Solaris and Linux tested) >Reporter: Filip Hanik >Assignee: Gary Tully >Priority: Critical > Fix For: 5.2.0 > > Attachments: jstack.blockedSession, patch-1-threadname-filter.patch, > patch-3-tcp-writetimeout.patch > > > the blocking Java Socket API doesn't have a timeout on socketWrite > invocations. > This means, if a TCP session is dropped or terminated without RST or FIN > packets, the operating system it left to eventually time out the session. On > the linux kernel this timeout usually takes 15 to 30minutes. > For this entire period, the AMQ server hangs, and producers and consumers are > unable to use a topic. > I have created two patches for this at the page: > http://www.hanik.com/covalent/amq/index.html > Let me show a bit more > - > "ActiveMQ Transport: tcp:///X.YYY.XXX.:2011" daemon prio=10 > tid=0x55d39000 nid=0xc78 runnable > [0x447c9000..0x447cac10] >java.lang.Thread.State: RUNNABLE > at java.net.SocketOutputStream.socketWrite0(Native Method) > at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92) > This is a thread stuck in blocking IO, and can be stuck for 30 minutes during > the kernel TCP retransmission attempts. > Unfortunately the thread dump is very misleading since the name of the > thread, is not the destination or even remotely related to the socket it is > operating on. > To mend this, a very simple (and configurable) ThreadNameFilter has been > suggested to the patch, that appends the destination and helps the system > administrator correctly identify the client that is about to receive data. > --- > at org.apache.activemq.broker.region.Topic.dispatch(Topic.java:581) > at org.apache.activemq.broker.region.Topic.doMessageSend(Topic.java:421) > - locked <0x2aaaec155818> (a > org.apache.activemq.broker.region.Topic) > at org.apache.activemq.broker.region.Topic.send(Topic.java:363) > The lock being held at this issue unfortunately makes the entire Topic single > threaded. > When this lock is being held, no other clients (producers and consumers) can > publish to/receive from this topic. > And this lock can hold up to 30 minutes. > I consider solving this single threaded behavior a 'feature enhancement' that > should be handled separately from this bug. Because even if it is solved, > threads still risk being stuck in socketWrite0 for dropped connections that > still appear to be established. > For this, I have implemented a socket timeout filter, based on a > TransportFilter, this filter only times out connections that are actually > writing data. > The two patches are at: > http://www.hanik.com/covalent/amq/patch-1-threadname-filter.patch > http://www.hanik.com/covalent/amq/patch-3-tcp-writetimeout.patch > the binary .jar applies to both 5.1 and trunk and can be used today in > existing environments. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (AMQ-1993) Systems hang due to inability to timeout socket write operation
[ https://issues.apache.org/activemq/browse/AMQ-1993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=52283#action_52283 ] Felix Ehm edited comment on AMQ-1993 at 6/16/09 2:10 AM: - Hi, I am using ActiveMQ 5.3.0 and apparently run into this reported problem again. Our clients weren't able to get any sessions and I first thought that it is a problem with the CopyOnWriteArray Object where consumers are kept. But then I saw one thread in RUNNABLE locking 3 mutexes and writing using java.net.SocketOutputStream.socketWrite0. I attached the thread dump file for investigations. Unfortunately I can't reproduce this situation with a unit test. The only additional information I got from our client was that he tried to restarted the connection within very short period (1-2 sec). was (Author: felixehm): Hi, I am using ActiveMQ 5.3.0 and apparently run into this reported problem again. Our clients weren't able to get any sessions and I first thought that it is a problem with the CopyOnWriteArray Object where consumers are kept. But then I saw one thread in RUNNABLE locking 3 mutexes and writing using java.net.SocketOutputStream.socketWrite0. I attached the thread dump file for investigations. > Systems hang due to inability to timeout socket write operation > --- > > Key: AMQ-1993 > URL: https://issues.apache.org/activemq/browse/AMQ-1993 > Project: ActiveMQ > Issue Type: Bug > Components: Broker >Affects Versions: 5.1.0, 5.2.0 > Environment: Unix (Solaris and Linux tested) >Reporter: Filip Hanik >Assignee: Gary Tully >Priority: Critical > Fix For: 5.2.0 > > Attachments: patch-1-threadname-filter.patch, > patch-3-tcp-writetimeout.patch > > > the blocking Java Socket API doesn't have a timeout on socketWrite > invocations. > This means, if a TCP session is dropped or terminated without RST or FIN > packets, the operating system it left to eventually time out the session. On > the linux kernel this timeout usually takes 15 to 30minutes. > For this entire period, the AMQ server hangs, and producers and consumers are > unable to use a topic. > I have created two patches for this at the page: > http://www.hanik.com/covalent/amq/index.html > Let me show a bit more > - > "ActiveMQ Transport: tcp:///X.YYY.XXX.:2011" daemon prio=10 > tid=0x55d39000 nid=0xc78 runnable > [0x447c9000..0x447cac10] >java.lang.Thread.State: RUNNABLE > at java.net.SocketOutputStream.socketWrite0(Native Method) > at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92) > This is a thread stuck in blocking IO, and can be stuck for 30 minutes during > the kernel TCP retransmission attempts. > Unfortunately the thread dump is very misleading since the name of the > thread, is not the destination or even remotely related to the socket it is > operating on. > To mend this, a very simple (and configurable) ThreadNameFilter has been > suggested to the patch, that appends the destination and helps the system > administrator correctly identify the client that is about to receive data. > --- > at org.apache.activemq.broker.region.Topic.dispatch(Topic.java:581) > at org.apache.activemq.broker.region.Topic.doMessageSend(Topic.java:421) > - locked <0x2aaaec155818> (a > org.apache.activemq.broker.region.Topic) > at org.apache.activemq.broker.region.Topic.send(Topic.java:363) > The lock being held at this issue unfortunately makes the entire Topic single > threaded. > When this lock is being held, no other clients (producers and consumers) can > publish to/receive from this topic. > And this lock can hold up to 30 minutes. > I consider solving this single threaded behavior a 'feature enhancement' that > should be handled separately from this bug. Because even if it is solved, > threads still risk being stuck in socketWrite0 for dropped connections that > still appear to be established. > For this, I have implemented a socket timeout filter, based on a > TransportFilter, this filter only times out connections that are actually > writing data. > The two patches are at: > http://www.hanik.com/covalent/amq/patch-1-threadname-filter.patch > http://www.hanik.com/covalent/amq/patch-3-tcp-writetimeout.patch > the binary .jar applies to both 5.1 and trunk and can be used today in > existing environments. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Reopened: (AMQ-1993) Systems hang due to inability to timeout socket write operation
[ https://issues.apache.org/activemq/browse/AMQ-1993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Ehm reopened AMQ-1993: Hi, I am using ActiveMQ 5.3.0 and apparently run into this reported problem again. Our clients weren't able to get any sessions and I first thought that it is a problem with the CopyOnWriteArray Object where consumers are kept. But then I saw one thread in RUNNABLE locking 3 mutexes and writing using java.net.SocketOutputStream.socketWrite0. I attached the thread dump file for investigations. > Systems hang due to inability to timeout socket write operation > --- > > Key: AMQ-1993 > URL: https://issues.apache.org/activemq/browse/AMQ-1993 > Project: ActiveMQ > Issue Type: Bug > Components: Broker >Affects Versions: 5.1.0, 5.2.0 > Environment: Unix (Solaris and Linux tested) >Reporter: Filip Hanik >Assignee: Gary Tully >Priority: Critical > Fix For: 5.2.0 > > Attachments: patch-1-threadname-filter.patch, > patch-3-tcp-writetimeout.patch > > > the blocking Java Socket API doesn't have a timeout on socketWrite > invocations. > This means, if a TCP session is dropped or terminated without RST or FIN > packets, the operating system it left to eventually time out the session. On > the linux kernel this timeout usually takes 15 to 30minutes. > For this entire period, the AMQ server hangs, and producers and consumers are > unable to use a topic. > I have created two patches for this at the page: > http://www.hanik.com/covalent/amq/index.html > Let me show a bit more > - > "ActiveMQ Transport: tcp:///X.YYY.XXX.:2011" daemon prio=10 > tid=0x55d39000 nid=0xc78 runnable > [0x447c9000..0x447cac10] >java.lang.Thread.State: RUNNABLE > at java.net.SocketOutputStream.socketWrite0(Native Method) > at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92) > This is a thread stuck in blocking IO, and can be stuck for 30 minutes during > the kernel TCP retransmission attempts. > Unfortunately the thread dump is very misleading since the name of the > thread, is not the destination or even remotely related to the socket it is > operating on. > To mend this, a very simple (and configurable) ThreadNameFilter has been > suggested to the patch, that appends the destination and helps the system > administrator correctly identify the client that is about to receive data. > --- > at org.apache.activemq.broker.region.Topic.dispatch(Topic.java:581) > at org.apache.activemq.broker.region.Topic.doMessageSend(Topic.java:421) > - locked <0x2aaaec155818> (a > org.apache.activemq.broker.region.Topic) > at org.apache.activemq.broker.region.Topic.send(Topic.java:363) > The lock being held at this issue unfortunately makes the entire Topic single > threaded. > When this lock is being held, no other clients (producers and consumers) can > publish to/receive from this topic. > And this lock can hold up to 30 minutes. > I consider solving this single threaded behavior a 'feature enhancement' that > should be handled separately from this bug. Because even if it is solved, > threads still risk being stuck in socketWrite0 for dropped connections that > still appear to be established. > For this, I have implemented a socket timeout filter, based on a > TransportFilter, this filter only times out connections that are actually > writing data. > The two patches are at: > http://www.hanik.com/covalent/amq/patch-1-threadname-filter.patch > http://www.hanik.com/covalent/amq/patch-3-tcp-writetimeout.patch > the binary .jar applies to both 5.1 and trunk and can be used today in > existing environments. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (AMQ-975) Setting "time to live" to >0 fails delivery from linux to window
[ https://issues.apache.org/activemq/browse/AMQ-975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=50273#action_50273 ] Felix Ehm commented on AMQ-975: --- Hi Dmitry, have you checked that the time on your machine is synchronized? If not there is a high chance that the message expires before the other machine can get it. Then you will never be able to receive it. ActiveMQSession.send(..) does the following to determine the expiration : expiration = timeToLive + System.currentTimeMillis(); Cheers, Felix > Setting "time to live" to >0 fails delivery from linux to window > > > Key: AMQ-975 > URL: https://issues.apache.org/activemq/browse/AMQ-975 > Project: ActiveMQ > Issue Type: Bug > Components: JMS client >Affects Versions: 4.0, 4.0.1 > Environment: Linux, window XP >Reporter: Wallaec Wong >Assignee: Rob Davies > Fix For: 5.0.0 > > > I found this problem on 4.0 and 4.0.1 running on Linux. > Problem reproduction steps: > 1. start activemq server on a linux server > 2. start a consumer client on a window XP > 3. start a producer client on same linux machine, setting timeToLive to any > value > 0 > 4. consumer client will not receive any messages > 5. set the time to live to 0, the consumer client will receive messages, or > 6. set time to live to 0 or not, consumer client running on linux will > receive messages too, or > 7. no problem too if producer runs on xp and consumer runs on linux > And apparently it does not really matter where the activemq server runs, > linix or xp. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
