[jira] [Commented] (AMQ-4986) tightEncoding=false + failover triggers Broker memory exhaust
[
https://issues.apache.org/jira/browse/AMQ-4986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14260850#comment-14260850
]
siddeshwar_hiremath commented on AMQ-4986:
--
Made following change:-
activemq.xml:-
connection.uri=nio://localhost:61616
Client broker url:-
failover:(nio://
127.0.0.1:61616?keepAlive=true&wireFormat.maxFrameSize=104857600)?jms.blobTransferPolicy.defaultUploadUrl=http://127.0.0.1:8161/fileserver/&initialReconnectDelay=1&useExponentialBackOff=false&randomize=false
I have the above setting and experiencing following error:-
nio protocol:-
---
2014-12-29 11:52:43,145 | DEBUG | Sending: WireFormatInfo { version=10,
properties={MaxFrameSize=9223372036854775807, CacheSize=1024,
CacheEnabled=true, SizePrefixDisabled=false,
MaxInactivityDurationInitalDelay=1, TcpNoDelayEnabled=true,
MaxInactivityDuration=3, TightEncodingEnabled=true,
StackTraceEnabled=true}, magic=[A,c,t,i,v,e,M,Q]} |
org.apache.activemq.transport.WireFormatNegotiator | ActiveMQ
BrokerService[localhost] Task-4
2014-12-29 11:52:43,591 | DEBUG | Transport Connection to:
tcp://127.0.0.1:51048 failed: java.io.IOException: Java heap space |
org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ NIO Worker 5
java.io.IOException: Java heap space
at
org.apache.activemq.util.IOExceptionSupport.create(IOExceptionSupport.java:39)[activemq-client-5.10.0.jar:5.10.0]
at
org.apache.activemq.transport.nio.NIOTransport.serviceRead(NIOTransport.java:151)[activemq-client-5.10.0.jar:5.10.0]
at
org.apache.activemq.transport.nio.NIOTransport$1.onSelect(NIOTransport.java:69)[activemq-client-5.10.0.jar:5.10.0]
at
org.apache.activemq.transport.nio.SelectorSelection.onSelect(SelectorSelection.java:94)[activemq-client-5.10.0.jar:5.10.0]
at
org.apache.activemq.transport.nio.SelectorWorker$1.run(SelectorWorker.java:119)[activemq-client-5.10.0.jar:5.10.0]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)[:1.7.0_40]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)[:1.7.0_40]
at java.lang.Thread.run(Thread.java:724)[:1.7.0_40]
2014-12-29 11:52:43,592 | DEBUG | Unregistering MBean
org.apache.activemq:type=Broker,brokerName=localhost,connector=clientConnectors,connectorName=nio_//localhost_61616&qe;transport.keepAlive&true&transport.wireFormat.maxFrameSize&104857600,connectionViewType=remoteAddress,connectionName=tcp_//127.0.0.1_51048
| org.apache.activemq.broker.jmx.ManagementContext | ActiveMQ NIO Worker 5
2014-12-29 11:52:43,605 | DEBUG | Stopping connection: tcp://127.0.0.1:51048 |
org.apache.activemq.broker.TransportConnection | ActiveMQ
BrokerService[localhost] Task-5
2014-12-29 11:52:43,609 | DEBUG | Stopping transport
tcp:///127.0.0.1:51048@61616 | org.apache.activemq.transport.tcp.TcpTransport |
ActiveMQ BrokerService[localhost] Task-5
2014-12-29 11:52:43,609 | DEBUG | Initialized TaskRunnerFactory[ActiveMQ Task]
using ExecutorService: null | org.apache.activemq.thread.TaskRunnerFactory |
ActiveMQ BrokerService[localhost] Task-5
2014-12-29 11:52:43,614 | DEBUG | Closed socket Socket[unconnected] |
org.apache.activemq.transport.tcp.TcpTransport | ActiveMQ Task-1
2014-12-29 11:52:43,614 | DEBUG | Stopped transport: null |
org.apache.activemq.broker.TransportConnection | ActiveMQ
BrokerService[localhost] Task-5
2014-12-29 11:52:43,614 | DEBUG | Connection Stopped: null |
org.apache.activemq.broker.TransportConnection | ActiveMQ
BrokerService[localhost] Task-5
-
With tcp protocol:-
-
ityDurationInitalDelay=1, TcpNoDelayEnabled=true,
MaxInactivityDuration=3, TightEncodingEnabled=true,
StackTraceEnabled=true}, magic=[A,c,t,i,v,e,M,Q]} |
org.apache.activemq.transport.WireFormatNegotiator | ActiveMQ
BrokerService[localhost] Task-8
2014-12-29 11:22:42,797 | DEBUG | Transport Connection to:
tcp://127.0.0.1:39382 failed: java.io.IOException: Unknown data type: 13 |
org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ Transport:
tcp:///127.0.0.1:39382@61616
java.io.IOException: Unknown data type: 13
at
org.apache.activemq.openwire.OpenWireFormat.looseUnmarshalNestedObject(OpenWireFormat.java:460)[activemq-client-5.10.0.jar:5.10.0]
at
org.apache.activemq.openwire.v10.BaseDataStreamMarshaller.looseUnmarsalNestedObject(BaseDataStreamMarshaller.java:466)[activemq-client-5.10.0.jar:5.10.0]
at
org.apache.activemq.openwire.v10.DataResponseMarshaller.looseUnmarshal(DataResponseMarshaller.java:113)[activemq-client-5.10.0.jar:5.10.0]
at
org.apache.activemq.openwir
[jira] [Commented] (AMQ-4986) tightEncoding=false + failover triggers Broker memory exhaust
[
https://issues.apache.org/jira/browse/AMQ-4986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14223146#comment-14223146
]
Felix Ehm commented on AMQ-4986:
In case this is of interest for you: I have removed this option from the URL
and was able to upgrade slowly the brokers.
I haven't really found a solution for this problem.
> tightEncoding=false + failover triggers Broker memory exhaust
> -
>
> Key: AMQ-4986
> URL: https://issues.apache.org/jira/browse/AMQ-4986
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
>Affects Versions: 5.6.0, 5.7.0, 5.8.0, 5.9.0
> Environment: java 1.7
> Client library 5.5.1
>Reporter: Felix Ehm
> Fix For: NEEDS_REVIEW
>
>
> We experience this problem in combination with 5.5.1 client and the
> wireformat.tightEncodingEnabled=false+ failover:
> Scenario:
> 1. start standard broker
> 2. start Client (with e.g. a MessageListener) with failover protocol: e.g.
> failover:(tcp://123.123.123.123:61616?wireformat.tightEncodingEnabled=false)
> 3. wait around 30sec (default for inactivity check)
> Result:
> The client closes the connection and re-tries to the broker which in turn
> throws the following exception:
> {code}
> 2014-01-21 20:12:49,568 [ActiveMQ Transport:
> tcp:///123.123.123.123:60156@61616] DEBUG Transport Transport Connection to:
> tcp://124.124.124.124:60156 failed: java.io.IOException: Unexpected error
> occured: java.lang.OutOfMemoryError: Java heap space
> java.io.IOException: Unexpected error occured: java.lang.OutOfMemoryError:
> Java heap space
> at
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:203)
> at java.lang.Thread.run(Thread.java:722)
> Caused by: java.lang.OutOfMemoryError: Java heap space
> at
> org.apache.activemq.openwire.v8.BaseDataStreamMarshaller.looseUnmarshalByteSequence(BaseDataStreamMarshaller.java:638)
> at
> org.apache.activemq.openwire.v8.WireFormatInfoMarshaller.looseUnmarshal(WireFormatInfoMarshaller.java:132)
> at
> org.apache.activemq.openwire.OpenWireFormat.doUnmarshal(OpenWireFormat.java:373)
> at
> org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:285)
> at
> org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:221)
> at
> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:213)
> at
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:196)
> ... 1 more
> {code}
> The problem here is that the BaseDataStreamMarshaller reads an int from the
> buffer and re-uses it immediately to allocate a byte array:
> {code}
> protected byte[] looseUnmarshalByteArray(DataInput dataIn) throws IOException
> {
> byte rc[] = null;
> if (dataIn.readBoolean()) {
> int size = dataIn.readInt();
> rc = new byte[size]; // PROBLEM! What happens if size has been
> read and interpreted wrongly ?
> dataIn.readFully(rc);
> }
> return rc;
> }
> {code}
> In our case the dadaIn.readInt() read an int number of 785.477.224 which
> triggers the broker to allocate blindly this amount of mem.
> We do not know yet what triggers the wrong byte sequence from the client, but
> on the brokers side, there should be a protection against this case.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (AMQ-4986) tightEncoding=false + failover triggers Broker memory exhaust
[
https://issues.apache.org/jira/browse/AMQ-4986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14223128#comment-14223128
]
Michael Dillon commented on AMQ-4986:
-
We are having similar issues as described above. The clients are running the
5.10 library and the broker is also 5.10. We see the client subscriber
reconnecting every 30 seconds. The difference is that we have the default
setting for tightEncodingEnabled=true. As a test we set this to false on our
client and the reconnecting issue went away. Our concern now is that we will
experience the OutOfMemory issue as described above. It would be great if we
could get some guidance on this issue.
> tightEncoding=false + failover triggers Broker memory exhaust
> -
>
> Key: AMQ-4986
> URL: https://issues.apache.org/jira/browse/AMQ-4986
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
>Affects Versions: 5.6.0, 5.7.0, 5.8.0, 5.9.0
> Environment: java 1.7
> Client library 5.5.1
>Reporter: Felix Ehm
> Fix For: NEEDS_REVIEW
>
>
> We experience this problem in combination with 5.5.1 client and the
> wireformat.tightEncodingEnabled=false+ failover:
> Scenario:
> 1. start standard broker
> 2. start Client (with e.g. a MessageListener) with failover protocol: e.g.
> failover:(tcp://123.123.123.123:61616?wireformat.tightEncodingEnabled=false)
> 3. wait around 30sec (default for inactivity check)
> Result:
> The client closes the connection and re-tries to the broker which in turn
> throws the following exception:
> {code}
> 2014-01-21 20:12:49,568 [ActiveMQ Transport:
> tcp:///123.123.123.123:60156@61616] DEBUG Transport Transport Connection to:
> tcp://124.124.124.124:60156 failed: java.io.IOException: Unexpected error
> occured: java.lang.OutOfMemoryError: Java heap space
> java.io.IOException: Unexpected error occured: java.lang.OutOfMemoryError:
> Java heap space
> at
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:203)
> at java.lang.Thread.run(Thread.java:722)
> Caused by: java.lang.OutOfMemoryError: Java heap space
> at
> org.apache.activemq.openwire.v8.BaseDataStreamMarshaller.looseUnmarshalByteSequence(BaseDataStreamMarshaller.java:638)
> at
> org.apache.activemq.openwire.v8.WireFormatInfoMarshaller.looseUnmarshal(WireFormatInfoMarshaller.java:132)
> at
> org.apache.activemq.openwire.OpenWireFormat.doUnmarshal(OpenWireFormat.java:373)
> at
> org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:285)
> at
> org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:221)
> at
> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:213)
> at
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:196)
> ... 1 more
> {code}
> The problem here is that the BaseDataStreamMarshaller reads an int from the
> buffer and re-uses it immediately to allocate a byte array:
> {code}
> protected byte[] looseUnmarshalByteArray(DataInput dataIn) throws IOException
> {
> byte rc[] = null;
> if (dataIn.readBoolean()) {
> int size = dataIn.readInt();
> rc = new byte[size]; // PROBLEM! What happens if size has been
> read and interpreted wrongly ?
> dataIn.readFully(rc);
> }
> return rc;
> }
> {code}
> In our case the dadaIn.readInt() read an int number of 785.477.224 which
> triggers the broker to allocate blindly this amount of mem.
> We do not know yet what triggers the wrong byte sequence from the client, but
> on the brokers side, there should be a protection against this case.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (AMQ-4986) tightEncoding=false + failover triggers Broker memory exhaust
[
https://issues.apache.org/jira/browse/AMQ-4986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13878723#comment-13878723
]
Felix Ehm commented on AMQ-4986:
Yes. We have a very large environment with > 200 applications where we are not
in the situation to be able to update client and broker setups always together
(all openwire).
According to Hiram this mixing should be ok and from my understanding the
openwire protocol should at least make sure the communication is fine. But
exactly here is the point where it breaks.
As I wrote, we're still trying to find out why the broker reads something
buggy from the wire from the client.
Did I mention that the client and broker come into a CON/DISCON loop ? The
client-side failover receives a invalid magic back and closes the connection,
and reconnects, and so on
The workaround of course is to remove this option, which I strongly consider.
Independently from this, the presented code should be looked at to prevent this
allocation. What do you think?
> tightEncoding=false + failover triggers Broker memory exhaust
> -
>
> Key: AMQ-4986
> URL: https://issues.apache.org/jira/browse/AMQ-4986
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
>Affects Versions: 5.6.0, 5.7.0, 5.8.0, 5.9.0
> Environment: java 1.7
> Client library 5.5.1
>Reporter: Felix Ehm
>Priority: Critical
> Fix For: 5.10.0
>
>
> We experience this problem in combination with 5.5.1 client and the
> wireformat.tightEncodingEnabled=false+ failover:
> Scenario:
> 1. start standard broker
> 2. start Client (with e.g. a MessageListener) with failover protocol: e.g.
> failover:(tcp://123.123.123.123:61616?wireformat.tightEncodingEnabled=false)
> 3. wait around 30sec (default for inactivity check)
> Result:
> The client closes the connection and re-tries to the broker which in turn
> throws the following exception:
> {code}
> 2014-01-21 20:12:49,568 [ActiveMQ Transport:
> tcp:///123.123.123.123:60156@61616] DEBUG Transport Transport Connection to:
> tcp://124.124.124.124:60156 failed: java.io.IOException: Unexpected error
> occured: java.lang.OutOfMemoryError: Java heap space
> java.io.IOException: Unexpected error occured: java.lang.OutOfMemoryError:
> Java heap space
> at
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:203)
> at java.lang.Thread.run(Thread.java:722)
> Caused by: java.lang.OutOfMemoryError: Java heap space
> at
> org.apache.activemq.openwire.v8.BaseDataStreamMarshaller.looseUnmarshalByteSequence(BaseDataStreamMarshaller.java:638)
> at
> org.apache.activemq.openwire.v8.WireFormatInfoMarshaller.looseUnmarshal(WireFormatInfoMarshaller.java:132)
> at
> org.apache.activemq.openwire.OpenWireFormat.doUnmarshal(OpenWireFormat.java:373)
> at
> org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:285)
> at
> org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:221)
> at
> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:213)
> at
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:196)
> ... 1 more
> {code}
> The problem here is that the BaseDataStreamMarshaller reads an int from the
> buffer and re-uses it immediately to allocate a byte array:
> {code}
> protected byte[] looseUnmarshalByteArray(DataInput dataIn) throws IOException
> {
> byte rc[] = null;
> if (dataIn.readBoolean()) {
> int size = dataIn.readInt();
> rc = new byte[size]; // PROBLEM! What happens if size has been
> read and interpreted wrongly ?
> dataIn.readFully(rc);
> }
> return rc;
> }
> {code}
> In our case the dadaIn.readInt() read an int number of 785.477.224 which
> triggers the broker to allocate blindly this amount of mem.
> We do not know yet what triggers the wrong byte sequence from the client, but
> on the brokers side, there should be a protection against this case.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (AMQ-4986) tightEncoding=false + failover triggers Broker memory exhaust
[
https://issues.apache.org/jira/browse/AMQ-4986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13878708#comment-13878708
]
Timothy Bish commented on AMQ-4986:
---
Are you mixing old versions of clients with newer broker versions ?
> tightEncoding=false + failover triggers Broker memory exhaust
> -
>
> Key: AMQ-4986
> URL: https://issues.apache.org/jira/browse/AMQ-4986
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
>Affects Versions: 5.6.0, 5.7.0, 5.8.0, 5.9.0
> Environment: java 1.7
> Client library 5.5.1
>Reporter: Felix Ehm
>Priority: Critical
> Fix For: 5.10.0
>
>
> We experience this problem in combination with 5.5.1 client and the
> wireformat.tightEncodingEnabled=false+ failover:
> Scenario:
> 1. start standard broker
> 2. start Client (with e.g. a MessageListener) with failover protocol: e.g.
> failover:(tcp://123.123.123.123:61616?wireformat.tightEncodingEnabled=false)
> 3. wait around 30sec (default for inactivity check)
> Result:
> The client closes the connection and re-tries to the broker which in turn
> throws the following exception:
> {code}
> 2014-01-21 20:12:49,568 [ActiveMQ Transport:
> tcp:///123.123.123.123:60156@61616] DEBUG Transport Transport Connection to:
> tcp://124.124.124.124:60156 failed: java.io.IOException: Unexpected error
> occured: java.lang.OutOfMemoryError: Java heap space
> java.io.IOException: Unexpected error occured: java.lang.OutOfMemoryError:
> Java heap space
> at
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:203)
> at java.lang.Thread.run(Thread.java:722)
> Caused by: java.lang.OutOfMemoryError: Java heap space
> at
> org.apache.activemq.openwire.v8.BaseDataStreamMarshaller.looseUnmarshalByteSequence(BaseDataStreamMarshaller.java:638)
> at
> org.apache.activemq.openwire.v8.WireFormatInfoMarshaller.looseUnmarshal(WireFormatInfoMarshaller.java:132)
> at
> org.apache.activemq.openwire.OpenWireFormat.doUnmarshal(OpenWireFormat.java:373)
> at
> org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:285)
> at
> org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:221)
> at
> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:213)
> at
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:196)
> ... 1 more
> {code}
> The problem here is that the BaseDataStreamMarshaller reads an int from the
> buffer and re-uses it immediately to allocate a byte array:
> {code}
> protected byte[] looseUnmarshalByteArray(DataInput dataIn) throws IOException
> {
> byte rc[] = null;
> if (dataIn.readBoolean()) {
> int size = dataIn.readInt();
> rc = new byte[size]; // PROBLEM! What happens if size has been
> read and interpreted wrongly ?
> dataIn.readFully(rc);
> }
> return rc;
> }
> {code}
> In our case the dadaIn.readInt() read an int number of 785.477.224 which
> triggers the broker to allocate blindly this amount of mem.
> We do not know yet what triggers the wrong byte sequence from the client, but
> on the brokers side, there should be a protection against this case.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
