[jira] [Commented] (AMQ-3819) high cpu with stomp+nio+ssl and many subscriptions
[
https://issues.apache.org/jira/browse/AMQ-3819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13264985#comment-13264985
]
Dejan Bosanac commented on AMQ-3819:
I created a fix in svn revision 1332230 and created a new snapshot
https://repository.apache.org/content/repositories/snapshots/org/apache/activemq/apache-activemq/5.7-SNAPSHOT/
It fixes the problem that transport get stuck in serviceRead() method and from
my local test it looks pretty much the same as plain stomp+ssl implementation.
Can you take it for a spin and let us know if it improves your use case.
high cpu with stomp+nio+ssl and many subscriptions
--
Key: AMQ-3819
URL: https://issues.apache.org/jira/browse/AMQ-3819
Project: ActiveMQ
Issue Type: Bug
Components: stomp
Affects Versions: 5.6.0
Environment: CentOS 6, RC of 5.6.0
java version 1.6.0_22
OpenJDK Runtime Environment (IcedTea6 1.10.6) (rhel-1.43.1.10.6.el6_2-x86_64)
OpenJDK 64-Bit Server VM (build 20.0-b11, mixed mode)
Reporter: R.I.Pienaar
Attachments: StompLoadTest.java, StompNIOSSLLoadTest.java
Switching an existing workload from a transport:
{quote}
transportConnector name=verified_stompssl
uri=stomp+ssl://0.0.0.0:6165?needClientAuth=true/
{quote}
to
{quote}
transportConnector name=verified_stompssl
uri=stomp+nio+ssl://0.0.0.0:6165?needClientAuth=true/
{quote}
showed the CPU profile to go from 1-5% to 300% constantly on a 8 core server
I was able to recreate this using a ruby client @
http://devco.net/rip/amq_560_stomp_nio_ssl_tester.rb
The important combinations are:
* I am connecting to a stomp+nio+ssl port
* I am creating the subscriptions to the 10 queus and topics
If I change either of these variables - like just commenting out the loop
that does those subscriptions - then the CPU load is acceptable.
I analysed the running VM with VisualVM and found that
transport.nio.NIOSSLTransport.serviceRead() is the busy thread.
My activemq.xml is:
{noformat}
beans
xmlns=http://www.springframework.org/schema/beans;
xmlns:amq=http://activemq.apache.org/schema/core;
xmlns:xsi=http://www.w3.org/2001/XMLSchema-instance;
xsi:schemaLocation=http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
http://activemq.apache.org/schema/core
http://activemq.apache.org/schema/core/activemq-core.xsd
http://activemq.apache.org/camel/schema/spring
http://activemq.apache.org/camel/schema/spring/camel-spring.xsd;
broker xmlns=http://activemq.apache.org/schema/core; brokerName=amq1
useJmx=true persistent=true schedulePeriodForDestinationPurge=6
destinationPolicy
policyMap
policyEntries
policyEntry topic= producerFlowControl=false/
policyEntry queue=*.reply. gcInactiveDestinations=true
inactiveTimoutBeforeGC=12 /
/policyEntries
/policyMap
/destinationPolicy
managementContext
managementContext connectorPort=1099
jmxDomainName=org.apache.activemq/
/managementContext
plugins
statisticsBrokerPlugin/
simpleAuthenticationPlugin
users
authenticationUser username=test password=test
groups=admins,everyone/
/users
/simpleAuthenticationPlugin
authorizationPlugin
map
authorizationMap
authorizationEntries
authorizationEntry queue= write=admins read=admins
admin=admins /
authorizationEntry topic= write=admins read=admins
admin=admins /
/authorizationEntries
/authorizationMap
/map
/authorizationPlugin
/plugins
sslContext
sslContext
keyStore=keystore.jks keyStorePassword=ohshahCu
trustStore=truststore.jks trustStorePassword=ohshahCu
/
/sslContext
systemUsage
systemUsage
memoryUsage
memoryUsage limit=200 mb /
/memoryUsage
storeUsage
storeUsage limit=1 gb /
/storeUsage
tempUsage
tempUsage limit=1 gb /
/tempUsage
/systemUsage
/systemUsage
transportConnectors
transportConnector name=openwire uri=tcp://0.0.0.0:6166/
transportConnector name=stomp+nio
uri=stomp+nio://0.0.0.0:6163/
transportConnector name=stompssl
uri=stomp+ssl://0.0.0.0:6164/
transportConnector name=verified_stompssl
uri=stomp+nio+ssl://0.0.0.0:6165?needClientAuth=true/
/transportConnectors
/broker
[jira] [Commented] (AMQ-3819) high cpu with stomp+nio+ssl and many subscriptions
[
https://issues.apache.org/jira/browse/AMQ-3819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13264993#comment-13264993
]
Timothy Bish commented on AMQ-3819:
---
Tested here with the fix Dejan posted and the NIO+SSL test works as expected
now on my machine. Would be good confirmation to see if that ruby client test
is working now.
high cpu with stomp+nio+ssl and many subscriptions
--
Key: AMQ-3819
URL: https://issues.apache.org/jira/browse/AMQ-3819
Project: ActiveMQ
Issue Type: Bug
Components: stomp
Affects Versions: 5.6.0
Environment: CentOS 6, RC of 5.6.0
java version 1.6.0_22
OpenJDK Runtime Environment (IcedTea6 1.10.6) (rhel-1.43.1.10.6.el6_2-x86_64)
OpenJDK 64-Bit Server VM (build 20.0-b11, mixed mode)
Reporter: R.I.Pienaar
Attachments: StompLoadTest.java, StompNIOSSLLoadTest.java
Switching an existing workload from a transport:
{quote}
transportConnector name=verified_stompssl
uri=stomp+ssl://0.0.0.0:6165?needClientAuth=true/
{quote}
to
{quote}
transportConnector name=verified_stompssl
uri=stomp+nio+ssl://0.0.0.0:6165?needClientAuth=true/
{quote}
showed the CPU profile to go from 1-5% to 300% constantly on a 8 core server
I was able to recreate this using a ruby client @
http://devco.net/rip/amq_560_stomp_nio_ssl_tester.rb
The important combinations are:
* I am connecting to a stomp+nio+ssl port
* I am creating the subscriptions to the 10 queus and topics
If I change either of these variables - like just commenting out the loop
that does those subscriptions - then the CPU load is acceptable.
I analysed the running VM with VisualVM and found that
transport.nio.NIOSSLTransport.serviceRead() is the busy thread.
My activemq.xml is:
{noformat}
beans
xmlns=http://www.springframework.org/schema/beans;
xmlns:amq=http://activemq.apache.org/schema/core;
xmlns:xsi=http://www.w3.org/2001/XMLSchema-instance;
xsi:schemaLocation=http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
http://activemq.apache.org/schema/core
http://activemq.apache.org/schema/core/activemq-core.xsd
http://activemq.apache.org/camel/schema/spring
http://activemq.apache.org/camel/schema/spring/camel-spring.xsd;
broker xmlns=http://activemq.apache.org/schema/core; brokerName=amq1
useJmx=true persistent=true schedulePeriodForDestinationPurge=6
destinationPolicy
policyMap
policyEntries
policyEntry topic= producerFlowControl=false/
policyEntry queue=*.reply. gcInactiveDestinations=true
inactiveTimoutBeforeGC=12 /
/policyEntries
/policyMap
/destinationPolicy
managementContext
managementContext connectorPort=1099
jmxDomainName=org.apache.activemq/
/managementContext
plugins
statisticsBrokerPlugin/
simpleAuthenticationPlugin
users
authenticationUser username=test password=test
groups=admins,everyone/
/users
/simpleAuthenticationPlugin
authorizationPlugin
map
authorizationMap
authorizationEntries
authorizationEntry queue= write=admins read=admins
admin=admins /
authorizationEntry topic= write=admins read=admins
admin=admins /
/authorizationEntries
/authorizationMap
/map
/authorizationPlugin
/plugins
sslContext
sslContext
keyStore=keystore.jks keyStorePassword=ohshahCu
trustStore=truststore.jks trustStorePassword=ohshahCu
/
/sslContext
systemUsage
systemUsage
memoryUsage
memoryUsage limit=200 mb /
/memoryUsage
storeUsage
storeUsage limit=1 gb /
/storeUsage
tempUsage
tempUsage limit=1 gb /
/tempUsage
/systemUsage
/systemUsage
transportConnectors
transportConnector name=openwire uri=tcp://0.0.0.0:6166/
transportConnector name=stomp+nio
uri=stomp+nio://0.0.0.0:6163/
transportConnector name=stompssl
uri=stomp+ssl://0.0.0.0:6164/
transportConnector name=verified_stompssl
uri=stomp+nio+ssl://0.0.0.0:6165?needClientAuth=true/
/transportConnectors
/broker
import resource=jetty.xml/
/beans
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
[jira] [Commented] (AMQ-3819) high cpu with stomp+nio+ssl and many subscriptions
[
https://issues.apache.org/jira/browse/AMQ-3819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13264996#comment-13264996
]
R.I.Pienaar commented on AMQ-3819:
--
Thanks guys, I'll test 'morrow UK time.
high cpu with stomp+nio+ssl and many subscriptions
--
Key: AMQ-3819
URL: https://issues.apache.org/jira/browse/AMQ-3819
Project: ActiveMQ
Issue Type: Bug
Components: stomp
Affects Versions: 5.6.0
Environment: CentOS 6, RC of 5.6.0
java version 1.6.0_22
OpenJDK Runtime Environment (IcedTea6 1.10.6) (rhel-1.43.1.10.6.el6_2-x86_64)
OpenJDK 64-Bit Server VM (build 20.0-b11, mixed mode)
Reporter: R.I.Pienaar
Attachments: StompLoadTest.java, StompNIOSSLLoadTest.java
Switching an existing workload from a transport:
{quote}
transportConnector name=verified_stompssl
uri=stomp+ssl://0.0.0.0:6165?needClientAuth=true/
{quote}
to
{quote}
transportConnector name=verified_stompssl
uri=stomp+nio+ssl://0.0.0.0:6165?needClientAuth=true/
{quote}
showed the CPU profile to go from 1-5% to 300% constantly on a 8 core server
I was able to recreate this using a ruby client @
http://devco.net/rip/amq_560_stomp_nio_ssl_tester.rb
The important combinations are:
* I am connecting to a stomp+nio+ssl port
* I am creating the subscriptions to the 10 queus and topics
If I change either of these variables - like just commenting out the loop
that does those subscriptions - then the CPU load is acceptable.
I analysed the running VM with VisualVM and found that
transport.nio.NIOSSLTransport.serviceRead() is the busy thread.
My activemq.xml is:
{noformat}
beans
xmlns=http://www.springframework.org/schema/beans;
xmlns:amq=http://activemq.apache.org/schema/core;
xmlns:xsi=http://www.w3.org/2001/XMLSchema-instance;
xsi:schemaLocation=http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
http://activemq.apache.org/schema/core
http://activemq.apache.org/schema/core/activemq-core.xsd
http://activemq.apache.org/camel/schema/spring
http://activemq.apache.org/camel/schema/spring/camel-spring.xsd;
broker xmlns=http://activemq.apache.org/schema/core; brokerName=amq1
useJmx=true persistent=true schedulePeriodForDestinationPurge=6
destinationPolicy
policyMap
policyEntries
policyEntry topic= producerFlowControl=false/
policyEntry queue=*.reply. gcInactiveDestinations=true
inactiveTimoutBeforeGC=12 /
/policyEntries
/policyMap
/destinationPolicy
managementContext
managementContext connectorPort=1099
jmxDomainName=org.apache.activemq/
/managementContext
plugins
statisticsBrokerPlugin/
simpleAuthenticationPlugin
users
authenticationUser username=test password=test
groups=admins,everyone/
/users
/simpleAuthenticationPlugin
authorizationPlugin
map
authorizationMap
authorizationEntries
authorizationEntry queue= write=admins read=admins
admin=admins /
authorizationEntry topic= write=admins read=admins
admin=admins /
/authorizationEntries
/authorizationMap
/map
/authorizationPlugin
/plugins
sslContext
sslContext
keyStore=keystore.jks keyStorePassword=ohshahCu
trustStore=truststore.jks trustStorePassword=ohshahCu
/
/sslContext
systemUsage
systemUsage
memoryUsage
memoryUsage limit=200 mb /
/memoryUsage
storeUsage
storeUsage limit=1 gb /
/storeUsage
tempUsage
tempUsage limit=1 gb /
/tempUsage
/systemUsage
/systemUsage
transportConnectors
transportConnector name=openwire uri=tcp://0.0.0.0:6166/
transportConnector name=stomp+nio
uri=stomp+nio://0.0.0.0:6163/
transportConnector name=stompssl
uri=stomp+ssl://0.0.0.0:6164/
transportConnector name=verified_stompssl
uri=stomp+nio+ssl://0.0.0.0:6165?needClientAuth=true/
/transportConnectors
/broker
import resource=jetty.xml/
/beans
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AMQ-3819) high cpu with stomp+nio+ssl and many subscriptions
[
https://issues.apache.org/jira/browse/AMQ-3819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13263757#comment-13263757
]
Timothy Bish commented on AMQ-3819:
---
Commit test cases on trunk for this scenario, one for each (TCP, SSL, NIO,
NIO+SSL). Only the NIO+SSL version has an issue.
high cpu with stomp+nio+ssl and many subscriptions
--
Key: AMQ-3819
URL: https://issues.apache.org/jira/browse/AMQ-3819
Project: ActiveMQ
Issue Type: Bug
Components: stomp
Affects Versions: 5.6.0
Environment: CentOS 6, RC of 5.6.0
java version 1.6.0_22
OpenJDK Runtime Environment (IcedTea6 1.10.6) (rhel-1.43.1.10.6.el6_2-x86_64)
OpenJDK 64-Bit Server VM (build 20.0-b11, mixed mode)
Reporter: R.I.Pienaar
Attachments: StompLoadTest.java, StompNIOSSLLoadTest.java
Switching an existing workload from a transport:
{quote}
transportConnector name=verified_stompssl
uri=stomp+ssl://0.0.0.0:6165?needClientAuth=true/
{quote}
to
{quote}
transportConnector name=verified_stompssl
uri=stomp+nio+ssl://0.0.0.0:6165?needClientAuth=true/
{quote}
showed the CPU profile to go from 1-5% to 300% constantly on a 8 core server
I was able to recreate this using a ruby client @
http://devco.net/rip/amq_560_stomp_nio_ssl_tester.rb
The important combinations are:
* I am connecting to a stomp+nio+ssl port
* I am creating the subscriptions to the 10 queus and topics
If I change either of these variables - like just commenting out the loop
that does those subscriptions - then the CPU load is acceptable.
I analysed the running VM with VisualVM and found that
transport.nio.NIOSSLTransport.serviceRead() is the busy thread.
My activemq.xml is:
{noformat}
beans
xmlns=http://www.springframework.org/schema/beans;
xmlns:amq=http://activemq.apache.org/schema/core;
xmlns:xsi=http://www.w3.org/2001/XMLSchema-instance;
xsi:schemaLocation=http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
http://activemq.apache.org/schema/core
http://activemq.apache.org/schema/core/activemq-core.xsd
http://activemq.apache.org/camel/schema/spring
http://activemq.apache.org/camel/schema/spring/camel-spring.xsd;
broker xmlns=http://activemq.apache.org/schema/core; brokerName=amq1
useJmx=true persistent=true schedulePeriodForDestinationPurge=6
destinationPolicy
policyMap
policyEntries
policyEntry topic= producerFlowControl=false/
policyEntry queue=*.reply. gcInactiveDestinations=true
inactiveTimoutBeforeGC=12 /
/policyEntries
/policyMap
/destinationPolicy
managementContext
managementContext connectorPort=1099
jmxDomainName=org.apache.activemq/
/managementContext
plugins
statisticsBrokerPlugin/
simpleAuthenticationPlugin
users
authenticationUser username=test password=test
groups=admins,everyone/
/users
/simpleAuthenticationPlugin
authorizationPlugin
map
authorizationMap
authorizationEntries
authorizationEntry queue= write=admins read=admins
admin=admins /
authorizationEntry topic= write=admins read=admins
admin=admins /
/authorizationEntries
/authorizationMap
/map
/authorizationPlugin
/plugins
sslContext
sslContext
keyStore=keystore.jks keyStorePassword=ohshahCu
trustStore=truststore.jks trustStorePassword=ohshahCu
/
/sslContext
systemUsage
systemUsage
memoryUsage
memoryUsage limit=200 mb /
/memoryUsage
storeUsage
storeUsage limit=1 gb /
/storeUsage
tempUsage
tempUsage limit=1 gb /
/tempUsage
/systemUsage
/systemUsage
transportConnectors
transportConnector name=openwire uri=tcp://0.0.0.0:6166/
transportConnector name=stomp+nio
uri=stomp+nio://0.0.0.0:6163/
transportConnector name=stompssl
uri=stomp+ssl://0.0.0.0:6164/
transportConnector name=verified_stompssl
uri=stomp+nio+ssl://0.0.0.0:6165?needClientAuth=true/
/transportConnectors
/broker
import resource=jetty.xml/
/beans
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see:
[jira] [Commented] (AMQ-3819) high cpu with stomp+nio+ssl and many subscriptions
[
https://issues.apache.org/jira/browse/AMQ-3819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13264038#comment-13264038
]
Timothy Bish commented on AMQ-3819:
---
The test case seems to indicate that its an issue with the send rate on the
connection. If you slow down the outbound send loop at the end of the test the
NIO+SSL test will start passing. It may be an issue of not properly handling
underflow situations in the secureRead method. When things go wrong it seems
like we get into a state of underflow but never try reading from the ssl
channel again.
high cpu with stomp+nio+ssl and many subscriptions
--
Key: AMQ-3819
URL: https://issues.apache.org/jira/browse/AMQ-3819
Project: ActiveMQ
Issue Type: Bug
Components: stomp
Affects Versions: 5.6.0
Environment: CentOS 6, RC of 5.6.0
java version 1.6.0_22
OpenJDK Runtime Environment (IcedTea6 1.10.6) (rhel-1.43.1.10.6.el6_2-x86_64)
OpenJDK 64-Bit Server VM (build 20.0-b11, mixed mode)
Reporter: R.I.Pienaar
Attachments: StompLoadTest.java, StompNIOSSLLoadTest.java
Switching an existing workload from a transport:
{quote}
transportConnector name=verified_stompssl
uri=stomp+ssl://0.0.0.0:6165?needClientAuth=true/
{quote}
to
{quote}
transportConnector name=verified_stompssl
uri=stomp+nio+ssl://0.0.0.0:6165?needClientAuth=true/
{quote}
showed the CPU profile to go from 1-5% to 300% constantly on a 8 core server
I was able to recreate this using a ruby client @
http://devco.net/rip/amq_560_stomp_nio_ssl_tester.rb
The important combinations are:
* I am connecting to a stomp+nio+ssl port
* I am creating the subscriptions to the 10 queus and topics
If I change either of these variables - like just commenting out the loop
that does those subscriptions - then the CPU load is acceptable.
I analysed the running VM with VisualVM and found that
transport.nio.NIOSSLTransport.serviceRead() is the busy thread.
My activemq.xml is:
{noformat}
beans
xmlns=http://www.springframework.org/schema/beans;
xmlns:amq=http://activemq.apache.org/schema/core;
xmlns:xsi=http://www.w3.org/2001/XMLSchema-instance;
xsi:schemaLocation=http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
http://activemq.apache.org/schema/core
http://activemq.apache.org/schema/core/activemq-core.xsd
http://activemq.apache.org/camel/schema/spring
http://activemq.apache.org/camel/schema/spring/camel-spring.xsd;
broker xmlns=http://activemq.apache.org/schema/core; brokerName=amq1
useJmx=true persistent=true schedulePeriodForDestinationPurge=6
destinationPolicy
policyMap
policyEntries
policyEntry topic= producerFlowControl=false/
policyEntry queue=*.reply. gcInactiveDestinations=true
inactiveTimoutBeforeGC=12 /
/policyEntries
/policyMap
/destinationPolicy
managementContext
managementContext connectorPort=1099
jmxDomainName=org.apache.activemq/
/managementContext
plugins
statisticsBrokerPlugin/
simpleAuthenticationPlugin
users
authenticationUser username=test password=test
groups=admins,everyone/
/users
/simpleAuthenticationPlugin
authorizationPlugin
map
authorizationMap
authorizationEntries
authorizationEntry queue= write=admins read=admins
admin=admins /
authorizationEntry topic= write=admins read=admins
admin=admins /
/authorizationEntries
/authorizationMap
/map
/authorizationPlugin
/plugins
sslContext
sslContext
keyStore=keystore.jks keyStorePassword=ohshahCu
trustStore=truststore.jks trustStorePassword=ohshahCu
/
/sslContext
systemUsage
systemUsage
memoryUsage
memoryUsage limit=200 mb /
/memoryUsage
storeUsage
storeUsage limit=1 gb /
/storeUsage
tempUsage
tempUsage limit=1 gb /
/tempUsage
/systemUsage
/systemUsage
transportConnectors
transportConnector name=openwire uri=tcp://0.0.0.0:6166/
transportConnector name=stomp+nio
uri=stomp+nio://0.0.0.0:6163/
transportConnector name=stompssl
uri=stomp+ssl://0.0.0.0:6164/
transportConnector name=verified_stompssl
uri=stomp+nio+ssl://0.0.0.0:6165?needClientAuth=true/
/transportConnectors
/broker
import
