Java broker OOM due to DirectMemory

2017-04-18 Thread Ramayan Tiwari 
Hi All,

We are using Java broker 6.0.5, with patch to use MultiQueueConsumer
feature. We just finished deploying to production and saw couple of
instances of broker OOM due to running out of DirectMemory buffer
(exceptions at the end of this email).

Here is our setup:
1. Max heap 12g, max direct memory 4g (this is opposite of what the
recommendation is, however, for our use cause message payload is really
small ~400bytes and is way less than the per message overhead of 1KB). In
perf testing, we were able to put 2 million messages without any issues.
2. ~400 connections to broker.
3. Each connection has 20 sessions and there is one multi queue consumer
attached to each session, listening to around 1000 queues.
4. We are still using 0.16 client (I know).

With the above setup, the baseline utilization (without any messages) for
direct memory was around 230mb (with 410 connection each taking 500KB).

Based on our understanding of broker memory allocation, message payload
should be the only thing adding to direct memory utilization (on top of
baseline), however, we are experiencing something completely different. In
our last broker crash, we see that broker is constantly running with 90%+
direct memory allocated, even when message payload sum from all the queues
is only 6-8% (these % are against available DM of 4gb). During these high
DM usage period, heap usage was around 60% (of 12gb).

We would like some help in understanding what could be the reason of these
high DM allocations. Are there things other than message payload and AMQP
connection, which use DM and could be contributing to these high usage?

Another thing where we are puzzled is the de-allocation of DM byte buffers.
>From log mining of heap and DM utilization, de-allocation of DM doesn't
correlate with heap GC. If anyone has seen any documentation related to
this, it would be very helpful if you could share that.

Thanks
Ramayan


*Exceptions*

java.lang.OutOfMemoryError: Direct buffer memory
at java.nio.Bits.reserveMemory(Bits.java:658) ~[na:1.8.0_40]
at java.nio.DirectByteBuffer.(DirectByteBuffer.java:123)
~[na:1.8.0_40]
at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311) ~[na:1.8.0_40]
at
org.apache.qpid.bytebuffer.QpidByteBuffer.allocateDirect(QpidByteBuffer.java:474)
~[qpid-common-6.0.5.jar:6.0.5]
at
org.apache.qpid.server.transport.NonBlockingConnectionPlainDelegate.restoreApplicationBufferForWrite(NonBlockingConnectionPlainDelegate.java:93)
~[qpid-broker-core-6.0.5.jar:6.0.5]
at
org.apache.qpid.server.transport.NonBlockingConnectionPlainDelegate.processData(NonBlockingConnectionPlainDelegate.java:60)
~[qpid-broker-core-6.0.5.jar:6.0.5]
at
org.apache.qpid.server.transport.NonBlockingConnection.doRead(NonBlockingConnection.java:506)
~[qpid-broker-core-6.0.5.jar:6.0.5]
at
org.apache.qpid.server.transport.NonBlockingConnection.doWork(NonBlockingConnection.java:285)
~[qpid-broker-core-6.0.5.jar:6.0.5]
at
org.apache.qpid.server.transport.NetworkConnectionScheduler.processConnection(NetworkConnectionScheduler.java:124)
~[qpid-broker-core-6.0.5.jar:6.0.5]
at
org.apache.qpid.server.transport.SelectorThread$ConnectionProcessor.processConnection(SelectorThread.java:504)
~[qpid-broker-core-6.0.5.jar:6.0.5]
at
org.apache.qpid.server.transport.SelectorThread$SelectionTask.performSelect(SelectorThread.java:337)
~[qpid-broker-core-6.0.5.jar:6.0.5]
at
org.apache.qpid.server.transport.SelectorThread$SelectionTask.run(SelectorThread.java:87)
~[qpid-broker-core-6.0.5.jar:6.0.5]
at
org.apache.qpid.server.transport.SelectorThread.run(SelectorThread.java:462)
~[qpid-broker-core-6.0.5.jar:6.0.5]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
~[na:1.8.0_40]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
~[na:1.8.0_40]
at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_40]



*Second exception*
java.lang.OutOfMemoryError: Direct buffer memory
at java.nio.Bits.reserveMemory(Bits.java:658) ~[na:1.8.0_40]
at java.nio.DirectByteBuffer.(DirectByteBuffer.java:123)
~[na:1.8.0_40]
at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311) ~[na:1.8.0_40]
at
org.apache.qpid.bytebuffer.QpidByteBuffer.allocateDirect(QpidByteBuffer.java:474)
~[qpid-common-6.0.5.jar:6.0.5]
at
org.apache.qpid.server.transport.NonBlockingConnectionPlainDelegate.(NonBlockingConnectionPlainDelegate.java:45)
~[qpid-broker-core-6.0.5.jar:6.0.5]
at
org.apache.qpid.server.transport.NonBlockingConnection.setTransportEncryption(NonBlockingConnection.java:625)
~[qpid-broker-core-6.0.5.jar:6.0.5]
at
org.apache.qpid.server.transport.NonBlockingConnection.(NonBlockingConnection.java:117)
~[qpid-broker-core-6.0.5.jar:6.0.5]
at
org.apache.qpid.server.transport.NonBlockingNetworkTransport.acceptSocketChannel(NonBlockingNetworkTransport.java:158)
~[qpid-broker-core-6.0.5.jar:6.0.5]
at
org.apache.qpid.server.transport.SelectorThread$SelectionTask$1.run(SelectorThread.java:191)
~[qpid-broker-core-6.0.5.jar:6.0.5]
at

Re: [VOTE] Release Apache Qpid JMS 0.22.0

2017-04-18 Thread Jakub Scholz 
+1 ... I used the staging repo and run my tests against different versions
of Qpid C++ and MRG-M brokers.

On Mon, Apr 17, 2017 at 7:34 PM, Robbie Gemmell 
wrote:

> Hi folks,
>
> I have put together a spin for a 0.22.0 Qpid JMS client release, please
> give it a test out and vote accordingly.
>
> The source and binary archives can be grabbed from:
> https://dist.apache.org/repos/dist/dev/qpid/jms/0.22.0-rc1/
>
> The maven artifacts are also staged for now at:
> https://repository.apache.org/content/repositories/orgapacheqpid-1104
>
> The JIRAs currently assigned are:
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?
> projectId=12314524=12340042
>
> Regards,
> Robbie
>
> P.S. If you want to test it out using maven (e.g with the examples src, or
> your own things), you can temporarily add this to your poms to access the
> staging repo:
>
>   
> 
>   staging
>   https://repository.apache.org/content/
> repositories/orgapacheqpid-1104
> 
>   
>
> The dependency for the client itself would then be:
>
>   
> org.apache.qpid
> qpid-jms-client
> 0.22.0
>   
>
> -
> To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
> For additional commands, e-mail: users-h...@qpid.apache.org
>
>

Re: [VOTE] Release Apache Qpid JMS 0.22.0

2017-04-18 Thread Timothy Bish 

On 04/17/2017 01:34 PM, Robbie Gemmell wrote:

Hi folks,

I have put together a spin for a 0.22.0 Qpid JMS client release, please
give it a test out and vote accordingly.

The source and binary archives can be grabbed from:
https://dist.apache.org/repos/dist/dev/qpid/jms/0.22.0-rc1/

The maven artifacts are also staged for now at:
https://repository.apache.org/content/repositories/orgapacheqpid-1104

The JIRAs currently assigned are:
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12314524=12340042

Regards,
Robbie

P.S. If you want to test it out using maven (e.g with the examples src, or
your own things), you can temporarily add this to your poms to access the
staging repo:

   
 
   staging
   
https://repository.apache.org/content/repositories/orgapacheqpid-1104
 
   

The dependency for the client itself would then be:

   
 org.apache.qpid
 qpid-jms-client
 0.22.0
   

-
To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
For additional commands, e-mail: users-h...@qpid.apache.org




+1

* validated signatures and checksums
* checked for license and notice files present in src and bin
* built from the source archive and ran tests
* ran the HelloWorld example against ActiveMQ 5.14.5 and Artemis 2.0.0
* built an ActiveMQ 5.x snapshot and ran the Qpid JMS based tests
* ran mvn apache-rat:check to ensure no missing license files.


--
Tim Bish
twitter: @tabish121
blog: http://timbish.blogspot.com/


-
To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
For additional commands, e-mail: users-h...@qpid.apache.org

Re: [VOTE] Release Apache Qpid JMS 0.22.0

2017-04-18 Thread Robbie Gemmell 
On 17 April 2017 at 18:34, Robbie Gemmell  wrote:
> Hi folks,
>
> I have put together a spin for a 0.22.0 Qpid JMS client release, please
> give it a test out and vote accordingly.
>
> The source and binary archives can be grabbed from:
> https://dist.apache.org/repos/dist/dev/qpid/jms/0.22.0-rc1/
>
> The maven artifacts are also staged for now at:
> https://repository.apache.org/content/repositories/orgapacheqpid-1104
>
> The JIRAs currently assigned are:
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12314524=12340042
>
> Regards,
> Robbie
>
> P.S. If you want to test it out using maven (e.g with the examples src, or
> your own things), you can temporarily add this to your poms to access the
> staging repo:
>
>   
> 
>   staging
>   
> https://repository.apache.org/content/repositories/orgapacheqpid-1104
> 
>   
>
> The dependency for the client itself would then be:
>
>   
> org.apache.qpid
> qpid-jms-client
> 0.22.0
>   

Adding my +1

I gave thigns a check over as follows:
- Verified the signature and checksum files.
- Checked LICENCE+NOTICE files are present/correct in src+bin archives.
- Ran mvn apache-rat:check to verify licence headers on the src archive.
- Ran the build and tests from the source release archive, no issues.
- Built the examples from the binary release archive using Maven with the
  staging repo and ran HelloWorld against Qpid Dispatch 0.8.0 rc3 tag,
  Qpid CPP master (both built against Proton 0.17.0), Qpid for Java 6.1.2,
  and ActiveMQ 5 master.
- Built the examples using javac manually and run to verify required libs
  were included in lib dir as expected.
- Ran the Joram JMS tests against the Qpid for Java master using the
  staging repo.
- Ran the ActiveMQ 5 master build and amqp tests using the staging repo.

Robbie

-
To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
For additional commands, e-mail: users-h...@qpid.apache.org

Re: End-to-end WebDriver test for Qpid Dispatch Router console

2017-04-18 Thread Matej Lesko 
Great job!!

Best regards,
Matej Leško
Middleware Messaging Quality Assurance Engineer

Red Hat Czech s.r.o., Purkynova 647/111, 612 00  Brno, Czech Republic

E-mail: lesko.matej...@gmail.com
phone: +421 949 478 066
IRC:   mlesko at #brno, #messaging, #messaging-qe, #brno-TPB

On Mon, Apr 17, 2017 at 10:29 PM, Jiri Danek  wrote:

>  Hello folks,
>
> for a while I've been working on WebDriver (Selenium 2.0) tests for the
> Dispatch web console. The idea is to have automatic check that the console
> is working and usable. I'd like to share it now in order to get feedback
> and possibly even adoption.
>
> This started as a learning project to get more familiar with pytest and
> webdriver. I would be glad for any suggestions and recommendations
> regarding what I've done wrong and what should be improved.
>
> Currently there is 10 tests, essentially all of them about connecting the
> console to a router.
> Source on GitHub
> https://github.com/jdanekrh/dispatch-console-tests/tree/update_to_9
>
> See it on Travis CI (running on Chrome and Firefox)
> https://travis-ci.org/jdanekrh/dispatch-console-tests/builds/222912530
>
> The way it runs on Travis CI is that it first downloads and runs two docker
> images which I've created, one for console and the other for router. The
> Dockerfiles are in the docker/ directory.
>
> When getting up to speed on UI tests, I tried to follow the idea of test
> pyramid [0] and chose to structure the tests around Page Objects[1][2],
> because it seems to be considered a good idea. This means a test might look
> like this
>
> @pytest.mark.nondestructive
> @pytest.mark.parametrize("when_correct_details", [
> lambda self, page: self.when_correct_details(page),
> lambda self, page: page.connect_to(self.console_ip)])
> def test_correct_details(self, when_correct_details):
> self.test_name = 'test_correct_details'
> page = self.given_connect_page()
> when_correct_details(self, page)
> page.connect_button.click()
> self.then_login_succeeds()
> self.then_no_js_error()
>
> If you are familiar with pytest and pytest-selenium, you'd know that by
> default only tests marked as nondestructive are executed. That is the
> meaning of the first decorator/annotation. The second annotation causes the
> test run twice, each time with different function as argument, the first
> function fills both ip and port, second function fills only ip on the
> initial connect screen.
>
> Here is a screencast of a complete test run in a Chrome browser. All
> software is running locally (meaning the test, the Chrome browser, Tomcat
> with the console and Dispatch Router).
>
> https://www.youtube.com/watch?v=A7XFCXPcIeE
>  (3 minutes)
>
> to run the same thing on the CLI, in the top level directory, run
> $ py.test --base-url http://127.0.0.1:8080/stand-alone --console
> stand-alone --local-chrome
>
> to use firefox, run
>
> $ py.test --base-url http://127.0.0.1:8080/stand-alone --console
> stand-alone  --capability marionette true --driver Firefox --driver-path
> /unless/in/PATH/then/path/to/geckodriver
>
> Regarding tests that fail in the video,
>
>
>- TestConnectPage::test_wrongip,port is not reported yet; I'd expect to
>see error message almost immediatelly, the way it used to work about 5
>months ago in hawtio version (when I tried it last)
>- TestConnectPage::test_correct_details(when_correct_details1) is
>reported as https://issues.apache.org/jira/browse/DISPATCH-746
>- TestHawtioLogsPage::test_open_hawtio_logs_page should not be tested
> on
>standalone console (and it passes because of the @pytest.mark.reproduces
>as explained below)
>- TestOverviewPage::test_expanding_tree should not be tested on
>standalone console
>
> There was idea that tests should never be failing. If there is a test that
> fails, then the test could be modified to succeed if the issue is present.
> I marked such tests with @pytest.mark.reproduces. Passing tests are marked
> with @pytest.mark.verifies. This is probably not a good idea, because it is
> chore to maintain. Better to fix the issue in the first place.
>
> Regarding CI, there is a Travis CI job linked to the test repository
> itself, and another Travis job to build Docker images. In the future, I'd
> like to run the image building job daily and have it trigger a job which
> will run the tests with the image. This way it will be immediatelly clear
> if some new test failed.
>
> If you have any suggestions regarding either the tests itself or ideas
> around what should be tested in general. I would be glad to hear it.
>
> [0]
> https://testing.googleblog.com/2015/04/just-say-no-to-
> more-end-to-end-tests.html
> [1]
> https://gojko.net/2010/04/13/how-to-implement-ui-testing-
> without-shooting-yourself-in-the-foot-2/
> [2] https://youtu.be/7tzA2nsg1jQ?t=14m
>
> Thanks for your help,
>
>  --
> Jiri
>

Start of svn to git migration of java components

2017-04-18 Thread Lorenz Quack 
Hello All,

We would like to notify all committers that we are starting the
svn to git migration of the Qpid Broker-J and Qpid JMS AMQP 0-x
components as discussed in [1].

Please do not commit to [2] and/or [3] or any subfolders until
further notice.

Kind regards,
Lorenz Quack


[1] 
http://qpid.2158936.n2.nabble.com/Plan-for-migration-of-Qpid-java-broker-svn-repo-into-git-td7662270.html
[2] https://svn.apache.org/repos/asf/qpid/java/
[3] https://svn.apache.org/repos/asf/qpid/qpid-jms-amqp-0-x/


-
To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
For additional commands, e-mail: users-h...@qpid.apache.org

Re: Plan for migration of Qpid java broker svn repo into git

2017-04-18 Thread Lorenz Quack 
On Tue, 2017-04-18 at 09:49 +0200, Rob Godfrey wrote:
> On 17 April 2017 at 17:41, Robbie Gemmell 
> wrote:
> 
> > 
> > Sounds good to me.
> > 
> > On 14 April 2017 at 14:54, Oleksandr Rudyy 
> > wrote:
> > > 
> > > Hi all,
> > > 
> > > As per vote [1] we would like to perform the migration of sources
> > > for
> > > Qpid-Broker-J (a.k.a. Qpid Java Broker) from svn repo into git.
> > > 
> > > Here is approximate plan of actions of how we would like to do
> > > that:
> > > 1) On Monday (17 of April), we are going to create git repo and
> > > send an
> > > email into user list asking to stop committing into svn repo [3].
> 
> I didn't see such an e-mail get sent yesterday... does that mean that
> the
> migration has been delayed/people should still feel free to commit to
> the
> svn repo, or simply that the mail got lost somewhere?
> 

Migration got delayed.  Expect to receive the email later today.
Until then, feel free to push any commits you have pending.

Kind regards,
Lorenz


> -- Rob
> 
> > 
> > 2) Immediately after that we will raise INFRA JIRA to suspend
> > commit email
> > > 
> > > notification on newly created git repo (in order to avoid email
> > > generated
> > > by push of svn commits into new git repo)
> > > 3) INFRA cannot not guarantee any time how quickly they are going
> > > to turn
> > > off the email notification. Thus, we need to wait until it is
> > implemented.
> > > 
> > > 4) As soon as commit emails are suspended, we will push svn
> > > commits from
> > > local git repo into Apache repo and raise another INFRA JIRA (or
> > > reuse
> > > existing INFRA JIRA) to resume commit notifications.
> > > 5) After commit notifications are resumed, we will send an email
> > > into
> > user
> > > 
> > > list about new git repo,  update the repo URLs on Qpid site and
> > > lock
> > > existing SVN repo to allow  read only access (by raising a
> > > separate INFRA
> > > JIRA for this).
> > > 
> > > Please, let me know if anyone has any concern regarding the plan
> > > above.
> > If
> > > 
> > > you have any in progress work, please, commit it into svn before
> > > Monday
> > or
> > > 
> > > wait for the new git repo.
> > > 
> > > Kind Regards,
> > > Alex
> > > 
> > > [1]
> > > http://qpid.2158936.n2.nabble.com/VOTE-Migrate-Qpid-Broker-
> > J-and-Qpid-JMS-AMQP-0-x-Client-from-SVN-to-GIT-td7661885.html
> > > 
> > > [2] Result:
> > > http://qpid.2158936.n2.nabble.com/RESULT-VOTE-Migrate-Qpid-
> > Broker-J-and-Qpid-JMS-AMQP-0-x-Client-from-SVN-to-GIT-
> > td7662060.html
> > > 
> > > [3] https://svn.apache.org/repos/asf/qpid/java
> > -
> > 
> > To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
> > For additional commands, e-mail: users-h...@qpid.apache.org
> > 
> > 

-
To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
For additional commands, e-mail: users-h...@qpid.apache.org

Re: Plan for migration of Qpid java broker svn repo into git

2017-04-18 Thread Rob Godfrey 
On 17 April 2017 at 17:41, Robbie Gemmell  wrote:

> Sounds good to me.
>
> On 14 April 2017 at 14:54, Oleksandr Rudyy  wrote:
> > Hi all,
> >
> > As per vote [1] we would like to perform the migration of sources for
> > Qpid-Broker-J (a.k.a. Qpid Java Broker) from svn repo into git.
> >
> > Here is approximate plan of actions of how we would like to do that:
> > 1) On Monday (17 of April), we are going to create git repo and send an
> > email into user list asking to stop committing into svn repo [3].
>


I didn't see such an e-mail get sent yesterday... does that mean that the
migration has been delayed/people should still feel free to commit to the
svn repo, or simply that the mail got lost somewhere?

-- Rob

> 2) Immediately after that we will raise INFRA JIRA to suspend commit email
> > notification on newly created git repo (in order to avoid email generated
> > by push of svn commits into new git repo)
> > 3) INFRA cannot not guarantee any time how quickly they are going to turn
> > off the email notification. Thus, we need to wait until it is
> implemented.
> > 4) As soon as commit emails are suspended, we will push svn commits from
> > local git repo into Apache repo and raise another INFRA JIRA (or reuse
> > existing INFRA JIRA) to resume commit notifications.
> > 5) After commit notifications are resumed, we will send an email into
> user
> > list about new git repo,  update the repo URLs on Qpid site and lock
> > existing SVN repo to allow  read only access (by raising a separate INFRA
> > JIRA for this).
> >
> > Please, let me know if anyone has any concern regarding the plan above.
> If
> > you have any in progress work, please, commit it into svn before Monday
> or
> > wait for the new git repo.
> >
> > Kind Regards,
> > Alex
> >
> > [1]
> > http://qpid.2158936.n2.nabble.com/VOTE-Migrate-Qpid-Broker-
> J-and-Qpid-JMS-AMQP-0-x-Client-from-SVN-to-GIT-td7661885.html
> > [2] Result:
> > http://qpid.2158936.n2.nabble.com/RESULT-VOTE-Migrate-Qpid-
> Broker-J-and-Qpid-JMS-AMQP-0-x-Client-from-SVN-to-GIT-td7662060.html
> > [3] https://svn.apache.org/repos/asf/qpid/java
>
> -
> To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
> For additional commands, e-mail: users-h...@qpid.apache.org
>
>