[
https://issues.apache.org/jira/browse/QPID-7753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lorenz Quack reopened QPID-7753:
--------------------------------
Assignee: (was: Keith Wall)
When Broker is shutting down the scheduling of reallocation memory task fails
with exception
{noformat}
java.util.concurrent.RejectedExecutionException: Task
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@e4d1fd0
rejected from
org.apache.qpid.server.util.HousekeepingExecutor@8af069[Terminated, pool size =
0, active threads = 0, queued tasks = 0, completed tasks = 3]
at
java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2047)
at
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823)
at
java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:326)
at
java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:533)
at
java.util.concurrent.ScheduledThreadPoolExecutor.submit(ScheduledThreadPoolExecutor.java:632)
at
org.apache.qpid.server.virtualhost.AbstractVirtualHost.reallocateMessages(AbstractVirtualHost.java:1407)
at
org.apache.qpid.server.model.BrokerImpl.compactMemoryInternal(BrokerImpl.java:1250)
at
org.apache.qpid.server.model.BrokerImpl.checkDirectMemoryUsage(BrokerImpl.java:449)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
{noformat}
> Sparsely occupied message buffers may lead to java.lang.OutOfMemoryError:
> Direct buffer memory
> ----------------------------------------------------------------------------------------------
>
> Key: QPID-7753
> URL: https://issues.apache.org/jira/browse/QPID-7753
> Project: Qpid
> Issue Type: Bug
> Components: Java Broker
> Affects Versions: qpid-java-6.0, qpid-java-6.1
> Reporter: Keith Wall
> Fix For: qpid-java-broker-7.0.0
>
> Attachments: flow-to-disk-based-on-used-direct-memory-6-0-x.diff
>
>
> Some Broker usage patterns can lead to the Broker failing with a
> "java.lang.OutOfMemoryError: Direct buffer memory" error.
> For the condition to manifest a producing application needs to use a single
> connection for the production of messages. Some messages need to be consumed
> quickly whilst others remain on the Broker. This pattern might result from:
> # the consuming application using message selectors to selectively consume
> some messages whilst leaving others on the Broker.
> # the use of 'out of order' queue types (priority/sorted etc) where lower
> priority items are left of the Broker.
> # the producing application routing messages to multiple queues and the
> consuming application draining some queues but not others.
> The problem arises owing to the buffering strategy under the IO layer.
> {{NonBlockingConnection}} allocates a {{netInputBuffer}} from pooled direct
> memory of size 256K. This buffer is used for all network reads until the
> space remaining in the buffer is less than the amount required to complete
> the AMQP frame that is currently being read, in which case a new
> netInputBuffer is allocated. The protocol layers identify the message
> payload/message headers as part of AMQP frame parsing and create a
> byte-buffer *views* onto the original input byte buffer. This byte buffer is
> retained by the store until the message is consumed. In the usage pattern
> described above, a single long lived message amongst a stream of shorted
> lived messages causes an entire 256K buffer chunk to be retained. Qpid
> cannot dispose or reuse the buffer until it is entirely unoccupied.
> The flow to disk feature is designed to prevent excessive direct memory use
> by flushing messages to disk if thresholds are breached. Flow to disk does
> not help the scenario described above because it considers the total payloads
> of live messages. Its algorithm does not consider direct memory occupancy.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org
For additional commands, e-mail: dev-h...@qpid.apache.org
