[ http://issues.apache.org/jira/browse/QPID-32?page=all ]
Marnie McCormack updated QPID-32:
---------------------------------
Fix Version/s: M2
> Amend persistence algorithm to introduce paging to physical memory
> ------------------------------------------------------------------
>
> Key: QPID-32
> URL: http://issues.apache.org/jira/browse/QPID-32
> Project: Qpid
> Issue Type: Improvement
> Components: Java Broker
> Reporter: Marnie McCormack
> Assigned To: Robert Greig
> Fix For: M2
>
>
> Currently all messages, including the headers and body, are held in memory in
> the broker. This means that even with persistent queues, all messages must be
> able to fit in the heap. Clearly that is a severe limitation.
> This effectively means that there is an upper limit for the number of
> messages a broker can handle, bound by the heap size available and defined by
> the size of the messages in transit.
> As we know that there are users who wish to combine high volume throughput
> with large message size (100Mb) we can expect that this limitation will be
> constraining fairly quickly, given finite available memory. We have had
> requests for the broker to be able to handle 5 million messages at any one
> time, with message size being frmo small (a few kb) to large (hundreds of Mb).
> We need to introduce a paging algorithm into the broker such that it can page
> messages in & out of memory as appropriate during transit. The broker needs
> to be enhanced so that messages do not need to be held in memory at all
> times. Ideally the redesign of this aspect will support flow-to-disk for
> non-persistent messages when configurable thresholds are reached.
> This may also affect transactional messages, persistent or not, as they are
> also held in memory during transit and thus are affected by the same
> constraint.
> The design of the solution for memory management is critical for the
> performance of the broker and should be published and reviewed, as well as
> having defined performance benchmarks to meet during testing. We need to be
> very confident that we're putting in a solution which is quick and efficient.
> We may also want to consider the configuration options available for memory
> management - allowing our users to tune deployed brokers in a way appropriate
> for their application profile i.e. if the app is sending large messages
> infrequently then the fastest solution could potentially differ from the
> appropriate model for an application sending mainly small messages and
> occasional very large messages.
> Also need to consider the impact of priority/qos settings on messages when
> designing the solution.
> The assignee for this work should expand the requirements in more detail once
> analysis is complete, prior to completion of development.
> Please add/amend the detail here if you have hands on experience of the
> existing persistence code. Thanks.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
