Hi Helen, thanks for taking the time to investigate this. Your suggestions sound very sensible, and it would be good to get them reviewed and incorporated as soon as possible. Would it be possible for you to raise a JIRA [1] for this and attach your changes as a patch file? - that will be the quickest way for us to be able to get these improvements applied.
Many thanks for your efforts, Rob [1] https://issues.apache.org/jira/browse/QPID On 7 May 2013 03:10, Helen Kwong <[email protected]> wrote: > Hello, > > My team is using the Java broker and Java client, version 0.16, and looking > to lower the client's memory footprint on our servers. We did some heap > analysis and found that the consumption is coming mostly from > AMQAnyDestination instances, each having a retained size of 2864B, and > since we have 6000 queues on each of our 2 brokers, this amounts to about > ~33MB, which is valuable real estate for us. In our analysis we found a few > possible optimizations in the Qpid code that would reduce the per-queue > heap consumption and which don't seem high risk, and were wondering how > feasible these changes are. > > The first optimization involves cacheing / resuing redundant maps, while > the rest involves making the default value of a variable the constant empty > Map or List, instead of a new empty Map or List. > > > 1. In Address / AddressParser, cacheing / reusing the options Maps for > queues created with the same options string. (This optimization gives us > the most significant savings, but has the highest risk.) > > All our queues are created with the same options string, which means each > corresponding AMQDestination has an Address that has an _options Map that > is the same for all queues, i.e., 12K copies of the same map. As far as we > can tell, the _options map is effectively immutable, i.e., there is no code > path by which an Address’s _options map can be modified. (Is this correct?) > So a possible improvement is that in > org.apache.qpid.messaging.util.AddressParser, we cache the options map for > each options string that we've already encountered, and if the options > string passed in has already been seen, we use the stored options map for > that Address. This way, for queues having the same options, their Address > options will reference the same Map. (For our queues, each Address _options > Map takes up 1384 B, so with 12K queues we'd save about 15.5MB.) > > > 2. AMQDestination's _link field -- > org.apache.qpid.client.messaging.address.Link$Subscription > > Subscription's args field is by default a new HashMap with default capacity > 16. In our use case it remains empty for all queues. A possible > optimization is to set the default value as Collections.emptyMap() instead. > As far was we can tell, Subscription.getArgs() is not used to get the map > and then modify it. For us this saves 128B per queue. > > > 3. AMQDestination's _targetNode field -- > org.apache.qpid.client.messaging.address.Node$ExchangeNode > > Optimization A: The field _exchangeOptions is never used and can be > removed. It's an empty new HashMap with default capacity 16. For us this > saves 128B per queue. > > Optimization B: The parent class Node has a _bindings List that is by > default a new ArrayList with the default capacity of 10. In our use case > _bindings remains empty for all queues. A possible optimization is to set > the default value as Collections.emptyList() instead, and if addBindings() > is called, lazily set it as a new empty list first. For us this saves 80B > per queue. > > > 4. AMQDestination's _sourceNode field -- > org.apache.qpid.client.messaging.address.Node$QueueNode > > Same 2 optimizations as (3) (with the unused field for QueueNode being > _queueOptions). > > > Overall: For us, the 5 optimizations in (2), (3), and (4) would save us > 128B x 3 + 80B x 2 = 544B per queue, and (1) would save us 1384B per queue; > all together this would save 1928B out of 2864B, or 67%, per queue for us. > > We'd appreciate feedback on how feasible these changes are and whether we > are making any incorrect assumptions. Thanks a lot! > > Helen >
