[jira] [Commented] (QPID-7830) Heap dominated by duplicates of common routing values / header values etc
[ https://issues.apache.org/jira/browse/QPID-7830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16491240#comment-16491240 ] ASF subversion and git services commented on QPID-7830: --- Commit 57bda83de7fbe6291aea507b5c1098fda5cf130d in qpid-broker-j's branch refs/heads/master from [~alex.rufous] [ https://git-wip-us.apache.org/repos/asf?p=qpid-broker-j.git;h=57bda83 ] QPID-7830: [Broker-J] Simplify string caching functionality > Heap dominated by duplicates of common routing values / header values etc > - > > Key: QPID-7830 > URL: https://issues.apache.org/jira/browse/QPID-7830 > Project: Qpid > Issue Type: Bug > Components: Broker-J >Reporter: Keith Wall >Assignee: Alex Rudyy >Priority: Major > Fix For: qpid-java-broker-7.1.0, qpid-java-broker-7.0.4 > > Attachments: > 0001-QPID-7830-Broker-J-Simplify-string-caching-functiona.patch > > > When used for store and forwarding, in some use cases the Broker's heap can > become dominated by duplicates of common values such as routing information > (e.g. {{amq.direct}} or an application's queue name) or common header values > (e.g a application/octet-stream or an application's user id). > On the 0-8..0-91 paths, every enqueued message gets its own > {{MessagePublishInfo}} referencing its own {{AMQShortString}} exchange and > routing keys. For some use-cases, these are drawn from a small set. On the > AMQP 1.0 path, {{Properties#to}} is an example. 0-10 is probably affected > too. > This unnecessarily increases the heap requirements of the Broker. > The Broker should adopt a sensible intern/caching policy with the same policy > applying regardless of whether messages follow the on-line enqueue or > recovery path. Note that in AMQP 1.0, values which are {{Symbols}} have > their underlying String automatically interned. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org For additional commands, e-mail: dev-h...@qpid.apache.org
[jira] [Commented] (QPID-7830) Heap dominated by duplicates of common routing values / header values etc
[ https://issues.apache.org/jira/browse/QPID-7830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16489319#comment-16489319 ] ASF subversion and git services commented on QPID-7830: --- Commit 28e73f3c1abad32d69d4e2f0937f1021a3ac61dd in qpid-broker-j's branch refs/heads/7.0.x from [~alex.rufous] [ https://git-wip-us.apache.org/repos/asf?p=qpid-broker-j.git;h=28e73f3 ] QPID-7830: [Broker-J] Fix failing test (cherry picked from commit 196b27065219b1283522bdb3a8f4fb1dda82ac16) > Heap dominated by duplicates of common routing values / header values etc > - > > Key: QPID-7830 > URL: https://issues.apache.org/jira/browse/QPID-7830 > Project: Qpid > Issue Type: Bug > Components: Broker-J >Reporter: Keith Wall >Assignee: Alex Rudyy >Priority: Major > Fix For: qpid-java-broker-7.1.0, qpid-java-broker-7.0.4 > > Attachments: > 0001-QPID-7830-Broker-J-Simplify-string-caching-functiona.patch > > > When used for store and forwarding, in some use cases the Broker's heap can > become dominated by duplicates of common values such as routing information > (e.g. {{amq.direct}} or an application's queue name) or common header values > (e.g a application/octet-stream or an application's user id). > On the 0-8..0-91 paths, every enqueued message gets its own > {{MessagePublishInfo}} referencing its own {{AMQShortString}} exchange and > routing keys. For some use-cases, these are drawn from a small set. On the > AMQP 1.0 path, {{Properties#to}} is an example. 0-10 is probably affected > too. > This unnecessarily increases the heap requirements of the Broker. > The Broker should adopt a sensible intern/caching policy with the same policy > applying regardless of whether messages follow the on-line enqueue or > recovery path. Note that in AMQP 1.0, values which are {{Symbols}} have > their underlying String automatically interned. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org For additional commands, e-mail: dev-h...@qpid.apache.org
[jira] [Commented] (QPID-7830) Heap dominated by duplicates of common routing values / header values etc
[ https://issues.apache.org/jira/browse/QPID-7830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16489321#comment-16489321 ] ASF subversion and git services commented on QPID-7830: --- Commit 63f7b7e949ac04ee7596ef6b78b8160ab146 in qpid-broker-j's branch refs/heads/7.0.x from [~alex.rufous] [ https://git-wip-us.apache.org/repos/asf?p=qpid-broker-j.git;h=63f7b7e ] QPID-7830: [Broker-J] Add null checks for 'subject' and 'to' (cherry picked from commit dc2e9d425a4547e0efbf6d8de70ec0c90ecdbdc4) > Heap dominated by duplicates of common routing values / header values etc > - > > Key: QPID-7830 > URL: https://issues.apache.org/jira/browse/QPID-7830 > Project: Qpid > Issue Type: Bug > Components: Broker-J >Reporter: Keith Wall >Assignee: Alex Rudyy >Priority: Major > Fix For: qpid-java-broker-7.1.0, qpid-java-broker-7.0.4 > > Attachments: > 0001-QPID-7830-Broker-J-Simplify-string-caching-functiona.patch > > > When used for store and forwarding, in some use cases the Broker's heap can > become dominated by duplicates of common values such as routing information > (e.g. {{amq.direct}} or an application's queue name) or common header values > (e.g a application/octet-stream or an application's user id). > On the 0-8..0-91 paths, every enqueued message gets its own > {{MessagePublishInfo}} referencing its own {{AMQShortString}} exchange and > routing keys. For some use-cases, these are drawn from a small set. On the > AMQP 1.0 path, {{Properties#to}} is an example. 0-10 is probably affected > too. > This unnecessarily increases the heap requirements of the Broker. > The Broker should adopt a sensible intern/caching policy with the same policy > applying regardless of whether messages follow the on-line enqueue or > recovery path. Note that in AMQP 1.0, values which are {{Symbols}} have > their underlying String automatically interned. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org For additional commands, e-mail: dev-h...@qpid.apache.org
[jira] [Commented] (QPID-7830) Heap dominated by duplicates of common routing values / header values etc
[ https://issues.apache.org/jira/browse/QPID-7830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16489320#comment-16489320 ] ASF subversion and git services commented on QPID-7830: --- Commit c14eb33b5062b81e9475215fc0f90b21026b4eae in qpid-broker-j's branch refs/heads/7.0.x from [~alex.rufous] [ https://git-wip-us.apache.org/repos/asf?p=qpid-broker-j.git;h=c14eb33 ] QPID-7830: [Broker-J] Cache AMQP 1.0 application property names and message properties 'to' and 'subject' (cherry picked from commit 85f1db6c4a2b5127ebec78cedcbe4563094cab47) > Heap dominated by duplicates of common routing values / header values etc > - > > Key: QPID-7830 > URL: https://issues.apache.org/jira/browse/QPID-7830 > Project: Qpid > Issue Type: Bug > Components: Broker-J >Reporter: Keith Wall >Assignee: Alex Rudyy >Priority: Major > Fix For: qpid-java-broker-7.1.0, qpid-java-broker-7.0.4 > > Attachments: > 0001-QPID-7830-Broker-J-Simplify-string-caching-functiona.patch > > > When used for store and forwarding, in some use cases the Broker's heap can > become dominated by duplicates of common values such as routing information > (e.g. {{amq.direct}} or an application's queue name) or common header values > (e.g a application/octet-stream or an application's user id). > On the 0-8..0-91 paths, every enqueued message gets its own > {{MessagePublishInfo}} referencing its own {{AMQShortString}} exchange and > routing keys. For some use-cases, these are drawn from a small set. On the > AMQP 1.0 path, {{Properties#to}} is an example. 0-10 is probably affected > too. > This unnecessarily increases the heap requirements of the Broker. > The Broker should adopt a sensible intern/caching policy with the same policy > applying regardless of whether messages follow the on-line enqueue or > recovery path. Note that in AMQP 1.0, values which are {{Symbols}} have > their underlying String automatically interned. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org For additional commands, e-mail: dev-h...@qpid.apache.org
[jira] [Commented] (QPID-7830) Heap dominated by duplicates of common routing values / header values etc
[ https://issues.apache.org/jira/browse/QPID-7830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16489318#comment-16489318 ] ASF subversion and git services commented on QPID-7830: --- Commit 7b675cb7359237a7023332f42471563c043fc4e5 in qpid-broker-j's branch refs/heads/7.0.x from [~alex.rufous] [ https://git-wip-us.apache.org/repos/asf?p=qpid-broker-j.git;h=7b675cb ] QPID-7830: [Broker-J] Cache encoded and decoded values for 0-10 str8 (cherry picked from commit 5b6dfcd94ef64544040de53d38e2463d327ebcb1) > Heap dominated by duplicates of common routing values / header values etc > - > > Key: QPID-7830 > URL: https://issues.apache.org/jira/browse/QPID-7830 > Project: Qpid > Issue Type: Bug > Components: Broker-J >Reporter: Keith Wall >Assignee: Alex Rudyy >Priority: Major > Fix For: qpid-java-broker-7.1.0, qpid-java-broker-7.0.4 > > Attachments: > 0001-QPID-7830-Broker-J-Simplify-string-caching-functiona.patch > > > When used for store and forwarding, in some use cases the Broker's heap can > become dominated by duplicates of common values such as routing information > (e.g. {{amq.direct}} or an application's queue name) or common header values > (e.g a application/octet-stream or an application's user id). > On the 0-8..0-91 paths, every enqueued message gets its own > {{MessagePublishInfo}} referencing its own {{AMQShortString}} exchange and > routing keys. For some use-cases, these are drawn from a small set. On the > AMQP 1.0 path, {{Properties#to}} is an example. 0-10 is probably affected > too. > This unnecessarily increases the heap requirements of the Broker. > The Broker should adopt a sensible intern/caching policy with the same policy > applying regardless of whether messages follow the on-line enqueue or > recovery path. Note that in AMQP 1.0, values which are {{Symbols}} have > their underlying String automatically interned. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org For additional commands, e-mail: dev-h...@qpid.apache.org
[jira] [Commented] (QPID-7830) Heap dominated by duplicates of common routing values / header values etc
[ https://issues.apache.org/jira/browse/QPID-7830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16489306#comment-16489306 ] ASF subversion and git services commented on QPID-7830: --- Commit dc2e9d425a4547e0efbf6d8de70ec0c90ecdbdc4 in qpid-broker-j's branch refs/heads/master from [~alex.rufous] [ https://git-wip-us.apache.org/repos/asf?p=qpid-broker-j.git;h=dc2e9d4 ] QPID-7830: [Broker-J] Add null checks for 'subject' and 'to' > Heap dominated by duplicates of common routing values / header values etc > - > > Key: QPID-7830 > URL: https://issues.apache.org/jira/browse/QPID-7830 > Project: Qpid > Issue Type: Bug > Components: Broker-J >Reporter: Keith Wall >Assignee: Alex Rudyy >Priority: Major > Fix For: qpid-java-broker-7.1.0, qpid-java-broker-7.0.4 > > Attachments: > 0001-QPID-7830-Broker-J-Simplify-string-caching-functiona.patch > > > When used for store and forwarding, in some use cases the Broker's heap can > become dominated by duplicates of common values such as routing information > (e.g. {{amq.direct}} or an application's queue name) or common header values > (e.g a application/octet-stream or an application's user id). > On the 0-8..0-91 paths, every enqueued message gets its own > {{MessagePublishInfo}} referencing its own {{AMQShortString}} exchange and > routing keys. For some use-cases, these are drawn from a small set. On the > AMQP 1.0 path, {{Properties#to}} is an example. 0-10 is probably affected > too. > This unnecessarily increases the heap requirements of the Broker. > The Broker should adopt a sensible intern/caching policy with the same policy > applying regardless of whether messages follow the on-line enqueue or > recovery path. Note that in AMQP 1.0, values which are {{Symbols}} have > their underlying String automatically interned. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org For additional commands, e-mail: dev-h...@qpid.apache.org
[jira] [Commented] (QPID-7830) Heap dominated by duplicates of common routing values / header values etc
[ https://issues.apache.org/jira/browse/QPID-7830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16488982#comment-16488982 ] Alex Rudyy commented on QPID-7830: -- The caching of strings in AMQP 0-8 and AMQP 1.0 is implemented only for certain fields. It looks like inconsistent approach to me. I think that caching should be implemented in the layer decoding the string regardless the string use case. Caching of string in AMQP 0-10 is implemented in decoder and looks to me like a more attractive approach than the one used for caching in AMQP 0-8 and AMQP 1.0. Though it is less effective as single-use object would hit the cache but it is easier to maintain and reason about. Currently, in AMQP 0-8 we do not cache message header names. Potentially caching of header names in AMQP 0-8 could further reduce memory consumption. I believe that caching functionality should be moved into {{StringTypeConstructor}} in AMQP 1.0 and implemented as part of {{construct}}, something like below {code} byte[] data = new byte[size]; in.get(data); ByteBuffer buffer = ByteBuffer.wrap(data); String cached = CACHE.get().getIfPresent(buffer); if (cached == null) { cached = new String(data, UTF_8); CACHE.get().put(buffer, cached); } return cached; {code} AMQP 0-8 {{AMQPShortStrings}} should be cached unconditionally in {{AMQShortString#readAMQShortString}} similar to code above and calls to {{intern}} should be removed. That should simplify the code and would allow to cache more duplicate string values than the current approach. I am going to attach a patch for the suggested changes. In meant time, I am going to merge implemented changes into 7.0.x > Heap dominated by duplicates of common routing values / header values etc > - > > Key: QPID-7830 > URL: https://issues.apache.org/jira/browse/QPID-7830 > Project: Qpid > Issue Type: Bug > Components: Broker-J >Reporter: Keith Wall >Assignee: Alex Rudyy >Priority: Major > Fix For: qpid-java-broker-7.1.0, qpid-java-broker-7.0.4 > > > When used for store and forwarding, in some use cases the Broker's heap can > become dominated by duplicates of common values such as routing information > (e.g. {{amq.direct}} or an application's queue name) or common header values > (e.g a application/octet-stream or an application's user id). > On the 0-8..0-91 paths, every enqueued message gets its own > {{MessagePublishInfo}} referencing its own {{AMQShortString}} exchange and > routing keys. For some use-cases, these are drawn from a small set. On the > AMQP 1.0 path, {{Properties#to}} is an example. 0-10 is probably affected > too. > This unnecessarily increases the heap requirements of the Broker. > The Broker should adopt a sensible intern/caching policy with the same policy > applying regardless of whether messages follow the on-line enqueue or > recovery path. Note that in AMQP 1.0, values which are {{Symbols}} have > their underlying String automatically interned. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org For additional commands, e-mail: dev-h...@qpid.apache.org
[jira] [Commented] (QPID-7830) Heap dominated by duplicates of common routing values / header values etc
[ https://issues.apache.org/jira/browse/QPID-7830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16488919#comment-16488919 ] ASF subversion and git services commented on QPID-7830: --- Commit 85f1db6c4a2b5127ebec78cedcbe4563094cab47 in qpid-broker-j's branch refs/heads/master from [~alex.rufous] [ https://git-wip-us.apache.org/repos/asf?p=qpid-broker-j.git;h=85f1db6 ] QPID-7830: [Broker-J] Cache AMQP 1.0 application property names and message properties 'to' and 'subject' > Heap dominated by duplicates of common routing values / header values etc > - > > Key: QPID-7830 > URL: https://issues.apache.org/jira/browse/QPID-7830 > Project: Qpid > Issue Type: Bug > Components: Broker-J >Reporter: Keith Wall >Priority: Major > Fix For: Future > > > When used for store and forwarding, in some use cases the Broker's heap can > become dominated by duplicates of common values such as routing information > (e.g. {{amq.direct}} or an application's queue name) or common header values > (e.g a application/octet-stream or an application's user id). > On the 0-8..0-91 paths, every enqueued message gets its own > {{MessagePublishInfo}} referencing its own {{AMQShortString}} exchange and > routing keys. For some use-cases, these are drawn from a small set. On the > AMQP 1.0 path, {{Properties#to}} is an example. 0-10 is probably affected > too. > This unnecessarily increases the heap requirements of the Broker. > The Broker should adopt a sensible intern/caching policy with the same policy > applying regardless of whether messages follow the on-line enqueue or > recovery path. Note that in AMQP 1.0, values which are {{Symbols}} have > their underlying String automatically interned. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org For additional commands, e-mail: dev-h...@qpid.apache.org
[jira] [Commented] (QPID-7830) Heap dominated by duplicates of common routing values / header values etc
[ https://issues.apache.org/jira/browse/QPID-7830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16472218#comment-16472218 ] ASF subversion and git services commented on QPID-7830: --- Commit d79537d2951dc6eabc7fabe6fa0760cb0f07c11d in qpid-broker-j's branch refs/heads/7.0.x from [~k-wall] [ https://git-wip-us.apache.org/repos/asf?p=qpid-broker-j.git;h=d79537d ] QPID-7830: [Broker-J] [AMQP 0-8..0-91] Move caching responsubility to virtualhost (cherry picked from commit ddc519a551061c682877784068e755677e2c6313) > Heap dominated by duplicates of common routing values / header values etc > - > > Key: QPID-7830 > URL: https://issues.apache.org/jira/browse/QPID-7830 > Project: Qpid > Issue Type: Bug > Components: Broker-J >Reporter: Keith Wall >Priority: Major > Fix For: Future > > > When used for store and forwarding, in some use cases the Broker's heap can > become dominated by duplicates of common values such as routing information > (e.g. {{amq.direct}} or an application's queue name) or common header values > (e.g a application/octet-stream or an application's user id). > On the 0-8..0-91 paths, every enqueued message gets its own > {{MessagePublishInfo}} referencing its own {{AMQShortString}} exchange and > routing keys. For some use-cases, these are drawn from a small set. On the > AMQP 1.0 path, {{Properties#to}} is an example. 0-10 is probably affected > too. > This unnecessarily increases the heap requirements of the Broker. > The Broker should adopt a sensible intern/caching policy with the same policy > applying regardless of whether messages follow the on-line enqueue or > recovery path. Note that in AMQP 1.0, values which are {{Symbols}} have > their underlying String automatically interned. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org For additional commands, e-mail: dev-h...@qpid.apache.org
[jira] [Commented] (QPID-7830) Heap dominated by duplicates of common routing values / header values etc
[ https://issues.apache.org/jira/browse/QPID-7830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16472215#comment-16472215 ] ASF subversion and git services commented on QPID-7830: --- Commit 83c6dfebc3c815157ed3f04403a6968cc05eabae in qpid-broker-j's branch refs/heads/7.0.x from [~k-wall] [ https://git-wip-us.apache.org/repos/asf?p=qpid-broker-j.git;h=83c6dfe ] QPID-7830: [Broker-J] [AMQP 0-8..0-91] Mechanically refactor AMQPShortString introducing factory methods and hiding constructors (cherry picked from commit 0a3dfc883b7af756fbdf076665eaae4ad8202bf7) > Heap dominated by duplicates of common routing values / header values etc > - > > Key: QPID-7830 > URL: https://issues.apache.org/jira/browse/QPID-7830 > Project: Qpid > Issue Type: Bug > Components: Broker-J >Reporter: Keith Wall >Priority: Major > Fix For: Future > > > When used for store and forwarding, in some use cases the Broker's heap can > become dominated by duplicates of common values such as routing information > (e.g. {{amq.direct}} or an application's queue name) or common header values > (e.g a application/octet-stream or an application's user id). > On the 0-8..0-91 paths, every enqueued message gets its own > {{MessagePublishInfo}} referencing its own {{AMQShortString}} exchange and > routing keys. For some use-cases, these are drawn from a small set. On the > AMQP 1.0 path, {{Properties#to}} is an example. 0-10 is probably affected > too. > This unnecessarily increases the heap requirements of the Broker. > The Broker should adopt a sensible intern/caching policy with the same policy > applying regardless of whether messages follow the on-line enqueue or > recovery path. Note that in AMQP 1.0, values which are {{Symbols}} have > their underlying String automatically interned. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org For additional commands, e-mail: dev-h...@qpid.apache.org
[jira] [Commented] (QPID-7830) Heap dominated by duplicates of common routing values / header values etc
[ https://issues.apache.org/jira/browse/QPID-7830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16472217#comment-16472217 ] ASF subversion and git services commented on QPID-7830: --- Commit 995d535bdea6fb5efcab9c165dbd9c807f94e72e in qpid-broker-j's branch refs/heads/7.0.x from [~k-wall] [ https://git-wip-us.apache.org/repos/asf?p=qpid-broker-j.git;h=995d535 ] QPID-7830: [Broker-J] [AMQP 0-8..0-91] Cache AMQPShortStrings that relate to exchanges/routing keys and header values (that are usually drawn from a small domain) in a time/size bound cache. Intent is to reduce the amount of tenured garbage produced when messages are repeatedly sent to same destination. This should reduce frequency and length of GC pauses. (cherry picked from commit 7d7b50824ad9f2a99b7de034ef36a529129b00ac) > Heap dominated by duplicates of common routing values / header values etc > - > > Key: QPID-7830 > URL: https://issues.apache.org/jira/browse/QPID-7830 > Project: Qpid > Issue Type: Bug > Components: Broker-J >Reporter: Keith Wall >Priority: Major > Fix For: Future > > > When used for store and forwarding, in some use cases the Broker's heap can > become dominated by duplicates of common values such as routing information > (e.g. {{amq.direct}} or an application's queue name) or common header values > (e.g a application/octet-stream or an application's user id). > On the 0-8..0-91 paths, every enqueued message gets its own > {{MessagePublishInfo}} referencing its own {{AMQShortString}} exchange and > routing keys. For some use-cases, these are drawn from a small set. On the > AMQP 1.0 path, {{Properties#to}} is an example. 0-10 is probably affected > too. > This unnecessarily increases the heap requirements of the Broker. > The Broker should adopt a sensible intern/caching policy with the same policy > applying regardless of whether messages follow the on-line enqueue or > recovery path. Note that in AMQP 1.0, values which are {{Symbols}} have > their underlying String automatically interned. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org For additional commands, e-mail: dev-h...@qpid.apache.org
[jira] [Commented] (QPID-7830) Heap dominated by duplicates of common routing values / header values etc
[ https://issues.apache.org/jira/browse/QPID-7830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16470550#comment-16470550 ] ASF subversion and git services commented on QPID-7830: --- Commit 196b27065219b1283522bdb3a8f4fb1dda82ac16 in qpid-broker-j's branch refs/heads/master from [~alex.rufous] [ https://git-wip-us.apache.org/repos/asf?p=qpid-broker-j.git;h=196b270 ] QPID-7830: [Broker-J] Fix failing test > Heap dominated by duplicates of common routing values / header values etc > - > > Key: QPID-7830 > URL: https://issues.apache.org/jira/browse/QPID-7830 > Project: Qpid > Issue Type: Bug > Components: Broker-J >Reporter: Keith Wall >Priority: Major > Fix For: Future > > > When used for store and forwarding, in some use cases the Broker's heap can > become dominated by duplicates of common values such as routing information > (e.g. {{amq.direct}} or an application's queue name) or common header values > (e.g a application/octet-stream or an application's user id). > On the 0-8..0-91 paths, every enqueued message gets its own > {{MessagePublishInfo}} referencing its own {{AMQShortString}} exchange and > routing keys. For some use-cases, these are drawn from a small set. On the > AMQP 1.0 path, {{Properties#to}} is an example. 0-10 is probably affected > too. > This unnecessarily increases the heap requirements of the Broker. > The Broker should adopt a sensible intern/caching policy with the same policy > applying regardless of whether messages follow the on-line enqueue or > recovery path. Note that in AMQP 1.0, values which are {{Symbols}} have > their underlying String automatically interned. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org For additional commands, e-mail: dev-h...@qpid.apache.org
[jira] [Commented] (QPID-7830) Heap dominated by duplicates of common routing values / header values etc
[ https://issues.apache.org/jira/browse/QPID-7830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16466107#comment-16466107 ] ASF subversion and git services commented on QPID-7830: --- Commit 5b6dfcd94ef64544040de53d38e2463d327ebcb1 in qpid-broker-j's branch refs/heads/master from [~alex.rufous] [ https://git-wip-us.apache.org/repos/asf?p=qpid-broker-j.git;h=5b6dfcd ] QPID-7830: [Broker-J] Cache encoded and decoded values for 0-10 str8 > Heap dominated by duplicates of common routing values / header values etc > - > > Key: QPID-7830 > URL: https://issues.apache.org/jira/browse/QPID-7830 > Project: Qpid > Issue Type: Bug > Components: Broker-J >Reporter: Keith Wall >Priority: Major > Fix For: Future > > > When used for store and forwarding, in some use cases the Broker's heap can > become dominated by duplicates of common values such as routing information > (e.g. {{amq.direct}} or an application's queue name) or common header values > (e.g a application/octet-stream or an application's user id). > On the 0-8..0-91 paths, every enqueued message gets its own > {{MessagePublishInfo}} referencing its own {{AMQShortString}} exchange and > routing keys. For some use-cases, these are drawn from a small set. On the > AMQP 1.0 path, {{Properties#to}} is an example. 0-10 is probably affected > too. > This unnecessarily increases the heap requirements of the Broker. > The Broker should adopt a sensible intern/caching policy with the same policy > applying regardless of whether messages follow the on-line enqueue or > recovery path. Note that in AMQP 1.0, values which are {{Symbols}} have > their underlying String automatically interned. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org For additional commands, e-mail: dev-h...@qpid.apache.org
[jira] [Commented] (QPID-7830) Heap dominated by duplicates of common routing values / header values etc
[ https://issues.apache.org/jira/browse/QPID-7830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16463990#comment-16463990 ] Alex Rudyy commented on QPID-7830: -- I reviewed the changes. They look good to me. Though, I would add additional caching for 0-10 routing keys, exchange names (in DeliveryProperties), contentType (MessageProperties) and 1-0 "to" field (in Properties) as it would save us additional heap space and reduce frequency and length of GC pauses. > Heap dominated by duplicates of common routing values / header values etc > - > > Key: QPID-7830 > URL: https://issues.apache.org/jira/browse/QPID-7830 > Project: Qpid > Issue Type: Bug > Components: Broker-J >Reporter: Keith Wall >Priority: Major > Fix For: Future > > > When used for store and forwarding, in some use cases the Broker's heap can > become dominated by duplicates of common values such as routing information > (e.g. {{amq.direct}} or an application's queue name) or common header values > (e.g a application/octet-stream or an application's user id). > On the 0-8..0-91 paths, every enqueued message gets its own > {{MessagePublishInfo}} referencing its own {{AMQShortString}} exchange and > routing keys. For some use-cases, these are drawn from a small set. On the > AMQP 1.0 path, {{Properties#to}} is an example. 0-10 is probably affected > too. > This unnecessarily increases the heap requirements of the Broker. > The Broker should adopt a sensible intern/caching policy with the same policy > applying regardless of whether messages follow the on-line enqueue or > recovery path. Note that in AMQP 1.0, values which are {{Symbols}} have > their underlying String automatically interned. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org For additional commands, e-mail: dev-h...@qpid.apache.org
[jira] [Commented] (QPID-7830) Heap dominated by duplicates of common routing values / header values etc
[ https://issues.apache.org/jira/browse/QPID-7830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16456290#comment-16456290 ] ASF subversion and git services commented on QPID-7830: --- Commit ddc519a551061c682877784068e755677e2c6313 in qpid-broker-j's branch refs/heads/master from [~k-wall] [ https://git-wip-us.apache.org/repos/asf?p=qpid-broker-j.git;h=ddc519a ] QPID-7830: [Broker-J] [AMQP 0-8..0-91] Move caching responsubility to virtualhost > Heap dominated by duplicates of common routing values / header values etc > - > > Key: QPID-7830 > URL: https://issues.apache.org/jira/browse/QPID-7830 > Project: Qpid > Issue Type: Bug > Components: Broker-J >Reporter: Keith Wall >Priority: Major > Fix For: Future > > > When used for store and forwarding, in some use cases the Broker's heap can > become dominated by duplicates of common values such as routing information > (e.g. {{amq.direct}} or an application's queue name) or common header values > (e.g a application/octet-stream or an application's user id). > On the 0-8..0-91 paths, every enqueued message gets its own > {{MessagePublishInfo}} referencing its own {{AMQShortString}} exchange and > routing keys. For some use-cases, these are drawn from a small set. On the > AMQP 1.0 path, {{Properties#to}} is an example. 0-10 is probably affected > too. > This unnecessarily increases the heap requirements of the Broker. > The Broker should adopt a sensible intern/caching policy with the same policy > applying regardless of whether messages follow the on-line enqueue or > recovery path. Note that in AMQP 1.0, values which are {{Symbols}} have > their underlying String automatically interned. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org For additional commands, e-mail: dev-h...@qpid.apache.org
[jira] [Commented] (QPID-7830) Heap dominated by duplicates of common routing values / header values etc
[ https://issues.apache.org/jira/browse/QPID-7830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16454415#comment-16454415 ] ASF subversion and git services commented on QPID-7830: --- Commit 0a3dfc883b7af756fbdf076665eaae4ad8202bf7 in qpid-broker-j's branch refs/heads/master from [~k-wall] [ https://git-wip-us.apache.org/repos/asf?p=qpid-broker-j.git;h=0a3dfc8 ] QPID-7830: [Broker-J] [AMQP 0-8..0-91] Mechanically refactor AMQPShortString introducing factory methods and hiding constructors > Heap dominated by duplicates of common routing values / header values etc > - > > Key: QPID-7830 > URL: https://issues.apache.org/jira/browse/QPID-7830 > Project: Qpid > Issue Type: Bug > Components: Broker-J >Reporter: Keith Wall >Priority: Major > Fix For: Future > > > When used for store and forwarding, in some use cases the Broker's heap can > become dominated by duplicates of common values such as routing information > (e.g. {{amq.direct}} or an application's queue name) or common header values > (e.g a application/octet-stream or an application's user id). > On the 0-8..0-91 paths, every enqueued message gets its own > {{MessagePublishInfo}} referencing its own {{AMQShortString}} exchange and > routing keys. For some use-cases, these are drawn from a small set. On the > AMQP 1.0 path, {{Properties#to}} is an example. 0-10 is probably affected > too. > This unnecessarily increases the heap requirements of the Broker. > The Broker should adopt a sensible intern/caching policy with the same policy > applying regardless of whether messages follow the on-line enqueue or > recovery path. Note that in AMQP 1.0, values which are {{Symbols}} have > their underlying String automatically interned. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org For additional commands, e-mail: dev-h...@qpid.apache.org
[jira] [Commented] (QPID-7830) Heap dominated by duplicates of common routing values / header values etc
[ https://issues.apache.org/jira/browse/QPID-7830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16454416#comment-16454416 ] ASF subversion and git services commented on QPID-7830: --- Commit 7d7b50824ad9f2a99b7de034ef36a529129b00ac in qpid-broker-j's branch refs/heads/master from [~k-wall] [ https://git-wip-us.apache.org/repos/asf?p=qpid-broker-j.git;h=7d7b508 ] QPID-7830: [Broker-J] [AMQP 0-8..0-91] Cache AMQPShortStrings that relate to exchanges/routing keys and header values (that are usually drawn from a small domain) in a time/size bound cache. Intent is to reduce the amount of tenured garbage produced when messages are repeatedly sent to same destination. This should reduce frequency and length of GC pauses. > Heap dominated by duplicates of common routing values / header values etc > - > > Key: QPID-7830 > URL: https://issues.apache.org/jira/browse/QPID-7830 > Project: Qpid > Issue Type: Bug > Components: Broker-J >Reporter: Keith Wall >Priority: Major > Fix For: Future > > > When used for store and forwarding, in some use cases the Broker's heap can > become dominated by duplicates of common values such as routing information > (e.g. {{amq.direct}} or an application's queue name) or common header values > (e.g a application/octet-stream or an application's user id). > On the 0-8..0-91 paths, every enqueued message gets its own > {{MessagePublishInfo}} referencing its own {{AMQShortString}} exchange and > routing keys. For some use-cases, these are drawn from a small set. On the > AMQP 1.0 path, {{Properties#to}} is an example. 0-10 is probably affected > too. > This unnecessarily increases the heap requirements of the Broker. > The Broker should adopt a sensible intern/caching policy with the same policy > applying regardless of whether messages follow the on-line enqueue or > recovery path. Note that in AMQP 1.0, values which are {{Symbols}} have > their underlying String automatically interned. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org For additional commands, e-mail: dev-h...@qpid.apache.org
[jira] [Commented] (QPID-7830) Heap dominated by duplicates of common routing values / header values etc
[ https://issues.apache.org/jira/browse/QPID-7830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16220081#comment-16220081 ] Lorenz Quack commented on QPID-7830: All this sounds awfully complicated. What is wrong with something simpler like using a static class in broker-core {{StringCache}} that uses a [guava cache|https://github.com/google/guava/wiki/CachesExplained] with a [size limit|https://github.com/google/guava/wiki/CachesExplained#size-based-eviction] and [time based eviction|https://github.com/google/guava/wiki/CachesExplained#timed-eviction]? When a protocol layer then needs a String which we know might be repetitive (like the cases mentioned above) it calls {{StringCache#getString()}} and guava takes care of the rest. This might be a naive approach but I would like to at least discuss why it is not sufficient? > Heap dominated by duplicates of common routing values / header values etc > - > > Key: QPID-7830 > URL: https://issues.apache.org/jira/browse/QPID-7830 > Project: Qpid > Issue Type: Bug > Components: Java Broker >Reporter: Keith Wall > Fix For: Future > > > When used for store and forwarding, in some use cases the Broker's heap can > become dominated by duplicates of common values such as routing information > (e.g. {{amq.direct}} or an application's queue name) or common header values > (e.g a application/octet-stream or an application's user id). > On the 0-8..0-91 paths, every enqueued message gets its own > {{MessagePublishInfo}} referencing its own {{AMQShortString}} exchange and > routing keys. For some use-cases, these are drawn from a small set. On the > AMQP 1.0 path, {{Properties#to}} is an example. 0-10 is probably affected > too. > This unnecessarily increases the heap requirements of the Broker. > The Broker should adopt a sensible intern/caching policy with the same policy > applying regardless of whether messages follow the on-line enqueue or > recovery path. Note that in AMQP 1.0, values which are {{Symbols}} have > their underlying String automatically interned. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org For additional commands, e-mail: dev-h...@qpid.apache.org
[jira] [Commented] (QPID-7830) Heap dominated by duplicates of common routing values / header values etc
[ https://issues.apache.org/jira/browse/QPID-7830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16218271#comment-16218271 ] Keith Wall commented on QPID-7830: -- I was pondering the problem of which object(s) in the Broker would be responsible for holding the caches of commonly used names such as destinations or content-types. Global caches are awkward as they don't work well in the embedded use-cases ({{String.intern}} falls into the category). I think that we would want separate caches for different protocols - which will avoid polluting AMQP 1.0 with AMQP 0-9..0-10 concerns. I think we should have the VirtualHost additionally expose a set ProtocolContext objects. ProtocolContext objects would be service loaded using the Broker's existing pluggable pattern (factory etc). A protocol implementations could one provide a factory if it wished. The protocol layer itself would lookup its ProtocolContext from the VirtualHost. In a sense a ProtocolObject would serve as a protocol specific 'singleton' scoped to the virtualhost. Turning back to the caching problem, in the 0-8..0-91 case,. its ProtocolContext would expose a destination cache, content type cache etc. The 0-8 protocol layer, on the enqueue path, would consult these caches and substitute the values from the cache if present. In the case of the destination cache, we could make it respond to changes in the model so that adding a queue or exchange automatically added a value to the destination cache and removing one purged it. Perhaps for things like routingKeys, contentTypes etc we keep a bounded LRU cache. > Heap dominated by duplicates of common routing values / header values etc > - > > Key: QPID-7830 > URL: https://issues.apache.org/jira/browse/QPID-7830 > Project: Qpid > Issue Type: Bug > Components: Java Broker >Reporter: Keith Wall > Fix For: Future > > > When used for store and forwarding, in some use cases the Broker's heap can > become dominated by duplicates of common values such as routing information > (e.g. {{amq.direct}} or an application's queue name) or common header values > (e.g a application/octet-stream or an application's user id). > On the 0-8..0-91 paths, every enqueued message gets its own > {{MessagePublishInfo}} referencing its own {{AMQShortString}} exchange and > routing keys. For some use-cases, these are drawn from a small set. On the > AMQP 1.0 path, {{Properties#to}} is an example. 0-10 is probably affected > too. > This unnecessarily increases the heap requirements of the Broker. > The Broker should adopt a sensible intern/caching policy with the same policy > applying regardless of whether messages follow the on-line enqueue or > recovery path. Note that in AMQP 1.0, values which are {{Symbols}} have > their underlying String automatically interned. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org For additional commands, e-mail: dev-h...@qpid.apache.org
[jira] [Commented] (QPID-7830) Heap dominated by duplicates of common routing values / header values etc
[ https://issues.apache.org/jira/browse/QPID-7830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16055947#comment-16055947 ] Rob Godfrey commented on QPID-7830: --- {quote} On the AMQP 0-8..0-91, with things like contentType, path I wonder if keeping the BasicContentHeaderProperties#_contentType as a separate AMQShortString is useful. {quote} I agree - it would be better here to simply use Strings for all the string like properties in the content header properties, and we can have a static list of "common" types, and if the string is in that list we'll use the static value. > Heap dominated by duplicates of common routing values / header values etc > - > > Key: QPID-7830 > URL: https://issues.apache.org/jira/browse/QPID-7830 > Project: Qpid > Issue Type: Bug > Components: Java Broker >Reporter: Keith Wall > > When used for store and forwarding, in some use cases the Broker's heap can > become dominated by duplicates of common values such as routing information > (e.g. {{amq.direct}} or an application's queue name) or common header values > (e.g a application/octet-stream or an application's user id). > On the 0-8..0-91 paths, every enqueued message gets its own > {{MessagePublishInfo}} referencing its own {{AMQShortString}} exchange and > routing keys. For some use-cases, these are drawn from a small set. On the > AMQP 1.0 path, {{Properties#to}} is an example. 0-10 is probably affected > too. > This unnecessarily increases the heap requirements of the Broker. > The Broker should adopt a sensible intern/caching policy with the same policy > applying regardless of whether messages follow the on-line enqueue or > recovery path. Note that in AMQP 1.0, values which are {{Symbols}} have > their underlying String automatically interned. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org For additional commands, e-mail: dev-h...@qpid.apache.org
[jira] [Commented] (QPID-7830) Heap dominated by duplicates of common routing values / header values etc
[ https://issues.apache.org/jira/browse/QPID-7830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16055895#comment-16055895 ] Keith Wall commented on QPID-7830: -- IIRC we removed the old AMQShortString caching as our algorithm didn't discriminate about queue names/routing keys that related to temporary queues. That caused our cache to appear to leak in some use cases. If we were to reintroduce caching we'd need to deal with this case (perhaps by purging the cache of a corresponding queue/exchange mapping when a exchange or queue is deleted). On the AMQP 0-8..0-91, with things like contentType, path I wonder if keeping the BasicContentHeaderProperties#_contentType as a separate AMQShortString is useful. Our main use case for this is routing (selector evaluation) and this ultimately needs a {{java.lang.String}}. We only use the {{AMQShortString}} form if the {{BasicContentHeaderProperties #_encodedForm}} has been cleared (flow to disk). In this event, the AMQShortString form could always be reconstructed. If we were to switch _contentType from {{AMQShortString}} to {{String}}, we can then use {{String.intern}} (with perhaps a whitelist of our commonly used values). > Heap dominated by duplicates of common routing values / header values etc > - > > Key: QPID-7830 > URL: https://issues.apache.org/jira/browse/QPID-7830 > Project: Qpid > Issue Type: Bug > Components: Java Broker >Reporter: Keith Wall > > When used for store and forwarding, in some use cases the Broker's heap can > become dominated by duplicates of common values such as routing information > (e.g. {{amq.direct}} or an application's queue name) or common header values > (e.g a application/octet-stream or an application's user id). > On the 0-8..0-91 paths, every enqueued message gets its own > {{MessagePublishInfo}} referencing its own {{AMQShortString}} exchange and > routing keys. For some use-cases, these are drawn from a small set. On the > AMQP 1.0 path, {{Properties#to}} is an example. 0-10 is probably affected > too. > This unnecessarily increases the heap requirements of the Broker. > The Broker should adopt a sensible intern/caching policy. Note that in AMQP > 1.0, values which are {{Symbols}} have their underlying String automatically > interned. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org For additional commands, e-mail: dev-h...@qpid.apache.org
[jira] [Commented] (QPID-7830) Heap dominated by duplicates of common routing values / header values etc
[ https://issues.apache.org/jira/browse/QPID-7830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16053913#comment-16053913 ] Rob Godfrey commented on QPID-7830: --- One approach might be something like keeping a pool of string which are likely to be reused (such as queue names, exchange names, etc), and checking in the pool to find the shared copy. I'd be hesitant to try to pool routing keys explicitly since by design these may pattern match... however for the common routing-key=queue-name case, simply caching the queue names would be sufficient. Historically we did cache short string values, but this was removed at some point as I don't think it was adding much value. > Heap dominated by duplicates of common routing values / header values etc > - > > Key: QPID-7830 > URL: https://issues.apache.org/jira/browse/QPID-7830 > Project: Qpid > Issue Type: Bug > Components: Java Broker >Reporter: Keith Wall > > When used for store and forwarding, in some use cases the Broker's heap can > become dominated by duplicates of common values such as routing information > (e.g. {{amq.direct}} or an application's queue name) or common header values > (e.g a application/octet-stream or an application's user id). > On the 0-8..0-91 paths, every enqueued message gets its own > {{MessagePublishInfo}} referencing its own {{AMQShortString}} exchange and > routing keys. For some use-cases, these are drawn from a small set. On the > AMQP 1.0 path, {{Properties#to}} is an example. 0-10 is probably affected > too. > This unnecessarily increases the heap requirements of the Broker. > The Broker should adopt a sensible intern/caching policy. Note that in AMQP > 1.0, values which are {{Symbols}} have their underlying String automatically > interned. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org For additional commands, e-mail: dev-h...@qpid.apache.org