[jira] [Commented] (AMQ-6477) ReduceMemoryFootprint should apply to non-persistent messages and subscriptions
[ https://issues.apache.org/jira/browse/AMQ-6477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15625572#comment-15625572 ] ASF subversion and git services commented on AMQ-6477: -- Commit fc3206864d895a469af39ec2f3f6808ad936b908 in activemq's branch refs/heads/activemq-5.14.x from [~cshannon] [ https://git-wip-us.apache.org/repos/asf?p=activemq.git;h=fc32068 ] https://issues.apache.org/jira/browse/AMQ-6477 Fixing potential concurrent modification exception (cherry picked from commit 5c80eda321e7edb5f34ffd62c71523310d26b2ca) > ReduceMemoryFootprint should apply to non-persistent messages and > subscriptions > --- > > Key: AMQ-6477 > URL: https://issues.apache.org/jira/browse/AMQ-6477 > Project: ActiveMQ > Issue Type: Bug > Components: Broker >Affects Versions: 5.14.1 >Reporter: Christopher L. Shannon >Assignee: Christopher L. Shannon > Fix For: 5.15.0, 5.14.2 > > > There is a flag called reduceMemoryFootprint which will clear out the > unmarshalled state of a message after it is added to a queue/topic because > that state isn't counted towards the size and can lead to OOM errors. > However, even setting this flag, I am still seeing some brokers run out of > memory. After analyzing the heap dumps I have found 2 reasons for this: > 1) Non-persistent messages do not have their unmarshalled state cleared. > This was done because when a message is persisted it is guaranteed to have > the marshalled state. However we can still clear the memory for > non-persistent messages as long as we check to make sure the marshalled state > exists first, which it will for transports like TCP but won't exist for the > VM transport. > 2) When a message is added to a subscription the properties are needed to > check which subscription messages can be dispatched to. This causes the > memory to be unmarshalled again. > In the topic case, we should probably defer the clearing of the state until > after the message is added to a subscription because messages are immediately > dispatched to topic subs so we don't unnecessarily have to convert the data > twice. In the queue case it probably makes sense to clear memory on add to > the queue and on add to the subscription. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AMQ-6477) ReduceMemoryFootprint should apply to non-persistent messages and subscriptions
[ https://issues.apache.org/jira/browse/AMQ-6477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15625570#comment-15625570 ] ASF subversion and git services commented on AMQ-6477: -- Commit 5c80eda321e7edb5f34ffd62c71523310d26b2ca in activemq's branch refs/heads/master from [~cshannon] [ https://git-wip-us.apache.org/repos/asf?p=activemq.git;h=5c80eda ] https://issues.apache.org/jira/browse/AMQ-6477 Fixing potential concurrent modification exception > ReduceMemoryFootprint should apply to non-persistent messages and > subscriptions > --- > > Key: AMQ-6477 > URL: https://issues.apache.org/jira/browse/AMQ-6477 > Project: ActiveMQ > Issue Type: Bug > Components: Broker >Affects Versions: 5.14.1 >Reporter: Christopher L. Shannon >Assignee: Christopher L. Shannon > Fix For: 5.15.0, 5.14.2 > > > There is a flag called reduceMemoryFootprint which will clear out the > unmarshalled state of a message after it is added to a queue/topic because > that state isn't counted towards the size and can lead to OOM errors. > However, even setting this flag, I am still seeing some brokers run out of > memory. After analyzing the heap dumps I have found 2 reasons for this: > 1) Non-persistent messages do not have their unmarshalled state cleared. > This was done because when a message is persisted it is guaranteed to have > the marshalled state. However we can still clear the memory for > non-persistent messages as long as we check to make sure the marshalled state > exists first, which it will for transports like TCP but won't exist for the > VM transport. > 2) When a message is added to a subscription the properties are needed to > check which subscription messages can be dispatched to. This causes the > memory to be unmarshalled again. > In the topic case, we should probably defer the clearing of the state until > after the message is added to a subscription because messages are immediately > dispatched to topic subs so we don't unnecessarily have to convert the data > twice. In the queue case it probably makes sense to clear memory on add to > the queue and on add to the subscription. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AMQ-6477) ReduceMemoryFootprint should apply to non-persistent messages and subscriptions
[ https://issues.apache.org/jira/browse/AMQ-6477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15613019#comment-15613019 ] ASF subversion and git services commented on AMQ-6477: -- Commit 4cbe692bccb5bb246a2099f8504e103ac90079be in activemq's branch refs/heads/activemq-5.14.x from [~cshannon] [ https://git-wip-us.apache.org/repos/asf?p=activemq.git;h=4cbe692 ] https://issues.apache.org/jira/browse/AMQ-6477 simplifying isMarshalled method (cherry picked from commit 0a80165a99bff33bfaeeb8fe1dc7b5a8e6f50830) > ReduceMemoryFootprint should apply to non-persistent messages and > subscriptions > --- > > Key: AMQ-6477 > URL: https://issues.apache.org/jira/browse/AMQ-6477 > Project: ActiveMQ > Issue Type: Bug > Components: Broker >Affects Versions: 5.14.1 >Reporter: Christopher L. Shannon >Assignee: Christopher L. Shannon > Fix For: 5.15.0, 5.14.2 > > > There is a flag called reduceMemoryFootprint which will clear out the > unmarshalled state of a message after it is added to a queue/topic because > that state isn't counted towards the size and can lead to OOM errors. > However, even setting this flag, I am still seeing some brokers run out of > memory. After analyzing the heap dumps I have found 2 reasons for this: > 1) Non-persistent messages do not have their unmarshalled state cleared. > This was done because when a message is persisted it is guaranteed to have > the marshalled state. However we can still clear the memory for > non-persistent messages as long as we check to make sure the marshalled state > exists first, which it will for transports like TCP but won't exist for the > VM transport. > 2) When a message is added to a subscription the properties are needed to > check which subscription messages can be dispatched to. This causes the > memory to be unmarshalled again. > In the topic case, we should probably defer the clearing of the state until > after the message is added to a subscription because messages are immediately > dispatched to topic subs so we don't unnecessarily have to convert the data > twice. In the queue case it probably makes sense to clear memory on add to > the queue and on add to the subscription. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AMQ-6477) ReduceMemoryFootprint should apply to non-persistent messages and subscriptions
[ https://issues.apache.org/jira/browse/AMQ-6477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15613017#comment-15613017 ] ASF subversion and git services commented on AMQ-6477: -- Commit 0a80165a99bff33bfaeeb8fe1dc7b5a8e6f50830 in activemq's branch refs/heads/master from [~cshannon] [ https://git-wip-us.apache.org/repos/asf?p=activemq.git;h=0a80165 ] https://issues.apache.org/jira/browse/AMQ-6477 simplifying isMarshalled method > ReduceMemoryFootprint should apply to non-persistent messages and > subscriptions > --- > > Key: AMQ-6477 > URL: https://issues.apache.org/jira/browse/AMQ-6477 > Project: ActiveMQ > Issue Type: Bug > Components: Broker >Affects Versions: 5.14.1 >Reporter: Christopher L. Shannon >Assignee: Christopher L. Shannon > Fix For: 5.15.0, 5.14.2 > > > There is a flag called reduceMemoryFootprint which will clear out the > unmarshalled state of a message after it is added to a queue/topic because > that state isn't counted towards the size and can lead to OOM errors. > However, even setting this flag, I am still seeing some brokers run out of > memory. After analyzing the heap dumps I have found 2 reasons for this: > 1) Non-persistent messages do not have their unmarshalled state cleared. > This was done because when a message is persisted it is guaranteed to have > the marshalled state. However we can still clear the memory for > non-persistent messages as long as we check to make sure the marshalled state > exists first, which it will for transports like TCP but won't exist for the > VM transport. > 2) When a message is added to a subscription the properties are needed to > check which subscription messages can be dispatched to. This causes the > memory to be unmarshalled again. > In the topic case, we should probably defer the clearing of the state until > after the message is added to a subscription because messages are immediately > dispatched to topic subs so we don't unnecessarily have to convert the data > twice. In the queue case it probably makes sense to clear memory on add to > the queue and on add to the subscription. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AMQ-6477) ReduceMemoryFootprint should apply to non-persistent messages and subscriptions
[ https://issues.apache.org/jira/browse/AMQ-6477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15605444#comment-15605444 ] ASF subversion and git services commented on AMQ-6477: -- Commit e0116d0458769881396d9f848d63496421fd59d5 in activemq's branch refs/heads/activemq-5.14.x from [~cshannon] [ https://git-wip-us.apache.org/repos/asf?p=activemq.git;h=e0116d0 ] https://issues.apache.org/jira/browse/AMQ-6477 ReduceMemoryFootprint now applies to non-persistent messages if they have been marshalled and topics now clear memory after the recovery policy check (cherry picked from commit 7c3bb401007b4047c540287b53b435b20d3161c0) > ReduceMemoryFootprint should apply to non-persistent messages and > subscriptions > --- > > Key: AMQ-6477 > URL: https://issues.apache.org/jira/browse/AMQ-6477 > Project: ActiveMQ > Issue Type: Bug > Components: Broker >Affects Versions: 5.14.1 >Reporter: Christopher L. Shannon >Assignee: Christopher L. Shannon > > There is a flag called reduceMemoryFootprint which will clear out the > unmarshalled state of a message after it is added to a queue/topic because > that state isn't counted towards the size and can lead to OOM errors. > However, even setting this flag, I am still seeing some brokers run out of > memory. After analyzing the heap dumps I have found 2 reasons for this: > 1) Non-persistent messages do not have their unmarshalled state cleared. > This was done because when a message is persisted it is guaranteed to have > the marshalled state. However we can still clear the memory for > non-persistent messages as long as we check to make sure the marshalled state > exists first, which it will for transports like TCP but won't exist for the > VM transport. > 2) When a message is added to a subscription the properties are needed to > check which subscription messages can be dispatched to. This causes the > memory to be unmarshalled again. > In the topic case, we should probably defer the clearing of the state until > after the message is added to a subscription because messages are immediately > dispatched to topic subs so we don't unnecessarily have to convert the data > twice. In the queue case it probably makes sense to clear memory on add to > the queue and on add to the subscription. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AMQ-6477) ReduceMemoryFootprint should apply to non-persistent messages and subscriptions
[ https://issues.apache.org/jira/browse/AMQ-6477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15605442#comment-15605442 ] ASF subversion and git services commented on AMQ-6477: -- Commit 7c3bb401007b4047c540287b53b435b20d3161c0 in activemq's branch refs/heads/master from [~cshannon] [ https://git-wip-us.apache.org/repos/asf?p=activemq.git;h=7c3bb40 ] https://issues.apache.org/jira/browse/AMQ-6477 ReduceMemoryFootprint now applies to non-persistent messages if they have been marshalled and topics now clear memory after the recovery policy check > ReduceMemoryFootprint should apply to non-persistent messages and > subscriptions > --- > > Key: AMQ-6477 > URL: https://issues.apache.org/jira/browse/AMQ-6477 > Project: ActiveMQ > Issue Type: Bug > Components: Broker >Affects Versions: 5.14.1 >Reporter: Christopher L. Shannon >Assignee: Christopher L. Shannon > > There is a flag called reduceMemoryFootprint which will clear out the > unmarshalled state of a message after it is added to a queue/topic because > that state isn't counted towards the size and can lead to OOM errors. > However, even setting this flag, I am still seeing some brokers run out of > memory. After analyzing the heap dumps I have found 2 reasons for this: > 1) Non-persistent messages do not have their unmarshalled state cleared. > This was done because when a message is persisted it is guaranteed to have > the marshalled state. However we can still clear the memory for > non-persistent messages as long as we check to make sure the marshalled state > exists first, which it will for transports like TCP but won't exist for the > VM transport. > 2) When a message is added to a subscription the properties are needed to > check which subscription messages can be dispatched to. This causes the > memory to be unmarshalled again. > In the topic case, we should probably defer the clearing of the state until > after the message is added to a subscription because messages are immediately > dispatched to topic subs so we don't unnecessarily have to convert the data > twice. In the queue case it probably makes sense to clear memory on add to > the queue and on add to the subscription. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AMQ-6477) ReduceMemoryFootprint should apply to non-persistent messages and subscriptions
[ https://issues.apache.org/jira/browse/AMQ-6477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15605338#comment-15605338 ] Christopher L. Shannon commented on AMQ-6477: - [~gtully], I've decided to be slightly more conservative for this patch. I don't feel super comfortable putting the memory clearing into the add method of the subscription like I did in my original patch because we would get into a state where we have to unmarshall/marshal the properties over and over again for each sub if there is a selector. Plus there could be race condition issues at that point since the message has now been handed off to the different subscriptions. So the new plan is to: 1) Still handle clearing the memory for non-persistent messages by checking if content and marshalledProperties are not null (or if both marshalledProperties and properties are null) 2) For topics, move the clear memory method call from doMessageSend to the dispatch method and put it after the subscriptionRecoveryPolicy check (which needs to use the properties and triggers and unmarshal) but before the dispatch itself. This will at least fix non-persistent messages and it will also clear memory for topic subscriptions as long as there are no selectors (which is a less common case). For the selector case where the dispatch policy triggers an unmarshall I would need to think about it a bit more or maybe we'd just need to do something like you mentioned earlier where we actually count the size of both objects in memory. > ReduceMemoryFootprint should apply to non-persistent messages and > subscriptions > --- > > Key: AMQ-6477 > URL: https://issues.apache.org/jira/browse/AMQ-6477 > Project: ActiveMQ > Issue Type: Bug > Components: Broker >Affects Versions: 5.14.1 >Reporter: Christopher L. Shannon >Assignee: Christopher L. Shannon > > There is a flag called reduceMemoryFootprint which will clear out the > unmarshalled state of a message after it is added to a queue/topic because > that state isn't counted towards the size and can lead to OOM errors. > However, even setting this flag, I am still seeing some brokers run out of > memory. After analyzing the heap dumps I have found 2 reasons for this: > 1) Non-persistent messages do not have their unmarshalled state cleared. > This was done because when a message is persisted it is guaranteed to have > the marshalled state. However we can still clear the memory for > non-persistent messages as long as we check to make sure the marshalled state > exists first, which it will for transports like TCP but won't exist for the > VM transport. > 2) When a message is added to a subscription the properties are needed to > check which subscription messages can be dispatched to. This causes the > memory to be unmarshalled again. > In the topic case, we should probably defer the clearing of the state until > after the message is added to a subscription because messages are immediately > dispatched to topic subs so we don't unnecessarily have to convert the data > twice. In the queue case it probably makes sense to clear memory on add to > the queue and on add to the subscription. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AMQ-6477) ReduceMemoryFootprint should apply to non-persistent messages and subscriptions
[ https://issues.apache.org/jira/browse/AMQ-6477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15602395#comment-15602395 ] Christopher L. Shannon commented on AMQ-6477: - May not need the sync after all since the dispatch happens in the same thread sequentially (the dispatch policy just iterates over each subscription 1 at a time) > ReduceMemoryFootprint should apply to non-persistent messages and > subscriptions > --- > > Key: AMQ-6477 > URL: https://issues.apache.org/jira/browse/AMQ-6477 > Project: ActiveMQ > Issue Type: Bug > Components: Broker >Affects Versions: 5.14.1 >Reporter: Christopher L. Shannon >Assignee: Christopher L. Shannon > > There is a flag called reduceMemoryFootprint which will clear out the > unmarshalled state of a message after it is added to a queue/topic because > that state isn't counted towards the size and can lead to OOM errors. > However, even setting this flag, I am still seeing some brokers run out of > memory. After analyzing the heap dumps I have found 2 reasons for this: > 1) Non-persistent messages do not have their unmarshalled state cleared. > This was done because when a message is persisted it is guaranteed to have > the marshalled state. However we can still clear the memory for > non-persistent messages as long as we check to make sure the marshalled state > exists first, which it will for transports like TCP but won't exist for the > VM transport. > 2) When a message is added to a subscription the properties are needed to > check which subscription messages can be dispatched to. This causes the > memory to be unmarshalled again. > In the topic case, we should probably defer the clearing of the state until > after the message is added to a subscription because messages are immediately > dispatched to topic subs so we don't unnecessarily have to convert the data > twice. In the queue case it probably makes sense to clear memory on add to > the queue and on add to the subscription. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AMQ-6477) ReduceMemoryFootprint should apply to non-persistent messages and subscriptions
[
https://issues.apache.org/jira/browse/AMQ-6477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15602290#comment-15602290
]
Christopher L. Shannon commented on AMQ-6477:
-
That's a good point about not sharing the message while doing the clearing
which I think could be an issue with topic subscriptions since the message will
potentially be passed to multiple subs.
When doing the clear in the subscriptions maybe we need to do something like
add a synchronize around the clear block:
{noformat}
if (isReduceMemoryFootprint()) {
synchronized(node.getMessage()) {
if (node.getMessage().isMarshalled()) {
node.getMessage().clearUnMarshalledState();
}
}
{noformat}
Ultimately, I agree about having the size count both states. That would
protect form OOM errors for when the message contains both of the states while
also trying to save memory when it can. The hard part is trying to estimate
the size of the properties map.
> ReduceMemoryFootprint should apply to non-persistent messages and
> subscriptions
> ---
>
> Key: AMQ-6477
> URL: https://issues.apache.org/jira/browse/AMQ-6477
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
>Affects Versions: 5.14.1
>Reporter: Christopher L. Shannon
>Assignee: Christopher L. Shannon
>
> There is a flag called reduceMemoryFootprint which will clear out the
> unmarshalled state of a message after it is added to a queue/topic because
> that state isn't counted towards the size and can lead to OOM errors.
> However, even setting this flag, I am still seeing some brokers run out of
> memory. After analyzing the heap dumps I have found 2 reasons for this:
> 1) Non-persistent messages do not have their unmarshalled state cleared.
> This was done because when a message is persisted it is guaranteed to have
> the marshalled state. However we can still clear the memory for
> non-persistent messages as long as we check to make sure the marshalled state
> exists first, which it will for transports like TCP but won't exist for the
> VM transport.
> 2) When a message is added to a subscription the properties are needed to
> check which subscription messages can be dispatched to. This causes the
> memory to be unmarshalled again.
> In the topic case, we should probably defer the clearing of the state until
> after the message is added to a subscription because messages are immediately
> dispatched to topic subs so we don't unnecessarily have to convert the data
> twice. In the queue case it probably makes sense to clear memory on add to
> the queue and on add to the subscription.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (AMQ-6477) ReduceMemoryFootprint should apply to non-persistent messages and subscriptions
[ https://issues.apache.org/jira/browse/AMQ-6477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15602237#comment-15602237 ] Gary Tully commented on AMQ-6477: - [~cshannon] looks good one thought: I guess it also means that on each dispatch, so when prefetched messages are redelivered, there will be an unmarshall/clear. it makes me think that maybe doing something to ensure that properties in both states are part of the size. That would tackle directly the OOM. However the non persistent case is valid and I guess the selector case is less common. One important aspect is ensuring no sharing of a message when the destructive change to clear the state is made. With composite sends or virtual topics (mostly there is copy) but some care there may be needed. Also consumer that use a composite dest to subscribe to multiple dests. Recall the test that had us add/remove some sync in message in the past. I think it will be fine, just be sure there is some existing sync point that will protect you in the case of sharing. > ReduceMemoryFootprint should apply to non-persistent messages and > subscriptions > --- > > Key: AMQ-6477 > URL: https://issues.apache.org/jira/browse/AMQ-6477 > Project: ActiveMQ > Issue Type: Bug > Components: Broker >Affects Versions: 5.14.1 >Reporter: Christopher L. Shannon >Assignee: Christopher L. Shannon > > There is a flag called reduceMemoryFootprint which will clear out the > unmarshalled state of a message after it is added to a queue/topic because > that state isn't counted towards the size and can lead to OOM errors. > However, even setting this flag, I am still seeing some brokers run out of > memory. After analyzing the heap dumps I have found 2 reasons for this: > 1) Non-persistent messages do not have their unmarshalled state cleared. > This was done because when a message is persisted it is guaranteed to have > the marshalled state. However we can still clear the memory for > non-persistent messages as long as we check to make sure the marshalled state > exists first, which it will for transports like TCP but won't exist for the > VM transport. > 2) When a message is added to a subscription the properties are needed to > check which subscription messages can be dispatched to. This causes the > memory to be unmarshalled again. > In the topic case, we should probably defer the clearing of the state until > after the message is added to a subscription because messages are immediately > dispatched to topic subs so we don't unnecessarily have to convert the data > twice. In the queue case it probably makes sense to clear memory on add to > the queue and on add to the subscription. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AMQ-6477) ReduceMemoryFootprint should apply to non-persistent messages and subscriptions
[ https://issues.apache.org/jira/browse/AMQ-6477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15602083#comment-15602083 ] Christopher L. Shannon commented on AMQ-6477: - [~gtully], Mind taking a quick look at this patch I created? I haven't written tests yet but this should give you an idea the direction I am going with this and I wanted to see if you thought I was overlooking anything. https://github.com/cshannon/activemq/commit/c07b156ac1308bf8a5cbc52f97a96be730166f1d > ReduceMemoryFootprint should apply to non-persistent messages and > subscriptions > --- > > Key: AMQ-6477 > URL: https://issues.apache.org/jira/browse/AMQ-6477 > Project: ActiveMQ > Issue Type: Bug > Components: Broker >Affects Versions: 5.14.1 >Reporter: Christopher L. Shannon >Assignee: Christopher L. Shannon > > There is a flag called reduceMemoryFootprint which will clear out the > unmarshalled state of a message after it is added to a queue/topic because > that state isn't counted towards the size and can lead to OOM errors. > However, even setting this flag, I am still seeing some brokers run out of > memory. After analyzing the heap dumps I have found 2 reasons for this: > 1) Non-persistent messages do not have their unmarshalled state cleared. > This was done because when a message is persisted it is guaranteed to have > the marshalled state. However we can still clear the memory for > non-persistent messages as long as we check to make sure the marshalled state > exists first, which it will for transports like TCP but won't exist for the > VM transport. > 2) When a message is added to a subscription the properties are needed to > check which subscription messages can be dispatched to. This causes the > memory to be unmarshalled again. > In the topic case, we should probably defer the clearing of the state until > after the message is added to a subscription because messages are immediately > dispatched to topic subs so we don't unnecessarily have to convert the data > twice. In the queue case it probably makes sense to clear memory on add to > the queue and on add to the subscription. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
