[jira] [Commented] (FELIX-6529) Improve memory usage ManifestParser using String deduplication
[ https://issues.apache.org/jira/browse/FELIX-6529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17540913#comment-17540913 ] Amit Mondal commented on FELIX-6529: [~karlpauls] The duplicate strings I encountered earlier were scattered in different places in framework. With this change, a bit over few KBs are spared which I feel is a really good improvement. I will, therefore, try to look into the other places in the framework that can be improved as well by removing the duplicate strings. Hence, there is nothing to be done as of now in reference to this ticket. Thanks a lot for your cooperation (y) > Improve memory usage ManifestParser using String deduplication > -- > > Key: FELIX-6529 > URL: https://issues.apache.org/jira/browse/FELIX-6529 > Project: Felix > Issue Type: Improvement >Reporter: Johannes Edmeier >Assignee: Karl Pauls >Priority: Major > Fix For: framework-7.0.4 > > Attachments: ManifestParser.patch, image-2022-05-13-14-16-39-509.png, > image-2022-05-13-14-17-55-965.png > > > In my heap dump I've seen a lot of duplicate Strings produced by the > ManifestParser. > It creates a lot of equal strings for the keys in the manifest but doesn't > deduplicate them and they're hold forever producing a lot on retained heap. > I've patched the ManifestParser to deduplicate just the keys and could save > ~8 Megs of heap. > Total usage before: 38MB after: 30MB > Duplicated Strings before: > !image-2022-05-13-14-16-39-509.png|width=658,height=203! > Duplicated Strings after: > !image-2022-05-13-14-17-55-965.png|width=793,height=200! > See patch attached. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (FELIX-6529) Improve memory usage ManifestParser using String deduplication
[ https://issues.apache.org/jira/browse/FELIX-6529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17540889#comment-17540889 ] Karl Pauls commented on FELIX-6529: --- [~amitmondal], I might be wrong, but I don't see these being related to keys anymore. Maybe you could help in finding out where they come from (looks like at a minimum quite some are parsed filters) and open a new issue(s) for further optimizations. Additionally, for this issue, I would still be interested in the overall impact. You said that just interning the keys did save you 8mb in your use-case. What did you save with this fix? > Improve memory usage ManifestParser using String deduplication > -- > > Key: FELIX-6529 > URL: https://issues.apache.org/jira/browse/FELIX-6529 > Project: Felix > Issue Type: Improvement >Reporter: Johannes Edmeier >Assignee: Karl Pauls >Priority: Major > Fix For: framework-7.0.4 > > Attachments: ManifestParser.patch, image-2022-05-13-14-16-39-509.png, > image-2022-05-13-14-17-55-965.png > > > In my heap dump I've seen a lot of duplicate Strings produced by the > ManifestParser. > It creates a lot of equal strings for the keys in the manifest but doesn't > deduplicate them and they're hold forever producing a lot on retained heap. > I've patched the ManifestParser to deduplicate just the keys and could save > ~8 Megs of heap. > Total usage before: 38MB after: 30MB > Duplicated Strings before: > !image-2022-05-13-14-16-39-509.png|width=658,height=203! > Duplicated Strings after: > !image-2022-05-13-14-17-55-965.png|width=793,height=200! > See patch attached. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (FELIX-6529) Improve memory usage ManifestParser using String deduplication
[ https://issues.apache.org/jira/browse/FELIX-6529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17540869#comment-17540869 ] Amit Mondal commented on FELIX-6529: [~karlpauls] I have just tested and analyzed the Framework 7.1.0-SNASHOT. Unfortunately, there exist still lots of different duplicate strings that originate from the Felix FW. For your further analysis, I have extracted the dominant ones in CSV files that include only the GC roots for the instances created in the FW. Here is the [link|https://amitinside-my.sharepoint.com/:u:/p/admin/EQ5xg7SFlgVNnVdFZUExOt8Bxxk1WCJl9AZMeirU-XMcTQ?e=6jXNBx] to download the ZIP file comprising the CSV files with the GC roots to the dominators. > Improve memory usage ManifestParser using String deduplication > -- > > Key: FELIX-6529 > URL: https://issues.apache.org/jira/browse/FELIX-6529 > Project: Felix > Issue Type: Improvement >Reporter: Johannes Edmeier >Assignee: Karl Pauls >Priority: Major > Fix For: framework-7.0.4 > > Attachments: ManifestParser.patch, image-2022-05-13-14-16-39-509.png, > image-2022-05-13-14-17-55-965.png > > > In my heap dump I've seen a lot of duplicate Strings produced by the > ManifestParser. > It creates a lot of equal strings for the keys in the manifest but doesn't > deduplicate them and they're hold forever producing a lot on retained heap. > I've patched the ManifestParser to deduplicate just the keys and could save > ~8 Megs of heap. > Total usage before: 38MB after: 30MB > Duplicated Strings before: > !image-2022-05-13-14-16-39-509.png|width=658,height=203! > Duplicated Strings after: > !image-2022-05-13-14-17-55-965.png|width=793,height=200! > See patch attached. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (FELIX-6529) Improve memory usage ManifestParser using String deduplication
[ https://issues.apache.org/jira/browse/FELIX-6529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17539843#comment-17539843 ] Amit Mondal commented on FELIX-6529: [~karlpauls] Thanks a lot for your quick response (y) I didn't know about this remote repo URL where snapshots are deployed. Since I am using Bndtools, I will configure it accordingly to consume the snapshot version of the framework and give it a thorough test. I will keep you updated. > Improve memory usage ManifestParser using String deduplication > -- > > Key: FELIX-6529 > URL: https://issues.apache.org/jira/browse/FELIX-6529 > Project: Felix > Issue Type: Improvement >Reporter: Johannes Edmeier >Assignee: Karl Pauls >Priority: Major > Fix For: framework-7.0.4 > > Attachments: ManifestParser.patch, image-2022-05-13-14-16-39-509.png, > image-2022-05-13-14-17-55-965.png > > > In my heap dump I've seen a lot of duplicate Strings produced by the > ManifestParser. > It creates a lot of equal strings for the keys in the manifest but doesn't > deduplicate them and they're hold forever producing a lot on retained heap. > I've patched the ManifestParser to deduplicate just the keys and could save > ~8 Megs of heap. > Total usage before: 38MB after: 30MB > Duplicated Strings before: > !image-2022-05-13-14-16-39-509.png|width=658,height=203! > Duplicated Strings after: > !image-2022-05-13-14-17-55-965.png|width=793,height=200! > See patch attached. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (FELIX-6529) Improve memory usage ManifestParser using String deduplication
[
https://issues.apache.org/jira/browse/FELIX-6529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17539842#comment-17539842
]
Karl Pauls commented on FELIX-6529:
---
[~amitmondal]:
I think you can just add it to your settings, something like:
{noformat}
apache.snapshots
Apache Development Snapshot Repository
https://repository.apache.org/content/repositories/snapshots/
false
true
{noformat}
> Improve memory usage ManifestParser using String deduplication
> --
>
> Key: FELIX-6529
> URL: https://issues.apache.org/jira/browse/FELIX-6529
> Project: Felix
> Issue Type: Improvement
>Reporter: Johannes Edmeier
>Assignee: Karl Pauls
>Priority: Major
> Fix For: framework-7.0.4
>
> Attachments: ManifestParser.patch, image-2022-05-13-14-16-39-509.png,
> image-2022-05-13-14-17-55-965.png
>
>
> In my heap dump I've seen a lot of duplicate Strings produced by the
> ManifestParser.
> It creates a lot of equal strings for the keys in the manifest but doesn't
> deduplicate them and they're hold forever producing a lot on retained heap.
> I've patched the ManifestParser to deduplicate just the keys and could save
> ~8 Megs of heap.
> Total usage before: 38MB after: 30MB
> Duplicated Strings before:
> !image-2022-05-13-14-16-39-509.png|width=658,height=203!
> Duplicated Strings after:
> !image-2022-05-13-14-17-55-965.png|width=793,height=200!
> See patch attached.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Commented] (FELIX-6529) Improve memory usage ManifestParser using String deduplication
[ https://issues.apache.org/jira/browse/FELIX-6529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17539838#comment-17539838 ] Amit Mondal commented on FELIX-6529: [~karlpauls] I can surely test the snapshot version as well. I am just wondering where I can get the snapshot version from. I can only find the latest 7.0.3 release. > Improve memory usage ManifestParser using String deduplication > -- > > Key: FELIX-6529 > URL: https://issues.apache.org/jira/browse/FELIX-6529 > Project: Felix > Issue Type: Improvement >Reporter: Johannes Edmeier >Assignee: Karl Pauls >Priority: Major > Fix For: framework-7.0.4 > > Attachments: ManifestParser.patch, image-2022-05-13-14-16-39-509.png, > image-2022-05-13-14-17-55-965.png > > > In my heap dump I've seen a lot of duplicate Strings produced by the > ManifestParser. > It creates a lot of equal strings for the keys in the manifest but doesn't > deduplicate them and they're hold forever producing a lot on retained heap. > I've patched the ManifestParser to deduplicate just the keys and could save > ~8 Megs of heap. > Total usage before: 38MB after: 30MB > Duplicated Strings before: > !image-2022-05-13-14-16-39-509.png|width=658,height=203! > Duplicated Strings after: > !image-2022-05-13-14-17-55-965.png|width=793,height=200! > See patch attached. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (FELIX-6529) Improve memory usage ManifestParser using String deduplication
[ https://issues.apache.org/jira/browse/FELIX-6529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17539837#comment-17539837 ] Karl Pauls commented on FELIX-6529: --- [~amitmondal] - thanks, I deployed a new 7.1.0-SNAPSHOT for now so maybe you could test that one already. Otherwise, I'm planning to cut 7.0.4 soon as well. > Improve memory usage ManifestParser using String deduplication > -- > > Key: FELIX-6529 > URL: https://issues.apache.org/jira/browse/FELIX-6529 > Project: Felix > Issue Type: Improvement >Reporter: Johannes Edmeier >Assignee: Karl Pauls >Priority: Major > Fix For: framework-7.0.4 > > Attachments: ManifestParser.patch, image-2022-05-13-14-16-39-509.png, > image-2022-05-13-14-17-55-965.png > > > In my heap dump I've seen a lot of duplicate Strings produced by the > ManifestParser. > It creates a lot of equal strings for the keys in the manifest but doesn't > deduplicate them and they're hold forever producing a lot on retained heap. > I've patched the ManifestParser to deduplicate just the keys and could save > ~8 Megs of heap. > Total usage before: 38MB after: 30MB > Duplicated Strings before: > !image-2022-05-13-14-16-39-509.png|width=658,height=203! > Duplicated Strings after: > !image-2022-05-13-14-17-55-965.png|width=793,height=200! > See patch attached. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (FELIX-6529) Improve memory usage ManifestParser using String deduplication
[ https://issues.apache.org/jira/browse/FELIX-6529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17539828#comment-17539828 ] Amit Mondal commented on FELIX-6529: [~karlpauls] I just noticed your PR and it surely is very promising. Thanks a ton for fixing this issue with the duplicate strings. If possible, could you kindly also build a framework release? I would certainly give it a thorough test and let you know. > Improve memory usage ManifestParser using String deduplication > -- > > Key: FELIX-6529 > URL: https://issues.apache.org/jira/browse/FELIX-6529 > Project: Felix > Issue Type: Improvement >Reporter: Johannes Edmeier >Assignee: Karl Pauls >Priority: Major > Fix For: framework-7.0.4 > > Attachments: ManifestParser.patch, image-2022-05-13-14-16-39-509.png, > image-2022-05-13-14-17-55-965.png > > > In my heap dump I've seen a lot of duplicate Strings produced by the > ManifestParser. > It creates a lot of equal strings for the keys in the manifest but doesn't > deduplicate them and they're hold forever producing a lot on retained heap. > I've patched the ManifestParser to deduplicate just the keys and could save > ~8 Megs of heap. > Total usage before: 38MB after: 30MB > Duplicated Strings before: > !image-2022-05-13-14-16-39-509.png|width=658,height=203! > Duplicated Strings after: > !image-2022-05-13-14-17-55-965.png|width=793,height=200! > See patch attached. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (FELIX-6529) Improve memory usage ManifestParser using String deduplication
[ https://issues.apache.org/jira/browse/FELIX-6529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17538938#comment-17538938 ] Karl Pauls commented on FELIX-6529: --- [~amitmondal], could you try my PR and see if it helps in our case? https://github.com/apache/felix-dev/pull/151 > Improve memory usage ManifestParser using String deduplication > -- > > Key: FELIX-6529 > URL: https://issues.apache.org/jira/browse/FELIX-6529 > Project: Felix > Issue Type: Improvement >Reporter: Johannes Edmeier >Assignee: Karl Pauls >Priority: Major > Fix For: framework-7.0.4 > > Attachments: ManifestParser.patch, image-2022-05-13-14-16-39-509.png, > image-2022-05-13-14-17-55-965.png > > > In my heap dump I've seen a lot of duplicate Strings produced by the > ManifestParser. > It creates a lot of equal strings for the keys in the manifest but doesn't > deduplicate them and they're hold forever producing a lot on retained heap. > I've patched the ManifestParser to deduplicate just the keys and could save > ~8 Megs of heap. > Total usage before: 38MB after: 30MB > Duplicated Strings before: > !image-2022-05-13-14-16-39-509.png|width=658,height=203! > Duplicated Strings after: > !image-2022-05-13-14-17-55-965.png|width=793,height=200! > See patch attached. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (FELIX-6529) Improve memory usage ManifestParser using String deduplication
[ https://issues.apache.org/jira/browse/FELIX-6529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17538228#comment-17538228 ] Johannes Edmeier commented on FELIX-6529: - >From my knowledge the String.intern() can be more expensive than the map >lookup (see [https://stackoverflow.com/a/10628759/5834477)] The local map is much more smaller than the string pool. But either way I guess the parser isn't a very performance critical thing, right? > Improve memory usage ManifestParser using String deduplication > -- > > Key: FELIX-6529 > URL: https://issues.apache.org/jira/browse/FELIX-6529 > Project: Felix > Issue Type: Improvement >Reporter: Johannes Edmeier >Assignee: Karl Pauls >Priority: Major > Fix For: framework-7.0.4 > > Attachments: ManifestParser.patch, image-2022-05-13-14-16-39-509.png, > image-2022-05-13-14-17-55-965.png > > > In my heap dump I've seen a lot of duplicate Strings produced by the > ManifestParser. > It creates a lot of equal strings for the keys in the manifest but doesn't > deduplicate them and they're hold forever producing a lot on retained heap. > I've patched the ManifestParser to deduplicate just the keys and could save > ~8 Megs of heap. > Total usage before: 38MB after: 30MB > Duplicated Strings before: > !image-2022-05-13-14-16-39-509.png|width=658,height=203! > Duplicated Strings after: > !image-2022-05-13-14-17-55-965.png|width=793,height=200! > See patch attached. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (FELIX-6529) Improve memory usage ManifestParser using String deduplication
[ https://issues.apache.org/jira/browse/FELIX-6529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17538081#comment-17538081 ] Amit Mondal commented on FELIX-6529: String.intern() is not recommended for Java 6 for the fixed-size string pool. Since Felix FW uses Java 8 as its baseline, we can make use of StringTableSize which is by default set to ~6 (I don't remember the exact number) and will allow String.intern() to run in constant time as it internally uses a map too and hence, String.intern() would surely be effective in contrast to custom maps to store the strings. > Improve memory usage ManifestParser using String deduplication > -- > > Key: FELIX-6529 > URL: https://issues.apache.org/jira/browse/FELIX-6529 > Project: Felix > Issue Type: Improvement >Reporter: Johannes Edmeier >Assignee: Karl Pauls >Priority: Major > Fix For: framework-7.0.4 > > Attachments: ManifestParser.patch, image-2022-05-13-14-16-39-509.png, > image-2022-05-13-14-17-55-965.png > > > In my heap dump I've seen a lot of duplicate Strings produced by the > ManifestParser. > It creates a lot of equal strings for the keys in the manifest but doesn't > deduplicate them and they're hold forever producing a lot on retained heap. > I've patched the ManifestParser to deduplicate just the keys and could save > ~8 Megs of heap. > Total usage before: 38MB after: 30MB > Duplicated Strings before: > !image-2022-05-13-14-16-39-509.png|width=658,height=203! > Duplicated Strings after: > !image-2022-05-13-14-17-55-965.png|width=793,height=200! > See patch attached. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (FELIX-6529) Improve memory usage ManifestParser using String deduplication
[ https://issues.apache.org/jira/browse/FELIX-6529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17538039#comment-17538039 ] Karl Pauls commented on FELIX-6529: --- Right, I had the same question as [~gnodet], shouldn't we just use String.intern() instead - that would potentially even help in cases where there is more than one framework at the same time. > Improve memory usage ManifestParser using String deduplication > -- > > Key: FELIX-6529 > URL: https://issues.apache.org/jira/browse/FELIX-6529 > Project: Felix > Issue Type: Improvement >Reporter: Johannes Edmeier >Assignee: Karl Pauls >Priority: Major > Fix For: framework-7.0.4 > > Attachments: ManifestParser.patch, image-2022-05-13-14-16-39-509.png, > image-2022-05-13-14-17-55-965.png > > > In my heap dump I've seen a lot of duplicate Strings produced by the > ManifestParser. > It creates a lot of equal strings for the keys in the manifest but doesn't > deduplicate them and they're hold forever producing a lot on retained heap. > I've patched the ManifestParser to deduplicate just the keys and could save > ~8 Megs of heap. > Total usage before: 38MB after: 30MB > Duplicated Strings before: > !image-2022-05-13-14-16-39-509.png|width=658,height=203! > Duplicated Strings after: > !image-2022-05-13-14-17-55-965.png|width=793,height=200! > See patch attached. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (FELIX-6529) Improve memory usage ManifestParser using String deduplication
[ https://issues.apache.org/jira/browse/FELIX-6529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17538036#comment-17538036 ] Guillaume Nodet commented on FELIX-6529: Wouldn't it be easier to just use `String.intern()` instead ? > Improve memory usage ManifestParser using String deduplication > -- > > Key: FELIX-6529 > URL: https://issues.apache.org/jira/browse/FELIX-6529 > Project: Felix > Issue Type: Improvement >Reporter: Johannes Edmeier >Assignee: Karl Pauls >Priority: Major > Fix For: framework-7.0.4 > > Attachments: ManifestParser.patch, image-2022-05-13-14-16-39-509.png, > image-2022-05-13-14-17-55-965.png > > > In my heap dump I've seen a lot of duplicate Strings produced by the > ManifestParser. > It creates a lot of equal strings for the keys in the manifest but doesn't > deduplicate them and they're hold forever producing a lot on retained heap. > I've patched the ManifestParser to deduplicate just the keys and could save > ~8 Megs of heap. > Total usage before: 38MB after: 30MB > Duplicated Strings before: > !image-2022-05-13-14-16-39-509.png|width=658,height=203! > Duplicated Strings after: > !image-2022-05-13-14-17-55-965.png|width=793,height=200! > See patch attached. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (FELIX-6529) Improve memory usage ManifestParser using String deduplication
[ https://issues.apache.org/jira/browse/FELIX-6529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17538033#comment-17538033 ] Amit Mondal commented on FELIX-6529: This is a huge improvement. In cases where lots of bundles are installed (in our case 200+), more than 25 MB of heap can be spared. > Improve memory usage ManifestParser using String deduplication > -- > > Key: FELIX-6529 > URL: https://issues.apache.org/jira/browse/FELIX-6529 > Project: Felix > Issue Type: Improvement >Reporter: Johannes Edmeier >Assignee: Karl Pauls >Priority: Major > Fix For: framework-7.0.4 > > Attachments: ManifestParser.patch, image-2022-05-13-14-16-39-509.png, > image-2022-05-13-14-17-55-965.png > > > In my heap dump I've seen a lot of duplicate Strings produced by the > ManifestParser. > It creates a lot of equal strings for the keys in the manifest but doesn't > deduplicate them and they're hold forever producing a lot on retained heap. > I've patched the ManifestParser to deduplicate just the keys and could save > ~8 Megs of heap. > Total usage before: 38MB after: 30MB > Duplicated Strings before: > !image-2022-05-13-14-16-39-509.png|width=658,height=203! > Duplicated Strings after: > !image-2022-05-13-14-17-55-965.png|width=793,height=200! > See patch attached. -- This message was sent by Atlassian Jira (v8.20.7#820007)
