[jira] [Comment Edited] (KARAF-5314) The performance of profile builder used by karaf maven plugin has reduced significantly in 4.1 compared to 4.0
[ https://issues.apache.org/jira/browse/KARAF-5314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16135240#comment-16135240 ] Vinay Shankar edited comment on KARAF-5314 at 8/21/17 2:43 PM: --- [~ch...@die-schneider.net] - My original email to the mailing list. Was not sent on Friday for some reason. "We have a highly complex and huge feature dependency tree due to the number of OSGI bundles (components). When we were using 4.0.9, the plugin would take aroung 5min to build the distribution. We upgraded to karaf 4.1.1 and the build time has increased significantly. Now that we refactored our feature dependency tree a bit and added more feature files, the build time takes around 13min (7min with 4.0.9). I decided to debug and I noticed a performance bottleneck in the Builder class of the profile module. I have created a ticket for the same https://issues.apache.org/jira/browse/KARAF-5314. Looking in the code, there has been a change in 4.1.1 to use Java 8 Streams API to filter the required features from the Set of all available features. A “for” loop was being used in 4.0.9. I also noticed that there was no cache being used. Each time a required feature was found in the available set of features, it was not being cached (to avoid checking the set of available features again if the same feature was found later in the dependency tree). I have implemented the caching and I tested the code against our feature dependency tree by pointing to the SNAPSHOT version of the plugin I built locally. The build only takes 40 seconds to complete now. It is a huge improvement from 13min. Considering that we are a big team and each of us build locally and we also build on Bamboo servers for our CI and deployment on multiple environments, this will be a real time saver for us. I have pushed the fix to my forked repository and created a pull request for the same https://github.com/apache/karaf/pull/337. I have also attached the patch file here. Please take a look and review. I was also wondering (if the review is successful and fix merged to master), when the next release would be? We would like to have the fix released in 4.1.3." was (Author: vinayshankar): [~ch...@die-schneider.net] - My original email to the mailing list. Was not sent on Friday for some reason. "_We have a highly complex and huge feature dependency tree due to the number of OSGI bundles (components). When we were using 4.0.9, the plugin would take aroung 5min to build the distribution. We upgraded to karaf 4.1.1 and the build time has increased significantly. Now that we refactored our feature dependency tree a bit and added more feature files, the build time takes around 13min (7min with 4.0.9). I decided to debug and I noticed a performance bottleneck in the Builder class of the profile module. I have created a ticket for the same https://issues.apache.org/jira/browse/KARAF-5314. Looking in the code, there has been a change in 4.1.1 to use Java 8 Streams API to filter the required features from the Set of all available features. A “for” loop was being used in 4.0.9. I also noticed that there was no cache being used. Each time a required feature was found in the available set of features, it was not being cached (to avoid checking the set of available features again if the same feature was found later in the dependency tree). I have implemented the caching and I tested the code against our feature dependency tree by pointing to the SNAPSHOT version of the plugin I built locally. The build only takes 40 seconds to complete now. It is a huge improvement from 13min. Considering that we are a big team and each of us build locally and we also build on Bamboo servers for our CI and deployment on multiple environments, this will be a real time saver for us. I have pushed the fix to my forked repository and created a pull request for the same https://github.com/apache/karaf/pull/337. I have also attached the patch file here. Please take a look and review. I was also wondering (if the review is successful and fix merged to master), when the next release would be? We would like to have the fix released in 4.1.3_" > The performance of profile builder used by karaf maven plugin has reduced > significantly in 4.1 compared to 4.0 > -- > > Key: KARAF-5314 > URL: https://issues.apache.org/jira/browse/KARAF-5314 > Project: Karaf > Issue Type: Bug >Reporter: Vinay Shankar > Fix For: 4.2.0 > > > The performance of profile builder used by karaf maven plugin has reduced > significantly in 4.1 compared to 4.0. > The java streams API is being used in 4.1 to filter our the required features > from the set of all features present in the repositories that are part
[jira] [Comment Edited] (KARAF-5314) The performance of profile builder used by karaf maven plugin has reduced significantly in 4.1 compared to 4.0
[ https://issues.apache.org/jira/browse/KARAF-5314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16135240#comment-16135240 ] Vinay Shankar edited comment on KARAF-5314 at 8/21/17 2:43 PM: --- [~ch...@die-schneider.net] - My original email to the mailing list. Was not sent on Friday for some reason. _We have a highly complex and huge feature dependency tree due to the number of OSGI bundles (components). When we were using 4.0.9, the plugin would take aroung 5min to build the distribution. We upgraded to karaf 4.1.1 and the build time has increased significantly. Now that we refactored our feature dependency tree a bit and added more feature files, the build time takes around 13min (7min with 4.0.9). I decided to debug and I noticed a performance bottleneck in the Builder class of the profile module. I have created a ticket for the same https://issues.apache.org/jira/browse/KARAF-5314. Looking in the code, there has been a change in 4.1.1 to use Java 8 Streams API to filter the required features from the Set of all available features. A “for” loop was being used in 4.0.9. I also noticed that there was no cache being used. Each time a required feature was found in the available set of features, it was not being cached (to avoid checking the set of available features again if the same feature was found later in the dependency tree). I have implemented the caching and I tested the code against our feature dependency tree by pointing to the SNAPSHOT version of the plugin I built locally. The build only takes 40 seconds to complete now. It is a huge improvement from 13min. Considering that we are a big team and each of us build locally and we also build on Bamboo servers for our CI and deployment on multiple environments, this will be a real time saver for us. I have pushed the fix to my forked repository and created a pull request for the same https://github.com/apache/karaf/pull/337. I have also attached the patch file here. Please take a look and review. I was also wondering (if the review is successful and fix merged to master), when the next release would be? We would like to have the fix released in 4.1.3_ was (Author: vinayshankar): [~ch...@die-schneider.net] - My original email to the mailing list. Was not sent on Friday for some reason. _We have a highly complex and huge feature dependency tree due to the number of OSGI bundles (components). When we were using 4.0.9, the plugin would take aroung 5min to build the distribution. We upgraded to karaf 4.1.1 and the build time has increased significantly. Now that we refactored our feature dependency tree a bit and added more feature files, the build time takes around 13min (7min with 4.0.9). I decided to debug and I noticed a performance bottleneck in the Builder class of the profile module. I have created a ticket for the same https://issues.apache.org/jira/browse/KARAF-5314. Looking in the code, there has been a change in 4.1.1 to use Java 8 Streams API to filter the required features from the Set of all available features. A “for” loop was being used in 4.0.9. I also noticed that there was no cache being used. Each time a required feature was found in the available set of features, it was not being cached (to avoid checking the set of available features again if the same feature was found later in the dependency tree). I have implemented the caching and I tested the code against our feature dependency tree by pointing to the SNAPSHOT version of the plugin I built locally. The build only takes 40 seconds to complete now. It is a huge improvement from 13min. Considering that we are a big team and each of us build locally and we also build on Bamboo servers for our CI and deployment on multiple environments, this will be a real time saver for us. I have pushed the fix to my forked repository and created a pull request for the same https://github.com/apache/karaf/pull/337. I have also attached the patch file here. Please take a look and review. I was also wondering (if the review is successful and fix merged to master), when the next release would be? We would like to have the fix released in 4.1.3._ > The performance of profile builder used by karaf maven plugin has reduced > significantly in 4.1 compared to 4.0 > -- > > Key: KARAF-5314 > URL: https://issues.apache.org/jira/browse/KARAF-5314 > Project: Karaf > Issue Type: Bug >Reporter: Vinay Shankar > Fix For: 4.2.0 > > > The performance of profile builder used by karaf maven plugin has reduced > significantly in 4.1 compared to 4.0. > The java streams API is being used in 4.1 to filter our the required features > from the set of all features present in the repositories that are part
[jira] [Comment Edited] (KARAF-5314) The performance of profile builder used by karaf maven plugin has reduced significantly in 4.1 compared to 4.0
[ https://issues.apache.org/jira/browse/KARAF-5314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16135240#comment-16135240 ] Vinay Shankar edited comment on KARAF-5314 at 8/21/17 2:43 PM: --- [~ch...@die-schneider.net] - My original email to the mailing list. Was not sent on Friday for some reason. "_We have a highly complex and huge feature dependency tree due to the number of OSGI bundles (components). When we were using 4.0.9, the plugin would take aroung 5min to build the distribution. We upgraded to karaf 4.1.1 and the build time has increased significantly. Now that we refactored our feature dependency tree a bit and added more feature files, the build time takes around 13min (7min with 4.0.9). I decided to debug and I noticed a performance bottleneck in the Builder class of the profile module. I have created a ticket for the same https://issues.apache.org/jira/browse/KARAF-5314. Looking in the code, there has been a change in 4.1.1 to use Java 8 Streams API to filter the required features from the Set of all available features. A “for” loop was being used in 4.0.9. I also noticed that there was no cache being used. Each time a required feature was found in the available set of features, it was not being cached (to avoid checking the set of available features again if the same feature was found later in the dependency tree). I have implemented the caching and I tested the code against our feature dependency tree by pointing to the SNAPSHOT version of the plugin I built locally. The build only takes 40 seconds to complete now. It is a huge improvement from 13min. Considering that we are a big team and each of us build locally and we also build on Bamboo servers for our CI and deployment on multiple environments, this will be a real time saver for us. I have pushed the fix to my forked repository and created a pull request for the same https://github.com/apache/karaf/pull/337. I have also attached the patch file here. Please take a look and review. I was also wondering (if the review is successful and fix merged to master), when the next release would be? We would like to have the fix released in 4.1.3_" was (Author: vinayshankar): [~ch...@die-schneider.net] - My original email to the mailing list. Was not sent on Friday for some reason. _We have a highly complex and huge feature dependency tree due to the number of OSGI bundles (components). When we were using 4.0.9, the plugin would take aroung 5min to build the distribution. We upgraded to karaf 4.1.1 and the build time has increased significantly. Now that we refactored our feature dependency tree a bit and added more feature files, the build time takes around 13min (7min with 4.0.9). I decided to debug and I noticed a performance bottleneck in the Builder class of the profile module. I have created a ticket for the same https://issues.apache.org/jira/browse/KARAF-5314. Looking in the code, there has been a change in 4.1.1 to use Java 8 Streams API to filter the required features from the Set of all available features. A “for” loop was being used in 4.0.9. I also noticed that there was no cache being used. Each time a required feature was found in the available set of features, it was not being cached (to avoid checking the set of available features again if the same feature was found later in the dependency tree). I have implemented the caching and I tested the code against our feature dependency tree by pointing to the SNAPSHOT version of the plugin I built locally. The build only takes 40 seconds to complete now. It is a huge improvement from 13min. Considering that we are a big team and each of us build locally and we also build on Bamboo servers for our CI and deployment on multiple environments, this will be a real time saver for us. I have pushed the fix to my forked repository and created a pull request for the same https://github.com/apache/karaf/pull/337. I have also attached the patch file here. Please take a look and review. I was also wondering (if the review is successful and fix merged to master), when the next release would be? We would like to have the fix released in 4.1.3_ > The performance of profile builder used by karaf maven plugin has reduced > significantly in 4.1 compared to 4.0 > -- > > Key: KARAF-5314 > URL: https://issues.apache.org/jira/browse/KARAF-5314 > Project: Karaf > Issue Type: Bug >Reporter: Vinay Shankar > Fix For: 4.2.0 > > > The performance of profile builder used by karaf maven plugin has reduced > significantly in 4.1 compared to 4.0. > The java streams API is being used in 4.1 to filter our the required features > from the set of all features present in the repositories that are part
[jira] [Comment Edited] (KARAF-5314) The performance of profile builder used by karaf maven plugin has reduced significantly in 4.1 compared to 4.0
[ https://issues.apache.org/jira/browse/KARAF-5314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16135240#comment-16135240 ] Vinay Shankar edited comment on KARAF-5314 at 8/21/17 2:42 PM: --- [~ch...@die-schneider.net] - My original email to the mailing list. Was not sent on Friday for some reason. _ "We have a highly complex and huge feature dependency tree due to the number of OSGI bundles (components). When we were using 4.0.9, the plugin would take aroung 5min to build the distribution. We upgraded to karaf 4.1.1 and the build time has increased significantly. Now that we refactored our feature dependency tree a bit and added more feature files, the build time takes around 13min (7min with 4.0.9). I decided to debug and I noticed a performance bottleneck in the Builder class of the profile module. I have created a ticket for the same https://issues.apache.org/jira/browse/KARAF-5314. Looking in the code, there has been a change in 4.1.1 to use Java 8 Streams API to filter the required features from the Set of all available features. A “for” loop was being used in 4.0.9. I also noticed that there was no cache being used. Each time a required feature was found in the available set of features, it was not being cached (to avoid checking the set of available features again if the same feature was found later in the dependency tree). I have implemented the caching and I tested the code against our feature dependency tree by pointing to the SNAPSHOT version of the plugin I built locally. The build only takes 40 seconds to complete now. It is a huge improvement from 13min. Considering that we are a big team and each of us build locally and we also build on Bamboo servers for our CI and deployment on multiple environments, this will be a real time saver for us. I have pushed the fix to my forked repository and created a pull request for the same https://github.com/apache/karaf/pull/337. I have also attached the patch file here. Please take a look and review. I was also wondering (if the review is successful and fix merged to master), when the next release would be? We would like to have the fix released in 4.1.3." _ was (Author: vinayshankar): [~ch...@die-schneider.net] - My original email to the mailing list. Was not sent on Friday for some reason. _"We have a highly complex and huge feature dependency tree due to the number of OSGI bundles (components). When we were using 4.0.9, the plugin would take aroung 5min to build the distribution. We upgraded to karaf 4.1.1 and the build time has increased significantly. Now that we refactored our feature dependency tree a bit and added more feature files, the build time takes around 13min (7min with 4.0.9). I decided to debug and I noticed a performance bottleneck in the Builder class of the profile module. I have created a ticket for the same https://issues.apache.org/jira/browse/KARAF-5314. Looking in the code, there has been a change in 4.1.1 to use Java 8 Streams API to filter the required features from the Set of all available features. A “for” loop was being used in 4.0.9. I also noticed that there was no cache being used. Each time a required feature was found in the available set of features, it was not being cached (to avoid checking the set of available features again if the same feature was found later in the dependency tree). I have implemented the caching and I tested the code against our feature dependency tree by pointing to the SNAPSHOT version of the plugin I built locally. The build only takes 40 seconds to complete now. It is a huge improvement from 13min. Considering that we are a big team and each of us build locally and we also build on Bamboo servers for our CI and deployment on multiple environments, this will be a real time saver for us. I have pushed the fix to my forked repository and created a pull request for the same https://github.com/apache/karaf/pull/337. I have also attached the patch file here. Please take a look and review. I was also wondering (if the review is successful and fix merged to master), when the next release would be? We would like to have the fix released in 4.1.3."_ > The performance of profile builder used by karaf maven plugin has reduced > significantly in 4.1 compared to 4.0 > -- > > Key: KARAF-5314 > URL: https://issues.apache.org/jira/browse/KARAF-5314 > Project: Karaf > Issue Type: Bug >Reporter: Vinay Shankar > Fix For: 4.2.0 > > > The performance of profile builder used by karaf maven plugin has reduced > significantly in 4.1 compared to 4.0. > The java streams API is being used in 4.1 to filter our the required features > from the set of all features present in the repositories that are
[jira] [Comment Edited] (KARAF-5314) The performance of profile builder used by karaf maven plugin has reduced significantly in 4.1 compared to 4.0
[ https://issues.apache.org/jira/browse/KARAF-5314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16135240#comment-16135240 ] Vinay Shankar edited comment on KARAF-5314 at 8/21/17 2:42 PM: --- [~ch...@die-schneider.net] - My original email to the mailing list. Was not sent on Friday for some reason. _We have a highly complex and huge feature dependency tree due to the number of OSGI bundles (components). When we were using 4.0.9, the plugin would take aroung 5min to build the distribution. We upgraded to karaf 4.1.1 and the build time has increased significantly. Now that we refactored our feature dependency tree a bit and added more feature files, the build time takes around 13min (7min with 4.0.9). I decided to debug and I noticed a performance bottleneck in the Builder class of the profile module. I have created a ticket for the same https://issues.apache.org/jira/browse/KARAF-5314. Looking in the code, there has been a change in 4.1.1 to use Java 8 Streams API to filter the required features from the Set of all available features. A “for” loop was being used in 4.0.9. I also noticed that there was no cache being used. Each time a required feature was found in the available set of features, it was not being cached (to avoid checking the set of available features again if the same feature was found later in the dependency tree). I have implemented the caching and I tested the code against our feature dependency tree by pointing to the SNAPSHOT version of the plugin I built locally. The build only takes 40 seconds to complete now. It is a huge improvement from 13min. Considering that we are a big team and each of us build locally and we also build on Bamboo servers for our CI and deployment on multiple environments, this will be a real time saver for us. I have pushed the fix to my forked repository and created a pull request for the same https://github.com/apache/karaf/pull/337. I have also attached the patch file here. Please take a look and review. I was also wondering (if the review is successful and fix merged to master), when the next release would be? We would like to have the fix released in 4.1.3._ was (Author: vinayshankar): [~ch...@die-schneider.net] - My original email to the mailing list. Was not sent on Friday for some reason. _ "We have a highly complex and huge feature dependency tree due to the number of OSGI bundles (components). When we were using 4.0.9, the plugin would take aroung 5min to build the distribution. We upgraded to karaf 4.1.1 and the build time has increased significantly. Now that we refactored our feature dependency tree a bit and added more feature files, the build time takes around 13min (7min with 4.0.9). I decided to debug and I noticed a performance bottleneck in the Builder class of the profile module. I have created a ticket for the same https://issues.apache.org/jira/browse/KARAF-5314. Looking in the code, there has been a change in 4.1.1 to use Java 8 Streams API to filter the required features from the Set of all available features. A “for” loop was being used in 4.0.9. I also noticed that there was no cache being used. Each time a required feature was found in the available set of features, it was not being cached (to avoid checking the set of available features again if the same feature was found later in the dependency tree). I have implemented the caching and I tested the code against our feature dependency tree by pointing to the SNAPSHOT version of the plugin I built locally. The build only takes 40 seconds to complete now. It is a huge improvement from 13min. Considering that we are a big team and each of us build locally and we also build on Bamboo servers for our CI and deployment on multiple environments, this will be a real time saver for us. I have pushed the fix to my forked repository and created a pull request for the same https://github.com/apache/karaf/pull/337. I have also attached the patch file here. Please take a look and review. I was also wondering (if the review is successful and fix merged to master), when the next release would be? We would like to have the fix released in 4.1.3." _ > The performance of profile builder used by karaf maven plugin has reduced > significantly in 4.1 compared to 4.0 > -- > > Key: KARAF-5314 > URL: https://issues.apache.org/jira/browse/KARAF-5314 > Project: Karaf > Issue Type: Bug >Reporter: Vinay Shankar > Fix For: 4.2.0 > > > The performance of profile builder used by karaf maven plugin has reduced > significantly in 4.1 compared to 4.0. > The java streams API is being used in 4.1 to filter our the required features > from the set of all features present in the repositories that are
[jira] [Comment Edited] (KARAF-5314) The performance of profile builder used by karaf maven plugin has reduced significantly in 4.1 compared to 4.0
[ https://issues.apache.org/jira/browse/KARAF-5314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16134003#comment-16134003 ] Christian Schneider edited comment on KARAF-5314 at 8/19/17 7:41 AM: - I am not sure if the streams API is the root of the problem. Instead I think the problem is that we repeatedly go through the features to find a match for a given feature/version. So a cache is a good idea to speed this up. Currently there are two cases for a version. The version can be open if only the name is given or the version is 0.0.0. In this case all versions would match. The other case is that a version is given. Then only this version would match. The current code filters using a range which adds all matching features. The code in the PR uses a cache that holds only one feature. So I am not sure this will always do the same thing. was (Author: ch...@die-schneider.net): I am not sure if the streams API is the root of the problem. Instead I think the problem is that we repeatedly go through the features to find a match for a given feature/version. So a cache is a good idea to speed this up. Currently there are two cases for a version. The version can be open if only the name is given or the version is 0.0.0. In this case all versions would match. The other case is that a version is given. Then only this version would match. The current code filters using a range which would return add all matching features. The code in the PR uses a cache that holds only one feature. So I am not sure this will always do the same thing. > The performance of profile builder used by karaf maven plugin has reduced > significantly in 4.1 compared to 4.0 > -- > > Key: KARAF-5314 > URL: https://issues.apache.org/jira/browse/KARAF-5314 > Project: Karaf > Issue Type: Bug >Reporter: Vinay Shankar > Fix For: 4.1.3 > > > The performance of profile builder used by karaf maven plugin has reduced > significantly in 4.1 compared to 4.0. > The java streams API is being used in 4.1 to filter our the required features > from the set of all features present in the repositories that are part of the > profile. This is done in the "addFeatures" method in the "Builder.java" class > in the "org.apache.karaf.profile.core" bundle. This change (from 4.0) has > drastically reduced the performance. For a profile with ~900 features in all > the repositories in the profile and ~300 required features from a highly > complex feature dependency tree, this function is taking around 13min to > complete. The same execution took around 3-5min in 4.0 (where simple for > loops were being used). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
