Re: [rules-users] Migrating from 4.0.7 to 5.4.0.Final
Hi again, I just tried the updated 5.4.x branch with 50% perm gen threshold and all my tests are now working as expected. As a side note, I've also ran a memory analysis of the tests and I can note that there are definitely less class loaders being created. I see that CompositeClassLoader still retain classes (hard to tell why though), but since I have the magical option to control consumption of PermGen, I consider that everything is okay for me Again many thanks for your help, hoping to see 5.4.1 out soon :-) -- View this message in context: http://drools.46999.n3.nabble.com/Migrating-from-4-0-7-to-5-4-0-Final-tp4018215p4018543.html Sent from the Drools: User forum mailing list archive at Nabble.com. ___ rules-users mailing list rules-users@lists.jboss.org https://lists.jboss.org/mailman/listinfo/rules-users
Re: [rules-users] Migrating from 4.0.7 to 5.4.0.Final
I am out of office until Tuesday, 17th of July. If you have any urgent queries, you can contact Robert Doherty at robert.dohe...@nathean.com 01 6853001 ___ rules-users mailing list rules-users@lists.jboss.org https://lists.jboss.org/mailman/listinfo/rules-users
Re: [rules-users] Migrating from 4.0.7 to 5.4.0.Final
I am out of office until Tuesday, 17th of July. If you have any urgent queries, you can contact Robert Doherty at robert.dohe...@nathean.com 01 6853001 ___ rules-users mailing list rules-users@lists.jboss.org https://lists.jboss.org/mailman/listinfo/rules-users
Re: [rules-users] Migrating from 4.0.7 to 5.4.0.Final
Hi again, As you suggested I modified the thread that jits the constraint, so now it shouldn't hold any reference after he finishes its job and then avoid any ClassLoader leaking. I also added another option, named drools.permgenThreshold that allows to define a threshold that is the percentage of usage of the PermGen space above which the engine stops JITting other constraints and let them run in interpreted mode. By default this value is set to 90% but of course you could set it to 0 if you want to avoid any JITting. Be aware that if you want to give it a try you will need to update and recompile also the droolsjbpm-knowledge project ( https://github.com/droolsjbpm/droolsjbpm-knowledge/tree/5.4.x ) since the class defining this now option is there. If you try it let me know how it works for you. Thanks for your precious suggestions and help, Mario -- View this message in context: http://drools.46999.n3.nabble.com/Migrating-from-4-0-7-to-5-4-0-Final-tp4018215p4018494.html Sent from the Drools: User forum mailing list archive at Nabble.com. ___ rules-users mailing list rules-users@lists.jboss.org https://lists.jboss.org/mailman/listinfo/rules-users
Re: [rules-users] Migrating from 4.0.7 to 5.4.0.Final
Mario Fusco wrote > > Yes, of course it will be possible, even if my suggestion is to avoid this > in production otherwise you will have ALL the constraints running in > interpreted mode. Actually you could already achieve this if you can > rebuild drools from the sources (you already did it, so I assume it won't > be a problem). If you want to try open the class MvelConstraint and modify > the constant JIT_THRESOLD. That constant defines after how many times a > constraint can be evaluated in interpreted mode before to get JITted, so > if you set it to a very high value (e.g. Integer.MAX_VALUE) you should be > able to completely avoid any JITting. Let me know if this works. > > I hope this helps, > Mario > I just tested it with JIT_THRESOLD=Integer.MAX_VALUE and it effectively does wonders for both perm gen consumption (as classes no longer held by the executor thread) and - believe it or not - even performances in some cases :-) It would be great to have this configurable (either this or your other option which will limit the amount of memory used by Jitting) -- View this message in context: http://drools.46999.n3.nabble.com/Migrating-from-4-0-7-to-5-4-0-Final-tp4018215p4018481.html Sent from the Drools: User forum mailing list archive at Nabble.com. ___ rules-users mailing list rules-users@lists.jboss.org https://lists.jboss.org/mailman/listinfo/rules-users
Re: [rules-users] Migrating from 4.0.7 to 5.4.0.Final
JP Chemali wrote > > Yes in my case it would be nice for generated classes to be garbage > collected when no longer referenced : for long-lived virtual machines > where rules are changed a lot, you really hit the max perm gen quickly > enough. > I guess what you are describing is a production environment where rules > are loaded/compiled once in the VM's life (startup time most probably) and > that's it. That's not my case, it's rather the reverse in some cases, as > specialized users can test their rule sets in the same environment as the > production one and choose to release them. > Ok, now I see your point. I will try to refactor that bit to avoid that the thread holds a reference to the classes it generates. JP Chemali wrote > > Your option to control JITing space looks like a good way to keep things > in check. Would it possible to ensure if we set the space to 0, it > actually disables it? That would ensure that everything runs in my calling > threads and gets garbage collected right? > Yes, of course it will be possible, even if my suggestion is to avoid this in production otherwise you will have ALL the constraints running in interpreted mode. Actually you could already achieve this if you can rebuild drools from the sources (you already did it, so I assume it won't be a problem). If you want to try open the class MvelConstraint and modify the constant JIT_THRESOLD. That constant defines after how many times a constraint can be evaluated in interpreted mode before to get JITted, so if you set it to a very high value (e.g. Integer.MAX_VALUE) you should be able to completely avoid any JITting. Let me know if this works. I hope this helps, Mario -- View this message in context: http://drools.46999.n3.nabble.com/Migrating-from-4-0-7-to-5-4-0-Final-tp4018215p4018469.html Sent from the Drools: User forum mailing list archive at Nabble.com. ___ rules-users mailing list rules-users@lists.jboss.org https://lists.jboss.org/mailman/listinfo/rules-users
Re: [rules-users] Migrating from 4.0.7 to 5.4.0.Final
Hi again mario, Yes in my case it would be nice for generated classes to be garbage collected when no longer referenced : for long-lived virtual machines where rules are changed a lot, you really hit the max perm gen quickly enough. I guess what you are describing is a production environment where rules are loaded/compiled once in the VM's life (startup time most probably) and that's it. That's not my case, it's rather the reverse in some cases, as specialized users can test their rule sets in the same environment as the production one and choose to release them. Your option to control JITing space looks like a good way to keep things in check. Would it possible to ensure if we set the space to 0, it actually disables it? That would ensure that everything runs in my calling threads and gets garbage collected right? Again thanks a mil for your help > Date: Wed, 4 Jul 2012 06:31:00 -0700 > From: mario.fu...@gmail.com > To: rules-users@lists.jboss.org > Subject: Re: [rules-users] Migrating from 4.0.7 to 5.4.0.Final > > Hi again, > > You are looking at the correct branch and I didn't change the MvelConstraint > class indeed. What I did is to change the ClassGenerator to try to avoid the > creation of new ClassLoaders (when possible). > > Said that what I don't understand is why you expect the generated classes to > be garbage collected. They are used to evaluate the constraints in your > rules, so I don't think they could/should be CG'd anyway. Am I missing > something? > > Anyway we are realizing that the required PermGen space could be a problem > especially if you have a big rule base, so I am going to add a configuration > option that will allow you to define how many space you want to give to > these JITted constraints. Once the engine will hit this limit it will just > stop JITting further constraints leaving them run in interpreted mode. > Hopefully the users will be able to find a good trade-off between > performances and memory occupation in this way. Does this make sense to you? > > Mario > > -- > View this message in context: > http://drools.46999.n3.nabble.com/Migrating-from-4-0-7-to-5-4-0-Final-tp4018215p4018465.html > Sent from the Drools: User forum mailing list archive at Nabble.com. > ___ > rules-users mailing list > rules-users@lists.jboss.org > https://lists.jboss.org/mailman/listinfo/rules-users ___ rules-users mailing list rules-users@lists.jboss.org https://lists.jboss.org/mailman/listinfo/rules-users
Re: [rules-users] Migrating from 4.0.7 to 5.4.0.Final
Hi again, You are looking at the correct branch and I didn't change the MvelConstraint class indeed. What I did is to change the ClassGenerator to try to avoid the creation of new ClassLoaders (when possible). Said that what I don't understand is why you expect the generated classes to be garbage collected. They are used to evaluate the constraints in your rules, so I don't think they could/should be CG'd anyway. Am I missing something? Anyway we are realizing that the required PermGen space could be a problem especially if you have a big rule base, so I am going to add a configuration option that will allow you to define how many space you want to give to these JITted constraints. Once the engine will hit this limit it will just stop JITting further constraints leaving them run in interpreted mode. Hopefully the users will be able to find a good trade-off between performances and memory occupation in this way. Does this make sense to you? Mario -- View this message in context: http://drools.46999.n3.nabble.com/Migrating-from-4-0-7-to-5-4-0-Final-tp4018215p4018465.html Sent from the Drools: User forum mailing list archive at Nabble.com. ___ rules-users mailing list rules-users@lists.jboss.org https://lists.jboss.org/mailman/listinfo/rules-users
Re: [rules-users] Migrating from 4.0.7 to 5.4.0.Final
Hi Mario, Just tested it and unfortunately found the same problems, looking at your commit in (https://issues.jboss.org/browse/JBRULES-3567) there seem to be no real code changes in class MvelConstraint, the major leak being the ExecutorService holding on to the threads and thus all the generated classes. Have I missed something? Maybe I didn't check out the correct branch (https://github.com/droolsjbpm/drools/tree/5.4.x) ? Thanks for any input -- View this message in context: http://drools.46999.n3.nabble.com/Migrating-from-4-0-7-to-5-4-0-Final-tp4018215p4018463.html Sent from the Drools: User forum mailing list archive at Nabble.com. ___ rules-users mailing list rules-users@lists.jboss.org https://lists.jboss.org/mailman/listinfo/rules-users
Re: [rules-users] Migrating from 4.0.7 to 5.4.0.Final
Mario Fusco wrote > > I just fixed the leaks you reported on the master repository and > backported the fix to the 5.4.x branch. I also closed the JIRA ticket I > opened a few hours ago, but it is still possible to comment on it or even > reopen it if you will find further issues. I don't know if you have the > possibility to build Drools from the sources but if so it could be great > if you could give a try to this fix. > > Nevertheless it is still a fact that Drools 5.4 requires more PermGen > space, for the reasons that both Mark and myself explained in our former > emails. > > I hope this helps, > Mario > Thanks, that was quick! Yes I am aware if the PermGen consumption, but I'm hoping to tone it down by looking at the rules themselves (in 4.0.7 I had managed to reduce it by simply shortening the rule names, let's see) I'll try to access the drools sources and let you know if your patch fixes the problem, again thanks that was real quick -- View this message in context: http://drools.46999.n3.nabble.com/Migrating-from-4-0-7-to-5-4-0-Final-tp4018215p4018454.html Sent from the Drools: User forum mailing list archive at Nabble.com. ___ rules-users mailing list rules-users@lists.jboss.org https://lists.jboss.org/mailman/listinfo/rules-users
Re: [rules-users] Migrating from 4.0.7 to 5.4.0.Final
laune wrote
>
> It seems that the problems reported by the OP are mostly (if not all)
> related to building KnowledgePackages from DRL files. Drools has been
> through several 5.x versions with (among others) many extensions to
> the DRL language, so some increase in resource consumption is to be
> expected.
>
> Most production scenarios, however, use the recommended practice of
> loading compiled KnowledgePackages that were serialized in a separate
> runs.
>
> Does the reported increase in execution times ("2-3 times slower")
> relate to rule execution only, without compiling? If I understand the
> OPs statement correctly, this increase was observed after the patch
> reducing PermGen space.
>
> -W
>
Hi there,
Actually the tests are the same on both versions, and you are right they
include the compilation of the rule set. In production however, we use the
technique you mentioned.
However, difference in compilation times between both versions is acceptable
for us (about 30-40% slower in big sets), if I drill down a bit on the
execution part however I see that it can get to 6 times slower at times.
I've done comparative analysis with YourKit lately on both versions to get a
grasp of where the time was spent. I can see the JIT part with the
ClassGenerator come into place, by that's not what baffles me: I see a lot
more DefaultConsequenceInvoker.evaluate() calls on the stack in 5.4.0 than
4.0.7 and that's where most of the time is spent
Does that make any sense? Are consequence evaluated even if rules are not
fired ? Or is the way the rule is written that causes this (no-loop
statements, ...) ?
Any way I can deactivate the JIT behavior to have a more solid ground for
comparison?
More questions than answers I'm afraid, hoping to get to the bottom of this
:-)
--
View this message in context:
http://drools.46999.n3.nabble.com/Migrating-from-4-0-7-to-5-4-0-Final-tp4018215p4018453.html
Sent from the Drools: User forum mailing list archive at Nabble.com.
___
rules-users mailing list
rules-users@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/rules-users
Re: [rules-users] Migrating from 4.0.7 to 5.4.0.Final
I just fixed the leaks you reported on the master repository and backported the fix to the 5.4.x branch. I also closed the JIRA ticket I opened a few hours ago, but it is still possible to comment on it or even reopen it if you will find further issues. I don't know if you have the possibility to build Drools from the sources but if so it could be great if you could give a try to this fix. Nevertheless it is still a fact that Drools 5.4 requires more PermGen space, for the reasons that both Mark and myself explained in our former emails. I hope this helps, Mario -- View this message in context: http://drools.46999.n3.nabble.com/Migrating-from-4-0-7-to-5-4-0-Final-tp4018215p4018436.html Sent from the Drools: User forum mailing list archive at Nabble.com. ___ rules-users mailing list rules-users@lists.jboss.org https://lists.jboss.org/mailman/listinfo/rules-users
Re: [rules-users] Migrating from 4.0.7 to 5.4.0.Final
Hi, sorry if I took too much time to reply to this: last week almost the whole Drools team has been busy with the JUDCon/JBWorld. About the ClassLoader leaking you're right. I just opened a ticket in JIRA about it ( https://issues.jboss.org/browse/JBRULES-3567 ) and I am starting working on this issue just now. If you have further comments or suggestions to fix it please feel free to attach them to that ticket. In general, as Mark already wrote, we generate a lot more of bytecode in the release 5.4 in order to improve the performance execution. More in detail, after a constraint has been evaluated a give amount of times (currently 20), we JIT that constraint via ASM. Of course the trade-off for this is that Drools now requires more PermGen space, so my advice is to enlarge it when possible. Any suggestion to improve this is welcome. Mario -- View this message in context: http://drools.46999.n3.nabble.com/Migrating-from-4-0-7-to-5-4-0-Final-tp4018215p4018430.html Sent from the Drools: User forum mailing list archive at Nabble.com. ___ rules-users mailing list rules-users@lists.jboss.org https://lists.jboss.org/mailman/listinfo/rules-users
Re: [rules-users] Migrating from 4.0.7 to 5.4.0.Final
It seems that the problems reported by the OP are mostly (if not all)
related to building KnowledgePackages from DRL files. Drools has been
through several 5.x versions with (among others) many extensions to
the DRL language, so some increase in resource consumption is to be
expected.
Most production scenarios, however, use the recommended practice of
loading compiled KnowledgePackages that were serialized in a separate
runs.
Does the reported increase in execution times ("2-3 times slower")
relate to rule execution only, without compiling? If I understand the
OPs statement correctly, this increase was observed after the patch
reducing PermGen space.
-W
On 03/07/2012, pgras wrote:
> Hello,
>
> I'm considering migrating one of our key (several thousands of rules,
> stateful sessions, and beans with propertyChangeSupport) application from
> 4.0.7 to 5.4 and your message isn't very encouraging :)
>
> Have you opened some JIRA's or have you made some other discoveries ?
>
> I will post my experiences here if I discover something but I maybe will
> have to wait until august before I can start with the migration.
>
> -Patrick
>
> --
> View this message in context:
> http://drools.46999.n3.nabble.com/Migrating-from-4-0-7-to-5-4-0-Final-tp4018215p4018416.html
> Sent from the Drools: User forum mailing list archive at Nabble.com.
> ___
> rules-users mailing list
> rules-users@lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/rules-users
>
___
rules-users mailing list
rules-users@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/rules-users
Re: [rules-users] Migrating from 4.0.7 to 5.4.0.Final
We generate a lot more bytecode now, for improved performance execution. constraints start of being evaluated dynamically, as they were in 4.0.x, after they are executed a certain number of times we emit bytecode to allow JIT to provide improved execution times. Mark On 29/06/2012 08:19, Jean-Paul Shemali wrote: Hi again all, Am i the only one noting these issues on the version? If this is not the right place to discuss this, could someone point me in the right direction (do I enter an issue in JIRA with potential patches, ...)? Thanks in advance > Date: Mon, 25 Jun 2012 00:01:16 -0700 > From: jshem...@hotmail.com > To: rules-users@lists.jboss.org > Subject: [rules-users] Migrating from 4.0.7 to 5.4.0.Final > > Hi all, > > I'm currently in the process of migrating an application using drools 4.0.7 > to 5.4.0.Final and I've stumbled into a few issues with the new version that > I find disturbing: > > - Our unit tests failed because the JVM ran out of perm gen space, which > looks odd as in 4.0.7 we've had quite a margin to start with. > - Doing several Yourkit analysis, I finally find classloading leaks: > * *org.drools.rule.builder.dialect.asm.ClassGenerator.EMPTY_METHOD_BODY* > is a static instance which will hold to the last instance of > InternalClassLoader created > * *org.drools.rule.constraint.MvelConstraint* uses an ExecutorHolder that > spawns daemon threads and keeps them in a pool. Problem is these threads > create class loaders, and can only be garbage collected when the thread > dies, meaning never in my case... > - I've quick fixed both issues, the second one by simply disabling the > thread pool. > - Once this is done, the perm gen behaves correctly, but the execution times > are 2-3 times slower on very large set of rules (~1000). Looking at Yourkit > analysis again, I see that the number of classes generated and the perm gen > consumption is about 3 times higher in 5.4.0.Final. I honestly don't know > how to address this. > > I've tried to find some other posts concerning these issues, to no avail. I > don't see any work around this, short of code changes, and for reducing the > number of generated classes I simply have no idea. > > Please correct me if I am wrong, any help would be greatly appreciated > > -- > View this message in context: http://drools.46999.n3.nabble.com/Migrating-from-4-0-7-to-5-4-0-Final-tp4018215.html > Sent from the Drools: User forum mailing list archive at Nabble.com. > ___ > rules-users mailing list > rules-users@lists.jboss.org > https://lists.jboss.org/mailman/listinfo/rules-users ___ rules-users mailing list rules-users@lists.jboss.org https://lists.jboss.org/mailman/listinfo/rules-users ___ rules-users mailing list rules-users@lists.jboss.org https://lists.jboss.org/mailman/listinfo/rules-users
Re: [rules-users] Migrating from 4.0.7 to 5.4.0.Final
Hello, I'm considering migrating one of our key (several thousands of rules, stateful sessions, and beans with propertyChangeSupport) application from 4.0.7 to 5.4 and your message isn't very encouraging :) Have you opened some JIRA's or have you made some other discoveries ? I will post my experiences here if I discover something but I maybe will have to wait until august before I can start with the migration. -Patrick -- View this message in context: http://drools.46999.n3.nabble.com/Migrating-from-4-0-7-to-5-4-0-Final-tp4018215p4018416.html Sent from the Drools: User forum mailing list archive at Nabble.com. ___ rules-users mailing list rules-users@lists.jboss.org https://lists.jboss.org/mailman/listinfo/rules-users
Re: [rules-users] Migrating from 4.0.7 to 5.4.0.Final
Hi again all, Am i the only one noting these issues on the version? If this is not the right place to discuss this, could someone point me in the right direction (do I enter an issue in JIRA with potential patches, ...)? Thanks in advance > Date: Mon, 25 Jun 2012 00:01:16 -0700 > From: jshem...@hotmail.com > To: rules-users@lists.jboss.org > Subject: [rules-users] Migrating from 4.0.7 to 5.4.0.Final > > Hi all, > > I'm currently in the process of migrating an application using drools 4.0.7 > to 5.4.0.Final and I've stumbled into a few issues with the new version that > I find disturbing: > > - Our unit tests failed because the JVM ran out of perm gen space, which > looks odd as in 4.0.7 we've had quite a margin to start with. > - Doing several Yourkit analysis, I finally find classloading leaks: > * *org.drools.rule.builder.dialect.asm.ClassGenerator.EMPTY_METHOD_BODY* > is a static instance which will hold to the last instance of > InternalClassLoader created > * *org.drools.rule.constraint.MvelConstraint* uses an ExecutorHolder that > spawns daemon threads and keeps them in a pool. Problem is these threads > create class loaders, and can only be garbage collected when the thread > dies, meaning never in my case... > - I've quick fixed both issues, the second one by simply disabling the > thread pool. > - Once this is done, the perm gen behaves correctly, but the execution times > are 2-3 times slower on very large set of rules (~1000). Looking at Yourkit > analysis again, I see that the number of classes generated and the perm gen > consumption is about 3 times higher in 5.4.0.Final. I honestly don't know > how to address this. > > I've tried to find some other posts concerning these issues, to no avail. I > don't see any work around this, short of code changes, and for reducing the > number of generated classes I simply have no idea. > > Please correct me if I am wrong, any help would be greatly appreciated > > -- > View this message in context: > http://drools.46999.n3.nabble.com/Migrating-from-4-0-7-to-5-4-0-Final-tp4018215.html > Sent from the Drools: User forum mailing list archive at Nabble.com. > ___ > rules-users mailing list > rules-users@lists.jboss.org > https://lists.jboss.org/mailman/listinfo/rules-users ___ rules-users mailing list rules-users@lists.jboss.org https://lists.jboss.org/mailman/listinfo/rules-users
