Re: [rules-users] Drools Fusion Performance and RETE algorithm
Orlando, The Power of Events has a full chapter on EPN. Also, it is a common subject on the network. Edson 2010/8/18 Orlando Costa orlandorica...@gmail.com Hi Edson I was more refering to more sub-set of ESP/CEP in my question, because of the description you have made about EPN and agent architecture that could be used as a footprint to a possible solution. Thus, I was wondering if there was more technical resource just on that topic (and not so much in CEP theory/background in general, as I already material on that). Best regards Orlando. -- View this message in context: http://drools-java-rules-engine.46999.n3.nabble.com/Drools-Fusion-Performance-and-RETE-algorithm-tp1190098p1214947.html Sent from the Drools - User mailing list archive at Nabble.com. ___ rules-users mailing list rules-users@lists.jboss.org https://lists.jboss.org/mailman/listinfo/rules-users -- Edson Tirelli JBoss Drools Core Development JBoss by Red Hat @ www.jboss.com ___ rules-users mailing list rules-users@lists.jboss.org https://lists.jboss.org/mailman/listinfo/rules-users
Re: [rules-users] Drools Fusion Performance and RETE algorithm
Hi Edson, This was really the kind of information I was looking for, many thanks! (I think you should even put this text highlighted on the documentation available at Drools site, these are very valuable insights). I am kind of new in ESP/CEP (been reading Event Processing - Designing IT Systems for Agile Companies and many other online info), but when you talk about EPN (Event-Processing Network), this is a concept that I am not yet familiar. This paragraph were you say: Although, the chosen architecture for your kbase and sessions will obviously have a high impact on the performance metrics you will achieve. Since you mentioned that in your use case every single fact will match and cause rules to fire, and the actions involve heavy operations like database access, I would recommend looking into an agent architecture, where you partition your knowledge base into several kbases with related rules. The kbases will operate on the atomic (or raw) events and create the composite events that you can feed into entry points of sessions of subsequent kbases, effectively increasing the overall system throughput and simplifying the maintenance of each agent's kbase in particular. looks quite interesting to explore. Besides the book I mentioned (and I am also going to check this on Event Processing in Action by Etzion) do you recommend any resource (book, article, etc) that elaborates more on the concepts you have described? Best regards Orlando. -- View this message in context: http://drools-java-rules-engine.46999.n3.nabble.com/Drools-Fusion-Performance-and-RETE-algorithm-tp1190098p1213845.html Sent from the Drools - User mailing list archive at Nabble.com. ___ rules-users mailing list rules-users@lists.jboss.org https://lists.jboss.org/mailman/listinfo/rules-users
Re: [rules-users] Drools Fusion Performance and RETE algorithm
Hi Orlando, Dr. David Luckam's website has a lot of content on CEP: http://www.complexevents.com/ http://www.complexevents.com/ Also, his book The Power of Events is considered the publication that formalized the concept of CEP. It is a good book, very helpful in understanding the overall concepts. Just be aware that being a visionary book, on one hand, lots of requirements for CEP products described there are still wishful thinking that no product on the market implemented yet and on the other hand, some products offer several features that were not envisioned by him at the time. Edson 2010/8/18 Orlando Costa orlandorica...@gmail.com Hi Edson, This was really the kind of information I was looking for, many thanks! (I think you should even put this text highlighted on the documentation available at Drools site, these are very valuable insights). I am kind of new in ESP/CEP (been reading Event Processing - Designing IT Systems for Agile Companies and many other online info), but when you talk about EPN (Event-Processing Network), this is a concept that I am not yet familiar. This paragraph were you say: Although, the chosen architecture for your kbase and sessions will obviously have a high impact on the performance metrics you will achieve. Since you mentioned that in your use case every single fact will match and cause rules to fire, and the actions involve heavy operations like database access, I would recommend looking into an agent architecture, where you partition your knowledge base into several kbases with related rules. The kbases will operate on the atomic (or raw) events and create the composite events that you can feed into entry points of sessions of subsequent kbases, effectively increasing the overall system throughput and simplifying the maintenance of each agent's kbase in particular. looks quite interesting to explore. Besides the book I mentioned (and I am also going to check this on Event Processing in Action by Etzion) do you recommend any resource (book, article, etc) that elaborates more on the concepts you have described? Best regards Orlando. -- View this message in context: http://drools-java-rules-engine.46999.n3.nabble.com/Drools-Fusion-Performance-and-RETE-algorithm-tp1190098p1213845.html Sent from the Drools - User mailing list archive at Nabble.com. ___ rules-users mailing list rules-users@lists.jboss.org https://lists.jboss.org/mailman/listinfo/rules-users -- Edson Tirelli JBoss Drools Core Development JBoss by Red Hat @ www.jboss.com ___ rules-users mailing list rules-users@lists.jboss.org https://lists.jboss.org/mailman/listinfo/rules-users
Re: [rules-users] Drools Fusion Performance and RETE algorithm
Hi Edson I was more refering to more sub-set of ESP/CEP in my question, because of the description you have made about EPN and agent architecture that could be used as a footprint to a possible solution. Thus, I was wondering if there was more technical resource just on that topic (and not so much in CEP theory/background in general, as I already material on that). Best regards Orlando. -- View this message in context: http://drools-java-rules-engine.46999.n3.nabble.com/Drools-Fusion-Performance-and-RETE-algorithm-tp1190098p1214947.html Sent from the Drools - User mailing list archive at Nabble.com. ___ rules-users mailing list rules-users@lists.jboss.org https://lists.jboss.org/mailman/listinfo/rules-users
[rules-users] Drools Fusion Performance and RETE algorithm
Hi all Looking at Drools JBoss Rules 5 0 Developer's guide it addresses the Fusion component at page 134, stating: This problem cannot be solved by the standard Drools rule engine. The volume of events is huge and it happens asynchronously. If we simply inserted them into the knowledge session, we would soon run out of memory. While the Rete algorithm behind Drools doesn't have any theoretical limitation on number of objects in the session, we could use the processing power more wisely. Drools Fusion is the right candidate for this kind of task. Ok, but my question is if this is enough, because I would guess that Fusion could be used to derived a more complex/rich event from the stream of events, and afterwards that complex event would be inserted in Working Memory (but not the flow of events that have generated the more complex event), thus reducing RETE algorithm processing. But since in our case every event will most certainly fire a rule, I am not sure about Fusion's usage. This is because our project is more a kind of Event-Trigger-Rule/Action system, meaning we have a high volume of incoming events (say more than 5000/minute), but each of the events will have to trigger a rule (and usually a rule is more or less a complex action, invoking DB operations and data structures manipulation). So, by using Fusion entry-points, would that be different than just sending the events as normal facts to Working Memory, or by dividing this in entry-points( or so called streams) we would have benefits in the RETE processing (I am assuming Fusion also uses RETE someway)? Best regards Orlando. PS-Does anyone know where can I find more info on the coupling of Drools Expert and Fusion? ___ rules-users mailing list rules-users@lists.jboss.org https://lists.jboss.org/mailman/listinfo/rules-users
Re: [rules-users] Drools Fusion Performance and RETE algorithm
Thanks Mauricio, but since in general RETE algorithm does not get along well with high volumes of information, I was wondering if I could get more details on: 1. the implementation architecture 2. about Drools Fusion performance, does anyone has already some experience or links to share? Br Orlando. -- View this message in context: http://drools-java-rules-engine.46999.n3.nabble.com/Drools-Fusion-Performance-and-RETE-algorithm-tp1190098p1192480.html Sent from the Drools - User mailing list archive at Nabble.com. ___ rules-users mailing list rules-users@lists.jboss.org https://lists.jboss.org/mailman/listinfo/rules-users
Re: [rules-users] Drools Fusion Performance and RETE algorithm
Google for a presentation that shows how fedex monitor a lot of trucks in near real time with fusion. I can't find the link, but probably Edson have it. On Tue, Aug 17, 2010 at 3:00 PM, Orlando Costa orlandorica...@gmail.comwrote: Thanks Mauricio, but since in general RETE algorithm does not get along well with high volumes of information, I was wondering if I could get more details on: 1. the implementation architecture 2. about Drools Fusion performance, does anyone has already some experience or links to share? Br Orlando. -- View this message in context: http://drools-java-rules-engine.46999.n3.nabble.com/Drools-Fusion-Performance-and-RETE-algorithm-tp1190098p1192480.html Sent from the Drools - User mailing list archive at Nabble.com. ___ rules-users mailing list rules-users@lists.jboss.org https://lists.jboss.org/mailman/listinfo/rules-users -- - CTO @ http://www.plugtree.com - MyJourney @ http://salaboy.wordpress.com - Co-Founder @ http://www.jbug.com.ar - Salatino Salaboy Mauricio - ___ rules-users mailing list rules-users@lists.jboss.org https://lists.jboss.org/mailman/listinfo/rules-users
Re: [rules-users] Drools Fusion Performance and RETE algorithm
Thanks, I already have seen this some days ago: http://www.redhat.com/f/pdf/jbw/amollenkopf_430_applying_drools.pdf but it does not say the amount of events coming, and the internals are more about the whole solution itself than about the internal Drools Expert/Fusion architecture (though is shows good rules examples with Fusion use of course). Br Orlando. -- View this message in context: http://drools-java-rules-engine.46999.n3.nabble.com/Drools-Fusion-Performance-and-RETE-algorithm-tp1190098p1193090.html Sent from the Drools - User mailing list archive at Nabble.com. ___ rules-users mailing list rules-users@lists.jboss.org https://lists.jboss.org/mailman/listinfo/rules-users
Re: [rules-users] Drools Fusion Performance and RETE algorithm
Hi Orlando, Mauricio, While the goal of the whole Drools platform is to provide a seamless environment for users to model business behavior using rules, processes and events, it is a misconception to say that it is all executed on RETE. Besides implementing several optimizations on top of the classic RETE algorithm (as designed by Dr. Forgy back in the 70's), Drools has specific algorithms to which it delegates processing of specific use cases/features (read some fusion and flow features). Regarding your question, entry-points are partitions on the RETE network that allow for parts of it to be evaluated concurrently, besides creating separate name spaces for facts (scopes if you will). So it does help on reducing the overall load of the matching algorithm. Although, rules are still fired sequentially as to ensure consistency of the working memory. Regarding use cases, we have customers that use 24x7 stateful sessions, keeping over 1M facts in memory constantly, with new facts coming into the session and facts expiring from the session all the time. The throughput of such use cases is obviously dependent on the percentage of facts/rules matchings, so it is hard to foresee what you will achieve on your use case without knowing in detail your rules. Although, the chosen architecture for your kbase and sessions will obviously have a high impact on the performance metrics you will achieve. Since you mentioned that in your use case every single fact will match and cause rules to fire, and the actions involve heavy operations like database access, I would recommend looking into an agent architecture, where you partition your knowledge base into several kbases with related rules. The kbases will operate on the atomic (or raw) events and create the composite events that you can feed into entry points of sessions of subsequent kbases, effectively increasing the overall system throughput and simplifying the maintenance of each agent's kbase in particular. If you are used to CEP nomenclature, you know that the above is a simplified description of an EPN. Finally, you asked about the difference between regular facts and events, and basically all events are facts with special properties. These properties allow the engine to do some fancy stuff with them, like the use of sliding windows or event lifecycle management (i.e., automatically expiring events). Although, there is nothing wrong in modeling events like regular facts if that best fits your design. Hope the above helps, Edson 2010/8/17 Orlando Costa orlandorica...@gmail.com Thanks, I already have seen this some days ago: http://www.redhat.com/f/pdf/jbw/amollenkopf_430_applying_drools.pdf but it does not say the amount of events coming, and the internals are more about the whole solution itself than about the internal Drools Expert/Fusion architecture (though is shows good rules examples with Fusion use of course). Br Orlando. -- View this message in context: http://drools-java-rules-engine.46999.n3.nabble.com/Drools-Fusion-Performance-and-RETE-algorithm-tp1190098p1193090.html Sent from the Drools - User mailing list archive at Nabble.com. ___ rules-users mailing list rules-users@lists.jboss.org https://lists.jboss.org/mailman/listinfo/rules-users -- Edson Tirelli JBoss Drools Core Development JBoss by Red Hat @ www.jboss.com ___ rules-users mailing list rules-users@lists.jboss.org https://lists.jboss.org/mailman/listinfo/rules-users
