Ernest: I think that James Owen with KBSC originally reported that the JRULES-OPT performance was skewed - especially for the Manners benchmark. The JRules Manners-Optimized benchmark, upon close examination, revealed that there was a limited number of rules (two or three) being fired with only a cycling of two rules once the process got really kicked off. This meant that the rules engine was not being stressed as the benchmark had originally intended. In other words, it took a short cut that could not be duplicated with "real world" problems.
If anyone can duplicate that kind of performance with real world problems I, for one, would be MOST interested in how they did it. I tried it as well and could never get that kind of improvement (100 times or more) over the regular interpreted or compiled versions of JRules. About all I ever got was about 20% (1.2 times) improvement. Let me know if you find the "trick" to making it work in the real world. :-) Lying? Naaahhhh.... It's just a marketing thing. Even Oracle and Sybase and others did that with the "standard" benchmarks for databases a few years ago. SDG jco -----Original Message----- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]] Sent: Thursday, January 16, 2003 11:46 AM To: Jess Mailing List Subject: Re: JESS: Perfromance metrics for forward chaining inferencing engines I think ejfried wrote: > > A Jess user posted their results of comparing quite a few different > rule engines' performance on one of Miranker's programs: see > > http://www.mail-archive.com/[email protected]/msg03278.html > > The ensuing discussion, both on and off list, was interesting. In sum, > it turned out that the "JRULES-OPT" run is a more or less hard-coded > solution to the problem, and that, despite claims to the contrary, the > data point for Rete++ is accurate. > It has been pointed out to me that the statement above, made by me in late November, could be misinterpreted as my accusing some of the correspondents in the earlier discussion of lying about the performance of specific products. I want to make it clear that I didn't intend anything of the sort. All that I meant to say was that the posted benchmark results were, in fact, the genuine results obtained by the original poster in his specific test environment, and with his specific translation of the benchmark program -- i.e., there were no transcription errors or other problems. It is very likely that each individual vendor could obtain better results for their own product if they were allowed to tailor the benchmark. By way of example, for Jess, benchmark performance can be strongly influenced by the HotSpot VM garbage collector parameters. Enlarging the "object nursery" from its default size can improve Jess performance by a factor of 2 or more. It is quite possible that other vendor's products perform best with other settings, or with specific other JVMs, or using a specific translation of the benchmark program which differs in some important way from the one the original poster wrote. No third-party benchmark will show ay product in its most flattering light. Some products may suffer more than others by their treatment during the benchmark, and that may well have been the case here. For this reason, it's important not to take bake-offs like this too seriously. As they say, "YMMV." If performance is important to you, you'll want to evaluate a range of products for yourself, using your own environment and your own data. --------------------------------------------------------- Ernest Friedman-Hill Distributed Systems Research Phone: (925) 294-2154 Sandia National Labs FAX: (925) 294-2234 PO Box 969, MS 9012 [EMAIL PROTECTED] Livermore, CA 94550 http://herzberg.ca.sandia.gov -------------------------------------------------------------------- To unsubscribe, send the words 'unsubscribe jess-users [EMAIL PROTECTED]' in the BODY of a message to [EMAIL PROTECTED], NOT to the list (use your own address!) List problems? Notify [EMAIL PROTECTED] -------------------------------------------------------------------- -------------------------------------------------------------------------------------------------------------------------------------------------- Note: The information contained in this message may be privileged and confidential and protected from disclosure. If the reader of this message is not the intended recipient, or an employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by replying to the message and deleting it from your computer. Thank you. -------------------------------------------------------------------------------------------------------------------------------------------------- -------------------------------------------------------------------- To unsubscribe, send the words 'unsubscribe jess-users [EMAIL PROTECTED]' in the BODY of a message to [EMAIL PROTECTED], NOT to the list (use your own address!) List problems? Notify [EMAIL PROTECTED] --------------------------------------------------------------------
