RE: Lucene and Javolution: A good mix ?
I understand your concerns! I was a little skeptical at the beginning. But even with the 1.5 jvm, the improvements still holds. Lucene creates a lots of garbage (strings, tokens, ...) either at index time or query time. While the new garbage collector strategies did seriously improve since java 1.4, the gains are still there as the object creation is also a cost that javolution easily saves us from. What javolution requires to give the best is can is for you to make certain critical classes extends the RealtimeObject class and implement a Factory pattern inside. Once this is done, you can now fully profit from the Javolution features. Even without doing that, we could still benefit from the FastList/FastMap classes being already pooled and the possibility to 'thread-safely' iterate lists/maps without creating any objects. Javolution is also released for gcj, which is great since it won't interfere with the lucene's gcj effort. From what I can foresee, the pros/cons would be: Pros: Leaner memory footprint Saving many cpu cycles Cons: Adding a dependency to lucene codebase Lucene developers must get familiar with the Context concepts Jp -Original Message- From: robert engels [mailto:[EMAIL PROTECTED] Sent: Wednesday, April 04, 2007 10:31 PM To: java-dev@lucene.apache.org Subject: Re: Lucene and Javolution: A good mix ? I would suggest that the Javolution folks do their tests against modern JVM... I have followed the Javolution project for some time, and while I agree that some of the techniques should improve things, I think that modern JVMs do most of this work for you (and the latest class libraries also help - StringBuilder and others). I also think that when you start doing you own memory management you might as well write the code in C/C++ because you need to use similar techniques (similar to the resource management when using SWT). Just my thoughts. On Apr 4, 2007, at 8:54 PM, Jean-Philippe Robichaud wrote: Hello Dear Lucene coders! Some of you may remember, I'm using lucene for a product (and many other internal utilities). I'm also using another open source library called Javolution (www.javolution.org http://www.javolution.org/ ) which does many things, one of them being to offer excellent replacements for ArrayList/Map/... and a super good memory management extension to the java language. As I'm [trying to] follow the conversations on this list, I see that many of you are working towards optimizing lucene in term of memory footprint and speed. I just finished optimizing my code (not lucene itself, but my code written on top of it) using Javolution PoolContext and the FastList/FastMap/... classes. The resulting speedup is a 6 times faster code. Javolution make it easy to recycle objects and do some object allocation on the stack rather than on the head, which remove stress on the garbage collector. Javolution also offers 2 classes (Text and TextBuilder) to replace String/StringBuffer which are perfect for anything related to string manipulation and some C union/struct equivalent for java. The thing is really great. Would anyone be interested in doing Lucene a face lift and start using javolution as a core lucene dependency? I understand that right now, lucene is free of any dependencies, which is quite great, but anyone interested in doing fast/lean/stable java application should seriously consider using javolution anyway. Any thoughts? Jp - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Lucene and Javolution: A good mix ?
On 4/4/07, Jean-Philippe Robichaud [EMAIL PROTECTED] wrote: I understand your concerns! I was a little skeptical at the beginning. But even with the 1.5 jvm, the improvements still holds. Lucene creates a lots of garbage (strings, tokens, ...) either at index time or query time. While the new garbage collector strategies did seriously improve since java 1.4, the gains are still there as the object creation is also a cost that javolution easily saves us from. I think the best approach at convincing people would be to produce a patch that implements some of the suggested changes, and benchmark it. As it stands, the positives are all hypothetical and the negatives rather tangible. -MIke - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Lucene and Javolution: A good mix ?
What Mike said. Without seeing the Javalutionized Lucene in action we won't get very far. jean-Philippe, are you interested in making the changes to Lucene and showing the performance improvement? Note that you can use the super-nice and easy to use contrib/benchmark to compare the vanilla Lucene and the Javalutionized Lucene. Otis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Simpy -- http://www.simpy.com/ - Tag - Search - Share - Original Message From: Mike Klaas [EMAIL PROTECTED] To: java-dev@lucene.apache.org Sent: Thursday, April 5, 2007 1:58:38 PM Subject: Re: Lucene and Javolution: A good mix ? On 4/4/07, Jean-Philippe Robichaud [EMAIL PROTECTED] wrote: I understand your concerns! I was a little skeptical at the beginning. But even with the 1.5 jvm, the improvements still holds. Lucene creates a lots of garbage (strings, tokens, ...) either at index time or query time. While the new garbage collector strategies did seriously improve since java 1.4, the gains are still there as the object creation is also a cost that javolution easily saves us from. I think the best approach at convincing people would be to produce a patch that implements some of the suggested changes, and benchmark it. As it stands, the positives are all hypothetical and the negatives rather tangible. -MIke - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Lucene and Javolution: A good mix ?
Yes, I believe enough in this approach to try it. I'm already starting to play with it. I took the current trunk and I'm starting to play with it. That begin said, I'm quite busy right now so I can't promise any steady progress. Also, I won't apply patches that are already in JIRA, so the numbers I'll get won't be the 'up-to-date ones. I understand that before this idea gets any traction, we must have an idea of how much this could help. But before going deep with this work, I wanted to know if Lucene developers have any interest in this kind of work. If the gurus dislike the idea of adding a dependency to Lucene (which is not the case for others Apache projects!), then I won't spend too much time on this. Jp -Original Message- From: Otis Gospodnetic [mailto:[EMAIL PROTECTED] Sent: Thursday, April 05, 2007 3:01 PM To: java-dev@lucene.apache.org Subject: Re: Lucene and Javolution: A good mix ? What Mike said. Without seeing the Javalutionized Lucene in action we won't get very far. jean-Philippe, are you interested in making the changes to Lucene and showing the performance improvement? Note that you can use the super-nice and easy to use contrib/benchmark to compare the vanilla Lucene and the Javalutionized Lucene. Otis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Simpy -- http://www.simpy.com/ - Tag - Search - Share - Original Message From: Mike Klaas [EMAIL PROTECTED] To: java-dev@lucene.apache.org Sent: Thursday, April 5, 2007 1:58:38 PM Subject: Re: Lucene and Javolution: A good mix ? On 4/4/07, Jean-Philippe Robichaud [EMAIL PROTECTED] wrote: I understand your concerns! I was a little skeptical at the beginning. But even with the 1.5 jvm, the improvements still holds. Lucene creates a lots of garbage (strings, tokens, ...) either at index time or query time. While the new garbage collector strategies did seriously improve since java 1.4, the gains are still there as the object creation is also a cost that javolution easily saves us from. I think the best approach at convincing people would be to produce a patch that implements some of the suggested changes, and benchmark it. As it stands, the positives are all hypothetical and the negatives rather tangible. -MIke - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Lucene and Javolution: A good mix ?
I'm not in love with the dependency idea, though it's not that big of a deal for me. However, I think you will want to get some of the performance patched (e.g. LUCENE-843) in first, so you can compare the latest and greatest version of Lucene with your Javalutionized version. From what I gather from Mike's emails, he is doing a lot of object and array sharing and reusing in order to minimize object creation, memory allocation, and thus create less work for the garbage collector. My 2 pick a currency, say Levs. Otis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Simpy -- http://www.simpy.com/ - Tag - Search - Share - Original Message From: Jean-Philippe Robichaud [EMAIL PROTECTED] To: java-dev@lucene.apache.org Sent: Thursday, April 5, 2007 3:19:51 PM Subject: RE: Lucene and Javolution: A good mix ? Yes, I believe enough in this approach to try it. I'm already starting to play with it. I took the current trunk and I'm starting to play with it. That begin said, I'm quite busy right now so I can't promise any steady progress. Also, I won't apply patches that are already in JIRA, so the numbers I'll get won't be the 'up-to-date ones. I understand that before this idea gets any traction, we must have an idea of how much this could help. But before going deep with this work, I wanted to know if Lucene developers have any interest in this kind of work. If the gurus dislike the idea of adding a dependency to Lucene (which is not the case for others Apache projects!), then I won't spend too much time on this. Jp -Original Message- From: Otis Gospodnetic [mailto:[EMAIL PROTECTED] Sent: Thursday, April 05, 2007 3:01 PM To: java-dev@lucene.apache.org Subject: Re: Lucene and Javolution: A good mix ? What Mike said. Without seeing the Javalutionized Lucene in action we won't get very far. jean-Philippe, are you interested in making the changes to Lucene and showing the performance improvement? Note that you can use the super-nice and easy to use contrib/benchmark to compare the vanilla Lucene and the Javalutionized Lucene. Otis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Simpy -- http://www.simpy.com/ - Tag - Search - Share - Original Message From: Mike Klaas [EMAIL PROTECTED] To: java-dev@lucene.apache.org Sent: Thursday, April 5, 2007 1:58:38 PM Subject: Re: Lucene and Javolution: A good mix ? On 4/4/07, Jean-Philippe Robichaud [EMAIL PROTECTED] wrote: I understand your concerns! I was a little skeptical at the beginning. But even with the 1.5 jvm, the improvements still holds. Lucene creates a lots of garbage (strings, tokens, ...) either at index time or query time. While the new garbage collector strategies did seriously improve since java 1.4, the gains are still there as the object creation is also a cost that javolution easily saves us from. I think the best approach at convincing people would be to produce a patch that implements some of the suggested changes, and benchmark it. As it stands, the positives are all hypothetical and the negatives rather tangible. -MIke - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Lucene and Javolution: A good mix ?
I'm not saying I'm against it, but one of the things that makes Lucene so great is it's lack of dependencies in the core. It isn't necessarily a slippery slope, either, if we do add one dependency. Javolution is BSD license, AFAICT. I don't know if that is a good or bad license as far as Apache is concerned, but it should be looked into before you spend any time on it. This is not meant to be a discouragement. If it shows a significant improvement, people will notice and it will be taken seriously, especially if it is backward compatible, well tested and well documented. -Grant On Apr 5, 2007, at 3:19 PM, Jean-Philippe Robichaud wrote: Yes, I believe enough in this approach to try it. I'm already starting to play with it. I took the current trunk and I'm starting to play with it. That begin said, I'm quite busy right now so I can't promise any steady progress. Also, I won't apply patches that are already in JIRA, so the numbers I'll get won't be the 'up-to-date ones. I understand that before this idea gets any traction, we must have an idea of how much this could help. But before going deep with this work, I wanted to know if Lucene developers have any interest in this kind of work. If the gurus dislike the idea of adding a dependency to Lucene (which is not the case for others Apache projects!), then I won't spend too much time on this. Jp -Original Message- From: Otis Gospodnetic [mailto:[EMAIL PROTECTED] Sent: Thursday, April 05, 2007 3:01 PM To: java-dev@lucene.apache.org Subject: Re: Lucene and Javolution: A good mix ? What Mike said. Without seeing the Javalutionized Lucene in action we won't get very far. jean-Philippe, are you interested in making the changes to Lucene and showing the performance improvement? Note that you can use the super-nice and easy to use contrib/benchmark to compare the vanilla Lucene and the Javalutionized Lucene. Otis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Simpy -- http://www.simpy.com/ - Tag - Search - Share - Original Message From: Mike Klaas [EMAIL PROTECTED] To: java-dev@lucene.apache.org Sent: Thursday, April 5, 2007 1:58:38 PM Subject: Re: Lucene and Javolution: A good mix ? On 4/4/07, Jean-Philippe Robichaud Jean- [EMAIL PROTECTED] wrote: I understand your concerns! I was a little skeptical at the beginning. But even with the 1.5 jvm, the improvements still holds. Lucene creates a lots of garbage (strings, tokens, ...) either at index time or query time. While the new garbage collector strategies did seriously improve since java 1.4, the gains are still there as the object creation is also a cost that javolution easily saves us from. I think the best approach at convincing people would be to produce a patch that implements some of the suggested changes, and benchmark it. As it stands, the positives are all hypothetical and the negatives rather tangible. -MIke - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] -- Grant Ingersoll http://www.grantingersoll.com/ http://lucene.grantingersoll.com http://www.paperoftheweek.com/ - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Lucene and Javolution: A good mix ?
I would suggest that the Javolution folks do their tests against modern JVM... I have followed the Javolution project for some time, and while I agree that some of the techniques should improve things, I think that modern JVMs do most of this work for you (and the latest class libraries also help - StringBuilder and others). I also think that when you start doing you own memory management you might as well write the code in C/C++ because you need to use similar techniques (similar to the resource management when using SWT). Just my thoughts. On Apr 4, 2007, at 8:54 PM, Jean-Philippe Robichaud wrote: Hello Dear Lucene coders! Some of you may remember, I'm using lucene for a product (and many other internal utilities). I'm also using another open source library called Javolution (www.javolution.org http://www.javolution.org/ ) which does many things, one of them being to offer excellent replacements for ArrayList/Map/... and a super good memory management extension to the java language. As I'm [trying to] follow the conversations on this list, I see that many of you are working towards optimizing lucene in term of memory footprint and speed. I just finished optimizing my code (not lucene itself, but my code written on top of it) using Javolution PoolContext and the FastList/FastMap/... classes. The resulting speedup is a 6 times faster code. Javolution make it easy to recycle objects and do some object allocation on the stack rather than on the head, which remove stress on the garbage collector. Javolution also offers 2 classes (Text and TextBuilder) to replace String/StringBuffer which are perfect for anything related to string manipulation and some C union/struct equivalent for java. The thing is really great. Would anyone be interested in doing Lucene a face lift and start using javolution as a core lucene dependency? I understand that right now, lucene is free of any dependencies, which is quite great, but anyone interested in doing fast/lean/stable java application should seriously consider using javolution anyway. Any thoughts? Jp - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
