Re: Add perf issues

2010-05-11 Thread Emmanuel Lecharny
Some findings... I have added some counters in the code (ahhh, 24 years back in the past, when I was adding printlf() in my C code ;). Here is what I found : - most of the time is consumed in the CachedRecordManager.update() method, which calls BaseRecordManager.update( long recid, Object obj

Re: Add perf issues

2010-05-11 Thread Emmanuel Lécharny
On 5/11/10 11:53 AM, Alex Karasulu wrote: On Mon, May 10, 2010 at 11:03 AM, Alex Karasuluwrote: Hi Emmanuel, I modified a test and the JdbmTable implementation to expose exactly which data structure is being used and make sure the switch occurs at the appropriate time. The tests really were not

Re: Add perf issues

2010-05-11 Thread Alex Karasulu
On Mon, May 10, 2010 at 11:03 AM, Alex Karasulu wrote: > > > On Sun, May 9, 2010 at 11:51 PM, Alex Karasulu wrote: > >> Hi Emmanuel, >> >> On Sun, May 9, 2010 at 5:40 PM, Emmanuel Lecharny wrote: >> >>> Hi, >>> >>> while doing some perf test on the Add operation, I'm getting blocked >>> after havi

Re: Add perf issues

2010-05-10 Thread Alex Karasulu
On Mon, May 10, 2010 at 11:31 AM, Emmanuel Lécharny wrote: > On 5/10/10 10:03 AM, Alex Karasulu wrote: > >> >> I'm starting to look into the switch over to using secondary BTree's for >> duplicates after the threshold is reached to make sure this is working >> properly. I remember having a hard t

Re: Add perf issues

2010-05-10 Thread Emmanuel Lécharny
On 5/10/10 10:03 AM, Alex Karasulu wrote: I'm starting to look into the switch over to using secondary BTree's for duplicates after the threshold is reached to make sure this is working properly. I remember having a hard time writing a test case to make sure that this switch over occurs before

Re: Add perf issues

2010-05-10 Thread Alex Karasulu
On Sun, May 9, 2010 at 11:51 PM, Alex Karasulu wrote: > Hi Emmanuel, > > On Sun, May 9, 2010 at 5:40 PM, Emmanuel Lecharny wrote: > >> Hi, >> >> while doing some perf test on the Add operation, I'm getting blocked after >> having added around 3800 entries. After investigations, I found that some

Re: Add perf issues

2010-05-09 Thread Alex Karasulu
Hi Emmanuel, On Sun, May 9, 2010 at 5:40 PM, Emmanuel Lecharny wrote: > Hi, > > while doing some perf test on the Add operation, I'm getting blocked after > having added around 3800 entries. After investigations, I found that some > index are very expensive to update : > - the ObjectClass index q

Re: Add perf issues

2010-05-09 Thread Emmanuel Lecharny
>> Welcome to my nightmare. ;) I'm still looking for a fast/cheap index data structure that can be sparsely populated, is lossless, and scales to billions of entries. (Currently our index is fast/cheap but lossy. If you fill a slot with a million entries and then remove every other one, it

Re: Add perf issues

2010-05-09 Thread Howard Chu
Emmanuel Lecharny wrote: Hi, while doing some perf test on the Add operation, I'm getting blocked after having added around 3800 entries. After investigations, I found that some index are very expensive to update : - the ObjectClass index quickly get saturated, and adding a new entry into it cos

Add perf issues

2010-05-09 Thread Emmanuel Lecharny
Hi, while doing some perf test on the Add operation, I'm getting blocked after having added around 3800 entries. After investigations, I found that some index are very expensive to update : - the ObjectClass index quickly get saturated, and adding a new entry into it cost a hell of a time. -