Re: [solr-user] Upgrade from 1.2 to 1.3 gives 3x slowdown

2009-04-15 Thread Fergus McMenemie
On Apr 2, 2009, at 9:23 AM, Fergus McMenemie wrote: Grant, I should note, however, that the speed difference you are seeing may not be as pronounced as it appears. If I recall during ApacheCon, I commented on how long it takes to shutdown your Solr instance when exiting it. That time it

Re: [solr-user] Upgrade from 1.2 to 1.3 gives 3x slowdown

2009-04-15 Thread Ryan McKinley
The work being done is addressing the deletes, AIUI, but of course there are other things happening during shutdown, too. There are no deletes to do. It was a clean index to begin with and there were no duplicates. I have not followed this thread, so forgive me if this has already been

Re: [solr-user] Upgrade from 1.2 to 1.3 gives 3x slowdown

2009-04-02 Thread Fergus McMenemie
On Apr 1, 2009, at 9:39 AM, Fergus McMenemie wrote: Grant, Redoing the work with your patch applied does not seem to make a difference! Is this the expected result? No, I didn't expect Solr 1095 to fix the problem. Overwrite = false + 1095, does, however, AFAICT by your last line, right?

Re: [solr-user] Upgrade from 1.2 to 1.3 gives 3x slowdown

2009-04-02 Thread Grant Ingersoll
On Apr 2, 2009, at 4:02 AM, Fergus McMenemie wrote: Grant, Hmmm, the big difference is made by overwrite=false. But, can you explain why overwrite=false makes such a difference. I am starting off with an empty index and I have checked the content there are no duplicates in the uniqueKey field.

Re: [solr-user] Upgrade from 1.2 to 1.3 gives 3x slowdown

2009-04-02 Thread Grant Ingersoll
On Apr 2, 2009, at 9:23 AM, Fergus McMenemie wrote: Grant, I should note, however, that the speed difference you are seeing may not be as pronounced as it appears. If I recall during ApacheCon, I commented on how long it takes to shutdown your Solr instance when exiting it. That time it

Re: [solr-user] Upgrade from 1.2 to 1.3 gives 3x slowdown

2009-04-01 Thread Fergus McMenemie
Grant, Redoing the work with your patch applied does not seem to make a difference! Is this the expected result? I did run it again using the full file, this time using my Imac:- 643465took 22min 14sec 2008-04-01 734796 73min 58sec

Re: [solr-user] Upgrade from 1.2 to 1.3 gives 3x slowdown

2009-04-01 Thread Grant Ingersoll
On Apr 1, 2009, at 9:39 AM, Fergus McMenemie wrote: Grant, Redoing the work with your patch applied does not seem to make a difference! Is this the expected result? No, I didn't expect Solr 1095 to fix the problem. Overwrite = false + 1095, does, however, AFAICT by your last line,

Re: [solr-user] Upgrade from 1.2 to 1.3 gives 3x slowdown

2009-03-31 Thread Grant Ingersoll
svn co -r REV_NUM https://svn.apache.org/repos/asf/lucene/solr/trunk solr-REV_NUM -Grant On Mar 30, 2009, at 4:55 PM, Fergus McMenemie wrote: Can you verify that rev 701485 still performs reasonably well? This is from October 2008 and I get similar results to the earlier rev. Am now

Re: [solr-user] Upgrade from 1.2 to 1.3 gives 3x slowdown

2009-03-31 Thread Grant Ingersoll
Can you try adding overwrite=false and running against the latest version? My current working theory is that Solr/Lucene has changed how deletes are handled such that work that was deferred before is now not deferred as often. In fact, you are not seeing this cost paid (or at least not

Re: [solr-user] Upgrade from 1.2 to 1.3 gives 3x slowdown

2009-03-31 Thread Fergus McMenemie
Grant, I am messing with the script, and with your tip I expect I can make it recurse over as many releases as needed. I did run it again using the full file, this time using my Imac:- 643465took 22min 14sec 2008-04-01 734796 73min 58sec

Re: [solr-user] Upgrade from 1.2 to 1.3 gives 3x slowdown

2009-03-30 Thread Grant Ingersoll
Fergus, I think the problem may actually be due to something that was introduced by a change to Solr's StopFilterFactory and the way it loads the stop words set. See https://issues.apache.org/jira/browse/SOLR-1095 I am in the process of testing it out and will let you know. -Grant On

Re: [solr-user] Upgrade from 1.2 to 1.3 gives 3x slowdown

2009-03-30 Thread Grant Ingersoll
Fregus, Is rev 643465 the absolute latest you tried that still performs? i.e. every revision after is slower? -Grant On Mar 30, 2009, at 12:45 PM, Grant Ingersoll wrote: Fergus, I think the problem may actually be due to something that was introduced by a change to Solr's

Re: [solr-user] Upgrade from 1.2 to 1.3 gives 3x slowdown

2009-03-30 Thread Grant Ingersoll
Can you verify that rev 701485 still performs reasonably well? This is from October 2008 and I get similar results to the earlier rev. Am now trying some other versions between October and when you first reported the issue in November. -Grant On Mar 30, 2009, at 3:37 PM, Grant

Re: [solr-user] Upgrade from 1.2 to 1.3 gives 3x slowdown

2009-03-30 Thread Fergus McMenemie
Grant, After all my playing about at boot camp, I gave things a rest. It was not till months later that got back to looking at solr again. So after 643465 (2008-Apr-01) the next version I tried was 694377 from (2008-Sep-11). Nothing in between. Yep so 643465 is the latest version I tried that

Re: [solr-user] Upgrade from 1.2 to 1.3 gives 3x slowdown

2009-03-30 Thread Fergus McMenemie
Can you verify that rev 701485 still performs reasonably well? This is from October 2008 and I get similar results to the earlier rev. Am now trying some other versions between October and when you first reported the issue in November. OK. Can you tell me how to get a hold of revision

RE: [solr-user] Upgrade from 1.2 to 1.3 gives 3x slowdown

2009-03-28 Thread Grant Ingersoll
Hey Fergus, Finally got a chance to run your scripts, etc. per the thread: http://www.lucidimagination.com/search/document/5c3de15a4e61095c/upgrade_from_1_2_to_1_3_gives_3x_slowdown_script#8324a98d8840c623 I can reproduce your slowdown. One oddity with rev 643465 is: On the old version, there