[jira] Updated: (LUCENE-2272) PayloadNearQuery has hardwired explanation for 'AveragePayloadFunction'

2010-03-23 Thread Peter Keegan (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-2272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Keegan updated LUCENE-2272: - Attachment: PNQ-patch.txt There is a bug in PayloadNearQuery. If there are multiple top level

[jira] Updated: (LUCENE-2272) PayloadNearQuery has hardwired explanation for 'AveragePayloadFunction'

2010-02-19 Thread Peter Keegan (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-2272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Keegan updated LUCENE-2272: - Attachment: payloadfunctin-patch.txt This patch adds the 'explain' met

[jira] Created: (LUCENE-2272) PayloadNearQuery has hardwired explanation for 'AveragePayloadFunction'

2010-02-19 Thread Peter Keegan (JIRA)
ene - Java Issue Type: Bug Components: Search Reporter: Peter Keegan Attachments: payloadfunctin-patch.txt The 'explain' method in PayloadNearSpanScorer assumes the AveragePayloadFunction was used. This patch adds the 'explain' method to the 

[jira] Commented: (LUCENE-1986) NPE in NearSpansUnordered from PayloadNearQuery

2009-10-19 Thread Peter Keegan (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-1986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12767373#action_12767373 ] Peter Keegan commented on LUCENE-1986: -- + if (!more) { +return f

[jira] Updated: (LUCENE-1986) NPE in NearSpansUnordered from PayloadNearQuery

2009-10-16 Thread Peter Keegan (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-1986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Keegan updated LUCENE-1986: - Attachment: TestPayloadNearQuery1.java Unit test that causes NPE > NPE in NearSpansUnorde

[jira] Created: (LUCENE-1986) NPE in NearSpansUnordered from PayloadNearQuery

2009-10-16 Thread Peter Keegan (JIRA)
Affects Versions: 2.9 Reporter: Peter Keegan Attachments: TestPayloadNearQuery1.java The following query causes a NPE in NearSpansUnordered, and is reproducible with the the attached unit test. The failure occurs on the last document scored. -- This message is automatically

Re: Query Parsing was Fwd: Lab - Esqueranto

2009-09-25 Thread Peter Keegan
We're using Antlr for our query parsing. What I like about it: - flexibility of separate lexer/parser and tree api - excellent IDE for building/testing the grammar However, the learning curve was quite long for me, although this was my first real encounter with parsers. Peter On Fri, Sep 25, 2009

[jira] Updated: (LUCENE-1341) BoostingNearQuery class (prototype)

2009-08-05 Thread Peter Keegan (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-1341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Keegan updated LUCENE-1341: - Attachment: lucene-1341-new-2.patch New version that works with current trunk (8/5/09

Re: SpanQuery and BoostingTermQuery oddities

2009-08-05 Thread Peter Keegan
: protected Weight createWeight(Searcher searcher) throws IOException public Scorer scorer(IndexReader reader) throws IOException Peter On Wed, Aug 5, 2009 at 12:29 PM, Peter Keegan wrote: > I ran into the same problem trying to update the BoostingNearQuery patch in > *LUCENE-1341

Re: SpanQuery and BoostingTermQuery oddities

2009-08-05 Thread Peter Keegan
I ran into the same problem trying to update the BoostingNearQuery patch in *LUCENE-1341 . *The scorer never gets called. This used to work in 2.3.2 Peter On Wed, Aug 5, 2009 at 11:01 AM, Mark Miller wrote: > Grant Ingersoll wrote: > >> >> On A

Re: Common Bottlenecks

2009-06-24 Thread Peter Keegan
I just want to point out that I don't consider these bottlenecks to be an issue. This is exactly where I'd expect to see hotspots from the profiler, and the overall performance of Lucene search is still stunning. Peter On Wed, Jun 24, 2009 at 10:13 AM, Peter Keegan wrote: >

Re: Common Bottlenecks

2009-06-24 Thread Peter Keegan
Our biggest bottleneck in searching is in a custom scorer which calls AllTermDocs.next() very frequently. This class uses Lucene's own BitVector, which I think is already highly optimized. Farther down in the list are DocSetHitCollector.collect() and FieldSortedQueue.insert(). For indexing, the mai

[jira] Updated: (LUCENE-1341) BoostingNearQuery class (prototype)

2009-04-23 Thread Peter Keegan (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-1341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Keegan updated LUCENE-1341: - Attachment: lucene-1341-new-1.patch As I was debugging a unit test for BoostingNearQuery, I

[jira] Updated: (LUCENE-1341) BoostingNearQuery class (prototype)

2009-02-11 Thread Peter Keegan (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-1341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Keegan updated LUCENE-1341: - Attachment: LUCENE-1341-new.patch Here is an updated patch for the 2.4 branch. It's 6 m

Re: [jira] Resolved: (LUCENE-1316) Avoidable synchronization bottleneck in MatchAlldocsQuery$MatchAllScorer

2009-02-07 Thread Peter Keegan
< luc...@mikemccandless.com> wrote: > > Hi Peter, > > You'll definitely need to start from the full patch in LUCENE-1316, which > fixes SegmentReader & Multi*Reader to use AllTermDocs when given a null > Term. > > Mike > > > Peter Keegan wrote: > > Hi Mike

Re: [jira] Resolved: (LUCENE-1316) Avoidable synchronization bottleneck in MatchAlldocsQuery$MatchAllScorer

2009-02-07 Thread Peter Keegan
(IndexSearcher.java:146) at org.apache.lucene.search.Searcher.search(Searcher.java:118) Peter On Fri, Feb 6, 2009 at 7:13 PM, Michael McCandless < luc...@mikemccandless.com> wrote: > > OK I just put the patch on there -- let me know. Thanks, > > Mike > > > Peter Keeg

Re: [jira] Resolved: (LUCENE-1316) Avoidable synchronization bottleneck in MatchAlldocsQuery$MatchAllScorer

2009-02-06 Thread Peter Keegan
Thanks, Mike. I'll be happy to help test the patch. Peter On Fri, Feb 6, 2009 at 6:44 PM, Michael McCandless < luc...@mikemccandless.com> wrote: > > OK I created https://issues.apache.org/jira/browse/LUCENE-1538. > > Mike > > > Peter Keegan wrote: > > I

Re: [jira] Resolved: (LUCENE-1316) Avoidable synchronization bottleneck in MatchAlldocsQuery$MatchAllScorer

2009-02-06 Thread Peter Keegan
I just ran into the same bottleneck with ValueSourceScorer. Here's a stack trace: Name: QueryThread group 1,#4 State: BLOCKED on org.apache.lucene.index.segmentrea...@49d7fb83 owned by: QueryThread group 1,#8 Total blocked: 1,535,881 Total waited: 1,080 Stack trace: org.apache.lucene.index.Segme

[jira] Updated: (LUCENE-1341) BoostingNearQuery class (prototype)

2008-07-21 Thread Peter Keegan (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-1341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Keegan updated LUCENE-1341: - Attachment: BoostingNearQuery.java bnq.patch Here is version of patch for Java

[jira] Commented: (LUCENE-1341) BoostingNearQuery class (prototype)

2008-07-19 Thread Peter Keegan (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-1341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12615004#action_12615004 ] Peter Keegan commented on LUCENE-1341: -- Note that this patch requires java 1.

[jira] Updated: (LUCENE-1341) BoostingNearQuery class (prototype)

2008-07-19 Thread Peter Keegan (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-1341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Keegan updated LUCENE-1341: - Attachment: BoostingNearQuery.java bnq.patch > BoostingNearQuery cl

[jira] Created: (LUCENE-1341) BoostingNearQuery class (prototype)

2008-07-19 Thread Peter Keegan (JIRA)
Versions: 2.3.1 Reporter: Peter Keegan Priority: Minor Fix For: 2.3.2 This patch implements term boosting for SpanNearQuery. Refer to: http://www.gossamer-threads.com/lists/lucene/java-user/62779 This patch works but probably needs more work. I don't lik

Re: How to delete multiple documents

2008-02-21 Thread Peter Keegan
srinivas, Create your IndexWriter and Analyzer once before your loop. I suggest you follow up any questions to the 'java-user' list. Peter On Thu, 21 Feb 2008 13:14:21 -00, [EMAIL PROTECTED] <[EMAIL PROTECTED]> wrote: > > In the Following Code if i insert one document i was > able to delete but

Re: Lucene 2.3 RC 1 available for testing

2008-01-09 Thread Peter Keegan
I've built a production index with rc1 and blasted it with lots of concurrent queries and I've seen no problems. I plan to do a lot more development and testing with 2.3 in the next few months. Peter On Jan 7, 2008 10:27 PM, Michael Busch <[EMAIL PROTECTED]> wrote: > Hi all, > > I just created t

[jira] Updated: (LUCENE-1093) SpanFirstQuery modification to aid term boosting based on position.

2007-12-27 Thread Peter Keegan (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-1093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Keegan updated LUCENE-1093: - Attachment: TestBasics20071227.patch Here is a patch to 'TestBasics' that adds a tes

[jira] Created: (LUCENE-1093) SpanFirstQuery modification to aid term boosting based on position.

2007-12-18 Thread Peter Keegan (JIRA)
Type: Improvement Components: Query/Scoring Reporter: Peter Keegan This is a request for a modification to SpanFirstQuery that would allow term boosting based on a term's distance from the beginning of the document. This modification to SpanFirstQuery would be that the

[jira] Created: (LUCENE-1088) PriorityQueue 'wouldBeInserted' method

2007-12-11 Thread Peter Keegan (JIRA)
nts: Other Reporter: Peter Keegan This is a request for a new method in PriorityQueue public boolean wouldBeInserted(Object element) // returns true if doc would be inserted, without inserting This would allow an application to prevent duplicate entries from being added to the queue. Here is a re

Re: Performance Improvement for Search using PriorityQueue

2007-12-11 Thread Peter Keegan
See my similar request from last March: http://www.nabble.com/FieldSortedHitQueue-enhancement-to9733550.html#a9733550 Peter On Dec 11, 2007 11:54 AM, Nadav Har'El <[EMAIL PROTECTED]> wrote: > On Mon, Dec 10, 2007, Shai Erera wrote about "Performance Improvement for > Search using PriorityQueue":

[jira] Commented: (LUCENE-1017) BoostingTermQuery performance

2007-12-10 Thread Peter Keegan (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-1017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12550238 ] Peter Keegan commented on LUCENE-1017: -- > What's the use case? Is there something that isn't pos

[jira] Commented: (LUCENE-1017) BoostingTermQuery performance

2007-12-10 Thread Peter Keegan (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-1017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12550203 ] Peter Keegan commented on LUCENE-1017: -- Grant, Unfortunately, my performance test bed isn't suitabl

[jira] Updated: (LUCENE-1017) BoostingTermQuery performance

2007-10-04 Thread Peter Keegan (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-1017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Keegan updated LUCENE-1017: - Attachment: BoostingTermQuery.patch Here is new version that traverses the term positions

[jira] Commented: (LUCENE-1017) BoostingTermQuery performance

2007-10-03 Thread Peter Keegan (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-1017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12532094 ] Peter Keegan commented on LUCENE-1017: -- Grant, You are right about it not handling multiple payloads per

[jira] Updated: (LUCENE-1017) BoostingTermQuery performance

2007-10-02 Thread Peter Keegan (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-1017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Keegan updated LUCENE-1017: - Attachment: termquery.patch Changes to TermQuery and TermScorer for previous patch

[jira] Updated: (LUCENE-1017) BoostingTermQuery performance

2007-10-02 Thread Peter Keegan (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-1017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Keegan updated LUCENE-1017: - Attachment: BoostingTermQuery.java Suggested change to BoostingTermQuery to extend TermQuery

[jira] Created: (LUCENE-1017) BoostingTermQuery performance

2007-10-02 Thread Peter Keegan (JIRA)
Environment: all Reporter: Peter Keegan I have been experimenting with payloads and BoostingTermQuery, which I think are excellent additions to Lucene core. Currently, BoostingTermQuery extends SpanQuery. I would suggest changing this class to extend TermQuery and refactor the

[jira] Commented: (LUCENE-991) BoostingTermQuery.explain() bugs

2007-09-07 Thread Peter Keegan (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12525734 ] Peter Keegan commented on LUCENE-991: - Confirmed - thanks. > BoostingTermQuery.explain() b

[jira] Commented: (LUCENE-991) BoostingTermQuery.explain() bugs

2007-09-07 Thread Peter Keegan (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12525709 ] Peter Keegan commented on LUCENE-991: - Hi Grant, > TopDocs hits = searcher.search(query, null,

[jira] Updated: (LUCENE-991) BoostingTermQuery.explain() bugs

2007-08-31 Thread Peter Keegan (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Keegan updated LUCENE-991: Attachment: TestBoostingTermQuery.patch Added 'testNoPayload' > BoostingTermQuery.e

[jira] Created: (LUCENE-991) BoostingTermQuery.explain() bugs

2007-08-31 Thread Peter Keegan (JIRA)
Reporter: Peter Keegan Priority: Minor Attachments: TestBoostingTermQuery.patch There are a couple of minor bugs in BoostingTermQuery.explain(). 1. The computation of average payload score produces NaN if no payloads were found. It should probably be: float avgPayloadScore

Re: Best Practices for getting Strings from a position range

2007-08-15 Thread Peter Keegan
you comparing, > etc.? Sample queries would be good. I would like to write up a > contrib/Benchmark algorithm to begin investigating this and see if > there is anything that can be done. > > Thanks, > Grant > > On Aug 10, 2007, at 6:27 PM, Peter Keegan wrote: > &g

Re: Best Practices for getting Strings from a position range

2007-08-10 Thread Peter Keegan
ught you just wanted access to the > term info per position.I think we will have to add something to > the Spans like we talked about before. > > -Grant > > On Aug 10, 2007, at 11:03 AM, Peter Keegan wrote: > > > Grant, > > > > I'm afraid I don't

Re: MMapDirectory is broken in 2.2

2007-08-10 Thread Peter Keegan
Hi Mike, I tested the patch and it looks good. Thanks, Peter On 8/10/07, Peter Keegan <[EMAIL PROTECTED]> wrote: > > Heck, no need to apologize - no harm done. For all the great work by you > and others, the least I can do is help find problems ;-) > > Peter > > On

Re: MMapDirectory is broken in 2.2

2007-08-10 Thread Peter Keegan
Heck, no need to apologize - no harm done. For all the great work by you and others, the least I can do is help find problems ;-) Peter On 8/10/07, Michael McCandless <[EMAIL PROTECTED]> wrote: > > > OK, sorry about this :( I will fix. > > Mike > > "Peter K

Re: MMapDirectory is broken in 2.2

2007-08-10 Thread Peter Keegan
gnore the bufferSize? > > I will open a Jira issue. Thanks for catching this! > > Mike > > "Peter Keegan" <[EMAIL PROTECTED]> wrote: > > I have discovered a bug in 2.2 (trunk version) that prevents > > MMapDirectory > > from working. TermInfosRe

MMapDirectory is broken in 2.2

2007-08-10 Thread Peter Keegan
I have discovered a bug in 2.2 (trunk version) that prevents MMapDirectory from working. TermInfosReader has a new constructor that calls a new version of FSDirectory.openInput(String, int). This new openInput signature isn't implemented in MMapDirectory, so FSDirectory is used instead. Should I s

Re: Best Practices for getting Strings from a position range

2007-08-10 Thread Peter Keegan
modifying Spans, could you provide a bit more detail? Thanks, Peter On 8/9/07, Peter Keegan <[EMAIL PROTECTED]> wrote: > > Hi Grant, > > I'm hoping to check this out soon. > > Thanks, > Peter > > On 8/7/07, Grant Ingersoll <[EMAIL PROTECTED] > wrote: &g

Re: Best Practices for getting Strings from a position range

2007-08-09 Thread Peter Keegan
what ideally what you want, but I haven't had time to scope > that one out yet., > > -Grant > > On Jul 24, 2007, at 6:02 PM, Peter Keegan wrote: > > > Hi Grant, > > > > No problem - I know you are very busy. I just wanted to get a > > sense for the &g

Re: Best Practices for getting Strings from a position range

2007-07-24 Thread Peter Keegan
at the match issue. Maybe next week... Cheers, Grant On Jul 23, 2007, at 8:51 AM, Peter Keegan wrote: > Any idea on when this might be available (days, weeks...)? > > Peter > > On 7/16/07, Grant Ingersoll <[EMAIL PROTECTED]> wrote: >> >> >> On Jul 16, 2007, a

Re: Best Practices for getting Strings from a position range

2007-07-23 Thread Peter Keegan
Any idea on when this might be available (days, weeks...)? Peter On 7/16/07, Grant Ingersoll <[EMAIL PROTECTED]> wrote: On Jul 16, 2007, at 1:06 AM, Chris Hostetter wrote: > > : Do we have a best practice for going from, say a SpanQuery doc/ > : position information and retrieving the actual

Re: Post mortem kudos for (LUCENE-843) :)

2007-07-17 Thread Peter Keegan
I did some performance comparison testing of Lucene 2.0 vs. trunk (with LUCENE-843). I'm seeing at least a 4X increase in indexing rate with the new DocumentsWriter in LUCENE-843 (still doing single-threaded indexing). Better yet, the total time to build the index is much shorter because I can now

Re: Post mortem kudos for (LUCENE-843) :)

2007-07-13 Thread Peter Keegan
I see you have a 64-bit OS but 32-bit JVM. Have you tried a 64-bit JVM (with a bigger RAM buffer)? Peter On 7/13/07, eks dev <[EMAIL PROTECTED]> wrote: >Interesting. This matches the experience Doron had where adding more >RAM actually slowed things down a bit (posted to >LUCENE-843). I kno

Re: Caching in QueryFilter - why?

2007-04-06 Thread Peter Keegan
this seems like a potentially big surprise for people when upgrading ... old code will continue to work fine without warning, just get a lot less efficient. I agree. This would certainly be a big surprise to anyone not following this forum closely (like me - I just happened to stumble upon this

Re: Monster lucene geosearch

2007-03-16 Thread Peter Keegan
I will move/respond to this to an existing java-user thread. Peter On 3/16/07, Eric Cone <[EMAIL PROTECTED]> wrote: Hello Peter, Now that the monster lucene search is live, is performance pretty good? Are you still running it on a single 8 core server? Can you give me a rough idea on the numbe

Re: bad queryparser bug

2007-02-01 Thread Peter Keegan
/1/07, Peter Keegan <[EMAIL PROTECTED]> wrote: Correction: The query parser produces the correct query with the parenthesis. But, I'm still looking for a fix for this. I could use some advice on where to look in QueryParser to fix this. Thanks, Peter On 2/1/07, Peter Keegan <[EMAI

Re: bad queryparser bug

2007-02-01 Thread Peter Keegan
Correction: The query parser produces the correct query with the parenthesis. But, I'm still looking for a fix for this. I could use some advice on where to look in QueryParser to fix this. Thanks, Peter On 2/1/07, Peter Keegan <[EMAIL PROTECTED]> wrote: I have discovered a ser

bad queryparser bug

2007-02-01 Thread Peter Keegan
I have discovered a serious bug in QueryParser. The following query: contents:sales && contents:marketing || contents:industrial && contents:sales is parsed as: +contents:sales +contents:marketing +contents:industrial +contents:sales The same parsed query occurs even with parenthesis: (contents:

[jira] Commented: (LUCENE-693) ConjunctionScorer - more tuneup

2006-10-24 Thread Peter Keegan (JIRA)
[ http://issues.apache.org/jira/browse/LUCENE-693?page=comments#action_1236 ] Peter Keegan commented on LUCENE-693: - fwiw, my tests were done using 'real world' queries and index. Most queries have several required clauses.

[jira] Commented: (LUCENE-693) ConjunctionScorer - more tuneup

2006-10-24 Thread Peter Keegan (JIRA)
[ http://issues.apache.org/jira/browse/LUCENE-693?page=comments#action_1208 ] Peter Keegan commented on LUCENE-693: - Well, I'm seeing a good 7% increase over the trunk version. Conjunction scorer time is mostly in 'skipto&#

[jira] Commented: (LUCENE-693) ConjunctionScorer - more tuneup

2006-10-24 Thread Peter Keegan (JIRA)
[ http://issues.apache.org/jira/browse/LUCENE-693?page=comments#action_12444317 ] Peter Keegan commented on LUCENE-693: - Yonik, I tried out your patch, but it causes an exception on some boolean queries. This one occurred on a boolean query

Re: [jira] Resolved: (LUCENE-443) ConjunctionScorer tune-up

2006-10-23 Thread Peter Keegan
Can you open a JIRA issue for this? Yes, it's: http://issues.apache.org/jira/browse/LUCENE-693 Peter

Re: [jira] Resolved: (LUCENE-443) ConjunctionScorer tune-up

2006-10-23 Thread Peter Keegan
ter On 10/23/06, Paul Elschot <[EMAIL PROTECTED]> wrote: On Monday 23 October 2006 22:12, Peter Keegan wrote: > I did some profile testing with the new ConjuctionScorer in 2.1 and > discovered a new bottleneck in ConjunctionScorer.sortScorers. The > java.utils.Arrays.sort method is

Re: [jira] Resolved: (LUCENE-443) ConjunctionScorer tune-up

2006-10-23 Thread Peter Keegan
Huh... that's interesting. I wonder why Arrays.sort(int[]) is all in-place but sort(Object[]) is not. I was wondering that myself. Here's the code: public static void sort(T[] a, Comparator c) { T[] aux = (T[])a.clone(); if (c==null) mergeSort(aux, a, 0, a.length, 0);

[jira] Created: (LUCENE-693) ConjunctionScorer - more tuneup

2006-10-23 Thread Peter Keegan (JIRA)
Environment: Windows Server 2003 x64, Java 1.6, pretty large index Reporter: Peter Keegan (See also: #LUCENE-443) I did some profile testing with the new ConjuctionScorer in 2.1 and discovered a new bottleneck in ConjunctionScorer.sortScorers. The java.utils.Arrays.sort method is cloning

Re: [jira] Resolved: (LUCENE-443) ConjunctionScorer tune-up

2006-10-23 Thread Peter Keegan
I did some profile testing with the new ConjuctionScorer in 2.1 and discovered a new bottleneck in ConjunctionScorer.sortScorers. The java.utils.Arrays.sort method is cloning the Scorers array on every sort, which is quite expensive on large indexes because of the size of the 'norms' array within,

Re: Custom sorting - memory leaks

2006-08-21 Thread Peter Keegan
I forgot to mention that the query threads are created only once. Peter On 8/21/06, Peter Keegan <[EMAIL PROTECTED]> wrote: Aleksey, My app has precise and predictable control over when the index reader/searcher gets refreshed. When this occurs, the sorting values for all the docs in t

Re: Custom sorting - memory leaks

2006-08-21 Thread Peter Keegan
ave different center point value then we still in enormous memory usage problem. Aleksey On 8/21/06, Peter Keegan <[EMAIL PROTECTED]> wrote: > I avoided the cache by implementing the equals and hashCode methods > appropriately in my custom ScoreDocComparator as described here: > http

Re: Custom sorting - memory leaks

2006-08-21 Thread Peter Keegan
I avoided the cache by implementing the equals and hashCode methods appropriately in my custom ScoreDocComparator as described here: http://www.lucenebook.com/blog/errata/2006/03/01/memory_leak.html Peter On 8/21/06, Aleksey Serba <[EMAIL PROTECTED]> wrote: Hi Chris, On 5/17/06, Peter

Re: Nio File Caching & Performance Test

2006-06-01 Thread Peter Keegan
My search process is using MMapDirectory on a read-only index via: -Dorg.apache.lucene.FSDirectory.class=org.apache.lucene.store.MMapDirectory Another indexing process is building the next version of the index in a different directory. When it's time to switch to the new index, the search proces

Re: FieldsReader synchronized access vs. ThreadLocal ?

2006-05-19 Thread Peter Keegan
high, but it helps in measuring the limits of the system. Peter On 5/19/06, Yonik Seeley <[EMAIL PROTECTED]> wrote: On 5/19/06, Peter Keegan <[EMAIL PROTECTED]> wrote: > The client test program blasts queries from >50 threads over a socket > and runs on a separate server f

Re: FieldsReader synchronized access vs. ThreadLocal ?

2006-05-19 Thread Peter Keegan
gh we are comparing apples and oranges, unless you are performing some really expensive queries, I would expect your configuration to be MUCH faster. -Original Message- From: Peter Keegan [mailto:[EMAIL PROTECTED] Sent: Thursday, May 18, 2006 1:32 PM To: java-dev@lucene.apache.org; [EMAIL

Re: FieldsReader synchronized access vs. ThreadLocal ?

2006-05-18 Thread Peter Keegan
some of the CPUs were far less than 100%, and others were at 100% may be a good sign. How any query threads were you testing with? -Original Message- From: Peter Keegan [mailto:[EMAIL PROTECTED] Sent: Thursday, May 18, 2006 1:01 PM To: java-dev@lucene.apache.org; [EMAIL PROTECTED]

Re: FieldsReader synchronized access vs. ThreadLocal ?

2006-05-18 Thread Peter Keegan
Robert, Sorry I missed your questions. The test results seem hard to believe. Doubling the CPUs only increased through put by 20%??? Seems rather low for primarily a "read only" test. I think this refers to the test I did on a 16 cpu (32 hyperthreaded) server. This system was actually two 8