[jira] Commented: (LUCENE-1483) Change IndexSearcher to use MultiSearcher semantics for multiple subreaders

2008-12-23 Thread Michael McCandless (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-1483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12658810#action_12658810 ] Michael McCandless commented on LUCENE-1483: {quote} > In the case where 3 hit

[jira] Updated: (LUCENE-1483) Change IndexSearcher to use MultiSearcher semantics for multiple subreaders

2008-12-23 Thread Michael McCandless (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-1483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael McCandless updated LUCENE-1483: --- Attachment: LUCENE-1483.patch New patch attached: * I moved maxScore tracking up

Re: [jira] Commented: (LUCENE-1483) Change IndexSearcher to use MultiSearcher semantics for multiple subreaders

2008-12-23 Thread Mark Miller
Michael McCandless (JIRA) wrote: Well, it's tricky... and indeed all tests pass with the bug (which is spooky -- I think we need to add cases to TestSort where 1) the index has many segments, and 2) the number of hits is much greater than the queue size), but I'm pretty sure it's a bug Its a nast

[jira] Commented: (LUCENE-1483) Change IndexSearcher to use MultiSearcher semantics for multiple subreaders

2008-12-23 Thread Mark Miller (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-1483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12658837#action_12658837 ] Mark Miller commented on LUCENE-1483: - Just as a reminder to myself - we also need a c

[jira] Updated: (LUCENE-1483) Change IndexSearcher to use MultiSearcher semantics for multiple subreaders

2008-12-23 Thread Michael McCandless (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-1483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael McCandless updated LUCENE-1483: --- Attachment: LUCENE-1483.patch {quote} > If we changed to to track the maxSlot it sho

[jira] Commented: (LUCENE-1483) Change IndexSearcher to use MultiSearcher semantics for multiple subreaders

2008-12-23 Thread Michael McCandless (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-1483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12658843#action_12658843 ] Michael McCandless commented on LUCENE-1483: Current results with topN=1000:

Re: [jira] Commented: (LUCENE-1483) Change IndexSearcher to use MultiSearcher semantics for multiple subreaders

2008-12-23 Thread Mark Miller
Mark Miller wrote: Michael McCandless (JIRA) wrote: Well, it's tricky... and indeed all tests pass with the bug (which is spooky -- I think we need to add cases to TestSort where 1) the index has many segments, and 2) the number of hits is much greater than the queue size), but I'm pretty sure i

Re: [jira] Commented: (LUCENE-1483) Change IndexSearcher to use MultiSearcher semantics for multiple subreaders

2008-12-23 Thread Michael McCandless
Mark Miller wrote: Mark Miller wrote: Michael McCandless (JIRA) wrote: Well, it's tricky... and indeed all tests pass with the bug (which is spooky -- I think we need to add cases to TestSort where 1) the index has many segments, and 2) the number of hits is much greater than the queue

Re: [jira] Commented: (LUCENE-1483) Change IndexSearcher to use MultiSearcher semantics for multiple subreaders

2008-12-23 Thread Mark Miller
Michael McCandless wrote: Mark Miller wrote: Mark Miller wrote: Michael McCandless (JIRA) wrote: Well, it's tricky... and indeed all tests pass with the bug (which is spooky -- I think we need to add cases to TestSort where 1) the index has many segments, and 2) the number of hits is much gr

[jira] Commented: (LUCENE-1314) IndexReader.clone

2008-12-23 Thread Michael McCandless (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-1314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12658863#action_12658863 ] Michael McCandless commented on LUCENE-1314: {quote} > I request assistance in

[jira] Updated: (LUCENE-1314) IndexReader.clone

2008-12-23 Thread Jason Rutherglen (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-1314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Rutherglen updated LUCENE-1314: - Attachment: LUCENE-1314.patch LUCENE-1314.patch - Updated to work with trunk - Cleaned

[jira] Commented: (LUCENE-1314) IndexReader.clone

2008-12-23 Thread Michael McCandless (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-1314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12658917#action_12658917 ] Michael McCandless commented on LUCENE-1314: {quote} > The clone method is syn

[jira] Commented: (LUCENE-1314) IndexReader.clone

2008-12-23 Thread Jason Rutherglen (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-1314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12658921#action_12658921 ] Jason Rutherglen commented on LUCENE-1314: -- In the case of a reader that is alrea

[jira] Updated: (LUCENE-1314) IndexReader.clone

2008-12-23 Thread Jason Rutherglen (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-1314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Rutherglen updated LUCENE-1314: - Attachment: LUCENE-1314.patch LUCENE-1314.patch - DirectoryIndexReader.doReopen acquire

Realtime Search

2008-12-23 Thread Jason Rutherglen
We've discussed realtime search before, it looks like after the next release we can get some sort of realtime search working. I was going to open a new issue but decided it might be best to discuss realtime search on the dev list. Lucene can implement realtime search as the ability to add, update

Re: Realtime Search

2008-12-23 Thread Marvin Humphrey
On Tue, Dec 23, 2008 at 05:51:43PM -0800, Jason Rutherglen wrote: > Are there other implementation options? Here's the plan for Lucy/KS: 1) Design index formats that can be memory mapped rather than slurped, bringing the cost of opening/reopening an IndexReader down to a negligible l

Re: Realtime Search

2008-12-23 Thread robert engels
Is there something that I am missing? I see lots of references to using "memory mapped" files to "dramatically" improve performance. I don't think this is the case at all. At the lowest levels, it is somewhat more efficient from a CPU standpoint, but with a decent OS cache the IO performanc

Re: Realtime Search

2008-12-23 Thread Marvin Humphrey
On Tue, Dec 23, 2008 at 08:36:24PM -0600, robert engels wrote: > Is there something that I am missing? Yes. > I see lots of references to using "memory mapped" files to "dramatically" > improve performance. There have been substantial discussions about this design in JIRA, notably LUCENE-1458

Re: Realtime Search

2008-12-23 Thread robert engels
Seems doubtful you will be able to do this without increasing the index size dramatically. Since it will need to be stored "unpacked" (in order to have random access), yet the terms are variable length - leading to using a maximum=minimum size for every term. In the end I highly doubt it

Re: Realtime Search

2008-12-23 Thread robert engels
Also, if you are thinking that accessing the "buffer" directly will be faster than "parsing" the packed structure, I'm not so sure. You can review the source for the various buffers, and since the is no "struct" support in Java, you end up combining bytes to make longs, etc. Also, a lot of