[jira] Resolved: (LUCENE-691) Bob Carpenter's FuzzyTermEnum refactoring
[
https://issues.apache.org/jira/browse/LUCENE-691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Otis Gospodnetic resolved LUCENE-691.
-
Resolution: Duplicate
Assignee: Otis Gospodnetic
The patch for Bob's change suggestions is in LUCENE-1183, so this issue is
redundant.
> Bob Carpenter's FuzzyTermEnum refactoring
> -
>
> Key: LUCENE-691
> URL: https://issues.apache.org/jira/browse/LUCENE-691
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Search
>Reporter: Otis Gospodnetic
>Assignee: Otis Gospodnetic
>Priority: Minor
>
> I'll just paste Bob's complete email here.
> I refactored the org.apache.lucene.search.FuzzyTermEnum
> edit distance implementation. It now only uses a single
> pair of arrays, and those never get resized. That required
> turning the order of text/target around in the loops. You'll
> see that with the pair of arrays method, they get re-used
> hand-over-hand, and are assigned to local variables in the
> tight loops.
> I removed the calculation of min distance and replaced
> it with a boolean -- the min wasn't needed, only the test vs.
> the max. I also flipped some variables around so there's
> one less addition in the very inner loop and the arrays are
> now looping in the ordinary way (starting at 0 with a < comparison).
> I also commented out the redundant definition of the public close()
> [which just called super.close() and had none of its own doc.]
> I also just compute the max distance each time rather than
> fiddling with an array -- it's just a little arithmetic done once
> per term, but that could be put back.
> I also rewrote min(int,int,int) to get rid of intermediate
> assignments. Is there a lib for this kind of thing?
> An intermediate refactoring that does the hand-over-hand
> with the existing array and resizing strategy is in
> FuzzyTermEnum.intermed.java.
> I ran the unit tests as follows on both versions (my hat's off to the
> build.xml author(s) -- this all just worked out of the box and was
> really easy to follow the first through):
> C:\java\lucene-2.0.0>ant -Djunit.includes="" -Dtestcase=TestFuzzyQuery test
> Buildfile: build.xml
> javacc-uptodate-check:
> javacc-notice:
> init:
> common.compile-core:
> [javac] Compiling 1 source file to
> C:\java\lucene-2.0.0\build\classes\java
> compile-core:
> compile-demo:
> common.compile-test:
> compile-test:
> test:
> [junit] Testsuite: org.apache.lucene.search.TestFuzzyQuery
> [junit] Tests run: 2, Failures: 0, Errors: 0, Time elapsed: 0.453 sec
> BUILD SUCCESSFUL
> Total time: 2 seconds
> Does anyone have regression/performance test harnesses?
> The unit tests were pretty minimal (which is a good thing!).
> It'd be nice to test the min impl (ternary vs. if/then)
> and the assumption about not allocating an
> array of max distances. All told, the refactored version
> should be a modest speed improvement, mainly from
> unfolding the arrays so they're local one-dimensional refs.
> I don't know what the protocol is for one-off contributions.
> I'm happy with the Apache license, so that shouldn't
> be a problem. I also don't know whether you use tabs
> or spaces -- I untabified the final version and used your
> two-space format in emacs.
> - Bob Carpenter
> package org.apache.lucene.search;
> /**
> * Copyright 2004 The Apache Software Foundation
> *
> * Licensed under the Apache License, Version 2.0 (the "License");
> * you may not use this file except in compliance with the License.
> * You may obtain a copy of the License at
> *
> * http://www.apache.org/licenses/LICENSE-2.0
> *
> * Unless required by applicable law or agreed to in writing, software
> * distributed under the License is distributed on an "AS IS" BASIS,
> * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
> * See the License for the specific language governing permissions and
> * limitations under the License.
> */
> import org.apache.lucene.index.IndexReader;
> import org.apache.lucene.index.Term;
> import java.io.IOException;
> /** Subclass of FilteredTermEnum for enumerating all terms that are similiar
> * to the specified filter term.
> *
> * Term enumerations are always ordered by Term.compareTo(). Each term in
> * the enumeration is greater than all that precede it.
> */
> public final class FuzzyTermEnum extends FilteredTermEnum {
> /* This should be somewhere around the average long word.
>* If it is longer, we waste time and space. If it is shorter, we waste a
>* little bit of time growing the array as we encounter longer words.
>*/
> private static final int TYPICAL_LONGEST_WORD_IN_INDEX = 19;
> /* Allows us save time required to create a new array
>* everytime similarity is called. These are slices that
>* will be reused during dynamic programm
[jira] Commented: (LUCENE-954) Toggle score normalization in Hits
[ https://issues.apache.org/jira/browse/LUCENE-954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12600366#action_12600366 ] Otis Gospodnetic commented on LUCENE-954: - I suppose there is now suddenly no need to work on Hits. I'll resolve this as Won't Fix in a few days, unless somebody has some more thoughts on this. > Toggle score normalization in Hits > -- > > Key: LUCENE-954 > URL: https://issues.apache.org/jira/browse/LUCENE-954 > Project: Lucene - Java > Issue Type: Improvement > Components: Search >Affects Versions: 2.2, 2.3, 2.3.1, 2.4 > Environment: any >Reporter: Christian Kohlschütter >Assignee: Otis Gospodnetic > Fix For: 2.4 > > Attachments: hits-scoreNorm.patch, LUCENE-954.patch > > > The current implementation of the "Hits" class sometimes performs score > normalization. > In particular, whenever the top-ranked score is bigger than 1.0, it is > normalized to a maximum of 1.0. > In this case, Hits may return different score results than TopDocs-based > methods. > In my scenario (a federated search system), Hits delievered just plain wrong > results. > I was merging results from several sources, all having homogeneous statistics > (similar to MultiSearcher, but over the Internet using HTTP/XML-based > protocols). > Sometimes, some of the sources had a top-score greater than 1, so I ended up > with garbled results. > I suggest to add a switch to enable/disable this score-normalization at > runtime. > My patch (attached) has an additional peformance benefit, since score > normalization now occurs only when Hits#score() is called, not when creating > the Hits result list. Whenever scores are not required, you save one > multiplication per retrieved hit (i.e., at least 100 multiplications with the > current implementation of Hits). -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Updated: (LUCENE-1026) Provide a simple way to concurrently access a Lucene index from multiple threads
[ https://issues.apache.org/jira/browse/LUCENE-1026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Miller updated LUCENE-1026: Attachment: IndexAccessor-05.27.2008.zip Well I changed too much tonight to consider it a 1.0 type code dump (beyond a bunch of changes that I was waiting to release). Every time I look at this thing I feel the need to gut more of the original LuceneIndexAccessor stuff. I think all of the similarity stuff is actually totally unnecessary, which means no more searcherCache. Also, no real need for a Searcher reader and a Reading reader. So I have stripped things down a bit to just a little more than you'd need (still nice to be able to work by Searcher if you want to). Feedback most welcome. > Provide a simple way to concurrently access a Lucene index from multiple > threads > > > Key: LUCENE-1026 > URL: https://issues.apache.org/jira/browse/LUCENE-1026 > Project: Lucene - Java > Issue Type: New Feature > Components: Index, Search >Reporter: Mark Miller >Priority: Minor > Attachments: DefaultIndexAccessor.java, > DefaultMultiIndexAccessor.java, IndexAccessor-02.04.2008.zip, > IndexAccessor-02.07.2008.zip, IndexAccessor-02.28.2008.zip, > IndexAccessor-05.27.2008.zip, IndexAccessor-1.26.2008.zip, > IndexAccessor-2.15.2008.zip, IndexAccessor.04032008.zip, IndexAccessor.java, > IndexAccessor.zip, IndexAccessorFactory.java, MultiIndexAccessor.java, > shai-IndexAccessor-2.zip, shai-IndexAccessor.zip, shai-IndexAccessor3.zip, > SimpleSearchServer.java, StopWatch.java, TestIndexAccessor.java > > > For building interactive indexes accessed through a network/internet > (multiple threads). > This builds upon the LuceneIndexAccessor patch. That patch was not very > newbie friendly and did not properly handle MultiSearchers (or at the least > made it easy to get into trouble). > This patch simplifies things and provides out of the box support for sharing > the IndexAccessors across threads. There is also a simple test class and > example SearchServer to get you started. > Future revisions will be zipped. > Works pretty solid as is, but could use the ability to warm new Searchers. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Updated: (LUCENE-1026) Provide a simple way to concurrently access a Lucene index from multiple threads
[ https://issues.apache.org/jira/browse/LUCENE-1026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Miller updated LUCENE-1026: Comment: was deleted > Provide a simple way to concurrently access a Lucene index from multiple > threads > > > Key: LUCENE-1026 > URL: https://issues.apache.org/jira/browse/LUCENE-1026 > Project: Lucene - Java > Issue Type: New Feature > Components: Index, Search >Reporter: Mark Miller >Priority: Minor > Attachments: DefaultIndexAccessor.java, > DefaultMultiIndexAccessor.java, IndexAccessor-02.04.2008.zip, > IndexAccessor-02.07.2008.zip, IndexAccessor-02.28.2008.zip, > IndexAccessor-1.26.2008.zip, IndexAccessor-2.15.2008.zip, > IndexAccessor.04032008.zip, IndexAccessor.java, IndexAccessor.zip, > IndexAccessorFactory.java, MultiIndexAccessor.java, shai-IndexAccessor-2.zip, > shai-IndexAccessor.zip, shai-IndexAccessor3.zip, SimpleSearchServer.java, > StopWatch.java, TestIndexAccessor.java > > > For building interactive indexes accessed through a network/internet > (multiple threads). > This builds upon the LuceneIndexAccessor patch. That patch was not very > newbie friendly and did not properly handle MultiSearchers (or at the least > made it easy to get into trouble). > This patch simplifies things and provides out of the box support for sharing > the IndexAccessors across threads. There is also a simple test class and > example SearchServer to get you started. > Future revisions will be zipped. > Works pretty solid as is, but could use the ability to warm new Searchers. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Commented: (LUCENE-1026) Provide a simple way to concurrently access a Lucene index from multiple threads
[
https://issues.apache.org/jira/browse/LUCENE-1026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12600274#action_12600274
]
Sean Timm commented on LUCENE-1026:
---
bq. 2. Logger.isLoggable is supposed to be faster than logging at a level thats
not turned on.
That is certainly true if you are creating a string from an object, or
concatenating strings. I don't believe that is true for static strings.
E.g., {code}logger.fine("closing cached reading reader");{code} you might as
well skip the Logger.isLoggable. In the case {code}logger.fine("closing cached
searchers (" + cachedSearchers.size() + ")");{code} it is certainly better to
check the log level first.
> Provide a simple way to concurrently access a Lucene index from multiple
> threads
>
>
> Key: LUCENE-1026
> URL: https://issues.apache.org/jira/browse/LUCENE-1026
> Project: Lucene - Java
> Issue Type: New Feature
> Components: Index, Search
>Reporter: Mark Miller
>Priority: Minor
> Attachments: DefaultIndexAccessor.java,
> DefaultMultiIndexAccessor.java, IndexAccessor-02.04.2008.zip,
> IndexAccessor-02.07.2008.zip, IndexAccessor-02.28.2008.zip,
> IndexAccessor-1.26.2008.zip, IndexAccessor-2.15.2008.zip,
> IndexAccessor.04032008.zip, IndexAccessor.java, IndexAccessor.zip,
> IndexAccessorFactory.java, MultiIndexAccessor.java, shai-IndexAccessor-2.zip,
> shai-IndexAccessor.zip, shai-IndexAccessor3.zip, SimpleSearchServer.java,
> StopWatch.java, TestIndexAccessor.java
>
>
> For building interactive indexes accessed through a network/internet
> (multiple threads).
> This builds upon the LuceneIndexAccessor patch. That patch was not very
> newbie friendly and did not properly handle MultiSearchers (or at the least
> made it easy to get into trouble).
> This patch simplifies things and provides out of the box support for sharing
> the IndexAccessors across threads. There is also a simple test class and
> example SearchServer to get you started.
> Future revisions will be zipped.
> Works pretty solid as is, but could use the ability to warm new Searchers.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
Re: svn commit: r660010 - in /lucene/java/site: docs/whoweare.html docs/whoweare.pdf src/documentation/content/xdocs/whoweare.xml
Michael Busch wrote: Welcome, Mark! :) -Michael Thanks Michael. Very happy to be aboard and ready to help in whatever limited ways I can. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Issue Comment Edited: (LUCENE-1026) Provide a simple way to concurrently access a Lucene index from multiple threads
[ https://issues.apache.org/jira/browse/LUCENE-1026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12600266#action_12600266 ] [EMAIL PROTECTED] edited comment on LUCENE-1026 at 5/27/08 1:45 PM: -- Bunch of new code coming on this tonight or tomorrow...I consider it basically the 1.0 version. 1. In my testing the ReentrantLock is much faster than the sync blocks under contention. 2. Logger.isLoggable is supposed to be faster than logging at a level thats not turned on. 3. The param is not for create, but for autoCommit. Create is called when you ask for an accessor that doesnt exist yet. 4. Why do you ask? I can't remember at the moment, but at first glance it looks like its unnecessary. It may have been done just because its done in getSearcher (where it is needed - or where it was needed before switching to reopen...need to double check this now). Ill investigate further and possibly take it out. The new code fixes a bunch of issues with the Reader handling. The Searcher handling works great though. was (Author: [EMAIL PROTECTED]): Bunch of new code coming on this tonight or tomorrow...I consider it basically the 1.0 version. 1. In my testing the ReentrantLock is much faster than the sync blocks under contention. 2. Logger.isLoggable is supposed to be faster than logging at a level thats not turned on. 3. The param is not for create, but for autoCommit. Create is called when you ask for an accessor that doesnt exist yet. 4. Why do you ask? I can't remember at the moment, but at first glance it looks like its unnecessary. It may have been done just because its done in getSearcher (where it is needed). Ill investigate further and possibly take it out. The new code fixes a bunch of issues with the Reader handling. The Searcher handling works great though. > Provide a simple way to concurrently access a Lucene index from multiple > threads > > > Key: LUCENE-1026 > URL: https://issues.apache.org/jira/browse/LUCENE-1026 > Project: Lucene - Java > Issue Type: New Feature > Components: Index, Search >Reporter: Mark Miller >Priority: Minor > Attachments: DefaultIndexAccessor.java, > DefaultMultiIndexAccessor.java, IndexAccessor-02.04.2008.zip, > IndexAccessor-02.07.2008.zip, IndexAccessor-02.28.2008.zip, > IndexAccessor-1.26.2008.zip, IndexAccessor-2.15.2008.zip, > IndexAccessor.04032008.zip, IndexAccessor.java, IndexAccessor.zip, > IndexAccessorFactory.java, MultiIndexAccessor.java, shai-IndexAccessor-2.zip, > shai-IndexAccessor.zip, shai-IndexAccessor3.zip, SimpleSearchServer.java, > StopWatch.java, TestIndexAccessor.java > > > For building interactive indexes accessed through a network/internet > (multiple threads). > This builds upon the LuceneIndexAccessor patch. That patch was not very > newbie friendly and did not properly handle MultiSearchers (or at the least > made it easy to get into trouble). > This patch simplifies things and provides out of the box support for sharing > the IndexAccessors across threads. There is also a simple test class and > example SearchServer to get you started. > Future revisions will be zipped. > Works pretty solid as is, but could use the ability to warm new Searchers. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Commented: (LUCENE-1026) Provide a simple way to concurrently access a Lucene index from multiple threads
[ https://issues.apache.org/jira/browse/LUCENE-1026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12600270#action_12600270 ] Mark Miller commented on LUCENE-1026: - Bunch of new code coming on this tonight or tomorrow...I consider it basically the 1.0 version. 1. In my testing the ReentrantLock is much faster than the sync blocks under contention. 2. Logger.isLoggable is supposed to be faster than logging at a level thats not turned on. 3. The param is not for create, but for autoCommit. Create is called when you ask for an accessor that doesnt exist yet. 4. Why do you ask? I can't remember at the moment, but at first glance it looks like its unnecessary. It may have been done just because its done in getSearcher (where it is needed). Ill investigate further and possibly take it out. The new code fixes a bunch of issues with the Reader handling. The Searcher handling works great though. > Provide a simple way to concurrently access a Lucene index from multiple > threads > > > Key: LUCENE-1026 > URL: https://issues.apache.org/jira/browse/LUCENE-1026 > Project: Lucene - Java > Issue Type: New Feature > Components: Index, Search >Reporter: Mark Miller >Priority: Minor > Attachments: DefaultIndexAccessor.java, > DefaultMultiIndexAccessor.java, IndexAccessor-02.04.2008.zip, > IndexAccessor-02.07.2008.zip, IndexAccessor-02.28.2008.zip, > IndexAccessor-1.26.2008.zip, IndexAccessor-2.15.2008.zip, > IndexAccessor.04032008.zip, IndexAccessor.java, IndexAccessor.zip, > IndexAccessorFactory.java, MultiIndexAccessor.java, shai-IndexAccessor-2.zip, > shai-IndexAccessor.zip, shai-IndexAccessor3.zip, SimpleSearchServer.java, > StopWatch.java, TestIndexAccessor.java > > > For building interactive indexes accessed through a network/internet > (multiple threads). > This builds upon the LuceneIndexAccessor patch. That patch was not very > newbie friendly and did not properly handle MultiSearchers (or at the least > made it easy to get into trouble). > This patch simplifies things and provides out of the box support for sharing > the IndexAccessors across threads. There is also a simple test class and > example SearchServer to get you started. > Future revisions will be zipped. > Works pretty solid as is, but could use the ability to warm new Searchers. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Commented: (LUCENE-1026) Provide a simple way to concurrently access a Lucene index from multiple threads
[ https://issues.apache.org/jira/browse/LUCENE-1026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12600266#action_12600266 ] Mark Miller commented on LUCENE-1026: - Bunch of new code coming on this tonight or tomorrow...I consider it basically the 1.0 version. 1. In my testing the ReentrantLock is much faster than the sync blocks under contention. 2. Logger.isLoggable is supposed to be faster than logging at a level thats not turned on. 3. The param is not for create, but for autoCommit. Create is called when you ask for an accessor that doesnt exist yet. 4. Why do you ask? I can't remember at the moment, but at first glance it looks like its unnecessary. It may have been done just because its done in getSearcher (where it is needed). Ill investigate further and possibly take it out. The new code fixes a bunch of issues with the Reader handling. The Searcher handling works great though. > Provide a simple way to concurrently access a Lucene index from multiple > threads > > > Key: LUCENE-1026 > URL: https://issues.apache.org/jira/browse/LUCENE-1026 > Project: Lucene - Java > Issue Type: New Feature > Components: Index, Search >Reporter: Mark Miller >Priority: Minor > Attachments: DefaultIndexAccessor.java, > DefaultMultiIndexAccessor.java, IndexAccessor-02.04.2008.zip, > IndexAccessor-02.07.2008.zip, IndexAccessor-02.28.2008.zip, > IndexAccessor-1.26.2008.zip, IndexAccessor-2.15.2008.zip, > IndexAccessor.04032008.zip, IndexAccessor.java, IndexAccessor.zip, > IndexAccessorFactory.java, MultiIndexAccessor.java, shai-IndexAccessor-2.zip, > shai-IndexAccessor.zip, shai-IndexAccessor3.zip, SimpleSearchServer.java, > StopWatch.java, TestIndexAccessor.java > > > For building interactive indexes accessed through a network/internet > (multiple threads). > This builds upon the LuceneIndexAccessor patch. That patch was not very > newbie friendly and did not properly handle MultiSearchers (or at the least > made it easy to get into trouble). > This patch simplifies things and provides out of the box support for sharing > the IndexAccessors across threads. There is also a simple test class and > example SearchServer to get you started. > Future revisions will be zipped. > Works pretty solid as is, but could use the ability to warm new Searchers. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: [jira] Resolved: (LUCENE-1285) WeightedSpanTermExtractor incorrectly treats the same terms occurring in different query types
Hey Otis, maybe I am missing something, but it didn't seem like I had the ability to resolve it. Hope its not too obvious and I am just missing the link. Otis Gospodnetic (JIRA) wrote: [ https://issues.apache.org/jira/browse/LUCENE-1285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Otis Gospodnetic resolved LUCENE-1285. -- Resolution: Fixed Lucene Fields: [New, Patch Available] (was: [New]) It looks like Mark already committed this, but forgot resolve this issue, so I'm marking it as Fixed. WeightedSpanTermExtractor incorrectly treats the same terms occurring in different query types -- Key: LUCENE-1285 URL: https://issues.apache.org/jira/browse/LUCENE-1285 Project: Lucene - Java Issue Type: Bug Components: contrib/highlighter Affects Versions: 2.4 Reporter: Andrzej Bialecki Assignee: Otis Gospodnetic Fix For: 2.4 Attachments: highlighter-test.patch, highlighter.patch Given a BooleanQuery with multiple clauses, if a term occurs both in a Span / Phrase query, and in a TermQuery, the results of term extraction are unpredictable and depend on the order of clauses. Concequently, the result of highlighting are incorrect. Example text: t1 t2 t3 t4 t2 Example query: t2 t3 "t1 t2" Current highlighting: [t1 t2] [t3] t4 t2 Correct highlighting: [t1 t2] [t3] t4 [t2] The problem comes from the fact that we keep a Map, and if the same term occurs in a Phrase or Span query the resulting WeightedSpanTerm will have a positionSensitive=true, whereas terms added from TermQuery have positionSensitive=false. The end result for this particular term will depend on the order in which the clauses are processed. My fix is to use a subclass of Map, which on put() always sets the result to the most lax setting, i.e. if we already have a term with positionSensitive=true, and we try to put() a term with positionSensitive=false, we set the result positionSensitive=false, as it will match both cases. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: please vote for this JRE bug
: Here's the JRE bug that can corrupt the Lucene index when a large merge takes : place: : : http://bugs.sun.com/bugdatabase/addVote.do?bug_id=6707044 : : If you all can vote for it, it may help speed the process along ;) For the record: all "Sun Developer Network" members (what you have to become to vote on an issue) can aparently cast a total of 3 votes for Bugs, and multiple votes from the same person forthe same bug do apear to count ... so if you aren't already voting for anything else, go ahead and cast all three votes for this issue. (see link in left nav) -Hoss - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Resolved: (LUCENE-1189) QueryParser does not correctly handle escaped characters within quoted strings
[
https://issues.apache.org/jira/browse/LUCENE-1189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Busch resolved LUCENE-1189.
---
Resolution: Fixed
Fix Version/s: 2.4
Lucene Fields: [New, Patch Available] (was: [Patch Available, New])
Committed. Thanks, Tomer!
> QueryParser does not correctly handle escaped characters within quoted strings
> --
>
> Key: LUCENE-1189
> URL: https://issues.apache.org/jira/browse/LUCENE-1189
> Project: Lucene - Java
> Issue Type: Bug
> Components: QueryParser
>Affects Versions: 2.2, 2.3, 2.3.1
> Environment: Windows Vista Business (x86 and x64) as well as latest
> Ubuntu server, both cases under Tomcat 6.0.14.
> This shouldn't matter though.
>Reporter: Tomer Gabel
>Assignee: Michael Busch
>Priority: Minor
> Fix For: 2.4
>
> Attachments: lucene-1189.patch, QueryParser.jj.patch
>
>
> The Lucene query parser incorrectly handles escaped characters inside quoted
> strings; specifically, a quoted string that ends with an (escaped) backslash
> followed by any additional quoted string will not be properly tokenized.
> Consider the following example:
> bq. {{(name:"///mike\\") or (name:"alphonse")}}
> This is not a contrived example -- it derives from an actual bug we've
> encountered in our system. Running this query will throw an exception, but
> removing the second clause resolves the problem. After some digging I've
> found that the problem is with the way quoted strings are processed by the
> lexer: you'll notice that Mike's name is followed by three escaped
> backslashes right before the ending quote; looking at the JavaCC code for the
> query parser highlights the problem:
> {code:title=QueryParser.jj|borderStyle=solid}
> TOKEN : {
>
> |
> |
> |
> |
> |
> |
> |
> |
> | : Boost
> |
> ...
> {code}
> Take a look at the way the QUOTED token is constructed -- there is no lexical
> processing of the escaped characters within the quoted string itself. In the
> above query the lexer matches everything from the first quote through all the
> backslashes, _treating the end quote as an escaped character_, thus also
> matching the starting quote of the second term. This causes a lexer error,
> because the last quote is then considered the start of a new match.
> I've come to understand that the Lucene query handler is supposed to be able
> to handle unsanitized human input; indeed the lexer above would handle a
> query like {{"blah\"}} without complaining, but that's a "best-guess"
> approach that results in bugs with legal, automatically generated queries.
> I've attached a patch that fixes the erroneous behavior but does not maintain
> leniency with malformed queries; I believe this is the correct approach
> because the two design goals are fundamentally at odds. I'd appreciate any
> comments.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Commented: (LUCENE-1026) Provide a simple way to concurrently access a Lucene index from multiple threads
[ https://issues.apache.org/jira/browse/LUCENE-1026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12600169#action_12600169 ] John Patterson commented on LUCENE-1026: Just checked out the source from the svn link above. I've got a few questions about the code 1 - why use ReentrantLock rather than synchronized blocks? The advice I have come across is to use the standard mechanism unless you need the extra functionality. Can't see anywhere in the code that needs this e.g. obtaining and releasing the lock in different methods or using more than one condition. 2 - why call logger.isLoggable all the time for simple log statements? 3 - In getWriter(boolean) the parameter is ignored if a write already exists. 4 - Why is condition.signalAll() called from getReadingReader()? > Provide a simple way to concurrently access a Lucene index from multiple > threads > > > Key: LUCENE-1026 > URL: https://issues.apache.org/jira/browse/LUCENE-1026 > Project: Lucene - Java > Issue Type: New Feature > Components: Index, Search >Reporter: Mark Miller >Priority: Minor > Attachments: DefaultIndexAccessor.java, > DefaultMultiIndexAccessor.java, IndexAccessor-02.04.2008.zip, > IndexAccessor-02.07.2008.zip, IndexAccessor-02.28.2008.zip, > IndexAccessor-1.26.2008.zip, IndexAccessor-2.15.2008.zip, > IndexAccessor.04032008.zip, IndexAccessor.java, IndexAccessor.zip, > IndexAccessorFactory.java, MultiIndexAccessor.java, shai-IndexAccessor-2.zip, > shai-IndexAccessor.zip, shai-IndexAccessor3.zip, SimpleSearchServer.java, > StopWatch.java, TestIndexAccessor.java > > > For building interactive indexes accessed through a network/internet > (multiple threads). > This builds upon the LuceneIndexAccessor patch. That patch was not very > newbie friendly and did not properly handle MultiSearchers (or at the least > made it easy to get into trouble). > This patch simplifies things and provides out of the box support for sharing > the IndexAccessors across threads. There is also a simple test class and > example SearchServer to get you started. > Future revisions will be zipped. > Works pretty solid as is, but could use the ability to warm new Searchers. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Resolved: (LUCENE-1285) WeightedSpanTermExtractor incorrectly treats the same terms occurring in different query types
[ https://issues.apache.org/jira/browse/LUCENE-1285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Otis Gospodnetic resolved LUCENE-1285. -- Resolution: Fixed Lucene Fields: [New, Patch Available] (was: [New]) It looks like Mark already committed this, but forgot resolve this issue, so I'm marking it as Fixed. > WeightedSpanTermExtractor incorrectly treats the same terms occurring in > different query types > -- > > Key: LUCENE-1285 > URL: https://issues.apache.org/jira/browse/LUCENE-1285 > Project: Lucene - Java > Issue Type: Bug > Components: contrib/highlighter >Affects Versions: 2.4 >Reporter: Andrzej Bialecki >Assignee: Otis Gospodnetic > Fix For: 2.4 > > Attachments: highlighter-test.patch, highlighter.patch > > > Given a BooleanQuery with multiple clauses, if a term occurs both in a Span / > Phrase query, and in a TermQuery, the results of term extraction are > unpredictable and depend on the order of clauses. Concequently, the result of > highlighting are incorrect. > Example text: t1 t2 t3 t4 t2 > Example query: t2 t3 "t1 t2" > Current highlighting: [t1 t2] [t3] t4 t2 > Correct highlighting: [t1 t2] [t3] t4 [t2] > The problem comes from the fact that we keep a Map WeightedSpanTerm>, and if the same term occurs in a Phrase or Span query the > resulting WeightedSpanTerm will have a positionSensitive=true, whereas terms > added from TermQuery have positionSensitive=false. The end result for this > particular term will depend on the order in which the clauses are processed. > My fix is to use a subclass of Map, which on put() always sets the result to > the most lax setting, i.e. if we already have a term with > positionSensitive=true, and we try to put() a term with > positionSensitive=false, we set the result positionSensitive=false, as it > will match both cases. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Commented: (LUCENE-1290) Deprecate Hits
[ https://issues.apache.org/jira/browse/LUCENE-1290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12600168#action_12600168 ] Hoss Man commented on LUCENE-1290: -- i've been away from the party while all the exciting stuff was being discussed, but for my 2 cents... * I firmly believe deprecating Hits is the way to go * I recognize the value in having a simple API that implements the Iterator API like HitIterator, but that doesn't mean it needs to be powered by "Hits", An alternate approach would be to add an "iterator()" method to TopDocs and TopFieldDocs that returned an iterator over the ScoreDocs ... or even an "iterator(Searcher)" method that would return a Iterator of "Hit" objects -- this could have the exact same API as the HitIterator and Hit classes currently do -- just more efficiently and with different constructors. As Michael said: Hits is deprecated, not deleted. clients will have plenty of time to change, and the javadocs for Hits provides a good explanation for how to use TopDocs instead -- but if people want to implement an "Iterator" based alternative, we should probably do that in a new issue. > Deprecate Hits > -- > > Key: LUCENE-1290 > URL: https://issues.apache.org/jira/browse/LUCENE-1290 > Project: Lucene - Java > Issue Type: Task > Components: Search >Reporter: Michael Busch >Assignee: Michael Busch >Priority: Minor > Fix For: 2.4 > > Attachments: lucene-1290.patch, lucene-1290.patch, lucene-1290.patch > > > The Hits class has several drawbacks as pointed out in LUCENE-954. > The other search APIs that use TopDocCollector and TopDocs should be used > instead. > This patch: > - deprecates org/apache/lucene/search/Hits, Hit, and HitIterator, as well as > the Searcher.search( * ) methods which return a Hits Object. > - removes all references to Hits from the core and uses TopDocs and ScoreDoc > instead > - Changes the demo SearchFiles: adds the two modes 'paging search' and > 'streaming search', > each of which demonstrating a different way of using the search APIs. The > former > uses TopDocs and a TopDocCollector, the latter a custom HitCollector > implementation. > - Updates the online tutorial that descibes the demo. > All tests pass. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
please vote for this JRE bug
Here's the JRE bug that can corrupt the Lucene index when a large merge takes place: http://bugs.sun.com/bugdatabase/addVote.do?bug_id=6707044 If you all can vote for it, it may help speed the process along ;) Here's the corresponding Lucene issue: https://issues.apache.org/jira/browse/LUCENE-1282 Mike - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Commented: (LUCENE-1282) Sun hotspot compiler bug in 1.6.0_04/05 affects Lucene
[
https://issues.apache.org/jira/browse/LUCENE-1282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12600165#action_12600165
]
Michael McCandless commented on LUCENE-1282:
Here is the bug at Sun:
http://bugs.sun.com/bugdatabase/addVote.do?bug_id=6707044
> Sun hotspot compiler bug in 1.6.0_04/05 affects Lucene
> --
>
> Key: LUCENE-1282
> URL: https://issues.apache.org/jira/browse/LUCENE-1282
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
>Affects Versions: 2.3, 2.3.1
>Reporter: Michael McCandless
>Assignee: Michael McCandless
>Priority: Minor
> Fix For: 2.4
>
> Attachments: corrupt_merge_out15.txt, crashtest, crashtest.log,
> hs_err_pid27359.log
>
>
> This is not a Lucene bug. It's an as-yet not fully characterized Sun
> JRE bug, as best I can tell. I'm opening this to gather all things we
> know, and to work around it in Lucene if possible, and maybe open an
> issue with Sun if we can reduce it to a compact test case.
> It's hit at least 3 users:
>
> http://mail-archives.apache.org/mod_mbox/lucene-java-user/200803.mbox/[EMAIL
> PROTECTED]
>
> http://mail-archives.apache.org/mod_mbox/lucene-solr-user/200804.mbox/[EMAIL
> PROTECTED]
>
> http://mail-archives.apache.org/mod_mbox/lucene-java-user/200805.mbox/[EMAIL
> PROTECTED]
> It's specific to at least JRE 1.6.0_04 and 1.6.0_05, that affects
> Lucene. Whereas 1.6.0_03 works OK and it's unknown whether 1.6.0_06
> shows it.
> The bug affects bulk merging of stored fields. When it strikes, the
> segment produced by a merge is corrupt because its fdx file (stored
> fields index file) is missing one document. After iterating many
> times with the first user that hit this, adding diagnostics &
> assertions, its seems that a call to fieldsWriter.addDocument some
> either fails to run entirely, or, fails to invoke its call to
> indexStream.writeLong. It's as if when hotspot compiles a method,
> there's some sort of race condition in cutting over to the compiled
> code whereby a single method call fails to be invoked (speculation).
> Unfortunately, this corruption is silent when it occurs and only later
> detected when a merge tries to merge the bad segment, or an
> IndexReader tries to open it. Here's a typical merge exception:
> {code}
> Exception in thread "Thread-10"
> org.apache.lucene.index.MergePolicy$MergeException:
> org.apache.lucene.index.CorruptIndexException:
> doc counts differ for segment _3gh: fieldsReader shows 15999 but
> segmentInfo shows 16000
> at
> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:271)
> Caused by: org.apache.lucene.index.CorruptIndexException: doc counts differ
> for segment _3gh: fieldsReader shows 15999 but segmentInfo shows 16000
> at
> org.apache.lucene.index.SegmentReader.initialize(SegmentReader.java:313)
> at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:262)
> at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:221)
> at
> org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3099)
> at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:2834)
> at
> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:240)
> {code}
> and here's a typical exception hit when opening a searcher:
> {code}
> org.apache.lucene.index.CorruptIndexException: doc counts differ for segment
> _kk: fieldsReader shows 72670 but segmentInfo shows 72671
> at
> org.apache.lucene.index.SegmentReader.initialize(SegmentReader.java:313)
> at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:262)
> at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:230)
> at
> org.apache.lucene.index.DirectoryIndexReader$1.doBody(DirectoryIndexReader.java:73)
> at
> org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:636)
> at
> org.apache.lucene.index.DirectoryIndexReader.open(DirectoryIndexReader.java:63)
> at org.apache.lucene.index.IndexReader.open(IndexReader.java:209)
> at org.apache.lucene.index.IndexReader.open(IndexReader.java:173)
> at
> org.apache.lucene.search.IndexSearcher.(IndexSearcher.java:48)
> {code}
> Sometimes, adding -Xbatch (forces up front compilation) or -Xint
> (disables compilation) to the java command line works around the
> issue.
> Here are some of the OS's we've seen the failure on:
> {code}
> SuSE 10.0
> Linux phoebe 2.6.13-15-smp #1 SMP Tue Sep 13 14:56:15 UTC 2005 x86_64
> x86_64 x86_64 GNU/Linux
> SuSE 8.2
> Linux phobos 2.4.20-64GB-SMP #1 SMP Mon Mar 17 17:56:03 UTC 2003 i686
> unknown unknown GNU/Linux
> Red Hat En
[jira] Updated: (LUCENE-1189) QueryParser does not correctly handle escaped characters within quoted strings
[
https://issues.apache.org/jira/browse/LUCENE-1189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Busch updated LUCENE-1189:
--
Priority: Minor (was: Major)
Lucene Fields: [New, Patch Available] (was: [Patch Available, New])
> QueryParser does not correctly handle escaped characters within quoted strings
> --
>
> Key: LUCENE-1189
> URL: https://issues.apache.org/jira/browse/LUCENE-1189
> Project: Lucene - Java
> Issue Type: Bug
> Components: QueryParser
>Affects Versions: 2.2, 2.3, 2.3.1
> Environment: Windows Vista Business (x86 and x64) as well as latest
> Ubuntu server, both cases under Tomcat 6.0.14.
> This shouldn't matter though.
>Reporter: Tomer Gabel
>Assignee: Michael Busch
>Priority: Minor
> Attachments: lucene-1189.patch, QueryParser.jj.patch
>
>
> The Lucene query parser incorrectly handles escaped characters inside quoted
> strings; specifically, a quoted string that ends with an (escaped) backslash
> followed by any additional quoted string will not be properly tokenized.
> Consider the following example:
> bq. {{(name:"///mike\\") or (name:"alphonse")}}
> This is not a contrived example -- it derives from an actual bug we've
> encountered in our system. Running this query will throw an exception, but
> removing the second clause resolves the problem. After some digging I've
> found that the problem is with the way quoted strings are processed by the
> lexer: you'll notice that Mike's name is followed by three escaped
> backslashes right before the ending quote; looking at the JavaCC code for the
> query parser highlights the problem:
> {code:title=QueryParser.jj|borderStyle=solid}
> TOKEN : {
>
> |
> |
> |
> |
> |
> |
> |
> |
> | : Boost
> |
> ...
> {code}
> Take a look at the way the QUOTED token is constructed -- there is no lexical
> processing of the escaped characters within the quoted string itself. In the
> above query the lexer matches everything from the first quote through all the
> backslashes, _treating the end quote as an escaped character_, thus also
> matching the starting quote of the second term. This causes a lexer error,
> because the last quote is then considered the start of a new match.
> I've come to understand that the Lucene query handler is supposed to be able
> to handle unsanitized human input; indeed the lexer above would handle a
> query like {{"blah\"}} without complaining, but that's a "best-guess"
> approach that results in bugs with legal, automatically generated queries.
> I've attached a patch that fixes the erroneous behavior but does not maintain
> leniency with malformed queries; I believe this is the correct approach
> because the two design goals are fundamentally at odds. I'd appreciate any
> comments.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Updated: (LUCENE-1189) QueryParser does not correctly handle escaped characters within quoted strings
[
https://issues.apache.org/jira/browse/LUCENE-1189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Busch updated LUCENE-1189:
--
Attachment: lucene-1189.patch
Thanks for your patch, Tomer! Your approach certainly seems
correct to me.
The file I'm attaching has your fix to QueryParser.jj and also
a testcase similar to your example that fails before and passes
after applying the patch.
I'm planning to commit this in a day or so.
> QueryParser does not correctly handle escaped characters within quoted strings
> --
>
> Key: LUCENE-1189
> URL: https://issues.apache.org/jira/browse/LUCENE-1189
> Project: Lucene - Java
> Issue Type: Bug
> Components: QueryParser
>Affects Versions: 2.2, 2.3, 2.3.1
> Environment: Windows Vista Business (x86 and x64) as well as latest
> Ubuntu server, both cases under Tomcat 6.0.14.
> This shouldn't matter though.
>Reporter: Tomer Gabel
>Assignee: Michael Busch
> Attachments: lucene-1189.patch, QueryParser.jj.patch
>
>
> The Lucene query parser incorrectly handles escaped characters inside quoted
> strings; specifically, a quoted string that ends with an (escaped) backslash
> followed by any additional quoted string will not be properly tokenized.
> Consider the following example:
> bq. {{(name:"///mike\\") or (name:"alphonse")}}
> This is not a contrived example -- it derives from an actual bug we've
> encountered in our system. Running this query will throw an exception, but
> removing the second clause resolves the problem. After some digging I've
> found that the problem is with the way quoted strings are processed by the
> lexer: you'll notice that Mike's name is followed by three escaped
> backslashes right before the ending quote; looking at the JavaCC code for the
> query parser highlights the problem:
> {code:title=QueryParser.jj|borderStyle=solid}
> TOKEN : {
>
> |
> |
> |
> |
> |
> |
> |
> |
> | : Boost
> |
> ...
> {code}
> Take a look at the way the QUOTED token is constructed -- there is no lexical
> processing of the escaped characters within the quoted string itself. In the
> above query the lexer matches everything from the first quote through all the
> backslashes, _treating the end quote as an escaped character_, thus also
> matching the starting quote of the second term. This causes a lexer error,
> because the last quote is then considered the start of a new match.
> I've come to understand that the Lucene query handler is supposed to be able
> to handle unsanitized human input; indeed the lexer above would handle a
> query like {{"blah\"}} without complaining, but that's a "best-guess"
> approach that results in bugs with legal, automatically generated queries.
> I've attached a patch that fixes the erroneous behavior but does not maintain
> leniency with malformed queries; I believe this is the correct approach
> because the two design goals are fundamentally at odds. I'd appreciate any
> comments.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
