PAGE RANKING IN LUCENE?
I need an urgent help. I want to change the page ranking algorithm in lucene and I do not know where to start from and what file should I change? I do not know what classes are involved. I have only a few days to do so so please help me with your complete explanation as a big favor! Bests, Eric -- View this message in context: http://www.nabble.com/PAGE-RANKING-IN-LUCENE%3CNEED-URGENT-HELP%21%3E-tf3575000.html#a9989950 Sent from the Lucene - Java Developer mailing list archive at Nabble.com. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: [jira] Commented: (LUCENE-584) Decouple Filter from BitSet; relation with LUCENE-730
: > Hoss, would this work (is this what you said)?
: > public Matcher getMatcher(IndexReader reader) throws IOException {
: > if(bits() == null) throw new SomeException("Filter must implement at least
: one of...");
: > return new BitsMatcher(bits());
: > }
Assuming BitsMatcher does what i think it does then yes, that's what i had
in mind ... i was specificly saying to make a default Matcher
implementation out of the code in the patched version of IndexSearcher
that has the comment...
+} else { // bits for filtering, skipTo() not used on scorer:
: This will not work correctly when the Scorer for the query that is searched
: with a filter does not implement skipTo(), for example BooleanScorer.
: See also the javadoc of class IndexSearcher in the patch.
I don'tget it, how would a Scorer not implement skipTo? ...oh...
final class BooleanScorer extends Scorer {
...
public boolean skipTo(int target) {
throw new UnsupportedOperationException();
}
...so lemme see if i understand this:
What's happening in the current trunk is that the only situations
in which code will attempt to call skipTo on a Scorer are:
a) From the score(HitCollector hc) method of the same Scorer class
(you should know if you suport it, you're in the class)
b) From the skipTo method of an enclosing Scorer
(If you "add" Scorer X to a a wrapper Scorer Y, and Y implements
skipTo, it can assume that X implements skipTo).
Am I correct so far?
In the latest version of the Matcher patch...
https://issues.apache.org/jira/secure/attachment/12352057/Matcher20070226.patch
...this changes, such that IndexSearcher will assume a Scorer supports
skipTo iff a Filter is used which implements getMatcher (I guess the
assumption being that if the code being used is new enough to support Matchers,
it's
new enough to support Scorer.skipTo). *BUT* if it's an "old" Filter using
a BitSet the code in IndexSearcher will continue with the same old
assumptions about the Scorer.
And the change eks describes (which is a much better way to describe what
i was suggesting) would break this safety net by always assuming skipTo
was safe to call.
So really the issue is that the patch assumpes one thing (Scorer supports
skipTo) based on the presence of something that should be thought of as
"newer" (Filter supports getMatcher) and relying on documentation to
enforce this.
Am I caught up now?
Off the top of my head, the best solution i can think of to this issue
would be to add the naive implementation of skipTo to Scorer, remove
the UnsupportedOperationException of skipTo from all Scorers in the core,
and rev Lucene to version 3.0 since this would probably be considered a
serious API change (method sigs don't change, but now we're requiring
people to implement a method that we have said in the past (by example)
can be Unsupported.
In general i'm not fond of assuming Scorer.skipTo when Filter.getMatcher
... the concepts are really orthoginal and even if it's a decent
assumption to make today, it doens't help us tomorow when we want to add a
getMatcher method to all of the core Filter classes to improve
performance.
-Hoss
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
Re: [jira] Commented: (LUCENE-584) Decouple Filter from BitSet; relation with LUCENE-730
ok , i see, thanks for hand holding here.
the simplest solution would be (without making another bigger/riskier patch):
- commit LUCENE-584 as is; no harm to anyone but some temporary complexity in
IndexSearcher
- commit LUCENE-730 - does no harm
- open new Jura issue "Simplify Filter usage in IndexSearcher" and re-factor
Filter to behave as Hoss mentioned it
- Original Message
From: Paul Elschot <[EMAIL PROTECTED]>
To: [EMAIL PROTECTED]
Sent: Friday, 13 April, 2007 11:05:10 PM
Subject: Re: [jira] Commented: (LUCENE-584) Decouple Filter from BitSet;
relation with LUCENE-730
On Friday 13 April 2007 22:10, eks dev wrote:
> Hoss, would this work (is this what you said)?
>
> public BitSet bits(IndexReader reader) throws IOException{
> return null;
> }
>
> public Matcher getMatcher(IndexReader reader) throws IOException {
> if(bits() == null) throw new SomeException("Filter must implement at least
one of...");
> return new BitsMatcher(bits());
> }
This will not work correctly when the Scorer for the query that is searched
with a filter does not implement skipTo(), for example BooleanScorer.
See also the javadoc of class IndexSearcher in the patch.
LUCENE-730 explicitly uses BooleanScorer, but only for the non filtered case
with a top level disjunction.
I think that with LUCENE-730 also added, the filtered case with BooleanScorer
would go away, allowing to simplify this logic in IndexSearcher.
This simplification of IndexSearcher is not in the LUCENE-730 patch, because
LUCENE-584 is not committed. At the moment I don't know precisely what
IndexSearcher would look like after LUCENE-730.
With LUCENE-730 BooleanScorer.setUseScorer14() could also be
removed/deprecated, but that is also not yet in the LUCENE-730 patch.
Regards,
Paul Elschot
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
___
Yahoo! Answers - Got a question? Someone out there knows the answer. Try it
now.
http://uk.answers.yahoo.com/
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
Re: [jira] Commented: (LUCENE-584) Decouple Filter from BitSet; relation with LUCENE-730
On Friday 13 April 2007 22:10, eks dev wrote:
> Hoss, would this work (is this what you said)?
>
> public BitSet bits(IndexReader reader) throws IOException{
> return null;
> }
>
> public Matcher getMatcher(IndexReader reader) throws IOException {
> if(bits() == null) throw new SomeException("Filter must implement at least
one of...");
> return new BitsMatcher(bits());
> }
This will not work correctly when the Scorer for the query that is searched
with a filter does not implement skipTo(), for example BooleanScorer.
See also the javadoc of class IndexSearcher in the patch.
LUCENE-730 explicitly uses BooleanScorer, but only for the non filtered case
with a top level disjunction.
I think that with LUCENE-730 also added, the filtered case with BooleanScorer
would go away, allowing to simplify this logic in IndexSearcher.
This simplification of IndexSearcher is not in the LUCENE-730 patch, because
LUCENE-584 is not committed. At the moment I don't know precisely what
IndexSearcher would look like after LUCENE-730.
With LUCENE-730 BooleanScorer.setUseScorer14() could also be
removed/deprecated, but that is also not yet in the LUCENE-730 patch.
Regards,
Paul Elschot
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
Re: [jira] Commented: (LUCENE-584) Decouple Filter from BitSet
Hoss, would this work (is this what you said)?
public BitSet bits(IndexReader reader) throws IOException{
return null;
}
public Matcher getMatcher(IndexReader reader) throws IOException {
if(bits() == null) throw new SomeException("Filter must implement at least
one of...");
return new BitsMatcher(bits());
}
and IndexSearcher does not have any logic, just uses getMatcher()
current implementations would work, new as well
- Original Message
From: Hoss Man (JIRA) <[EMAIL PROTECTED]>
To: [EMAIL PROTECTED]
Sent: Friday, 13 April, 2007 8:01:16 PM
Subject: [jira] Commented: (LUCENE-584) Decouple Filter from BitSet
[
https://issues.apache.org/jira/browse/LUCENE-584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12488733
]
Hoss Man commented on LUCENE-584:
-
I'm still behind on following this issue, but Otis: if you are interested in
moving forward with this, you might consider trying the cahnges i proposed in
my "15/Mar/07 11:06 AM" Comment...
https://issues.apache.org/jira/browse/LUCENE-584#action_12481263
...I think it would keep IndexSearcher a little cleaner, and make it easier for
people to migrate existing Filter's gradually (without requiring extra work for
people writing new "Matcher" style Filters from scratch)
> Decouple Filter from BitSet
> ---
>
> Key: LUCENE-584
> URL: https://issues.apache.org/jira/browse/LUCENE-584
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Search
>Affects Versions: 2.0.1
>Reporter: Peter Schäfer
>Priority: Minor
> Attachments: bench-diff.txt, bench-diff.txt, BitsMatcher.java,
> Filter-20060628.patch, HitCollector-20060628.patch,
> IndexSearcher-20060628.patch, MatchCollector.java, Matcher.java,
> Matcher20070226.patch, Scorer-20060628.patch, Searchable-20060628.patch,
> Searcher-20060628.patch, Some Matchers.zip, SortedVIntList.java,
> TestSortedVIntList.java
>
>
> {code}
> package org.apache.lucene.search;
> public abstract class Filter implements java.io.Serializable
> {
> public abstract AbstractBitSet bits(IndexReader reader) throws IOException;
> }
> public interface AbstractBitSet
> {
> public boolean get(int index);
> }
> {code}
> It would be useful if the method =Filter.bits()= returned an abstract
> interface, instead of =java.util.BitSet=.
> Use case: there is a very large index, and, depending on the user's
> privileges, only a small portion of the index is actually visible.
> Sparsely populated =java.util.BitSet=s are not efficient and waste lots of
> memory. It would be desirable to have an alternative BitSet implementation
> with smaller memory footprint.
> Though it _is_ possibly to derive classes from =java.util.BitSet=, it was
> obviously not designed for that purpose.
> That's why I propose to use an interface instead. The default implementation
> could still delegate to =java.util.BitSet=.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
___
Yahoo! Mail is the world's favourite email. Don't settle for less, sign up for
your free account today
http://uk.rd.yahoo.com/evt=44106/*http://uk.docs.yahoo.com/mail/winter07.html
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
Re: Maven artifacts for Lucene.*
Chris Hostetter wrote: > Whatever files also need to be included along with the jars in order to > make the maven distribution complete that can't be built completley > dynamicly (ie: the md5 files) can certainly be commited into the > repository ... but if making a release requires a lot of manual upating to > those files, it's going to be a hinderane to the process ... things like > version number and date should ideally be filled in via variables to help > keep things automated. This starts to sound like a plan that can work. I'll see if I can hack something up as a patch fow a review. Do you think poms should live in a separate dir or should they be spread across dirs (modules). > jar dependencies are another matter ... as you say, for java-lucene the > issue is trivial since there are no dependencies, but for other projects > it could get complicated. Solr (for example) ships with the versions of > it's dependencies that it expects to use, and in some cases these version > may not be official release versions that you would ever find in a maven > repository. I'm notsure how apps that want to publish to maven but depend > on apss that do not publish to maven deal with this problem, but whatever > solution they use could also be used in this case. I have seen for example a solution where artifacts are published on somewhere else but official repositories, but to be frank I don't know what's the best (or at least acceptable) solution here. > : projects). IMO we should however try to look at the big picture also and > : not only try to solve the minimal part to get it out of lucene-java > : hands, because I am afraid that if the minimum is done here in > : lucene-java there might be caps to fill in other projects and the way > : things are done here is not usable in other sub projects as it is. > > each project has it's own community ... even if you find a perfect > solution to every problem anyone in the world might ever encounter, > discussing it on java-dev does nothing to get your solution adopted by the > nutch, hadoop, or solr communities. I understand that there are separate communities. I am not saying that everybody must accept the solution that will (if any) adopted by lucene-java. But still I am hoping that we lucene-java won't deliberately accept a solution that won't work for others (as you said it: "if a simple solution is found for our build file, it will probably lend itself to similar soluteions for hte other Lucne projects that use ant.") -- Sami Siren - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Packaging Lucene 2.1.0 for Debian; found 2 junit errors
I also saw those when I did my maven trials. I didn't dig any deeper.
--
Sami Siren
Jan-Pascal van Best wrote:
> Hi all,
>
> I'm busy packaging Lucene 2.1.0 for Debian. When I run "ant test" in
> contrib/highlighter and in contrib/spellchecker I get failed unit tests
> (see end of this e-mail). Are these two known issues, or do you think
> the may be a problem with either the build environment, or my packaging?
>
> I'm using Sun's java5 jdk, ant 1.6.5, ant-optional 1.6.5, and junit 3.8.1.1.
>
> Any ideas?
>
> Thanks for any help,
>
> Jan-Pascal
>
>
>
>
> [junit] Testsuite: org.apache.lucene.search.highlight.HighlighterTest
> [junit] Tests run: 26, Failures: 1, Errors: 0, Time elapsed: 3,072 sec
>
> [junit] - Standard Output ---
> [junit] Searching for: kennedy
> [junit] John Kennedy has been shot
> [junit] This piece of text refers to Kennedy... to
> Kennedy
> [junit] kennedy has been shot
> [junit] Searching for: kennedy
> [junit] John Kennedy has been shot
> [junit] refers to Kennedy... to Kennedy
> [junit] kennedy has been shot
> [junit] Searching for: keneddy^0.14285707 kennedy^0.71428573
> [junit] John Kennedy has been shot
> [junit] refers to Kennedy... to Kennedy
> [junit] kennedy has been shot
> [junit] to Keneddy
> [junit] Searching for: kennedy
> [junit] John Kennedy has been shot
> [junit] refers to Kennedy... to Kennedy
> [junit] kennedy has been shot
> [junit] Searching for: keneddy kennedy
> [junit] to Keneddy
> [junit] John Kennedy has been shot
> [junit] refers to Kennedy... to Kennedy
> [junit] kennedy has been shot
> [junit] Searching for: ConstantScore(contents:[kannedy-kznnedy])
> [junit]
> [junit]
> [junit]
> [junit]
> [junit] Searching for: "john kennedy"
> [junit] John Kennedy has been shot
> [junit] Searching for: spanNear([john, kennedy], 1, true)
> [junit] John Kennedy has been shot
> [junit] Searching for: filtered(spanNear([john, kennedy], 1,
> true))->contents:[john-john]
> [junit] John Kennedy has been shot
> [junit] Searching for: filtered("john kennedy")->contents:[john-john]
> [junit] John Kennedy has been shot
> [junit] Searching for: john (kennedy)
> [junit] John Kennedy has been shot
> [junit] refers to Kennedy... to Kennedy
> [junit] kennedy has been shot
> [junit] Searching for: jfk kennedy
> [junit] JFK has been shot
> [junit] John Kennedy has been shot
> [junit] refers to Kennedy... to Kennedy
> [junit] kennedy has been shot
> [junit] Searching for: kennedy
> [junit] John Kennedy has been shot
> [junit] This piece of text refers to Kennedy
> [junit] kennedy has been shot
> [junit] Searching for: kennedy
> [junit] John Kennedy has been shot
> [junit] This piece of text refers to Kennedy at the beginning
> then has a longer piece of text that is very
> [junit] is really here which says kennedy has been shot
> [junit] Searching for: kennedy
> [junit] John Kennedy has been shot
> [junit] This piece of text refers to Kennedy at the beginning then
> has a longer piece of text that is very long in the middle and finally ends
> with anothe
> r reference to Kennedy
> [junit] Hello this is a piece of text that is very long and contains too
> much preamble and the meat is really here which says kennedy has been
> shot
> [junit] Searching for: meat
> [junit] Searching with primitive query
>
>
>
>
> [junit] Searching for: aninvalidquerywhichshouldyieldnoresults
> [junit] Searching for: multi*
> [junit] multiOne
> [junit] multiTwo
> [junit] - ---
> [junit] Testcase:
> testGetRangeFragments(org.apache.lucene.search.highlight.HighlighterTest):
> FAILED
> [junit] Failed to find correct number of highlights 0 found
> [junit] junit.framework.AssertionFailedError: Failed to find correct
> number of highlights 0 found
> [junit] at
> org.apache.lucene.search.highlight.HighlighterTest.testGetRangeFragments(HighlighterTest.java:137)
>
>
> [junit] Test org.apache.lucene.search.highlight.HighlighterTest FAILED
>
> BUILD FAILED
>
>
> --- and ---
>
> [junit] Testsuite: org.apache.lucene.search.spell.TestSpellChecker
> [junit] Tests run: 1, Failures: 1, Errors: 0, Time elapsed: 1,234 sec
>
> [junit] Testcase:
> testBuild(org.apache.lucene.search.spell.TestSpellChecker): FAILED
> [junit] expected:<56> but was:<29>
> [junit] junit.framework.AssertionFailedError: expected:<56> but was:<29>
> [junit] at
> org.apache.lucene.search.spell.TestSpellChecker.testBuild(TestSpellChecker.java:60)
>
>
> [junit] Test org.apache.lu
[jira] Commented: (LUCENE-584) Decouple Filter from BitSet
[
https://issues.apache.org/jira/browse/LUCENE-584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12488733
]
Hoss Man commented on LUCENE-584:
-
I'm still behind on following this issue, but Otis: if you are interested in
moving forward with this, you might consider trying the cahnges i proposed in
my "15/Mar/07 11:06 AM" Comment...
https://issues.apache.org/jira/browse/LUCENE-584#action_12481263
...I think it would keep IndexSearcher a little cleaner, and make it easier for
people to migrate existing Filter's gradually (without requiring extra work for
people writing new "Matcher" style Filters from scratch)
> Decouple Filter from BitSet
> ---
>
> Key: LUCENE-584
> URL: https://issues.apache.org/jira/browse/LUCENE-584
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Search
>Affects Versions: 2.0.1
>Reporter: Peter Schäfer
>Priority: Minor
> Attachments: bench-diff.txt, bench-diff.txt, BitsMatcher.java,
> Filter-20060628.patch, HitCollector-20060628.patch,
> IndexSearcher-20060628.patch, MatchCollector.java, Matcher.java,
> Matcher20070226.patch, Scorer-20060628.patch, Searchable-20060628.patch,
> Searcher-20060628.patch, Some Matchers.zip, SortedVIntList.java,
> TestSortedVIntList.java
>
>
> {code}
> package org.apache.lucene.search;
> public abstract class Filter implements java.io.Serializable
> {
> public abstract AbstractBitSet bits(IndexReader reader) throws IOException;
> }
> public interface AbstractBitSet
> {
> public boolean get(int index);
> }
> {code}
> It would be useful if the method =Filter.bits()= returned an abstract
> interface, instead of =java.util.BitSet=.
> Use case: there is a very large index, and, depending on the user's
> privileges, only a small portion of the index is actually visible.
> Sparsely populated =java.util.BitSet=s are not efficient and waste lots of
> memory. It would be desirable to have an alternative BitSet implementation
> with smaller memory footprint.
> Though it _is_ possibly to derive classes from =java.util.BitSet=, it was
> obviously not designed for that purpose.
> That's why I propose to use an interface instead. The default implementation
> could still delegate to =java.util.BitSet=.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
Packaging Lucene 2.1.0 for Debian; found 2 junit errors
Hi all,
I'm busy packaging Lucene 2.1.0 for Debian. When I run "ant test" in
contrib/highlighter and in contrib/spellchecker I get failed unit tests
(see end of this e-mail). Are these two known issues, or do you think
the may be a problem with either the build environment, or my packaging?
I'm using Sun's java5 jdk, ant 1.6.5, ant-optional 1.6.5, and junit 3.8.1.1.
Any ideas?
Thanks for any help,
Jan-Pascal
[junit] Testsuite: org.apache.lucene.search.highlight.HighlighterTest
[junit] Tests run: 26, Failures: 1, Errors: 0, Time elapsed: 3,072 sec
[junit] - Standard Output ---
[junit] Searching for: kennedy
[junit] John Kennedy has been shot
[junit] This piece of text refers to Kennedy... to Kennedy
[junit] kennedy has been shot
[junit] Searching for: kennedy
[junit] John Kennedy has been shot
[junit] refers to Kennedy... to Kennedy
[junit] kennedy has been shot
[junit] Searching for: keneddy^0.14285707 kennedy^0.71428573
[junit] John Kennedy has been shot
[junit] refers to Kennedy... to Kennedy
[junit] kennedy has been shot
[junit] to Keneddy
[junit] Searching for: kennedy
[junit] John Kennedy has been shot
[junit] refers to Kennedy... to Kennedy
[junit] kennedy has been shot
[junit] Searching for: keneddy kennedy
[junit] to Keneddy
[junit] John Kennedy has been shot
[junit] refers to Kennedy... to Kennedy
[junit] kennedy has been shot
[junit] Searching for: ConstantScore(contents:[kannedy-kznnedy])
[junit]
[junit]
[junit]
[junit]
[junit] Searching for: "john kennedy"
[junit] John Kennedy has been shot
[junit] Searching for: spanNear([john, kennedy], 1, true)
[junit] John Kennedy has been shot
[junit] Searching for: filtered(spanNear([john, kennedy], 1,
true))->contents:[john-john]
[junit] John Kennedy has been shot
[junit] Searching for: filtered("john kennedy")->contents:[john-john]
[junit] John Kennedy has been shot
[junit] Searching for: john (kennedy)
[junit] John Kennedy has been shot
[junit] refers to Kennedy... to Kennedy
[junit] kennedy has been shot
[junit] Searching for: jfk kennedy
[junit] JFK has been shot
[junit] John Kennedy has been shot
[junit] refers to Kennedy... to Kennedy
[junit] kennedy has been shot
[junit] Searching for: kennedy
[junit] John Kennedy has been shot
[junit] This piece of text refers to Kennedy
[junit] kennedy has been shot
[junit] Searching for: kennedy
[junit] John Kennedy has been shot
[junit] This piece of text refers to Kennedy at the beginning
then has a longer piece of text that is very
[junit] is really here which says kennedy has been shot
[junit] Searching for: kennedy
[junit] John Kennedy has been shot
[junit] This piece of text refers to Kennedy at the beginning then
has a longer piece of text that is very long in the middle and finally ends
with anothe
r reference to Kennedy
[junit] Hello this is a piece of text that is very long and contains too
much preamble and the meat is really here which says kennedy has been
shot
[junit] Searching for: meat
[junit] Searching with primitive query
[junit] Searching for: aninvalidquerywhichshouldyieldnoresults
[junit] Searching for: multi*
[junit] multiOne
[junit] multiTwo
[junit] - ---
[junit] Testcase:
testGetRangeFragments(org.apache.lucene.search.highlight.HighlighterTest):
FAILED
[junit] Failed to find correct number of highlights 0 found
[junit] junit.framework.AssertionFailedError: Failed to find correct number
of highlights 0 found
[junit] at
org.apache.lucene.search.highlight.HighlighterTest.testGetRangeFragments(HighlighterTest.java:137)
[junit] Test org.apache.lucene.search.highlight.HighlighterTest FAILED
BUILD FAILED
--- and ---
[junit] Testsuite: org.apache.lucene.search.spell.TestSpellChecker
[junit] Tests run: 1, Failures: 1, Errors: 0, Time elapsed: 1,234 sec
[junit] Testcase:
testBuild(org.apache.lucene.search.spell.TestSpellChecker): FAILED
[junit] expected:<56> but was:<29>
[junit] junit.framework.AssertionFailedError: expected:<56> but was:<29>
[junit] at
org.apache.lucene.search.spell.TestSpellChecker.testBuild(TestSpellChecker.java:60)
[junit] Test org.apache.lucene.search.spell.TestSpellChecker FAILED
BUILD FAILED
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
