[jira] Commented: (LUCENE-1003) RussianAnalyzer's tokenizer skips numbers from input text,
[
https://issues.apache.org/jira/browse/LUCENE-1003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12528663
]
Nick Menere commented on LUCENE-1003:
-
Yeah,
I raised this on the dev list a few months ago and didn't get much response.
I think I might even be responsible for that code above. It was meant more as
hack to [get a customer up and
running|http://jira.atlassian.com/browse/JRA-12399].
Cheers,
Nick
> RussianAnalyzer's tokenizer skips numbers from input text,
> --
>
> Key: LUCENE-1003
> URL: https://issues.apache.org/jira/browse/LUCENE-1003
> Project: Lucene - Java
> Issue Type: Bug
> Components: Analysis
>Affects Versions: 2.2
>Reporter: TUSUR OpenTeam
>
> RussianAnalyzer's tokenizer skips numbers from input text, so that resulting
> token stream miss numbers. Problem can be solved by adding numbers to
> RussianCharsets.UnicodeRussian. See test case below for details.
> {code:title=TestRussianAnalyzer.java|borderStyle=solid}
> public class TestRussianAnalyzer extends TestCase {
> Reader reader = new StringReader("text 1000");
> // test FAILS
> public void testStemmer() {
> testAnalyzer(new RussianAnalyzer());
> }
> // test PASSES
> public void testFixedRussianAnalyzer() {
> testAnalyzer(new RussianAnalyzer(getRussianCharSet()));
> }
> private void testAnalyzer(RussianAnalyzer analyzer) {
> try {
> TokenStream stream = analyzer.tokenStream("text", reader);
> assertEquals("text", stream.next().termText());
> assertNotNull(stream.next());
> } catch (IOException e) {
> fail(e.getMessage());
> }
> }
> private char[] getRussianCharSet() {
> int length = RussianCharsets.UnicodeRussian.length;
> final char[] russianChars = new char[length + 10];
> System
> .arraycopy(RussianCharsets.UnicodeRussian, 0, russianChars, 0,
> length);
> russianChars[length++] = '0';
> russianChars[length++] = '1';
> russianChars[length++] = '2';
> russianChars[length++] = '3';
> russianChars[length++] = '4';
> russianChars[length++] = '5';
> russianChars[length++] = '6';
> russianChars[length++] = '7';
> russianChars[length++] = '8';
> russianChars[length] = '9';
> return russianChars;
> }
> }
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Updated: (LUCENE-1003) RussianAnalyzer's tokenizer skips numbers from input text,
[
https://issues.apache.org/jira/browse/LUCENE-1003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
TUSUR OpenTeam updated LUCENE-1003:
---
Description:
RussianAnalyzer's tokenizer skips numbers from input text, so that resulting
token stream miss numbers. Problem can be solved by adding numbers to
RussianCharsets.UnicodeRussian. See test case below for details.
{code:title=TestRussianAnalyzer.java|borderStyle=solid}
public class TestRussianAnalyzer extends TestCase {
Reader reader = new StringReader("text 1000");
// test FAILS
public void testStemmer() {
testAnalyzer(new RussianAnalyzer());
}
// test PASSES
public void testFixedRussianAnalyzer() {
testAnalyzer(new RussianAnalyzer(getRussianCharSet()));
}
private void testAnalyzer(RussianAnalyzer analyzer) {
try {
TokenStream stream = analyzer.tokenStream("text", reader);
assertEquals("text", stream.next().termText());
assertNotNull(stream.next());
} catch (IOException e) {
fail(e.getMessage());
}
}
private char[] getRussianCharSet() {
int length = RussianCharsets.UnicodeRussian.length;
final char[] russianChars = new char[length + 10];
System
.arraycopy(RussianCharsets.UnicodeRussian, 0, russianChars, 0, length);
russianChars[length++] = '0';
russianChars[length++] = '1';
russianChars[length++] = '2';
russianChars[length++] = '3';
russianChars[length++] = '4';
russianChars[length++] = '5';
russianChars[length++] = '6';
russianChars[length++] = '7';
russianChars[length++] = '8';
russianChars[length] = '9';
return russianChars;
}
}
{code}
was:
RussianAnalyzer's tokenizer skips numbers from input text, so that resulting
token stream miss numbers. Problem can be solved by adding numbers to
RussianCharsets.UnicodeRussian. See test case below for details.
{code:title=TestRussianAnalyzer.java|borderStyle=solid}
public class TestRussianAnalyzer extends TestCase {
Reader reader = new StringReader("text 1000");
public void testStemmer() {
testAnalyzer(new RussianAnalyzer());
}
public void testFixedRussianAnalyzer() {
testAnalyzer(new RussianAnalyzer(getRussianCharSet()));
}
private void testAnalyzer(RussianAnalyzer analyzer) {
try {
TokenStream stream = analyzer.tokenStream("text", reader);
assertEquals("text", stream.next().termText());
assertNotNull(stream.next());
} catch (IOException e) {
fail(e.getMessage());
}
}
private char[] getRussianCharSet() {
int length = RussianCharsets.UnicodeRussian.length;
final char[] russianChars = new char[length + 10];
System
.arraycopy(RussianCharsets.UnicodeRussian, 0, russianChars, 0, length);
russianChars[length++] = '0';
russianChars[length++] = '1';
russianChars[length++] = '2';
russianChars[length++] = '3';
russianChars[length++] = '4';
russianChars[length++] = '5';
russianChars[length++] = '6';
russianChars[length++] = '7';
russianChars[length++] = '8';
russianChars[length] = '9';
return russianChars;
}
}
{code}
> RussianAnalyzer's tokenizer skips numbers from input text,
> --
>
> Key: LUCENE-1003
> URL: https://issues.apache.org/jira/browse/LUCENE-1003
> Project: Lucene - Java
> Issue Type: Bug
> Components: Analysis
>Affects Versions: 2.2
>Reporter: TUSUR OpenTeam
>
> RussianAnalyzer's tokenizer skips numbers from input text, so that resulting
> token stream miss numbers. Problem can be solved by adding numbers to
> RussianCharsets.UnicodeRussian. See test case below for details.
> {code:title=TestRussianAnalyzer.java|borderStyle=solid}
> public class TestRussianAnalyzer extends TestCase {
> Reader reader = new StringReader("text 1000");
> // test FAILS
> public void testStemmer() {
> testAnalyzer(new RussianAnalyzer());
> }
> // test PASSES
> public void testFixedRussianAnalyzer() {
> testAnalyzer(new RussianAnalyzer(getRussianCharSet()));
> }
> private void testAnalyzer(RussianAnalyzer analyzer) {
> try {
> TokenStream stream = analyzer.tokenStream("text", reader);
> assertEquals("text", stream.next().termText());
> assertNotNull(stream.next());
> } catch (IOException e) {
> fail(e.getMessage());
> }
> }
> private char[] getRussianCharSet() {
> int length = RussianCharsets.UnicodeRussian.length;
> final char[] russianChars = new char[length + 10];
> System
> .arraycopy(RussianCharsets.UnicodeRussian, 0, russianChars, 0,
> length);
> russianChars[length++] = '0';
> russianChars[length++] = '1';
> russianChars[length++] = '2';
> russianChars[length++] = '3';
> russianChars[length++] = '4';
> russianChars[length++] = '5';
> rus
[jira] Created: (LUCENE-1003) RussianAnalyzer's tokenizer skips numbers from input text,
RussianAnalyzer's tokenizer skips numbers from input text,
--
Key: LUCENE-1003
URL: https://issues.apache.org/jira/browse/LUCENE-1003
Project: Lucene - Java
Issue Type: Bug
Components: Analysis
Affects Versions: 2.2
Reporter: TUSUR OpenTeam
RussianAnalyzer's tokenizer skips numbers from input text, so that resulting
token stream miss numbers. Problem can be solved by adding numbers to
RussianCharsets.UnicodeRussian. See test case below for details.
{code:title=TestRussianAnalyzer.java|borderStyle=solid}
public class TestRussianAnalyzer extends TestCase {
Reader reader = new StringReader("text 1000");
public void testStemmer() {
testAnalyzer(new RussianAnalyzer());
}
public void testFixedRussianAnalyzer() {
testAnalyzer(new RussianAnalyzer(getRussianCharSet()));
}
private void testAnalyzer(RussianAnalyzer analyzer) {
try {
TokenStream stream = analyzer.tokenStream("text", reader);
assertEquals("text", stream.next().termText());
assertNotNull(stream.next());
} catch (IOException e) {
fail(e.getMessage());
}
}
private char[] getRussianCharSet() {
int length = RussianCharsets.UnicodeRussian.length;
final char[] russianChars = new char[length + 10];
System
.arraycopy(RussianCharsets.UnicodeRussian, 0, russianChars, 0, length);
russianChars[length++] = '0';
russianChars[length++] = '1';
russianChars[length++] = '2';
russianChars[length++] = '3';
russianChars[length++] = '4';
russianChars[length++] = '5';
russianChars[length++] = '6';
russianChars[length++] = '7';
russianChars[length++] = '8';
russianChars[length] = '9';
return russianChars;
}
}
{code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Updated: (LUCENE-986) Refactor segmentInfos from IndexReader into its subclasses
[ https://issues.apache.org/jira/browse/LUCENE-986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Busch updated LUCENE-986: - Attachment: lucene-986.patch In addition to Hoss' changes this patch: * Removes the boolean directoryOwner variable from DirectoryIndexReader and checks for segmentInfos != null instead. I also added a comment to DirectoryIndexReader about this. * TestMultiReader now extends the new unit test TestMultiSegmentReader and overwrites the method openReader(). I'm planning to commit this in a day or so if nobody objects. > Refactor segmentInfos from IndexReader into its subclasses > -- > > Key: LUCENE-986 > URL: https://issues.apache.org/jira/browse/LUCENE-986 > Project: Lucene - Java > Issue Type: Improvement > Components: Index >Reporter: Michael Busch >Assignee: Michael Busch >Priority: Minor > Fix For: 2.3 > > Attachments: lucene-986.patch, lucene-986.patch, lucene-986.patch > > > References to segmentInfos in IndexReader cause different kinds of problems > for subclasses of IndexReader, like e. g. MultiReader. > Only subclasses of IndexReader that own the index directory, namely > SegmentReader and MultiSegmentReader, should have a SegmentInfos object > and be able to access it. > Further information: > http://www.gossamer-threads.com/lists/lucene/java-dev/51808 > http://www.gossamer-threads.com/lists/lucene/java-user/52460 > A part of the refactoring work was already done in LUCENE-781 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Commented: (LUCENE-584) Decouple Filter from BitSet
[
https://issues.apache.org/jira/browse/LUCENE-584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12528539
]
Paul Elschot commented on LUCENE-584:
-
The posted patch proposes to use this class to determine which documents should
be filtered:
public abstract class Matcher {
public abstract boolean next() throws IOException;
public abstract boolean skipTo(int target) throws IOException;
public abstract int doc();
// plus a few more methods
}
This class is then used as a superclass of org.apache.lucene.search.Scorer.
> Decouple Filter from BitSet
> ---
>
> Key: LUCENE-584
> URL: https://issues.apache.org/jira/browse/LUCENE-584
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Search
>Affects Versions: 2.0.1
>Reporter: Peter Schäfer
>Priority: Minor
> Attachments: bench-diff.txt, bench-diff.txt,
> Matcher-20070905-1ground.patch, Matcher-20070905-2default.patch,
> Matcher-20070905-3core.patch, Some Matchers.zip
>
>
> {code}
> package org.apache.lucene.search;
> public abstract class Filter implements java.io.Serializable
> {
> public abstract AbstractBitSet bits(IndexReader reader) throws IOException;
> }
> public interface AbstractBitSet
> {
> public boolean get(int index);
> }
> {code}
> It would be useful if the method =Filter.bits()= returned an abstract
> interface, instead of =java.util.BitSet=.
> Use case: there is a very large index, and, depending on the user's
> privileges, only a small portion of the index is actually visible.
> Sparsely populated =java.util.BitSet=s are not efficient and waste lots of
> memory. It would be desirable to have an alternative BitSet implementation
> with smaller memory footprint.
> Though it _is_ possibly to derive classes from =java.util.BitSet=, it was
> obviously not designed for that purpose.
> That's why I propose to use an interface instead. The default implementation
> could still delegate to =java.util.BitSet=.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Commented: (LUCENE-986) Refactor segmentInfos from IndexReader into its subclasses
[ https://issues.apache.org/jira/browse/LUCENE-986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12528522 ] Michael Busch commented on LUCENE-986: -- > I got the patch to apply cleanly (see mailing list for details) Thanks, Hoss! I'm using TortoiseSVN, I have to check how to set those default properties for new files correctly. >* just to clarify: IndexReader(Directory) is only around for > backwards compatibility in subclasses right? And the only reason > for the "private Directory" is to keep supporting the directory() > method for any subclasses relying on it? Yes, correct. >* IndexReader() says it can be removed once the other constructor is > removed ... why? is that just pointing out that we can rely on the > default constructor? Yes, just a suggested simplification. Keeping the constructor wouldn't hurt though. >* since one of the goals is that IndexReaders which don't own their > directory shouldn't utilize segmentInfos, would it make sense to > eliminate directoryOwner from DirectoryIndexReader and instead test > whether (null == segmentInfos) ? Sounds good, will do... >* the way TestMultiReader is setup with the "mode" is a bit confusing > ... perhaps instead we should make a subclass where only openReader > is overwritten (it will inherit the individual test methods) ? Yes, that's cleaner, will make that change as well. > here's the list of tweaks I made... The improvements look good to me. Thanks for the review, Hoss! I'll attach a new patch shortly. > Refactor segmentInfos from IndexReader into its subclasses > -- > > Key: LUCENE-986 > URL: https://issues.apache.org/jira/browse/LUCENE-986 > Project: Lucene - Java > Issue Type: Improvement > Components: Index >Reporter: Michael Busch >Assignee: Michael Busch >Priority: Minor > Fix For: 2.3 > > Attachments: lucene-986.patch, lucene-986.patch > > > References to segmentInfos in IndexReader cause different kinds of problems > for subclasses of IndexReader, like e. g. MultiReader. > Only subclasses of IndexReader that own the index directory, namely > SegmentReader and MultiSegmentReader, should have a SegmentInfos object > and be able to access it. > Further information: > http://www.gossamer-threads.com/lists/lucene/java-dev/51808 > http://www.gossamer-threads.com/lists/lucene/java-user/52460 > A part of the refactoring work was already done in LUCENE-781 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Resolved: (LUCENE-941) Benchmark alg line - {[AddDoc(4000)]: 4} : * - causes an infinite loop
[
https://issues.apache.org/jira/browse/LUCENE-941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Doron Cohen resolved LUCENE-941.
Resolution: Fixed
Lucene Fields: (was: [New])
fixed.
> Benchmark alg line - {[AddDoc(4000)]: 4} : * - causes an infinite loop
> ---
>
> Key: LUCENE-941
> URL: https://issues.apache.org/jira/browse/LUCENE-941
> Project: Lucene - Java
> Issue Type: Bug
> Components: contrib/benchmark
>Reporter: Doron Cohen
>Assignee: Doron Cohen
>Priority: Minor
> Fix For: 2.3
>
> Attachments: lucene-941-patch.txt, lucene-941-patch.txt
>
>
> Background in
> http://www.mail-archive.com/java-dev@lucene.apache.org/msg10831.html
> The line
>{[AddDoc(4000)]: 4} : *
> causes an infinite loop because the parallel sequence would mask the
> exhaustion from the outer sequential sequence.
> To fix this the DocMaker exhaustion check should be modified to rely on the
> doc maker instance only, and to be reset when the inputs are being reset.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Commented: (LUCENE-584) Decouple Filter from BitSet
[
https://issues.apache.org/jira/browse/LUCENE-584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12528420
]
robert engels commented on LUCENE-584:
--
I don't think
public interface AbstractBitSet
is according to standards.
It should just be
public interface BitSet
possibly
public interface IBitSet
if coming from the Windows world.
Since it is in a different package, there is no collision with the standard
BitSet class.
> Decouple Filter from BitSet
> ---
>
> Key: LUCENE-584
> URL: https://issues.apache.org/jira/browse/LUCENE-584
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Search
>Affects Versions: 2.0.1
>Reporter: Peter Schäfer
>Priority: Minor
> Attachments: bench-diff.txt, bench-diff.txt,
> Matcher-20070905-1ground.patch, Matcher-20070905-2default.patch,
> Matcher-20070905-3core.patch, Some Matchers.zip
>
>
> {code}
> package org.apache.lucene.search;
> public abstract class Filter implements java.io.Serializable
> {
> public abstract AbstractBitSet bits(IndexReader reader) throws IOException;
> }
> public interface AbstractBitSet
> {
> public boolean get(int index);
> }
> {code}
> It would be useful if the method =Filter.bits()= returned an abstract
> interface, instead of =java.util.BitSet=.
> Use case: there is a very large index, and, depending on the user's
> privileges, only a small portion of the index is actually visible.
> Sparsely populated =java.util.BitSet=s are not efficient and waste lots of
> memory. It would be desirable to have an alternative BitSet implementation
> with smaller memory footprint.
> Though it _is_ possibly to derive classes from =java.util.BitSet=, it was
> obviously not designed for that purpose.
> That's why I propose to use an interface instead. The default implementation
> could still delegate to =java.util.BitSet=.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Updated: (LUCENE-584) Decouple Filter from BitSet
[
https://issues.apache.org/jira/browse/LUCENE-584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Paul Elschot updated LUCENE-584:
Attachment: (was: Matcher5-contrib-queries-20070730.patch)
> Decouple Filter from BitSet
> ---
>
> Key: LUCENE-584
> URL: https://issues.apache.org/jira/browse/LUCENE-584
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Search
>Affects Versions: 2.0.1
>Reporter: Peter Schäfer
>Priority: Minor
> Attachments: bench-diff.txt, bench-diff.txt,
> Matcher-20070905-1ground.patch, Matcher-20070905-2default.patch,
> Matcher-20070905-3core.patch, Some Matchers.zip
>
>
> {code}
> package org.apache.lucene.search;
> public abstract class Filter implements java.io.Serializable
> {
> public abstract AbstractBitSet bits(IndexReader reader) throws IOException;
> }
> public interface AbstractBitSet
> {
> public boolean get(int index);
> }
> {code}
> It would be useful if the method =Filter.bits()= returned an abstract
> interface, instead of =java.util.BitSet=.
> Use case: there is a very large index, and, depending on the user's
> privileges, only a small portion of the index is actually visible.
> Sparsely populated =java.util.BitSet=s are not efficient and waste lots of
> memory. It would be desirable to have an alternative BitSet implementation
> with smaller memory footprint.
> Though it _is_ possibly to derive classes from =java.util.BitSet=, it was
> obviously not designed for that purpose.
> That's why I propose to use an interface instead. The default implementation
> could still delegate to =java.util.BitSet=.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Updated: (LUCENE-584) Decouple Filter from BitSet
[
https://issues.apache.org/jira/browse/LUCENE-584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Paul Elschot updated LUCENE-584:
Attachment: (was: Matcher3-core-20070730.patch)
> Decouple Filter from BitSet
> ---
>
> Key: LUCENE-584
> URL: https://issues.apache.org/jira/browse/LUCENE-584
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Search
>Affects Versions: 2.0.1
>Reporter: Peter Schäfer
>Priority: Minor
> Attachments: bench-diff.txt, bench-diff.txt,
> Matcher-20070905-1ground.patch, Matcher-20070905-2default.patch,
> Matcher-20070905-3core.patch, Some Matchers.zip
>
>
> {code}
> package org.apache.lucene.search;
> public abstract class Filter implements java.io.Serializable
> {
> public abstract AbstractBitSet bits(IndexReader reader) throws IOException;
> }
> public interface AbstractBitSet
> {
> public boolean get(int index);
> }
> {code}
> It would be useful if the method =Filter.bits()= returned an abstract
> interface, instead of =java.util.BitSet=.
> Use case: there is a very large index, and, depending on the user's
> privileges, only a small portion of the index is actually visible.
> Sparsely populated =java.util.BitSet=s are not efficient and waste lots of
> memory. It would be desirable to have an alternative BitSet implementation
> with smaller memory footprint.
> Though it _is_ possibly to derive classes from =java.util.BitSet=, it was
> obviously not designed for that purpose.
> That's why I propose to use an interface instead. The default implementation
> could still delegate to =java.util.BitSet=.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Updated: (LUCENE-584) Decouple Filter from BitSet
[
https://issues.apache.org/jira/browse/LUCENE-584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Paul Elschot updated LUCENE-584:
Attachment: (was: Matcher2-default-20070730.patch)
> Decouple Filter from BitSet
> ---
>
> Key: LUCENE-584
> URL: https://issues.apache.org/jira/browse/LUCENE-584
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Search
>Affects Versions: 2.0.1
>Reporter: Peter Schäfer
>Priority: Minor
> Attachments: bench-diff.txt, bench-diff.txt,
> Matcher-20070905-1ground.patch, Matcher-20070905-2default.patch,
> Matcher-20070905-3core.patch, Some Matchers.zip
>
>
> {code}
> package org.apache.lucene.search;
> public abstract class Filter implements java.io.Serializable
> {
> public abstract AbstractBitSet bits(IndexReader reader) throws IOException;
> }
> public interface AbstractBitSet
> {
> public boolean get(int index);
> }
> {code}
> It would be useful if the method =Filter.bits()= returned an abstract
> interface, instead of =java.util.BitSet=.
> Use case: there is a very large index, and, depending on the user's
> privileges, only a small portion of the index is actually visible.
> Sparsely populated =java.util.BitSet=s are not efficient and waste lots of
> memory. It would be desirable to have an alternative BitSet implementation
> with smaller memory footprint.
> Though it _is_ possibly to derive classes from =java.util.BitSet=, it was
> obviously not designed for that purpose.
> That's why I propose to use an interface instead. The default implementation
> could still delegate to =java.util.BitSet=.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Updated: (LUCENE-584) Decouple Filter from BitSet
[
https://issues.apache.org/jira/browse/LUCENE-584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Paul Elschot updated LUCENE-584:
Attachment: (was: Matcher6-contrib-xml-20070730.patch)
> Decouple Filter from BitSet
> ---
>
> Key: LUCENE-584
> URL: https://issues.apache.org/jira/browse/LUCENE-584
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Search
>Affects Versions: 2.0.1
>Reporter: Peter Schäfer
>Priority: Minor
> Attachments: bench-diff.txt, bench-diff.txt,
> Matcher-20070905-1ground.patch, Matcher-20070905-2default.patch,
> Matcher-20070905-3core.patch, Some Matchers.zip
>
>
> {code}
> package org.apache.lucene.search;
> public abstract class Filter implements java.io.Serializable
> {
> public abstract AbstractBitSet bits(IndexReader reader) throws IOException;
> }
> public interface AbstractBitSet
> {
> public boolean get(int index);
> }
> {code}
> It would be useful if the method =Filter.bits()= returned an abstract
> interface, instead of =java.util.BitSet=.
> Use case: there is a very large index, and, depending on the user's
> privileges, only a small portion of the index is actually visible.
> Sparsely populated =java.util.BitSet=s are not efficient and waste lots of
> memory. It would be desirable to have an alternative BitSet implementation
> with smaller memory footprint.
> Though it _is_ possibly to derive classes from =java.util.BitSet=, it was
> obviously not designed for that purpose.
> That's why I propose to use an interface instead. The default implementation
> could still delegate to =java.util.BitSet=.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Updated: (LUCENE-584) Decouple Filter from BitSet
[
https://issues.apache.org/jira/browse/LUCENE-584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Paul Elschot updated LUCENE-584:
Attachment: (was: Matcher1-ground-20070730.patch)
> Decouple Filter from BitSet
> ---
>
> Key: LUCENE-584
> URL: https://issues.apache.org/jira/browse/LUCENE-584
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Search
>Affects Versions: 2.0.1
>Reporter: Peter Schäfer
>Priority: Minor
> Attachments: bench-diff.txt, bench-diff.txt,
> Matcher-20070905-1ground.patch, Matcher-20070905-2default.patch,
> Matcher-20070905-3core.patch, Some Matchers.zip
>
>
> {code}
> package org.apache.lucene.search;
> public abstract class Filter implements java.io.Serializable
> {
> public abstract AbstractBitSet bits(IndexReader reader) throws IOException;
> }
> public interface AbstractBitSet
> {
> public boolean get(int index);
> }
> {code}
> It would be useful if the method =Filter.bits()= returned an abstract
> interface, instead of =java.util.BitSet=.
> Use case: there is a very large index, and, depending on the user's
> privileges, only a small portion of the index is actually visible.
> Sparsely populated =java.util.BitSet=s are not efficient and waste lots of
> memory. It would be desirable to have an alternative BitSet implementation
> with smaller memory footprint.
> Though it _is_ possibly to derive classes from =java.util.BitSet=, it was
> obviously not designed for that purpose.
> That's why I propose to use an interface instead. The default implementation
> could still delegate to =java.util.BitSet=.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Updated: (LUCENE-584) Decouple Filter from BitSet
[
https://issues.apache.org/jira/browse/LUCENE-584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Paul Elschot updated LUCENE-584:
Attachment: (was: Matcher4-contrib-misc-20070730.patch)
> Decouple Filter from BitSet
> ---
>
> Key: LUCENE-584
> URL: https://issues.apache.org/jira/browse/LUCENE-584
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Search
>Affects Versions: 2.0.1
>Reporter: Peter Schäfer
>Priority: Minor
> Attachments: bench-diff.txt, bench-diff.txt,
> Matcher-20070905-1ground.patch, Matcher-20070905-2default.patch,
> Matcher-20070905-3core.patch, Some Matchers.zip
>
>
> {code}
> package org.apache.lucene.search;
> public abstract class Filter implements java.io.Serializable
> {
> public abstract AbstractBitSet bits(IndexReader reader) throws IOException;
> }
> public interface AbstractBitSet
> {
> public boolean get(int index);
> }
> {code}
> It would be useful if the method =Filter.bits()= returned an abstract
> interface, instead of =java.util.BitSet=.
> Use case: there is a very large index, and, depending on the user's
> privileges, only a small portion of the index is actually visible.
> Sparsely populated =java.util.BitSet=s are not efficient and waste lots of
> memory. It would be desirable to have an alternative BitSet implementation
> with smaller memory footprint.
> Though it _is_ possibly to derive classes from =java.util.BitSet=, it was
> obviously not designed for that purpose.
> That's why I propose to use an interface instead. The default implementation
> could still delegate to =java.util.BitSet=.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
Fwd: BUILD for lucene (#315) on localhost was BROKEN: Script returned non-zero code "1"
I'm looking at this... Mike "Build Administrator" <[EMAIL PROTECTED]> wrote: > BUILD result > > > BUILD for lucene (#315) on localhost was BROKEN: Script returned non-zero > code "1" > > Build status: > > http://parabuild.viewtier.com:8080/parabuild/index.htm?view=detailed&buildid=1996 > > > Suspect Changes > > > Change(s) by mikemccand : > > - LUCENE-984: reduce TermVectorsWriter to just the merging use; >Time: 02:53 AM 09/18/2007; Change list: 576807 > > - LUCENE-845, LUCENE-847, LUCENE-870: factor MergePolicy & Mer; >Time: 02:27 AM 09/18/2007; Change list: 576798 > > Change(s) by doronc : > > - LUCENE-941: (leftover - add info in benchmark/CHANGES.txt en; >Time: 02:13 AM 09/18/2007; Change list: 576790 > > Change(s) Details: > > http://parabuild.viewtier.com:8080/parabuild/build/changes.htm?buildrunid=6038 > > BUILD logs > > > BUILD log: > http://parabuild.viewtier.com:8080/parabuild/build/log.htm?logid=8566 > Error Lines: > http://parabuild.viewtier.com:8080/parabuild/build/log.htm?logid=8567 > JUnit Tests: > http://parabuild.viewtier.com:8080/parabuild/build/log.htm?logid=8568 > > Log Lines Around Error > > > [junit] Tests run: 7, Failures: 0, Errors: 0, Time elapsed: 9.743 > sec > [junit] Testsuite: org.apache.lucene.search.TestThreadSafe > [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 26.839 > sec > [junit] Testsuite: org.apache.lucene.search.TestWildcard > [junit] Tests run: 5, Failures: 0, Errors: 0, Time elapsed: 2.198 > sec > [junit] Testsuite: > org.apache.lucene.search.function.TestCustomScoreQuery > [junit] Tests run: 4, Failures: 0, Errors: 0, Time elapsed: 2.414 > sec > [junit] Testsuite: > org.apache.lucene.search.function.TestFieldScoreQuery > [junit] Tests run: 12, Failures: 0, Errors: 0, Time elapsed: 2.501 > sec > [junit] Testsuite: org.apache.lucene.search.function.TestOrdValues > [junit] Tests run: 5, Failures: 0, Errors: 0, Time elapsed: 1.942 > sec > [junit] Testsuite: > org.apache.lucene.search.payloads.TestBoostingTermQuery > [junit] Tests run: 4, Failures: 0, Errors: 0, Time elapsed: 6.879 > sec > [junit] Testsuite: org.apache.lucene.search.spans.TestBasics > [junit] Tests run: 19, Failures: 0, Errors: 0, Time elapsed: 29.218 > sec > [junit] Testsuite: > org.apache.lucene.search.spans.TestNearSpansOrdered > [junit] Tests run: 10, Failures: 0, Errors: 0, Time elapsed: 2 sec > [junit] Testsuite: > org.apache.lucene.search.spans.TestSpanExplanations > [junit] Tests run: 31, Failures: 0, Errors: 0, Time elapsed: 2.974 > sec > [junit] Testsuite: > org.apache.lucene.search.spans.TestSpanExplanationsOfNonMatches > [junit] Tests run: 31, Failures: 0, Errors: 0, Time elapsed: 2.504 > sec > [junit] Testsuite: org.apache.lucene.search.spans.TestSpans > [junit] Tests run: 21, Failures: 0, Errors: 0, Time elapsed: 2.765 > sec > [junit] Testsuite: org.apache.lucene.search.spans.TestSpansAdvanced > [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 1.694 > sec > [junit] Testsuite: > org.apache.lucene.search.spans.TestSpansAdvanced2 > [junit] Tests run: 4, Failures: 0, Errors: 0, Time elapsed: 1.89 > sec > [junit] Testsuite: org.apache.lucene.store.TestBufferedIndexInput > [junit] Tests run: 4, Failures: 0, Errors: 0, Time elapsed: 5.711 > sec > [junit] Testsuite: org.apache.lucene.store.TestHugeRamFile > [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 16.99 > sec > [junit] Testsuite: org.apache.lucene.store.TestLock > [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 1.592 > sec > [junit] Testsuite: org.apache.lucene.store.TestLockFactory > [junit] Tests run: 13, Failures: 0, Errors: 0, Time elapsed: 12.777 > sec > [junit] Testsuite: org.apache.lucene.store.TestMMapDirectory > [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 1.038 > sec > [junit] Testsuite: org.apache.lucene.store.TestWindowsMMap > [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 4.521 > sec > [junit] Testsuite: org.apache.lucene.util.TestBitVector > [junit] Tests run: 6, Failures: 0, Errors: 0, Time elapsed: 6.877 > sec > [junit] Testsuite: org.apache.lucene.util.TestPriorityQueue > [junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 1.165 > sec > [junit] Testsuite: org.apache.lucene.util.TestSmallFloat > [junit] Tests run: 2, Failures: 0,
[jira] Resolved: (LUCENE-984) remove TermVectorsWriter (it's no longer used)
[ https://issues.apache.org/jira/browse/LUCENE-984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael McCandless resolved LUCENE-984. --- Resolution: Fixed > remove TermVectorsWriter (it's no longer used) > -- > > Key: LUCENE-984 > URL: https://issues.apache.org/jira/browse/LUCENE-984 > Project: Lucene - Java > Issue Type: Bug > Components: Index >Affects Versions: 2.3 >Reporter: Michael McCandless >Assignee: Michael McCandless >Priority: Trivial > Fix For: 2.3 > > Attachments: LUCENE-984.patch > > > We should remove TermVectorsWriter: it's no longer used now that > DocumentsWriter writes the term vectors directly to the index. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Resolved: (LUCENE-870) add concurrent merge policy
[ https://issues.apache.org/jira/browse/LUCENE-870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael McCandless resolved LUCENE-870. --- Resolution: Fixed > add concurrent merge policy > --- > > Key: LUCENE-870 > URL: https://issues.apache.org/jira/browse/LUCENE-870 > Project: Lucene - Java > Issue Type: New Feature > Components: Index >Reporter: Steven Parkes >Assignee: Steven Parkes > Fix For: 2.3 > > Attachments: CMP.patch.txt, concurrentMerge.patch, > LUCENE-870.take2.patch, LUCENE-870.take3.patch > > > Provide the ability to handle merges in one or more concurrent threads, i.e., > concurrent with other IndexWriter operations. > I'm factoring the code from LUCENE-847 for this. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Resolved: (LUCENE-847) Factor merge policy out of IndexWriter
[ https://issues.apache.org/jira/browse/LUCENE-847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael McCandless resolved LUCENE-847. --- Resolution: Fixed > Factor merge policy out of IndexWriter > -- > > Key: LUCENE-847 > URL: https://issues.apache.org/jira/browse/LUCENE-847 > Project: Lucene - Java > Issue Type: Improvement > Components: Index >Reporter: Steven Parkes >Assignee: Steven Parkes > Fix For: 2.3 > > Attachments: concurrentMerge.patch, LUCENE-847.patch.txt, > LUCENE-847.patch.txt, LUCENE-847.take3.patch, LUCENE-847.take4.patch, > LUCENE-847.take5.patch, LUCENE-847.take6.patch, LUCENE-847.take7.patch, > LUCENE-847.take8.patch, LUCENE-847.txt > > > If we factor the merge policy out of IndexWriter, we can make it pluggable, > making it possible for apps to choose a custom merge policy and for easier > experimenting with merge policy variants. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Resolved: (LUCENE-845) If you "flush by RAM usage" then IndexWriter may over-merge
[ https://issues.apache.org/jira/browse/LUCENE-845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael McCandless resolved LUCENE-845. --- Resolution: Fixed > If you "flush by RAM usage" then IndexWriter may over-merge > --- > > Key: LUCENE-845 > URL: https://issues.apache.org/jira/browse/LUCENE-845 > Project: Lucene - Java > Issue Type: Bug > Components: Index >Affects Versions: 2.1 >Reporter: Michael McCandless >Assignee: Michael McCandless >Priority: Minor > Fix For: 2.3 > > Attachments: LUCENE-845.patch > > > I think a good way to maximize performance of Lucene's indexing for a > given amount of RAM is to flush (writer.flush()) the added documents > whenever the RAM usage (writer.ramSizeInBytes()) has crossed the max > RAM you can afford. > But, this can confuse the merge policy and cause over-merging, unless > you set maxBufferedDocs properly. > This is because the merge policy looks at the current maxBufferedDocs > to figure out which segments are level 0 (first flushed) or level 1 > (merged from level 0 segments). > I'm not sure how to fix this. Maybe we can look at net size (bytes) > of a segment and "infer" level from this? Still we would have to be > resilient to the application suddenly increasing the RAM allowed. > The good news is to workaround this bug I think you just need to > ensure that your maxBufferedDocs is less than mergeFactor * > typical-number-of-docs-flushed. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Commented: (LUCENE-941) Benchmark alg line - {[AddDoc(4000)]: 4} : * - causes an infinite loop
[
https://issues.apache.org/jira/browse/LUCENE-941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12528313
]
Doron Cohen commented on LUCENE-941:
Two notes on current patch:
- Two ResetSystem Tasks now subclass ResetInputsTask.
- Unlike initially intended, fix is mainly in TaskSequence, not relying on
DocMakers, so that it can easily be extended to work with other "makers".
> Benchmark alg line - {[AddDoc(4000)]: 4} : * - causes an infinite loop
> ---
>
> Key: LUCENE-941
> URL: https://issues.apache.org/jira/browse/LUCENE-941
> Project: Lucene - Java
> Issue Type: Bug
> Components: contrib/benchmark
>Reporter: Doron Cohen
>Assignee: Doron Cohen
>Priority: Minor
> Fix For: 2.3
>
> Attachments: lucene-941-patch.txt, lucene-941-patch.txt
>
>
> Background in
> http://www.mail-archive.com/java-dev@lucene.apache.org/msg10831.html
> The line
>{[AddDoc(4000)]: 4} : *
> causes an infinite loop because the parallel sequence would mask the
> exhaustion from the outer sequential sequence.
> To fix this the DocMaker exhaustion check should be modified to rely on the
> doc maker instance only, and to be reset when the inputs are being reset.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Updated: (LUCENE-986) Refactor segmentInfos from IndexReader into its subclasses
[ https://issues.apache.org/jira/browse/LUCENE-986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hoss Man updated LUCENE-986: Attachment: lucene-986.patch I got the patch to apply cleanly (see mailing list for details) On the whole it looks really good, i'm attaching an updated version with some minor improvements (mainly javadocs), but first a few questions... * just to clarify: IndexReader(Directory) is only around for backwards compatibility in subclasses right? And the only reason for the "private Directory" is to keep supporting the directory() method for any subclasses relying on it? * IndexReader() says it can be removed once the other constructor is removed ... why? is that just pointing out that we can rely on the default constructor? * since one of the goals is that IndexReaders which don't own their directory shouldn't utilize segmentInfos, would it make sense to eliminate directoryOwner from DirectoryIndexReader and instead test whether (null == segmentInfos) ? * the way TestMultiReader is setup with the "mode" is a bit confusing ... perhaps instead we should make a subclass where only openReader is overwritten (it will inherit the individual test methods) ? here's the list of tweaks I made... * added a note in the IndexReader class javadocs about methods that throw UnSupOp but aren't abstract. * added javadocs to the IndexReader(Directory) constructor based on my understanding * added back javadocs to IndexReader methods that now throw UnSupOp to make it clear what they do to callers looking at the API, but made it clear tthe default impls throw UnSupOp * made IndexReader.directory() throw UnSupOp if directory is null, enhanced it's javadocs. * /* NOOP */ in IndexReader.acquireWriteLock so it's clear to code analysis tools that it's not a mistake. * small additions to javadocs for DirectoryIndexReader class, but these should probably be elaborated on (depending on what you think of my segmentInfos==null idea above) * made DirectoryIndexReader(...) call init(...) to eliminate a small about of duplication. > Refactor segmentInfos from IndexReader into its subclasses > -- > > Key: LUCENE-986 > URL: https://issues.apache.org/jira/browse/LUCENE-986 > Project: Lucene - Java > Issue Type: Improvement > Components: Index >Reporter: Michael Busch >Assignee: Michael Busch >Priority: Minor > Fix For: 2.3 > > Attachments: lucene-986.patch, lucene-986.patch > > > References to segmentInfos in IndexReader cause different kinds of problems > for subclasses of IndexReader, like e. g. MultiReader. > Only subclasses of IndexReader that own the index directory, namely > SegmentReader and MultiSegmentReader, should have a SegmentInfos object > and be able to access it. > Further information: > http://www.gossamer-threads.com/lists/lucene/java-dev/51808 > http://www.gossamer-threads.com/lists/lucene/java-user/52460 > A part of the refactoring work was already done in LUCENE-781 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
