[jira] [Commented] (LUCENE-3234) Provide limit on phrase analysis in FastVectorHighlighter
[ https://issues.apache.org/jira/browse/LUCENE-3234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13114484#comment-13114484 ] Koji Sekiguchi commented on LUCENE-3234: Ok, thank you, David. I opened SOLR-2794. > Provide limit on phrase analysis in FastVectorHighlighter > - > > Key: LUCENE-3234 > URL: https://issues.apache.org/jira/browse/LUCENE-3234 > Project: Lucene - Java > Issue Type: Improvement >Affects Versions: 2.9.4, 3.0.3, 3.1, 3.2, 3.3 >Reporter: Mike Sokolov >Assignee: Koji Sekiguchi > Fix For: 3.4, 4.0 > > Attachments: LUCENE-3234.patch, LUCENE-3234.patch, LUCENE-3234.patch, > LUCENE-3234.patch, LUCENE-3234.patch > > > With larger documents, FVH can spend a lot of time trying to find the > best-scoring snippet as it examines every possible phrase formed from > matching terms in the document. If one is willing to accept > less-than-perfect scoring by limiting the number of phrases that are > examined, substantial speedups are possible. This is analogous to the > Highlighter limit on the number of characters to analyze. > The patch includes an artifical test case that shows > 1000x speedup. In a > more normal test environment, with English documents and random queries, I am > seeing speedups of around 3-10x when setting phraseLimit=1, which has the > effect of selecting the first possible snippet in the document. Most of our > sites operate in this way (just show the first snippet), so this would be a > big win for us. > With phraseLimit = -1, you get the existing FVH behavior. At larger values of > phraseLimit, you may not get substantial speedup in the normal case, but you > do get the benefit of protection against blow-up in pathological cases. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Created] (SOLR-2794) change the default of hl.phraseLimit to 5000
change the default of hl.phraseLimit to 5000 Key: SOLR-2794 URL: https://issues.apache.org/jira/browse/SOLR-2794 Project: Solr Issue Type: Improvement Components: highlighter Affects Versions: 3.4, 4.0 Reporter: Koji Sekiguchi Priority: Trivial David Smiley suggested that the default of hl.phraseLimit parameter should be 5000 (5000 was suggested by Mike). See LUCENE-3234. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
Re: [JENKINS] Lucene-Solr-tests-only-trunk-java7 - Build # 511 - Failure
Fixed in revision 1175685. On Mon, Sep 26, 2011 at 6:59 PM, Chris Male wrote: > Can replicate. > > > On Mon, Sep 26, 2011 at 6:55 PM, Chris Male wrote: > >> This didn't fail locally, investigating. >> >> >> On Mon, Sep 26, 2011 at 6:40 PM, Apache Jenkins Server < >> jenk...@builds.apache.org> wrote: >> >>> Build: >>> https://builds.apache.org/job/Lucene-Solr-tests-only-trunk-java7/511/ >>> >>> 1 tests failed. >>> REGRESSION: >>> org.apache.lucene.index.TestLongPostings.testLongPostingsNoPositions >>> >>> Error Message: >>> end() called before incrementToken() returned false! >>> >>> Stack Trace: >>> junit.framework.AssertionFailedError: end() called before >>> incrementToken() returned false! >>>at >>> org.apache.lucene.util.LuceneTestCaseRunner.runChild(LuceneTestCaseRunner.java:148) >>>at >>> org.apache.lucene.util.LuceneTestCaseRunner.runChild(LuceneTestCaseRunner.java:50) >>>at >>> org.apache.lucene.analysis.MockTokenizer.end(MockTokenizer.java:193) >>>at org.apache.lucene.analysis.TokenFilter.end(TokenFilter.java:42) >>>at >>> org.apache.lucene.index.TestLongPostings.getRandomTerm(TestLongPostings.java:62) >>>at >>> org.apache.lucene.index.TestLongPostings.doTestLongPostingsNoPositions(TestLongPostings.java:278) >>>at >>> org.apache.lucene.index.TestLongPostings.testLongPostingsNoPositions(TestLongPostings.java:261) >>>at >>> org.apache.lucene.util.LuceneTestCase$2$1.evaluate(LuceneTestCase.java:611) >>> >>> >>> >>> >>> Build Log (for compile errors): >>> [...truncated 1897 lines...] >>> >>> >>> >>> - >>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org >>> For additional commands, e-mail: dev-h...@lucene.apache.org >>> >>> >> >> >> -- >> Chris Male | Software Developer | JTeam BV.| www.jteam.nl >> > > > > -- > Chris Male | Software Developer | JTeam BV.| www.jteam.nl > -- Chris Male | Software Developer | JTeam BV.| www.jteam.nl
Re: [JENKINS] Lucene-Solr-tests-only-trunk-java7 - Build # 511 - Failure
Can replicate. On Mon, Sep 26, 2011 at 6:55 PM, Chris Male wrote: > This didn't fail locally, investigating. > > > On Mon, Sep 26, 2011 at 6:40 PM, Apache Jenkins Server < > jenk...@builds.apache.org> wrote: > >> Build: >> https://builds.apache.org/job/Lucene-Solr-tests-only-trunk-java7/511/ >> >> 1 tests failed. >> REGRESSION: >> org.apache.lucene.index.TestLongPostings.testLongPostingsNoPositions >> >> Error Message: >> end() called before incrementToken() returned false! >> >> Stack Trace: >> junit.framework.AssertionFailedError: end() called before incrementToken() >> returned false! >>at >> org.apache.lucene.util.LuceneTestCaseRunner.runChild(LuceneTestCaseRunner.java:148) >>at >> org.apache.lucene.util.LuceneTestCaseRunner.runChild(LuceneTestCaseRunner.java:50) >>at >> org.apache.lucene.analysis.MockTokenizer.end(MockTokenizer.java:193) >>at org.apache.lucene.analysis.TokenFilter.end(TokenFilter.java:42) >>at >> org.apache.lucene.index.TestLongPostings.getRandomTerm(TestLongPostings.java:62) >>at >> org.apache.lucene.index.TestLongPostings.doTestLongPostingsNoPositions(TestLongPostings.java:278) >>at >> org.apache.lucene.index.TestLongPostings.testLongPostingsNoPositions(TestLongPostings.java:261) >>at >> org.apache.lucene.util.LuceneTestCase$2$1.evaluate(LuceneTestCase.java:611) >> >> >> >> >> Build Log (for compile errors): >> [...truncated 1897 lines...] >> >> >> >> - >> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org >> For additional commands, e-mail: dev-h...@lucene.apache.org >> >> > > > -- > Chris Male | Software Developer | JTeam BV.| www.jteam.nl > -- Chris Male | Software Developer | JTeam BV.| www.jteam.nl
Re: [JENKINS] Lucene-Solr-tests-only-trunk-java7 - Build # 511 - Failure
This didn't fail locally, investigating. On Mon, Sep 26, 2011 at 6:40 PM, Apache Jenkins Server < jenk...@builds.apache.org> wrote: > Build: > https://builds.apache.org/job/Lucene-Solr-tests-only-trunk-java7/511/ > > 1 tests failed. > REGRESSION: > org.apache.lucene.index.TestLongPostings.testLongPostingsNoPositions > > Error Message: > end() called before incrementToken() returned false! > > Stack Trace: > junit.framework.AssertionFailedError: end() called before incrementToken() > returned false! >at > org.apache.lucene.util.LuceneTestCaseRunner.runChild(LuceneTestCaseRunner.java:148) >at > org.apache.lucene.util.LuceneTestCaseRunner.runChild(LuceneTestCaseRunner.java:50) >at > org.apache.lucene.analysis.MockTokenizer.end(MockTokenizer.java:193) >at org.apache.lucene.analysis.TokenFilter.end(TokenFilter.java:42) >at > org.apache.lucene.index.TestLongPostings.getRandomTerm(TestLongPostings.java:62) >at > org.apache.lucene.index.TestLongPostings.doTestLongPostingsNoPositions(TestLongPostings.java:278) >at > org.apache.lucene.index.TestLongPostings.testLongPostingsNoPositions(TestLongPostings.java:261) >at > org.apache.lucene.util.LuceneTestCase$2$1.evaluate(LuceneTestCase.java:611) > > > > > Build Log (for compile errors): > [...truncated 1897 lines...] > > > > - > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org > For additional commands, e-mail: dev-h...@lucene.apache.org > > -- Chris Male | Software Developer | JTeam BV.| www.jteam.nl
[jira] [Commented] (LUCENE-3234) Provide limit on phrase analysis in FastVectorHighlighter
[ https://issues.apache.org/jira/browse/LUCENE-3234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13114472#comment-13114472 ] David Smiley commented on LUCENE-3234: -- Koji; you didn't react to my comment that an app would be incredibly unlikely to actually depend on the unlimited behavior; thus there is no back-compat. Again, I think it should be 5000. > Provide limit on phrase analysis in FastVectorHighlighter > - > > Key: LUCENE-3234 > URL: https://issues.apache.org/jira/browse/LUCENE-3234 > Project: Lucene - Java > Issue Type: Improvement >Affects Versions: 2.9.4, 3.0.3, 3.1, 3.2, 3.3 >Reporter: Mike Sokolov >Assignee: Koji Sekiguchi > Fix For: 3.4, 4.0 > > Attachments: LUCENE-3234.patch, LUCENE-3234.patch, LUCENE-3234.patch, > LUCENE-3234.patch, LUCENE-3234.patch > > > With larger documents, FVH can spend a lot of time trying to find the > best-scoring snippet as it examines every possible phrase formed from > matching terms in the document. If one is willing to accept > less-than-perfect scoring by limiting the number of phrases that are > examined, substantial speedups are possible. This is analogous to the > Highlighter limit on the number of characters to analyze. > The patch includes an artifical test case that shows > 1000x speedup. In a > more normal test environment, with English documents and random queries, I am > seeing speedups of around 3-10x when setting phraseLimit=1, which has the > effect of selecting the first possible snippet in the document. Most of our > sites operate in this way (just show the first snippet), so this would be a > big win for us. > With phraseLimit = -1, you get the existing FVH behavior. At larger values of > phraseLimit, you may not get substantial speedup in the normal case, but you > do get the benefit of protection against blow-up in pathological cases. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[JENKINS] Lucene-Solr-tests-only-trunk-java7 - Build # 511 - Failure
Build: https://builds.apache.org/job/Lucene-Solr-tests-only-trunk-java7/511/ 1 tests failed. REGRESSION: org.apache.lucene.index.TestLongPostings.testLongPostingsNoPositions Error Message: end() called before incrementToken() returned false! Stack Trace: junit.framework.AssertionFailedError: end() called before incrementToken() returned false! at org.apache.lucene.util.LuceneTestCaseRunner.runChild(LuceneTestCaseRunner.java:148) at org.apache.lucene.util.LuceneTestCaseRunner.runChild(LuceneTestCaseRunner.java:50) at org.apache.lucene.analysis.MockTokenizer.end(MockTokenizer.java:193) at org.apache.lucene.analysis.TokenFilter.end(TokenFilter.java:42) at org.apache.lucene.index.TestLongPostings.getRandomTerm(TestLongPostings.java:62) at org.apache.lucene.index.TestLongPostings.doTestLongPostingsNoPositions(TestLongPostings.java:278) at org.apache.lucene.index.TestLongPostings.testLongPostingsNoPositions(TestLongPostings.java:261) at org.apache.lucene.util.LuceneTestCase$2$1.evaluate(LuceneTestCase.java:611) Build Log (for compile errors): [...truncated 1897 lines...] - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (LUCENE-3455) All Analysis Consumers should use reusableTokenStream
[ https://issues.apache.org/jira/browse/LUCENE-3455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Male updated LUCENE-3455: --- Attachment: LUCENE-3455.patch Patch which converts the last consumers over to using reusableTokenStream(). Couple of changes to the classic QP included in this. Rather than silently returning null on an IOException, a ParseException is now thrown. Equally, where before the IOException was simply ignored, an Exception is thrown. Going to commit this shortly and then change reusableTokenStream() to tokenStream(). > All Analysis Consumers should use reusableTokenStream > - > > Key: LUCENE-3455 > URL: https://issues.apache.org/jira/browse/LUCENE-3455 > Project: Lucene - Java > Issue Type: Sub-task > Components: modules/analysis >Reporter: Chris Male >Assignee: Chris Male > Attachments: LUCENE-3455-test-consumers.patch, > LUCENE-3455-test-consumers.patch, LUCENE-3455.patch > > > With Analyzer now using TokenStreamComponents, theres no reason for Analysis > consumers to use tokenStream() (it just gives bad performance). Consequently > all consumers will be moved over to using reusableTokenStream(). The only > challenge here is that reusableTokenStream throws an IOException which many > consumers are not rigged to deal with. > Once all consumers have been moved, we can rename reusableTokenStream() back > to tokenStream(). -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-3455) All Analysis Consumers should use reusableTokenStream
[ https://issues.apache.org/jira/browse/LUCENE-3455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13114458#comment-13114458 ] Chris Male commented on LUCENE-3455: Committed revision 1175670. > All Analysis Consumers should use reusableTokenStream > - > > Key: LUCENE-3455 > URL: https://issues.apache.org/jira/browse/LUCENE-3455 > Project: Lucene - Java > Issue Type: Sub-task > Components: modules/analysis >Reporter: Chris Male >Assignee: Chris Male > Attachments: LUCENE-3455-test-consumers.patch, > LUCENE-3455-test-consumers.patch > > > With Analyzer now using TokenStreamComponents, theres no reason for Analysis > consumers to use tokenStream() (it just gives bad performance). Consequently > all consumers will be moved over to using reusableTokenStream(). The only > challenge here is that reusableTokenStream throws an IOException which many > consumers are not rigged to deal with. > Once all consumers have been moved, we can rename reusableTokenStream() back > to tokenStream(). -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
Re: [jira] [Commented] (LUCENE-3454) rename optimize to a less cool-sounding name
One note. changing the name to something besides optimize *may* cause people to actually look before they use it. I mean, I *know* what optimize means, and it is good. Unfortunately what I know about the term isn't what it does, so changing the name to SqueezeTheCharmin will at least have a chance of making me dig deeper before just using it. FWIW Erick Who here remembers Mr. Whipple? On Sun, Sep 25, 2011 at 4:27 PM, Mike Sokolov (JIRA) wrote: > > [ > https://issues.apache.org/jira/browse/LUCENE-3454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13114391#comment-13114391 > ] > > Mike Sokolov commented on LUCENE-3454: > -- > > squeeze()? You apply some pressure - maybe it merges, and maybe it doesn't. > Depending how hard you squeeze. squeeze(1) is squeezing harder than > squeeze(10), which is a bit weird, but that's just down to units of > squishiness. > >> rename optimize to a less cool-sounding name >> >> >> Key: LUCENE-3454 >> URL: https://issues.apache.org/jira/browse/LUCENE-3454 >> Project: Lucene - Java >> Issue Type: Improvement >> Affects Versions: 4.0 >> Reporter: Robert Muir >> >> I think users see the name optimize and feel they must do this, because who >> wants a suboptimal system? but this probably just results in wasted time and >> resources. >> maybe rename to collapseSegments or something? > > -- > This message is automatically generated by JIRA. > For more information on JIRA, see: http://www.atlassian.com/software/jira > > > > - > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org > For additional commands, e-mail: dev-h...@lucene.apache.org > > - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[JENKINS] Lucene-trunk - Build # 1685 - Still Failing
Build: https://builds.apache.org/job/Lucene-trunk/1685/ 1 tests failed. REGRESSION: org.apache.lucene.util.automaton.TestCompiledAutomaton.testRandom Error Message: Java heap space Stack Trace: java.lang.OutOfMemoryError: Java heap space at org.apache.lucene.util.automaton.RunAutomaton.(RunAutomaton.java:128) at org.apache.lucene.util.automaton.ByteRunAutomaton.(ByteRunAutomaton.java:28) at org.apache.lucene.util.automaton.CompiledAutomaton.(CompiledAutomaton.java:134) at org.apache.lucene.util.automaton.TestCompiledAutomaton.build(TestCompiledAutomaton.java:39) at org.apache.lucene.util.automaton.TestCompiledAutomaton.testTerms(TestCompiledAutomaton.java:55) at org.apache.lucene.util.automaton.TestCompiledAutomaton.testRandom(TestCompiledAutomaton.java:101) at org.apache.lucene.util.LuceneTestCase$2$1.evaluate(LuceneTestCase.java:611) at org.apache.lucene.util.LuceneTestCaseRunner.runChild(LuceneTestCaseRunner.java:148) at org.apache.lucene.util.LuceneTestCaseRunner.runChild(LuceneTestCaseRunner.java:50) Build Log (for compile errors): [...truncated 12907 lines...] - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-3456) Analysis Consumers should end and close any TokenStreams
[ https://issues.apache.org/jira/browse/LUCENE-3456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1311#comment-1311 ] Robert Muir commented on LUCENE-3456: - ok, i committed my stuff, just test bugs (ill backport though)... go ahead! > Analysis Consumers should end and close any TokenStreams > > > Key: LUCENE-3456 > URL: https://issues.apache.org/jira/browse/LUCENE-3456 > Project: Lucene - Java > Issue Type: Sub-task >Reporter: Chris Male > Attachments: LUCENE-3456.patch, LUCENE-3456.patch > > > While converting consumers over to using reusableTokenStream, I notice many > don't call end() or close() on TokenStreams. > Even if they are test TSs only created once, we should follow the defined > usage pattern. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-2793) A SolrIndexSearcher can be left open if the executor rejects a task.
[ https://issues.apache.org/jira/browse/SOLR-2793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13114443#comment-13114443 ] Mark Miller commented on SOLR-2793: --- And of course this should be done for the other submits...just hit this same issue on a different submit call in the autocommit test... > A SolrIndexSearcher can be left open if the executor rejects a task. > > > Key: SOLR-2793 > URL: https://issues.apache.org/jira/browse/SOLR-2793 > Project: Solr > Issue Type: Bug >Reporter: Mark Miller >Assignee: Mark Miller > Fix For: 3.5, 4.0 > > Attachments: SOLR-2793.patch > > > This is starting to really bug me because tests almost never pass on my linux > box due to this issue. Time to fix it. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-2793) A SolrIndexSearcher can be left open if the executor rejects a task.
[ https://issues.apache.org/jira/browse/SOLR-2793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13114442#comment-13114442 ] Mark Miller commented on SOLR-2793: --- As I'm stressing tests to track down other failures, I've learned of another subtlety in this - in some cases you need to decrement twice on error. > A SolrIndexSearcher can be left open if the executor rejects a task. > > > Key: SOLR-2793 > URL: https://issues.apache.org/jira/browse/SOLR-2793 > Project: Solr > Issue Type: Bug >Reporter: Mark Miller >Assignee: Mark Miller > Fix For: 3.5, 4.0 > > Attachments: SOLR-2793.patch > > > This is starting to really bug me because tests almost never pass on my linux > box due to this issue. Time to fix it. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-3456) Analysis Consumers should end and close any TokenStreams
[ https://issues.apache.org/jira/browse/LUCENE-3456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13114438#comment-13114438 ] Chris Male commented on LUCENE-3456: Alright I'll stick to Lucene and modules for a bit. > Analysis Consumers should end and close any TokenStreams > > > Key: LUCENE-3456 > URL: https://issues.apache.org/jira/browse/LUCENE-3456 > Project: Lucene - Java > Issue Type: Sub-task >Reporter: Chris Male > Attachments: LUCENE-3456.patch, LUCENE-3456.patch > > > While converting consumers over to using reusableTokenStream, I notice many > don't call end() or close() on TokenStreams. > Even if they are test TSs only created once, we should follow the defined > usage pattern. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-3456) Analysis Consumers should end and close any TokenStreams
[ https://issues.apache.org/jira/browse/LUCENE-3456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13114437#comment-13114437 ] Robert Muir commented on LUCENE-3456: - i have a new patch i made on the airplane, but it found more bugs... ill fix them tonight and commit! > Analysis Consumers should end and close any TokenStreams > > > Key: LUCENE-3456 > URL: https://issues.apache.org/jira/browse/LUCENE-3456 > Project: Lucene - Java > Issue Type: Sub-task >Reporter: Chris Male > Attachments: LUCENE-3456.patch, LUCENE-3456.patch > > > While converting consumers over to using reusableTokenStream, I notice many > don't call end() or close() on TokenStreams. > Even if they are test TSs only created once, we should follow the defined > usage pattern. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-3456) Analysis Consumers should end and close any TokenStreams
[ https://issues.apache.org/jira/browse/LUCENE-3456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13114436#comment-13114436 ] Chris Male commented on LUCENE-3456: Where are we at with this Robert? I'm about to complete converting over to reusableTokenStream() in Solr. Should I work on that or wait for you to finish here? > Analysis Consumers should end and close any TokenStreams > > > Key: LUCENE-3456 > URL: https://issues.apache.org/jira/browse/LUCENE-3456 > Project: Lucene - Java > Issue Type: Sub-task >Reporter: Chris Male > Attachments: LUCENE-3456.patch, LUCENE-3456.patch > > > While converting consumers over to using reusableTokenStream, I notice many > don't call end() or close() on TokenStreams. > Even if they are test TSs only created once, we should follow the defined > usage pattern. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-1895) ManifoldCF SearchComponent plugin for enforcing ManifoldCF security at search time
[ https://issues.apache.org/jira/browse/SOLR-1895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karl Wright updated SOLR-1895: -- Attachment: SOLR-1895-queries.patch Attached SOLR-1895-queries.patch, for query version of search component. > ManifoldCF SearchComponent plugin for enforcing ManifoldCF security at search > time > -- > > Key: SOLR-1895 > URL: https://issues.apache.org/jira/browse/SOLR-1895 > Project: Solr > Issue Type: New Feature > Components: SearchComponents - other >Reporter: Karl Wright > Labels: document, security, solr > Fix For: 3.5, 4.0 > > Attachments: LCFSecurityFilter.java, LCFSecurityFilter.java, > LCFSecurityFilter.java, LCFSecurityFilter.java, SOLR-1895-queries.patch, > SOLR-1895-service-plugin.patch, SOLR-1895-service-plugin.patch, > SOLR-1895.patch, SOLR-1895.patch, SOLR-1895.patch, SOLR-1895.patch, > SOLR-1895.patch, SOLR-1895.patch > > > I've written an LCF SearchComponent which filters returned results based on > access tokens provided by LCF's authority service. The component requires > you to configure the appropriate authority service URL base, e.g.: > > > name="AuthorityServiceBaseURL">http://localhost:8080/lcf-authority-service > > Also required are the following schema.xml additions: > > stored="false" multiValued="true"/> > stored="false" multiValued="true"/> > stored="false" multiValued="true"/> > multiValued="true"/> > Finally, to tie it into the standard request handler, it seems to need to run > last: > > > lcfSecurity > > ... > I have not set a package for this code. Nor have I been able to get it > reviewed by someone as conversant with Solr as I would prefer. It is my > hope, however, that this module will become part of the standard Solr 1.5 > suite of search components, since that would tie it in with LCF nicely. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-3460) Move handling of query only containing MUST_NOT to QueryParser (and remove QueryUtils.makeQueryable() hack in Solr)
[ https://issues.apache.org/jira/browse/LUCENE-3460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13114414#comment-13114414 ] Hoss Man commented on LUCENE-3460: -- a) watch out how/if you remove QueryUtils.makeQueryable ... this "hack" isn't just about adding a MatchAllDocsQuery, it's also got logic for dealing with WrappedQuery instances. b) the original reason why QueryUtils.makeQueryable was added (instead of putting that logic in SolrQueryParser) was so that Solr plugins could use the underlying lucene parser to parse user input (even if the resulting query parsed to all negative clauses, or no clauses) and then programaticly manipulate that query in a manner of their choosing -- calling makeQueryable if that was the desire. The three useages Uwe describes would eliminate this possibility (specificly bacause of "QP should simply parse to an empty BQ in the case of only prohibited clauses." statement) ie: Perhaps i want to allow my users to specify all MUST_NOT clauses, and then i want to add my *own* MUST clause instead of a MatchAllDocsQuery. ...all of that said: i'm definitely in favor of more "signals" coming out of the query parser of things that the caller should be aware of. particularly if we can help deal with the situation of *nested* clauses that are semanticly invalid (ie: "bar +(foo +(a the))" > Move handling of query only containing MUST_NOT to QueryParser (and remove > QueryUtils.makeQueryable() hack in Solr) > --- > > Key: LUCENE-3460 > URL: https://issues.apache.org/jira/browse/LUCENE-3460 > Project: Lucene - Java > Issue Type: Sub-task > Components: core/queryparser >Affects Versions: 3.4, 4.0 >Reporter: Uwe Schindler >Assignee: Uwe Schindler > Fix For: 4.0 > > > With the parent issue, users entering (a -b) into the queryparser can simply > fail with an UnsupportedOperationException, if "a" is a stopword. > Solr already has a hack to add a MatchAllDocsQuery, if a query only contains > prohibited clauses. > The other issue (not affecting prohibited clauses) with stopwords is: If the > user enters (a the) into queryparser, the query will return no results, as > "a" and "the" are stopwords. This confuses lots of people (not only > developers, even ordinary users of our interfaces). If somebody queries for a > stopword, the correct way to handle this is to return *all* documents > (MatchAllDocsQuery). > A possible solution, as suggested by Chris Male on IRC was: Add a flag to > QueryParser to enable a "no-should-or-must-clauses" mode, where this is > replaced by MatchAllDocs automatically. This would also solve the prohibited > clause case, too. > The stopword case is bad, but the opposite is as bad as returning all > documents. > At least this issue should somehow handle the only-prohibited case like Solr > and remove the hack from Solr. > Changing this in QueryParser is the more correct solution than doing this > hidden in BQ. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[JENKINS] Lucene-Solr-tests-only-trunk-java7 - Build # 508 - Still Failing
Build: https://builds.apache.org/job/Lucene-Solr-tests-only-trunk-java7/508/ 1 tests failed. REGRESSION: org.apache.solr.core.TestJmxIntegration.testJmxOnCoreReload Error Message: Number of registered MBeans is not the same as info registry size expected:<53> but was:<51> Stack Trace: junit.framework.AssertionFailedError: Number of registered MBeans is not the same as info registry size expected:<53> but was:<51> at org.apache.lucene.util.LuceneTestCaseRunner.runChild(LuceneTestCaseRunner.java:148) at org.apache.lucene.util.LuceneTestCaseRunner.runChild(LuceneTestCaseRunner.java:50) at org.apache.solr.core.TestJmxIntegration.testJmxOnCoreReload(TestJmxIntegration.java:134) at org.apache.lucene.util.LuceneTestCase$2$1.evaluate(LuceneTestCase.java:611) Build Log (for compile errors): [...truncated 10966 lines...] - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-3440) FastVectorHighlighter: IDF-weighted terms for ordered fragments
[
https://issues.apache.org/jira/browse/LUCENE-3440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13114402#comment-13114402
]
Koji Sekiguchi commented on LUCENE-3440:
{quote}
Hm, I thought about something like that:
{code:xml}
{code}
For Solr-users (like me). If somebody would like to use the boost-based
ordering, he could. Maybe, for some use-cases the boost-based approach is
better than the weighted one.
{quote}
I thought that, too. But I saw the following in the patch:
{code}
public List getWeightedFragInfoList( List
src ) {
Collections.sort( src, new ScoreComparator() );
//Collections.sort( src, new WeightComparator() );
return src;
}
{code}
And I thought you wanted to use WeightComparator from
ScoreOrderFragmentsBuilder. :)
Well now, let's introduce WeightOrderFragmentsBuilder.
> FastVectorHighlighter: IDF-weighted terms for ordered fragments
>
>
> Key: LUCENE-3440
> URL: https://issues.apache.org/jira/browse/LUCENE-3440
> Project: Lucene - Java
> Issue Type: Improvement
> Components: modules/highlighter
>Affects Versions: 3.5, 4.0
>Reporter: sebastian L.
>Priority: Minor
> Labels: FastVectorHighlighter
> Fix For: 3.5, 4.0
>
> Attachments: LUCENE-3.5-SNAPSHOT-3440-3.patch,
> LUCENE-4.0-SNAPSHOT-3440-3.patch
>
>
> The FastVectorHighlighter uses for every term found in a fragment an equal
> weight, which causes a higher ranking for fragments with a high number of
> words or, in the worst case, a high number of very common words than
> fragments that contains *all* of the terms used in the original query.
> This patch provides ordered fragments with IDF-weighted terms:
> total weight = total weight + IDF for unique term per fragment * boost of
> query;
> The ranking-formula should be the same, or at least similar, to that one used
> in org.apache.lucene.search.highlight.QueryTermScorer.
> The patch is simple, but it works for us.
> Some ideas:
> - A better approach would be moving the whole fragments-scoring into a
> separate class.
> - Switch scoring via parameter
> - Exact phrases should be given a even better score, regardless if a
> phrase-query was executed or not
> - edismax/dismax-parameters pf, ps and pf^boost should be observed and
> corresponding fragments should be ranked higher
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-1895) ManifoldCF SearchComponent plugin for enforcing ManifoldCF security at search time
[ https://issues.apache.org/jira/browse/SOLR-1895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13114400#comment-13114400 ] Karl Wright commented on SOLR-1895: --- The token vs. wildcard difference may be small because the number of distinct indexed token values is small. I'm going to try something bigger to see if the difference gets larger. There are good reasons to stick with the wildcard approach, having to do with documents that aren't indexed using Manifold. You really want them to be treated as if they are "open"; the token-based approach will cause them to be excluded, unfortunately. So I'm hoping the numbers stay good for wildcards. I'm going to try a few tricks Simon taught me to limit the number of query rewrites on those clauses, in order to make it as fast as possible. I don't understand the filter being so slow, however. That is indeed a surprise. As a result, I'll be attaching a query-based patch shortly. > ManifoldCF SearchComponent plugin for enforcing ManifoldCF security at search > time > -- > > Key: SOLR-1895 > URL: https://issues.apache.org/jira/browse/SOLR-1895 > Project: Solr > Issue Type: New Feature > Components: SearchComponents - other >Reporter: Karl Wright > Labels: document, security, solr > Fix For: 3.5, 4.0 > > Attachments: LCFSecurityFilter.java, LCFSecurityFilter.java, > LCFSecurityFilter.java, LCFSecurityFilter.java, > SOLR-1895-service-plugin.patch, SOLR-1895-service-plugin.patch, > SOLR-1895.patch, SOLR-1895.patch, SOLR-1895.patch, SOLR-1895.patch, > SOLR-1895.patch, SOLR-1895.patch > > > I've written an LCF SearchComponent which filters returned results based on > access tokens provided by LCF's authority service. The component requires > you to configure the appropriate authority service URL base, e.g.: > > > name="AuthorityServiceBaseURL">http://localhost:8080/lcf-authority-service > > Also required are the following schema.xml additions: > > stored="false" multiValued="true"/> > stored="false" multiValued="true"/> > stored="false" multiValued="true"/> > multiValued="true"/> > Finally, to tie it into the standard request handler, it seems to need to run > last: > > > lcfSecurity > > ... > I have not set a package for this code. Nor have I been able to get it > reviewed by someone as conversant with Solr as I would prefer. It is my > hope, however, that this module will become part of the standard Solr 1.5 > suite of search components, since that would tie it in with LCF nicely. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-3454) rename optimize to a less cool-sounding name
[ https://issues.apache.org/jira/browse/LUCENE-3454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13114391#comment-13114391 ] Mike Sokolov commented on LUCENE-3454: -- squeeze()? You apply some pressure - maybe it merges, and maybe it doesn't. Depending how hard you squeeze. squeeze(1) is squeezing harder than squeeze(10), which is a bit weird, but that's just down to units of squishiness. > rename optimize to a less cool-sounding name > > > Key: LUCENE-3454 > URL: https://issues.apache.org/jira/browse/LUCENE-3454 > Project: Lucene - Java > Issue Type: Improvement >Affects Versions: 4.0 >Reporter: Robert Muir > > I think users see the name optimize and feel they must do this, because who > wants a suboptimal system? but this probably just results in wasted time and > resources. > maybe rename to collapseSegments or something? -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-3451) Remove special handling of pure negative Filters in BooleanFilter, disallow pure negative queries in BooleanQuery
[ https://issues.apache.org/jira/browse/LUCENE-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13114389#comment-13114389 ] Uwe Schindler commented on LUCENE-3451: --- Mike, See Sub-Task LUCENE-3460 for an explanation. > Remove special handling of pure negative Filters in BooleanFilter, disallow > pure negative queries in BooleanQuery > - > > Key: LUCENE-3451 > URL: https://issues.apache.org/jira/browse/LUCENE-3451 > Project: Lucene - Java > Issue Type: Bug >Reporter: Uwe Schindler >Assignee: Uwe Schindler > Fix For: 4.0 > > Attachments: LUCENE-3451.patch, LUCENE-3451.patch, LUCENE-3451.patch, > LUCENE-3451.patch > > > We should at least in Lucene 4.0 remove the hack in BooleanFilter that allows > pure negative Filter clauses. This is not supported by BooleanQuery and > confuses users (I think that's the problem in LUCENE-3450). > The hack is buggy, as it does not respect deleted documents and returns them > in its DocIdSet. > Also we should think about disallowing pure-negative Queries at all and throw > UOE. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Issue Comment Edited] (LUCENE-3451) Remove special handling of pure negative Filters in BooleanFilter, disallow pure negative queries in BooleanQuery
[
https://issues.apache.org/jira/browse/LUCENE-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13114385#comment-13114385
]
Mike Sokolov edited comment on LUCENE-3451 at 9/25/11 11:19 PM:
Just wondering if there is a reason not to "fix" the user's query by adding a
{noformat}*:*{noformat} (maybe implicitly), rather than throwing an Exception?
This is invariably the fix users are instructed to apply in this case, and it
does seem to be the logical implication of a pure not-query.
Hmm - I just found the discussion over in LUCENE-3460, which addresses this
point.
was (Author: sokolov):
Just wondering if there is a reason not to "fix" the user's query by adding
a {noformat}*:*{noformat} (maybe implicitly), rather than throwing an
Exception? This is invariably the fix users are instructed to apply in this
case, and it does seem to be the logical implication of a pure not-query.
> Remove special handling of pure negative Filters in BooleanFilter, disallow
> pure negative queries in BooleanQuery
> -
>
> Key: LUCENE-3451
> URL: https://issues.apache.org/jira/browse/LUCENE-3451
> Project: Lucene - Java
> Issue Type: Bug
>Reporter: Uwe Schindler
>Assignee: Uwe Schindler
> Fix For: 4.0
>
> Attachments: LUCENE-3451.patch, LUCENE-3451.patch, LUCENE-3451.patch,
> LUCENE-3451.patch
>
>
> We should at least in Lucene 4.0 remove the hack in BooleanFilter that allows
> pure negative Filter clauses. This is not supported by BooleanQuery and
> confuses users (I think that's the problem in LUCENE-3450).
> The hack is buggy, as it does not respect deleted documents and returns them
> in its DocIdSet.
> Also we should think about disallowing pure-negative Queries at all and throw
> UOE.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Issue Comment Edited] (LUCENE-3451) Remove special handling of pure negative Filters in BooleanFilter, disallow pure negative queries in BooleanQuery
[
https://issues.apache.org/jira/browse/LUCENE-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13114385#comment-13114385
]
Mike Sokolov edited comment on LUCENE-3451 at 9/25/11 10:57 PM:
Just wondering if there is a reason not to "fix" the user's query by adding a
{noformat}*:*{noformat} (maybe implicitly), rather than throwing an Exception?
This is invariably the fix users are instructed to apply in this case, and it
does seem to be the logical implication of a pure not-query.
was (Author: sokolov):
Just wondering if there is a reason not to "fix" the user's query by adding
a *:* (maybe implicitly), rather than throwing an Exception? This is
invariably the fix users are instructed to apply in this case, and it does seem
to be the logical implication of a pure not-query.
> Remove special handling of pure negative Filters in BooleanFilter, disallow
> pure negative queries in BooleanQuery
> -
>
> Key: LUCENE-3451
> URL: https://issues.apache.org/jira/browse/LUCENE-3451
> Project: Lucene - Java
> Issue Type: Bug
>Reporter: Uwe Schindler
>Assignee: Uwe Schindler
> Fix For: 4.0
>
> Attachments: LUCENE-3451.patch, LUCENE-3451.patch, LUCENE-3451.patch,
> LUCENE-3451.patch
>
>
> We should at least in Lucene 4.0 remove the hack in BooleanFilter that allows
> pure negative Filter clauses. This is not supported by BooleanQuery and
> confuses users (I think that's the problem in LUCENE-3450).
> The hack is buggy, as it does not respect deleted documents and returns them
> in its DocIdSet.
> Also we should think about disallowing pure-negative Queries at all and throw
> UOE.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-3451) Remove special handling of pure negative Filters in BooleanFilter, disallow pure negative queries in BooleanQuery
[ https://issues.apache.org/jira/browse/LUCENE-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13114385#comment-13114385 ] Mike Sokolov commented on LUCENE-3451: -- Just wondering if there is a reason not to "fix" the user's query by adding a *:* (maybe implicitly), rather than throwing an Exception? This is invariably the fix users are instructed to apply in this case, and it does seem to be the logical implication of a pure not-query. > Remove special handling of pure negative Filters in BooleanFilter, disallow > pure negative queries in BooleanQuery > - > > Key: LUCENE-3451 > URL: https://issues.apache.org/jira/browse/LUCENE-3451 > Project: Lucene - Java > Issue Type: Bug >Reporter: Uwe Schindler >Assignee: Uwe Schindler > Fix For: 4.0 > > Attachments: LUCENE-3451.patch, LUCENE-3451.patch, LUCENE-3451.patch, > LUCENE-3451.patch > > > We should at least in Lucene 4.0 remove the hack in BooleanFilter that allows > pure negative Filter clauses. This is not supported by BooleanQuery and > confuses users (I think that's the problem in LUCENE-3450). > The hack is buggy, as it does not respect deleted documents and returns them > in its DocIdSet. > Also we should think about disallowing pure-negative Queries at all and throw > UOE. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Issue Comment Edited] (LUCENE-3360) Move FieldCache to IndexReader
[
https://issues.apache.org/jira/browse/LUCENE-3360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13114375#comment-13114375
]
Martijn van Groningen edited comment on LUCENE-3360 at 9/25/11 9:45 PM:
Makes sense to wait to commit this change for trunk. I've updated the 3x
version of the patch with latest changes in 3x. This version only deprecates FC.
was (Author: martijn.v.groningen):
Makes sense to wait committing for trunk. I've updated the 3x version of
the patch with latest changes in 3x. This version only deprecates FC.
> Move FieldCache to IndexReader
> --
>
> Key: LUCENE-3360
> URL: https://issues.apache.org/jira/browse/LUCENE-3360
> Project: Lucene - Java
> Issue Type: Improvement
>Reporter: Martijn van Groningen
> Fix For: 3.5, 4.0
>
> Attachments: LUCENE-3360-3x.patch, LUCENE-3360-3x.patch,
> LUCENE-3360.patch, LUCENE-3360.patch, LUCENE-3360.patch, LUCENE-3360.patch,
> LUCENE-3360.patch, LUCENE-3360.patch, LUCENE-3360.patch
>
>
> Move the static FieldCache.DEFAULT field instance to atomic IndexReaders, so
> that FieldCache insanity caused by the WeakHashMap no longer occurs.
> * Add a new method to IndexReader that by default throws an UOE:
> {code}public FieldCache getFieldCache(){code}
> * The SegmentReader implements this method and returns its own internal
> FieldCache implementation. This implementation just uses a
> HashMap,Object>> to store entries.
> * The SlowMultiReaderWrapper implements this method as well and basically
> behaves the same as the current FieldCacheImpl.
> This issue won't solve the insanity that comes from inconsistent usage of a
> single field (for example retrieve both int[] and DocTermIndex for the same
> field).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (LUCENE-3360) Move FieldCache to IndexReader
[
https://issues.apache.org/jira/browse/LUCENE-3360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Martijn van Groningen updated LUCENE-3360:
--
Attachment: LUCENE-3360-3x.patch
Makes sense to wait committing for trunk. I've updated the 3x version of the
patch with latest changes in 3x. This version only deprecates FC.
> Move FieldCache to IndexReader
> --
>
> Key: LUCENE-3360
> URL: https://issues.apache.org/jira/browse/LUCENE-3360
> Project: Lucene - Java
> Issue Type: Improvement
>Reporter: Martijn van Groningen
> Fix For: 3.5, 4.0
>
> Attachments: LUCENE-3360-3x.patch, LUCENE-3360-3x.patch,
> LUCENE-3360.patch, LUCENE-3360.patch, LUCENE-3360.patch, LUCENE-3360.patch,
> LUCENE-3360.patch, LUCENE-3360.patch, LUCENE-3360.patch
>
>
> Move the static FieldCache.DEFAULT field instance to atomic IndexReaders, so
> that FieldCache insanity caused by the WeakHashMap no longer occurs.
> * Add a new method to IndexReader that by default throws an UOE:
> {code}public FieldCache getFieldCache(){code}
> * The SegmentReader implements this method and returns its own internal
> FieldCache implementation. This implementation just uses a
> HashMap,Object>> to store entries.
> * The SlowMultiReaderWrapper implements this method as well and basically
> behaves the same as the current FieldCacheImpl.
> This issue won't solve the insanity that comes from inconsistent usage of a
> single field (for example retrieve both int[] and DocTermIndex for the same
> field).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-2793) A SolrIndexSearcher can be left open if the executor rejects a task.
[ https://issues.apache.org/jira/browse/SOLR-2793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13114374#comment-13114374 ] Mark Miller commented on SOLR-2793: --- whoops - patch is slightly stale - what I will actually commit also propagates the exception - right after decref, throw e. > A SolrIndexSearcher can be left open if the executor rejects a task. > > > Key: SOLR-2793 > URL: https://issues.apache.org/jira/browse/SOLR-2793 > Project: Solr > Issue Type: Bug >Reporter: Mark Miller >Assignee: Mark Miller > Fix For: 3.5, 4.0 > > Attachments: SOLR-2793.patch > > > This is starting to really bug me because tests almost never pass on my linux > box due to this issue. Time to fix it. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-2793) A SolrIndexSearcher can be left open if the executor rejects a task.
[ https://issues.apache.org/jira/browse/SOLR-2793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Miller updated SOLR-2793: -- Attachment: SOLR-2793.patch patch simply decrefs newSearchHolder if the executor rejects the task > A SolrIndexSearcher can be left open if the executor rejects a task. > > > Key: SOLR-2793 > URL: https://issues.apache.org/jira/browse/SOLR-2793 > Project: Solr > Issue Type: Bug >Reporter: Mark Miller >Assignee: Mark Miller > Fix For: 3.5, 4.0 > > Attachments: SOLR-2793.patch > > > This is starting to really bug me because tests almost never pass on my linux > box due to this issue. Time to fix it. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Created] (SOLR-2793) A SolrIndexSearcher can be left open if the executor rejects a task.
A SolrIndexSearcher can be left open if the executor rejects a task. Key: SOLR-2793 URL: https://issues.apache.org/jira/browse/SOLR-2793 Project: Solr Issue Type: Bug Reporter: Mark Miller Assignee: Mark Miller Fix For: 3.5, 4.0 This is starting to really bug me because tests almost never pass on my linux box due to this issue. Time to fix it. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[JENKINS] Lucene-Solr-tests-only-trunk-java7 - Build # 507 - Failure
Build: https://builds.apache.org/job/Lucene-Solr-tests-only-trunk-java7/507/ 1 tests failed. FAILED: junit.framework.TestSuite.org.apache.solr.update.AutoCommitTest Error Message: ERROR: SolrIndexSearcher opens=21 closes=20 Stack Trace: junit.framework.AssertionFailedError: ERROR: SolrIndexSearcher opens=21 closes=20 at org.apache.solr.SolrTestCaseJ4.endTrackingSearchers(SolrTestCaseJ4.java:129) at org.apache.solr.util.AbstractSolrTestCase.afterClassAbstractSolrTestCase(AbstractSolrTestCase.java:105) Build Log (for compile errors): [...truncated 11015 lines...] - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[JENKINS] Lucene-Solr-tests-only-3.x - Build # 10664 - Still Failing
Build: https://builds.apache.org/job/Lucene-Solr-tests-only-3.x/10664/ No tests ran. Build Log (for compile errors): [...truncated 4400 lines...] clover: common.compile-core: [mkdir] Created dir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-tests-only-3.x/checkout/solr/build/contrib/solr-uima/classes/java [javac] Compiling 9 source files to /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-tests-only-3.x/checkout/solr/build/contrib/solr-uima/classes/java [copy] Copying 13 files to /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-tests-only-3.x/checkout/solr/build/contrib/solr-uima/classes/java common-solr.compile-core: compile-core: compile: compile-solr-test-framework: [echo] Building solr-test-framework... compile-solr-core: compile-test-framework: jflex-uptodate-check: jflex-notice: javacc-uptodate-check: javacc-notice: init: clover.setup: clover.info: [echo] [echo] Clover not found. Code coverage reports disabled. [echo] clover: common.compile-core: compile-core: compile-test-framework: compile-core: [mkdir] Created dir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-tests-only-3.x/checkout/solr/build/solr-test-framework/classes/java [javac] Compiling 12 source files to /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-tests-only-3.x/checkout/solr/build/solr-test-framework/classes/java [javac] /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-tests-only-3.x/checkout/solr/test-framework/src/java/org/apache/solr/SolrTestCaseJ4.java:785: warning: [unchecked] unchecked call to compareTo(T) as a member of the raw type java.lang.Comparable [javac] return this.id.compareTo(other.id); [javac] ^ [javac] /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-tests-only-3.x/checkout/solr/test-framework/src/java/org/apache/solr/SolrTestCaseJ4.java:904: warning: [unchecked] unchecked cast [javac] found : java.lang.Object [javac] required: java.util.List [javac] List docList = (List)response; [javac]^ [javac] /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-tests-only-3.x/checkout/solr/test-framework/src/java/org/apache/solr/SolrTestCaseJ4.java:1016: warning: [unchecked] unchecked call to compareTo(T) as a member of the raw type java.lang.Comparable [javac] c = v1.compareTo(v2); [javac] ^ [javac] /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-tests-only-3.x/checkout/solr/test-framework/src/java/org/apache/solr/BaseDistributedSearchTestCase.java:580: warning: [unchecked] unchecked call to add(E) as a member of the raw type java.util.Set [javac] if (uniqueValues.add(v)) return v; [javac] ^ [javac] /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-tests-only-3.x/checkout/solr/test-framework/src/java/org/apache/solr/util/TestHarness.java:585: warning: [unchecked] unchecked conversion [javac] found : org.apache.solr.common.util.NamedList.NamedListEntry[] [javac] required: java.util.Map.Entry[] [javac] Map.Entry [] entries = new NamedListEntry[q.length / 2]; [javac] ^ [javac] /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-tests-only-3.x/checkout/solr/test-framework/src/java/org/apache/solr/util/TestHarness.java:589: warning: [unchecked] unchecked call to NamedList(java.util.Map.Entry[]) as a member of the raw type org.apache.solr.common.util.NamedList [javac] return new LocalSolrQueryRequest(TestHarness.this.getCore(), new NamedList(entries)); [javac]^ [javac] /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-tests-only-3.x/checkout/solr/test-framework/src/java/org/apache/solr/JSONTestUtil.java:232: warning: [unchecked] unchecked cast [javac] found : java.lang.Object [javac] required: java.util.Map [javac] Map expectedMap = (Map)expected; [javac] ^ [javac] /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-tests-only-3.x/checkout/solr/test-framework/src/java/org/apache/solr/JSONTestUtil.java:247: warning: [unchecked] unchecked call to HashSet(java.util.Collection) as a member of the raw type java.util.HashSet [javac] match = new HashSet(StrUtils.splitSmart(matchList,",",false)); [javac] ^ [javac] /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-tests-only-3.x/checkout/solr/test-framework/src/java/org/apache/solr/JSONTestUtil.java:247: warning: [unchecked] unchecked conversion [javac] found : java.util.HashSet [javac] required: java.util.Set [javac] match = new HashSet(StrUtils.splitSmart(matchList,",",false)); [javac] ^ [javac] /usr/home/hudson/hudson-slave/worksp
[jira] [Commented] (LUCENE-3360) Move FieldCache to IndexReader
[
https://issues.apache.org/jira/browse/LUCENE-3360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13114358#comment-13114358
]
Uwe Schindler commented on LUCENE-3360:
---
Hi Martijn,
nice work! But I would recommend to wait with commiting those changes. We redo
FieldCache for 3.x and wanted to forward-port that to trunk, as the FieldCache
implementation is very complicated there: LUCENE-3443
I think we should work together and solve the stuff identical like in 3.x, but
bound to IndexReader.
> Move FieldCache to IndexReader
> --
>
> Key: LUCENE-3360
> URL: https://issues.apache.org/jira/browse/LUCENE-3360
> Project: Lucene - Java
> Issue Type: Improvement
>Reporter: Martijn van Groningen
> Fix For: 3.5, 4.0
>
> Attachments: LUCENE-3360-3x.patch, LUCENE-3360.patch,
> LUCENE-3360.patch, LUCENE-3360.patch, LUCENE-3360.patch, LUCENE-3360.patch,
> LUCENE-3360.patch, LUCENE-3360.patch
>
>
> Move the static FieldCache.DEFAULT field instance to atomic IndexReaders, so
> that FieldCache insanity caused by the WeakHashMap no longer occurs.
> * Add a new method to IndexReader that by default throws an UOE:
> {code}public FieldCache getFieldCache(){code}
> * The SegmentReader implements this method and returns its own internal
> FieldCache implementation. This implementation just uses a
> HashMap,Object>> to store entries.
> * The SlowMultiReaderWrapper implements this method as well and basically
> behaves the same as the current FieldCacheImpl.
> This issue won't solve the insanity that comes from inconsistent usage of a
> single field (for example retrieve both int[] and DocTermIndex for the same
> field).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-3464) Rename IndexReader.reopen to make it clear that reopen may not happen
[ https://issues.apache.org/jira/browse/LUCENE-3464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13114354#comment-13114354 ] Mark Miller commented on LUCENE-3464: - bq. reopenIfNeeded +1 - can't think of anything better yet > Rename IndexReader.reopen to make it clear that reopen may not happen > - > > Key: LUCENE-3464 > URL: https://issues.apache.org/jira/browse/LUCENE-3464 > Project: Lucene - Java > Issue Type: Bug >Reporter: Michael McCandless >Assignee: Michael McCandless > Fix For: 3.5, 4.0 > > > Spinoff from LUCENE-3454 where Shai noted this inconsistency. > IR.reopen sounds like an unconditional operation, which has trapped users in > the past into always closing the old reader instead of only closing it if the > returned reader is new. > I think this hidden maybe-ness is trappy and we should rename it > (maybeReopen? reopenIfNeeded?). > In addition, instead of returning "this" when the reopen didn't happen, I > think we should return null to enforce proper usage of the maybe-ness of this > API. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[JENKINS] Lucene-Solr-tests-only-3.x-java7 - Build # 514 - Still Failing
Build: https://builds.apache.org/job/Lucene-Solr-tests-only-3.x-java7/514/ No tests ran. Build Log (for compile errors): [...truncated 4424 lines...] clover: common.compile-core: [mkdir] Created dir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-tests-only-3.x-java7/checkout/solr/build/contrib/solr-uima/classes/java [javac] Compiling 9 source files to /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-tests-only-3.x-java7/checkout/solr/build/contrib/solr-uima/classes/java [copy] Copying 13 files to /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-tests-only-3.x-java7/checkout/solr/build/contrib/solr-uima/classes/java common-solr.compile-core: compile-core: compile: compile-solr-test-framework: [echo] Building solr-test-framework... compile-solr-core: compile-test-framework: jflex-uptodate-check: jflex-notice: javacc-uptodate-check: javacc-notice: init: clover.setup: clover.info: [echo] [echo] Clover not found. Code coverage reports disabled. [echo] clover: common.compile-core: compile-core: compile-test-framework: compile-core: [mkdir] Created dir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-tests-only-3.x-java7/checkout/solr/build/solr-test-framework/classes/java [javac] Compiling 12 source files to /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-tests-only-3.x-java7/checkout/solr/build/solr-test-framework/classes/java [javac] /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-tests-only-3.x-java7/checkout/solr/test-framework/src/java/org/apache/solr/SolrTestCaseJ4.java:785: warning: [unchecked] unchecked call to compareTo(T) as a member of the raw type java.lang.Comparable [javac] return this.id.compareTo(other.id); [javac] ^ [javac] /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-tests-only-3.x-java7/checkout/solr/test-framework/src/java/org/apache/solr/SolrTestCaseJ4.java:904: warning: [unchecked] unchecked cast [javac] found : java.lang.Object [javac] required: java.util.List [javac] List docList = (List)response; [javac]^ [javac] /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-tests-only-3.x-java7/checkout/solr/test-framework/src/java/org/apache/solr/SolrTestCaseJ4.java:1016: warning: [unchecked] unchecked call to compareTo(T) as a member of the raw type java.lang.Comparable [javac] c = v1.compareTo(v2); [javac] ^ [javac] /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-tests-only-3.x-java7/checkout/solr/test-framework/src/java/org/apache/solr/BaseDistributedSearchTestCase.java:580: warning: [unchecked] unchecked call to add(E) as a member of the raw type java.util.Set [javac] if (uniqueValues.add(v)) return v; [javac] ^ [javac] /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-tests-only-3.x-java7/checkout/solr/test-framework/src/java/org/apache/solr/util/TestHarness.java:585: warning: [unchecked] unchecked conversion [javac] found : org.apache.solr.common.util.NamedList.NamedListEntry[] [javac] required: java.util.Map.Entry[] [javac] Map.Entry [] entries = new NamedListEntry[q.length / 2]; [javac] ^ [javac] /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-tests-only-3.x-java7/checkout/solr/test-framework/src/java/org/apache/solr/util/TestHarness.java:589: warning: [unchecked] unchecked call to NamedList(java.util.Map.Entry[]) as a member of the raw type org.apache.solr.common.util.NamedList [javac] return new LocalSolrQueryRequest(TestHarness.this.getCore(), new NamedList(entries)); [javac]^ [javac] /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-tests-only-3.x-java7/checkout/solr/test-framework/src/java/org/apache/solr/JSONTestUtil.java:232: warning: [unchecked] unchecked cast [javac] found : java.lang.Object [javac] required: java.util.Map [javac] Map expectedMap = (Map)expected; [javac] ^ [javac] /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-tests-only-3.x-java7/checkout/solr/test-framework/src/java/org/apache/solr/JSONTestUtil.java:247: warning: [unchecked] unchecked call to HashSet(java.util.Collection) as a member of the raw type java.util.HashSet [javac] match = new HashSet(StrUtils.splitSmart(matchList,",",false)); [javac] ^ [javac] /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-tests-only-3.x-java7/checkout/solr/test-framework/src/java/org/apache/solr/JSONTestUtil.java:247: warning: [unchecked] unchecked conversion [javac] found : java.util.HashSet [javac] required: java.util.Set [javac] match = new HashSet(StrUtils.splitSmart(matchList,",
[JENKINS] Lucene-Solr-tests-only-3.x - Build # 10663 - Failure
Build: https://builds.apache.org/job/Lucene-Solr-tests-only-3.x/10663/ No tests ran. Build Log (for compile errors): [...truncated 4437 lines...] clover: common.compile-core: [mkdir] Created dir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-tests-only-3.x/checkout/solr/build/contrib/solr-uima/classes/java [javac] Compiling 9 source files to /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-tests-only-3.x/checkout/solr/build/contrib/solr-uima/classes/java [copy] Copying 13 files to /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-tests-only-3.x/checkout/solr/build/contrib/solr-uima/classes/java common-solr.compile-core: compile-core: compile: compile-solr-test-framework: [echo] Building solr-test-framework... compile-solr-core: compile-test-framework: jflex-uptodate-check: jflex-notice: javacc-uptodate-check: javacc-notice: init: clover.setup: clover.info: [echo] [echo] Clover not found. Code coverage reports disabled. [echo] clover: common.compile-core: compile-core: compile-test-framework: compile-core: [mkdir] Created dir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-tests-only-3.x/checkout/solr/build/solr-test-framework/classes/java [javac] Compiling 12 source files to /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-tests-only-3.x/checkout/solr/build/solr-test-framework/classes/java [javac] /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-tests-only-3.x/checkout/solr/test-framework/src/java/org/apache/solr/SolrTestCaseJ4.java:785: warning: [unchecked] unchecked call to compareTo(T) as a member of the raw type java.lang.Comparable [javac] return this.id.compareTo(other.id); [javac] ^ [javac] /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-tests-only-3.x/checkout/solr/test-framework/src/java/org/apache/solr/SolrTestCaseJ4.java:904: warning: [unchecked] unchecked cast [javac] found : java.lang.Object [javac] required: java.util.List [javac] List docList = (List)response; [javac]^ [javac] /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-tests-only-3.x/checkout/solr/test-framework/src/java/org/apache/solr/SolrTestCaseJ4.java:1016: warning: [unchecked] unchecked call to compareTo(T) as a member of the raw type java.lang.Comparable [javac] c = v1.compareTo(v2); [javac] ^ [javac] /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-tests-only-3.x/checkout/solr/test-framework/src/java/org/apache/solr/BaseDistributedSearchTestCase.java:580: warning: [unchecked] unchecked call to add(E) as a member of the raw type java.util.Set [javac] if (uniqueValues.add(v)) return v; [javac] ^ [javac] /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-tests-only-3.x/checkout/solr/test-framework/src/java/org/apache/solr/util/TestHarness.java:585: warning: [unchecked] unchecked conversion [javac] found : org.apache.solr.common.util.NamedList.NamedListEntry[] [javac] required: java.util.Map.Entry[] [javac] Map.Entry [] entries = new NamedListEntry[q.length / 2]; [javac] ^ [javac] /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-tests-only-3.x/checkout/solr/test-framework/src/java/org/apache/solr/util/TestHarness.java:589: warning: [unchecked] unchecked call to NamedList(java.util.Map.Entry[]) as a member of the raw type org.apache.solr.common.util.NamedList [javac] return new LocalSolrQueryRequest(TestHarness.this.getCore(), new NamedList(entries)); [javac]^ [javac] /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-tests-only-3.x/checkout/solr/test-framework/src/java/org/apache/solr/JSONTestUtil.java:232: warning: [unchecked] unchecked cast [javac] found : java.lang.Object [javac] required: java.util.Map [javac] Map expectedMap = (Map)expected; [javac] ^ [javac] /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-tests-only-3.x/checkout/solr/test-framework/src/java/org/apache/solr/JSONTestUtil.java:247: warning: [unchecked] unchecked call to HashSet(java.util.Collection) as a member of the raw type java.util.HashSet [javac] match = new HashSet(StrUtils.splitSmart(matchList,",",false)); [javac] ^ [javac] /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-tests-only-3.x/checkout/solr/test-framework/src/java/org/apache/solr/JSONTestUtil.java:247: warning: [unchecked] unchecked conversion [javac] found : java.util.HashSet [javac] required: java.util.Set [javac] match = new HashSet(StrUtils.splitSmart(matchList,",",false)); [javac] ^ [javac] /usr/home/hudson/hudson-slave/worksp
[JENKINS] Lucene-Solr-tests-only-trunk - Build # 10640 - Failure
Build: https://builds.apache.org/job/Lucene-Solr-tests-only-trunk/10640/ 1 tests failed. FAILED: junit.framework.TestSuite.org.apache.solr.update.AutoCommitTest Error Message: java.lang.AssertionError: directory of test was not closed, opened from: org.apache.solr.core.MockDirectoryFactory.create(MockDirectoryFactory.java:33) Stack Trace: java.lang.RuntimeException: java.lang.AssertionError: directory of test was not closed, opened from: org.apache.solr.core.MockDirectoryFactory.create(MockDirectoryFactory.java:33) at org.apache.lucene.util.LuceneTestCase.afterClassLuceneTestCaseJ4(LuceneTestCase.java:470) at org.apache.lucene.util.LuceneTestCase.checkResourcesAfterClass(LuceneTestCase.java:528) at org.apache.lucene.util.LuceneTestCase.afterClassLuceneTestCaseJ4(LuceneTestCase.java:438) Build Log (for compile errors): [...truncated 7923 lines...] - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Issue Comment Edited] (LUCENE-3360) Move FieldCache to IndexReader
[
https://issues.apache.org/jira/browse/LUCENE-3360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13114343#comment-13114343
]
Martijn van Groningen edited comment on LUCENE-3360 at 9/25/11 8:06 PM:
Attached an updated version for trunk.
* FieldCache and FieldCacheImpl have been removed completely from the source
code.
* DocTerms and DocTermsIndex have been moved to o.a.l.search.cache
* Parsers have been moved to o.a.l.search.cache.parser
* Functionality that relies on top level caches (e.g. faceting and some
functions) now uses SlowMultiReaderWrapper: {code}new
SlowMultiReaderWrapper(topReader).getFieldCache(){code}
was (Author: martijn.v.groningen):
Attached an updated version for trunk.
* FieldCache and FieldCacheImpl have been removed completely from the source
code.
* DocTerms and DocTermsIndex have been moved to o.a.l.search.cache
* Parsers have been moved to o.a.l.search.cache.parser
* Functionality that relies on top level caches (e.g. faceting and some
functions) now uses SlowMultiReaderWrapper: {code}new
SlowMultiReaderWrapper(topReader)).getFieldCache(){code}
> Move FieldCache to IndexReader
> --
>
> Key: LUCENE-3360
> URL: https://issues.apache.org/jira/browse/LUCENE-3360
> Project: Lucene - Java
> Issue Type: Improvement
>Reporter: Martijn van Groningen
> Fix For: 3.5, 4.0
>
> Attachments: LUCENE-3360-3x.patch, LUCENE-3360.patch,
> LUCENE-3360.patch, LUCENE-3360.patch, LUCENE-3360.patch, LUCENE-3360.patch,
> LUCENE-3360.patch, LUCENE-3360.patch
>
>
> Move the static FieldCache.DEFAULT field instance to atomic IndexReaders, so
> that FieldCache insanity caused by the WeakHashMap no longer occurs.
> * Add a new method to IndexReader that by default throws an UOE:
> {code}public FieldCache getFieldCache(){code}
> * The SegmentReader implements this method and returns its own internal
> FieldCache implementation. This implementation just uses a
> HashMap,Object>> to store entries.
> * The SlowMultiReaderWrapper implements this method as well and basically
> behaves the same as the current FieldCacheImpl.
> This issue won't solve the insanity that comes from inconsistent usage of a
> single field (for example retrieve both int[] and DocTermIndex for the same
> field).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-3454) rename optimize to a less cool-sounding name
[ https://issues.apache.org/jira/browse/LUCENE-3454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13114347#comment-13114347 ] Mark Miller commented on LUCENE-3454: - In spitball land... requestAggressiveMerge? > rename optimize to a less cool-sounding name > > > Key: LUCENE-3454 > URL: https://issues.apache.org/jira/browse/LUCENE-3454 > Project: Lucene - Java > Issue Type: Improvement >Affects Versions: 4.0 >Reporter: Robert Muir > > I think users see the name optimize and feel they must do this, because who > wants a suboptimal system? but this probably just results in wasted time and > resources. > maybe rename to collapseSegments or something? -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (LUCENE-3360) Move FieldCache to IndexReader
[
https://issues.apache.org/jira/browse/LUCENE-3360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Martijn van Groningen updated LUCENE-3360:
--
Attachment: LUCENE-3360.patch
Attached an updated version for trunk.
* FieldCache and FieldCacheImpl have been removed completely from the source
code.
* DocTerms and DocTermsIndex have been moved to o.a.l.search.cache
* Parsers have been moved to o.a.l.search.cache.parser
* Functionality that relies on top level caches (e.g. faceting and some
functions) now uses SlowMultiReaderWrapper: {code}new
SlowMultiReaderWrapper(topReader)).getFieldCache(){code}
> Move FieldCache to IndexReader
> --
>
> Key: LUCENE-3360
> URL: https://issues.apache.org/jira/browse/LUCENE-3360
> Project: Lucene - Java
> Issue Type: Improvement
>Reporter: Martijn van Groningen
> Fix For: 3.5, 4.0
>
> Attachments: LUCENE-3360-3x.patch, LUCENE-3360.patch,
> LUCENE-3360.patch, LUCENE-3360.patch, LUCENE-3360.patch, LUCENE-3360.patch,
> LUCENE-3360.patch, LUCENE-3360.patch
>
>
> Move the static FieldCache.DEFAULT field instance to atomic IndexReaders, so
> that FieldCache insanity caused by the WeakHashMap no longer occurs.
> * Add a new method to IndexReader that by default throws an UOE:
> {code}public FieldCache getFieldCache(){code}
> * The SegmentReader implements this method and returns its own internal
> FieldCache implementation. This implementation just uses a
> HashMap,Object>> to store entries.
> * The SlowMultiReaderWrapper implements this method as well and basically
> behaves the same as the current FieldCacheImpl.
> This issue won't solve the insanity that comes from inconsistent usage of a
> single field (for example retrieve both int[] and DocTermIndex for the same
> field).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[JENKINS] Lucene-Solr-tests-only-3.x-java7 - Build # 513 - Failure
Build: https://builds.apache.org/job/Lucene-Solr-tests-only-3.x-java7/513/ 1 tests failed. REGRESSION: org.apache.solr.TestDistributedSearch.testDistribSearch Error Message: java.lang.AssertionError: Some threads threw uncaught exceptions! Stack Trace: java.lang.RuntimeException: java.lang.AssertionError: Some threads threw uncaught exceptions! at org.apache.lucene.util.LuceneTestCase.tearDown(LuceneTestCase.java:574) at org.apache.solr.SolrTestCaseJ4.tearDown(SolrTestCaseJ4.java:96) at org.apache.solr.BaseDistributedSearchTestCase.tearDown(BaseDistributedSearchTestCase.java:160) at org.apache.lucene.util.LuceneTestCaseRunner.runChild(LuceneTestCaseRunner.java:147) at org.apache.lucene.util.LuceneTestCaseRunner.runChild(LuceneTestCaseRunner.java:50) at org.apache.lucene.util.LuceneTestCase.checkUncaughtExceptionsAfter(LuceneTestCase.java:602) at org.apache.lucene.util.LuceneTestCase.tearDown(LuceneTestCase.java:546) Build Log (for compile errors): [...truncated 13830 lines...] - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Resolved] (LUCENE-3457) Upgrade to commons-compress 1.2
[ https://issues.apache.org/jira/browse/LUCENE-3457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Doron Cohen resolved LUCENE-3457. - Resolution: Fixed Fixed: - 1175475 - trunk - 1175528 - 3x > Upgrade to commons-compress 1.2 > --- > > Key: LUCENE-3457 > URL: https://issues.apache.org/jira/browse/LUCENE-3457 > Project: Lucene - Java > Issue Type: Bug > Components: modules/benchmark >Reporter: Doron Cohen >Assignee: Doron Cohen >Priority: Minor > Fix For: 3.5, 4.0 > > Attachments: LUCENE-3457.patch, test.out.gz > > > Commons Compress bug COMPRESS-127 was fixed in 1.2, so the workaround in > benchmark's StreamUtils is no longer required. Compress is also used in solr. > Replace with new jar in both benchmark and solr and get rid of that > workaround. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Assigned] (LUCENE-3464) Rename IndexReader.reopen to make it clear that reopen may not happen
[ https://issues.apache.org/jira/browse/LUCENE-3464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael McCandless reassigned LUCENE-3464: -- Assignee: Michael McCandless > Rename IndexReader.reopen to make it clear that reopen may not happen > - > > Key: LUCENE-3464 > URL: https://issues.apache.org/jira/browse/LUCENE-3464 > Project: Lucene - Java > Issue Type: Bug >Reporter: Michael McCandless >Assignee: Michael McCandless > Fix For: 3.5, 4.0 > > > Spinoff from LUCENE-3454 where Shai noted this inconsistency. > IR.reopen sounds like an unconditional operation, which has trapped users in > the past into always closing the old reader instead of only closing it if the > returned reader is new. > I think this hidden maybe-ness is trappy and we should rename it > (maybeReopen? reopenIfNeeded?). > In addition, instead of returning "this" when the reopen didn't happen, I > think we should return null to enforce proper usage of the maybe-ness of this > API. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Created] (LUCENE-3464) Rename IndexReader.reopen to make it clear that reopen may not happen
Rename IndexReader.reopen to make it clear that reopen may not happen - Key: LUCENE-3464 URL: https://issues.apache.org/jira/browse/LUCENE-3464 Project: Lucene - Java Issue Type: Bug Reporter: Michael McCandless Fix For: 3.5, 4.0 Spinoff from LUCENE-3454 where Shai noted this inconsistency. IR.reopen sounds like an unconditional operation, which has trapped users in the past into always closing the old reader instead of only closing it if the returned reader is new. I think this hidden maybe-ness is trappy and we should rename it (maybeReopen? reopenIfNeeded?). In addition, instead of returning "this" when the reopen didn't happen, I think we should return null to enforce proper usage of the maybe-ness of this API. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-3454) rename optimize to a less cool-sounding name
[ https://issues.apache.org/jira/browse/LUCENE-3454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13114323#comment-13114323 ] Michael McCandless commented on LUCENE-3454: bq. Why do we have IR.reopen and not maybeReopen? That's a good point... and I actually don't like that naming either!! I think it trips users up because of it's hidden maybe-ness, ie, users who always close the old reader on calling this API (which, if no reopen actually occurred, closes the very same reader it had just returned). Also, IW already has a merge method, taking a OneMerge parameter; that naming I think is correct because it's unconditional: IW will carry out the merge you passed to it. Naming is the hardest part! > rename optimize to a less cool-sounding name > > > Key: LUCENE-3454 > URL: https://issues.apache.org/jira/browse/LUCENE-3454 > Project: Lucene - Java > Issue Type: Improvement >Affects Versions: 4.0 >Reporter: Robert Muir > > I think users see the name optimize and feel they must do this, because who > wants a suboptimal system? but this probably just results in wasted time and > resources. > maybe rename to collapseSegments or something? -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (LUCENE-3456) Analysis Consumers should end and close any TokenStreams
[ https://issues.apache.org/jira/browse/LUCENE-3456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Muir updated LUCENE-3456: Attachment: LUCENE-3456.patch attached is a patch cutting over all uses of whitespace->mocktokenizer in solr test configs, and fixing the errors this found. i'd like to commit shortly, we can improve the coverage more by looking at other configs and cutting over things like keywordtokenizer > Analysis Consumers should end and close any TokenStreams > > > Key: LUCENE-3456 > URL: https://issues.apache.org/jira/browse/LUCENE-3456 > Project: Lucene - Java > Issue Type: Sub-task >Reporter: Chris Male > Attachments: LUCENE-3456.patch, LUCENE-3456.patch > > > While converting consumers over to using reusableTokenStream, I notice many > don't call end() or close() on TokenStreams. > Even if they are test TSs only created once, we should follow the defined > usage pattern. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
Re: svn commit: r1175455 - in /lucene/dev/branches/branch_3x/lucene/contrib/misc/src/test/org/apache/lucene: index/TestNRTManager.java search/TestSearcherManager.java
Ugh!
Thanks for fixing Uwe.
Mike McCandless
http://blog.mikemccandless.com
On Sun, Sep 25, 2011 at 2:11 PM, wrote:
> Author: uschindler
> Date: Sun Sep 25 18:11:20 2011
> New Revision: 1175455
>
> URL: http://svn.apache.org/viewvc?rev=1175455&view=rev
> Log:
> More interface @Overrides
>
> Modified:
>
> lucene/dev/branches/branch_3x/lucene/contrib/misc/src/test/org/apache/lucene/index/TestNRTManager.java
>
> lucene/dev/branches/branch_3x/lucene/contrib/misc/src/test/org/apache/lucene/search/TestSearcherManager.java
>
> Modified:
> lucene/dev/branches/branch_3x/lucene/contrib/misc/src/test/org/apache/lucene/index/TestNRTManager.java
> URL:
> http://svn.apache.org/viewvc/lucene/dev/branches/branch_3x/lucene/contrib/misc/src/test/org/apache/lucene/index/TestNRTManager.java?rev=1175455&r1=1175454&r2=1175455&view=diff
> ==
> ---
> lucene/dev/branches/branch_3x/lucene/contrib/misc/src/test/org/apache/lucene/index/TestNRTManager.java
> (original)
> +++
> lucene/dev/branches/branch_3x/lucene/contrib/misc/src/test/org/apache/lucene/index/TestNRTManager.java
> Sun Sep 25 18:11:20 2011
> @@ -181,7 +181,7 @@ public class TestNRTManager extends Thre
>
> nrt = new NRTManager(writer, es,
> new SearcherWarmer() {
> - @Override
> + // Not with Java 5: @Override
> public void warm(IndexSearcher s) throws
> IOException {
> TestNRTManager.this.warmCalled = true;
> s.search(new TermQuery(new Term("body",
> "united")), 10);
>
> Modified:
> lucene/dev/branches/branch_3x/lucene/contrib/misc/src/test/org/apache/lucene/search/TestSearcherManager.java
> URL:
> http://svn.apache.org/viewvc/lucene/dev/branches/branch_3x/lucene/contrib/misc/src/test/org/apache/lucene/search/TestSearcherManager.java?rev=1175455&r1=1175454&r2=1175455&view=diff
> ==
> ---
> lucene/dev/branches/branch_3x/lucene/contrib/misc/src/test/org/apache/lucene/search/TestSearcherManager.java
> (original)
> +++
> lucene/dev/branches/branch_3x/lucene/contrib/misc/src/test/org/apache/lucene/search/TestSearcherManager.java
> Sun Sep 25 18:11:20 2011
> @@ -47,7 +47,7 @@ public class TestSearcherManager extends
> writer.commit();
> mgr = new SearcherManager(dir,
> new SearcherWarmer() {
> - @Override
> + // Not with Java 5: @Override
> public void warm(IndexSearcher s) throws
> IOException {
> TestSearcherManager.this.warmCalled = true;
> s.search(new TermQuery(new Term("body",
> "united")), 10);
>
>
>
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-3456) Analysis Consumers should end and close any TokenStreams
[ https://issues.apache.org/jira/browse/LUCENE-3456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13114311#comment-13114311 ] Robert Muir commented on LUCENE-3456: - I'm iterating on this patch a bit, so we don't do duplicate work. it probably won't find all the problems, but it will fix some. > Analysis Consumers should end and close any TokenStreams > > > Key: LUCENE-3456 > URL: https://issues.apache.org/jira/browse/LUCENE-3456 > Project: Lucene - Java > Issue Type: Sub-task >Reporter: Chris Male > Attachments: LUCENE-3456.patch > > > While converting consumers over to using reusableTokenStream, I notice many > don't call end() or close() on TokenStreams. > Even if they are test TSs only created once, we should follow the defined > usage pattern. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-3454) rename optimize to a less cool-sounding name
[ https://issues.apache.org/jira/browse/LUCENE-3454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13114304#comment-13114304 ] DM Smith commented on LUCENE-3454: -- When I started w/ Lucene, I read the docs and was drawn to call optimize because of its "cool name." However, it was reading the documentation at the time that convinced me that it was appropriate for my use case: Creating an index that once created would never be modified. It needed to be as fast as possible for search on low performance computing devices (old laptops, ancient computers, netbooks, phones, ...). Maybe I misunderstood, but wasn't it and isn't it still appropriate for that? And I have no idea what NooSegments means. If you want a really uncool name how about dumbDown()? But either way, please document the appropriate use cases for it. > rename optimize to a less cool-sounding name > > > Key: LUCENE-3454 > URL: https://issues.apache.org/jira/browse/LUCENE-3454 > Project: Lucene - Java > Issue Type: Improvement >Affects Versions: 4.0 >Reporter: Robert Muir > > I think users see the name optimize and feel they must do this, because who > wants a suboptimal system? but this probably just results in wasted time and > resources. > maybe rename to collapseSegments or something? -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[JENKINS-MAVEN] Lucene-Solr-Maven-3.x #252: POMs out of sync
Build: https://builds.apache.org/job/Lucene-Solr-Maven-3.x/252/ No tests ran. Build Log (for compile errors): [...truncated 9609 lines...] clover.info: [echo] [echo] Clover not found. Code coverage reports disabled. [echo] clover: common.compile-core: compile-core: compile-test-framework: common.compile-test: [mkdir] Created dir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Maven-3.x/checkout/lucene/build/contrib/memory/classes/test [javac] Compiling 1 source file to /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Maven-3.x/checkout/lucene/build/contrib/memory/classes/test [copy] Copying 2 files to /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Maven-3.x/checkout/lucene/build/contrib/memory/classes/test build-artifacts-and-tests: [echo] Building misc... common.init: compile-lucene-core: jflex-uptodate-check: jflex-notice: javacc-uptodate-check: javacc-notice: init: clover.setup: clover.info: [echo] [echo] Clover not found. Code coverage reports disabled. [echo] clover: common.compile-core: compile-core: init: clover.setup: clover.info: [echo] [echo] Clover not found. Code coverage reports disabled. [echo] clover: compile-core: jar-core: [jar] Building jar: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Maven-3.x/checkout/lucene/build/contrib/misc/lucene-misc-3.5-SNAPSHOT.jar jar: compile-test: [echo] Building misc... common.init: compile-lucene-core: init: clover.setup: clover.info: [echo] [echo] Clover not found. Code coverage reports disabled. [echo] clover: compile-core: compile-test-framework: common.compile-test: [mkdir] Created dir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Maven-3.x/checkout/lucene/build/contrib/misc/classes/test [javac] Compiling 12 source files to /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Maven-3.x/checkout/lucene/build/contrib/misc/classes/test [javac] /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Maven-3.x/checkout/lucene/contrib/misc/src/test/org/apache/lucene/index/TestNRTManager.java:184: method does not override a method from its superclass [javac]@Override [javac] ^ [javac] /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Maven-3.x/checkout/lucene/contrib/misc/src/test/org/apache/lucene/search/TestSearcherManager.java:50: method does not override a method from its superclass [javac] @Override [javac] ^ [javac] Note: Some input files use or override a deprecated API. [javac] Note: Recompile with -Xlint:deprecation for details. [javac] 2 errors [...truncated 15 lines...] - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-3457) Upgrade to commons-compress 1.2
[ https://issues.apache.org/jira/browse/LUCENE-3457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13114302#comment-13114302 ] Doron Cohen commented on LUCENE-3457: - ok great, thanks Robert, so this has nothing to do with the comprees jar update. I'll commit shortly. > Upgrade to commons-compress 1.2 > --- > > Key: LUCENE-3457 > URL: https://issues.apache.org/jira/browse/LUCENE-3457 > Project: Lucene - Java > Issue Type: Bug > Components: modules/benchmark >Reporter: Doron Cohen >Assignee: Doron Cohen >Priority: Minor > Fix For: 3.5, 4.0 > > Attachments: LUCENE-3457.patch, test.out.gz > > > Commons Compress bug COMPRESS-127 was fixed in 1.2, so the workaround in > benchmark's StreamUtils is no longer required. Compress is also used in solr. > Replace with new jar in both benchmark and solr and get rid of that > workaround. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-2279) eliminate pathological performance on StopFilter when using a Set instead of CharArraySet
[
https://issues.apache.org/jira/browse/LUCENE-2279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13114300#comment-13114300
]
Uwe Schindler commented on LUCENE-2279:
---
You misunderstood the response: StopFilter indeed did not change. The change is
now that in Lucene 4.0 all Analyzers are required to reuse TokenStream
instances, so the StopFilter is only produced only once in your application
(when the Analyzer is created).
> eliminate pathological performance on StopFilter when using a Set
> instead of CharArraySet
> -
>
> Key: LUCENE-2279
> URL: https://issues.apache.org/jira/browse/LUCENE-2279
> Project: Lucene - Java
> Issue Type: Improvement
> Components: modules/analysis
>Reporter: thushara wijeratna
>Priority: Minor
>
> passing a Set to a StopFilter instead of a CharArraySet results in a
> very slow filter.
> this is because for each document, Analyzer.tokenStream() is called, which
> ends up calling the StopFilter (if used). And if a regular Set is
> used in the StopFilter all the elements of the set are copied to a
> CharArraySet, as we can see in it's ctor:
> public StopFilter(boolean enablePositionIncrements, TokenStream input, Set
> stopWords, boolean ignoreCase)
> {
> super(input);
> if (stopWords instanceof CharArraySet) {
> this.stopWords = (CharArraySet)stopWords;
> } else {
> this.stopWords = new CharArraySet(stopWords.size(), ignoreCase);
> this.stopWords.addAll(stopWords);
> }
> this.enablePositionIncrements = enablePositionIncrements;
> init();
> }
> i feel we should make the StopFilter signature specific, as in specifying
> CharArraySet vs Set, and there should be a JavaDoc warning on using the other
> variants of the StopFilter as they all result in a copy for each invocation
> of Analyzer.tokenStream().
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-2279) eliminate pathological performance on StopFilter when using a Set instead of CharArraySet
[
https://issues.apache.org/jira/browse/LUCENE-2279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13114298#comment-13114298
]
thushara wijeratna commented on LUCENE-2279:
I have not used this in a while. I just looked at the StopFilter source, and it
seems identical to what I noticed before. I think the argument you make is that
users should not use that form of the StopFilter constructor. I don't have a
strong opinion on this. In any case, it was not a *major* issue to start with.
Thx for the response.
> eliminate pathological performance on StopFilter when using a Set
> instead of CharArraySet
> -
>
> Key: LUCENE-2279
> URL: https://issues.apache.org/jira/browse/LUCENE-2279
> Project: Lucene - Java
> Issue Type: Improvement
> Components: modules/analysis
>Reporter: thushara wijeratna
>Priority: Minor
>
> passing a Set to a StopFilter instead of a CharArraySet results in a
> very slow filter.
> this is because for each document, Analyzer.tokenStream() is called, which
> ends up calling the StopFilter (if used). And if a regular Set is
> used in the StopFilter all the elements of the set are copied to a
> CharArraySet, as we can see in it's ctor:
> public StopFilter(boolean enablePositionIncrements, TokenStream input, Set
> stopWords, boolean ignoreCase)
> {
> super(input);
> if (stopWords instanceof CharArraySet) {
> this.stopWords = (CharArraySet)stopWords;
> } else {
> this.stopWords = new CharArraySet(stopWords.size(), ignoreCase);
> this.stopWords.addAll(stopWords);
> }
> this.enablePositionIncrements = enablePositionIncrements;
> init();
> }
> i feel we should make the StopFilter signature specific, as in specifying
> CharArraySet vs Set, and there should be a JavaDoc warning on using the other
> variants of the StopFilter as they all result in a copy for each invocation
> of Analyzer.tokenStream().
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
problems writing a custom Similarity class
Hi all,
I'm just starting to get into solr development and I want to try writing a
custom Scoring Class. I copied the DefaultSimilarity class and renamed it
to TestSimilarity and then I just overwrote the function idf to return 1 so
that scores only look at the tf:
@Override
public float idf(int docFreq, int numDocs) {
return 1.0f;
}
I then make sure my TestSimilarity is always used by editing
conf/schema.xml to have this line:
Scoring seems to still be using an idf score that is not 1 and returning
results sorted by rareness of a phrase instead of frequency of the word.
I am following this tutorial:
http://www.lucenetutorial.com/advanced-topics/scoring.html
Is there something else I need to do to sort by only term frequency?
I'd appreciate any help oor suggestions on this.
Thanks,
Jason
[jira] [Commented] (SOLR-2769) HunspellStemFilterFactory
[ https://issues.apache.org/jira/browse/SOLR-2769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13114296#comment-13114296 ] Jan Høydahl commented on SOLR-2769: --- Thanks for the @see fix, Chris. Robert, you're probably right. I've rephrased some of the statements and added reservations about dictionary quality. > HunspellStemFilterFactory > - > > Key: SOLR-2769 > URL: https://issues.apache.org/jira/browse/SOLR-2769 > Project: Solr > Issue Type: New Feature > Components: Schema and Analysis >Reporter: Jan Høydahl > Labels: stemming > Fix For: 3.5, 4.0 > > Attachments: SOLR-2769-branch_3x.patch, SOLR-2769-branch_3x.patch, > SOLR-2769.patch, SOLR-2769.patch, SOLR-2769.patch, SOLR-2769.patch > > > Now that Hunspell stemmer is added to Lucene (LUCENE-3414), let's make a > Factory for it in Solr -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Created] (LUCENE-3463) Jenins trunk tests (nightly only) fail quite often with OOM in Automaton/FST tests
Jenins trunk tests (nightly only) fail quite often with OOM in Automaton/FST
tests
--
Key: LUCENE-3463
URL: https://issues.apache.org/jira/browse/LUCENE-3463
Project: Lucene - Java
Issue Type: Bug
Reporter: Uwe Schindler
The nightly Job Lucene-trunk quite often fails with OOM (in several methods,
not always in the same test):
Example from last night (this time a huge Automaton):
{noformat}
[junit] java.lang.OutOfMemoryError: Java heap space
[junit] Dumping heap to
/home/hudson/hudson-slave/workspace/Lucene-trunk/heapdumps/java_pid38855.hprof
...
[junit] Heap dump file created [86965954 bytes in 1.186 secs]
[junit] Testsuite: org.apache.lucene.index.TestTermsEnum
[junit] Testcase: testIntersectRandom(org.apache.lucene.index.TestTermsEnum):
Caused an ERROR
[junit] Java heap space
[junit] java.lang.OutOfMemoryError: Java heap space
[junit] at
org.apache.lucene.util.automaton.RunAutomaton.(RunAutomaton.java:128)
[junit] at
org.apache.lucene.util.automaton.ByteRunAutomaton.(ByteRunAutomaton.java:28)
[junit] at
org.apache.lucene.util.automaton.CompiledAutomaton.(CompiledAutomaton.java:134)
[junit] at
org.apache.lucene.index.TestTermsEnum.testIntersectRandom(TestTermsEnum.java:266)
[junit] at
org.apache.lucene.util.LuceneTestCase$2$1.evaluate(LuceneTestCase.java:611)
[junit] at
org.apache.lucene.util.LuceneTestCaseRunner.runChild(LuceneTestCaseRunner.java:148)
[junit] at
org.apache.lucene.util.LuceneTestCaseRunner.runChild(LuceneTestCaseRunner.java:50)
[junit]
[junit]
[junit] Tests run: 6, Failures: 0, Errors: 1, Time elapsed: 11.699 sec
{noformat}
Other traces:
{noformat}
[junit] Testsuite: org.apache.lucene.util.fst.TestFSTs
[junit] Testcase: testRealTerms(org.apache.lucene.util.fst.TestFSTs): Caused
an ERROR
[junit] Java heap space
[junit] java.lang.OutOfMemoryError: Java heap space
[junit] at org.apache.lucene.util.ArrayUtil.grow(ArrayUtil.java:338)
[junit] at
org.apache.lucene.util.fst.FST$BytesWriter.writeBytes(FST.java:927)
[junit] at
org.apache.lucene.util.fst.ByteSequenceOutputs.write(ByteSequenceOutputs.java:113)
[junit] at
org.apache.lucene.util.fst.ByteSequenceOutputs.write(ByteSequenceOutputs.java:32)
[junit] at org.apache.lucene.util.fst.FST.addNode(FST.java:451)
[junit] at org.apache.lucene.util.fst.NodeHash.add(NodeHash.java:122)
[junit] at
org.apache.lucene.util.fst.Builder.compileNode(Builder.java:180)
[junit] at org.apache.lucene.util.fst.Builder.finish(Builder.java:495)
[junit] at
org.apache.lucene.index.codecs.memory.MemoryCodec$TermsWriter.finish(MemoryCodec.java:232)
[junit] at
org.apache.lucene.index.FreqProxTermsWriterPerField.flush(FreqProxTermsWriterPerField.java:414)
[junit] at
org.apache.lucene.index.FreqProxTermsWriter.flush(FreqProxTermsWriter.java:92)
[junit] at org.apache.lucene.index.TermsHash.flush(TermsHash.java:117)
[junit] at
org.apache.lucene.index.DocInverter.flush(DocInverter.java:80)
[junit] at
org.apache.lucene.index.DocFieldProcessor.flush(DocFieldProcessor.java:78)
[junit] at
org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:472)
[junit] at
org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:420)
[junit] at
org.apache.lucene.index.DocumentsWriter.flushAllThreads(DocumentsWriter.java:568)
[junit] at
org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:366)
[junit] at
org.apache.lucene.index.IndexReader.open(IndexReader.java:317)
[junit] at
org.apache.lucene.util.fst.TestFSTs.testRealTerms(TestFSTs.java:1034)
[junit] at
org.apache.lucene.util.LuceneTestCase$2$1.evaluate(LuceneTestCase.java:611)
{noformat}
or:
{noformat}
[junit] Testsuite: org.apache.lucene.util.automaton.TestCompiledAutomaton
[junit] Testcase:
testRandom(org.apache.lucene.util.automaton.TestCompiledAutomaton): Caused an
ERROR
[junit] Java heap space
[junit] java.lang.OutOfMemoryError: Java heap space
[junit] at
org.apache.lucene.util.automaton.RunAutomaton.(RunAutomaton.java:128)
[junit] at
org.apache.lucene.util.automaton.ByteRunAutomaton.(ByteRunAutomaton.java:28)
[junit] at
org.apache.lucene.util.automaton.CompiledAutomaton.(CompiledAutomaton.java:134)
[junit] at
org.apache.lucene.util.automaton.TestCompiledAutomaton.build(TestCompiledAutomaton.java:39)
[junit] at
org.apache.lucene.util.automaton.TestCompiledAutomaton.testTerms(TestCompiledAutomaton.java:55)
[junit] at
org.apache.lucene.util.automaton.TestCompiledAutomaton.testRandom(TestCompiledAutomaton.java:101)
[junit] at
org.apache.lucene.util.LuceneTestCase$2$1.evaluate(LuceneTestCase.java:611)
[junit]
Client-side handling of JSON documents in SolrJ
The SolrJ code has a class DirectXmlRequest to send XML documents to Solr, but I don't see anything similar for JSON. Is it worth creating a JIRA? I didn't see anything on a quick scan. Speaking of client-side handling, there are a couple of utility routines in ClientUtils for handling XML, should there be another JIRA for providing something similar for JSON? The two methods in ClientUtils I'm thinking of in particular are: writeXML and toXML. Erick - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[JENKINS] Lucene-trunk - Build # 1684 - Still Failing
Build: https://builds.apache.org/job/Lucene-trunk/1684/ 1 tests failed. FAILED: org.apache.lucene.index.TestTermsEnum.testIntersectRandom Error Message: Java heap space Stack Trace: java.lang.OutOfMemoryError: Java heap space at org.apache.lucene.util.automaton.RunAutomaton.(RunAutomaton.java:128) at org.apache.lucene.util.automaton.ByteRunAutomaton.(ByteRunAutomaton.java:28) at org.apache.lucene.util.automaton.CompiledAutomaton.(CompiledAutomaton.java:134) at org.apache.lucene.index.TestTermsEnum.testIntersectRandom(TestTermsEnum.java:266) at org.apache.lucene.util.LuceneTestCase$2$1.evaluate(LuceneTestCase.java:611) at org.apache.lucene.util.LuceneTestCaseRunner.runChild(LuceneTestCaseRunner.java:148) at org.apache.lucene.util.LuceneTestCaseRunner.runChild(LuceneTestCaseRunner.java:50) Build Log (for compile errors): [...truncated 12889 lines...] - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-3440) FastVectorHighlighter: IDF-weighted terms for ordered fragments
[
https://issues.apache.org/jira/browse/LUCENE-3440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13114291#comment-13114291
]
sebastian L. commented on LUCENE-3440:
--
bq. Patch looks great!
Thanks.
bq. 1. For the new totalWeight, add getter method and modify toString() in
WeightedFragInfo().
Okay.
bq. 2. The patch uses hard-coded DefaultSimilarity to calculate idf. I don't
think that a custom similarity can be used here, too. If so, how about just
copying idf method rather than creating a similarity object?
I played a little with log(numDocs - docFreq + 0.5 / docFreq + 0.5) but is
seems to make no difference. If I'm not mistaken there is no method
IndexReader.getSimilarity() or IndexReader.getDefaultSimilarity().
Therefore: Okay.
bq. 3. Please do not hesitate to update ScoreComparator (do not add
WeightOrderFragmentsBuilder)
Hm, I thought about something like that:
{code:xml}
{code}
For Solr-users (like me). If somebody would like to use the boost-based
ordering, he could. Maybe, for some use-cases the boost-based approach is
better than the weighted one.
bq. 4 Could you update package javadoc (
https://builds.apache.org//job/Lucene-trunk/javadoc/contrib-highlighter/org/apache/lucene/search/vectorhighlight/package-summary.html#package_description
) and insert totalWeight into description and figures.
Okay.
bq. 5. use docFreq(String field, BytesRef term) version for trunk to avoid
creating Term object.
Okay.
bq. I agree. I think if there is a table so that we can compare totalBoost
(current) and totalWeight (patch) with real values, it helps a lot.
I'll write some Proof-of-concept Test-Class. But this may take some time.
I discovered a little problem with overlapping terms, depending on the
analyzing-process:
WeightedPhraseInfo.addIfNoOverlap() dumps the second part of hyphenated words
(for example: social-economics). The result is that all informations in
TermInfo are lost and not available for computing the fragments weight. I
simple modified WeightedPhraseInfo.addIfNoOverlap() a little to change this
behavior:
{code:java}
void addIfNoOverlap( WeightedPhraseInfo wpi ){
for( WeightedPhraseInfo existWpi : phraseList ){
if( existWpi.isOffsetOverlap( wpi ) ) {
existWpi.termInfos.addAll( wpi.termInfos );
return;
}
}
phraseList.add( wpi );
}
{code}
But I am not sure if there could be some unforeseen site-effects?
> FastVectorHighlighter: IDF-weighted terms for ordered fragments
>
>
> Key: LUCENE-3440
> URL: https://issues.apache.org/jira/browse/LUCENE-3440
> Project: Lucene - Java
> Issue Type: Improvement
> Components: modules/highlighter
>Affects Versions: 3.5, 4.0
>Reporter: sebastian L.
>Priority: Minor
> Labels: FastVectorHighlighter
> Fix For: 3.5, 4.0
>
> Attachments: LUCENE-3.5-SNAPSHOT-3440-3.patch,
> LUCENE-4.0-SNAPSHOT-3440-3.patch
>
>
> The FastVectorHighlighter uses for every term found in a fragment an equal
> weight, which causes a higher ranking for fragments with a high number of
> words or, in the worst case, a high number of very common words than
> fragments that contains *all* of the terms used in the original query.
> This patch provides ordered fragments with IDF-weighted terms:
> total weight = total weight + IDF for unique term per fragment * boost of
> query;
> The ranking-formula should be the same, or at least similar, to that one used
> in org.apache.lucene.search.highlight.QueryTermScorer.
> The patch is simple, but it works for us.
> Some ideas:
> - A better approach would be moving the whole fragments-scoring into a
> separate class.
> - Switch scoring via parameter
> - Exact phrases should be given a even better score, regardless if a
> phrase-query was executed or not
> - edismax/dismax-parameters pf, ps and pf^boost should be observed and
> corresponding fragments should be ranked higher
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-3454) rename optimize to a less cool-sounding name
[ https://issues.apache.org/jira/browse/LUCENE-3454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13114290#comment-13114290 ] Eks Dev commented on LUCENE-3454: - as a user, - +1 to parameter, if you do not know what NooSegments mean, you shouldn't be invoking this method. - what about tryMerge(int ), attempMerge > rename optimize to a less cool-sounding name > > > Key: LUCENE-3454 > URL: https://issues.apache.org/jira/browse/LUCENE-3454 > Project: Lucene - Java > Issue Type: Improvement >Affects Versions: 4.0 >Reporter: Robert Muir > > I think users see the name optimize and feel they must do this, because who > wants a suboptimal system? but this probably just results in wasted time and > resources. > maybe rename to collapseSegments or something? -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
RE: [JENKINS] Lucene-3.x - Build # 502 - Failure
Fixed Javadocs warnings. - Uwe Schindler H.-H.-Meier-Allee 63, D-28213 Bremen http://www.thetaphi.de eMail: u...@thetaphi.de > -Original Message- > From: Apache Jenkins Server [mailto:jenk...@builds.apache.org] > Sent: Sunday, September 25, 2011 6:26 PM > To: dev@lucene.apache.org > Subject: [JENKINS] Lucene-3.x - Build # 502 - Failure > > Build: https://builds.apache.org/job/Lucene-3.x/502/ > > No tests ran. > > Build Log (for compile errors): > [...truncated 11619 lines...] > > javacc-uptodate-check: > > javacc-notice: > > init: > > clover.setup: > > clover.info: > [echo] > [echo] Clover not found. Code coverage reports disabled. > [echo] > > clover: > > common.compile-core: > > compile-core: > > init: > > clover.setup: > > clover.info: > [echo] > [echo] Clover not found. Code coverage reports disabled. > [echo] > > clover: > > common.compile-core: > > compile-core: > > javadocs: > [mkdir] Created dir: /usr/home/hudson/hudson-slave/workspace/Lucene- > 3.x/checkout/lucene/build/docs/api/contrib-memory > [copy] Copying 1 file to /usr/home/hudson/hudson-slave/workspace/Lucene- > 3.x/checkout/lucene/build/docs/api/contrib-memory > [copy] Copying 1 file to /usr/home/hudson/hudson-slave/workspace/Lucene- > 3.x/checkout/lucene/build/docs/api/contrib-memory > [javadoc] Generating Javadoc > [javadoc] Javadoc execution > [javadoc] Loading source files for package org.apache.lucene.index.memory... > [javadoc] Constructing Javadoc information... > [javadoc] Standard Doclet version 1.5.0_16-p9 > [javadoc] Building tree for all the packages and classes... > [javadoc] Generating /usr/home/hudson/hudson-slave/workspace/Lucene- > 3.x/checkout/lucene/build/docs/api/contrib-memory/serialized-form.html... > [javadoc] Copying file /usr/home/hudson/hudson-slave/workspace/Lucene- > 3.x/checkout/lucene/src/tools/prettify/stylesheet+prettify.css to file > /usr/home/hudson/hudson-slave/workspace/Lucene- > 3.x/checkout/lucene/build/docs/api/contrib-memory/stylesheet+prettify.css... > [javadoc] Building index for all the packages and classes... > [javadoc] Building index for all classes... > [javadoc] Generating /usr/home/hudson/hudson-slave/workspace/Lucene- > 3.x/checkout/lucene/build/docs/api/contrib-memory/help-doc.html... > [javadoc] Note: Custom tags that were not seen: @lucene.experimental, > @lucene.internal > [jar] Building jar: /usr/home/hudson/hudson-slave/workspace/Lucene- > 3.x/checkout/lucene/build/contrib/memory/lucene-memory-3.5-2011-09- > 25_16-17-03-javadoc.jar > [echo] Building misc... > > common.init: > > compile-lucene-core: > > jflex-uptodate-check: > > jflex-notice: > > javacc-uptodate-check: > > javacc-notice: > > init: > > clover.setup: > > clover.info: > [echo] > [echo] Clover not found. Code coverage reports disabled. > [echo] > > clover: > > common.compile-core: > > compile-core: > > init: > > clover.setup: > > clover.info: > [echo] > [echo] Clover not found. Code coverage reports disabled. > [echo] > > clover: > > compile-core: > [mkdir] Created dir: /usr/home/hudson/hudson-slave/workspace/Lucene- > 3.x/checkout/lucene/build/contrib/misc/classes/java > [javac] Compiling 19 source files to /usr/home/hudson/hudson- > slave/workspace/Lucene-3.x/checkout/lucene/build/contrib/misc/classes/java > [javac] /usr/home/hudson/hudson-slave/workspace/Lucene- > 3.x/checkout/lucene/contrib/misc/src/java/org/apache/lucene/search/Searche > rManager.java:197: method does not override a method from its superclass > [javac] @Override > [javac]^ > [javac] Note: Some input files use or override a deprecated API. > [javac] Note: Recompile with -Xlint:deprecation for details. > [javac] 1 error > [...truncated 1593 lines...] > > init: > > clover.setup: > > clover.info: > [echo] > [echo] Clover not found. Code coverage reports disabled. > [echo] > > clover: > > common.compile-core: > > compile-core: > > jar-core: > [jar] Building jar: /usr/home/hudson/hudson-slave/workspace/Lucene- > 3.x/checkout/lucene/build/contrib/memory/lucene-memory-3.5-2011-09- > 25_16-17-03.jar > > jar: > > compile-test: > [echo] Building memory... > > check-queryparser-uptodate: > > jar-queryparser: > > common.init: > > compile-lucene-core: > > init: > > clover.setup: > > clover.info: > [echo] > [echo] Clover not found. Code coverage reports disabled. > [echo] > > clover: > > common.compile-core: > > compile-core: > > compile-test-framework: > > common.compile-test: > [mkdir] Created dir: /usr/home/hudson/hudson-slave/workspace/Lucene- > 3.x/checkout/lucene/build/contrib/memory/classes/test > [javac] Compiling 1 source file to /usr/home/hudson/hudson- > slave/workspace/Lucene- > 3.x/checkout/lucene/build/contrib/memory/classes/test > [copy] Copying 2
[JENKINS] Lucene-3.x - Build # 502 - Failure
Build: https://builds.apache.org/job/Lucene-3.x/502/ No tests ran. Build Log (for compile errors): [...truncated 11619 lines...] javacc-uptodate-check: javacc-notice: init: clover.setup: clover.info: [echo] [echo] Clover not found. Code coverage reports disabled. [echo] clover: common.compile-core: compile-core: init: clover.setup: clover.info: [echo] [echo] Clover not found. Code coverage reports disabled. [echo] clover: common.compile-core: compile-core: javadocs: [mkdir] Created dir: /usr/home/hudson/hudson-slave/workspace/Lucene-3.x/checkout/lucene/build/docs/api/contrib-memory [copy] Copying 1 file to /usr/home/hudson/hudson-slave/workspace/Lucene-3.x/checkout/lucene/build/docs/api/contrib-memory [copy] Copying 1 file to /usr/home/hudson/hudson-slave/workspace/Lucene-3.x/checkout/lucene/build/docs/api/contrib-memory [javadoc] Generating Javadoc [javadoc] Javadoc execution [javadoc] Loading source files for package org.apache.lucene.index.memory... [javadoc] Constructing Javadoc information... [javadoc] Standard Doclet version 1.5.0_16-p9 [javadoc] Building tree for all the packages and classes... [javadoc] Generating /usr/home/hudson/hudson-slave/workspace/Lucene-3.x/checkout/lucene/build/docs/api/contrib-memory/serialized-form.html... [javadoc] Copying file /usr/home/hudson/hudson-slave/workspace/Lucene-3.x/checkout/lucene/src/tools/prettify/stylesheet+prettify.css to file /usr/home/hudson/hudson-slave/workspace/Lucene-3.x/checkout/lucene/build/docs/api/contrib-memory/stylesheet+prettify.css... [javadoc] Building index for all the packages and classes... [javadoc] Building index for all classes... [javadoc] Generating /usr/home/hudson/hudson-slave/workspace/Lucene-3.x/checkout/lucene/build/docs/api/contrib-memory/help-doc.html... [javadoc] Note: Custom tags that were not seen: @lucene.experimental, @lucene.internal [jar] Building jar: /usr/home/hudson/hudson-slave/workspace/Lucene-3.x/checkout/lucene/build/contrib/memory/lucene-memory-3.5-2011-09-25_16-17-03-javadoc.jar [echo] Building misc... common.init: compile-lucene-core: jflex-uptodate-check: jflex-notice: javacc-uptodate-check: javacc-notice: init: clover.setup: clover.info: [echo] [echo] Clover not found. Code coverage reports disabled. [echo] clover: common.compile-core: compile-core: init: clover.setup: clover.info: [echo] [echo] Clover not found. Code coverage reports disabled. [echo] clover: compile-core: [mkdir] Created dir: /usr/home/hudson/hudson-slave/workspace/Lucene-3.x/checkout/lucene/build/contrib/misc/classes/java [javac] Compiling 19 source files to /usr/home/hudson/hudson-slave/workspace/Lucene-3.x/checkout/lucene/build/contrib/misc/classes/java [javac] /usr/home/hudson/hudson-slave/workspace/Lucene-3.x/checkout/lucene/contrib/misc/src/java/org/apache/lucene/search/SearcherManager.java:197: method does not override a method from its superclass [javac] @Override [javac]^ [javac] Note: Some input files use or override a deprecated API. [javac] Note: Recompile with -Xlint:deprecation for details. [javac] 1 error [...truncated 1593 lines...] init: clover.setup: clover.info: [echo] [echo] Clover not found. Code coverage reports disabled. [echo] clover: common.compile-core: compile-core: jar-core: [jar] Building jar: /usr/home/hudson/hudson-slave/workspace/Lucene-3.x/checkout/lucene/build/contrib/memory/lucene-memory-3.5-2011-09-25_16-17-03.jar jar: compile-test: [echo] Building memory... check-queryparser-uptodate: jar-queryparser: common.init: compile-lucene-core: init: clover.setup: clover.info: [echo] [echo] Clover not found. Code coverage reports disabled. [echo] clover: common.compile-core: compile-core: compile-test-framework: common.compile-test: [mkdir] Created dir: /usr/home/hudson/hudson-slave/workspace/Lucene-3.x/checkout/lucene/build/contrib/memory/classes/test [javac] Compiling 1 source file to /usr/home/hudson/hudson-slave/workspace/Lucene-3.x/checkout/lucene/build/contrib/memory/classes/test [copy] Copying 2 files to /usr/home/hudson/hudson-slave/workspace/Lucene-3.x/checkout/lucene/build/contrib/memory/classes/test build-artifacts-and-tests: [echo] Building misc... common.init: compile-lucene-core: jflex-uptodate-check: jflex-notice: javacc-uptodate-check: javacc-notice: init: clover.setup: clover.info: [echo] [echo] Clover not found. Code coverage reports disabled. [echo] clover: common.compile-core: compile-core: init: clover.setup: clover.info: [echo] [echo] Clover not found. Code coverage reports disabled. [echo] clover: compile-core: [javac] Compiling 5 source files to /usr/home/hudson/hudso
[jira] [Resolved] (LUCENE-3445) Add SearcherManager, to manage IndexSearcher usage across threads and reopens
[ https://issues.apache.org/jira/browse/LUCENE-3445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael McCandless resolved LUCENE-3445. Resolution: Fixed > Add SearcherManager, to manage IndexSearcher usage across threads and reopens > - > > Key: LUCENE-3445 > URL: https://issues.apache.org/jira/browse/LUCENE-3445 > Project: Lucene - Java > Issue Type: Improvement >Reporter: Michael McCandless >Assignee: Michael McCandless > Fix For: 3.5, 4.0 > > Attachments: LUCENE-3445.patch, LUCENE-3445.patch > > > This is a simple helper class I wrote for Lucene in Action 2nd ed. > I'd like to commit under Lucene (contrib/misc). > It simplifies using & reopening an IndexSearcher across multiple > threads, by using IndexReader's ref counts to know when it's safe > to close the reader. > In the process I also factored out a test base class for tests that > want to make lots of simultaneous indexing and searching threads, and > fixed TestNRTThreads (core), TestNRTManager (contrib/misc) and the new > TestSearcherManager (contrib/misc) to use this base class. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-3454) rename optimize to a less cool-sounding name
[ https://issues.apache.org/jira/browse/LUCENE-3454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13114283#comment-13114283 ] Shai Erera commented on LUCENE-3454: It's just that maybeMerge feels like asking a question, to which you expect an answer. Why is it so important to have this 'maybe' or 'ifNeeded' in the API? just like optimize(), MP decides what to merge, and optimize() can end up doing nothing as well ... Why do we have IR.reopen and not maybeReopen? That that it returns something is not much different than IW.merge(), IMO. > rename optimize to a less cool-sounding name > > > Key: LUCENE-3454 > URL: https://issues.apache.org/jira/browse/LUCENE-3454 > Project: Lucene - Java > Issue Type: Improvement >Affects Versions: 4.0 >Reporter: Robert Muir > > I think users see the name optimize and feel they must do this, because who > wants a suboptimal system? but this probably just results in wasted time and > resources. > maybe rename to collapseSegments or something? -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[JENKINS] Lucene-Solr-tests-only-3.x - Build # 10661 - Failure
Build: https://builds.apache.org/job/Lucene-Solr-tests-only-3.x/10661/ All tests passed Build Log (for compile errors): [...truncated 18060 lines...] - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-3462) Jenkins builds hang quite often in TestIndexWriterWithThreads.testCloseWithThreads
[
https://issues.apache.org/jira/browse/LUCENE-3462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13114280#comment-13114280
]
Uwe Schindler commented on LUCENE-3462:
---
Also interesting: No other thread is running at the same time, so it simply
waits and waits for nothing else (the rest of stack trace is only internal JVM
cleanup threads and the Jenkins server itsself).
> Jenkins builds hang quite often in
> TestIndexWriterWithThreads.testCloseWithThreads
> --
>
> Key: LUCENE-3462
> URL: https://issues.apache.org/jira/browse/LUCENE-3462
> Project: Lucene - Java
> Issue Type: Bug
>Affects Versions: 4.0
>Reporter: Uwe Schindler
>
> Last hung test run:
> [https://builds.apache.org/job/Lucene-Solr-tests-only-trunk/10638/console]
> {noformat}
> [junit] "main" prio=5 tid=0x000801ef3800 nid=0x1965c waiting on condition
> [0x7fbfd000]
> [junit]java.lang.Thread.State: WAITING (parking)
> [junit] at sun.misc.Unsafe.park(Native Method)
> [junit] - parking to wait for <0x000825d853a8> (a
> java.util.concurrent.locks.ReentrantLock$NonfairSync)
> [junit] at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> [junit] at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:838)
> [junit] at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:871)
> [junit] at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1201)
> [junit] at
> java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:214)
> [junit] at
> java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:290)
> [junit] at
> org.apache.lucene.index.DocumentsWriterFlushControl.markForFullFlush(DocumentsWriterFlushControl.java:403)
> [junit] at
> org.apache.lucene.index.DocumentsWriter.flushAllThreads(DocumentsWriter.java:557)
> [junit] - locked <0x000825d81998> (a
> org.apache.lucene.index.DocumentsWriter)
> [junit] at
> org.apache.lucene.index.IndexWriter.prepareCommit(IndexWriter.java:2776)
> [junit] - locked <0x000825d7d840> (a java.lang.Object)
> [junit] at
> org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2904)
> [junit] - locked <0x000825d7d830> (a java.lang.Object)
> [junit] at
> org.apache.lucene.index.IndexWriter.closeInternal(IndexWriter.java:1156)
> [junit] at
> org.apache.lucene.index.IndexWriter.close(IndexWriter.java:1099)
> [junit] at
> org.apache.lucene.index.TestIndexWriterWithThreads.testCloseWithThreads(TestIndexWriterWithThreads.java:200)
> [junit] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> [junit] at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> [junit] at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> [junit] at java.lang.reflect.Method.invoke(Method.java:616)
> [junit] at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
> [junit] at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> [junit] at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
> [junit] at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
> [junit] at org.junit.rules.TestWatchman$1.evaluate(TestWatchman.java:48)
> [junit] at
> org.apache.lucene.util.LuceneTestCase$2$1.evaluate(LuceneTestCase.java:611)
> [junit] at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
> [junit] at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
> [junit] at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:76)
> [junit] at
> org.apache.lucene.util.LuceneTestCaseRunner.runChild(LuceneTestCaseRunner.java:148)
> [junit] at
> org.apache.lucene.util.LuceneTestCaseRunner.runChild(LuceneTestCaseRunner.java:50)
> [junit] at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
> [junit] at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
> [junit] at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191)
> [junit] at
> org.junit.runners.ParentRunner.access$000(ParentRunner.java:42)
> [junit] at
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184)
> [junit] at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
> [junit] at
> org.junit.internal.runners
[jira] [Commented] (LUCENE-3462) Jenkins builds hang quite often in TestIndexWriterWithThreads.testCloseWithThreads
[
https://issues.apache.org/jira/browse/LUCENE-3462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13114279#comment-13114279
]
Uwe Schindler commented on LUCENE-3462:
---
I hung previously several times, but this time I requested stack trace. Seems
to be related to DWPT, never happens in 3.x.
Hung several times last 4 weeks (about 10 times).
> Jenkins builds hang quite often in
> TestIndexWriterWithThreads.testCloseWithThreads
> --
>
> Key: LUCENE-3462
> URL: https://issues.apache.org/jira/browse/LUCENE-3462
> Project: Lucene - Java
> Issue Type: Bug
>Affects Versions: 4.0
>Reporter: Uwe Schindler
>
> Last hung test run:
> [https://builds.apache.org/job/Lucene-Solr-tests-only-trunk/10638/console]
> {noformat}
> [junit] "main" prio=5 tid=0x000801ef3800 nid=0x1965c waiting on condition
> [0x7fbfd000]
> [junit]java.lang.Thread.State: WAITING (parking)
> [junit] at sun.misc.Unsafe.park(Native Method)
> [junit] - parking to wait for <0x000825d853a8> (a
> java.util.concurrent.locks.ReentrantLock$NonfairSync)
> [junit] at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> [junit] at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:838)
> [junit] at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:871)
> [junit] at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1201)
> [junit] at
> java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:214)
> [junit] at
> java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:290)
> [junit] at
> org.apache.lucene.index.DocumentsWriterFlushControl.markForFullFlush(DocumentsWriterFlushControl.java:403)
> [junit] at
> org.apache.lucene.index.DocumentsWriter.flushAllThreads(DocumentsWriter.java:557)
> [junit] - locked <0x000825d81998> (a
> org.apache.lucene.index.DocumentsWriter)
> [junit] at
> org.apache.lucene.index.IndexWriter.prepareCommit(IndexWriter.java:2776)
> [junit] - locked <0x000825d7d840> (a java.lang.Object)
> [junit] at
> org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2904)
> [junit] - locked <0x000825d7d830> (a java.lang.Object)
> [junit] at
> org.apache.lucene.index.IndexWriter.closeInternal(IndexWriter.java:1156)
> [junit] at
> org.apache.lucene.index.IndexWriter.close(IndexWriter.java:1099)
> [junit] at
> org.apache.lucene.index.TestIndexWriterWithThreads.testCloseWithThreads(TestIndexWriterWithThreads.java:200)
> [junit] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> [junit] at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> [junit] at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> [junit] at java.lang.reflect.Method.invoke(Method.java:616)
> [junit] at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
> [junit] at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> [junit] at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
> [junit] at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
> [junit] at org.junit.rules.TestWatchman$1.evaluate(TestWatchman.java:48)
> [junit] at
> org.apache.lucene.util.LuceneTestCase$2$1.evaluate(LuceneTestCase.java:611)
> [junit] at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
> [junit] at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
> [junit] at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:76)
> [junit] at
> org.apache.lucene.util.LuceneTestCaseRunner.runChild(LuceneTestCaseRunner.java:148)
> [junit] at
> org.apache.lucene.util.LuceneTestCaseRunner.runChild(LuceneTestCaseRunner.java:50)
> [junit] at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
> [junit] at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
> [junit] at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191)
> [junit] at
> org.junit.runners.ParentRunner.access$000(ParentRunner.java:42)
> [junit] at
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184)
> [junit] at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
> [junit] at
> org.junit.internal.runners.statements.RunAfters.evaluate(
[jira] [Created] (LUCENE-3462) Jenkins builds hang quite often in TestIndexWriterWithThreads.testCloseWithThreads
Jenkins builds hang quite often in
TestIndexWriterWithThreads.testCloseWithThreads
--
Key: LUCENE-3462
URL: https://issues.apache.org/jira/browse/LUCENE-3462
Project: Lucene - Java
Issue Type: Bug
Affects Versions: 4.0
Reporter: Uwe Schindler
Last hung test run:
[https://builds.apache.org/job/Lucene-Solr-tests-only-trunk/10638/console]
{noformat}
[junit] "main" prio=5 tid=0x000801ef3800 nid=0x1965c waiting on condition
[0x7fbfd000]
[junit]java.lang.Thread.State: WAITING (parking)
[junit] at sun.misc.Unsafe.park(Native Method)
[junit] - parking to wait for <0x000825d853a8> (a
java.util.concurrent.locks.ReentrantLock$NonfairSync)
[junit] at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
[junit] at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:838)
[junit] at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:871)
[junit] at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1201)
[junit] at
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:214)
[junit] at
java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:290)
[junit] at
org.apache.lucene.index.DocumentsWriterFlushControl.markForFullFlush(DocumentsWriterFlushControl.java:403)
[junit] at
org.apache.lucene.index.DocumentsWriter.flushAllThreads(DocumentsWriter.java:557)
[junit] - locked <0x000825d81998> (a
org.apache.lucene.index.DocumentsWriter)
[junit] at
org.apache.lucene.index.IndexWriter.prepareCommit(IndexWriter.java:2776)
[junit] - locked <0x000825d7d840> (a java.lang.Object)
[junit] at
org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2904)
[junit] - locked <0x000825d7d830> (a java.lang.Object)
[junit] at
org.apache.lucene.index.IndexWriter.closeInternal(IndexWriter.java:1156)
[junit] at
org.apache.lucene.index.IndexWriter.close(IndexWriter.java:1099)
[junit] at
org.apache.lucene.index.TestIndexWriterWithThreads.testCloseWithThreads(TestIndexWriterWithThreads.java:200)
[junit] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[junit] at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
[junit] at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[junit] at java.lang.reflect.Method.invoke(Method.java:616)
[junit] at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
[junit] at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
[junit] at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
[junit] at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
[junit] at org.junit.rules.TestWatchman$1.evaluate(TestWatchman.java:48)
[junit] at
org.apache.lucene.util.LuceneTestCase$2$1.evaluate(LuceneTestCase.java:611)
[junit] at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
[junit] at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
[junit] at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:76)
[junit] at
org.apache.lucene.util.LuceneTestCaseRunner.runChild(LuceneTestCaseRunner.java:148)
[junit] at
org.apache.lucene.util.LuceneTestCaseRunner.runChild(LuceneTestCaseRunner.java:50)
[junit] at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
[junit] at
org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
[junit] at
org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191)
[junit] at
org.junit.runners.ParentRunner.access$000(ParentRunner.java:42)
[junit] at
org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184)
[junit] at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
[junit] at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
[junit] at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
[junit] at
junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:39)
[junit] at
org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:420)
[junit] at
org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:911)
[junit] at
org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:743)
{nofo
[jira] [Updated] (LUCENE-3461) Adding same IndexDocValuesField twice trips assert
[ https://issues.apache.org/jira/browse/LUCENE-3461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael McCandless updated LUCENE-3461: --- Attachment: LUCENE-3461.patch Patch, catching more than one IDV fields in the same doc, and throwing IllegalArgExc. Test passes now. > Adding same IndexDocValuesField twice trips assert > -- > > Key: LUCENE-3461 > URL: https://issues.apache.org/jira/browse/LUCENE-3461 > Project: Lucene - Java > Issue Type: Bug >Reporter: Michael McCandless > Fix For: 4.0 > > Attachments: LUCENE-3461.patch, LUCENE-3461.patch > > > Doc values fields are single-valued by design, ie a given field name can only > occur once in the document. > But if you accidentally add it more than once, you get an assert error, which > is spooky because if you run w/o asserts maybe something eviler happens. > I think we should explicitly check for this and throw clear exc since user > could easily do this by accident. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-3460) Move handling of query only containing MUST_NOT to QueryParser (and remove QueryUtils.makeQueryable() hack in Solr)
[
https://issues.apache.org/jira/browse/LUCENE-3460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13114275#comment-13114275
]
Uwe Schindler commented on LUCENE-3460:
---
I also agree partly with Robert; with partly i mean: Returning nothing for
stop-words only confuses users, too, so we should do something.
Just to make it clear, this was on IRC:
{quote}
[15:11] chrismale: If someones query is all stopwords, then matching everything
makes sense.
[15:11] chrismale: every term got filtered out
[15:12] chrismale: Leaving you nothing
[15:12] chrismale: Nothing is parsed to MatchAllDocs
{quote}
So the issue description is *right* :-)
My idea was to add a setter to QP:
- the default is to work as it did before: To prevent the UOE, QP should simply
parse to an empty BQ in the case of only prohibited clauses. This would
preserve backwards compatibility for QP, it will never throw Exception
- add a "Solr" mode: This would add MatchAllDocsQuery with Occur.MUST, so this
would be identical behaviour like Solr does in QueryUtils. The special case in
Solr can be removed, QP never throws Exception.
- my favourite is this mode: If the user enters a query that only has
stop-words as positive clauses (we can easily detect this by counting terms),
the QP should throw a good ParseException explaining that the entered query
string does no produce a meaningful query, as all clauses *may* be stop words
(it could also be stripped of by other filters, not only stop filter - so the
message should be more generic). It would do this also if no negative queries
are involved. So a user entering "a the" would get a meaningfull explanation
that this is an invalid query.
> Move handling of query only containing MUST_NOT to QueryParser (and remove
> QueryUtils.makeQueryable() hack in Solr)
> ---
>
> Key: LUCENE-3460
> URL: https://issues.apache.org/jira/browse/LUCENE-3460
> Project: Lucene - Java
> Issue Type: Sub-task
> Components: core/queryparser
>Affects Versions: 3.4, 4.0
>Reporter: Uwe Schindler
>Assignee: Uwe Schindler
> Fix For: 4.0
>
>
> With the parent issue, users entering (a -b) into the queryparser can simply
> fail with an UnsupportedOperationException, if "a" is a stopword.
> Solr already has a hack to add a MatchAllDocsQuery, if a query only contains
> prohibited clauses.
> The other issue (not affecting prohibited clauses) with stopwords is: If the
> user enters (a the) into queryparser, the query will return no results, as
> "a" and "the" are stopwords. This confuses lots of people (not only
> developers, even ordinary users of our interfaces). If somebody queries for a
> stopword, the correct way to handle this is to return *all* documents
> (MatchAllDocsQuery).
> A possible solution, as suggested by Chris Male on IRC was: Add a flag to
> QueryParser to enable a "no-should-or-must-clauses" mode, where this is
> replaced by MatchAllDocs automatically. This would also solve the prohibited
> clause case, too.
> The stopword case is bad, but the opposite is as bad as returning all
> documents.
> At least this issue should somehow handle the only-prohibited case like Solr
> and remove the hack from Solr.
> Changing this in QueryParser is the more correct solution than doing this
> hidden in BQ.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
Re: showItems for LRUCache missing?
I created https://issues.apache.org/jira/browse/SOLR-2789 that has a patch for adding showItems. It also has a patch for displaying more details of what a query key is. That maybe should be it's own patch though? Eric On Sep 24, 2011, at 11:55 PM, Bill Bell wrote: > Let's add it. Add a SOLR ticket in JIRA. > > On 9/22/11 7:59 AM, "Eric Pugh" wrote: > >> Folks, >> >> I was trying to figure out what explicitly was in my various Solr caches. >> After building a custom request handler and using Reflection to gain >> access to the private Map's in the caches, I realized that showItems is >> an option on both fieldValueCache and FastLRUCache, but isn't an option >> on LRUCache... Is there a reason for that? If I wanted to submit a >> patch, is it best to do it against Trunk? >> >> Eric >> >> - >> Eric Pugh | Principal | OpenSource Connections, LLC | 434.466.1467 | >> http://www.opensourceconnections.com >> Co-Author: Solr 1.4 Enterprise Search Server available from >> http://www.packtpub.com/solr-1-4-enterprise-search-server >> This e-mail and all contents, including attachments, is considered to be >> Company Confidential unless explicitly stated otherwise, regardless of >> whether attachments are marked as such. >> >> >> >> >> >> >> >> >> >> >> - >> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org >> For additional commands, e-mail: dev-h...@lucene.apache.org >> > > > > - > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org > For additional commands, e-mail: dev-h...@lucene.apache.org > - Eric Pugh | Principal | OpenSource Connections, LLC | 434.466.1467 | http://www.opensourceconnections.com Co-Author: Solr 1.4 Enterprise Search Server available from http://www.packtpub.com/solr-1-4-enterprise-search-server This e-mail and all contents, including attachments, is considered to be Company Confidential unless explicitly stated otherwise, regardless of whether attachments are marked as such. - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-3454) rename optimize to a less cool-sounding name
[ https://issues.apache.org/jira/browse/LUCENE-3454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13114269#comment-13114269 ] Michael McCandless commented on LUCENE-3454: Hmm... I like maybeMerge() better because it makes it clear that there may be no merging done? But maybe we can find a different word from maybe instead :) mergeIfNeeded? > rename optimize to a less cool-sounding name > > > Key: LUCENE-3454 > URL: https://issues.apache.org/jira/browse/LUCENE-3454 > Project: Lucene - Java > Issue Type: Improvement >Affects Versions: 4.0 >Reporter: Robert Muir > > I think users see the name optimize and feel they must do this, because who > wants a suboptimal system? but this probably just results in wasted time and > resources. > maybe rename to collapseSegments or something? -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-3454) rename optimize to a less cool-sounding name
[ https://issues.apache.org/jira/browse/LUCENE-3454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13114266#comment-13114266 ] Shai Erera commented on LUCENE-3454: Fine. Can we use the opportunity to remove 'maybe'? Just merge(). If it ends up doing nothing, it's still ok, with me at least. > rename optimize to a less cool-sounding name > > > Key: LUCENE-3454 > URL: https://issues.apache.org/jira/browse/LUCENE-3454 > Project: Lucene - Java > Issue Type: Improvement >Affects Versions: 4.0 >Reporter: Robert Muir > > I think users see the name optimize and feel they must do this, because who > wants a suboptimal system? but this probably just results in wasted time and > resources. > maybe rename to collapseSegments or something? -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-3460) Move handling of query only containing MUST_NOT to QueryParser (and remove QueryUtils.makeQueryable() hack in Solr)
[ https://issues.apache.org/jira/browse/LUCENE-3460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13114265#comment-13114265 ] Chris Male commented on LUCENE-3460: I do kind of agree with Robert here. If there are no terms remaining (after analysis), then there shouldn't really be a Query at all. > Move handling of query only containing MUST_NOT to QueryParser (and remove > QueryUtils.makeQueryable() hack in Solr) > --- > > Key: LUCENE-3460 > URL: https://issues.apache.org/jira/browse/LUCENE-3460 > Project: Lucene - Java > Issue Type: Sub-task > Components: core/queryparser >Affects Versions: 3.4, 4.0 >Reporter: Uwe Schindler >Assignee: Uwe Schindler > Fix For: 4.0 > > > With the parent issue, users entering (a -b) into the queryparser can simply > fail with an UnsupportedOperationException, if "a" is a stopword. > Solr already has a hack to add a MatchAllDocsQuery, if a query only contains > prohibited clauses. > The other issue (not affecting prohibited clauses) with stopwords is: If the > user enters (a the) into queryparser, the query will return no results, as > "a" and "the" are stopwords. This confuses lots of people (not only > developers, even ordinary users of our interfaces). If somebody queries for a > stopword, the correct way to handle this is to return *all* documents > (MatchAllDocsQuery). > A possible solution, as suggested by Chris Male on IRC was: Add a flag to > QueryParser to enable a "no-should-or-must-clauses" mode, where this is > replaced by MatchAllDocs automatically. This would also solve the prohibited > clause case, too. > The stopword case is bad, but the opposite is as bad as returning all > documents. > At least this issue should somehow handle the only-prohibited case like Solr > and remove the hack from Solr. > Changing this in QueryParser is the more correct solution than doing this > hidden in BQ. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-3453) remove IndexDocValuesField
[ https://issues.apache.org/jira/browse/LUCENE-3453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13114264#comment-13114264 ] Michael McCandless commented on LUCENE-3453: I opened LUCENE-3461 for catching accidental multi-valued DV fields. > remove IndexDocValuesField > -- > > Key: LUCENE-3453 > URL: https://issues.apache.org/jira/browse/LUCENE-3453 > Project: Lucene - Java > Issue Type: Bug >Affects Versions: 4.0 >Reporter: Robert Muir >Assignee: Chris Male > > Its confusing how we present CSF functionality to the user, its actually not > a "field" but an "attribute" of a field like STORED or INDEXED. > Otherwise, its really hard to think about CSF because there is a mismatch > between the APIs and the index format. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (LUCENE-3461) Adding same IndexDocValuesField twice trips assert
[ https://issues.apache.org/jira/browse/LUCENE-3461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael McCandless updated LUCENE-3461: --- Attachment: LUCENE-3461.patch Patch w/ failing test case. Not yet sure where we should check... I think the assert is too low (it should remain) but higher up we should catch this. > Adding same IndexDocValuesField twice trips assert > -- > > Key: LUCENE-3461 > URL: https://issues.apache.org/jira/browse/LUCENE-3461 > Project: Lucene - Java > Issue Type: Bug >Reporter: Michael McCandless > Fix For: 4.0 > > Attachments: LUCENE-3461.patch > > > Doc values fields are single-valued by design, ie a given field name can only > occur once in the document. > But if you accidentally add it more than once, you get an assert error, which > is spooky because if you run w/o asserts maybe something eviler happens. > I think we should explicitly check for this and throw clear exc since user > could easily do this by accident. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Created] (LUCENE-3461) Adding same IndexDocValuesField twice trips assert
Adding same IndexDocValuesField twice trips assert -- Key: LUCENE-3461 URL: https://issues.apache.org/jira/browse/LUCENE-3461 Project: Lucene - Java Issue Type: Bug Reporter: Michael McCandless Fix For: 4.0 Doc values fields are single-valued by design, ie a given field name can only occur once in the document. But if you accidentally add it more than once, you get an assert error, which is spooky because if you run w/o asserts maybe something eviler happens. I think we should explicitly check for this and throw clear exc since user could easily do this by accident. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (LUCENE-3460) Move handling of query only containing MUST_NOT to QueryParser (and remove QueryUtils.makeQueryable() hack in Solr)
[ https://issues.apache.org/jira/browse/LUCENE-3460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uwe Schindler updated LUCENE-3460: -- Description: With the parent issue, users entering (a -b) into the queryparser can simply fail with an UnsupportedOperationException, if "a" is a stopword. Solr already has a hack to add a MatchAllDocsQuery, if a query only contains prohibited clauses. The other issue (not affecting prohibited clauses) with stopwords is: If the user enters (a the) into queryparser, the query will return no results, as "a" and "the" are stopwords. This confuses lots of people (not only developers, even ordinary users of our interfaces). If somebody queries for a stopword, the correct way to handle this is to return *all* documents (MatchAllDocsQuery). A possible solution, as suggested by Chris Male on IRC was: Add a flag to QueryParser to enable a "no-should-or-must-clauses" mode, where this is replaced by MatchAllDocs automatically. This would also solve the prohibited clause case, too. The stopword case is bad, but the opposite is as bad as returning all documents. At least this issue should somehow handle the only-prohibited case like Solr and remove the hack from Solr. Changing this in QueryParser is the more correct solution than doing this hidden in BQ. was: With the parent issue, users entering (a -b) into the queryparser can simply fail with an UnsupportedOperationException, if "a" is a stopword. Solr already has a hack to add a MatchAllDocsQuery, if a query only contains prohibited clauses. The other issue (not affecting prohibited clauses) with stopwords is: If the user enters (a the) into queryparser, the query will return no results, as "a" and "the" are stopwords. This confuses lots of people (not only developers, even ordinary users of our interfaces). If somebody queries for a stopword, the correct way to handle this is to return *all* documents (MatchAllDocsQuery). I propose to add a flag to QueryParser to enable a "no-should-or-must-clauses" mode, where this is replaced by MatchAllDocs automatically. This would also solve the prohibited clause case, too. Changing this in QueryParser is the more correct solution than doing this hidden in BQ. > Move handling of query only containing MUST_NOT to QueryParser (and remove > QueryUtils.makeQueryable() hack in Solr) > --- > > Key: LUCENE-3460 > URL: https://issues.apache.org/jira/browse/LUCENE-3460 > Project: Lucene - Java > Issue Type: Sub-task > Components: core/queryparser >Affects Versions: 3.4, 4.0 >Reporter: Uwe Schindler >Assignee: Uwe Schindler > Fix For: 4.0 > > > With the parent issue, users entering (a -b) into the queryparser can simply > fail with an UnsupportedOperationException, if "a" is a stopword. > Solr already has a hack to add a MatchAllDocsQuery, if a query only contains > prohibited clauses. > The other issue (not affecting prohibited clauses) with stopwords is: If the > user enters (a the) into queryparser, the query will return no results, as > "a" and "the" are stopwords. This confuses lots of people (not only > developers, even ordinary users of our interfaces). If somebody queries for a > stopword, the correct way to handle this is to return *all* documents > (MatchAllDocsQuery). > A possible solution, as suggested by Chris Male on IRC was: Add a flag to > QueryParser to enable a "no-should-or-must-clauses" mode, where this is > replaced by MatchAllDocs automatically. This would also solve the prohibited > clause case, too. > The stopword case is bad, but the opposite is as bad as returning all > documents. > At least this issue should somehow handle the only-prohibited case like Solr > and remove the hack from Solr. > Changing this in QueryParser is the more correct solution than doing this > hidden in BQ. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-3460) Move handling of query only containing MUST_NOT to QueryParser (and remove QueryUtils.makeQueryable() hack in Solr)
[
https://issues.apache.org/jira/browse/LUCENE-3460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13114260#comment-13114260
]
Robert Muir commented on LUCENE-3460:
-
{quote}
If the user enters (a the) into queryparser, the query will return no results,
as "a" and "the" are stopwords. This confuses lots of people (not only
developers, even ordinary users of our interfaces). If somebody queries for a
stopword, the correct way to handle this is to return all documents
(MatchAllDocsQuery).
{quote}
-1.
There is no terms to search for, and no evidence that all documents are
relevant to the query.
> Move handling of query only containing MUST_NOT to QueryParser (and remove
> QueryUtils.makeQueryable() hack in Solr)
> ---
>
> Key: LUCENE-3460
> URL: https://issues.apache.org/jira/browse/LUCENE-3460
> Project: Lucene - Java
> Issue Type: Sub-task
> Components: core/queryparser
>Affects Versions: 3.4, 4.0
>Reporter: Uwe Schindler
>Assignee: Uwe Schindler
> Fix For: 4.0
>
>
> With the parent issue, users entering (a -b) into the queryparser can simply
> fail with an UnsupportedOperationException, if "a" is a stopword.
> Solr already has a hack to add a MatchAllDocsQuery, if a query only contains
> prohibited clauses.
> The other issue (not affecting prohibited clauses) with stopwords is: If the
> user enters (a the) into queryparser, the query will return no results, as
> "a" and "the" are stopwords. This confuses lots of people (not only
> developers, even ordinary users of our interfaces). If somebody queries for a
> stopword, the correct way to handle this is to return *all* documents
> (MatchAllDocsQuery).
> I propose to add a flag to QueryParser to enable a
> "no-should-or-must-clauses" mode, where this is replaced by MatchAllDocs
> automatically. This would also solve the prohibited clause case, too.
> Changing this in QueryParser is the more correct solution than doing this
> hidden in BQ.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-3453) remove IndexDocValuesField
[ https://issues.apache.org/jira/browse/LUCENE-3453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13114259#comment-13114259 ] Michael McCandless commented on LUCENE-3453: That sounds great! This makes our expert APIs (direct Field/FieldType construction) much cleaner for creating a index doc values field. After this we can separately take up what sugar classes/methods we can add to make it easy for non-expert users to create doc values. Maybe static methods like NumericField.newIntValueField(17) and BinaryField.newFixedValueField(bytes) for example... I'll open a new issue about accidentally adding same DV field twice... > remove IndexDocValuesField > -- > > Key: LUCENE-3453 > URL: https://issues.apache.org/jira/browse/LUCENE-3453 > Project: Lucene - Java > Issue Type: Bug >Affects Versions: 4.0 >Reporter: Robert Muir >Assignee: Chris Male > > Its confusing how we present CSF functionality to the user, its actually not > a "field" but an "attribute" of a field like STORED or INDEXED. > Otherwise, its really hard to think about CSF because there is a mismatch > between the APIs and the index format. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (LUCENE-3455) All Analysis Consumers should use reusableTokenStream
[ https://issues.apache.org/jira/browse/LUCENE-3455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Male updated LUCENE-3455: --- Attachment: LUCENE-3455-test-consumers.patch Fixed some failing tests. Now everything is passing. Committing shortly. > All Analysis Consumers should use reusableTokenStream > - > > Key: LUCENE-3455 > URL: https://issues.apache.org/jira/browse/LUCENE-3455 > Project: Lucene - Java > Issue Type: Sub-task > Components: modules/analysis >Reporter: Chris Male >Assignee: Chris Male > Attachments: LUCENE-3455-test-consumers.patch, > LUCENE-3455-test-consumers.patch > > > With Analyzer now using TokenStreamComponents, theres no reason for Analysis > consumers to use tokenStream() (it just gives bad performance). Consequently > all consumers will be moved over to using reusableTokenStream(). The only > challenge here is that reusableTokenStream throws an IOException which many > consumers are not rigged to deal with. > Once all consumers have been moved, we can rename reusableTokenStream() back > to tokenStream(). -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-3453) remove IndexDocValuesField
[ https://issues.apache.org/jira/browse/LUCENE-3453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13114256#comment-13114256 ] Chris Male commented on LUCENE-3453: Okay so a battle plan: DocValues is basically an attribute of a Field and is a way of processing that Field's value. So to remove the need for IndexDocValues, lets: - Move ValueType to FieldType. FieldType is the appropriate place for that kind of metadata. This forces the user to define the ValueType of their DocValues when they initialize the FT. - Remove PerFieldDocValues docValues() from Field as its implicit when you consider the ValueType in combination with the actual value. - Change DocValuesConsumer to consume an IndexableField. It'll be responsible for then creating the PerFieldDocValues by looking at the ValueType. When we get round to introducing StorableField/Type, the DocValues ValueType will go over to StorableFieldType, more closely aligning DocValues and traditional value storing. > remove IndexDocValuesField > -- > > Key: LUCENE-3453 > URL: https://issues.apache.org/jira/browse/LUCENE-3453 > Project: Lucene - Java > Issue Type: Bug >Affects Versions: 4.0 >Reporter: Robert Muir >Assignee: Chris Male > > Its confusing how we present CSF functionality to the user, its actually not > a "field" but an "attribute" of a field like STORED or INDEXED. > Otherwise, its really hard to think about CSF because there is a mismatch > between the APIs and the index format. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-3460) Move handling of query only containing MUST_NOT to QueryParser (and remove QueryUtils.makeQueryable() hack in Solr)
[ https://issues.apache.org/jira/browse/LUCENE-3460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13114255#comment-13114255 ] Michael McCandless commented on LUCENE-3460: +1 > Move handling of query only containing MUST_NOT to QueryParser (and remove > QueryUtils.makeQueryable() hack in Solr) > --- > > Key: LUCENE-3460 > URL: https://issues.apache.org/jira/browse/LUCENE-3460 > Project: Lucene - Java > Issue Type: Sub-task > Components: core/queryparser >Affects Versions: 3.4, 4.0 >Reporter: Uwe Schindler >Assignee: Uwe Schindler > Fix For: 4.0 > > > With the parent issue, users entering (a -b) into the queryparser can simply > fail with an UnsupportedOperationException, if "a" is a stopword. > Solr already has a hack to add a MatchAllDocsQuery, if a query only contains > prohibited clauses. > The other issue (not affecting prohibited clauses) with stopwords is: If the > user enters (a the) into queryparser, the query will return no results, as > "a" and "the" are stopwords. This confuses lots of people (not only > developers, even ordinary users of our interfaces). If somebody queries for a > stopword, the correct way to handle this is to return *all* documents > (MatchAllDocsQuery). > I propose to add a flag to QueryParser to enable a > "no-should-or-must-clauses" mode, where this is replaced by MatchAllDocs > automatically. This would also solve the prohibited clause case, too. > Changing this in QueryParser is the more correct solution than doing this > hidden in BQ. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Created] (LUCENE-3460) Move handling of query only containing MUST_NOT to QueryParser (and remove QueryUtils.makeQueryable() hack in Solr)
Move handling of query only containing MUST_NOT to QueryParser (and remove QueryUtils.makeQueryable() hack in Solr) --- Key: LUCENE-3460 URL: https://issues.apache.org/jira/browse/LUCENE-3460 Project: Lucene - Java Issue Type: Sub-task Components: core/queryparser Affects Versions: 3.4, 4.0 Reporter: Uwe Schindler Assignee: Uwe Schindler With the parent issue, users entering (a -b) into the queryparser can simply fail with an UnsupportedOperationException, if "a" is a stopword. Solr already has a hack to add a MatchAllDocsQuery, if a query only contains prohibited clauses. The other issue (not affecting prohibited clauses) with stopwords is: If the user enters (a the) into queryparser, the query will return no results, as "a" and "the" are stopwords. This confuses lots of people (not only developers, even ordinary users of our interfaces). If somebody queries for a stopword, the correct way to handle this is to return *all* documents (MatchAllDocsQuery). I propose to add a flag to QueryParser to enable a "no-should-or-must-clauses" mode, where this is replaced by MatchAllDocs automatically. This would also solve the prohibited clause case, too. Changing this in QueryParser is the more correct solution than doing this hidden in BQ. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-3454) rename optimize to a less cool-sounding name
[ https://issues.apache.org/jira/browse/LUCENE-3454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13114248#comment-13114248 ] Michael McCandless commented on LUCENE-3454: How about maybeMerge() and maybeMerge(int maxSegCount) (since we have to absorb optimize(int maxSegCount) too)? This way what used to be optimize is now maybeMerge(1). I like that this forces the user to make an explicit decision about how many segments they require the index to merge down to, so they realize by picking 1 they are asking for tons of merging. > rename optimize to a less cool-sounding name > > > Key: LUCENE-3454 > URL: https://issues.apache.org/jira/browse/LUCENE-3454 > Project: Lucene - Java > Issue Type: Improvement >Affects Versions: 4.0 >Reporter: Robert Muir > > I think users see the name optimize and feel they must do this, because who > wants a suboptimal system? but this probably just results in wasted time and > resources. > maybe rename to collapseSegments or something? -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (LUCENE-3451) Remove special handling of pure negative Filters in BooleanFilter, disallow pure negative queries in BooleanQuery
[ https://issues.apache.org/jira/browse/LUCENE-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uwe Schindler updated LUCENE-3451: -- Attachment: LUCENE-3451.patch Updated patch after committing BF cleanup (LUCENE-3458). > Remove special handling of pure negative Filters in BooleanFilter, disallow > pure negative queries in BooleanQuery > - > > Key: LUCENE-3451 > URL: https://issues.apache.org/jira/browse/LUCENE-3451 > Project: Lucene - Java > Issue Type: Bug >Reporter: Uwe Schindler >Assignee: Uwe Schindler > Fix For: 4.0 > > Attachments: LUCENE-3451.patch, LUCENE-3451.patch, LUCENE-3451.patch, > LUCENE-3451.patch > > > We should at least in Lucene 4.0 remove the hack in BooleanFilter that allows > pure negative Filter clauses. This is not supported by BooleanQuery and > confuses users (I think that's the problem in LUCENE-3450). > The hack is buggy, as it does not respect deleted documents and returns them > in its DocIdSet. > Also we should think about disallowing pure-negative Queries at all and throw > UOE. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Resolved] (LUCENE-3458) Change BooleanFilter to have only a single clauses ArrayList (so toString() works fine, clauses() method could be added) so it behaves more lik BooleanQuery
[ https://issues.apache.org/jira/browse/LUCENE-3458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uwe Schindler resolved LUCENE-3458. --- Resolution: Fixed Committed 3.x branch revision: 1175386 > Change BooleanFilter to have only a single clauses ArrayList (so toString() > works fine, clauses() method could be added) so it behaves more lik > BooleanQuery > > > Key: LUCENE-3458 > URL: https://issues.apache.org/jira/browse/LUCENE-3458 > Project: Lucene - Java > Issue Type: Task > Components: modules/other >Affects Versions: 3.4, 4.0 >Reporter: Uwe Schindler >Assignee: Uwe Schindler > Fix For: 3.5, 4.0 > > Attachments: LUCENE-3458.patch > > > This is unrelated to the other BF changes, but should be done -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-3234) Provide limit on phrase analysis in FastVectorHighlighter
[ https://issues.apache.org/jira/browse/LUCENE-3234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13114233#comment-13114233 ] Koji Sekiguchi commented on LUCENE-3234: It could be the suggested 5000. I don't have a persistence on it. The current default is just for back-compat. > Provide limit on phrase analysis in FastVectorHighlighter > - > > Key: LUCENE-3234 > URL: https://issues.apache.org/jira/browse/LUCENE-3234 > Project: Lucene - Java > Issue Type: Improvement >Affects Versions: 2.9.4, 3.0.3, 3.1, 3.2, 3.3 >Reporter: Mike Sokolov >Assignee: Koji Sekiguchi > Fix For: 3.4, 4.0 > > Attachments: LUCENE-3234.patch, LUCENE-3234.patch, LUCENE-3234.patch, > LUCENE-3234.patch, LUCENE-3234.patch > > > With larger documents, FVH can spend a lot of time trying to find the > best-scoring snippet as it examines every possible phrase formed from > matching terms in the document. If one is willing to accept > less-than-perfect scoring by limiting the number of phrases that are > examined, substantial speedups are possible. This is analogous to the > Highlighter limit on the number of characters to analyze. > The patch includes an artifical test case that shows > 1000x speedup. In a > more normal test environment, with English documents and random queries, I am > seeing speedups of around 3-10x when setting phraseLimit=1, which has the > effect of selecting the first possible snippet in the document. Most of our > sites operate in this way (just show the first snippet), so this would be a > big win for us. > With phraseLimit = -1, you get the existing FVH behavior. At larger values of > phraseLimit, you may not get substantial speedup in the normal case, but you > do get the benefit of protection against blow-up in pathological cases. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-3458) Change BooleanFilter to have only a single clauses ArrayList (so toString() works fine, clauses() method could be added) so it behaves more lik BooleanQuery
[ https://issues.apache.org/jira/browse/LUCENE-3458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13114231#comment-13114231 ] Uwe Schindler commented on LUCENE-3458: --- Committed trunk revision: 1175385 > Change BooleanFilter to have only a single clauses ArrayList (so toString() > works fine, clauses() method could be added) so it behaves more lik > BooleanQuery > > > Key: LUCENE-3458 > URL: https://issues.apache.org/jira/browse/LUCENE-3458 > Project: Lucene - Java > Issue Type: Task > Components: modules/other >Affects Versions: 3.4, 4.0 >Reporter: Uwe Schindler >Assignee: Uwe Schindler > Fix For: 3.5, 4.0 > > Attachments: LUCENE-3458.patch > > > This is unrelated to the other BF changes, but should be done -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (LUCENE-3458) Change BooleanFilter to have only a single clauses ArrayList (so toString() works fine, clauses() method could be added) so it behaves more lik BooleanQuery
[ https://issues.apache.org/jira/browse/LUCENE-3458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uwe Schindler updated LUCENE-3458: -- Attachment: LUCENE-3458.patch Patch, will commit soon and update other issues. BF API now behaves like BQ (Iterable, toString() returns clauses in orginal order,...) > Change BooleanFilter to have only a single clauses ArrayList (so toString() > works fine, clauses() method could be added) so it behaves more lik > BooleanQuery > > > Key: LUCENE-3458 > URL: https://issues.apache.org/jira/browse/LUCENE-3458 > Project: Lucene - Java > Issue Type: Task > Components: modules/other >Affects Versions: 3.4, 4.0 >Reporter: Uwe Schindler >Assignee: Uwe Schindler > Fix For: 3.5, 4.0 > > Attachments: LUCENE-3458.patch > > > This is unrelated to the other BF changes, but should be done -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-3457) Upgrade to commons-compress 1.2
[ https://issues.apache.org/jira/browse/LUCENE-3457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13114225#comment-13114225 ] Robert Muir commented on LUCENE-3457: - I know whats happening, a test failed (happens often with solr tests), but some assertions in the base solr test classes then cause every test following to fail. this happened to me too, i'll commit a fix. > Upgrade to commons-compress 1.2 > --- > > Key: LUCENE-3457 > URL: https://issues.apache.org/jira/browse/LUCENE-3457 > Project: Lucene - Java > Issue Type: Bug > Components: modules/benchmark >Reporter: Doron Cohen >Assignee: Doron Cohen >Priority: Minor > Fix For: 3.5, 4.0 > > Attachments: LUCENE-3457.patch, test.out.gz > > > Commons Compress bug COMPRESS-127 was fixed in 1.2, so the workaround in > benchmark's StreamUtils is no longer required. Compress is also used in solr. > Replace with new jar in both benchmark and solr and get rid of that > workaround. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (LUCENE-3457) Upgrade to commons-compress 1.2
[ https://issues.apache.org/jira/browse/LUCENE-3457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Doron Cohen updated LUCENE-3457: Attachment: test.out.gz Still it fails - this time running 'clean test' from trunk, all lucene tests pass, some of solr tests failed: - org.apache.solr.handler.TestReplicationHandler [junit] Tests run: 1, Failures: 1, Errors: 0, Time elapsed: 43.703 sec - org.apache.solr.handler.component.DebugComponentTest [junit] Tests run: 2, Failures: 1, Errors: 0, Time elapsed: 1 sec - org.apache.solr.handler.component.TermVectorComponentTest [junit] Tests run: 4, Failures: 1, Errors: 0, Time elapsed: 1.375 sec - org.apache.solr.request.JSONWriterTest [junit] Tests run: 2, Failures: 1, Errors: 0, Time elapsed: 1.078 sec - org.apache.solr.schema.BadIndexSchemaTest [junit] Tests run: 5, Failures: 1, Errors: 0, Time elapsed: 1.266 sec - org.apache.solr.schema.RequiredFieldsTest [junit] Tests run: 3, Failures: 1, Errors: 0, Time elapsed: 1.422 sec - org.apache.solr.search.QueryParsingTest [junit] Tests run: 2, Failures: 1, Errors: 0, Time elapsed: 0.641 sec - org.apache.solr.search.SpatialFilterTest [junit] Tests run: 3, Failures: 1, Errors: 0, Time elapsed: 1.438 sec - org.apache.solr.search.TestQueryTypes [junit] Tests run: 1, Failures: 1, Errors: 0, Time elapsed: 0.953 sec - org.apache.solr.servlet.CacheHeaderTest [junit] Tests run: 5, Failures: 1, Errors: 0, Time elapsed: 0.984 sec - org.apache.solr.spelling.SpellCheckCollatorTest [junit] Tests run: 4, Failures: 1, Errors: 0, Time elapsed: 1.281 sec - org.apache.solr.update.DocumentBuilderTest [junit] Tests run: 4, Failures: 1, Errors: 0, Time elapsed: 0.734 sec - org.apache.solr.util.SolrPluginUtilsTest [junit] Tests run: 7, Failures: 1, Errors: 0, Time elapsed: 0.766 sec Running alone, TestReplicationHandler for example passes. Same for DebugComponentTest. I am not sure what is happenning here. Attaching the test output in case someone wants take a look. > Upgrade to commons-compress 1.2 > --- > > Key: LUCENE-3457 > URL: https://issues.apache.org/jira/browse/LUCENE-3457 > Project: Lucene - Java > Issue Type: Bug > Components: modules/benchmark >Reporter: Doron Cohen >Assignee: Doron Cohen >Priority: Minor > Fix For: 3.5, 4.0 > > Attachments: LUCENE-3457.patch, test.out.gz > > > Commons Compress bug COMPRESS-127 was fixed in 1.2, so the workaround in > benchmark's StreamUtils is no longer required. Compress is also used in solr. > Replace with new jar in both benchmark and solr and get rid of that > workaround. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Issue Comment Edited] (LUCENE-3457) Upgrade to commons-compress 1.2
[ https://issues.apache.org/jira/browse/LUCENE-3457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13114219#comment-13114219 ] Doron Cohen edited comment on LUCENE-3457 at 9/25/11 11:44 AM: --- Thanks Chris, almost sure I did a clean, will try again. was (Author: doronc): Thanks Chriss, almost sure I did a clean, will try again. > Upgrade to commons-compress 1.2 > --- > > Key: LUCENE-3457 > URL: https://issues.apache.org/jira/browse/LUCENE-3457 > Project: Lucene - Java > Issue Type: Bug > Components: modules/benchmark >Reporter: Doron Cohen >Assignee: Doron Cohen >Priority: Minor > Fix For: 3.5, 4.0 > > Attachments: LUCENE-3457.patch > > > Commons Compress bug COMPRESS-127 was fixed in 1.2, so the workaround in > benchmark's StreamUtils is no longer required. Compress is also used in solr. > Replace with new jar in both benchmark and solr and get rid of that > workaround. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (LUCENE-3456) Analysis Consumers should end and close any TokenStreams
[ https://issues.apache.org/jira/browse/LUCENE-3456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Muir updated LUCENE-3456: Attachment: LUCENE-3456.patch here is a patch cutting over solr/core... this looks to have uncovered some bugs, i didnt fix any of them. additionally i think we should do the same thing for solrj, contrib/*, etc. > Analysis Consumers should end and close any TokenStreams > > > Key: LUCENE-3456 > URL: https://issues.apache.org/jira/browse/LUCENE-3456 > Project: Lucene - Java > Issue Type: Sub-task >Reporter: Chris Male > Attachments: LUCENE-3456.patch > > > While converting consumers over to using reusableTokenStream, I notice many > don't call end() or close() on TokenStreams. > Even if they are test TSs only created once, we should follow the defined > usage pattern. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-3456) Analysis Consumers should end and close any TokenStreams
[ https://issues.apache.org/jira/browse/LUCENE-3456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13114221#comment-13114221 ] Chris Male commented on LUCENE-3456: Sorting this out for the current codebase isn't a huge challenge. I work from incrementToken() / reuseable/TokenStream() calls, and see whats being called in their vicinity. But going forward, I agree using Mock* in Solr more makes sense. > Analysis Consumers should end and close any TokenStreams > > > Key: LUCENE-3456 > URL: https://issues.apache.org/jira/browse/LUCENE-3456 > Project: Lucene - Java > Issue Type: Sub-task >Reporter: Chris Male > > While converting consumers over to using reusableTokenStream, I notice many > don't call end() or close() on TokenStreams. > Even if they are test TSs only created once, we should follow the defined > usage pattern. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-3457) Upgrade to commons-compress 1.2
[ https://issues.apache.org/jira/browse/LUCENE-3457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13114219#comment-13114219 ] Doron Cohen commented on LUCENE-3457: - Thanks Chriss, almost sure I did a clean, will try again. > Upgrade to commons-compress 1.2 > --- > > Key: LUCENE-3457 > URL: https://issues.apache.org/jira/browse/LUCENE-3457 > Project: Lucene - Java > Issue Type: Bug > Components: modules/benchmark >Reporter: Doron Cohen >Assignee: Doron Cohen >Priority: Minor > Fix For: 3.5, 4.0 > > Attachments: LUCENE-3457.patch > > > Commons Compress bug COMPRESS-127 was fixed in 1.2, so the workaround in > benchmark's StreamUtils is no longer required. Compress is also used in solr. > Replace with new jar in both benchmark and solr and get rid of that > workaround. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (LUCENE-3459) Change ChainedFilter to use FixedBitSet
[ https://issues.apache.org/jira/browse/LUCENE-3459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uwe Schindler updated LUCENE-3459: -- Component/s: modules/other Affects Version/s: 4.0 3.4 Fix Version/s: 4.0 Assignee: Uwe Schindler > Change ChainedFilter to use FixedBitSet > --- > > Key: LUCENE-3459 > URL: https://issues.apache.org/jira/browse/LUCENE-3459 > Project: Lucene - Java > Issue Type: Task > Components: modules/other >Affects Versions: 3.4, 4.0 >Reporter: Uwe Schindler >Assignee: Uwe Schindler > Fix For: 4.0 > > > ChainedFilter also uses OpenBitSet(DISI) at the moment. It should also be > changed to use FixedBitSet. There are two issues: > - It exposes sometimes OpenBitSetDISI to it's public API - we should remove > those methods like in BooleanFilter and break backwards > - It allows a XOR operation. This is not yet supported by FixedBitSet, but > it's easy to add (like for BooleanFilter). On the other hand, this XOR > operation is bogus, as it may mark documents in the BitSet that are deleted, > breaking new features like applying Filters down-low (LUCENE-1536). We should > remove the XOR operation maybe or force it to use IR.validDocs() (trunk) or > IR.isDeleted() -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Created] (LUCENE-3459) Change ChainedFilter to use FixedBitSet
Change ChainedFilter to use FixedBitSet --- Key: LUCENE-3459 URL: https://issues.apache.org/jira/browse/LUCENE-3459 Project: Lucene - Java Issue Type: Task Reporter: Uwe Schindler ChainedFilter also uses OpenBitSet(DISI) at the moment. It should also be changed to use FixedBitSet. There are two issues: - It exposes sometimes OpenBitSetDISI to it's public API - we should remove those methods like in BooleanFilter and break backwards - It allows a XOR operation. This is not yet supported by FixedBitSet, but it's easy to add (like for BooleanFilter). On the other hand, this XOR operation is bogus, as it may mark documents in the BitSet that are deleted, breaking new features like applying Filters down-low (LUCENE-1536). We should remove the XOR operation maybe or force it to use IR.validDocs() (trunk) or IR.isDeleted() -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
