Re: test buffering causes OOM errors

2012-07-21 Thread Dawid Weiss
> +1, I also miss simple streamed output option when running a test. > > And it's fine if it only works if you run one test, one JVM, etc. There is an issue for 1-jvm streaming on github. Like I said -- it is possible to decrease memory consumption (stdout buffering) although there will always be

Re: test buffering causes OOM errors

2012-07-21 Thread Dawid Weiss
> +1, I also miss simple streamed output option when running a test. > > And it's fine if it only works if you run one test, one JVM, etc. There is an issue for 1-jvm streaming on github. Like I said -- it is possible to decrease memory consumption (stdout buffering) although there will always be

Re: literalsOverride param in Solr Cell doesn't exist???

2012-07-21 Thread Eric Pugh
Thank you! I didn't real so much had gone on between the two cuts. That explains things! It's totally an edge case, and a good incentive to "ship it"! Eric On Jul 21, 2012, at 7:50 PM, Jack Krupansky wrote: > Interesting logistical issue - wiki tagging as to what is in "4.0" vs. "4.0 > Alph

Re: literalsOverride param in Solr Cell doesn't exist???

2012-07-21 Thread Jack Krupansky
Interesting logistical issue - wiki tagging as to what is in "4.0" vs. "4.0 Alpha". So, this feature is in "4.0" but not "4.0 Alpha". The 4.0 Alpha download artifacts are dated 6/29 and this change was dated 6/27. The "staged" artifacts are dated 6/25, so this change just missed the Alpha cuto

[jira] [Commented] (SOLR-3653) Custom bigramming filter for to handle Smart Chinese edge cases

2012-07-21 Thread Lance Norskog (JIRA)
[ https://issues.apache.org/jira/browse/SOLR-3653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13419963#comment-13419963 ] Lance Norskog commented on SOLR-3653: - Yeah, good point. > Custom bigr

[jira] [Comment Edited] (LUCENE-3151) Make all of Analysis completely independent from Lucene Core

2012-07-21 Thread Lance Norskog (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-3151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13419957#comment-13419957 ] Lance Norskog edited comment on LUCENE-3151 at 7/21/12 10:02 PM: --

[jira] [Commented] (LUCENE-3151) Make all of Analysis completely independent from Lucene Core

2012-07-21 Thread Lance Norskog (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-3151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13419957#comment-13419957 ] Lance Norskog commented on LUCENE-3151: --- Is it intended to support jars from differ

[jira] [Commented] (LUCENE-3151) Make all of Analysis completely independent from Lucene Core

2012-07-21 Thread Grant Ingersoll (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-3151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13419955#comment-13419955 ] Grant Ingersoll commented on LUCENE-3151: - For: # IndexReader -- It's mostly jus

[jira] [Commented] (LUCENE-4239) Provide access to PackedInts' low-level blocks <-> values conversion methods

2012-07-21 Thread Adrien Grand (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-4239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13419917#comment-13419917 ] Adrien Grand commented on LUCENE-4239: -- I just committed. Let's see how we can impro

literalsOverride param in Solr Cell doesn't exist???

2012-07-21 Thread Eric Pugh
I was playing around with Solr Cell, and I think I found a bug in the docs: http://wiki.apache.org/solr/ExtractingRequestHandler. I was trying to use "literalsOverride=true" to override what Tika was returning with my own data, and no joy. Digging around in the 4.0 Alpha tagged code, I can't f

[jira] [Assigned] (SOLR-3656) use same data dir in core reload

2012-07-21 Thread Mark Miller (JIRA)
[ https://issues.apache.org/jira/browse/SOLR-3656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Miller reassigned SOLR-3656: - Assignee: Mark Miller > use same data dir in core reload > >

[jira] [Commented] (LUCENE-4239) Provide access to PackedInts' low-level blocks <-> values conversion methods

2012-07-21 Thread Han Jiang (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-4239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13419902#comment-13419902 ] Han Jiang commented on LUCENE-4239: --- Hi Adrien, you're right! There is something wrong

[jira] [Comment Edited] (SOLR-3502) [Copy Field][Importing] Copy Field duplicate a field and a multivalued field can be created without having field multivalued in schema

2012-07-21 Thread Jun Ohtani (JIRA)
[ https://issues.apache.org/jira/browse/SOLR-3502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13419901#comment-13419901 ] Jun Ohtani edited comment on SOLR-3502 at 7/21/12 6:30 PM: --- I try

[jira] [Commented] (SOLR-3502) [Copy Field][Importing] Copy Field duplicate a field and a multivalued field can be created without having field multivalued in schema

2012-07-21 Thread Jun Ohtani (JIRA)
[ https://issues.apache.org/jira/browse/SOLR-3502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13419901#comment-13419901 ] Jun Ohtani commented on SOLR-3502: -- I try "partial update" for 4.0-ALPHA using exampledocs

[jira] [Commented] (SOLR-3657) error message only refers to "source" field when problem parsing value for "dest" field of copyField

2012-07-21 Thread Jun Ohtani (JIRA)
[ https://issues.apache.org/jira/browse/SOLR-3657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13419899#comment-13419899 ] Jun Ohtani commented on SOLR-3657: -- maybe related this issue. https://issues.apache.org/ji

[jira] [Commented] (LUCENE-4239) Provide access to PackedInts' low-level blocks <-> values conversion methods

2012-07-21 Thread Adrien Grand (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-4239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13419893#comment-13419893 ] Adrien Grand commented on LUCENE-4239: -- Han, perhaps the patch didn't apply well on

[jira] [Commented] (SOLR-3658) SolrCmdDistributor can briefly create spikes of threads in the thousands.

2012-07-21 Thread Mark Miller (JIRA)
[ https://issues.apache.org/jira/browse/SOLR-3658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13419883#comment-13419883 ] Mark Miller commented on SOLR-3658: --- (although i don't know if they were just using 32-bi

[jira] [Commented] (SOLR-3658) SolrCmdDistributor can briefly create spikes of threads in the thousands.

2012-07-21 Thread Mark Miller (JIRA)
[ https://issues.apache.org/jira/browse/SOLR-3658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13419881#comment-13419881 ] Mark Miller commented on SOLR-3658: --- Yeah, I'm not sure of the nitty gritty - but a coupl

RE: VOTE: Lucene/Solr 3.6.1

2012-07-21 Thread Uwe Schindler
Hi, The vote has passed. I tagged the release, branch 3.6.1 is open again for committing Roberts fixes for Disjunctions and other stuff. I will start publishing artifacts to apache and maven (Steven, I need your help), soon. Website will be updated tomorrow. Uwe - Uwe Schindler H.-H.-Meier-

[JENKINS] Lucene-Solr-tests-only-4.x - Build # 303 - Failure

2012-07-21 Thread Apache Jenkins Server
Build: https://builds.apache.org/job/Lucene-Solr-tests-only-4.x/303/ 1 tests failed. REGRESSION: org.apache.solr.update.SoftAutoCommitTest.testSoftAndHardCommitMaxTimeDelete Error Message: searcher529 wasn't soon enough after soft529: 1342887657845 !< 1342887657574 + 100 (fudge) Stack Trace:

[JENKINS] Lucene-Solr-tests-only-trunk - Build # 14909 - Failure

2012-07-21 Thread Apache Jenkins Server
Build: https://builds.apache.org/job/Lucene-Solr-tests-only-trunk/14909/ 1 tests failed. REGRESSION: org.apache.solr.update.SoftAutoCommitTest.testSoftAndHardCommitMaxTimeDelete Error Message: soft529 wasn't fast enough Stack Trace: java.lang.AssertionError: soft529 wasn't fast enough

[jira] [Commented] (LUCENE-4242) UnInverted cache uses term freq to filter out terms (but deleted docs are included in the freq count)

2012-07-21 Thread Michael McCandless (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-4242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13419860#comment-13419860 ] Michael McCandless commented on LUCENE-4242: bq. liveDocsRatio will be the sa

Re: [JENKINS] Lucene-Solr-trunk-Linux-Java7-64 - Build # 655 - Still Failing!

2012-07-21 Thread Michael McCandless
On Sat, Jul 21, 2012 at 8:15 AM, Robert Muir wrote: > I committed fixes for this, but things we should do: > > 1. Add checks to TestPostingsFormat that payloads work correctly when > scanning over deleted documents, and change the values in the returned > BytesRef itself from getPayload (bytes/off

[jira] [Commented] (LUCENE-4242) UnInverted cache uses term freq to filter out terms (but deleted docs are included in the freq count)

2012-07-21 Thread Yonik Seeley (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-4242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13419855#comment-13419855 ] Yonik Seeley commented on LUCENE-4242: -- This patch doesn't seem to make sense. liveD

[jira] [Updated] (LUCENE-4242) UnInverted cache uses term freq to filter out terms (but deleted docs are included in the freq count)

2012-07-21 Thread Michael McCandless (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-4242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael McCandless updated LUCENE-4242: --- Attachment: LUCENE-4242.patch I cleaned up the patch a bit ... only issue now is it

[jira] [Commented] (LUCENE-4242) UnInverted cache uses term freq to filter out terms (but deleted docs are included in the freq count)

2012-07-21 Thread Michael McCandless (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-4242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13419838#comment-13419838 ] Michael McCandless commented on LUCENE-4242: maxDoc is indeed a confusing nam

Re: test buffering causes OOM errors

2012-07-21 Thread Michael McCandless
On Sat, Jul 21, 2012 at 5:49 AM, Yonik Seeley wrote: > Logs are most useful in big stress tests where things are not easily > reproducible or debuggable with a debugger - and these also tend to > produce a lot of logging. +1, I also miss simple streamed output option when running a test. And it

[jira] [Commented] (LUCENE-3312) Break out StorableField from IndexableField

2012-07-21 Thread Robert Muir (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-3312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13419832#comment-13419832 ] Robert Muir commented on LUCENE-3312: - Yes. There aren't many tests that test that Lu

[jira] [Commented] (LUCENE-3312) Break out StorableField from IndexableField

2012-07-21 Thread Nikola Tankovic (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-3312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13419831#comment-13419831 ] Nikola Tankovic commented on LUCENE-3312: - I keep getting this error: {code} [jun

Re: [JENKINS] Lucene-Solr-trunk-Linux-Java7-64 - Build # 655 - Still Failing!

2012-07-21 Thread Robert Muir
I committed fixes for this, but things we should do: 1. Add checks to TestPostingsFormat that payloads work correctly when scanning over deleted documents, and change the values in the returned BytesRef itself from getPayload (bytes/offset/length) to ensure callers actually set all 3 of these in g

[jira] [Updated] (SOLR-3662) core reload can go back in time

2012-07-21 Thread Yonik Seeley (JIRA)
[ https://issues.apache.org/jira/browse/SOLR-3662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yonik Seeley updated SOLR-3662: --- Attachment: SOLR-3662.patch This patch improves a number of areas related to reload: - the VersionInfo

[JENKINS] Lucene-Solr-tests-only-trunk-java7 - Build # 2939 - Failure

2012-07-21 Thread Apache Jenkins Server
Build: https://builds.apache.org/job/Lucene-Solr-tests-only-trunk-java7/2939/ 1 tests failed. REGRESSION: org.apache.lucene.index.TestDocumentsWriterStallControl.testSimpleStall Error Message: Left-over thread detected (and terminated): Thread[Thread-1377,5,] (stack trace is a snapshot locati

Re: [JENKINS] Lucene-Solr-trunk-Linux-Java7-64 - Build # 655 - Still Failing!

2012-07-21 Thread Robert Muir
OK: actually my fix is correct, there are 2 bugs 1. this bug fixes scanning over deleted documents for the payloads case: it fixes TestDuelingCodecs 2. the second bug (TestPayloads) is this line: // Just to ensure all codecs can // handle a caller that

Re: [JENKINS] Lucene-Solr-trunk-Linux-Java7-64 - Build # 655 - Still Failing!

2012-07-21 Thread Robert Muir
I give up on this thing for now. but the nextDoc() in LowFreqD&PEnum looks bogus if payloads are present. I tried this (doesnt work), but i still feel like this is on the right path: Index: src/java/org/apache/lucene/codecs/memory/DirectPostingsFormat.java

Re: [JENKINS] Lucene-Solr-trunk-Linux-Java7-64 - Build # 655 - Still Failing!

2012-07-21 Thread Robert Muir
Maybe after this bug gets resolved we can merge it over? I just svn copied it to trunk as a test, and it compiles and seems to work. Seems like a good test to have sooner than later... On Sat, Jul 21, 2012 at 7:17 AM, Michael McCandless wrote: > I want to do that! It lives only on the block bra

Re: [JENKINS] Lucene-Solr-trunk-Linux-Java7-64 - Build # 655 - Still Failing!

2012-07-21 Thread Michael McCandless
I want to do that! It lives only on the block branch for now ... Mike McCandless http://blog.mikemccandless.com On Sat, Jul 21, 2012 at 7:13 AM, Robert Muir wrote: > We should see if the new TestPostingsFormat fails with that set to > Integer.MAX_VALUE too! > > On Sat, Jul 21, 2012 at 7:12 AM,

Re: [JENKINS] Lucene-Solr-trunk-Linux-Java7-64 - Build # 655 - Still Failing!

2012-07-21 Thread Robert Muir
We should see if the new TestPostingsFormat fails with that set to Integer.MAX_VALUE too! On Sat, Jul 21, 2012 at 7:12 AM, Michael McCandless wrote: > OK that helps, thanks! > > Mike McCandless > > http://blog.mikemccandless.com > > On Sat, Jul 21, 2012 at 7:08 AM, Robert Muir wrote: >> Test pas

Re: [JENKINS] Lucene-Solr-trunk-Linux-Java7-64 - Build # 655 - Still Failing!

2012-07-21 Thread Michael McCandless
OK that helps, thanks! Mike McCandless http://blog.mikemccandless.com On Sat, Jul 21, 2012 at 7:08 AM, Robert Muir wrote: > Test passes if i do this: > > this.lowFreqCutoff = 1; // nocommit lowFreqCutoff; > > If i wire it to Integer.MAX_VALUE, easier tests fail like > TestPayloads.testPaylo

Re: [JENKINS] Lucene-Solr-trunk-Linux-Java7-64 - Build # 655 - Still Failing!

2012-07-21 Thread Robert Muir
Test passes if i do this: this.lowFreqCutoff = 1; // nocommit lowFreqCutoff; If i wire it to Integer.MAX_VALUE, easier tests fail like TestPayloads.testPayloadsEncoding On Sat, Jul 21, 2012 at 5:43 AM, Policeman Jenkins Server wrote: > Build: > http://jenkins.sd-datasolutions.de/job/Lucene

Re: [JENKINS] Lucene-Solr-trunk-Linux-Java7-64 - Build # 655 - Still Failing!

2012-07-21 Thread Michael McCandless
I'll dig. Mike McCandless http://blog.mikemccandless.com On Sat, Jul 21, 2012 at 6:42 AM, Robert Muir wrote: > This looks like a gremlin in DirectPostingsFormat (LUCENE-4227), its > retrieving the wrong bytes for the payload > > On Sat, Jul 21, 2012 at 5:43 AM, Policeman Jenkins Server > wrot

Re: test buffering causes OOM errors

2012-07-21 Thread Dawid Weiss
I am on mobile but I will try to justify the design decisions I made later on. As for now, quickly, tests.iters has been fixed and should not oom unless you run with many thousands of repetitions (if so it is better to loop manually). As for the oom on large sysouts - incease ant memory or redire

Re: [JENKINS] Lucene-Solr-trunk-Linux-Java7-64 - Build # 655 - Still Failing!

2012-07-21 Thread Robert Muir
This looks like a gremlin in DirectPostingsFormat (LUCENE-4227), its retrieving the wrong bytes for the payload On Sat, Jul 21, 2012 at 5:43 AM, Policeman Jenkins Server wrote: > Build: > http://jenkins.sd-datasolutions.de/job/Lucene-Solr-trunk-Linux-Java7-64/655/ > > 1 tests failed. > REGRESSIO

[jira] [Commented] (SOLR-3653) Custom bigramming filter for to handle Smart Chinese edge cases

2012-07-21 Thread Robert Muir (JIRA)
[ https://issues.apache.org/jira/browse/SOLR-3653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13419791#comment-13419791 ] Robert Muir commented on SOLR-3653: --- Why not just use a synonymfilter for your special ca

[jira] [Commented] (LUCENE-3151) Make all of Analysis completely independent from Lucene Core

2012-07-21 Thread Grant Ingersoll (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-3151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13419790#comment-13419790 ] Grant Ingersoll commented on LUCENE-3151: - bq. For example, I just searched for o

Re: test buffering causes OOM errors

2012-07-21 Thread Yonik Seeley
On Sat, Jul 21, 2012 at 5:36 AM, Dawid Weiss wrote: > Full stack trace would me more helpful Full stack trace below. > but I think the problem is that solr > tests emit megabytes of data to the console and this has to be buffered and > passed to report listeners. I've not looked into the test f

[JENKINS] Lucene-Solr-trunk-Windows-Java6-64 - Build # 850 - Failure!

2012-07-21 Thread Policeman Jenkins Server
Build: http://jenkins.sd-datasolutions.de/job/Lucene-Solr-trunk-Windows-Java6-64/850/ 1 tests failed. REGRESSION: org.apache.solr.spelling.suggest.SuggesterFSTTest.testSuggestions Error Message: Exception during query Stack Trace: java.lang.RuntimeException: Exception during query at

[JENKINS] Lucene-Solr-trunk-Linux-Java7-64 - Build # 655 - Still Failing!

2012-07-21 Thread Policeman Jenkins Server
Build: http://jenkins.sd-datasolutions.de/job/Lucene-Solr-trunk-Linux-Java7-64/655/ 1 tests failed. REGRESSION: org.apache.lucene.index.TestDuelingCodecs.testEquals Error Message: left: SimpleText / right: Lucene40: {docid=MockVariableIntBlock(baseBlockSize=116), body=PostingsFormat(name=Dire

Re: test buffering causes OOM errors

2012-07-21 Thread Dawid Weiss
Full stack trace would me more helpful but I think the problem is that solr tests emit megabytes of data to the console and this has to be buffered and passed to report listeners. It is possible to spill to disk of course but not a trivial refactoring. An easy workaround for now would be to redir

test buffering causes OOM errors

2012-07-21 Thread Yonik Seeley
Is there a way we can write to a file instead of buffering test output in memory? This has been causing exceptions for me lately. [junit4:junit4] Event bus dispatch error: java.lang.OutOfMemoryError: Java heap space [junit4:junit4] java.lang.OutOfMemoryError: Java heap space [junit4:junit4]

[jira] [Created] (SOLR-3662) core reload can go back in time

2012-07-21 Thread Yonik Seeley (JIRA)
Yonik Seeley created SOLR-3662: -- Summary: core reload can go back in time Key: SOLR-3662 URL: https://issues.apache.org/jira/browse/SOLR-3662 Project: Solr Issue Type: Bug Reporter:

[jira] [Assigned] (SOLR-3662) core reload can go back in time

2012-07-21 Thread Yonik Seeley (JIRA)
[ https://issues.apache.org/jira/browse/SOLR-3662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yonik Seeley reassigned SOLR-3662: -- Assignee: Yonik Seeley > core reload can go back in time > --- >

[jira] [Comment Edited] (SOLR-3658) SolrCmdDistributor can briefly create spikes of threads in the thousands.

2012-07-21 Thread Yonik Seeley (JIRA)
[ https://issues.apache.org/jira/browse/SOLR-3658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13419755#comment-13419755 ] Yonik Seeley edited comment on SOLR-3658 at 7/21/12 7:41 AM: - O

[jira] [Commented] (SOLR-3658) SolrCmdDistributor can briefly create spikes of threads in the thousands.

2012-07-21 Thread Yonik Seeley (JIRA)
[ https://issues.apache.org/jira/browse/SOLR-3658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13419755#comment-13419755 ] Yonik Seeley commented on SOLR-3658: Other thing to keep in mind memory-wise is that ea

[jira] [Created] (SOLR-3661) RequestHandlerBase getStatistics method has atom problem

2012-07-21 Thread oznyang (JIRA)
oznyang created SOLR-3661: - Summary: RequestHandlerBase getStatistics method has atom problem Key: SOLR-3661 URL: https://issues.apache.org/jira/browse/SOLR-3661 Project: Solr Issue Type: Bug