date:20120828

Reproduces for me with:

> [junit4:junit4]   2> NOTE: reproduce with: ant test  
> -Dtestcase=GroupFacetCollectorTest 
> -Dtests.method=testMVGroupedFacetingWithDeletes -Dtests.seed=BF2B1FB32B155A3 
> -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=sk_SK 
> -Dtests.timezone=America/Atikokan -Dtests.file.encoding=UTF-8

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-3765) Wrong handling of documents with same id in cross collection searches

2012-08-28 Thread Per Steffensen (JIRA)


 [ 
https://issues.apache.org/jira/browse/SOLR-3765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Per Steffensen updated SOLR-3765:
-

Description: 
Dialog with myself from solr-users mailing list:

Per Steffensen skrev:
{quote} 
Hi

Due to what we have seen in recent tests I got in doubt how Solr search is 
actually supposed to behave
* Searching with "distrib=true&q=*:*&rows=10&collection=x,y,z&sort=timestamp 
asc"
** Is Solr supposed to return the 10 documents with the lowest timestamp across 
all documents in all slices of collection x, y and z, or is it supposed to just 
pick 10 random documents from those slices and just sort those 10 randomly 
selected documents?
** Put in another way - is this search supposed to be consistent, returning 
exactly the same set of documents when performed several times (no documents 
are updated between consecutive searches)?
{quote}

Fortunately I believe the answer is, that it ought to "return the 10 documents 
with the lowest timestamp across all documents in all slices of collection x, y 
and Z". The reason I asked was because I got different responses for 
consecutive simular requests. Now I believe it can be explained by the bug 
described below. I guess they you do cross-collection/shard searches, the 
"request-handling" Solr forwards the query to all involved shards simultanious 
and merges sub-results into the final result as they are returned from the 
shards. Because of the "consider documents with same id as the same document 
even though the come from different collections"-bug it is kinda random 
(depending on which shards responds first/last), for a given id, what 
collection the document with that specific id is taken from. And if documents 
with the same id from different collections has different timestamp it is 
random where that document ends up in the final sorted result.

So i believe this inconsistency can be explained by the bug described below.

{quote}
* A search returns a "numFound"-field telling how many documents all in all 
matches the search-criteria, even though not all those documents are returned 
by the search. It is a crazy question to ask, but I will do it anyway because 
we actually see a problem with this. Isnt it correct that two searches which 
only differs on the "rows"-number (documents to be returned) should always 
return the same value for "numFound"?
{quote}

Well I found out myself what the problem is (or seems to be) - see:
http://lucene.472066.n3.nabble.com/Changing-value-of-start-parameter-affects-numFound-td2460645.html
http://lucene.472066.n3.nabble.com/numFound-inconsistent-for-different-rows-param-td3997269.html
http://lucene.472066.n3.nabble.com/Solr-v3-5-0-numFound-changes-when-paging-through-results-on-8-shard-cluster-td3990400.html

Until 4.0 this "bug" could be "ignored" because it was ok for a cross-shards 
search to consider documents with identical id's as dublets and therefore only 
returning/counting one of them. It is still, in 4.0, ok within the same 
collection, but across collections identical id's should not be considered 
dublicates and should not reduce documents returned/counted. So i believe this 
"feature" has now become a bug in 4.0 when it comes to cross-collections 
searches.

{quote}
Thanks!

Regards, Steff
{quote}



  was:
Dialog with myself from solr-users mailing list:

Per Steffensen skrev:
{qoute} 
Hi

Due to what we have seen in recent tests I got in doubt how Solr search is 
actually supposed to behave
* Searching with "distrib=true&q=*:*&rows=10&collection=x,y,z&sort=timestamp 
asc"
** Is Solr supposed to return the 10 documents with the lowest timestamp across 
all documents in all slices of collection x, y and z, or is it supposed to just 
pick 10 random documents from those slices and just sort those 10 randomly 
selected documents?
** Put in another way - is this search supposed to be consistent, returning 
exactly the same set of documents when performed several times (no documents 
are updated between consecutive searches)?
{quote}

Fortunately I believe the answer is, that it ought to "return the 10 documents 
with the lowest timestamp across all documents in all slices of collection x, y 
and Z". The reason I asked was because I got different responses for 
consecutive simular requests. Now I believe it can be explained by the bug 
described below. I guess they you do cross-collection/shard searches, the 
"request-handling" Solr forwards the query to all involved shards simultanious 
and merges sub-results into the final result as they are returned from the 
shards. Because of the "consider documents with same id as the same document 
even though the come from different collections"-bug it is kinda random 
(depending on which shards responds first/last), for a given id, what 
collection the document with that specific id is taken from. And if documents 
with the same id from different collections has different

[jira] [Updated] (SOLR-3765) Wrong handling of documents with same id in cross collection searches

2012-08-28 Thread Per Steffensen (JIRA)


 [ 
https://issues.apache.org/jira/browse/SOLR-3765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Per Steffensen updated SOLR-3765:
-

Description: 
Dialog with myself from solr-users mailing list:

Per Steffensen skrev:
{qoute} 
Hi

Due to what we have seen in recent tests I got in doubt how Solr search is 
actually supposed to behave
* Searching with "distrib=true&q=*:*&rows=10&collection=x,y,z&sort=timestamp 
asc"
** Is Solr supposed to return the 10 documents with the lowest timestamp across 
all documents in all slices of collection x, y and z, or is it supposed to just 
pick 10 random documents from those slices and just sort those 10 randomly 
selected documents?
** Put in another way - is this search supposed to be consistent, returning 
exactly the same set of documents when performed several times (no documents 
are updated between consecutive searches)?
{quote}

Fortunately I believe the answer is, that it ought to "return the 10 documents 
with the lowest timestamp across all documents in all slices of collection x, y 
and Z". The reason I asked was because I got different responses for 
consecutive simular requests. Now I believe it can be explained by the bug 
described below. I guess they you do cross-collection/shard searches, the 
"request-handling" Solr forwards the query to all involved shards simultanious 
and merges sub-results into the final result as they are returned from the 
shards. Because of the "consider documents with same id as the same document 
even though the come from different collections"-bug it is kinda random 
(depending on which shards responds first/last), for a given id, what 
collection the document with that specific id is taken from. And if documents 
with the same id from different collections has different timestamp it is 
random where that document ends up in the final sorted result.

So i believe this inconsistency can be explained by the bug described below.

{quote}
* A search returns a "numFound"-field telling how many documents all in all 
matches the search-criteria, even though not all those documents are returned 
by the search. It is a crazy question to ask, but I will do it anyway because 
we actually see a problem with this. Isnt it correct that two searches which 
only differs on the "rows"-number (documents to be returned) should always 
return the same value for "numFound"?
{quote}

Well I found out myself what the problem is (or seems to be) - see:
http://lucene.472066.n3.nabble.com/Changing-value-of-start-parameter-affects-numFound-td2460645.html
http://lucene.472066.n3.nabble.com/numFound-inconsistent-for-different-rows-param-td3997269.html
http://lucene.472066.n3.nabble.com/Solr-v3-5-0-numFound-changes-when-paging-through-results-on-8-shard-cluster-td3990400.html

Until 4.0 this "bug" could be "ignored" because it was ok for a cross-shards 
search to consider documents with identical id's as dublets and therefore only 
returning/counting one of them. It is still, in 4.0, ok within the same 
collection, but across collections identical id's should not be considered 
dublicates and should not reduce documents returned/counted. So i believe this 
"feature" has now become a bug in 4.0 when it comes to cross-collections 
searches.

{quote}
Thanks!

Regards, Steff
{quote}



  was:
Dialog with myself from solr-users mailing list:

Per Steffensen skrev:
> Hi
>
> Due to what we have seen in recent tests I got in doubt how Solr search is 
> actually supposed to behave
>
> * Searching with "distrib=true&q=*:*&rows=10&collection=x,y,z&sort=timestamp 
> asc"
> ** Is Solr supposed to return the 10 documents with the lowest timestamp 
> across all documents in all slices of collection x, y and z, or is it 
> supposed to just pick 10 random documents from those slices and just sort 
> those 10 randomly selected documents?
> ** Put in another way - is this search supposed to be consistent, returning 
> exactly the same set of documents when performed several times (no documents 
> are updated between consecutive searches)?
Fortunately I believe the answer is, that it ought to "return the 10 documents 
with the lowest timestamp across all documents in all slices of collection x, y 
and Z". The reason I asked was because I got different responses for 
consecutive simular requests. Now I believe it can be explained by the bug 
described below. I guess they you do cross-collection/shard searches, the 
"request-handling" Solr forwards the query to all involved shards simultanious 
and merges sub-results into the final result as they are returned from the 
shards. Because of the "consider documents with same id as the same document 
even though the come from different collections"-bug it is kinda random 
(depending on which shards responds first/last), for a given id, what 
collection the document with that specific id is taken from. And if documents 
with the same id from different collections has

[jira] [Created] (SOLR-3765) Wrong handling of documents with same id in cross collection searches

2012-08-28 Thread Per Steffensen (JIRA)

Per Steffensen created SOLR-3765:


 Summary: Wrong handling of documents with same id in cross 
collection searches
 Key: SOLR-3765
 URL: https://issues.apache.org/jira/browse/SOLR-3765
 Project: Solr
  Issue Type: Bug
  Components: search, SolrCloud
Affects Versions: 4.0
 Environment: Self-build version of Solr fra 4.x branch (revision )
Reporter: Per Steffensen


Dialog with myself from solr-users mailing list:

Per Steffensen skrev:
> Hi
>
> Due to what we have seen in recent tests I got in doubt how Solr search is 
> actually supposed to behave
>
> * Searching with "distrib=true&q=*:*&rows=10&collection=x,y,z&sort=timestamp 
> asc"
> ** Is Solr supposed to return the 10 documents with the lowest timestamp 
> across all documents in all slices of collection x, y and z, or is it 
> supposed to just pick 10 random documents from those slices and just sort 
> those 10 randomly selected documents?
> ** Put in another way - is this search supposed to be consistent, returning 
> exactly the same set of documents when performed several times (no documents 
> are updated between consecutive searches)?
Fortunately I believe the answer is, that it ought to "return the 10 documents 
with the lowest timestamp across all documents in all slices of collection x, y 
and Z". The reason I asked was because I got different responses for 
consecutive simular requests. Now I believe it can be explained by the bug 
described below. I guess they you do cross-collection/shard searches, the 
"request-handling" Solr forwards the query to all involved shards simultanious 
and merges sub-results into the final result as they are returned from the 
shards. Because of the "consider documents with same id as the same document 
even though the come from different collections"-bug it is kinda random 
(depending on which shards responds first/last), for a given id, what 
collection the document with that specific id is taken from. And if documents 
with the same id from different collections has different timestamp it is 
random where that document ends up in the final sorted result.

So i believe this inconsistency can be explained by the bug described below.
>
> * A search returns a "numFound"-field telling how many documents all in all 
> matches the search-criteria, even though not all those documents are returned 
> by the search. It is a crazy question to ask, but I will do it anyway because 
> we actually see a problem with this. Isnt it correct that two searches which 
> only differs on the "rows"-number (documents to be returned) should always 
> return the same value for "numFound"?
Well I found out myself what the problem is (or seems to be) - see:
http://lucene.472066.n3.nabble.com/Changing-value-of-start-parameter-affects-numFound-td2460645.html
http://lucene.472066.n3.nabble.com/numFound-inconsistent-for-different-rows-param-td3997269.html
http://lucene.472066.n3.nabble.com/Solr-v3-5-0-numFound-changes-when-paging-through-results-on-8-shard-cluster-td3990400.html

Until 4.0 this "bug" could be "ignored" because it was ok for a cross-shards 
search to consider documents with identical id's as dublets and therefore only 
returning/counting one of them. It is still, in 4.0, ok within the same 
collection, but across collections identical id's should not be considered 
dublicates and should not reduce documents returned/counted. So i believe this 
"feature" has now become a bug in 4.0 when it comes to cross-collections 
searches.
>
> Thanks!
>
> Regards, Steff
>



--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Resolved] (SOLR-2747) Include formatted Changes.html for release

2012-08-28 Thread Steven Rowe (JIRA)


 [ 
https://issues.apache.org/jira/browse/SOLR-2747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steven Rowe resolved SOLR-2747.
---

   Resolution: Fixed
Fix Version/s: (was: 4.1)
   5.0
   4.0

Committed:

* branch_4x: [r1378442|http://svn.apache.org/viewvc?view=rev&rev=1378442], 
[r1378449|http://svn.apache.org/viewvc?view=rev&rev=1378449]
* trunk: [r1378452|http://svn.apache.org/viewvc?view=rev&rev=1378452]

> Include formatted Changes.html for release
> --
>
> Key: SOLR-2747
> URL: https://issues.apache.org/jira/browse/SOLR-2747
> Project: Solr
>  Issue Type: Improvement
>Reporter: Martijn van Groningen
>Assignee: Steven Rowe
>Priority: Minor
> Fix For: 4.0, 5.0
>
> Attachments: SOLR-2747.patch, SOLR-2747.patch, SOLR-2747.patch, 
> SOLR-2747.patch, SOLR-2747.patch
>
>
> Just like when releasing Lucene, Solr should also have a html formatted 
> changes file.
> The Lucene Perl script (lucene/src/site/changes/changes2html.pl) should be 
> reused.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: [JENKINS] Lucene-Solr-Tests-4.x-java7 - Build # 366 - Failure

> [junit4:junit4] ERROR: JVM J0 ended with an exception, command line: 
> /usr/local/openjdk7/jre/bin/java -XX:+UseG1GC -Dtests.prefix=tests -

I checked again, this is caused by two interfering issues --

1) BasicDistributedZk2Test.testDistribSearch hangs on socket read (I
posted a stack trace a while ago).

2) The JVM crashes without any core dump or any other info right after
some of the hanging threads are interrupted when the timeout occurs.

I filed an issue with the randomized runner too because some of the
events (timeout messages, jstack stack traces) are truncated from the
console log (when the JVM terminates the event stream is corrupted so
the runner never completes the last suite's execution). This is
recoverable from the *.events file though. I couldn't reproduce this
issue either locally or on jenkins so I assume it is a race condition
(possibly distributed) somewhere.

Dawid

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-4.x-Linux (32bit/jdk1.6.0_34) - Build # 723 - Failure!

2012-08-28 Thread Policeman Jenkins Server

Build: http://jenkins.sd-datasolutions.de/job/Lucene-Solr-4.x-Linux/723/
Java: 32bit/jdk1.6.0_34 -client -XX:+UseConcMarkSweepGC

1 tests failed.
REGRESSION:  
org.apache.lucene.search.grouping.GroupFacetCollectorTest.testMVGroupedFacetingWithDeletes

Error Message:
limit=1 actual=2

Stack Trace:
java.lang.AssertionError: limit=1 actual=2
at 
__randomizedtesting.SeedInfo.seed([BF2B1FB32B155A3:6E4CF04CB463B524]:0)
at 
org.apache.lucene.index.RandomIndexWriter.doRandomForceMerge(RandomIndexWriter.java:395)
at 
org.apache.lucene.index.RandomIndexWriter.close(RandomIndexWriter.java:439)
at 
org.apache.lucene.search.grouping.GroupFacetCollectorTest.testMVGroupedFacetingWithDeletes(GroupFacetCollectorTest.java:274)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at 
org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:345)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:769)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:429)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:40)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:345)
at java.lang.Thread.run(Thread.java:662)




Build Log:
[...truncated 4707 lines...]
[junit4:junit4] Suite: org.apache.lucene.search.grouping.GroupFacetCollectorTest
[junit4:junit4]   2> NOTE: reproduce with: ant test  
-Dtestcase=GroupFacetCollectorTest 
-Dtests.method=testMVGroupedFacetingWithDeletes -Dtests.seed=BF2B1FB32B155A3 
-Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=sk_SK 
-Dtests.timezone=America/Atikokan -Dtests.file.encoding=UTF-8
[junit4:junit4] FAILURE 0.05s J1 | 
GroupFacetCollectorTest.testMVGro

RE: Build failed in Jenkins: Lucene-trunk-Linux-Java7-64-test-only #3430

Simon has granted me access on flonkings Jenkins, and I've set up the 
hugerriblific build failure regex in the notification email configuration for 
the jobs hosted there. - Steve

-Original Message-
From: Steven A Rowe [mailto:sar...@syr.edu] 
Sent: Saturday, August 25, 2012 8:13 PM
To: dev@lucene.apache.org
Subject: RE: Build failed in Jenkins: Lucene-trunk-Linux-Java7-64-test-only 
#3430

I'm pretty sure the AMBFR isn't being using on flonkings.  I don't have access, 
so I haven't put it there. - Steve

-Original Message-
From: Michael McCandless [mailto:luc...@mikemccandless.com] 
Sent: Saturday, August 25, 2012 6:12 PM
To: dev@lucene.apache.org
Cc: buil...@flonkings.com; sim...@apache.org
Subject: Re: Build failed in Jenkins: Lucene-trunk-Linux-Java7-64-test-only 
#3430

Also, it looks like that awesome, massive build failure regexp is
failing to properly splice out this failure?

Mike McCandless

http://blog.mikemccandless.com


On Sat, Aug 25, 2012 at 6:11 PM, Michael McCandless
 wrote:
> +1 to remove it.
>
> But why doesn't it fail all the time...?
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
> On Sat, Aug 25, 2012 at 5:28 PM, Robert Muir  wrote:
>> I think this test method is useless and we should remove it.
>>
>> On Sat, Aug 25, 2012 at 5:17 PM,   wrote:
>>> See 
>>>
>>> --
>>> [...truncated 1008 lines...]
>>> [junit4:junit4] Completed on J5 in 0.43s, 4 tests
>>> [junit4:junit4]
>>> [junit4:junit4] Suite: org.apache.lucene.search.TestRegexpRandom
>>> [junit4:junit4] Completed on J2 in 1.07s, 1 test
>>> [junit4:junit4]
>>> [junit4:junit4] Suite: org.apache.lucene.search.TestWildcard
>>> [junit4:junit4] Completed on J0 in 0.85s, 8 tests
>>> [junit4:junit4]
>>> [junit4:junit4] Suite: 
>>> org.apache.lucene.search.TestSimpleExplanationsOfNonMatches
>>> [junit4:junit4] Completed on J4 in 1.42s, 69 tests
>>> [junit4:junit4]
>>> [junit4:junit4] Suite: org.apache.lucene.index.TestUniqueTermCount
>>> [junit4:junit4] Completed on J2 in 0.37s, 1 test
>>> [junit4:junit4]
>>> [junit4:junit4] Suite: 
>>> org.apache.lucene.search.TestComplexExplanationsOfNonMatches
>>> [junit4:junit4] Completed on J7 in 0.59s, 22 tests
>>> [junit4:junit4]
>>> [junit4:junit4] Suite: org.apache.lucene.index.TestSumDocFreq
>>> [junit4:junit4] Completed on J5 in 0.60s, 1 test
>>> [junit4:junit4]
>>> [junit4:junit4] Suite: org.apache.lucene.index.TestBinaryTerms
>>> [junit4:junit4] Completed on J2 in 0.18s, 1 test
>>> [junit4:junit4]
>>> [junit4:junit4] Suite: org.apache.lucene.index.TestParallelReaderEmptyIndex
>>> [junit4:junit4] Completed on J7 in 0.19s, 2 tests
>>> [junit4:junit4]
>>> [junit4:junit4] Suite: org.apache.lucene.index.TestPerSegmentDeletes
>>> [junit4:junit4] Completed on J4 in 0.36s, 1 test
>>> [junit4:junit4]
>>> [junit4:junit4] Suite: org.apache.lucene.index.TestIndexWriterConfig
>>> [junit4:junit4] Completed on J5 in 0.22s, 9 tests
>>> [junit4:junit4]
>>> [junit4:junit4] Suite: org.apache.lucene.TestSearchForDuplicates
>>> [junit4:junit4] Completed on J0 in 0.53s, 1 test
>>> [junit4:junit4]
>>> [junit4:junit4] Suite: org.apache.lucene.search.TestFilteredSearch
>>> [junit4:junit4] Completed on J2 in 0.20s, 1 test
>>> [junit4:junit4]
>>> [junit4:junit4] Suite: org.apache.lucene.index.TestForTooMuchCloning
>>> [junit4:junit4] Completed on J3 in 3.60s, 1 test
>>> [junit4:junit4]
>>> [junit4:junit4] Suite: org.apache.lucene.index.Test2BPostings
>>> [junit4:junit4] IGNOR/A 0.14s J0 | Test2BPostings.test
>>> [junit4:junit4]> Assumption #1: 'nightly' test group is disabled 
>>> (@Nightly)
>>> [junit4:junit4] Completed on J0 in 0.22s, 1 test, 1 skipped
>>> [junit4:junit4]
>>> [junit4:junit4] Suite: org.apache.lucene.index.TestFilterAtomicReader
>>> [junit4:junit4] Completed on J2 in 0.23s, 2 tests
>>> [junit4:junit4]
>>> [junit4:junit4] Suite: org.apache.lucene.search.TestCachingWrapperFilter
>>> [junit4:junit4] Completed on J4 in 0.38s, 5 tests
>>> [junit4:junit4]
>>> [junit4:junit4] Suite: org.apache.lucene.search.TestDocValuesScoring
>>> [junit4:junit4] Completed on J7 in 0.43s, 1 test
>>> [junit4:junit4]
>>> [junit4:junit4] Suite: org.apache.lucene.search.TestAutomatonQuery
>>> [junit4:junit4] Completed on J6 in 1.93s, 6 tests
>>> [junit4:junit4]
>>> [junit4:junit4] Suite: 
>>> org.apache.lucene.util.junitcompat.TestSeedFromUncaught
>>> [junit4:junit4] Completed on J0 in 0.14s, 1 test
>>> [junit4:junit4]
>>> [junit4:junit4] Suite: org.apache.lucene.codecs.appending.TestAppendingCodec
>>> [junit4:junit4] Completed on J2 in 0.12s, 2 tests
>>> [junit4:junit4]
>>> [junit4:junit4] Suite: org.apache.lucene.store.TestFileSwitchDirectory
>>> [junit4:junit4] Completed on J3 in 0.26s, 4 tests
>>> [junit4:junit4]
>>> [junit4:junit4] Suite: org.apache.lucene.search.TestPhrasePrefixQuery
>>> [junit4:junit4] Completed on J6 in 0.12s, 1 test
>>> [junit4:junit4]
>>> [junit4:junit4] Suite: org.apache.lucene.search.TestDocIdSet
>>> [junit4:junit4] Completed on J5

[JENKINS] Lucene-Solr-Tests-4.x-java7 - Build # 366 - Failure

2012-08-28 Thread Apache Jenkins Server

Build: https://builds.apache.org/job/Lucene-Solr-Tests-4.x-java7/366/

All tests passed

Build Log:
[...truncated 7606 lines...]
[junit4:junit4] ERROR: JVM J0 ended with an exception, command line: 
/usr/local/openjdk7/jre/bin/java -XX:+UseG1GC -Dtests.prefix=tests 
-Dtests.seed=597A59E4DFFC3345 -Xmx512M -Dtests.iters= -Dtests.verbose=false 
-Dtests.infostream=false 
-Dtests.lockdir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/lucene/build
 -Dtests.codec=random -Dtests.postingsformat=random -Dtests.locale=random 
-Dtests.timezone=random -Dtests.directory=random 
-Dtests.linedocsfile=europarl.lines.txt.gz -Dtests.luceneMatchVersion=4.0 
-Dtests.cleanthreads=perClass 
-Djava.util.logging.config.file=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/testlogging.properties
 -Dtests.nightly=false -Dtests.weekly=false -Dtests.slow=true 
-Dtests.asserts.gracious=false -Dtests.multiplier=3 -DtempDir=. 
-Djava.io.tmpdir=. -Dlucene.version=4.0-SNAPSHOT -Djetty.testMode=1 
-Djetty.insecurerandom=1 
-Dsolr.directoryFactory=org.apache.solr.core.MockDirectoryFactory 
-Dfile.encoding=US-ASCII -classpath 
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/classes/test:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-test-framework/classes/java:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test-files:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/lucene/build/test-framework/classes/java:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-solrj/classes/java:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/classes/java:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/lucene/build/analysis/common/lucene-analyzers-common-4.0-SNAPSHOT.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/lucene/build/analysis/kuromoji/lucene-analyzers-kuromoji-4.0-SNAPSHOT.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/lucene/build/analysis/phonetic/lucene-analyzers-phonetic-4.0-SNAPSHOT.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/lucene/build/highlighter/lucene-highlighter-4.0-SNAPSHOT.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/lucene/build/memory/lucene-memory-4.0-SNAPSHOT.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/lucene/build/misc/lucene-misc-4.0-SNAPSHOT.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/lucene/build/spatial/lucene-spatial-4.0-SNAPSHOT.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/lucene/build/suggest/lucene-suggest-4.0-SNAPSHOT.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/lucene/build/grouping/lucene-grouping-4.0-SNAPSHOT.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/lucene/build/queries/lucene-queries-4.0-SNAPSHOT.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/lucene/build/queryparser/lucene-queryparser-4.0-SNAPSHOT.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/lib/commons-cli-1.2.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/lib/commons-codec-1.6.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/lib/commons-fileupload-1.2.1.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/lib/commons-io-2.1.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/lib/commons-lang-2.6.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/lib/easymock-2.2.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/lib/guava-r05.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/lib/httpclient-4.1.3.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/lib/httpcore-4.1.4.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/lib/httpmime-4.1.3.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/lib/javax.servlet-api-3.0.1.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/lib/jcl-over-slf4j-1.6.4.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/lib/log4j-over-slf4j-1.6.4.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/lib/slf4j-api-1.6.4.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/lib/slf4j-jdk14-1.6.4.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/lib/spatial4j-0.2.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/lib/wstx-asl-3.2.7.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/lib/zookeeper-3.3.6.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-

Re: Lucene3xCodec doesn't allow deletions in 4x?

2012-08-28 Thread Robert Muir

On Tue, Aug 28, 2012 at 10:06 PM, Chris Hostetter
 wrote:
> On the Solr side of things, i think we should even want to consider
> automaticly running IndexUpgrader on startup if we detect that the
> Lucene3xCodec is in use to simplify things -- we can't even suggest running
> "optimize" as a quick/easy way to force and index format upgrade because if
> the 3x index as already optimized then it's a no-op and the index stays in
> the 3x format.
>

By the way: i don't really agree with that. If we want to be able to
make deletes against 3.x segments, thats a trivial patch (hint:
uncomment out the UOE)

But we just need to decide thats what we want to do (and if should set
any precedent for future backwards compat).

Personally I wouldn't be against it here because of how we do sneaky
stuff for 3.x segments on commit already (e.g. 'upgrade the SI'), so
we
already support 2 codecs (preflex and preflex-upgraded). this wouldn't
make the situation any worse.

Index: 
lucene/test-framework/src/java/org/apache/lucene/codecs/lucene3x/PreFlexRWCodec.java
===
--- 
lucene/test-framework/src/java/org/apache/lucene/codecs/lucene3x/PreFlexRWCodec.java
(revision
1378430)
+++ 
lucene/test-framework/src/java/org/apache/lucene/codecs/lucene3x/PreFlexRWCodec.java
(working
copy)
@@ -38,8 +38,6 @@
   private final TermVectorsFormat termVectors = new
PreFlexRWTermVectorsFormat();
   private final SegmentInfoFormat segmentInfos = new
PreFlexRWSegmentInfoFormat();
   private final StoredFieldsFormat storedFields = new
PreFlexRWStoredFieldsFormat();
-  // TODO: this should really be a different impl
-  private final LiveDocsFormat liveDocs = new Lucene40LiveDocsFormat();

   @Override
   public PostingsFormat postingsFormat() {
@@ -87,15 +85,6 @@
   }

   @Override
-  public LiveDocsFormat liveDocsFormat() {
-if (LuceneTestCase.PREFLEX_IMPERSONATION_IS_ACTIVE) {
-  return liveDocs;
-} else {
-  return super.liveDocsFormat();
-}
-  }
-
-  @Override
   public StoredFieldsFormat storedFieldsFormat() {
 if (LuceneTestCase.PREFLEX_IMPERSONATION_IS_ACTIVE) {
   return storedFields;
Index: 
lucene/core/src/test/org/apache/lucene/index/TestBackwardsCompatibility.java
===
--- 
lucene/core/src/test/org/apache/lucene/index/TestBackwardsCompatibility.java
(revision
1378430)
+++ 
lucene/core/src/test/org/apache/lucene/index/TestBackwardsCompatibility.java
(working
copy)
@@ -301,6 +301,22 @@
   dir.close();
 }
   }
+
+  public void testDeleteOldIndex() throws IOException {
+for (String name : oldNames) {
+  if (VERBOSE) {
+System.out.println("TEST: oldName=" + name);
+  }
+  Directory dir = newDirectory(oldIndexDirs.get(name));
+  IndexWriter iw = new IndexWriter(dir, new
IndexWriterConfig(TEST_VERSION_CURRENT, null));
+  iw.deleteDocuments(new Term("id", "3"));
+  iw.close();
+  IndexReader ir = DirectoryReader.open(dir);
+  assertEquals(34, ir.numDocs());
+  ir.close();
+  dir.close();
+}
+  }

   private void doTestHits(ScoreDoc[] hits, int expectedCount,
IndexReader reader) throws IOException {
 final int hitCount = hits.length;
Index: lucene/core/src/java/org/apache/lucene/codecs/lucene3x/Lucene3xCodec.java
===
--- lucene/core/src/java/org/apache/lucene/codecs/lucene3x/Lucene3xCodec.java   
(revision
1378430)
+++ lucene/core/src/java/org/apache/lucene/codecs/lucene3x/Lucene3xCodec.java   
(working
copy)
@@ -68,12 +68,7 @@
   static final String COMPOUND_FILE_STORE_EXTENSION = "cfx";

   // TODO: this should really be a different impl
-  private final LiveDocsFormat liveDocsFormat = new Lucene40LiveDocsFormat() {
-@Override
-public void writeLiveDocs(MutableBits bits, Directory dir,
SegmentInfoPerCommit info, int newDelCount, IOContext context) throws
IOException {
-  throw new UnsupportedOperationException("this codec can only be
used for reading");
-}
-  };
+  private final LiveDocsFormat liveDocsFormat = new Lucene40LiveDocsFormat();

   // 3.x doesn't support docvalues
   private final DocValuesFormat docValuesFormat = new DocValuesFormat() {

-- 
lucidworks.com

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Lucene3xCodec doesn't allow deletions in 4x?

2012-08-28 Thread Robert Muir

On Tue, Aug 28, 2012 at 10:06 PM, Chris Hostetter
 wrote:
>
> On the solr-user list, Dirk Högemann recently mentioned a problem he was
> seeing when he tried upgrading his existing solr setup from 3.x to 4.0-BETA.
> Specifically this exception getting logged...
>
> http://find.searchhub.org/document/cdb30099bfea30c6
>
> auto commit error...:java.lang.UnsupportedOperationException: this codec can
> only be used for reading

right, the codec is read-only in every way (as documented).

>
> if it's expected, then it seems like we should definitely spell out this
> limitation in MIGRATE.txt and advocate either full rebuilds, or the use of
> IndexUpgrader for anyone who's indexes are non-static.

I don't think it belongs in MIGRATE.txt (the purpose is as an API
migration guide). But more in CHANGES.txt is fine, e.g. "we are
serious when we say read-only".

-- 
lucidworks.com

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Lucene3xCodec doesn't allow deletions in 4x?

2012-08-28 Thread Chris Hostetter



On the solr-user list, Dirk Högemann recently mentioned a problem he was 
seeing when he tried upgrading his existing solr setup from 3.x to 
4.0-BETA.  Specifically this exception getting logged...


http://find.searchhub.org/document/cdb30099bfea30c6

auto commit error...:java.lang.UnsupportedOperationException: this codec can 
only be used for reading
at 
org.apache.lucene.codecs.lucene3x.Lucene3xCodec$1.writeLiveDocs(Lucene3xCodec.java:74)
at 
org.apache.lucene.index.ReadersAndLiveDocs.writeLiveDocs(ReadersAndLiveDocs.java:278)
at 
org.apache.lucene.index.IndexWriter$ReaderPool.release(IndexWriter.java:435)
at 
org.apache.lucene.index.BufferedDeletesStream.applyDeletes(BufferedDeletesStream.java:278)
at 
org.apache.lucene.index.IndexWriter.applyAllDeletes(IndexWriter.java:2928)
at 
org.apache.lucene.index.IndexWriter.maybeApplyDeletes(IndexWriter.java:2919)
at 
org.apache.lucene.index.IndexWriter.prepareCommit(IndexWriter.java:2666)
at 
org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2793)
at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2773)
at 
org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:531)
at org.apache.solr.update.CommitTracker.run(CommitTracker.java:214)

Dirk was able to work arround this by completely re-indexing, but it 
seemed strange to me that this would happen.


My understanding is that even though an IndexUpgrader tool was now 
available, it wasn't going to be required for users to use it when 
upgrading from 3.x to 4.x.  Explicitly upgrading the index format might be 
a good idea, and might make hte index more performant, but as I understood 
it, the way things had been implemented with codecs explicitly upgrading 
the index format wasn't strictly neccessary, and that users should be able 
to upgrade their lucene apps same way that was supported with other index 
format upgrades in the past: the old index can be read, and as changes are 
made new segments will be re-written in the new format.  (Note in 
particular: at the moment we don't mention IndexUpgrader in 
MIGRATE.txt at all.)


It appears however, based on this stack trace and some other experiements 
i tried, that any attempts to "delete" documents in a segment that is 
using the Lucene3xCodec will fail.


This seems like a really scary time bomb sitaution, because if you 
upgrade, things will seem to be working -- you can even add documents, and 
depending on the order that you do things, some "old" segments may get 
merged and use the new format, so *some* deletes of "old" documents (in 
those merged/upgraded) segments may work, but then somewhere down the 
road, you may try to a delete that affects docs in a still un-merge/upgraded 
segment, and that delete will fail -- 5 minutes later, if another merge 
has happened, attempting to do the exact same delete may succeed.


All of which begs the question: is this a known/intended limitation of the 
Lucene3xCodec, or an oversight in the Lucene3xCodec?


if it's expected, then it seems like we should definitely spell out this 
limitation in MIGRATE.txt and advocate either full rebuilds, or the use of 
IndexUpgrader for anyone who's indexes are non-static.


On the Solr side of things, i think we should even want to consider 
automaticly running IndexUpgrader on startup if we detect that the 
Lucene3xCodec is in use to simplify things -- we can't even suggest 
running "optimize" as a quick/easy way to force and index format upgrade 
because if the 3x index as already optimized then it's a no-op and the 
index stays in the 3x format.



-Hoss

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-3755) shard splitting

2012-08-28 Thread Yonik Seeley (JIRA)


 [ 
https://issues.apache.org/jira/browse/SOLR-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yonik Seeley updated SOLR-3755:
---

Attachment: SOLR-3755.patch

Draft (unfinished) patch just to let people know where I'm going... 

> shard splitting
> ---
>
> Key: SOLR-3755
> URL: https://issues.apache.org/jira/browse/SOLR-3755
> Project: Solr
>  Issue Type: New Feature
>  Components: SolrCloud
>Reporter: Yonik Seeley
> Attachments: SOLR-3755.patch
>
>
> We can currently easily add replicas to handle increases in query volume, but 
> we should also add a way to add additional shards dynamically by splitting 
> existing shards.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Resolved] (SOLR-3764) Upgrading to Solr 4 from Solr 3.x doesn't support "single core" mode Solr Home dirs (ie: all 3.x examples)


 [ 
https://issues.apache.org/jira/browse/SOLR-3764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hoss Man resolved SOLR-3764.


Resolution: Cannot Reproduce

user error ... i had a typo in my path (big surprise, i know)

> Upgrading to Solr 4 from Solr 3.x doesn't support "single core" mode Solr 
> Home dirs (ie: all 3.x examples)
> --
>
> Key: SOLR-3764
> URL: https://issues.apache.org/jira/browse/SOLR-3764
> Project: Solr
>  Issue Type: Bug
>Affects Versions: 4.0-ALPHA, 4.0-BETA
>Reporter: Hoss Man
>Priority: Blocker
> Fix For: 4.0
>
>
> if users were previously using a "Solr Home" based on the solr example, and 
> attempt to upgrade to Solr 4 using their existing Solr Home dir, they will 
> get an error because the implicit behavior when no solr.xml file is found 
> seems to be assuming that a "./collection1" instanceDir will exist.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: large messages from Jenkins failures

2012-08-28 Thread Michael McCandless

On Tue, Aug 28, 2012 at 5:42 PM, Chris Hostetter
 wrote:

> If folks are concerned that certian tests fail to frequently to be
> considered "stable" and included in the main build, then let's:
>
>  1) slap a special "@UnstableTest" annotation on them
>  2) set up a new jenkins job that *only* runs these @UnstableTest jobs
>  3) configure this new jenkins job to not send any email

+1

Mike McCandless

http://blog.mikemccandless.com

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-3764) Upgrading to Solr 4 from Solr 3.x doesn't support "single core" mode Solr Home dirs (ie: all 3.x examples)


[ 
https://issues.apache.org/jira/browse/SOLR-3764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13443691#comment-13443691
 ] 

Hoss Man commented on SOLR-3764:


Steps to reproduce...

1) run the Solr 3.6.1 example and index some data
2) checkout the 4x branch, and run "cd solr && ant example && cd example"
3) run 4x, pointed to the existing 3.6.1 solr home...

{noformat}
hossman@frisbee:~/lucene/4x_dev/solr/example$ java 
-Dsolr.solr.home=/home/hossman/lucene-3.6.1_tag/solr/example/solr -jar start.jar
2012-08-28 17:20:01.656:INFO:oejs.Server:jetty-8.1.2.v20120308
2012-08-28 17:20:01.668:INFO:oejdp.ScanningAppProvider:Deployment monitor 
/home/hossman/lucene/4x_dev/solr/example/contexts at interval 0
2012-08-28 17:20:01.671:INFO:oejd.DeploymentManager:Deployable added: 
/home/hossman/lucene/4x_dev/solr/example/contexts/solr.xml
2012-08-28 17:20:02.208:INFO:oejw.StandardDescriptorProcessor:NO JSP Support 
for /solr, did not find org.apache.jasper.servlet.JspServlet
2012-08-28 17:20:02.225:INFO:oejsh.ContextHandler:started 
o.e.j.w.WebAppContext{/solr,file:/home/hossman/lucene/4x_dev/solr/example/solr-webapp/webapp/},/home/hossman/lucene/4x_dev/solr/example/webapps/solr.war
2012-08-28 17:20:02.225:INFO:oejsh.ContextHandler:started 
o.e.j.w.WebAppContext{/solr,file:/home/hossman/lucene/4x_dev/solr/example/solr-webapp/webapp/},/home/hossman/lucene/4x_dev/solr/example/webapps/solr.war
Aug 28, 2012 5:20:02 PM org.apache.solr.core.SolrResourceLoader locateSolrHome
INFO: JNDI not configured for solr (NoInitialContextEx)
Aug 28, 2012 5:20:02 PM org.apache.solr.core.SolrResourceLoader locateSolrHome
INFO: using system property solr.solr.home: 
/home/hossman/lucene-3.6.1_tag/solr/example/solr
Aug 28, 2012 5:20:02 PM org.apache.solr.core.SolrResourceLoader 
INFO: new SolrResourceLoader for deduced Solr Home: 
'/home/hossman/lucene-3.6.1_tag/solr/example/solr/'
Aug 28, 2012 5:20:02 PM org.apache.solr.servlet.SolrDispatchFilter init
INFO: SolrDispatchFilter.init()
Aug 28, 2012 5:20:02 PM org.apache.solr.core.SolrResourceLoader locateSolrHome
INFO: JNDI not configured for solr (NoInitialContextEx)
Aug 28, 2012 5:20:02 PM org.apache.solr.core.SolrResourceLoader locateSolrHome
INFO: using system property solr.solr.home: 
/home/hossman/lucene-3.6.1_tag/solr/example/solr
Aug 28, 2012 5:20:02 PM org.apache.solr.core.CoreContainer$Initializer 
initialize
INFO: looking for solr.xml: 
/home/hossman/lucene-3.6.1_tag/solr/example/solr/solr.xml
Aug 28, 2012 5:20:02 PM org.apache.solr.core.CoreContainer 
INFO: New CoreContainer 420745445
Aug 28, 2012 5:20:02 PM org.apache.solr.core.CoreContainer$Initializer 
initialize
INFO: no solr.xml file found - using default
Aug 28, 2012 5:20:02 PM org.apache.solr.core.CoreContainer load
INFO: Loading CoreContainer using Solr Home: 
'/home/hossman/lucene-3.6.1_tag/solr/example/solr/'
Aug 28, 2012 5:20:02 PM org.apache.solr.core.SolrResourceLoader 
INFO: new SolrResourceLoader for directory: 
'/home/hossman/lucene-3.6.1_tag/solr/example/solr/'
Aug 28, 2012 5:20:02 PM org.apache.solr.core.CoreContainer load
INFO: Registering Log Listener
Aug 28, 2012 5:20:02 PM org.apache.solr.core.CoreContainer create
INFO: Creating SolrCore 'collection1' using instanceDir: 
/home/hossman/lucene-3.6.1_tag/solr/example/solr/collection1
Aug 28, 2012 5:20:02 PM org.apache.solr.core.SolrResourceLoader 
INFO: new SolrResourceLoader for directory: 
'/home/hossman/lucene-3.6.1_tag/solr/example/solr/collection1/'
Aug 28, 2012 5:20:02 PM org.apache.solr.common.SolrException log
SEVERE: null:java.io.IOException: Can't find resource 'solrconfig.xml' in 
classpath or 
'/home/hossman/lucene-3.6.1_tag/solr/example/solr/collection1/conf/', 
cwd=/home/hossman/lucene/4x_dev/solr/example
at 
org.apache.solr.core.SolrResourceLoader.openResource(SolrResourceLoader.java:317)
at 
org.apache.solr.core.SolrResourceLoader.openConfig(SolrResourceLoader.java:283)
at org.apache.solr.core.Config.(Config.java:111)
at org.apache.solr.core.Config.(Config.java:78)
at org.apache.solr.core.SolrConfig.(SolrConfig.java:117)
at org.apache.solr.core.CoreContainer.create(CoreContainer.java:780)
at org.apache.solr.core.CoreContainer.load(CoreContainer.java:539)
at 
org.apache.solr.core.CoreContainer$Initializer.initialize(CoreContainer.java:312)
at 
org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:106)
at org.eclipse.jetty.servlet.FilterHolder.doStart(FilterHolder.java:114)
at 
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:59)
at 
org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:754)
at 
org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:258)
at 
org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1221)
at 
org.eclipse.jett

[jira] [Created] (SOLR-3764) Upgrading to Solr 4 from Solr 3.x doesn't support "single core" mode Solr Home dirs (ie: all 3.x examples)

Hoss Man created SOLR-3764:
--

 Summary: Upgrading to Solr 4 from Solr 3.x doesn't support "single 
core" mode Solr Home dirs (ie: all 3.x examples)
 Key: SOLR-3764
 URL: https://issues.apache.org/jira/browse/SOLR-3764
 Project: Solr
  Issue Type: Bug
Affects Versions: 4.0-BETA, 4.0-ALPHA
Reporter: Hoss Man
Priority: Blocker
 Fix For: 4.0


if users were previously using a "Solr Home" based on the solr example, and 
attempt to upgrade to Solr 4 using their existing Solr Home dir, they will get 
an error because the implicit behavior when no solr.xml file is found seems to 
be assuming that a "./collection1" instanceDir will exist.


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-2810) Explore Alternate Stored Field approaches for highly redundant data


[ 
https://issues.apache.org/jira/browse/LUCENE-2810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13443679#comment-13443679
 ] 

Adrien Grand commented on LUCENE-2810:
--

Oops, I didn't know of this issue when I opened LUCENE-4226. It tries to solve 
a very similar issue I think!

> Explore Alternate Stored Field approaches for highly redundant data
> ---
>
> Key: LUCENE-2810
> URL: https://issues.apache.org/jira/browse/LUCENE-2810
> Project: Lucene - Core
>  Issue Type: Improvement
>  Components: core/store
>Reporter: Grant Ingersoll
>Assignee: Grant Ingersoll
>
> In some cases (logs, HTML pages w/ boilerplate, etc.), the stored fields for 
> documents contain a lot of redundant information and end up wasting a lot of 
> space across a large collection of documents.  For instance, simply 
> compressing a typical log file often results in > 75% compression rates.  We 
> should explore mechanisms for applying compression across all the documents 
> for a field (or fields) while still maintaining relatively fast lookup (that 
> being said, in most logging applications, fast retrieval of a given event is 
> not always critical.)  For instance, perhaps it is possible to have a part of 
> storage that contains the set of unique values for all the fields and the 
> document field value simply contains a reference (could be as small as a few 
> bits depending on the number of uniq. items) to that value instead of having 
> a full copy.  Extending this, perhaps we can leverage some existing 
> compression capabilities in Java to provide this as well.  
> It may make sense to implement this as a Directory, but it might also make 
> sense as a Codec, if and when we have support for changing storage Codecs.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (LUCENE-4226) Efficient compression of small to medium stored fields


 [ 
https://issues.apache.org/jira/browse/LUCENE-4226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Adrien Grand updated LUCENE-4226:
-

Attachment: CompressionBenchmark.java
LUCENE-4226.patch

New patch as well as the code I used to benchmark.

Documents are still compressed into chunks, but I removed the ability to select 
the compression algorithm on a per-field basis in order to make the patch 
simpler and to handle cross-field compression.

I also added an index in front of compressed data using packed ints, so that 
uncompressors can stop uncompressing when enough data has been uncompressed.

The JDK only includes a moderately fast compression algorithm (deflate), but 
for this kind of use-case, we would probably be more interested in fast 
compression and uncompression algorithms such as LZ4 
(http://code.google.com/p/lz4/) or Snappy (http://code.google.com/p/snappy/). 
Since lucene-core has no dependency, I ported LZ4 to Java (included in the 
patch, see o.a.l.util.compress).

LZ4 has a very fast uncompressor and two compression modes :
 - fast scan, which looks for the last offset in the stream that has at least 4 
common bytes (using a hash table) and adds a reference to it,
 - high compression, which looks for the last 256 offsets in the stream that 
have at least 4 common bytes, takes the one that has the longest common 
sequence, and then performs trade-offs between overlapping matches in order to 
improve the compression ratio.

(In case you are curious about LZ4, I did some benchmarking with other 
compression algorithms in 
http://blog.jpountz.net/post/28092106032/wow-lz4-is-fast, unfortunately the 
high-compression Java impl is not included in the benchmark.)

I ran a similar benchmark as for my first patch, but this time I only 
compressed and stored the 1kb text field (the title field being too small was 
unfair for document-level compression with deflate). Here are the results :

{noformat}
Format   Chunk size  Compression ratio IndexReader.document time

uncompressed   100% 100%
doc/deflate 1   58% 579%
doc/deflate 9   57% 577%
index/deflate 1  4K 50%1057%
index/deflate 9  4K 48%1037%
index/lz4 scan   4K 70% 329%
index/lz4 hc 4K 66% 321%
index/deflate 1   1 60% 457%
index/deflate 9   1 59% 454%
index/lz4 scan1 81% 171%
index/lz4 hc  1 79% 176%
{noformat}

NOTE: chunk size = 1 means that there was only one document in the chunk (there 
is a compress+flush every time the byte size of documents is >= the chunk size).

NOTE: these number have been computed with the whole index fitting in the I/O 
cache. The performance should be more in favor of the compressing formats as 
soon as the index does not fit in the I/O cache anymore.

There are still a few nocommits in the patch, but it should be easy to get rid 
of them. I'd be very happy to have some feedback. :-)

> Efficient compression of small to medium stored fields
> --
>
> Key: LUCENE-4226
> URL: https://issues.apache.org/jira/browse/LUCENE-4226
> Project: Lucene - Core
>  Issue Type: Improvement
>  Components: core/index
>Reporter: Adrien Grand
>Priority: Trivial
> Attachments: CompressionBenchmark.java, CompressionBenchmark.java, 
> LUCENE-4226.patch, LUCENE-4226.patch, SnappyCompressionAlgorithm.java
>
>
> I've been doing some experiments with stored fields lately. It is very common 
> for an index with stored fields enabled to have most of its space used by the 
> .fdt index file. To prevent this .fdt file from growing too much, one option 
> is to compress stored fields. Although compression works rather well for 
> large fields, this is not the case for small fields and the compression ratio 
> can be very close to 100%, even with efficient compression algorithms.
> In order to improve the compression ratio for small fields, I've written a 
> {{StoredFieldsFormat}} that compresses several documents in a single chunk of 
> data. To see how it behaves in terms of document deserialization speed and 
> compression ratio, I've run several tests with different index compression 
> strategies on 100,000 docs from Mike's 1K Wikipedia articles (title and text 
> were indexed

[jira] [Resolved] (SOLR-3757) Add coment in solrConfig.xml about master/slave SolrReplication


 [ 
https://issues.apache.org/jira/browse/SOLR-3757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hoss Man resolved SOLR-3757.


Resolution: Fixed
  Assignee: Hoss Man

Thanks for pointing that out Essoussi,  I've tried to simplify things by only 
having a single /replication handler in the example, with comments explaining 
how it's used for SolrCloud, and how to cahnge it if you just wnat simple 
master/slave replication.

Committed revision 1378387.
Committed revision 1378388.


> Add coment in solrConfig.xml about master/slave SolrReplication
> ---
>
> Key: SOLR-3757
> URL: https://issues.apache.org/jira/browse/SOLR-3757
> Project: Solr
>  Issue Type: Bug
>  Components: replication (java)
>Affects Versions: 4.0-BETA
>Reporter: ESSOUSSI Jamel
>Assignee: Hoss Man
> Fix For: 4.0
>
>
> Hi,
> We should add a notification that if we would use master/slave replication we 
> should comment the SolrCloud replicationHandler in the solrConfig.xml file.
>  startup="lazy" /> 
> else it will overload the replication handler.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: RecursivePrefixTreeFieldType class not found

2012-08-28 Thread Jones, Dan

THANK you Mr. Smiley!

I waited days for a response from someone. Thank you so much



Hi Dan,

That page is the in-progress documentation for this: 
https://issues.apache.org/jira/browse/SOLR-3304 which hasn't been committed
and so it requires you check out Lucene/Solr and apply the patch. The class
reference will be solr.SpatialRecursivePrefixTreeFieldType. I updated the wiki.

~ David Smiley

On Aug 27, 2012, at 2:36 PM, Jones, Dan wrote:

According to the document I was reading here: 
http://wiki.apache.org/solr/SolrAdaptersForLuceneSpatial4

First, you must register a spatial field type in the Solr schema.xml file. The
instructions in this whole document imply
theRecursivePrefixTreeStrategy
based field type used in a geospatial context.



I need to set the fieldType to RecursivePrefixTreeFieldType and of course, I'm
getting class not found. I'm using the latest solr 4.0.0-BETA

I have a field that I would like to import into solr that is a MULTIPOLYGON

For Example: TUV Tuvalu MULTIPOLYGON (((179.21322733454343 -8.561290924154292,
179.20240933453334 -8.465417924064994, 179.2183813345482 -8.481890924080346,
179.2251453345545 -8.492217924089957, 179.23109133456006 -8.50491792410179,
179.23228133456115 -8.51841792411436, 179.23149133456042 -8.533499924128407,
179.22831833455746 -8.543426924137648, 179.22236333455191 -8.554145924147633,
179.21322733454343 -8.561290924154292)), ((177.2902543327525 -6.114445921875486,
177.28137233274424 -6.109863921871224, 177.27804533274116 -6.099445921861516,
177.28137233274424 -6.089445921852203, 177.3055273327667 -6.10597292186759,
177.2958093327577 -6.113890921874969, 177.2902543327525 -6.114445921875486)),
((176.30636333183617 -6.288335922037433, 176.29871833182904 -6.285135922034456,
176.29525433182584 -6.274581922024623, 176.30601833183584 -6.260135922011173,
176.31198133184142 -6.28215492203168, 176.30636333183617 -6.288335922037433)),
((178.69580033406152 -7.484163923151129, 178.68885433405507 -7.480835923148035,
178.68878133405497 -7.467572923135677, 178.7017813340671 -7.475208923142787,
178.69580033406152 -7.484163923151129)))

Since the LSP was moved into Solr, would there be a different name for the
class? (I'm not sure the factory class above can be found yet either)

Any help would be much appreciated!



This communication (including all attachments) is intended solely for
the use of the person(s) to whom it is addressed and should be treated
as a confidential AAA communication. If you are not the intended
recipient, any use, distribution, printing, or copying of this email is
strictly prohibited. If you received this email in error, please
immediately delete it from your system and notify the originator. Your
cooperation is appreciated.

RE: [JENKINS-MAVEN] Lucene-Solr-Maven-4.x #64: POMs out of sync

On August 12th, roughly the same day BasicDistributedZkTest#testDistribSearch 
started failing regularly under the Maven builds, I reduced Solr log output 
under the Maven builds, because Jenkins wasn't sending any notification emails 
when the Maven builds failed, AFAICT because the voluminous Solr logs were 
triggering OOMs in Jenkins JVMs.  (I don't know of a way for 
maven-surefire-plugin to emulate randomizedtesting's 
print-stderr-and-stdout-only-on-failure behavior, so *all* Solr log output was 
ending up in the Jenkins build logs under Maven: 54MB in the last build where 
this was still enabled.)

If it helps, though, there was one build (Lucene-Solr-Maven-4.x build #56) 
under which the failure occurred while full logging was still happening:


 

In that log, the schema name is always shown as:

oass.IndexSchema.readSchema Schema name=test

I.e., the version isn't included in the schema name.  (Maybe 
IndexSchema#readSchema should also log the schema version?)

Oh, I may know what's happening: because Maven dependency model can't represent 
the dependencies among solr-core, test-framework, and solrj, the maven builds 
run the solrj and solr-core tests together, and so both modules' resources are 
copied to the same directory.  I see that solrj has a test schema.xml file with 
.

Steve

-Original Message-
From: Chris Hostetter [mailto:hossman_luc...@fucit.org] 
Sent: Tuesday, August 28, 2012 5:58 PM
To: dev@lucene.apache.org
Subject: Re: [JENKINS-MAVEN] Lucene-Solr-Maven-4.x #64: POMs out of sync


: Off of the top of my head, the only thing I can figure is that the
: maven based tests are somehow getting the wrong schema sometimes.
: Maybe if there's some different with how solr homes are set between
: ant and maven?

that should be easy to sanity check right?  add something like this into 
the @Before method...

  assertEquals("test-schema-1.0", core.getSchema().getSchemaName())

...and then double check that all of the test schema files have unique 
name attributes in their XML.


-Hoss

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org


-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Resolved] (LUCENE-4336) javacc tasks should use ivy


 [ 
https://issues.apache.org/jira/browse/LUCENE-4336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Uwe Schindler resolved LUCENE-4336.
---

   Resolution: Fixed
Fix Version/s: 4.0
   5.0

Committed trunk revision: 1378381
Committed 4.x revision: 1378382

> javacc tasks should use ivy
> ---
>
> Key: LUCENE-4336
> URL: https://issues.apache.org/jira/browse/LUCENE-4336
> Project: Lucene - Core
>  Issue Type: Task
>Reporter: Robert Muir
>Assignee: Uwe Schindler
> Fix For: 5.0, 4.0
>
> Attachments: LUCENE-4336.patch
>
>
> its a hassle to set this up currently. we should be able to
> just download javacc this way to run those tasks instead of
> making you download it yourself from the java.net site and
> setting build.properties options and stuff.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Comment Edited] (LUCENE-4336) javacc tasks should use ivy


[ 
https://issues.apache.org/jira/browse/LUCENE-4336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13443644#comment-13443644
 ] 

Uwe Schindler edited comment on LUCENE-4336 at 8/29/12 10:21 AM:
-

Here the patch. The javacc task is a little bit complicated (needed some help 
from blogs,...) because javacc.jar does not contain the ant task. The task is 
shipped with ANT and ANT expects a attribute called "javacchome", that must 
point to the home folder of an extracted javacc distribution (although it only 
uses the JAR).  looks inside this dir for bin/lib and inside this dir 
for a file named javacc.jar (without version).
So we cannot use ivy:cachepath and ivy:retrieve is too unflexible (it allows to 
setup directory layout, but filename always contains version).
The trick here is a "handwritten" resolve/rewrite:
- get ivy:cachefileset for javacc-5.0.jar artifact
- create a "fake" release folder in ${build.dir} and then copy the cachefileset 
into it with a .
- the javacc.home dir is then located in ${build.dir} and nuked on ant clean.

This patch also cleans up javacc at all. We only use javacc in the queryparser 
module so I moved all ant logic there. I also removed some unused tasks and 
properties.

I will commit soon.

  was (Author: thetaphi):
Here the patch. The javacc task is a little bit complicated (needed some 
help from blogs,...) because javacc.jar does not contain the ant task. The task 
is shipped with ANT and ANT expects a attribute called "javacchome", that must 
point to the folder of an extracte javacc distribution (although it only uses 
the JAR).  looks inside this dir for bin/lib and inside this dir for a 
file named javacc.jar (without version).
So we cannot use cachepath and ivy:retrieve is too unflexible (it allows to 
setup directory layout, but filename always contains version).
The trick here is a "handwritten" resolve/rewrite:
- get ivy:cachefileset for javacc-5.0.jar
- create a "fake" release folder in ${build.dir} and then copy the cachefileset 
into it with the mergemapper (to javacc.jar).

This patch also cleans up javacc at all. We only use javacc in the queryparser 
module so I moved all ant logic there. I also removed some unused tasks and 
properties.

I will commit soon.
  
> javacc tasks should use ivy
> ---
>
> Key: LUCENE-4336
> URL: https://issues.apache.org/jira/browse/LUCENE-4336
> Project: Lucene - Core
>  Issue Type: Task
>Reporter: Robert Muir
>Assignee: Uwe Schindler
> Attachments: LUCENE-4336.patch
>
>
> its a hassle to set this up currently. we should be able to
> just download javacc this way to run those tasks instead of
> making you download it yourself from the java.net site and
> setting build.properties options and stuff.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (LUCENE-4336) javacc tasks should use ivy


 [ 
https://issues.apache.org/jira/browse/LUCENE-4336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Uwe Schindler updated LUCENE-4336:
--

Attachment: LUCENE-4336.patch

Here the patch. The javacc task is a little bit complicated (needed some help 
from blogs,...) because javacc.jar does not contain the ant task. The task is 
shipped with ANT and ANT expects a attribute called "javacchome", that must 
point to the folder of an extracte javacc distribution (although it only uses 
the JAR).  looks inside this dir for bin/lib and inside this dir for a 
file named javacc.jar (without version).
So we cannot use cachepath and ivy:retrieve is too unflexible (it allows to 
setup directory layout, but filename always contains version).
The trick here is a "handwritten" resolve/rewrite:
- get ivy:cachefileset for javacc-5.0.jar
- create a "fake" release folder in ${build.dir} and then copy the cachefileset 
into it with the mergemapper (to javacc.jar).

This patch also cleans up javacc at all. We only use javacc in the queryparser 
module so I moved all ant logic there. I also removed some unused tasks and 
properties.

I will commit soon.

> javacc tasks should use ivy
> ---
>
> Key: LUCENE-4336
> URL: https://issues.apache.org/jira/browse/LUCENE-4336
> Project: Lucene - Core
>  Issue Type: Task
>Reporter: Robert Muir
>Assignee: Uwe Schindler
> Attachments: LUCENE-4336.patch
>
>
> its a hassle to set this up currently. we should be able to
> just download javacc this way to run those tasks instead of
> making you download it yourself from the java.net site and
> setting build.properties options and stuff.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Assigned] (LUCENE-4336) javacc tasks should use ivy


 [ 
https://issues.apache.org/jira/browse/LUCENE-4336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Uwe Schindler reassigned LUCENE-4336:
-

Assignee: Uwe Schindler

> javacc tasks should use ivy
> ---
>
> Key: LUCENE-4336
> URL: https://issues.apache.org/jira/browse/LUCENE-4336
> Project: Lucene - Core
>  Issue Type: Task
>Reporter: Robert Muir
>Assignee: Uwe Schindler
>
> its a hassle to set this up currently. we should be able to
> just download javacc this way to run those tasks instead of
> making you download it yourself from the java.net site and
> setting build.properties options and stuff.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: [JENKINS-MAVEN] Lucene-Solr-Maven-4.x #64: POMs out of sync

2012-08-28 Thread Chris Hostetter


: Off of the top of my head, the only thing I can figure is that the
: maven based tests are somehow getting the wrong schema sometimes.
: Maybe if there's some different with how solr homes are set between
: ant and maven?

that should be easy to sanity check right?  add something like this into 
the @Before method...

  assertEquals("test-schema-1.0", core.getSchema().getSchemaName())

...and then double check that all of the test schema files have unique 
name attributes in their XML.


-Hoss

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: [JENKINS-MAVEN] Lucene-Solr-Maven-4.x #64: POMs out of sync

On Fri, Aug 17, 2012 at 9:42 AM, Steven A Rowe  wrote:
> Five out of the last seven builds have failed with this exact same ERROR.
>
> I can't reproduce on my Win7+Cygwin environment.
>
> Does anybody know what's happening here?
>
> If not, I'll ignore this test under Maven.

Pretty strange.  I can't reproduce locally.

"multiple values encountered for non multiValued field val_i: [10, 20]"

This should be very deterministic (i.e. it should always fail if it
were actually a non multiValued field).
The *_i fields are multivalued according to schema.xml, so this
exception should not happen (the version=1.0 in schema.xml means
multiValued=true by default).

Off of the top of my head, the only thing I can figure is that the
maven based tests are somehow getting the wrong schema sometimes.
Maybe if there's some different with how solr homes are set between
ant and maven?

-Yonik
http://lucidworks.com

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: large messages from Jenkins failures

>  1) slap a special "@UnstableTest" annotation on them
>  2) set up a new jenkins job that *only* runs these @UnstableTest jobs
>  3) configure this new jenkins job to not send any email
>
> ...seems like that would satisfy everyone right?

I'm all for it. We can rename @BadApple to @Unstable and make it
disabled by default. As for (2) this will be tricky because there's no
way to run just a specific group. I like this idea as a feature though
so if there's no vetos I'll add it to the runner.

Dawid

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: large messages from Jenkins failures

2012-08-28 Thread Chris Hostetter


: I didn't say that. I said the opposite - that having imperfect tests
: (or rather tests that cannot be fixed for whatever reason) discourages
: from looking at test failures and makes one just unsubscribe from the
: jenkins mails. If this is the case then yes, I think not having a test
: like that at all is better than having it.

As i've said before...

Running these problematic tests in jenkins on machines like 
builds.apache.org is still very helpful because in many cases folks are 
unable to reproduce the failures anywhere else (or in some cases: some 
people can reproduce them, but not the people who have the 
knowledge/energy to fix them)

If folks are concerned that certian tests fail to frequently to be
considered "stable" and included in the main build, then let's:

 1) slap a special "@UnstableTest" annotation on them
 2) set up a new jenkins job that *only* runs these @UnstableTest jobs
 3) configure this new jenkins job to not send any email

...seems like that would satisfy everyone right?

-Hoss

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-4335) Builds should regenerate all generated sources

2012-08-28 Thread Steven Rowe (JIRA)


[ 
https://issues.apache.org/jira/browse/LUCENE-4335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13443560#comment-13443560
 ] 

Steven Rowe commented on LUCENE-4335:
-

I'm not sure about Javacc, but I've seen JFlex reorder cases in switch 
statements, even when there are no .jflex source changes, when run under 
different JVM versions.  I recall seeing this specifically when generating 
under Java5 and Java6, both Oracle JVMs on Windows.

I'll look into the generator to see how to fix the output order.

> Builds should regenerate all generated sources
> --
>
> Key: LUCENE-4335
> URL: https://issues.apache.org/jira/browse/LUCENE-4335
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Michael McCandless
>
> We have more and more sources that are generated programmatically (query 
> parsers, fuzzy levN tables from Moman, packed ints specialized decoders, 
> etc.), and it's dangerous because developers may directly edit the generated 
> sources and forget to edit the meta-source.  It's happened to me several 
> times ... most recently just after landing the BlockPostingsFormat branch.
> I think we should re-gen all of these in our builds and fail the build if 
> this creates a difference.  I know some generators (eg JavaCC) embed 
> timestamps and so always create mods ... we can leave them out of this for 
> starters (or maybe post-process the sources to remove the timestamps) ...

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Resolved] (LUCENE-4338) Redirect java.io.tmpdir so that each JVM gets their own tmpdir under a build/test


 [ 
https://issues.apache.org/jira/browse/LUCENE-4338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dawid Weiss resolved LUCENE-4338.
-

   Resolution: Fixed
Fix Version/s: 4.0
   5.0

> Redirect java.io.tmpdir so that each JVM gets their own tmpdir under a 
> build/test
> -
>
> Key: LUCENE-4338
> URL: https://issues.apache.org/jira/browse/LUCENE-4338
> Project: Lucene - Core
>  Issue Type: Task
>Reporter: Dawid Weiss
>Assignee: Dawid Weiss
> Fix For: 5.0, 4.0
>
> Attachments: LUCENE-4338.patch
>
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-4338) Redirect java.io.tmpdir so that each JVM gets their own tmpdir under a build/test


[ 
https://issues.apache.org/jira/browse/LUCENE-4338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13443536#comment-13443536
 ] 

Dawid Weiss commented on LUCENE-4338:
-

Simple but does the job. I'm running tests right now to see if nothing breaks, 
will commit soon.

> Redirect java.io.tmpdir so that each JVM gets their own tmpdir under a 
> build/test
> -
>
> Key: LUCENE-4338
> URL: https://issues.apache.org/jira/browse/LUCENE-4338
> Project: Lucene - Core
>  Issue Type: Task
>Reporter: Dawid Weiss
>Assignee: Dawid Weiss
> Attachments: LUCENE-4338.patch
>
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-4338) Redirect java.io.tmpdir so that each JVM gets their own tmpdir under a build/test


[ 
https://issues.apache.org/jira/browse/LUCENE-4338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13443534#comment-13443534
 ] 

Robert Muir commented on LUCENE-4338:
-

I don't know why I thought it would be trickier than that :) commit it!

> Redirect java.io.tmpdir so that each JVM gets their own tmpdir under a 
> build/test
> -
>
> Key: LUCENE-4338
> URL: https://issues.apache.org/jira/browse/LUCENE-4338
> Project: Lucene - Core
>  Issue Type: Task
>Reporter: Dawid Weiss
>Assignee: Dawid Weiss
> Attachments: LUCENE-4338.patch
>
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (LUCENE-4338) Redirect java.io.tmpdir so that each JVM gets their own tmpdir under a build/test


 [ 
https://issues.apache.org/jira/browse/LUCENE-4338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dawid Weiss updated LUCENE-4338:


Attachment: LUCENE-4338.patch

> Redirect java.io.tmpdir so that each JVM gets their own tmpdir under a 
> build/test
> -
>
> Key: LUCENE-4338
> URL: https://issues.apache.org/jira/browse/LUCENE-4338
> Project: Lucene - Core
>  Issue Type: Task
>Reporter: Dawid Weiss
>Assignee: Dawid Weiss
> Attachments: LUCENE-4338.patch
>
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Created] (LUCENE-4338) Redirect java.io.tmpdir so that each JVM gets their own tmpdir under a build/test

Dawid Weiss created LUCENE-4338:
---

 Summary: Redirect java.io.tmpdir so that each JVM gets their own 
tmpdir under a build/test
 Key: LUCENE-4338
 URL: https://issues.apache.org/jira/browse/LUCENE-4338
 Project: Lucene - Core
  Issue Type: Task
Reporter: Dawid Weiss
Assignee: Dawid Weiss




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-4332) Integrate PiTest mutation coverage tool into build


[ 
https://issues.apache.org/jira/browse/LUCENE-4332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13443520#comment-13443520
 ] 

Dawid Weiss commented on LUCENE-4332:
-

Sure, all I'm saying is that I know very little about SMs so if something 
breaks I'll just stare at you blindly ;)

> Integrate PiTest mutation coverage tool into build
> --
>
> Key: LUCENE-4332
> URL: https://issues.apache.org/jira/browse/LUCENE-4332
> Project: Lucene - Core
>  Issue Type: Improvement
>Affects Versions: 4.1, 5.0
>Reporter: Greg Bowyer
>Assignee: Greg Bowyer
>  Labels: build
> Attachments: 
> LUCENE-4332-Integrate-PiTest-mutation-coverage-tool-into-build.patch, 
> LUCENE-4332-Integrate-PiTest-mutation-coverage-tool-into-build.patch
>
>
> As discussed briefly on the mailing list, this patch is an attempt to 
> integrate the PiTest mutation coverage tool into the lucene build

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-4332) Integrate PiTest mutation coverage tool into build


[ 
https://issues.apache.org/jira/browse/LUCENE-4332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13443513#comment-13443513
 ] 

Greg Bowyer commented on LUCENE-4332:
-

I can codify a security manager, they are somewhat complex but I see our needs 
here as very simple (essentially assert file paths)

> Integrate PiTest mutation coverage tool into build
> --
>
> Key: LUCENE-4332
> URL: https://issues.apache.org/jira/browse/LUCENE-4332
> Project: Lucene - Core
>  Issue Type: Improvement
>Affects Versions: 4.1, 5.0
>Reporter: Greg Bowyer
>Assignee: Greg Bowyer
>  Labels: build
> Attachments: 
> LUCENE-4332-Integrate-PiTest-mutation-coverage-tool-into-build.patch, 
> LUCENE-4332-Integrate-PiTest-mutation-coverage-tool-into-build.patch
>
>
> As discussed briefly on the mailing list, this patch is an attempt to 
> integrate the PiTest mutation coverage tool into the lucene build

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-4332) Integrate PiTest mutation coverage tool into build


[ 
https://issues.apache.org/jira/browse/LUCENE-4332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13443512#comment-13443512
 ] 

Uwe Schindler commented on LUCENE-4332:
---

You would need a simple Security manager that allows all, except 
creating/deleting some files outside a working directory. We can check this out 
(by enabling a noop security manager), if it then still passes we are fine.

Then we can go forward. The security manager would only be activated by LTC 
before after a test class and then disabled again. I would not restrict file 
access too much. For Pitest, I would only disallow everything outside working 
directory root and later disable more. I think a simple restriction to build/ 
dir would also help to prevent solr creating files in test-files src folder.

An all-allowing security manager is easy, template is available to extend. The 
problem you are talking about are too complex security restrictions dictated by 
J2EE (that limit things like creating classes or classloader) - we dont want to 
do this, we only want a hook into (new FileOutputStream) and want to throw an 
Exception on wrong path. If you allow all other security manager requests, 
there is no issue.

> Integrate PiTest mutation coverage tool into build
> --
>
> Key: LUCENE-4332
> URL: https://issues.apache.org/jira/browse/LUCENE-4332
> Project: Lucene - Core
>  Issue Type: Improvement
>Affects Versions: 4.1, 5.0
>Reporter: Greg Bowyer
>Assignee: Greg Bowyer
>  Labels: build
> Attachments: 
> LUCENE-4332-Integrate-PiTest-mutation-coverage-tool-into-build.patch, 
> LUCENE-4332-Integrate-PiTest-mutation-coverage-tool-into-build.patch
>
>
> As discussed briefly on the mailing list, this patch is an attempt to 
> integrate the PiTest mutation coverage tool into the lucene build

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-4332) Integrate PiTest mutation coverage tool into build


[ 
https://issues.apache.org/jira/browse/LUCENE-4332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13443509#comment-13443509
 ] 

Robert Muir commented on LUCENE-4332:
-

I agree with you, i dont know anything about security managers either. But it 
seems like we could use such a thing to find test bugs.

Of course our security manager would have vulnerabilities (possibly introduced 
intentionally in case we need to backdoor it). But this
is more like locking your front door so it won't blow open when its windy 
outside.

> Integrate PiTest mutation coverage tool into build
> --
>
> Key: LUCENE-4332
> URL: https://issues.apache.org/jira/browse/LUCENE-4332
> Project: Lucene - Core
>  Issue Type: Improvement
>Affects Versions: 4.1, 5.0
>Reporter: Greg Bowyer
>Assignee: Greg Bowyer
>  Labels: build
> Attachments: 
> LUCENE-4332-Integrate-PiTest-mutation-coverage-tool-into-build.patch, 
> LUCENE-4332-Integrate-PiTest-mutation-coverage-tool-into-build.patch
>
>
> As discussed briefly on the mailing list, this patch is an attempt to 
> integrate the PiTest mutation coverage tool into the lucene build

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: large messages from Jenkins failures

> Imperfect test coverage is better than no test coverage?
> Seems like we could simply disable all of our tests and then be happy
> because they will never fail ;-)

I didn't say that. I said the opposite - that having imperfect tests
(or rather tests that cannot be fixed for whatever reason) discourages
from looking at test failures and makes one just unsubscribe from the
jenkins mails. If this is the case then yes, I think not having a test
like that at all is better than having it.

> Some of these tests fail because of threads left over that are hard to
> control - we have a lot more moving parts like jetty and zookeeper.

I understand that but these tests have been failing long before those
checks were added. I also understand the complexity involved -- like I
said, I also tried to fix those tests and failed.

> timely manner, it seems like the most logical thing to do is relax the
> checks - that maximises test coverage.

These thread leak checks are meant to isolate test suites from each
other and I think they do a good job at it.

> It is bad, but disabling seems even worse, unless we're just not
> worried about test code coverage at all.

We have different viewpoints on this, sorry.

Dawid

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: large messages from Jenkins failures

On Tue, Aug 28, 2012 at 4:03 PM, Michael McCandless
 wrote:
>> Another option is to redirect solr fails to a different mailing list
>> that only those that care about solr development can follow.
>
> I don't think splintering the dev community is healthy.

Well, it seems like some people would prefer tests that fail sometimes
to be disabled so they don't see the failure messages.
Others (like me) find those tests to be extremely valuable since they
represent coverage for key features.  How else to resolve that?

"Just fix the test" isn't an answer... unless one is personally
committing the time to do it themselves.

-Yonik
http://lucidworks.com

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-4332) Integrate PiTest mutation coverage tool into build


[ 
https://issues.apache.org/jira/browse/LUCENE-4332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13443501#comment-13443501
 ] 

Dawid Weiss commented on LUCENE-4332:
-

bq. Well if its for test purposes only and not enforcing actual security, we 
should be able to give the runner a nice backdoor (e.g. static boolean 
BACKDOORED) if we really need to right?

I don't know what it will do and how it will fail. I never run with a security 
manager, I think the design of SM is complex, weird and poorly documented (as 
evidenced by the security vuln. published today...). I just don't think of it 
much because I know very few people that actually run with a SM enabled. I am 
fairly sure I do many things in the code that I know won't work with an 
installed SM -- scanning properties, opening files, etc. I'm sure it is 
possible to configure it do grant permissions to certain packages but I've no 
idea how to do it.

I'm not saying no to the idea, but I'm just letting you know that things may 
break in unexpected ways and I will have limited time to learn SM internals... 
Not that I'm especially looking forward to that. :)



> Integrate PiTest mutation coverage tool into build
> --
>
> Key: LUCENE-4332
> URL: https://issues.apache.org/jira/browse/LUCENE-4332
> Project: Lucene - Core
>  Issue Type: Improvement
>Affects Versions: 4.1, 5.0
>Reporter: Greg Bowyer
>Assignee: Greg Bowyer
>  Labels: build
> Attachments: 
> LUCENE-4332-Integrate-PiTest-mutation-coverage-tool-into-build.patch, 
> LUCENE-4332-Integrate-PiTest-mutation-coverage-tool-into-build.patch
>
>
> As discussed briefly on the mailing list, this patch is an attempt to 
> integrate the PiTest mutation coverage tool into the lucene build

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-4332) Integrate PiTest mutation coverage tool into build


[ 
https://issues.apache.org/jira/browse/LUCENE-4332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13443497#comment-13443497
 ] 

Dawid Weiss commented on LUCENE-4332:
-

As for generating a common seed for all runs -- if you slurp junit4 taskdefs at 
the main level somewhere then you can do:
{code}

 
{code}

and as long as this property is passed to subants it will remain the same. But 
you can just as well just generate it, even from the shell level:
{code}
openssl rand -hex 8
{code}

> Integrate PiTest mutation coverage tool into build
> --
>
> Key: LUCENE-4332
> URL: https://issues.apache.org/jira/browse/LUCENE-4332
> Project: Lucene - Core
>  Issue Type: Improvement
>Affects Versions: 4.1, 5.0
>Reporter: Greg Bowyer
>Assignee: Greg Bowyer
>  Labels: build
> Attachments: 
> LUCENE-4332-Integrate-PiTest-mutation-coverage-tool-into-build.patch, 
> LUCENE-4332-Integrate-PiTest-mutation-coverage-tool-into-build.patch
>
>
> As discussed briefly on the mailing list, this patch is an attempt to 
> integrate the PiTest mutation coverage tool into the lucene build

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-4332) Integrate PiTest mutation coverage tool into build


[ 
https://issues.apache.org/jira/browse/LUCENE-4332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13443491#comment-13443491
 ] 

Robert Muir commented on LUCENE-4332:
-

{quote}
I didn't cater for the presence of a security manager in the runner and it will 
probably break things in the runner that will be tough to debug. Just a fair 
warning. You will probably have to give the runner a policy of being able to do 
everything and it still may fail to run.
{quote}

Well if its for test purposes only and not enforcing actual security, we should 
be able to give the runner a nice backdoor (e.g. static boolean BACKDOORED) if
we really need to right?


> Integrate PiTest mutation coverage tool into build
> --
>
> Key: LUCENE-4332
> URL: https://issues.apache.org/jira/browse/LUCENE-4332
> Project: Lucene - Core
>  Issue Type: Improvement
>Affects Versions: 4.1, 5.0
>Reporter: Greg Bowyer
>Assignee: Greg Bowyer
>  Labels: build
> Attachments: 
> LUCENE-4332-Integrate-PiTest-mutation-coverage-tool-into-build.patch, 
> LUCENE-4332-Integrate-PiTest-mutation-coverage-tool-into-build.patch
>
>
> As discussed briefly on the mailing list, this patch is an attempt to 
> integrate the PiTest mutation coverage tool into the lucene build

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-4332) Integrate PiTest mutation coverage tool into build


[ 
https://issues.apache.org/jira/browse/LUCENE-4332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13443489#comment-13443489
 ] 

Dawid Weiss commented on LUCENE-4332:
-

bq. OK, cool: its something we never did before that I always thought should be 
done...

Yes, I checked. Don't let this confuse you:
{code}
http://labs.carrotsearch.com/download/randomizedtesting/2.0.0-SNAPSHOT/docs/junit4-ant/Tasks/junit4.html

I looked at the common-build.xml file though and I see only this:
{code}

 
{code}

I was wrong then, the default 'java.io.tmpdir' is not overriden here and I 
think it should. I wrote a small test asking for a File.createTempFile and it 
did use a global temp (not good).


> Integrate PiTest mutation coverage tool into build
> --
>
> Key: LUCENE-4332
> URL: https://issues.apache.org/jira/browse/LUCENE-4332
> Project: Lucene - Core
>  Issue Type: Improvement
>Affects Versions: 4.1, 5.0
>Reporter: Greg Bowyer
>Assignee: Greg Bowyer
>  Labels: build
> Attachments: 
> LUCENE-4332-Integrate-PiTest-mutation-coverage-tool-into-build.patch, 
> LUCENE-4332-Integrate-PiTest-mutation-coverage-tool-into-build.patch
>
>
> As discussed briefly on the mailing list, this patch is an attempt to 
> integrate the PiTest mutation coverage tool into the lucene build

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Created] (LUCENE-4337) Create Java security manager for forcible asserting behaviours in testing

Greg Bowyer created LUCENE-4337:
---

 Summary: Create Java security manager for forcible asserting 
behaviours in testing
 Key: LUCENE-4337
 URL: https://issues.apache.org/jira/browse/LUCENE-4337
 Project: Lucene - Core
  Issue Type: Bug
Affects Versions: 4.1, 5.0, 4.0
Reporter: Greg Bowyer
Assignee: Greg Bowyer


Following on from conversations about mutation testing, there is an interest in 
building a Java security manager that is able to assert / guarantee certain 
behaviours 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-4332) Integrate PiTest mutation coverage tool into build


[ 
https://issues.apache.org/jira/browse/LUCENE-4332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13443477#comment-13443477
 ] 

Robert Muir commented on LUCENE-4332:
-

{quote}
They already do I think because tmpdir property is overridden with "." and cwd 
is set to J0/J1/J2/JN under the test dir.
{quote}

OK, cool: its something we never did before that I always thought should be 
done... 

> Integrate PiTest mutation coverage tool into build
> --
>
> Key: LUCENE-4332
> URL: https://issues.apache.org/jira/browse/LUCENE-4332
> Project: Lucene - Core
>  Issue Type: Improvement
>Affects Versions: 4.1, 5.0
>Reporter: Greg Bowyer
>Assignee: Greg Bowyer
>  Labels: build
> Attachments: 
> LUCENE-4332-Integrate-PiTest-mutation-coverage-tool-into-build.patch, 
> LUCENE-4332-Integrate-PiTest-mutation-coverage-tool-into-build.patch
>
>
> As discussed briefly on the mailing list, this patch is an attempt to 
> integrate the PiTest mutation coverage tool into the lucene build

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: large messages from Jenkins failures

On Tue, Aug 28, 2012 at 3:57 PM, Dawid Weiss
 wrote:
> I don't agree with you here. I think having two or three failures
> daily from the same test (and typically with the same message) is far
> worse than not having it at all.

Imperfect test coverage is better than no test coverage?
Seems like we could simply disable all of our tests and then be happy
because they will never fail ;-)

Some of these tests fail because of threads left over that are hard to
control - we have a lot more moving parts like jetty and zookeeper.
Some tests started failing more often because of more stringent checks
(like threads left over after a test).  If these can't be fixed in a
timely manner, it seems like the most logical thing to do is relax the
checks - that maximises test coverage.

> You get used to having failing tests
> and this is bad. A test failure should be a red flag, something you
> eagerly look into because you're curious about what happened. I
> stopped having that feeling after a while, this seems bad to me.

It is bad, but disabling seems even worse, unless we're just not
worried about test code coverage at all.

-Yonik
http://lucidworks.com

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-4332) Integrate PiTest mutation coverage tool into build


[ 
https://issues.apache.org/jira/browse/LUCENE-4332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13443471#comment-13443471
 ] 

Dawid Weiss commented on LUCENE-4332:
-

bq. toFacyDawidWeissSeedFormat

That's just a hex representation of eight bytes (unsigned long), nothing fancy.

bq. We should install a SecurityManager always and only allow files in 
build/test

I didn't cater for the presence of a security manager in the runner and it will 
probably break things in the runner that will be tough to debug. Just a fair 
warning. You will probably have to give the runner a policy of being able to do 
everything and it still may fail to run.

bq. also we should ask Dawid if each forked jvm can get its own sandbox'ed 
java.io.tmpdir in build/test:

They already do I think because tmpdir property is overridden with "." and cwd 
is set to J0/J1/J2/JN under the test dir.

> Integrate PiTest mutation coverage tool into build
> --
>
> Key: LUCENE-4332
> URL: https://issues.apache.org/jira/browse/LUCENE-4332
> Project: Lucene - Core
>  Issue Type: Improvement
>Affects Versions: 4.1, 5.0
>Reporter: Greg Bowyer
>Assignee: Greg Bowyer
>  Labels: build
> Attachments: 
> LUCENE-4332-Integrate-PiTest-mutation-coverage-tool-into-build.patch, 
> LUCENE-4332-Integrate-PiTest-mutation-coverage-tool-into-build.patch
>
>
> As discussed briefly on the mailing list, this patch is an attempt to 
> integrate the PiTest mutation coverage tool into the lucene build

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: large messages from Jenkins failures

2012-08-28 Thread Michael McCandless

On Tue, Aug 28, 2012 at 3:04 PM, Yonik Seeley  wrote:
> On Tue, Aug 28, 2012 at 2:43 PM, Dawid Weiss
>  wrote:
>> 2) I think Solr emits a LOT of logging information to the console. I
>> don't know if all of it is really useful -- I doubt it, really.
>>
>> The solutions I see are simple -- disable the tests that fail 3-5
>> times and we still don't know what causes the problem. Disable them
>> and file a JIRA issue.
>
> Another option is to redirect solr fails to a different mailing list
> that only those that care about solr development can follow.

I don't think splintering the dev community is healthy.

What I really want is for the tests (or the bugs in Solr/Lucene
causing the test failures) to be fixed, for a Solr dev who understands
the test to dig into it.

> Tests that fail a small percent of the time are still hugely valuable
> (i.e. when they fail for a different reason than usual, or they start
> failing much more often).  Simply disabling them is far worse for the
> project.

I  agree, for tests that don't fail frequently.  This is the
power/purpose of having a test.

The problem is certain Solr tests fail very frequently and nobody
jumps on those failures / we become complacent: such failures quickly
stop being helpful.  I know Mark has jumped on some of the test
failures (thank you!), but he's only one person and we still have
certain Solr tests failing frequently.

This really reflects a deeper problem: Solr doesn't have enough dev
coverage, or devs that have time/itch/energy to dig into hard test
failures.  When a test fails devs should be eager to fix it.  That's
the polar opposite of Solr's failures today.

Mike McCandless

http://blog.mikemccandless.com

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-4336) javacc tasks should use ivy


[ 
https://issues.apache.org/jira/browse/LUCENE-4336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13443465#comment-13443465
 ] 

Robert Muir commented on LUCENE-4336:
-

The javacc 5.0 is in maven repo (I upgraded to it locally because it fixes some 
bugs in generated javadocs code).

So it seems possible.

Jflex is harder, i think 1.5 is still unreleased? But we can do just javacc for 
now: its an improvement.

> javacc tasks should use ivy
> ---
>
> Key: LUCENE-4336
> URL: https://issues.apache.org/jira/browse/LUCENE-4336
> Project: Lucene - Core
>  Issue Type: Task
>Reporter: Robert Muir
>
> its a hassle to set this up currently. we should be able to
> just download javacc this way to run those tasks instead of
> making you download it yourself from the java.net site and
> setting build.properties options and stuff.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-4335) Builds should regenerate all generated sources


[ 
https://issues.apache.org/jira/browse/LUCENE-4335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13443464#comment-13443464
 ] 

Robert Muir commented on LUCENE-4335:
-

{quote}
We should check what causes this.
{quote}

I agree, this is always scary when it happens. It makes it harder to tell if 
something really changed.

> Builds should regenerate all generated sources
> --
>
> Key: LUCENE-4335
> URL: https://issues.apache.org/jira/browse/LUCENE-4335
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Michael McCandless
>
> We have more and more sources that are generated programmatically (query 
> parsers, fuzzy levN tables from Moman, packed ints specialized decoders, 
> etc.), and it's dangerous because developers may directly edit the generated 
> sources and forget to edit the meta-source.  It's happened to me several 
> times ... most recently just after landing the BlockPostingsFormat branch.
> I think we should re-gen all of these in our builds and fail the build if 
> this creates a difference.  I know some generators (eg JavaCC) embed 
> timestamps and so always create mods ... we can leave them out of this for 
> starters (or maybe post-process the sources to remove the timestamps) ...

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: large messages from Jenkins failures

> Another option is to redirect solr fails to a different mailing list
> that only those that care about solr development can follow.

I don't make a distinction between solr and lucene development, call me odd.
I did try to help with those few tests (and I fixed some others) but no luck.

> Tests that fail a small percent of the time are still hugely valuable
> (i.e. when they fail for a different reason than usual, or they start
> failing much more often).  Simply disabling them is far worse for the
> project.

I don't agree with you here. I think having two or three failures
daily from the same test (and typically with the same message) is far
worse than not having it at all. You get used to having failing tests
and this is bad. A test failure should be a red flag, something you
eagerly look into because you're curious about what happened. I
stopped having that feeling after a while, this seems bad to me.

Dawid

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

RE: large messages from Jenkins failures

Actually, after discussing with Uwe on #lucene-dev IRC, I'm looking into 
another mechanism to reduce the size of email messages: the Jenkins Email-Ext 
plugin has a per-build-job configuration item named "Pre-send script" that 
allows you to modify the MimeMessage object representing an email via a Groovy 
script.  Here's what I've got so far - I'm going to enable this now on all the 
jobs on Uwe's Jenkins (the "msg" variable, of type MimeMessage, is made 
available by the plugin to the script):

maxLength = 20;
trailingLength = 1;
content = msg.getContent(); // assumption: mime type is "text/plain"
contentLength = content.length();
if (content.length() > maxLength) {
  text = content.substring(0, maxLength - trailingLength) 
   + "\n\n[... truncated too long message ...]\n\n"
   + content.substring(contentLength - trailingLength);
  msg.setText(text, "UTF-8");
}

Steve

-Original Message-
From: ysee...@gmail.com [mailto:ysee...@gmail.com] On Behalf Of Yonik Seeley
Sent: Tuesday, August 28, 2012 3:11 PM
To: dev@lucene.apache.org
Subject: Re: large messages from Jenkins failures

On Mon, Aug 20, 2012 at 2:22 PM, Dawid Weiss
 wrote:
> Oh, one more thing -- if we suppress the console output we would
> absolutely have to keep (at jenkins) multiple tests-report.txt files
> because these always contain full output dumps (regardless of console
> settings). Otherwise we'd suppress potentially important info.

+1 to not forward truckloads of info to the mailing lists, as long as
we can easily get at it via jenkins or some other mechanism.

-Yonik
http://lucidworks.com

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org


-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Comment Edited] (SOLR-3755) shard splitting

2012-08-28 Thread Yonik Seeley (JIRA)


[ 
https://issues.apache.org/jira/browse/SOLR-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13440698#comment-13440698
 ] 

Yonik Seeley edited comment on SOLR-3755 at 8/29/12 6:51 AM:
-

We need to associate hash ranges with shards and allow overlapping shards (i.e. 
1-10, 1-5,6-10)

General Strategy for splitting w/ no service interruptions:
 - Bring up 2 new cores on the same node, covering the new hash ranges
 - Both cores should go into recovery mode (i.e. leader should start
forwarding updates)
   - leaders either need to consider these new smaller shards as replicas, or 
they need to forward to the "leader" for the new smaller shard
   - searches should no longer go across all shards, but should just span the 
complete hash range
 - leader does a hard commit and splits the index
 - Smaller indexes are installed on the new cores
 - Overseer should create new replicas for new shards
 - Mark old shard as “retired” – some mechanism to shut it down (after there is 
an acceptable amount of coverage of the new shards via replicas)

Future: allow splitting even with “custom” shards

  was (Author: ysee...@gmail.com):
We need to associate hash ranges with shards and allow overlapping shards 
(i.e. 1-10, 1-5,6-10)

General Strategy for splitting w/ no service interruptions:
 - Bring up 2 new cores on the same node, covering the new hash ranges
 - Both cores should go into recovery mode (i.e. leader should start
forwarding updates)
 - leader does a hard commit and splits the index
 - Smaller indexes are installed on the new cores
 - Overseer should create new replicas for new shards
 - Mark old shard as “retired” – some mechanism to shut it down (after there is 
an acceptable amount of coverage of the new shards via replicas)

Future: allow splitting even with “custom” shards
  
> shard splitting
> ---
>
> Key: SOLR-3755
> URL: https://issues.apache.org/jira/browse/SOLR-3755
> Project: Solr
>  Issue Type: New Feature
>  Components: SolrCloud
>Reporter: Yonik Seeley
>
> We can currently easily add replicas to handle increases in query volume, but 
> we should also add a way to add additional shards dynamically by splitting 
> existing shards.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-4335) Builds should regenerate all generated sources


[ 
https://issues.apache.org/jira/browse/LUCENE-4335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13443455#comment-13443455
 ] 

Uwe Schindler commented on LUCENE-4335:
---

Thats a good idea, there is one problem with one of the tools, not sure if 
jflex or javacc. It happens that one of these tools reorders the switch 
statement's "case XX:" labels and so creating different source. This seems to 
depend on JDK version used, if you regen again its the same, but often i 
changed the metafile (like fixing /** to /* for license) and regened, it was 
different order. The pattern looks like one of these tools use a 
HashSet/HashMap of "case" statements, where the order is undefined.

We should check what causes this.

bq. then we could have an 'ant regenerate' command that regens all sources, and 
our usual 'svn status' check would ensure nothing changed.

We have to extend that one to also detect modifications. The current checker 
task only looks for unversioned files and checks properties. By this you can 
run it before commit. This one would need to check for mods, too.

> Builds should regenerate all generated sources
> --
>
> Key: LUCENE-4335
> URL: https://issues.apache.org/jira/browse/LUCENE-4335
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Michael McCandless
>
> We have more and more sources that are generated programmatically (query 
> parsers, fuzzy levN tables from Moman, packed ints specialized decoders, 
> etc.), and it's dangerous because developers may directly edit the generated 
> sources and forget to edit the meta-source.  It's happened to me several 
> times ... most recently just after landing the BlockPostingsFormat branch.
> I think we should re-gen all of these in our builds and fail the build if 
> this creates a difference.  I know some generators (eg JavaCC) embed 
> timestamps and so always create mods ... we can leave them out of this for 
> starters (or maybe post-process the sources to remove the timestamps) ...

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Comment Edited] (LUCENE-4332) Integrate PiTest mutation coverage tool into build


[ 
https://issues.apache.org/jira/browse/LUCENE-4332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13443449#comment-13443449
 ] 

Greg Bowyer edited comment on LUCENE-4332 at 8/29/12 6:48 AM:
--

Following up it turns out to be *very* simple to do the security manager trick

{code:java}
import java.io.File;

public class Test {

public static void main(String... args) {
System.setSecurityManager(new SecurityManager() {
public void checkDelete(String file) throws SecurityException {
File fp = new File(file);
String path = fp.getAbsolutePath();

if (!path.startsWith("/tmp")) {
throw new SecurityException("Bang!");
}
}
});

new File("/home/greg/test").delete();
}
}
{code}

{code}
Exception in thread "main" java.lang.SecurityException: Bang!
at Test$1.checkDelete(Test.java:12)
at java.io.File.delete(File.java:971)
at Test.main(Test.java:17)
{code}

There is a lot of scope here if you want to abuse checking for all sorts of 
things (files, sockets, threads etc)

  was (Author: gbow...@fastmail.co.uk):
Following up it turns out to be *very* simple to do the security manager 
trick

{code:java}
import java.io.File;

public class Test {

public static void main(String... args) {
System.setSecurityManager(new SecurityManager() {
public void checkDelete(String file) throws SecurityException {
File fp = new File(file);
String path = fp.getAbsolutePath();

if (!path.startsWith("/tmp")) {
throw new SecurityException("Bang!");
}
}
});

new File("/home/greg/test").delete();
}
}
{code}

{code}
Exception in thread "main" java.lang.SecurityException: Bang!
at Test$1.checkDelete(Test.java:12)
at java.io.File.delete(File.java:971)
at Test.main(Test.java:17)
{code}
  
> Integrate PiTest mutation coverage tool into build
> --
>
> Key: LUCENE-4332
> URL: https://issues.apache.org/jira/browse/LUCENE-4332
> Project: Lucene - Core
>  Issue Type: Improvement
>Affects Versions: 4.1, 5.0
>Reporter: Greg Bowyer
>Assignee: Greg Bowyer
>  Labels: build
> Attachments: 
> LUCENE-4332-Integrate-PiTest-mutation-coverage-tool-into-build.patch, 
> LUCENE-4332-Integrate-PiTest-mutation-coverage-tool-into-build.patch
>
>
> As discussed briefly on the mailing list, this patch is an attempt to 
> integrate the PiTest mutation coverage tool into the lucene build

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-4332) Integrate PiTest mutation coverage tool into build


[ 
https://issues.apache.org/jira/browse/LUCENE-4332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13443449#comment-13443449
 ] 

Greg Bowyer commented on LUCENE-4332:
-

Following up it turns out to be *very* simple to do the security manager trick

{code:java}
import java.io.File;

public class Test {

public static void main(String... args) {
System.setSecurityManager(new SecurityManager() {
public void checkDelete(String file) throws SecurityException {
File fp = new File(file);
String path = fp.getAbsolutePath();

if (!path.startsWith("/tmp")) {
throw new SecurityException("Bang!");
}
}
});

new File("/home/greg/test").delete();
}
}
{code}

{code}
Exception in thread "main" java.lang.SecurityException: Bang!
at Test$1.checkDelete(Test.java:12)
at java.io.File.delete(File.java:971)
at Test.main(Test.java:17)
{code}

> Integrate PiTest mutation coverage tool into build
> --
>
> Key: LUCENE-4332
> URL: https://issues.apache.org/jira/browse/LUCENE-4332
> Project: Lucene - Core
>  Issue Type: Improvement
>Affects Versions: 4.1, 5.0
>Reporter: Greg Bowyer
>Assignee: Greg Bowyer
>  Labels: build
> Attachments: 
> LUCENE-4332-Integrate-PiTest-mutation-coverage-tool-into-build.patch, 
> LUCENE-4332-Integrate-PiTest-mutation-coverage-tool-into-build.patch
>
>
> As discussed briefly on the mailing list, this patch is an attempt to 
> integrate the PiTest mutation coverage tool into the lucene build

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-4336) javacc tasks should use ivy


[ 
https://issues.apache.org/jira/browse/LUCENE-4336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13443448#comment-13443448
 ] 

Uwe Schindler commented on LUCENE-4336:
---

+1, I can look into this! I hope this can be solved from maven repo with a 
simple taskdef using cachepath.

> javacc tasks should use ivy
> ---
>
> Key: LUCENE-4336
> URL: https://issues.apache.org/jira/browse/LUCENE-4336
> Project: Lucene - Core
>  Issue Type: Task
>Reporter: Robert Muir
>
> its a hassle to set this up currently. we should be able to
> just download javacc this way to run those tasks instead of
> making you download it yourself from the java.net site and
> setting build.properties options and stuff.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-139) Support updateable/modifiable documents

2012-08-28 Thread Dilip Maddi (JIRA)


[ 
https://issues.apache.org/jira/browse/SOLR-139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13443446#comment-13443446
 ] 

Dilip Maddi commented on SOLR-139:
--

Christopher, 
Here is how I am able to update a document by posting an XML

  
VA902B
300
  


> Support updateable/modifiable documents
> ---
>
> Key: SOLR-139
> URL: https://issues.apache.org/jira/browse/SOLR-139
> Project: Solr
>  Issue Type: New Feature
>  Components: update
>Reporter: Ryan McKinley
> Attachments: Eriks-ModifiableDocument.patch, 
> Eriks-ModifiableDocument.patch, Eriks-ModifiableDocument.patch, 
> Eriks-ModifiableDocument.patch, Eriks-ModifiableDocument.patch, 
> Eriks-ModifiableDocument.patch, getStoredFields.patch, getStoredFields.patch, 
> getStoredFields.patch, getStoredFields.patch, getStoredFields.patch, 
> SOLR-139_createIfNotExist.patch, SOLR-139-IndexDocumentCommand.patch, 
> SOLR-139-IndexDocumentCommand.patch, SOLR-139-IndexDocumentCommand.patch, 
> SOLR-139-IndexDocumentCommand.patch, SOLR-139-IndexDocumentCommand.patch, 
> SOLR-139-IndexDocumentCommand.patch, SOLR-139-IndexDocumentCommand.patch, 
> SOLR-139-IndexDocumentCommand.patch, SOLR-139-IndexDocumentCommand.patch, 
> SOLR-139-IndexDocumentCommand.patch, SOLR-139-IndexDocumentCommand.patch, 
> SOLR-139-ModifyInputDocuments.patch, SOLR-139-ModifyInputDocuments.patch, 
> SOLR-139-ModifyInputDocuments.patch, SOLR-139-ModifyInputDocuments.patch, 
> SOLR-139.patch, SOLR-139.patch, SOLR-139-XmlUpdater.patch, 
> SOLR-269+139-ModifiableDocumentUpdateProcessor.patch
>
>
> It would be nice to be able to update some fields on a document without 
> having to insert the entire document.
> Given the way lucene is structured, (for now) one can only modify stored 
> fields.
> While we are at it, we can support incrementing an existing value - I think 
> this only makes sense for numbers.
> for background, see:
> http://www.nabble.com/loading-many-documents-by-ID-tf3145666.html#a8722293

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Created] (LUCENE-4336) javacc tasks should use ivy

Robert Muir created LUCENE-4336:
---

 Summary: javacc tasks should use ivy
 Key: LUCENE-4336
 URL: https://issues.apache.org/jira/browse/LUCENE-4336
 Project: Lucene - Core
  Issue Type: Task
Reporter: Robert Muir


its a hassle to set this up currently. we should be able to
just download javacc this way to run those tasks instead of
making you download it yourself from the java.net site and
setting build.properties options and stuff.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-4332) Integrate PiTest mutation coverage tool into build


[ 
https://issues.apache.org/jira/browse/LUCENE-4332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13443430#comment-13443430
 ] 

Robert Muir commented on LUCENE-4332:
-

i think a security manager would be useful for that reason separately.

also we should ask Dawid if each forked jvm can get its own sandbox'ed 
java.io.tmpdir in build/test: this would only prevent problems
and would be nice for various external libs that might write to java.io.tmpdir 
or whatever.

> Integrate PiTest mutation coverage tool into build
> --
>
> Key: LUCENE-4332
> URL: https://issues.apache.org/jira/browse/LUCENE-4332
> Project: Lucene - Core
>  Issue Type: Improvement
>Affects Versions: 4.1, 5.0
>Reporter: Greg Bowyer
>Assignee: Greg Bowyer
>  Labels: build
> Attachments: 
> LUCENE-4332-Integrate-PiTest-mutation-coverage-tool-into-build.patch, 
> LUCENE-4332-Integrate-PiTest-mutation-coverage-tool-into-build.patch
>
>
> As discussed briefly on the mailing list, this patch is an attempt to 
> integrate the PiTest mutation coverage tool into the lucene build

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-4335) Builds should regenerate all generated sources


[ 
https://issues.apache.org/jira/browse/LUCENE-4335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13443427#comment-13443427
 ] 

Robert Muir commented on LUCENE-4335:
-

I think we should use replaceRegexp commands (like that are already there) to 
remove the various system information (time, paths, etc) that jflex/javacc/etc 
add from the generated code.

then we could have an 'ant regenerate' command that regens all sources, and our 
usual 'svn status' check would ensure nothing changed.


> Builds should regenerate all generated sources
> --
>
> Key: LUCENE-4335
> URL: https://issues.apache.org/jira/browse/LUCENE-4335
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Michael McCandless
>
> We have more and more sources that are generated programmatically (query 
> parsers, fuzzy levN tables from Moman, packed ints specialized decoders, 
> etc.), and it's dangerous because developers may directly edit the generated 
> sources and forget to edit the meta-source.  It's happened to me several 
> times ... most recently just after landing the BlockPostingsFormat branch.
> I think we should re-gen all of these in our builds and fail the build if 
> this creates a difference.  I know some generators (eg JavaCC) embed 
> timestamps and so always create mods ... we can leave them out of this for 
> starters (or maybe post-process the sources to remove the timestamps) ...

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-4332) Integrate PiTest mutation coverage tool into build


[ 
https://issues.apache.org/jira/browse/LUCENE-4332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13443424#comment-13443424
 ] 

Greg Bowyer commented on LUCENE-4332:
-

{quote}
Thats a cool idea also for our own tests! We should install a SecurityManager 
always and only allow files in build/test. LuceneTestCase can enforce this 
SecurityManager installed! And if a test writes outside, fail it!
{quote}

Should we split out that as a separate thing and get a security manager built 
that hooks into the awesome carrot testing stuffs

> Integrate PiTest mutation coverage tool into build
> --
>
> Key: LUCENE-4332
> URL: https://issues.apache.org/jira/browse/LUCENE-4332
> Project: Lucene - Core
>  Issue Type: Improvement
>Affects Versions: 4.1, 5.0
>Reporter: Greg Bowyer
>Assignee: Greg Bowyer
>  Labels: build
> Attachments: 
> LUCENE-4332-Integrate-PiTest-mutation-coverage-tool-into-build.patch, 
> LUCENE-4332-Integrate-PiTest-mutation-coverage-tool-into-build.patch
>
>
> As discussed briefly on the mailing list, this patch is an attempt to 
> integrate the PiTest mutation coverage tool into the lucene build

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Created] (LUCENE-4335) Builds should regenerate all generated sources

2012-08-28 Thread Michael McCandless (JIRA)

Michael McCandless created LUCENE-4335:
--

 Summary: Builds should regenerate all generated sources
 Key: LUCENE-4335
 URL: https://issues.apache.org/jira/browse/LUCENE-4335
 Project: Lucene - Core
  Issue Type: Improvement
Reporter: Michael McCandless


We have more and more sources that are generated programmatically (query 
parsers, fuzzy levN tables from Moman, packed ints specialized decoders, etc.), 
and it's dangerous because developers may directly edit the generated sources 
and forget to edit the meta-source.  It's happened to me several times ... most 
recently just after landing the BlockPostingsFormat branch.

I think we should re-gen all of these in our builds and fail the build if this 
creates a difference.  I know some generators (eg JavaCC) embed timestamps and 
so always create mods ... we can leave them out of this for starters (or maybe 
post-process the sources to remove the timestamps) ...

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-3763) Make solr use lucene filters directly


[ 
https://issues.apache.org/jira/browse/SOLR-3763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13443409#comment-13443409
 ] 

Greg Bowyer commented on SOLR-3763:
---

I guess my next step is to get caching working, I am not sure quite how to take 
baby steps with this beyond getting to feature parity.

> Make solr use lucene filters directly
> -
>
> Key: SOLR-3763
> URL: https://issues.apache.org/jira/browse/SOLR-3763
> Project: Solr
>  Issue Type: Improvement
>Affects Versions: 4.0, 4.1, 5.0
>Reporter: Greg Bowyer
>Assignee: Greg Bowyer
> Attachments: SOLR-3763-Make-solr-use-lucene-filters-directly.patch
>
>
> Presently solr uses bitsets, queries and collectors to implement the concept 
> of filters. This has proven to be very powerful, but does come at the cost of 
> introducing a large body of code into solr making it harder to optimise and 
> maintain.
> Another issue here is that filters currently cache sub-optimally given the 
> changes in lucene towards atomic readers.
> Rather than patch these issues, this is an attempt to rework the filters in 
> solr to leverage the Filter subsystem from lucene as much as possible.
> In good time the aim is to get this to do the following:
> ∘ Handle setting up filter implementations that are able to correctly cache 
> with reference to the AtomicReader that they are caching for rather that for 
> the entire index at large
> ∘ Get the post filters working, I am thinking that this can be done via 
> lucenes chained filter, with the ‟expensive” filters being put towards the 
> end of the chain - this has different semantics internally to the original 
> implementation but IMHO should have the same result for end users
> ∘ Learn how to create filters that are potentially more efficient, at present 
> solr basically runs a simple query that gathers a DocSet that relates to the 
> documents that we want filtered; it would be interesting to make use of 
> filter implementations that are in theory faster than query filters (for 
> instance there are filters that are able to query the FieldCache)
> ∘ Learn how to decompose filters so that a complex filter query can be cached 
> (potentially) as its constituent parts; for example the filter below 
> currently needs love, care and feeding to ensure that the filter cache is not 
> unduly stressed
> {code}
>   'category:(100) OR category:(200) OR category:(300)'
> {code}
> Really there is no reason not to express this in a cached form as 
> {code}
> BooleanFilter(
> FilterClause(CachedFilter(TermFilter(Term("category", 100))), SHOULD),
> FilterClause(CachedFilter(TermFilter(Term("category", 200))), SHOULD),
> FilterClause(CachedFilter(TermFilter(Term("category", 300))), SHOULD)
>   )
> {code}
> This would yeild better cache usage I think as we can resuse docsets across 
> multiple queries as well as avoid issues when filters are presented in 
> differing orders
> ∘ Instead of end users providing costing we might (and this is a big might 
> FWIW), be able to create a sort of execution plan of filters, leveraging a 
> combination of what the index is able to tell us as well as sampling and 
> ‟educated guesswork”; in essence this is what some DBMS software, for example 
> postgresql does - it has a genetic algo that attempts to solve the travelling 
> salesman - to great effect
> ∘ I am sure I will probably come up with other ambitious ideas to plug in 
> here . :S 
> Patches obviously forthcoming but the bulk of the work can be followed here 
> https://github.com/GregBowyer/lucene-solr/commits/solr-uses-lucene-filters

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: large messages from Jenkins failures

On Mon, Aug 20, 2012 at 2:22 PM, Dawid Weiss
 wrote:
> Oh, one more thing -- if we suppress the console output we would
> absolutely have to keep (at jenkins) multiple tests-report.txt files
> because these always contain full output dumps (regardless of console
> settings). Otherwise we'd suppress potentially important info.

+1 to not forward truckloads of info to the mailing lists, as long as
we can easily get at it via jenkins or some other mechanism.

-Yonik
http://lucidworks.com

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: large messages from Jenkins failures

On Tue, Aug 28, 2012 at 2:43 PM, Dawid Weiss
 wrote:
> 2) I think Solr emits a LOT of logging information to the console. I
> don't know if all of it is really useful -- I doubt it, really.
>
> The solutions I see are simple -- disable the tests that fail 3-5
> times and we still don't know what causes the problem. Disable them
> and file a JIRA issue.

Another option is to redirect solr fails to a different mailing list
that only those that care about solr development can follow.

Tests that fail a small percent of the time are still hugely valuable
(i.e. when they fail for a different reason than usual, or they start
failing much more often).  Simply disabling them is far worse for the
project.

-Yonik
http://lucidworks.com

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-4322) Can we make oal.util.packed.BulkOperation* smaller?


[ 
https://issues.apache.org/jira/browse/LUCENE-4322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13443388#comment-13443388
 ] 

Uwe Schindler commented on LUCENE-4322:
---

+1!! Smaller, smaller, smaller :-)

> Can we make oal.util.packed.BulkOperation* smaller?
> ---
>
> Key: LUCENE-4322
> URL: https://issues.apache.org/jira/browse/LUCENE-4322
> Project: Lucene - Core
>  Issue Type: Bug
>Reporter: Michael McCandless
> Fix For: 5.0, 4.0
>
> Attachments: LUCENE-4322.patch
>
>
> These source files add up to a lot of sources ... it caused problems when 
> compiling under Maven and InteliJ.
> I committed a change to make separates files, but in aggregate this is still 
> a lot ...
> EG maybe we don't need to specialize encode?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: large messages from Jenkins failures

> Unfortunately, -Dtests.showOutput=never penalizes all tests that don't have 
> megabytes of failure output because some do.

It doesn't penalize them, it says exactly what it does. A "full"
report is always written to disk, including all sysouts -- look at
tests-report.txt if I recall right.

> What do you think of adding an option to limit output size (e.g. 
> -Dtests.outputLimitKB=10), and truncating to that size if it's exceeded?  If 
> you think this would be reasonable, I'm willing to (try to) do the work.

I don't know... this seems like monkey patching for something that is
wrong in the first place. Here are my thoughts on this:

1) the problem is not really in big e-mails but that they're frequent
failures resulting from pretty much a fixed set of classes that we
don't know how to stabilize.

2) I think Solr emits a LOT of logging information to the console. I
don't know if all of it is really useful -- I doubt it, really.

The solutions I see are simple -- disable the tests that fail 3-5
times and we still don't know what causes the problem. Disable them
and file a JIRA issue.

An alternative is to redirect these logs on Solr tests to a file or a
circular memory buffer and only emit like a tail of N most recent
messages if we know a test failed (which is easy to do with a simple
rule).

Patching the test runner to truncate log output is doable of course
but I think it's powdering the corpse or whatever the English idiom
for that is, you get me.

Dawid

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-4322) Can we make oal.util.packed.BulkOperation* smaller?


[ 
https://issues.apache.org/jira/browse/LUCENE-4322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13443343#comment-13443343
 ] 

Dawid Weiss commented on LUCENE-4322:
-

Good idea, Andrien. More classes are definitely worse than more static methods, 
this is a normal result of how ZIP format works (each file is encoded 
individually, compression dictionaries are inefficient for many small files).

> Can we make oal.util.packed.BulkOperation* smaller?
> ---
>
> Key: LUCENE-4322
> URL: https://issues.apache.org/jira/browse/LUCENE-4322
> Project: Lucene - Core
>  Issue Type: Bug
>Reporter: Michael McCandless
> Fix For: 5.0, 4.0
>
> Attachments: LUCENE-4322.patch
>
>
> These source files add up to a lot of sources ... it caused problems when 
> compiling under Maven and InteliJ.
> I committed a change to make separates files, but in aggregate this is still 
> a lot ...
> EG maybe we don't need to specialize encode?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

RE: svn commit: r1378140 - /lucene/dev/trunk/lucene/core/src/java/org/apache/lucene/search/DisjunctionMaxQuery.java

2012-08-28 Thread Uwe Schindler

Glad to help you!

- {@code ...} creates a ... environment, so ist formatted like 
code.
- there is also {@literal ...} which makes the thing escaped, but plain text 
(so its same font as remaining text)

Uwe

-
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: u...@thetaphi.de


> -Original Message-
> From: Michael McCandless [mailto:luc...@mikemccandless.com]
> Sent: Tuesday, August 28, 2012 6:16 PM
> To: dev@lucene.apache.org
> Subject: Re: svn commit: r1378140 -
> /lucene/dev/trunk/lucene/core/src/java/org/apache/lucene/search/Disjunction
> MaxQuery.java
> 
> Aha!  Most excellent.  I'll switch to {@code ...}.  Thanks for the pointer.
> 
> Mike McCandless
> 
> http://blog.mikemccandless.com
> 
> On Tue, Aug 28, 2012 at 10:35 AM, Uwe Schindler  wrote:
> > Hi Mike,
> >
> > The easier way (and also looks more code-style like):
> > {@code Collection} - This escapes automatically and you can read it
> better. It also added  around!
> >
> > See my recent additions to WeakIdentityMap :-)
> >
> > Uwe
> >
> > -
> > Uwe Schindler
> > H.-H.-Meier-Allee 63, D-28213 Bremen
> > http://www.thetaphi.de
> > eMail: u...@thetaphi.de
> >
> >
> >> -Original Message-
> >> From: mikemcc...@apache.org [mailto:mikemcc...@apache.org]
> >> Sent: Tuesday, August 28, 2012 4:02 PM
> >> To: comm...@lucene.apache.org
> >> Subject: svn commit: r1378140 -
> >> /lucene/dev/trunk/lucene/core/src/java/org/apache/lucene/search/Disju
> >> nction
> >> MaxQuery.java
> >>
> >> Author: mikemccand
> >> Date: Tue Aug 28 14:02:19 2012
> >> New Revision: 1378140
> >>
> >> URL: http://svn.apache.org/viewvc?rev=1378140&view=rev
> >> Log:
> >> escape generics to HTML
> >>
> >> Modified:
> >>
> >> lucene/dev/trunk/lucene/core/src/java/org/apache/lucene/search/Disjun
> >> ction
> >> MaxQuery.java
> >>
> >> Modified:
> >> lucene/dev/trunk/lucene/core/src/java/org/apache/lucene/search/Disjun
> >> ction
> >> MaxQuery.java
> >> URL:
> >> http://svn.apache.org/viewvc/lucene/dev/trunk/lucene/core/src/java/or
> >> g/apac
> >>
> he/lucene/search/DisjunctionMaxQuery.java?rev=1378140&r1=1378139&r2=1
> >> 378140&view=diff
> >>
> 
> >> ==
> >> ---
> >> lucene/dev/trunk/lucene/core/src/java/org/apache/lucene/search/Disjun
> >> ction
> >> MaxQuery.java (original)
> >> +++ lucene/dev/trunk/lucene/core/src/java/org/apache/lucene/search/Di
> >> +++ sju nctionMaxQuery.java Tue Aug 28 14:02:19 2012
> >> @@ -61,7 +61,7 @@ public class DisjunctionMaxQuery extends
> >>
> >>/**
> >> * Creates a new DisjunctionMaxQuery
> >> -   * @param disjuncts a Collection of all the disjuncts to add
> >> +   * @param disjuncts a Collection of all the disjuncts
> >> + to add
> >> * @param tieBreakerMultiplier   the weight to give to each matching
> non-
> >> maximum disjunct
> >> */
> >>public DisjunctionMaxQuery(Collection disjuncts, float
> >> tieBreakerMultiplier) { @@ -77,14 +77,14 @@ public class
> >> DisjunctionMaxQuery extends
> >>}
> >>
> >>/** Add a collection of disjuncts to this disjunction
> >> -   * via Iterable
> >> +   * via Iterable
> >> * @param disjuncts a collection of queries to add as disjuncts.
> >> */
> >>public void add(Collection disjuncts) {
> >>  this.disjuncts.addAll(disjuncts);
> >>}
> >>
> >> -  /** @return An Iterator over the disjuncts */
> >> +  /** @return An Iterator over the disjuncts */
> >>public Iterator iterator() {
> >>  return disjuncts.iterator();
> >>}
> >
> >
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For
> > additional commands, e-mail: dev-h...@lucene.apache.org
> >
> 
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional
> commands, e-mail: dev-h...@lucene.apache.org


-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-4332) Integrate PiTest mutation coverage tool into build


[ 
https://issues.apache.org/jira/browse/LUCENE-4332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13443332#comment-13443332
 ] 

Hoss Man commented on LUCENE-4332:
--

I had never heard of this technique until Greg mentioned it on the list, but 
the key thing that really impresses me about it is that (from what i can see) 
it can help us find code whose behavior does not affect the output of the test 
-- this is something no "code coverage" tool like clover and emma can do.  

Clover is great for reporting that when the tests are run, method "bar()" is 
executed 200 times but "foo()" is never executed at all, but that doesn't tell 
us anything about whether the success of a test is actually dependent on the 
results of "bar()" being correct.  With these kinds of mutation testing, we 
will be able to see reports that say "bar() was executed 200 times, but when i 
munged the result of of bar() it didn't cause any tests to fail" which could be 
a big help for identifying tests where we assert the results of method calls, 
but forget to assert the "side effects" of those calls.



> Integrate PiTest mutation coverage tool into build
> --
>
> Key: LUCENE-4332
> URL: https://issues.apache.org/jira/browse/LUCENE-4332
> Project: Lucene - Core
>  Issue Type: Improvement
>Affects Versions: 4.1, 5.0
>Reporter: Greg Bowyer
>Assignee: Greg Bowyer
>  Labels: build
> Attachments: 
> LUCENE-4332-Integrate-PiTest-mutation-coverage-tool-into-build.patch, 
> LUCENE-4332-Integrate-PiTest-mutation-coverage-tool-into-build.patch
>
>
> As discussed briefly on the mailing list, this patch is an attempt to 
> integrate the PiTest mutation coverage tool into the lucene build

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-4322) Can we make oal.util.packed.BulkOperation* smaller?

2012-08-28 Thread Michael McCandless (JIRA)


[ 
https://issues.apache.org/jira/browse/LUCENE-4322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13443313#comment-13443313
 ] 

Michael McCandless commented on LUCENE-4322:


+1 for the first iteration.

> Can we make oal.util.packed.BulkOperation* smaller?
> ---
>
> Key: LUCENE-4322
> URL: https://issues.apache.org/jira/browse/LUCENE-4322
> Project: Lucene - Core
>  Issue Type: Bug
>Reporter: Michael McCandless
> Fix For: 5.0, 4.0
>
> Attachments: LUCENE-4322.patch
>
>
> These source files add up to a lot of sources ... it caused problems when 
> compiling under Maven and InteliJ.
> I committed a change to make separates files, but in aggregate this is still 
> a lot ...
> EG maybe we don't need to specialize encode?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Comment Edited] (LUCENE-4332) Integrate PiTest mutation coverage tool into build

[
https://issues.apache.org/jira/browse/LUCENE-4332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13443304#comment-13443304
]

Greg Bowyer edited comment on LUCENE-4332 at 8/29/12 4:51 AM:
--

Wow lot of interest .

I will try to answer some of the salient points

Core was missing until today as one test (TestLuceneConstantVersion) didn't run
correctly as it was lacking the Lucene version system property. Currently pit
refuses to run unless the underlying suite is all green (a good thing IMHO) so
I didn't have core from my first run (its there now).

This takes a long time to run, all of the ancillary Lucene packages take
roughly 4 hours to run on the largest CPU ec2 instance, core takes 8 hours
(this was the other reason core was missing, I was waiting for it to finish
crunching)

As to the random seed, I completely agree and it was one of the things I
mentioned on the mailing list that makes the output of this tool not perfect. I
do feel that the tests that are randomised typically do a better job at gaining
coverage, but its a good idea to stabilise the seed.

Jars and build.xml, I have no problems changing this to whatever people think
fits best into the build. My impression was that clover is handled the way it
is because it is not technically opensource and as a result has screwball
licensing concerns, essentially I didn't know any better :S I will try to get a
chance to make it use the ivy:cachepath approach.

Regarding the risks posed by mutations, I cannot prove or say there are no
risks; however mutation testing is not random in the mutations applied, they
are formulaic and quite simple. It will not permute arguments nor will it
mutate complex objects (it can and does mess with object references turning
references in arguments to nulls). I can conceive of ways in which it could
screwup mutated code making it possible to delete random files but I don't
think they are going to be extremely likely situations. FWIW I would be less
worried about this deleting something on the filesystem and far more worried
about it accidentally leaving corpses of undeleted files.

Sandboxing it could solve that issue, if that is too much effort another
approach might be to work with the pitest team and build a security manager
that is militant about file access, disallowing anything that canonicalises
outside of a given path.

Oh and as Robert suggested we can always point it away from key things.

At the end of the day its a tool like any other, I have exactly the same
feelings as Robert on this
{quote}

This is cool. I'd say lets get it up and going on jenkins (even weekly
or something). why worry about the imperfections in any of these
coverage tools, whats way more important is when the results find
situations where you thought you were testing something, but really
arent, etc (here was a recent one found by clover
http://svn.apache.org/viewvc?rev=1376722&view=rev).

so imo just another tool to be able to identify serious gaps/test-bugs
after things are up and running. and especially looking at deltas from
line coverage to identify stuff thats 'executing' but not actually
being tested.
{quote}

was (Author: gbow...@fastmail.co.uk):
Wow lot of interest .

I will try to answer some of the salient points

[jira] [Commented] (LUCENE-4332) Integrate PiTest mutation coverage tool into build


[ 
https://issues.apache.org/jira/browse/LUCENE-4332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13443304#comment-13443304
 ] 

Greg Bowyer commented on LUCENE-4332:
-

Wow lot of interest .

I will try to answer some of the salient points

Core was missing until today as one test (TestLuceneConstantVersion) didn't run 
correctly as it was lacking the Lucene version system property. Currently pit 
refuses to run unless the underlying suite is all green (a good thing IMHO) so 
I didn't have core from my first run (its there now).

This takes a long time to run, all of the ancillary Lucene packages take 
roughly 4 hours to run on the largest CPU ec2 instance, core takes 8 hours 
(this was the other reason core was missing, I was waiting for it to finish 
crunching)

As to the random seed, I completely agree and it was one of the things I 
mentioned on the mailing list that makes the output of this tool not perfect. I 
do feel that the tests that are randomised typically do a better job at gaining 
coverage, but its a good idea to stabilise the seed.

Jars and build.xml, I have no problems changing this to whatever people think 
fits best into the build. My impression was that clover is handled the way it 
is because it is not technically opensource and as a result has screwball 
licensing concerns, essentially I didn't know any better :S I will try to get a 
chance to make it use the ivy:cachepath approach.

Regarding the risks posed by mutations, I cannot prove or say there are no 
risks; however mutation testing is not random in the mutations applied, they 
are formulaic and quite simple. It will not permute arguments nor will it 
mutate complex objects (it can and does mess with object references turning 
references in arguments to nulls). I can conceive of ways in which it could 
screwup mutated code making it possible to delete random files but I don't 
think they are going to be extremely likely situations. FWIW I would be less 
worried about this deleting something on the filesystem and far more worried 
about it accidentally leaving corpses of undeleted files. 

Sandboxing it could solve that issue, if that is too much effort another 
approach might be to work with the pitest team and build a security manager 
that is militant about file access, disallowing anything that canonicalises 
outside of a given path.

Oh and as Robert suggested we can always point it away from key things.

> Integrate PiTest mutation coverage tool into build
> --
>
> Key: LUCENE-4332
> URL: https://issues.apache.org/jira/browse/LUCENE-4332
> Project: Lucene - Core
>  Issue Type: Improvement
>Affects Versions: 4.1, 5.0
>Reporter: Greg Bowyer
>Assignee: Greg Bowyer
>  Labels: build
> Attachments: 
> LUCENE-4332-Integrate-PiTest-mutation-coverage-tool-into-build.patch, 
> LUCENE-4332-Integrate-PiTest-mutation-coverage-tool-into-build.patch
>
>
> As discussed briefly on the mailing list, this patch is an attempt to 
> integrate the PiTest mutation coverage tool into the lucene build

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-4324) extend checkJavaDocs.py to methods,constants,fields


[ 
https://issues.apache.org/jira/browse/LUCENE-4324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13443299#comment-13443299
 ] 

Robert Muir commented on LUCENE-4324:
-

I think this is working pretty well now as far as output. also the overridden 
method situation
mentioned by Uwe is resolved, and Mike has it detecting more broken html.

I think we should port to java6 now before it gets any harder. If no one beats 
me to it I'll
try to look at this later tonight (but please feel free, otherwise I will be 
hacking on trying
to fill in missing docs themselves)

> extend checkJavaDocs.py to methods,constants,fields
> ---
>
> Key: LUCENE-4324
> URL: https://issues.apache.org/jira/browse/LUCENE-4324
> Project: Lucene - Core
>  Issue Type: New Feature
>  Components: general/build
>Reporter: Robert Muir
> Attachments: LUCENE-4322.patch, LUCENE-4324_crawl.patch
>
>
> We have a large amount of classes in the source code, its nice that we have 
> checkJavaDocs.py to ensure packages and classes have some human-level 
> description.
> But I think we need it for methods etc too. (it is also part of our 
> contribution/style guidelines: 
> http://wiki.apache.org/lucene-java/HowToContribute#Making_Changes)
> The reason is that like classes and packages, once we can enforce this in the 
> build, people will quickly add forgotten documentation soon after their 
> commit when its fresh in their mind.
> Otherwise, its likely to never happen.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-4322) Can we make oal.util.packed.BulkOperation* smaller?


[ 
https://issues.apache.org/jira/browse/LUCENE-4322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13443297#comment-13443297
 ] 

Robert Muir commented on LUCENE-4322:
-

+1 for the first iteration.

> Can we make oal.util.packed.BulkOperation* smaller?
> ---
>
> Key: LUCENE-4322
> URL: https://issues.apache.org/jira/browse/LUCENE-4322
> Project: Lucene - Core
>  Issue Type: Bug
>Reporter: Michael McCandless
> Fix For: 5.0, 4.0
>
> Attachments: LUCENE-4322.patch
>
>
> These source files add up to a lot of sources ... it caused problems when 
> compiling under Maven and InteliJ.
> I committed a change to make separates files, but in aggregate this is still 
> a lot ...
> EG maybe we don't need to specialize encode?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-4322) Can we make oal.util.packed.BulkOperation* smaller?


[ 
https://issues.apache.org/jira/browse/LUCENE-4322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13443284#comment-13443284
 ] 

Adrien Grand commented on LUCENE-4322:
--

bq. We could still shrink things further by doing less loop unrolling 
ourselves? Eg for BulkOperationPacked2, when decoding from byte[], that code is 
replicated 8 times but could be done as just 8* iters.

Yes, I had planned to work on it too! Unless someone doesn't like my last 
patch, I'll commit it and will start working on this loop unrolling issue 
soon...


> Can we make oal.util.packed.BulkOperation* smaller?
> ---
>
> Key: LUCENE-4322
> URL: https://issues.apache.org/jira/browse/LUCENE-4322
> Project: Lucene - Core
>  Issue Type: Bug
>Reporter: Michael McCandless
> Fix For: 5.0, 4.0
>
> Attachments: LUCENE-4322.patch
>
>
> These source files add up to a lot of sources ... it caused problems when 
> compiling under Maven and InteliJ.
> I committed a change to make separates files, but in aggregate this is still 
> a lot ...
> EG maybe we don't need to specialize encode?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-4322) Can we make oal.util.packed.BulkOperation* smaller?

2012-08-28 Thread Michael McCandless (JIRA)


[ 
https://issues.apache.org/jira/browse/LUCENE-4322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13443267#comment-13443267
 ] 

Michael McCandless commented on LUCENE-4322:


Patch looks good!  That's a nice reduction.

Too bad we have to duplicate decode 4 times (from long[]/byte[] to 
long[]/int[]).

We could still shrink things further by doing less loop unrolling ourselves?  
Eg for BulkOperationPacked2, when decoding from byte[], that code is replicated 
8 times but could be done as just 8* iters.

> Can we make oal.util.packed.BulkOperation* smaller?
> ---
>
> Key: LUCENE-4322
> URL: https://issues.apache.org/jira/browse/LUCENE-4322
> Project: Lucene - Core
>  Issue Type: Bug
>Reporter: Michael McCandless
> Fix For: 5.0, 4.0
>
> Attachments: LUCENE-4322.patch
>
>
> These source files add up to a lot of sources ... it caused problems when 
> compiling under Maven and InteliJ.
> I committed a change to make separates files, but in aggregate this is still 
> a lot ...
> EG maybe we don't need to specialize encode?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (LUCENE-4322) Can we make oal.util.packed.BulkOperation* smaller?


 [ 
https://issues.apache.org/jira/browse/LUCENE-4322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Adrien Grand updated LUCENE-4322:
-

Attachment: LUCENE-4322.patch

Patch that tries to reduce the JAR size:
 - unspecialized encode methods,
 - specialized decode methods only when 0 < bitsPerValue <= 24.

Overall, it makes the core jar 361kb bytes smaller (2700542 bytes before 
applying the patch, 2330514 after).

I ran a quick run of lucene-util in debug mode with blockPostingsFormat=For and 
it showed no performance difference.

> Can we make oal.util.packed.BulkOperation* smaller?
> ---
>
> Key: LUCENE-4322
> URL: https://issues.apache.org/jira/browse/LUCENE-4322
> Project: Lucene - Core
>  Issue Type: Bug
>Reporter: Michael McCandless
> Fix For: 5.0, 4.0
>
> Attachments: LUCENE-4322.patch
>
>
> These source files add up to a lot of sources ... it caused problems when 
> compiling under Maven and InteliJ.
> I committed a change to make separates files, but in aggregate this is still 
> a lot ...
> EG maybe we don't need to specialize encode?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-4322) Can we make oal.util.packed.BulkOperation* smaller?


[ 
https://issues.apache.org/jira/browse/LUCENE-4322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13443246#comment-13443246
 ] 

Adrien Grand commented on LUCENE-4322:
--

It looks like one of the reasons of the size increase of lucene-core.jar is its 
number of classes. Although some code has been moved to modules, its number of 
classes has increased from 972 in 3.6.1 to 1471 in trunk.

> Can we make oal.util.packed.BulkOperation* smaller?
> ---
>
> Key: LUCENE-4322
> URL: https://issues.apache.org/jira/browse/LUCENE-4322
> Project: Lucene - Core
>  Issue Type: Bug
>Reporter: Michael McCandless
> Fix For: 5.0, 4.0
>
>
> These source files add up to a lot of sources ... it caused problems when 
> compiling under Maven and InteliJ.
> I committed a change to make separates files, but in aggregate this is still 
> a lot ...
> EG maybe we don't need to specialize encode?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: svn commit: r1378140 - /lucene/dev/trunk/lucene/core/src/java/org/apache/lucene/search/DisjunctionMaxQuery.java

2012-08-28 Thread Michael McCandless

Aha!  Most excellent.  I'll switch to {@code ...}.  Thanks for the pointer.

Mike McCandless

http://blog.mikemccandless.com

On Tue, Aug 28, 2012 at 10:35 AM, Uwe Schindler  wrote:
> Hi Mike,
>
> The easier way (and also looks more code-style like):
> {@code Collection} - This escapes automatically and you can read it 
> better. It also added  around!
>
> See my recent additions to WeakIdentityMap :-)
>
> Uwe
>
> -
> Uwe Schindler
> H.-H.-Meier-Allee 63, D-28213 Bremen
> http://www.thetaphi.de
> eMail: u...@thetaphi.de
>
>
>> -Original Message-
>> From: mikemcc...@apache.org [mailto:mikemcc...@apache.org]
>> Sent: Tuesday, August 28, 2012 4:02 PM
>> To: comm...@lucene.apache.org
>> Subject: svn commit: r1378140 -
>> /lucene/dev/trunk/lucene/core/src/java/org/apache/lucene/search/Disjunction
>> MaxQuery.java
>>
>> Author: mikemccand
>> Date: Tue Aug 28 14:02:19 2012
>> New Revision: 1378140
>>
>> URL: http://svn.apache.org/viewvc?rev=1378140&view=rev
>> Log:
>> escape generics to HTML
>>
>> Modified:
>>
>> lucene/dev/trunk/lucene/core/src/java/org/apache/lucene/search/Disjunction
>> MaxQuery.java
>>
>> Modified:
>> lucene/dev/trunk/lucene/core/src/java/org/apache/lucene/search/Disjunction
>> MaxQuery.java
>> URL:
>> http://svn.apache.org/viewvc/lucene/dev/trunk/lucene/core/src/java/org/apac
>> he/lucene/search/DisjunctionMaxQuery.java?rev=1378140&r1=1378139&r2=1
>> 378140&view=diff
>> 
>> ==
>> ---
>> lucene/dev/trunk/lucene/core/src/java/org/apache/lucene/search/Disjunction
>> MaxQuery.java (original)
>> +++ lucene/dev/trunk/lucene/core/src/java/org/apache/lucene/search/Disju
>> +++ nctionMaxQuery.java Tue Aug 28 14:02:19 2012
>> @@ -61,7 +61,7 @@ public class DisjunctionMaxQuery extends
>>
>>/**
>> * Creates a new DisjunctionMaxQuery
>> -   * @param disjuncts a Collection of all the disjuncts to add
>> +   * @param disjuncts a Collection of all the disjuncts to
>> + add
>> * @param tieBreakerMultiplier   the weight to give to each matching non-
>> maximum disjunct
>> */
>>public DisjunctionMaxQuery(Collection disjuncts, float
>> tieBreakerMultiplier) { @@ -77,14 +77,14 @@ public class
>> DisjunctionMaxQuery extends
>>}
>>
>>/** Add a collection of disjuncts to this disjunction
>> -   * via Iterable
>> +   * via Iterable
>> * @param disjuncts a collection of queries to add as disjuncts.
>> */
>>public void add(Collection disjuncts) {
>>  this.disjuncts.addAll(disjuncts);
>>}
>>
>> -  /** @return An Iterator over the disjuncts */
>> +  /** @return An Iterator over the disjuncts */
>>public Iterator iterator() {
>>  return disjuncts.iterator();
>>}
>
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

RE: large messages from Jenkins failures

Hi Dawid,

Unfortunately, -Dtests.showOutput=never penalizes all tests that don't have 
megabytes of failure output because some do.

What do you think of adding an option to limit output size (e.g. 
-Dtests.outputLimitKB=10), and truncating to that size if it's exceeded?  If 
you think this would be reasonable, I'm willing to (try to) do the work.

Steve

-Original Message-
From: Steven A Rowe [mailto:sar...@syr.edu] 
Sent: Monday, August 20, 2012 4:10 PM
To: dev@lucene.apache.org
Subject: RE: large messages from Jenkins failures

+1 to using -Dtests.showOutput=never for Jenkins jobs. - Steve

-Original Message-
From: dawid.we...@gmail.com [mailto:dawid.we...@gmail.com] On Behalf Of Dawid 
Weiss
Sent: Monday, August 20, 2012 2:20 PM
To: dev@lucene.apache.org
Subject: Re: large messages from Jenkins failures

This is partially aggregated by solr failure logs (the output for
successful suites is not emitted to the console). As for myself I
don't look at those e-mails directly, I typicall click on the jenkins
link to see the full output. Alternatively we could suppress the
console output for failures too (it would still show the stack trace
and everything, just not the stdout/sysouts) -- this is relatively
easy to override even from jenkins level:

-Dtests.showOutput=never

Dawid

On Fri, Aug 17, 2012 at 5:04 PM, Dyer, James
 wrote:
> Is there any way we can limit the size of the messages Jenkins emails this
> list?  Responsing to a "your mailbox is full" warning, I found I had 32
> recent Jenkins messages all over 1mb (a few were >10mb).  A few weeks ago I
> returned from vacation to find my mail account partially disabled because
> Jenkins had used up most of my storage.  Maybe, if the log is more than so
> many lines to just supplies a link to it than have the whole thing in the
> email?  I realize a lot of you have unlimited storage on your email
> accounts, but unfortunately I do not.
>
>
>
> James Dyer
>
> E-Commerce Systems
>
> Ingram Content Group
>
> (615) 213-4311
>
>

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org


-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-3763) Make solr use lucene filters directly

2012-08-28 Thread Yonik Seeley (JIRA)


[ 
https://issues.apache.org/jira/browse/SOLR-3763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13443202#comment-13443202
 ] 

Yonik Seeley commented on SOLR-3763:


Interesting work Greg!  A few points:

bq. Another issue here is that filters currently cache sub-optimally given the 
changes in lucene towards atomic readers.

This really depends on the problem - sometimes top-level cache is more optimal, 
and sometimes per-segment caches are more optimal.  IMO, we shouldn't force 
either, but add the ability to cache per-segment.

There are already issues open for caching disjunction clauses separately too - 
it's a rather orthogonal issue.

It might be a better idea to start off small: we could make a QParser that 
creates a CachingWrapperFilter wrapped in a FilteredQuery and hence will cache 
per-segment.  That should be simple and non-invasive enough to make it into 4.0

> Make solr use lucene filters directly
> -
>
> Key: SOLR-3763
> URL: https://issues.apache.org/jira/browse/SOLR-3763
> Project: Solr
>  Issue Type: Improvement
>Affects Versions: 4.0, 4.1, 5.0
>Reporter: Greg Bowyer
>Assignee: Greg Bowyer
> Attachments: SOLR-3763-Make-solr-use-lucene-filters-directly.patch
>
>
> Presently solr uses bitsets, queries and collectors to implement the concept 
> of filters. This has proven to be very powerful, but does come at the cost of 
> introducing a large body of code into solr making it harder to optimise and 
> maintain.
> Another issue here is that filters currently cache sub-optimally given the 
> changes in lucene towards atomic readers.
> Rather than patch these issues, this is an attempt to rework the filters in 
> solr to leverage the Filter subsystem from lucene as much as possible.
> In good time the aim is to get this to do the following:
> ∘ Handle setting up filter implementations that are able to correctly cache 
> with reference to the AtomicReader that they are caching for rather that for 
> the entire index at large
> ∘ Get the post filters working, I am thinking that this can be done via 
> lucenes chained filter, with the ‟expensive” filters being put towards the 
> end of the chain - this has different semantics internally to the original 
> implementation but IMHO should have the same result for end users
> ∘ Learn how to create filters that are potentially more efficient, at present 
> solr basically runs a simple query that gathers a DocSet that relates to the 
> documents that we want filtered; it would be interesting to make use of 
> filter implementations that are in theory faster than query filters (for 
> instance there are filters that are able to query the FieldCache)
> ∘ Learn how to decompose filters so that a complex filter query can be cached 
> (potentially) as its constituent parts; for example the filter below 
> currently needs love, care and feeding to ensure that the filter cache is not 
> unduly stressed
> {code}
>   'category:(100) OR category:(200) OR category:(300)'
> {code}
> Really there is no reason not to express this in a cached form as 
> {code}
> BooleanFilter(
> FilterClause(CachedFilter(TermFilter(Term("category", 100))), SHOULD),
> FilterClause(CachedFilter(TermFilter(Term("category", 200))), SHOULD),
> FilterClause(CachedFilter(TermFilter(Term("category", 300))), SHOULD)
>   )
> {code}
> This would yeild better cache usage I think as we can resuse docsets across 
> multiple queries as well as avoid issues when filters are presented in 
> differing orders
> ∘ Instead of end users providing costing we might (and this is a big might 
> FWIW), be able to create a sort of execution plan of filters, leveraging a 
> combination of what the index is able to tell us as well as sampling and 
> ‟educated guesswork”; in essence this is what some DBMS software, for example 
> postgresql does - it has a genetic algo that attempts to solve the travelling 
> salesman - to great effect
> ∘ I am sure I will probably come up with other ambitious ideas to plug in 
> here . :S 
> Patches obviously forthcoming but the bulk of the work can be followed here 
> https://github.com/GregBowyer/lucene-solr/commits/solr-uses-lucene-filters

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

RE: svn commit: r1378140 - /lucene/dev/trunk/lucene/core/src/java/org/apache/lucene/search/DisjunctionMaxQuery.java

2012-08-28 Thread Uwe Schindler

Hi Mike,

The easier way (and also looks more code-style like):
{@code Collection} - This escapes automatically and you can read it 
better. It also added  around!

See my recent additions to WeakIdentityMap :-)

Uwe

-
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: u...@thetaphi.de


> -Original Message-
> From: mikemcc...@apache.org [mailto:mikemcc...@apache.org]
> Sent: Tuesday, August 28, 2012 4:02 PM
> To: comm...@lucene.apache.org
> Subject: svn commit: r1378140 -
> /lucene/dev/trunk/lucene/core/src/java/org/apache/lucene/search/Disjunction
> MaxQuery.java
> 
> Author: mikemccand
> Date: Tue Aug 28 14:02:19 2012
> New Revision: 1378140
> 
> URL: http://svn.apache.org/viewvc?rev=1378140&view=rev
> Log:
> escape generics to HTML
> 
> Modified:
> 
> lucene/dev/trunk/lucene/core/src/java/org/apache/lucene/search/Disjunction
> MaxQuery.java
> 
> Modified:
> lucene/dev/trunk/lucene/core/src/java/org/apache/lucene/search/Disjunction
> MaxQuery.java
> URL:
> http://svn.apache.org/viewvc/lucene/dev/trunk/lucene/core/src/java/org/apac
> he/lucene/search/DisjunctionMaxQuery.java?rev=1378140&r1=1378139&r2=1
> 378140&view=diff
> 
> ==
> ---
> lucene/dev/trunk/lucene/core/src/java/org/apache/lucene/search/Disjunction
> MaxQuery.java (original)
> +++ lucene/dev/trunk/lucene/core/src/java/org/apache/lucene/search/Disju
> +++ nctionMaxQuery.java Tue Aug 28 14:02:19 2012
> @@ -61,7 +61,7 @@ public class DisjunctionMaxQuery extends
> 
>/**
> * Creates a new DisjunctionMaxQuery
> -   * @param disjuncts a Collection of all the disjuncts to add
> +   * @param disjuncts a Collection of all the disjuncts to
> + add
> * @param tieBreakerMultiplier   the weight to give to each matching non-
> maximum disjunct
> */
>public DisjunctionMaxQuery(Collection disjuncts, float
> tieBreakerMultiplier) { @@ -77,14 +77,14 @@ public class
> DisjunctionMaxQuery extends
>}
> 
>/** Add a collection of disjuncts to this disjunction
> -   * via Iterable
> +   * via Iterable
> * @param disjuncts a collection of queries to add as disjuncts.
> */
>public void add(Collection disjuncts) {
>  this.disjuncts.addAll(disjuncts);
>}
> 
> -  /** @return An Iterator over the disjuncts */
> +  /** @return An Iterator over the disjuncts */
>public Iterator iterator() {
>  return disjuncts.iterator();
>}



-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-3088) create shard placeholders


 [ 
https://issues.apache.org/jira/browse/SOLR-3088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mark Miller updated SOLR-3088:
--

Fix Version/s: (was: 4.1)
   5.0
   4.0

> create shard placeholders
> -
>
> Key: SOLR-3088
> URL: https://issues.apache.org/jira/browse/SOLR-3088
> Project: Solr
>  Issue Type: New Feature
>  Components: SolrCloud
>Reporter: Yonik Seeley
>Assignee: Sami Siren
> Fix For: 4.0, 5.0
>
> Attachments: SOLR-3088.patch, SOLR-3088.patch
>
>
> When creating a new collection, a placeholder for each shard should be 
> created.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Resolved] (SOLR-3088) create shard placeholders


 [ 
https://issues.apache.org/jira/browse/SOLR-3088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mark Miller resolved SOLR-3088.
---

Resolution: Fixed
  Assignee: Sami Siren

> create shard placeholders
> -
>
> Key: SOLR-3088
> URL: https://issues.apache.org/jira/browse/SOLR-3088
> Project: Solr
>  Issue Type: New Feature
>  Components: SolrCloud
>Reporter: Yonik Seeley
>Assignee: Sami Siren
> Fix For: 4.1
>
> Attachments: SOLR-3088.patch, SOLR-3088.patch
>
>
> When creating a new collection, a placeholder for each shard should be 
> created.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-3659) non-reproducible failures from RecoveryZkTest - mostly NRTCachingDirectory.deleteFile


[ 
https://issues.apache.org/jira/browse/SOLR-3659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13443162#comment-13443162
 ] 

Dawid Weiss commented on SOLR-3659:
---

Thanks Mark.

> non-reproducible failures from RecoveryZkTest - mostly 
> NRTCachingDirectory.deleteFile
> -
>
> Key: SOLR-3659
> URL: https://issues.apache.org/jira/browse/SOLR-3659
> Project: Solr
>  Issue Type: Bug
>Reporter: Hoss Man
> Attachments: just-failures.txt, 
> RecoveryZkTest.testDistribSearch-100-tests-failures.txt.tgz
>
>
> Since getting my new laptop, i've noticed some sporadic failures from 
> RecoveryZkTest, so last night tried running 100 iterations againts trunk 
> (r1363555), and got 5 errors/failures...
> * 3 asertion failures from NRTCachingDirectory.deleteFile
> * 1 node recovery assertion from 
> AbstractDistributedZkTestCase.waitForRecoveriesToFinish caused by OOM
> * 1 searcher leak assertion: opens=1658 closes=1652 (possibly lingering 
> affects from OOM?)
> see comments/attachments for details

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-3347) deleteByQuery failing with SolrCloud without _version_ in schema.xml


 [ 
https://issues.apache.org/jira/browse/SOLR-3347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mark Miller updated SOLR-3347:
--

Fix Version/s: 5.0

> deleteByQuery failing with SolrCloud without _version_ in schema.xml
> 
>
> Key: SOLR-3347
> URL: https://issues.apache.org/jira/browse/SOLR-3347
> Project: Solr
>  Issue Type: Bug
>  Components: SolrCloud
>Reporter: Benson Margulies
> Fix For: 4.0, 5.0
>
> Attachments: 
> 0001-Attempt-to-repro-problem-with-del-and-SolrCloud.patch, 
> provision-and-start.sh, schema.xml, solrconfig.xml
>
>
> Distributed execution of deleteByQuery("\*:\*") depends on the existence of a 
> field \_version\_ in the schema. The default schema has no comment on this 
> field to indicate its important or relevance to SolrCloud, and no message is 
> logged nor error status returned when there is no such field. The code in 
> DistributedUpdateProcessor just has an if statement that never ever does any 
> local deleting without it.
> I don't know whether the intention was that this should work or not. If 
> someone would clue me in, I'd make a patch for schema.xml to add comments, or 
> a patch to D-U-P to add logging. If it was supposed to work, I'm probably not 
> qualified to make the fix to make it work.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-3465) Replication Causes Two Searcher Warmups


 [ 
https://issues.apache.org/jira/browse/SOLR-3465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mark Miller updated SOLR-3465:
--

Fix Version/s: 5.0
   4.0

> Replication Causes Two Searcher Warmups 
> 
>
> Key: SOLR-3465
> URL: https://issues.apache.org/jira/browse/SOLR-3465
> Project: Solr
>  Issue Type: Bug
>  Components: replication (java)
>Affects Versions: 4.0-ALPHA
>Reporter: Michael Garski
>Assignee: Mark Miller
> Fix For: 4.0, 5.0
>
>
> I'm doing some testing with the current trunk, and am seeing that when a 
> slave retrieves index updates from the master the warmup & searcher 
> registration is performed twice.  Here is a snippet of the log that 
> demonstrates this:
> May 16, 2012 6:02:02 PM org.apache.solr.handler.SnapPuller fetchLatestIndex
> INFO: Total time taken for download : 92 secs
> May 16, 2012 6:02:02 PM org.apache.solr.core.SolrDeletionPolicy onInit
> INFO: SolrDeletionPolicy.onInit: commits:num=2
> 
> commit{dir=/Users/mgarski/Code/indexes/solr2/geo/index,segFN=segments_1,generation=1,filenames=[segments_1]
> 
> commit{dir=/Users/mgarski/Code/indexes/solr2/geo/index,segFN=segments_10,generation=36,filenames=[_45_0.tim,
>  _45.fdt, segments_10, _45_0.tip, _45.fdx, _45.fnm, _45_0.frq, _45.per, 
> _45_0.prx]
> May 16, 2012 6:02:02 PM org.apache.solr.core.SolrDeletionPolicy updateCommits
> INFO: newest commit = 36
> May 16, 2012 6:02:02 PM org.apache.solr.search.SolrIndexSearcher 
> INFO: Opening Searcher@559fe5e6 main
> May 16, 2012 6:02:02 PM org.apache.solr.core.QuerySenderListener newSearcher
> INFO: QuerySenderListener sending requests to Searcher@559fe5e6 
> main{StandardDirectoryReader(segments_10:335:nrt _45(4.0):C1096375)}
> May 16, 2012 6:02:02 PM org.apache.solr.core.QuerySenderListener newSearcher
> INFO: QuerySenderListener done.
> May 16, 2012 6:02:02 PM org.apache.solr.core.SolrCore registerSearcher
> INFO: [geo] Registered new searcher Searcher@559fe5e6 
> main{StandardDirectoryReader(segments_10:335:nrt _45(4.0):C1096375)}
> May 16, 2012 6:02:02 PM org.apache.solr.update.DirectUpdateHandler2 commit
> INFO: start 
> commit{flags=0,version=0,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false}
> May 16, 2012 6:02:02 PM org.apache.solr.search.SolrIndexSearcher 
> INFO: Opening Searcher@42101da9 main
> May 16, 2012 6:02:02 PM org.apache.solr.update.DirectUpdateHandler2 commit
> INFO: end_commit_flush
> May 16, 2012 6:02:02 PM org.apache.solr.core.QuerySenderListener newSearcher
> INFO: QuerySenderListener sending requests to Searcher@42101da9 
> main{StandardDirectoryReader(segments_10:335:nrt _45(4.0):C1096375)}
> May 16, 2012 6:02:02 PM org.apache.solr.core.QuerySenderListener newSearcher
> INFO: QuerySenderListener done.
> May 16, 2012 6:02:02 PM org.apache.solr.core.SolrCore registerSearcher
> INFO: [geo] Registered new searcher Searcher@42101da9 
> main{StandardDirectoryReader(segments_10:335:nrt _45(4.0):C1096375)}
> I am trying to determine the cause, does anyone have any idea where to start?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Assigned] (SOLR-3465) Replication Causes Two Searcher Warmups


 [ 
https://issues.apache.org/jira/browse/SOLR-3465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mark Miller reassigned SOLR-3465:
-

Assignee: Mark Miller

> Replication Causes Two Searcher Warmups 
> 
>
> Key: SOLR-3465
> URL: https://issues.apache.org/jira/browse/SOLR-3465
> Project: Solr
>  Issue Type: Bug
>  Components: replication (java)
>Affects Versions: 4.0-ALPHA
>Reporter: Michael Garski
>Assignee: Mark Miller
> Fix For: 4.0, 5.0
>
>
> I'm doing some testing with the current trunk, and am seeing that when a 
> slave retrieves index updates from the master the warmup & searcher 
> registration is performed twice.  Here is a snippet of the log that 
> demonstrates this:
> May 16, 2012 6:02:02 PM org.apache.solr.handler.SnapPuller fetchLatestIndex
> INFO: Total time taken for download : 92 secs
> May 16, 2012 6:02:02 PM org.apache.solr.core.SolrDeletionPolicy onInit
> INFO: SolrDeletionPolicy.onInit: commits:num=2
> 
> commit{dir=/Users/mgarski/Code/indexes/solr2/geo/index,segFN=segments_1,generation=1,filenames=[segments_1]
> 
> commit{dir=/Users/mgarski/Code/indexes/solr2/geo/index,segFN=segments_10,generation=36,filenames=[_45_0.tim,
>  _45.fdt, segments_10, _45_0.tip, _45.fdx, _45.fnm, _45_0.frq, _45.per, 
> _45_0.prx]
> May 16, 2012 6:02:02 PM org.apache.solr.core.SolrDeletionPolicy updateCommits
> INFO: newest commit = 36
> May 16, 2012 6:02:02 PM org.apache.solr.search.SolrIndexSearcher 
> INFO: Opening Searcher@559fe5e6 main
> May 16, 2012 6:02:02 PM org.apache.solr.core.QuerySenderListener newSearcher
> INFO: QuerySenderListener sending requests to Searcher@559fe5e6 
> main{StandardDirectoryReader(segments_10:335:nrt _45(4.0):C1096375)}
> May 16, 2012 6:02:02 PM org.apache.solr.core.QuerySenderListener newSearcher
> INFO: QuerySenderListener done.
> May 16, 2012 6:02:02 PM org.apache.solr.core.SolrCore registerSearcher
> INFO: [geo] Registered new searcher Searcher@559fe5e6 
> main{StandardDirectoryReader(segments_10:335:nrt _45(4.0):C1096375)}
> May 16, 2012 6:02:02 PM org.apache.solr.update.DirectUpdateHandler2 commit
> INFO: start 
> commit{flags=0,version=0,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false}
> May 16, 2012 6:02:02 PM org.apache.solr.search.SolrIndexSearcher 
> INFO: Opening Searcher@42101da9 main
> May 16, 2012 6:02:02 PM org.apache.solr.update.DirectUpdateHandler2 commit
> INFO: end_commit_flush
> May 16, 2012 6:02:02 PM org.apache.solr.core.QuerySenderListener newSearcher
> INFO: QuerySenderListener sending requests to Searcher@42101da9 
> main{StandardDirectoryReader(segments_10:335:nrt _45(4.0):C1096375)}
> May 16, 2012 6:02:02 PM org.apache.solr.core.QuerySenderListener newSearcher
> INFO: QuerySenderListener done.
> May 16, 2012 6:02:02 PM org.apache.solr.core.SolrCore registerSearcher
> INFO: [geo] Registered new searcher Searcher@42101da9 
> main{StandardDirectoryReader(segments_10:335:nrt _45(4.0):C1096375)}
> I am trying to determine the cause, does anyone have any idea where to start?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-3659) non-reproducible failures from RecoveryZkTest - mostly NRTCachingDirectory.deleteFile


[ 
https://issues.apache.org/jira/browse/SOLR-3659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13443160#comment-13443160
 ] 

Mark Miller commented on SOLR-3659:
---

This should be fixed now.

> non-reproducible failures from RecoveryZkTest - mostly 
> NRTCachingDirectory.deleteFile
> -
>
> Key: SOLR-3659
> URL: https://issues.apache.org/jira/browse/SOLR-3659
> Project: Solr
>  Issue Type: Bug
>Reporter: Hoss Man
> Attachments: just-failures.txt, 
> RecoveryZkTest.testDistribSearch-100-tests-failures.txt.tgz
>
>
> Since getting my new laptop, i've noticed some sporadic failures from 
> RecoveryZkTest, so last night tried running 100 iterations againts trunk 
> (r1363555), and got 5 errors/failures...
> * 3 asertion failures from NRTCachingDirectory.deleteFile
> * 1 node recovery assertion from 
> AbstractDistributedZkTestCase.waitForRecoveriesToFinish caused by OOM
> * 1 searcher leak assertion: opens=1658 closes=1652 (possibly lingering 
> affects from OOM?)
> see comments/attachments for details

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-3507) Refactor parts of solr doing inter node communication to use shardhandlerfactory/shardhandler

2012-08-28 Thread Sami Siren (JIRA)


 [ 
https://issues.apache.org/jira/browse/SOLR-3507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sami Siren updated SOLR-3507:
-

Attachment: SOLR-3507.patch

new patch updated to latest trunk with cleanups.

> Refactor parts of solr doing inter node communication to use 
> shardhandlerfactory/shardhandler
> -
>
> Key: SOLR-3507
> URL: https://issues.apache.org/jira/browse/SOLR-3507
> Project: Solr
>  Issue Type: Improvement
>Reporter: Sami Siren
>Assignee: Sami Siren
>Priority: Minor
> Attachments: SOLR-3507.patch, SOLR-3507.patch, SOLR-3507.patch
>
>
> Sequal to SOLR-3480, the aim is to change most (all?) parts of solr that need 
> to talk to different nodes to use ShardHandlerFacory from corecontainer.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Resolved] (LUCENE-4334) remove unnecessary ant-junit dependency


 [ 
https://issues.apache.org/jira/browse/LUCENE-4334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Muir resolved LUCENE-4334.
-

   Resolution: Fixed
Fix Version/s: 4.0
   5.0

seemed straightforward to me. Hope I'm not missing anything.

We should look for any other unused jars.

> remove unnecessary ant-junit dependency
> ---
>
> Key: LUCENE-4334
> URL: https://issues.apache.org/jira/browse/LUCENE-4334
> Project: Lucene - Core
>  Issue Type: Bug
>  Components: general/build
>Reporter: Robert Muir
> Fix For: 5.0, 4.0
>
> Attachments: LUCENE-4334.patch
>
>
> We don't use this integration anymore for running tests: we use 
> randomizedtesting's junit4.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Created] (LUCENE-4334) remove unnecessary ant-junit dependency

Robert Muir created LUCENE-4334:
---

 Summary: remove unnecessary ant-junit dependency
 Key: LUCENE-4334
 URL: https://issues.apache.org/jira/browse/LUCENE-4334
 Project: Lucene - Core
  Issue Type: Bug
  Components: general/build
Reporter: Robert Muir


We don't use this integration anymore for running tests: we use 
randomizedtesting's junit4.


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (LUCENE-4334) remove unnecessary ant-junit dependency


 [ 
https://issues.apache.org/jira/browse/LUCENE-4334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Muir updated LUCENE-4334:


Attachment: LUCENE-4334.patch

> remove unnecessary ant-junit dependency
> ---
>
> Key: LUCENE-4334
> URL: https://issues.apache.org/jira/browse/LUCENE-4334
> Project: Lucene - Core
>  Issue Type: Bug
>  Components: general/build
>Reporter: Robert Muir
> Attachments: LUCENE-4334.patch
>
>
> We don't use this integration anymore for running tests: we use 
> randomizedtesting's junit4.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-4332) Integrate PiTest mutation coverage tool into build