[jira] [Updated] (SOLR-4143) setRequestHandler - option to not set qt parameter

2012-12-19 Thread Shawn Heisey (JIRA)

 [ 
https://issues.apache.org/jira/browse/SOLR-4143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shawn Heisey updated SOLR-4143:
---

Attachment: SOLR-4143-alternate.patch

Deleted the most recent trunk and 4x alternate patches.  Adding a new alternate 
patch, updated to latest trunk and including an incomplete addition to 
CHANGES.txt that will need final tweaking before commit.

> setRequestHandler - option to not set qt parameter
> --
>
> Key: SOLR-4143
> URL: https://issues.apache.org/jira/browse/SOLR-4143
> Project: Solr
>  Issue Type: Improvement
>  Components: clients - java
>Affects Versions: 4.0
> Environment: solr-impl4.1-SNAPSHOT 1416639M - ncindex - 
> 2012-12-03 12:54:38
>Reporter: Shawn Heisey
> Fix For: 4.1, 5.0
>
> Attachments: SOLR-4143-alternate.patch, SOLR-4143-alternate.patch, 
> SOLR-4143-alternate-testsfailing.patch, SOLR-4143.patch, SOLR-4143.patch, 
> SOLR-4143.patch, SOLR-4143.patch, SOLR-4143.patch
>
>
> The setRequestHandler method does what I expect in one way - it changes the 
> URL path from /select to the String argument.  It is however doing something 
> that I did not expect, which is setting the qt parameter on the query as 
> well.  Here is the code:
> private static final String PING_HANDLER = "/admin/ping";
> query.setRequestHandler(PING_HANDLER);
> This is resulting in the following exception being logged in my Solr 3.5.0 
> servers.  I am not including the whole exception, because this issue is for 
> SolrJ 4, not Solr 3.5, and the 3.5 version is working as it was designed.
> {code}
> Dec 3, 2012 4:04:01 PM org.apache.solr.common.SolrException log
> SEVERE: org.apache.solr.common.SolrException: Cannot execute the 
> PingRequestHandler recursively
> at 
> org.apache.solr.handler.PingRequestHandler.handleRequestBody(PingRequestHandler.java:60)
> at 
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:208)
> {code}
> I believe it would be useful to include an alternate setRequestHandler method 
> that includes a boolean option deciding whether or not to also set the qt 
> parameter.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



[jira] [Updated] (SOLR-4143) setRequestHandler - option to not set qt parameter

2012-12-19 Thread Shawn Heisey (JIRA)

 [ 
https://issues.apache.org/jira/browse/SOLR-4143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shawn Heisey updated SOLR-4143:
---

Attachment: (was: SOLR-4143-alternate-trunk.patch)

> setRequestHandler - option to not set qt parameter
> --
>
> Key: SOLR-4143
> URL: https://issues.apache.org/jira/browse/SOLR-4143
> Project: Solr
>  Issue Type: Improvement
>  Components: clients - java
>Affects Versions: 4.0
> Environment: solr-impl4.1-SNAPSHOT 1416639M - ncindex - 
> 2012-12-03 12:54:38
>Reporter: Shawn Heisey
> Fix For: 4.1, 5.0
>
> Attachments: SOLR-4143-alternate.patch, 
> SOLR-4143-alternate-testsfailing.patch, SOLR-4143.patch, SOLR-4143.patch, 
> SOLR-4143.patch, SOLR-4143.patch, SOLR-4143.patch
>
>
> The setRequestHandler method does what I expect in one way - it changes the 
> URL path from /select to the String argument.  It is however doing something 
> that I did not expect, which is setting the qt parameter on the query as 
> well.  Here is the code:
> private static final String PING_HANDLER = "/admin/ping";
> query.setRequestHandler(PING_HANDLER);
> This is resulting in the following exception being logged in my Solr 3.5.0 
> servers.  I am not including the whole exception, because this issue is for 
> SolrJ 4, not Solr 3.5, and the 3.5 version is working as it was designed.
> {code}
> Dec 3, 2012 4:04:01 PM org.apache.solr.common.SolrException log
> SEVERE: org.apache.solr.common.SolrException: Cannot execute the 
> PingRequestHandler recursively
> at 
> org.apache.solr.handler.PingRequestHandler.handleRequestBody(PingRequestHandler.java:60)
> at 
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:208)
> {code}
> I believe it would be useful to include an alternate setRequestHandler method 
> that includes a boolean option deciding whether or not to also set the qt 
> parameter.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



[jira] [Updated] (SOLR-4143) setRequestHandler - option to not set qt parameter

2012-12-19 Thread Shawn Heisey (JIRA)

 [ 
https://issues.apache.org/jira/browse/SOLR-4143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shawn Heisey updated SOLR-4143:
---

Attachment: (was: SOLR-4143-alternate-4x.patch)

> setRequestHandler - option to not set qt parameter
> --
>
> Key: SOLR-4143
> URL: https://issues.apache.org/jira/browse/SOLR-4143
> Project: Solr
>  Issue Type: Improvement
>  Components: clients - java
>Affects Versions: 4.0
> Environment: solr-impl4.1-SNAPSHOT 1416639M - ncindex - 
> 2012-12-03 12:54:38
>Reporter: Shawn Heisey
> Fix For: 4.1, 5.0
>
> Attachments: SOLR-4143-alternate.patch, 
> SOLR-4143-alternate-testsfailing.patch, SOLR-4143.patch, SOLR-4143.patch, 
> SOLR-4143.patch, SOLR-4143.patch, SOLR-4143.patch
>
>
> The setRequestHandler method does what I expect in one way - it changes the 
> URL path from /select to the String argument.  It is however doing something 
> that I did not expect, which is setting the qt parameter on the query as 
> well.  Here is the code:
> private static final String PING_HANDLER = "/admin/ping";
> query.setRequestHandler(PING_HANDLER);
> This is resulting in the following exception being logged in my Solr 3.5.0 
> servers.  I am not including the whole exception, because this issue is for 
> SolrJ 4, not Solr 3.5, and the 3.5 version is working as it was designed.
> {code}
> Dec 3, 2012 4:04:01 PM org.apache.solr.common.SolrException log
> SEVERE: org.apache.solr.common.SolrException: Cannot execute the 
> PingRequestHandler recursively
> at 
> org.apache.solr.handler.PingRequestHandler.handleRequestBody(PingRequestHandler.java:60)
> at 
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:208)
> {code}
> I believe it would be useful to include an alternate setRequestHandler method 
> that includes a boolean option deciding whether or not to also set the qt 
> parameter.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



[jira] [Commented] (SOLR-1337) Spans and Payloads Query Support

2012-12-19 Thread Dmitry Kan (JIRA)

[ 
https://issues.apache.org/jira/browse/SOLR-1337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13536838#comment-13536838
 ] 

Dmitry Kan commented on SOLR-1337:
--

[Jan Høydahl] at Lucene query parser level. New token FUZZY_SLOP_SHARP (name 
isn't probably the best sound, but can be changed) has been introduced in the 
QueryParser.jj and supportive code implemented. The syntax is same as that of ~ 
operator, i.e. "term1 term2 ... termn" #slope.

> Spans and Payloads Query Support
> 
>
> Key: SOLR-1337
> URL: https://issues.apache.org/jira/browse/SOLR-1337
> Project: Solr
>  Issue Type: New Feature
>Reporter: Grant Ingersoll
>Assignee: Grant Ingersoll
> Fix For: 4.1
>
>
> It would be really nice to have query side support for: Spans and Payloads.  
> The main ingredient missing at this point is QueryParser support and a output 
> format for the spans and the payload spans.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



Override fq via local params?

2012-12-19 Thread Dawid Weiss
Attention Solr developers :) Is this something that should be possible
(and doesn't work) or is there a misconception somewhere? Thanks for
your help in advance; quoting:

I have tested on a range of SOLR releases from 1.6.0 to 3.6.0 and all
exhibit the same behavior, which is:

If I add a default filter query to the search hander, in this case
Section_Title:description


   
 
   explicit
   10
   DRECONTENT
   Section_Title:description
 

 Then I am not able to override this default with LocalParams, for
example the request to filter query on a different field is ignored

{!fq=Section_Title:summary}space

However, the same filer query does work if I put it in the URL

http://localhost:8983/solr/select/?q={!fq%3DSection_Title%3Asummary}space&version=2.2&start=0&rows=10&indent=on&fq=Section_Title:summary

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



[jira] [Commented] (LUCENE-4639) Improving _TestUtil.getTempDir

2012-12-19 Thread Shai Erera (JIRA)

[ 
https://issues.apache.org/jira/browse/LUCENE-4639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13536829#comment-13536829
 ] 

Shai Erera commented on LUCENE-4639:


Here's a test which reproduces the Access is denied exception on Windows, if 
run with the same seed (pick any seed, I used: )

{code}
  @Test
  public void testAccessDenied() throws Exception {
_TestUtil.getTempDir("testAccessDenied");
fail("msg");
  }
{code}

I cannot actually commit that test, but if you run it twice on Windows (w/ the 
same seed), you will hit the Access is denied.
However it passes with the changes to getTempDir (calling mkdir()). I will 
verify that no test breaks b/c getTempDir now creates the directory.

> Improving _TestUtil.getTempDir
> --
>
> Key: LUCENE-4639
> URL: https://issues.apache.org/jira/browse/LUCENE-4639
> Project: Lucene - Core
>  Issue Type: Bug
>Reporter: Shai Erera
>Assignee: Shai Erera
>Priority: Minor
>
> Spinoff from here: 
> http://lucene.472066.n3.nabble.com/TestUtil-getTempFile-may-fail-on-quot-Access-Denied-quot-td4028048.html.
> _TestUtil.getTempDir uses createTempFile and then deletes the file. While 
> this usually works, if someone runs tests by multiple JVMs and does not 
> ensure each JVM gets an isolated temp.dir to work in, that my result in two 
> JVMs sharing the same directory.
> Also, on Windows, if you call getTempDir on an existing directory, you get an 
> "Access is denied" exception.
> Dawid proposed a simple solution to just call mkdirs() continuously until 
> success. I'd like to try that.
> Also, I think that genTempFile could use some house cleaning, e.g.:
> * tempFileLocker can be just an Object instance? Why do we need a class?
> * If we initialize counter and counterBase in a static clause, we can avoid 
> checking if counter==0 as well as passing Random to genTempFile (that will 
> remove any suspicion that it does anything randomly)
> ** Also, instead of synchronizing on tempFileLocker, can we just use 
> AtomicInteger for the counter?
> I'll modify getTempDir first. It documents "does not create the directory", I 
> want to make sure no test fails due that.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



[jira] [Created] (LUCENE-4639) Improving _TestUtil.getTempDir

2012-12-19 Thread Shai Erera (JIRA)
Shai Erera created LUCENE-4639:
--

 Summary: Improving _TestUtil.getTempDir
 Key: LUCENE-4639
 URL: https://issues.apache.org/jira/browse/LUCENE-4639
 Project: Lucene - Core
  Issue Type: Bug
Reporter: Shai Erera
Assignee: Shai Erera
Priority: Minor


Spinoff from here: 
http://lucene.472066.n3.nabble.com/TestUtil-getTempFile-may-fail-on-quot-Access-Denied-quot-td4028048.html.

_TestUtil.getTempDir uses createTempFile and then deletes the file. While this 
usually works, if someone runs tests by multiple JVMs and does not ensure each 
JVM gets an isolated temp.dir to work in, that my result in two JVMs sharing 
the same directory.

Also, on Windows, if you call getTempDir on an existing directory, you get an 
"Access is denied" exception.

Dawid proposed a simple solution to just call mkdirs() continuously until 
success. I'd like to try that.

Also, I think that genTempFile could use some house cleaning, e.g.:
* tempFileLocker can be just an Object instance? Why do we need a class?
* If we initialize counter and counterBase in a static clause, we can avoid 
checking if counter==0 as well as passing Random to genTempFile (that will 
remove any suspicion that it does anything randomly)
** Also, instead of synchronizing on tempFileLocker, can we just use 
AtomicInteger for the counter?

I'll modify getTempDir first. It documents "does not create the directory", I 
want to make sure no test fails due that.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



[jira] [Assigned] (SOLR-2976) stats.facet no longer works on single valued trie fields that don't use precision step

2012-12-19 Thread Ryan McKinley (JIRA)

 [ 
https://issues.apache.org/jira/browse/SOLR-2976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ryan McKinley reassigned SOLR-2976:
---

Assignee: (was: Ryan McKinley)

did not know this was assigned to me  i'll step out since I don't fully 
understand what is happening

> stats.facet no longer works on single valued trie fields that don't use 
> precision step
> --
>
> Key: SOLR-2976
> URL: https://issues.apache.org/jira/browse/SOLR-2976
> Project: Solr
>  Issue Type: Bug
>Affects Versions: 3.5
>Reporter: Hoss Man
> Attachments: SOLR-2976_3.4_test.patch, SOLR-2976.patch
>
>
> As reported on the mailing list, 3.5 introduced a regression that prevents 
> single valued Trie fields that don't use precision steps (to add course 
> grained terms) from being used in stats.facet.
> two immediately obvious problems...
> 1) in 3.5 the stats component is checking if isTokenzed() is true for the 
> field type (which is probably wise) but regardless of the precisionStep used, 
> TrieField.isTokenized is hardcoded to return true
> 2) the 3.5 stats faceting will fail if the FieldType is multivalued - it 
> doesn't check if the SchemaField is configured to be single valued 
> (overriding the FieldType)
> so even if a user has something like this in their schema...
> {code}
>  omitNorms="true" />
>  multiValued="false" />
> {code}
> ...stats.facet will not work.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



Re: VOTE: release 3.6.2

2012-12-19 Thread Andi Vajda


+1

I prepared a PyLucene 3.6.2 release from the tip of the lucene_solr_3_6 
branch and all tests passed.


Andi..

On Wed, 19 Dec 2012, Robert Muir wrote:


Hello everyone, given the bug Mike fixed in
https://issues.apache.org/jira/browse/LUCENE-4635, I think we should
push out a 3.6.2 bugfix release with the fix.
Tom (the reporter of the bug) confirmed his checkindex passed
successfully last night, and besides that we have a few other nice
bugfixes there.

Artifacts are here: http://s.apache.org/lusolr362rc0

If you want to use the smoketester, of course you must run it from the
lucene_solr_3_6 branch. This passes for me on linux. I'm not sure it
works on other platforms.

Thanks,
Robert

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org




-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



[jira] [Commented] (SOLR-2976) stats.facet no longer works on single valued trie fields that don't use precision step

2012-12-19 Thread Yonik Seeley (JIRA)

[ 
https://issues.apache.org/jira/browse/SOLR-2976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13536703#comment-13536703
 ] 

Yonik Seeley commented on SOLR-2976:


Does this faceting limitation simply date back to before trie fields with 
precision step > 0 couldn't use the fieldCache?  Was introduced over 3 years 
ago (SOLR-1492).  Maybe we just forgot to remove it...

> stats.facet no longer works on single valued trie fields that don't use 
> precision step
> --
>
> Key: SOLR-2976
> URL: https://issues.apache.org/jira/browse/SOLR-2976
> Project: Solr
>  Issue Type: Bug
>Affects Versions: 3.5
>Reporter: Hoss Man
>Assignee: Ryan McKinley
> Attachments: SOLR-2976_3.4_test.patch, SOLR-2976.patch
>
>
> As reported on the mailing list, 3.5 introduced a regression that prevents 
> single valued Trie fields that don't use precision steps (to add course 
> grained terms) from being used in stats.facet.
> two immediately obvious problems...
> 1) in 3.5 the stats component is checking if isTokenzed() is true for the 
> field type (which is probably wise) but regardless of the precisionStep used, 
> TrieField.isTokenized is hardcoded to return true
> 2) the 3.5 stats faceting will fail if the FieldType is multivalued - it 
> doesn't check if the SchemaField is configured to be single valued 
> (overriding the FieldType)
> so even if a user has something like this in their schema...
> {code}
>  omitNorms="true" />
>  multiValued="false" />
> {code}
> ...stats.facet will not work.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



[jira] [Commented] (SOLR-2976) stats.facet no longer works on single valued trie fields that don't use precision step

2012-12-19 Thread Yonik Seeley (JIRA)

[ 
https://issues.apache.org/jira/browse/SOLR-2976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13536688#comment-13536688
 ] 

Yonik Seeley commented on SOLR-2976:


Verified the problem.
Just do: 
{code}
http://localhost:8983/solr/select?q=*:*&facet=true&facet.field=foo_ti
{code}

And then observe the log:
{code}
Dec 19, 2012 9:00:12 PM org.apache.solr.request.UnInvertedField 
INFO: UnInverted multi-valued field 
{field=foo_ti,memSize=4288,tindexSize=0,time=0,phase1=0,nTerms=0,bigTerms=0,termInstances=0,uses=0}
{code}


> stats.facet no longer works on single valued trie fields that don't use 
> precision step
> --
>
> Key: SOLR-2976
> URL: https://issues.apache.org/jira/browse/SOLR-2976
> Project: Solr
>  Issue Type: Bug
>Affects Versions: 3.5
>Reporter: Hoss Man
>Assignee: Ryan McKinley
> Attachments: SOLR-2976_3.4_test.patch, SOLR-2976.patch
>
>
> As reported on the mailing list, 3.5 introduced a regression that prevents 
> single valued Trie fields that don't use precision steps (to add course 
> grained terms) from being used in stats.facet.
> two immediately obvious problems...
> 1) in 3.5 the stats component is checking if isTokenzed() is true for the 
> field type (which is probably wise) but regardless of the precisionStep used, 
> TrieField.isTokenized is hardcoded to return true
> 2) the 3.5 stats faceting will fail if the FieldType is multivalued - it 
> doesn't check if the SchemaField is configured to be single valued 
> (overriding the FieldType)
> so even if a user has something like this in their schema...
> {code}
>  omitNorms="true" />
>  multiValued="false" />
> {code}
> ...stats.facet will not work.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



[jira] [Commented] (SOLR-4221) Custom sharding

2012-12-19 Thread Commit Tag Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/SOLR-4221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13536681#comment-13536681
 ] 

Commit Tag Bot commented on SOLR-4221:
--

[branch_4x commit] Yonik Seeley
http://svn.apache.org/viewvc?view=revision&revision=1424265

SOLR-4221: pick correct router for collection props


> Custom sharding
> ---
>
> Key: SOLR-4221
> URL: https://issues.apache.org/jira/browse/SOLR-4221
> Project: Solr
>  Issue Type: New Feature
>Reporter: Yonik Seeley
>
> Features to let users control everything about sharding/routing.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



[jira] [Commented] (SOLR-4221) Custom sharding

2012-12-19 Thread Commit Tag Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/SOLR-4221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13536676#comment-13536676
 ] 

Commit Tag Bot commented on SOLR-4221:
--

[trunk commit] Yonik Seeley
http://svn.apache.org/viewvc?view=revision&revision=1424263

SOLR-4221: pick correct router for collection props


> Custom sharding
> ---
>
> Key: SOLR-4221
> URL: https://issues.apache.org/jira/browse/SOLR-4221
> Project: Solr
>  Issue Type: New Feature
>Reporter: Yonik Seeley
>
> Features to let users control everything about sharding/routing.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



[jira] [Commented] (SOLR-4196) Untangle XML-specific nature of Config and Container classes

2012-12-19 Thread Mark Miller (JIRA)

[ 
https://issues.apache.org/jira/browse/SOLR-4196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13536651#comment-13536651
 ] 

Mark Miller commented on SOLR-4196:
---

Yeah, but since the xml is not very complex, most of it just seemed like grunt 
work, and perhaps not too much of it? I have not looked too closely in a while, 
but I would have approached it by just making a 'solr.xml' java object that I 
stuck the settings on and then pass it around same as the xml dom was passed 
around. Its just replacing one value holder for another right?

bootstrapConf for example, just wants to get a couple of the properties off the 
cores. This is probably going to be a common thing - getting both corecontainer 
and core level properties...that is what solr.xml is all about.

So like in the persist code I would make a class to represent a cores 
properties and then I'd make a class to represent the corecontainers 
properties. Then I'd compose these classes based on teh dir layout and configs 
and sys properties and end up with a core container object with n properties 
and y core objects with x properties. boostrapConf would get to easily pull the 
core properties it wants, and so would every other place that ends up needing 
to look at a cores properties. CoreContainer level properties would also be 
available to pass around.

> Untangle XML-specific nature of Config and Container classes
> 
>
> Key: SOLR-4196
> URL: https://issues.apache.org/jira/browse/SOLR-4196
> Project: Solr
>  Issue Type: Improvement
>  Components: Schema and Analysis
>Reporter: Erick Erickson
>Assignee: Erick Erickson
>Priority: Minor
> Fix For: 4.1, 5.0
>
>
> sub-task for SOLR-4083. If we're going to try to obsolete solr.xml, we need 
> to pull all of the specific XML processing out of Config and Container. 
> Currently, we refer to xpaths all over the place. This JIRA is about 
> providing a thunking layer to isolate the XML-esque nature of solr.xml and 
> allow a simple properties file to be used instead which will lead, 
> eventually, to solr.xml going away.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



[jira] [Commented] (SOLR-2976) stats.facet no longer works on single valued trie fields that don't use precision step

2012-12-19 Thread Robert Muir (JIRA)

[ 
https://issues.apache.org/jira/browse/SOLR-2976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13536644#comment-13536644
 ] 

Robert Muir commented on SOLR-2976:
---

Sorry Ryan, I didnt mean to assign myself. My browser just went psycho for a 
bit!

> stats.facet no longer works on single valued trie fields that don't use 
> precision step
> --
>
> Key: SOLR-2976
> URL: https://issues.apache.org/jira/browse/SOLR-2976
> Project: Solr
>  Issue Type: Bug
>Affects Versions: 3.5
>Reporter: Hoss Man
>Assignee: Ryan McKinley
> Attachments: SOLR-2976_3.4_test.patch, SOLR-2976.patch
>
>
> As reported on the mailing list, 3.5 introduced a regression that prevents 
> single valued Trie fields that don't use precision steps (to add course 
> grained terms) from being used in stats.facet.
> two immediately obvious problems...
> 1) in 3.5 the stats component is checking if isTokenzed() is true for the 
> field type (which is probably wise) but regardless of the precisionStep used, 
> TrieField.isTokenized is hardcoded to return true
> 2) the 3.5 stats faceting will fail if the FieldType is multivalued - it 
> doesn't check if the SchemaField is configured to be single valued 
> (overriding the FieldType)
> so even if a user has something like this in their schema...
> {code}
>  omitNorms="true" />
>  multiValued="false" />
> {code}
> ...stats.facet will not work.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



[jira] [Updated] (SOLR-2976) stats.facet no longer works on single valued trie fields that don't use precision step

2012-12-19 Thread Robert Muir (JIRA)

 [ 
https://issues.apache.org/jira/browse/SOLR-2976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Muir updated SOLR-2976:
--

Assignee: Ryan McKinley  (was: Robert Muir)

> stats.facet no longer works on single valued trie fields that don't use 
> precision step
> --
>
> Key: SOLR-2976
> URL: https://issues.apache.org/jira/browse/SOLR-2976
> Project: Solr
>  Issue Type: Bug
>Affects Versions: 3.5
>Reporter: Hoss Man
>Assignee: Ryan McKinley
> Attachments: SOLR-2976_3.4_test.patch, SOLR-2976.patch
>
>
> As reported on the mailing list, 3.5 introduced a regression that prevents 
> single valued Trie fields that don't use precision steps (to add course 
> grained terms) from being used in stats.facet.
> two immediately obvious problems...
> 1) in 3.5 the stats component is checking if isTokenzed() is true for the 
> field type (which is probably wise) but regardless of the precisionStep used, 
> TrieField.isTokenized is hardcoded to return true
> 2) the 3.5 stats faceting will fail if the FieldType is multivalued - it 
> doesn't check if the SchemaField is configured to be single valued 
> (overriding the FieldType)
> so even if a user has something like this in their schema...
> {code}
>  omitNorms="true" />
>  multiValued="false" />
> {code}
> ...stats.facet will not work.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



[jira] [Commented] (SOLR-2976) stats.facet no longer works on single valued trie fields that don't use precision step

2012-12-19 Thread Robert Muir (JIRA)

[ 
https://issues.apache.org/jira/browse/SOLR-2976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13536641#comment-13536641
 ] 

Robert Muir commented on SOLR-2976:
---

Yonik: Ill port over my test. But i think this is the piece of the code that 
causes it in SimpleFacets:
{code}
boolean multiToken = sf.multiValued() || ft.multiValuedFieldCache();

if (TrieField.getMainValuePrefix(ft) != null) {
  // A TrieField with multiple parts indexed per value... currently only
  // UnInvertedField can handle this case, so force it's use.
  enumMethod = false;
  multiToken = true;
}
{code}

This second "if" I think causes the problem? It only returns null if 
precisionStep is 0:
{code}
if (trie.precisionStep  == Integer.MAX_VALUE)
  return null;
{code}

> stats.facet no longer works on single valued trie fields that don't use 
> precision step
> --
>
> Key: SOLR-2976
> URL: https://issues.apache.org/jira/browse/SOLR-2976
> Project: Solr
>  Issue Type: Bug
>Affects Versions: 3.5
>Reporter: Hoss Man
>Assignee: Robert Muir
> Attachments: SOLR-2976_3.4_test.patch, SOLR-2976.patch
>
>
> As reported on the mailing list, 3.5 introduced a regression that prevents 
> single valued Trie fields that don't use precision steps (to add course 
> grained terms) from being used in stats.facet.
> two immediately obvious problems...
> 1) in 3.5 the stats component is checking if isTokenzed() is true for the 
> field type (which is probably wise) but regardless of the precisionStep used, 
> TrieField.isTokenized is hardcoded to return true
> 2) the 3.5 stats faceting will fail if the FieldType is multivalued - it 
> doesn't check if the SchemaField is configured to be single valued 
> (overriding the FieldType)
> so even if a user has something like this in their schema...
> {code}
>  omitNorms="true" />
>  multiValued="false" />
> {code}
> ...stats.facet will not work.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



[jira] [Assigned] (SOLR-2976) stats.facet no longer works on single valued trie fields that don't use precision step

2012-12-19 Thread Robert Muir (JIRA)

 [ 
https://issues.apache.org/jira/browse/SOLR-2976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Muir reassigned SOLR-2976:
-

Assignee: Robert Muir  (was: Ryan McKinley)

> stats.facet no longer works on single valued trie fields that don't use 
> precision step
> --
>
> Key: SOLR-2976
> URL: https://issues.apache.org/jira/browse/SOLR-2976
> Project: Solr
>  Issue Type: Bug
>Affects Versions: 3.5
>Reporter: Hoss Man
>Assignee: Robert Muir
> Attachments: SOLR-2976_3.4_test.patch, SOLR-2976.patch
>
>
> As reported on the mailing list, 3.5 introduced a regression that prevents 
> single valued Trie fields that don't use precision steps (to add course 
> grained terms) from being used in stats.facet.
> two immediately obvious problems...
> 1) in 3.5 the stats component is checking if isTokenzed() is true for the 
> field type (which is probably wise) but regardless of the precisionStep used, 
> TrieField.isTokenized is hardcoded to return true
> 2) the 3.5 stats faceting will fail if the FieldType is multivalued - it 
> doesn't check if the SchemaField is configured to be single valued 
> (overriding the FieldType)
> so even if a user has something like this in their schema...
> {code}
>  omitNorms="true" />
>  multiValued="false" />
> {code}
> ...stats.facet will not work.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



[jira] [Commented] (SOLR-2976) stats.facet no longer works on single valued trie fields that don't use precision step

2012-12-19 Thread Yonik Seeley (JIRA)

[ 
https://issues.apache.org/jira/browse/SOLR-2976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13536637#comment-13536637
 ] 

Yonik Seeley commented on SOLR-2976:


bq. Its also the case that if precisionStep != 0, faceting on a single-valued 
numeric field builds an UninvertedField (which is unnecessary, as the 
fieldcache can handle this just fine).

That shouldn't be the case.  The current algorithms check Solr's idea of single 
valued vs multi-valued - it doesn't matter how many tokens are indexed per 
value.

> stats.facet no longer works on single valued trie fields that don't use 
> precision step
> --
>
> Key: SOLR-2976
> URL: https://issues.apache.org/jira/browse/SOLR-2976
> Project: Solr
>  Issue Type: Bug
>Affects Versions: 3.5
>Reporter: Hoss Man
>Assignee: Ryan McKinley
> Attachments: SOLR-2976_3.4_test.patch, SOLR-2976.patch
>
>
> As reported on the mailing list, 3.5 introduced a regression that prevents 
> single valued Trie fields that don't use precision steps (to add course 
> grained terms) from being used in stats.facet.
> two immediately obvious problems...
> 1) in 3.5 the stats component is checking if isTokenzed() is true for the 
> field type (which is probably wise) but regardless of the precisionStep used, 
> TrieField.isTokenized is hardcoded to return true
> 2) the 3.5 stats faceting will fail if the FieldType is multivalued - it 
> doesn't check if the SchemaField is configured to be single valued 
> (overriding the FieldType)
> so even if a user has something like this in their schema...
> {code}
>  omitNorms="true" />
>  multiValued="false" />
> {code}
> ...stats.facet will not work.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



Re: VOTE: release 3.6.2

2012-12-19 Thread Michael McCandless
I don't think these javadocs warnings require a re-spin so +1 to
release current artifacts.

Mike McCandless

http://blog.mikemccandless.com


On Wed, Dec 19, 2012 at 3:46 PM, Michael McCandless
 wrote:
> On Wed, Dec 19, 2012 at 2:01 PM, Robert Muir  wrote:
>> On Wed, Dec 19, 2012 at 1:48 PM, Michael McCandless
>>  wrote:
>>> Hmm my smoke test run was angry about this javadocs warning:
>>>
>>>   [javadoc] 
>>> /l/36x/tmp/unpack/apache-solr-3.6.2/lucene/contrib/facet/src/java/org/apache/lucene/facet/taxonomy/writercache/lru/NameIntCacheLRU.java:76:
>>> warning - @return tag has no arguments.
>>>
>>> It hit this when running javadocs with 1.7.0_07.
>>
>> Thanks Mike... can you fix? I have no idea why different versions of
>> java7 have different levels of pickiness.
>>
>> This is no issue for e.g. 4.x+, because we have the eclipse checker as
>> part of our build which fails on this. But for 3.x we don't have as
>> many tools unfortunately.
>>
>> I dont think we should put a lot of effort into this: but when
>> backporting bugfixes to old branches like this please be really
>> careful about this stuff.
>
> OK I committed that fix and a couple other javadocs warnings.
>
> Thanks for spinning 3.6.2 Robert!
>
> Mike McCandless
>
> http://blog.mikemccandless.com

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



[jira] [Commented] (SOLR-2976) stats.facet no longer works on single valued trie fields that don't use precision step

2012-12-19 Thread Robert Muir (JIRA)

[ 
https://issues.apache.org/jira/browse/SOLR-2976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13536575#comment-13536575
 ] 

Robert Muir commented on SOLR-2976:
---

Its also the case that if precisionStep != 0, faceting on a single-valued 
numeric field builds an UninvertedField (which is unnecessary, as the 
fieldcache can handle this just fine).


> stats.facet no longer works on single valued trie fields that don't use 
> precision step
> --
>
> Key: SOLR-2976
> URL: https://issues.apache.org/jira/browse/SOLR-2976
> Project: Solr
>  Issue Type: Bug
>Affects Versions: 3.5
>Reporter: Hoss Man
>Assignee: Ryan McKinley
> Attachments: SOLR-2976_3.4_test.patch, SOLR-2976.patch
>
>
> As reported on the mailing list, 3.5 introduced a regression that prevents 
> single valued Trie fields that don't use precision steps (to add course 
> grained terms) from being used in stats.facet.
> two immediately obvious problems...
> 1) in 3.5 the stats component is checking if isTokenzed() is true for the 
> field type (which is probably wise) but regardless of the precisionStep used, 
> TrieField.isTokenized is hardcoded to return true
> 2) the 3.5 stats faceting will fail if the FieldType is multivalued - it 
> doesn't check if the SchemaField is configured to be single valued 
> (overriding the FieldType)
> so even if a user has something like this in their schema...
> {code}
>  omitNorms="true" />
>  multiValued="false" />
> {code}
> ...stats.facet will not work.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



[jira] [Commented] (SOLR-3969) Admin dashboard -- include cache/buffer memory utilization

2012-12-19 Thread Shawn Heisey (JIRA)

[ 
https://issues.apache.org/jira/browse/SOLR-3969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13536556#comment-13536556
 ] 

Shawn Heisey commented on SOLR-3969:


bq. Shawn, perhaps you can provide an patch for your favorite OS?

If I can figure out how, sure.  I currently have no idea how the existing graph 
data is gathered and displayed.  It'll be a while before I can get to it, too.

> Admin dashboard -- include cache/buffer memory utilization
> --
>
> Key: SOLR-3969
> URL: https://issues.apache.org/jira/browse/SOLR-3969
> Project: Solr
>  Issue Type: Improvement
>  Components: web gui
>Affects Versions: 4.0
> Environment: Linux bigindy5 2.6.32-279.9.1.el6.centos.plus.x86_64 #1 
> SMP Wed Sep 26 03:52:55 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux
> java version "1.7.0_07"
> Java(TM) SE Runtime Environment (build 1.7.0_07-b10)
> Java HotSpot(TM) 64-Bit Server VM (build 23.3-b01, mixed mode)
>Reporter: Shawn Heisey
>Priority: Minor
> Fix For: 4.1
>
>
> The admin dashboard includes memory utilization, but there is no way on that 
> screen to see how much memory is being used for cache and buffers -- two 
> additional numbers shown by top.
> The most visually appealing way to display this would be to break the 
> Physical Memory graph into multiple colors, but if that's not practical, more 
> graphs could be added.  I'm not even sure it's possible, but if it is, it 
> would make a great addition.  Hopefully there would also be a cross-platform 
> way of doing it so that the major platforms could all be supported.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



Re: VOTE: release 3.6.2

2012-12-19 Thread Tom Burton-West
Thanks Robert,

Ok, I can see that logic.   People who want the "new feature" can just
apply the patch.

Tom

On Wed, Dec 19, 2012 at 5:53 PM, Robert Muir  wrote:

> On Wed, Dec 19, 2012 at 5:50 PM, Tom Burton-West 
> wrote:
> > Hi Robert,
> >
> > Would it be possible to fold in also LUCENE-4286?
> > I don't see the 3.6 backport listed in the JIRA issue, but it would be
> nice
> > to have that flag available for people still on the 3.6.x branch.
> >
>
> I think i would prefer not to: even though its just a "new
> option/feature", 3.6.x is basically in bugfix mode.
>
> I know this isn't great for people that want the feature, but i feel
> we should be really careful with these bugfix releases myself.
>
> Terms index bug is a whole different story though!
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>
>


Re: VOTE: release 3.6.2

2012-12-19 Thread Robert Muir
On Wed, Dec 19, 2012 at 5:50 PM, Tom Burton-West  wrote:
> Hi Robert,
>
> Would it be possible to fold in also LUCENE-4286?
> I don't see the 3.6 backport listed in the JIRA issue, but it would be nice
> to have that flag available for people still on the 3.6.x branch.
>

I think i would prefer not to: even though its just a "new
option/feature", 3.6.x is basically in bugfix mode.

I know this isn't great for people that want the feature, but i feel
we should be really careful with these bugfix releases myself.

Terms index bug is a whole different story though!

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



Re: VOTE: release 3.6.2

2012-12-19 Thread Tom Burton-West
Hi Robert,

Would it be possible to fold in also LUCENE-4286?
I don't see the 3.6 backport listed in the JIRA issue, but it would be nice
to have that flag available for people still on the 3.6.x branch.

Tom

On Wed, Dec 19, 2012 at 3:46 PM, Michael McCandless <
luc...@mikemccandless.com> wrote:

> On Wed, Dec 19, 2012 at 2:01 PM, Robert Muir  wrote:
> > On Wed, Dec 19, 2012 at 1:48 PM, Michael McCandless
> >  wrote:
> >> Hmm my smoke test run was angry about this javadocs warning:
> >>
> >>   [javadoc]
> /l/36x/tmp/unpack/apache-solr-3.6.2/lucene/contrib/facet/src/java/org/apache/lucene/facet/taxonomy/writercache/lru/NameIntCacheLRU.java:76:
> >> warning - @return tag has no arguments.
> >>
> >> It hit this when running javadocs with 1.7.0_07.
> >
> > Thanks Mike... can you fix? I have no idea why different versions of
> > java7 have different levels of pickiness.
> >
> > This is no issue for e.g. 4.x+, because we have the eclipse checker as
> > part of our build which fails on this. But for 3.x we don't have as
> > many tools unfortunately.
> >
> > I dont think we should put a lot of effort into this: but when
> > backporting bugfixes to old branches like this please be really
> > careful about this stuff.
>
> OK I committed that fix and a couple other javadocs warnings.
>
> Thanks for spinning 3.6.2 Robert!
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>
>


[jira] [Commented] (SOLR-2976) stats.facet no longer works on single valued trie fields that don't use precision step

2012-12-19 Thread yuanyun.cn (JIRA)

[ 
https://issues.apache.org/jira/browse/SOLR-2976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13536499#comment-13536499
 ] 

yuanyun.cn commented on SOLR-2976:
--

Hit this problem recently.
To fix this problem. and support facet search on this field, I have to create 
another field,  with precisionStep="2147483647"(Integer,MAX_VALUE), this is not 
good, as it takes more disk size, and it's hard to explain to customers why we 
need this field.

So do we have plan to fix this problem? Thanks :)

> stats.facet no longer works on single valued trie fields that don't use 
> precision step
> --
>
> Key: SOLR-2976
> URL: https://issues.apache.org/jira/browse/SOLR-2976
> Project: Solr
>  Issue Type: Bug
>Affects Versions: 3.5
>Reporter: Hoss Man
>Assignee: Ryan McKinley
> Attachments: SOLR-2976_3.4_test.patch, SOLR-2976.patch
>
>
> As reported on the mailing list, 3.5 introduced a regression that prevents 
> single valued Trie fields that don't use precision steps (to add course 
> grained terms) from being used in stats.facet.
> two immediately obvious problems...
> 1) in 3.5 the stats component is checking if isTokenzed() is true for the 
> field type (which is probably wise) but regardless of the precisionStep used, 
> TrieField.isTokenized is hardcoded to return true
> 2) the 3.5 stats faceting will fail if the FieldType is multivalued - it 
> doesn't check if the SchemaField is configured to be single valued 
> (overriding the FieldType)
> so even if a user has something like this in their schema...
> {code}
>  omitNorms="true" />
>  multiValued="false" />
> {code}
> ...stats.facet will not work.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



[jira] [Commented] (SOLR-4204) Make SolrCloud tests more friendly to FreeBSD blackhole 2 environments.

2012-12-19 Thread Mark Miller (JIRA)

[ 
https://issues.apache.org/jira/browse/SOLR-4204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13536494#comment-13536494
 ] 

Mark Miller commented on SOLR-4204:
---

Not there yet, but very close!

> Make SolrCloud tests more friendly to FreeBSD blackhole 2 environments.
> ---
>
> Key: SOLR-4204
> URL: https://issues.apache.org/jira/browse/SOLR-4204
> Project: Solr
>  Issue Type: Improvement
>  Components: SolrCloud
>Reporter: Mark Miller
>Assignee: Mark Miller
> Fix For: 4.1, 5.0
>
> Attachments: SOLR-4204.patch
>
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



Re: [JENKINS-MAVEN] Lucene-Solr-Maven-trunk #712: POMs out of sync

2012-12-19 Thread Mark Miller
Only 1 test fail on the maven build!

And I can easily duplicate on my FreeBSD env!

I'm so tired that I don't know that I will fix this tonight, but soon!

- Mark

On Dec 19, 2012, at 4:58 PM, Apache Jenkins Server  
wrote:

> Build: https://builds.apache.org/job/Lucene-Solr-Maven-trunk/712/
> 
> 1 tests failed.
> REGRESSION:  
> org.apache.solr.handler.component.DistributedMLTComponentTest.testDistribSearch
> 
> Error Message:
> Server at http://127.0.0.1:20964/ua_z/hz returned non ok status:500, 
> message:Server Error
> 
> Stack Trace:
> org.apache.solr.common.SolrException: Server at 
> http://127.0.0.1:20964/ua_z/hz returned non ok status:500, message:Server 
> Error
>   at 
> __randomizedtesting.SeedInfo.seed([8761BA8083FBD7E1:6873498F4A4B7DD]:0)
>   at 
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:372)
>   at 
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:181)
>   at 
> org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:90)
>   at org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:301)
>   at 
> org.apache.solr.BaseDistributedSearchTestCase.queryServer(BaseDistributedSearchTestCase.java:470)
>   at 
> org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearchTestCase.java:493)
>   at 
> org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearchTestCase.java:475)
>   at 
> org.apache.solr.handler.component.DistributedMLTComponentTest.doTest(DistributedMLTComponentTest.java:119)
>   at 
> org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:794)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:616)
>   at 
> com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
>   at 
> com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
>   at 
> com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
>   at 
> com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
>   at 
> com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
>   at 
> com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
>   at 
> org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
>   at 
> org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
>   at 
> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>   at 
> com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
>   at 
> org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
>   at 
> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
>   at 
> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>   at 
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>   at 
> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
>   at 
> com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
>   at 
> com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
>   at 
> com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
>   at 
> com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
>   at 
> com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
>   at 
> com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
>   at 
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>   at 
> com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
>   at 
> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>   at 
> org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
>   at 
> com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
>   at 
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoS

[JENKINS-MAVEN] Lucene-Solr-Maven-trunk #712: POMs out of sync

2012-12-19 Thread Apache Jenkins Server
Build: https://builds.apache.org/job/Lucene-Solr-Maven-trunk/712/

1 tests failed.
REGRESSION:  
org.apache.solr.handler.component.DistributedMLTComponentTest.testDistribSearch

Error Message:
Server at http://127.0.0.1:20964/ua_z/hz returned non ok status:500, 
message:Server Error

Stack Trace:
org.apache.solr.common.SolrException: Server at http://127.0.0.1:20964/ua_z/hz 
returned non ok status:500, message:Server Error
at 
__randomizedtesting.SeedInfo.seed([8761BA8083FBD7E1:6873498F4A4B7DD]:0)
at 
org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:372)
at 
org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:181)
at 
org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:90)
at org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:301)
at 
org.apache.solr.BaseDistributedSearchTestCase.queryServer(BaseDistributedSearchTestCase.java:470)
at 
org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearchTestCase.java:493)
at 
org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearchTestCase.java:475)
at 
org.apache.solr.handler.component.DistributedMLTComponentTest.doTest(DistributedMLTComponentTest.java:119)
at 
org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:794)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at 
org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.Test

[jira] [Commented] (LUCENE-4246) Fix IndexWriter.close() to not commit or wait for pending merges

2012-12-19 Thread Shai Erera (JIRA)

[ 
https://issues.apache.org/jira/browse/LUCENE-4246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13536424#comment-13536424
 ] 

Shai Erera commented on LUCENE-4246:


Today a simple app just calls close(). Now you propose that it will call 
close(), but try-catch two exceptions RunningMerges and UncommittedChanges. 
Then the poor developer will need to decide what to do with them .. should he 
call rollback()? should he call commit()? then close(), only to try-catch those 
two ex, now documenting "it's fine now"?

While the app today can choose to call waitForMerges, or close(false), or 
commit, then close.

The changes will make the code more verbose and I'm afraid will raise the bar 
for simple apps. Aren't you the one that always pushes for simple, more 
approachable API?

And with that solution, what would rollback() do? Unless we change rollback to 
not close, it's just an alias, as Robert put it, to close, only it doesn't 
throw the two new exceptions?

> Fix IndexWriter.close() to not commit or wait for pending merges
> 
>
> Key: LUCENE-4246
> URL: https://issues.apache.org/jira/browse/LUCENE-4246
> Project: Lucene - Core
>  Issue Type: Bug
>Reporter: Robert Muir
>Assignee: Robert Muir
> Fix For: 4.1
>
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



Re: _TestUtil.getTempFile may fail on "Access Denied"

2012-12-19 Thread Shai Erera
I prefer that LuceneTestCase will not assume anything about how it's being
run. E.g. if someone writes a script, not through the runner, which forks
JVMs, it would be good if LTC was still safe.

I don't mind giving mkdirs() a try. I think that it's better than
createNewFile() -> delete -> mkdirs().

Shai


On Wed, Dec 19, 2012 at 10:54 PM, Dawid Weiss
wrote:

> > Well, mkdirs() does state that it returns true iff all path components
> were
> > created, false otherwise. It doesn't say anything about atomicity, which
> > makes me nervous ...
>
> I'd assume it's the filesystem that guarantees this. Or so I hope.
>
> > But if others think that's fine, then this will be more resilient to
> > multiple JVMs sharing the same temp.dir.
>
> But this should never be the case -- we're physically splitting cwd's
> of all forked JVMs and any temporary folders should be created inside
> those (the security manager and Uwe are making sure you can't write
> out of your assigned directory)?
>
> D.
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>
>


Re: _TestUtil.getTempFile may fail on "Access Denied"

2012-12-19 Thread Dawid Weiss
> Well, mkdirs() does state that it returns true iff all path components were
> created, false otherwise. It doesn't say anything about atomicity, which
> makes me nervous ...

I'd assume it's the filesystem that guarantees this. Or so I hope.

> But if others think that's fine, then this will be more resilient to
> multiple JVMs sharing the same temp.dir.

But this should never be the case -- we're physically splitting cwd's
of all forked JVMs and any temporary folders should be created inside
those (the security manager and Uwe are making sure you can't write
out of your assigned directory)?

D.

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



Re: VOTE: release 3.6.2

2012-12-19 Thread Michael McCandless
On Wed, Dec 19, 2012 at 2:01 PM, Robert Muir  wrote:
> On Wed, Dec 19, 2012 at 1:48 PM, Michael McCandless
>  wrote:
>> Hmm my smoke test run was angry about this javadocs warning:
>>
>>   [javadoc] 
>> /l/36x/tmp/unpack/apache-solr-3.6.2/lucene/contrib/facet/src/java/org/apache/lucene/facet/taxonomy/writercache/lru/NameIntCacheLRU.java:76:
>> warning - @return tag has no arguments.
>>
>> It hit this when running javadocs with 1.7.0_07.
>
> Thanks Mike... can you fix? I have no idea why different versions of
> java7 have different levels of pickiness.
>
> This is no issue for e.g. 4.x+, because we have the eclipse checker as
> part of our build which fails on this. But for 3.x we don't have as
> many tools unfortunately.
>
> I dont think we should put a lot of effort into this: but when
> backporting bugfixes to old branches like this please be really
> careful about this stuff.

OK I committed that fix and a couple other javadocs warnings.

Thanks for spinning 3.6.2 Robert!

Mike McCandless

http://blog.mikemccandless.com

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



[jira] [Commented] (SOLR-4222) create custom sharded collection via collections API

2012-12-19 Thread Yonik Seeley (JIRA)

[ 
https://issues.apache.org/jira/browse/SOLR-4222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13536414#comment-13536414
 ] 

Yonik Seeley commented on SOLR-4222:


The following call currently results in nothing being done:
curl 
"http://localhost:8983/solr/admin/collections?action=CREATE&name=mycollection2";

This seems to be because the OverseerCollectionsProcessor sends out messages to 
nodes to create new cores, and as a side effect, the collection is created by 
one of those cores.  We need to be able to create a collection independent of 
any cores.


> create custom sharded collection via collections API
> 
>
> Key: SOLR-4222
> URL: https://issues.apache.org/jira/browse/SOLR-4222
> Project: Solr
>  Issue Type: Sub-task
>Reporter: Yonik Seeley
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



Re: _TestUtil.getTempFile may fail on "Access Denied"

2012-12-19 Thread Shai Erera
Well, mkdirs() does state that it returns true iff all path components were
created, false otherwise. It doesn't say anything about atomicity, which
makes me nervous ...

But if others think that's fine, then this will be more resilient to
multiple JVMs sharing the same temp.dir.
We can then document that getTempFile should be used for files only. You'll
also not be able to create a directory out of it, since you'll receive an
already created file ...

Shai


On Wed, Dec 19, 2012 at 10:31 PM, Dawid Weiss
wrote:

> This may be lame but why not generate a filename candidates in a loop
> and File.mkdir() instead until successful?
>
> D.
>
> On Wed, Dec 19, 2012 at 9:22 PM, Shai Erera  wrote:
> > Tests running in the same JVM will still create directories atomically
> > because genTempFile returns a unique file name (it synchronizes around
> > counter). The problem may be across JVMs, because e.g. P1 and P2, which
> > don't sync around the same counter, may create the same file name, one
> may
> > fail on the IOE. But now that the build sets a unique temp.dir per JVM,
> this
> > won't happen I think? Still, here's a patch which protects around that
> too,
> > even though it's a bit longer:
> >
> > Index:
> lucene/test-framework/src/java/org/apache/lucene/util/_TestUtil.java
> > ===
> > --- lucene/test-framework/src/java/org/apache/lucene/util/_TestUtil.java
> > (revision 1424071)
> > +++ lucene/test-framework/src/java/org/apache/lucene/util/_TestUtil.java
> > (working copy)
> > @@ -748,9 +748,19 @@
> >  // already exist. otherwise tests might not reproduce, depending on
> > when you last
> >  // ran 'ant clean'
> >
> >  final Random random = new
> > Random(RandomizedContext.current().getRandom().nextLong());
> > -do {
> > +while (true) {
> >
> >result = genTempFile(random, prefix, newSuffix, directory);
> > -} while (!result.createNewFile());
> > +  try {
> > +if (result.createNewFile()) break;
> > +  } catch (IOException e) {
> > +// if 'result' exists and is a directory, Windows throws this
> bogus
> > exception
> > +// specializing on the msg text, rather than ignoring any IOE,
> just
> > in case
> > +// createNewFile may throw a real IOE.
> > +if (!(e.getMessage().equalsIgnoreCase("access is deined"))) {
> > +  throw e;
> > +}
> > +  }
> > +}
> >  return result;
> >}
> >
> > Note though that if we don't rely on the isolation of every JVM, then
> > getTempDir is totally broken too. Because it first createNewFile, then
> > deletes it, so potentially two JVMs can successfully create the same file
> > (one after the other) and from there on share it !
> >
> > Which of the patches is better to commit? The simpler one, which relies
> on
> > JVM isolation, or the safer one?
> >
> > Shai
> >
> >
> > On Wed, Dec 19, 2012 at 9:42 PM, Michael McCandless
> >  wrote:
> >>
> >> There is some risk with your patch: if two tests try to create the
> >> same dir at the same time ... with the patch, one of the tests will
> >> create the dir, and the 2nd test will see it already exists and then
> >> use that directory illegally (ie, share it with the first test).  That
> >> will lead to very confusing test failures (if it happens).
> >>
> >> Ie, the idea of this API is atomicity: the file/dir did not exist
> >> before, and this caller succeeded in creating it, it returns true,
> >> meaning 1) the file/dir is newly created, and 2) is "private" to you.
> >>
> >> Mike McCandless
> >>
> >> http://blog.mikemccandless.com
> >>
> >> On Wed, Dec 19, 2012 at 1:57 PM, Shai Erera  wrote:
> >> > If nobody objects, I'd like to commit this simple fix. Currently, if a
> >> > test
> >> > fails, I cannot  re-run it w/ same seed unless I delete the directory
> >> > manually, or in the test itself (since CloseableFile doesn't delete
> the
> >> > dir
> >> > if the test failed).
> >> >
> >> > Shai
> >> >
> >> >
> >> > On Wed, Dec 19, 2012 at 5:15 PM, Shai Erera  wrote:
> >> >>>
> >> >>> I patched _TestUtil like so, and the test passes
> >> >>
> >> >>
> >> >> Sorry, I meant to say that if fails where I expect it to fail, not on
> >> >> Access is Denied.
> >> >>
> >> >> Shai
> >> >>
> >> >>
> >> >> On Wed, Dec 19, 2012 at 5:14 PM, Shai Erera 
> wrote:
> >> >>>
> >> >>> Hi
> >> >>>
> >> >>> When a test fails, the file system directories that it created are
> not
> >> >>> deleted.
> >> >>>
> >> >>> At least on Windows, when you run the test with the same seed again,
> >> >>> _TestUtil.getTempDir fails on  "Access Denied". The reason is that
> it
> >> >>> calls
> >> >>> file.createNewFile() on a directory. I don't know if it's specific
> to
> >> >>> Windows, but I tried this test with both J9 and Oracle JVMs:
> >> >>>
> >> >>>   @Test
> >> >>>   public void testAccessDenied() throws Exception {
> >> >>> _TestUtil.getTempDir("testAccessDenied").mkdirs();
> >> >>> fail("msg

[jira] [Created] (SOLR-4222) create custom sharded collection via collections API

2012-12-19 Thread Yonik Seeley (JIRA)
Yonik Seeley created SOLR-4222:
--

 Summary: create custom sharded collection via collections API
 Key: SOLR-4222
 URL: https://issues.apache.org/jira/browse/SOLR-4222
 Project: Solr
  Issue Type: Sub-task
Reporter: Yonik Seeley




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



[jira] [Created] (SOLR-4221) Custom sharding

2012-12-19 Thread Yonik Seeley (JIRA)
Yonik Seeley created SOLR-4221:
--

 Summary: Custom sharding
 Key: SOLR-4221
 URL: https://issues.apache.org/jira/browse/SOLR-4221
 Project: Solr
  Issue Type: New Feature
Reporter: Yonik Seeley


Features to let users control everything about sharding/routing.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



Re: _TestUtil.getTempFile may fail on "Access Denied"

2012-12-19 Thread Dawid Weiss
This may be lame but why not generate a filename candidates in a loop
and File.mkdir() instead until successful?

D.

On Wed, Dec 19, 2012 at 9:22 PM, Shai Erera  wrote:
> Tests running in the same JVM will still create directories atomically
> because genTempFile returns a unique file name (it synchronizes around
> counter). The problem may be across JVMs, because e.g. P1 and P2, which
> don't sync around the same counter, may create the same file name, one may
> fail on the IOE. But now that the build sets a unique temp.dir per JVM, this
> won't happen I think? Still, here's a patch which protects around that too,
> even though it's a bit longer:
>
> Index: lucene/test-framework/src/java/org/apache/lucene/util/_TestUtil.java
> ===
> --- lucene/test-framework/src/java/org/apache/lucene/util/_TestUtil.java
> (revision 1424071)
> +++ lucene/test-framework/src/java/org/apache/lucene/util/_TestUtil.java
> (working copy)
> @@ -748,9 +748,19 @@
>  // already exist. otherwise tests might not reproduce, depending on
> when you last
>  // ran 'ant clean'
>
>  final Random random = new
> Random(RandomizedContext.current().getRandom().nextLong());
> -do {
> +while (true) {
>
>result = genTempFile(random, prefix, newSuffix, directory);
> -} while (!result.createNewFile());
> +  try {
> +if (result.createNewFile()) break;
> +  } catch (IOException e) {
> +// if 'result' exists and is a directory, Windows throws this bogus
> exception
> +// specializing on the msg text, rather than ignoring any IOE, just
> in case
> +// createNewFile may throw a real IOE.
> +if (!(e.getMessage().equalsIgnoreCase("access is deined"))) {
> +  throw e;
> +}
> +  }
> +}
>  return result;
>}
>
> Note though that if we don't rely on the isolation of every JVM, then
> getTempDir is totally broken too. Because it first createNewFile, then
> deletes it, so potentially two JVMs can successfully create the same file
> (one after the other) and from there on share it !
>
> Which of the patches is better to commit? The simpler one, which relies on
> JVM isolation, or the safer one?
>
> Shai
>
>
> On Wed, Dec 19, 2012 at 9:42 PM, Michael McCandless
>  wrote:
>>
>> There is some risk with your patch: if two tests try to create the
>> same dir at the same time ... with the patch, one of the tests will
>> create the dir, and the 2nd test will see it already exists and then
>> use that directory illegally (ie, share it with the first test).  That
>> will lead to very confusing test failures (if it happens).
>>
>> Ie, the idea of this API is atomicity: the file/dir did not exist
>> before, and this caller succeeded in creating it, it returns true,
>> meaning 1) the file/dir is newly created, and 2) is "private" to you.
>>
>> Mike McCandless
>>
>> http://blog.mikemccandless.com
>>
>> On Wed, Dec 19, 2012 at 1:57 PM, Shai Erera  wrote:
>> > If nobody objects, I'd like to commit this simple fix. Currently, if a
>> > test
>> > fails, I cannot  re-run it w/ same seed unless I delete the directory
>> > manually, or in the test itself (since CloseableFile doesn't delete the
>> > dir
>> > if the test failed).
>> >
>> > Shai
>> >
>> >
>> > On Wed, Dec 19, 2012 at 5:15 PM, Shai Erera  wrote:
>> >>>
>> >>> I patched _TestUtil like so, and the test passes
>> >>
>> >>
>> >> Sorry, I meant to say that if fails where I expect it to fail, not on
>> >> Access is Denied.
>> >>
>> >> Shai
>> >>
>> >>
>> >> On Wed, Dec 19, 2012 at 5:14 PM, Shai Erera  wrote:
>> >>>
>> >>> Hi
>> >>>
>> >>> When a test fails, the file system directories that it created are not
>> >>> deleted.
>> >>>
>> >>> At least on Windows, when you run the test with the same seed again,
>> >>> _TestUtil.getTempDir fails on  "Access Denied". The reason is that it
>> >>> calls
>> >>> file.createNewFile() on a directory. I don't know if it's specific to
>> >>> Windows, but I tried this test with both J9 and Oracle JVMs:
>> >>>
>> >>>   @Test
>> >>>   public void testAccessDenied() throws Exception {
>> >>> _TestUtil.getTempDir("testAccessDenied").mkdirs();
>> >>> fail("msg");
>> >>>   }
>> >>>
>> >>> Because of the fail() in the end, the directory is not deleted. If you
>> >>> run it with the same seed twice (say, -Dtests.seed=6EDF27F12DB68F8D),
>> >>> then
>> >>> you'll get:
>> >>>
>> >>> java.lang.RuntimeException: java.io.IOException: Access is denied
>> >>> at
>> >>>
>> >>> __randomizedtesting.SeedInfo.seed([6EDF27F12DB68F8D:668298B50F63FCD2]:0)
>> >>> at org.apache.lucene.util._TestUtil.getTempDir(_TestUtil.java:107)
>> >>> at
>> >>>
>> >>> org.apache.lucene.TestAssertions.testAccessDenied(TestAssertions.java:62)
>> >>>
>> >>> I patched _TestUtil like so, and the test passes:
>> >>>
>> >>> Index:
>> >>> lucene/test-framework/src/java/org/apache/lucene/util/_TestUtil.java
>> >>> =

[jira] [Commented] (LUCENE-4609) Write a PackedIntsEncoder/Decoder for facets

2012-12-19 Thread Gilad Barkai (JIRA)

[ 
https://issues.apache.org/jira/browse/LUCENE-4609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13536391#comment-13536391
 ] 

Gilad Barkai commented on LUCENE-4609:
--

In a unique encoding dgap, there's no zero, so in order to save that little 
extra bit, every gap could be encoded as a (gap - 1). Tricky is the word :)

> Write a PackedIntsEncoder/Decoder for facets
> 
>
> Key: LUCENE-4609
> URL: https://issues.apache.org/jira/browse/LUCENE-4609
> Project: Lucene - Core
>  Issue Type: New Feature
>  Components: modules/facet
>Reporter: Shai Erera
>Priority: Minor
> Attachments: LUCENE-4609.patch
>
>
> Today the facets API lets you write IntEncoder/Decoder to encode/decode the 
> category ordinals. We have several such encoders, including VInt (default), 
> and block encoders.
> It would be interesting to implement and benchmark a 
> PackedIntsEncoder/Decoder, with potentially two variants: (1) receives 
> bitsPerValue up front, when you e.g. know that you have a small taxonomy and 
> the max value you can see and (2) one that decides for each doc on the 
> optimal bitsPerValue, writes it as a header in the byte[] or something.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



[jira] [Commented] (LUCENE-4609) Write a PackedIntsEncoder/Decoder for facets

2012-12-19 Thread Michael McCandless (JIRA)

[ 
https://issues.apache.org/jira/browse/LUCENE-4609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13536386#comment-13536386
 ] 

Michael McCandless commented on LUCENE-4609:


{quote}
I'm missing something.. if there are 2 bits per value, and the codec knows its 
only 1 byte, there could be either 1, 2, 3 or 4 values in that single byte. How 
could the decoder know when to stop without knowing how many bits should not be 
encoded at the end?
{quote}

Ahh you're right!  OK.  So I think the custom header must include both bpv and 
"wasted bits".  Hmm, but only if bpv is "small enough" to be ambiguous right?

I guess another option is to leave those bits as 0s so that the decoded ord is 
0, which is the "reserved" root ord and so counting that is harmless maybe?  
Tricky ...

> Write a PackedIntsEncoder/Decoder for facets
> 
>
> Key: LUCENE-4609
> URL: https://issues.apache.org/jira/browse/LUCENE-4609
> Project: Lucene - Core
>  Issue Type: New Feature
>  Components: modules/facet
>Reporter: Shai Erera
>Priority: Minor
> Attachments: LUCENE-4609.patch
>
>
> Today the facets API lets you write IntEncoder/Decoder to encode/decode the 
> category ordinals. We have several such encoders, including VInt (default), 
> and block encoders.
> It would be interesting to implement and benchmark a 
> PackedIntsEncoder/Decoder, with potentially two variants: (1) receives 
> bitsPerValue up front, when you e.g. know that you have a small taxonomy and 
> the max value you can see and (2) one that decides for each doc on the 
> optimal bitsPerValue, writes it as a header in the byte[] or something.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



Re: _TestUtil.getTempFile may fail on "Access Denied"

2012-12-19 Thread Shai Erera
Tests running in the same JVM will still create directories atomically
because genTempFile returns a unique file name (it synchronizes around
counter). The problem may be across JVMs, because e.g. P1 and P2, which
don't sync around the same counter, may create the same file name, one may
fail on the IOE. But now that the build sets a unique temp.dir per JVM,
this won't happen I think? Still, here's a patch which protects around that
too, even though it's a bit longer:

Index: lucene/test-framework/src/java/org/apache/lucene/util/_TestUtil.java
===
---
lucene/test-framework/src/java/org/apache/lucene/util/_TestUtil.java
(revision 1424071)
+++
lucene/test-framework/src/java/org/apache/lucene/util/_TestUtil.java
(working copy)
@@ -748,9 +748,19 @@
 // already exist. otherwise tests might not reproduce, depending on
when you last
 // ran 'ant clean'
 final Random random = new
Random(RandomizedContext.current().getRandom().nextLong());
-do {
+while (true) {
   result = genTempFile(random, prefix, newSuffix, directory);
-} while (!result.createNewFile());
+  try {
+if (result.createNewFile()) break;
+  } catch (IOException e) {
+// if 'result' exists and is a directory, Windows throws this
bogus exception
+// specializing on the msg text, rather than ignoring any IOE,
just in case
+// createNewFile may throw a real IOE.
+if (!(e.getMessage().equalsIgnoreCase("access is deined"))) {
+  throw e;
+}
+  }
+}
 return result;
   }

Note though that if we don't rely on the isolation of every JVM, then
getTempDir is totally broken too. Because it first createNewFile, then
deletes it, so potentially two JVMs can successfully create the same file
(one after the other) and from there on share it !

Which of the patches is better to commit? The simpler one, which relies on
JVM isolation, or the safer one?

Shai


On Wed, Dec 19, 2012 at 9:42 PM, Michael McCandless <
luc...@mikemccandless.com> wrote:

> There is some risk with your patch: if two tests try to create the
> same dir at the same time ... with the patch, one of the tests will
> create the dir, and the 2nd test will see it already exists and then
> use that directory illegally (ie, share it with the first test).  That
> will lead to very confusing test failures (if it happens).
>
> Ie, the idea of this API is atomicity: the file/dir did not exist
> before, and this caller succeeded in creating it, it returns true,
> meaning 1) the file/dir is newly created, and 2) is "private" to you.
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
> On Wed, Dec 19, 2012 at 1:57 PM, Shai Erera  wrote:
> > If nobody objects, I'd like to commit this simple fix. Currently, if a
> test
> > fails, I cannot  re-run it w/ same seed unless I delete the directory
> > manually, or in the test itself (since CloseableFile doesn't delete the
> dir
> > if the test failed).
> >
> > Shai
> >
> >
> > On Wed, Dec 19, 2012 at 5:15 PM, Shai Erera  wrote:
> >>>
> >>> I patched _TestUtil like so, and the test passes
> >>
> >>
> >> Sorry, I meant to say that if fails where I expect it to fail, not on
> >> Access is Denied.
> >>
> >> Shai
> >>
> >>
> >> On Wed, Dec 19, 2012 at 5:14 PM, Shai Erera  wrote:
> >>>
> >>> Hi
> >>>
> >>> When a test fails, the file system directories that it created are not
> >>> deleted.
> >>>
> >>> At least on Windows, when you run the test with the same seed again,
> >>> _TestUtil.getTempDir fails on  "Access Denied". The reason is that it
> calls
> >>> file.createNewFile() on a directory. I don't know if it's specific to
> >>> Windows, but I tried this test with both J9 and Oracle JVMs:
> >>>
> >>>   @Test
> >>>   public void testAccessDenied() throws Exception {
> >>> _TestUtil.getTempDir("testAccessDenied").mkdirs();
> >>> fail("msg");
> >>>   }
> >>>
> >>> Because of the fail() in the end, the directory is not deleted. If you
> >>> run it with the same seed twice (say, -Dtests.seed=6EDF27F12DB68F8D),
> then
> >>> you'll get:
> >>>
> >>> java.lang.RuntimeException: java.io.IOException: Access is denied
> >>> at
> >>>
> __randomizedtesting.SeedInfo.seed([6EDF27F12DB68F8D:668298B50F63FCD2]:0)
> >>> at org.apache.lucene.util._TestUtil.getTempDir(_TestUtil.java:107)
> >>> at
> >>>
> org.apache.lucene.TestAssertions.testAccessDenied(TestAssertions.java:62)
> >>>
> >>> I patched _TestUtil like so, and the test passes:
> >>>
> >>> Index:
> >>> lucene/test-framework/src/java/org/apache/lucene/util/_TestUtil.java
> >>> ===
> >>> ---
> lucene/test-framework/src/java/org/apache/lucene/util/_TestUtil.java
> >>> (revision 1423868)
> >>> +++
> lucene/test-framework/src/java/org/apache/lucene/util/_TestUtil.java
> >>> (working copy)
> >>> @@ -750,7 +750,7 @@
> >>>  final Random random = new
> >>> Random(RandomizedContext.curre

[jira] [Commented] (LUCENE-4246) Fix IndexWriter.close() to not commit or wait for pending merges

2012-12-19 Thread Michael McCandless (JIRA)

[ 
https://issues.apache.org/jira/browse/LUCENE-4246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13536385#comment-13536385
 ] 

Michael McCandless commented on LUCENE-4246:


bq. Can't we just decide (I think that's the 3rd time I'm proposing it) to 
never wait for merges on close(), and keep close() committing changes?

I don't think we can never wait for merges on close.

That can easily lead to an accidental "denial of service attack on big merges", 
which would be an awful trap.  Ie, a big merge kicks off, but never has a 
chance to finish because the app closes/opens new IWs frequently.  Then every 
IW that's opened will restart the merge, spend CPU/IO resources, only to abort 
when the IW is closed.

I've never liked that close "hides" this wait-for-massive-merge to finish, but 
I also don't like this "just abort the massive merge" solution: it would create 
a nasty trap.  I would prefer  that it's explicit (abortMerges or 
waitForMerges).

> Fix IndexWriter.close() to not commit or wait for pending merges
> 
>
> Key: LUCENE-4246
> URL: https://issues.apache.org/jira/browse/LUCENE-4246
> Project: Lucene - Core
>  Issue Type: Bug
>Reporter: Robert Muir
>Assignee: Robert Muir
> Fix For: 4.1
>
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



[jira] [Commented] (LUCENE-4609) Write a PackedIntsEncoder/Decoder for facets

2012-12-19 Thread Gilad Barkai (JIRA)

[ 
https://issues.apache.org/jira/browse/LUCENE-4609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13536384#comment-13536384
 ] 

Gilad Barkai commented on LUCENE-4609:
--

bq. Hopefully you don't need to separately encode "leftover unused bits" ... ie 
byte[].length (which is "free" here, since codec already stores this) should 
suffice.

I'm missing something.. if there are 2 bits per value, and the codec knows its 
only 1 byte, there could be either 1, 2, 3 or 4 values in that single byte. How 
could the decoder know when to stop without knowing how many bits should not be 
encoded at the end? 

> Write a PackedIntsEncoder/Decoder for facets
> 
>
> Key: LUCENE-4609
> URL: https://issues.apache.org/jira/browse/LUCENE-4609
> Project: Lucene - Core
>  Issue Type: New Feature
>  Components: modules/facet
>Reporter: Shai Erera
>Priority: Minor
> Attachments: LUCENE-4609.patch
>
>
> Today the facets API lets you write IntEncoder/Decoder to encode/decode the 
> category ordinals. We have several such encoders, including VInt (default), 
> and block encoders.
> It would be interesting to implement and benchmark a 
> PackedIntsEncoder/Decoder, with potentially two variants: (1) receives 
> bitsPerValue up front, when you e.g. know that you have a small taxonomy and 
> the max value you can see and (2) one that decides for each doc on the 
> optimal bitsPerValue, writes it as a header in the byte[] or something.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



[jira] [Commented] (LUCENE-4609) Write a PackedIntsEncoder/Decoder for facets

2012-12-19 Thread Michael McCandless (JIRA)

[ 
https://issues.apache.org/jira/browse/LUCENE-4609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13536379#comment-13536379
 ] 

Michael McCandless commented on LUCENE-4609:


bq. Well, It's a 'end point' encoder, meaning it encodes whatever values are 
received directly to the output.

Ahh right, OK.

bq. PForDelta is indeed slower. But we've met scenarios in which most dgaps are 
small - hence the NOnes, and the Four/Eight Flag encoders. 

OK makes sense.

{quote}
If indeed most values are small, say, could fit in 4 bits, but there's also one 
or two larger values which would require 12 or 14 bits, we could benefit hear 
greatly.
This is all relevant only where there are large amount of categories per 
document.
{quote}

Right ... I'm just wondering how often this happens in "typical" (if there is 
such a thing) facet aps.  Decode speed trumps compression ratios here, I think.

bq. That is right. To be frank, I'm not 100% sure what PackedInts does.. nor 
how large its header is.. 

The header is very large ... really you should only need 1) bpv, and 2) 
bytes.length (which I think you already have, via both payloads and DocValues). 
 If the PackedInts API isn't flexible enough for you to feed it bpv and 
bytes.length then let's fix that!

bq. For bits-per-value smaller than the size of a byte, there's a need to know 
how many bits should be left out from the last read byte.

Hopefully you don't need to separately encode "leftover unused bits" ... ie 
byte[].length (which is "free" here, since codec already stores this) should 
suffice.

> Write a PackedIntsEncoder/Decoder for facets
> 
>
> Key: LUCENE-4609
> URL: https://issues.apache.org/jira/browse/LUCENE-4609
> Project: Lucene - Core
>  Issue Type: New Feature
>  Components: modules/facet
>Reporter: Shai Erera
>Priority: Minor
> Attachments: LUCENE-4609.patch
>
>
> Today the facets API lets you write IntEncoder/Decoder to encode/decode the 
> category ordinals. We have several such encoders, including VInt (default), 
> and block encoders.
> It would be interesting to implement and benchmark a 
> PackedIntsEncoder/Decoder, with potentially two variants: (1) receives 
> bitsPerValue up front, when you e.g. know that you have a small taxonomy and 
> the max value you can see and (2) one that decides for each doc on the 
> optimal bitsPerValue, writes it as a header in the byte[] or something.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



[jira] [Resolved] (LUCENE-4606) The test framework should report forked JVM PIDs at the start of test logs

2012-12-19 Thread Dawid Weiss (JIRA)

 [ 
https://issues.apache.org/jira/browse/LUCENE-4606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dawid Weiss resolved LUCENE-4606.
-

Resolution: Fixed

> The test framework should report forked JVM PIDs at the start of test logs
> --
>
> Key: LUCENE-4606
> URL: https://issues.apache.org/jira/browse/LUCENE-4606
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Dawid Weiss
>Assignee: Dawid Weiss
>Priority: Trivial
> Fix For: 4.1, 5.0
>
>
> A follow-up to LUCENE-4603

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



Re: DocsEnum.freq()

2012-12-19 Thread Shai Erera
>
> Also, this will only work if you run it from command-line (an ant task)
>

That's fine. The purpose (for me) is to ensure the robustness of a test /
code change.
Once a seed is found which fails the test, we should be able to debug the
test in eclipse
using -Dtests.seed, at least for single-threaded tests.

Apologies for the inconvenience it causes.
>

No apologies needed !
Your work is a blessing. This runner is really great.

Shai


On Wed, Dec 19, 2012 at 9:40 PM, Dawid Weiss
wrote:

> > BTW, I don't know if it should be like that or not, but I ran this test
> with
> > -Dtests.iters=1000 and printed at the end of each iteration
> > Codec.getDefault().
> > For all iterations it printed the same Codec, but if I ran the test many
> > times (no iteration), it printed different Codecs in each run.
> > I thought that tests.iters should simulate the behavior of running the
> test
> > many times? And therefore pick a new seed + Codec (among other things)
> for
> > each iteration?
>
> Sorry, I missed this. What you're observing is correct -- the Codec is
> picked at the class level (in TestRuleSetupAndRestoreClassEnv.java).
> -Dtests.iters duplicates each test much like if you put two or more
> test methods in a suite class. Any static rules and static hooks
> execute *once* for a class, regardless of the number of individual
> tests, so the codec is picked once, depending on the master seed (not
> the per-method derivative seed).
>
> This would be of course solved if tests.dups could run each class with
> a different seed. I wish I had more time to look into this; I kind of
> know how to solve this I think but it'll require a significant effort
> put into rewriting core parts of the runner. Also, this will only work
> if you run it from command-line (an ant task); there is no way to
> duplicate an entire test suite in Eclipse or other IDEs.
>
> I'll try to look into this, be patient. Apologies for the
> inconvenience it causes.
>
> Dawid
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>
>


Re: _TestUtil.getTempFile may fail on "Access Denied"

2012-12-19 Thread Michael McCandless
There is some risk with your patch: if two tests try to create the
same dir at the same time ... with the patch, one of the tests will
create the dir, and the 2nd test will see it already exists and then
use that directory illegally (ie, share it with the first test).  That
will lead to very confusing test failures (if it happens).

Ie, the idea of this API is atomicity: the file/dir did not exist
before, and this caller succeeded in creating it, it returns true,
meaning 1) the file/dir is newly created, and 2) is "private" to you.

Mike McCandless

http://blog.mikemccandless.com

On Wed, Dec 19, 2012 at 1:57 PM, Shai Erera  wrote:
> If nobody objects, I'd like to commit this simple fix. Currently, if a test
> fails, I cannot  re-run it w/ same seed unless I delete the directory
> manually, or in the test itself (since CloseableFile doesn't delete the dir
> if the test failed).
>
> Shai
>
>
> On Wed, Dec 19, 2012 at 5:15 PM, Shai Erera  wrote:
>>>
>>> I patched _TestUtil like so, and the test passes
>>
>>
>> Sorry, I meant to say that if fails where I expect it to fail, not on
>> Access is Denied.
>>
>> Shai
>>
>>
>> On Wed, Dec 19, 2012 at 5:14 PM, Shai Erera  wrote:
>>>
>>> Hi
>>>
>>> When a test fails, the file system directories that it created are not
>>> deleted.
>>>
>>> At least on Windows, when you run the test with the same seed again,
>>> _TestUtil.getTempDir fails on  "Access Denied". The reason is that it calls
>>> file.createNewFile() on a directory. I don't know if it's specific to
>>> Windows, but I tried this test with both J9 and Oracle JVMs:
>>>
>>>   @Test
>>>   public void testAccessDenied() throws Exception {
>>> _TestUtil.getTempDir("testAccessDenied").mkdirs();
>>> fail("msg");
>>>   }
>>>
>>> Because of the fail() in the end, the directory is not deleted. If you
>>> run it with the same seed twice (say, -Dtests.seed=6EDF27F12DB68F8D), then
>>> you'll get:
>>>
>>> java.lang.RuntimeException: java.io.IOException: Access is denied
>>> at
>>> __randomizedtesting.SeedInfo.seed([6EDF27F12DB68F8D:668298B50F63FCD2]:0)
>>> at org.apache.lucene.util._TestUtil.getTempDir(_TestUtil.java:107)
>>> at
>>> org.apache.lucene.TestAssertions.testAccessDenied(TestAssertions.java:62)
>>>
>>> I patched _TestUtil like so, and the test passes:
>>>
>>> Index:
>>> lucene/test-framework/src/java/org/apache/lucene/util/_TestUtil.java
>>> ===
>>> --- lucene/test-framework/src/java/org/apache/lucene/util/_TestUtil.java
>>> (revision 1423868)
>>> +++ lucene/test-framework/src/java/org/apache/lucene/util/_TestUtil.java
>>> (working copy)
>>> @@ -750,7 +750,7 @@
>>>  final Random random = new
>>> Random(RandomizedContext.current().getRandom().nextLong());
>>>  do {
>>>result = genTempFile(random, prefix, newSuffix, directory);
>>> -} while (!result.createNewFile());
>>> +} while (result.exists() || !result.createNewFile());
>>>  return result;
>>>}
>>>
>>> Basically, adding an exists() check before createNewFile(). Even though
>>> createNewFile() documents that it returns false if the file exists, it does
>>> not specify the behavior when the file exists and is a directory, and
>>> obviously fails.
>>>
>>> I made sure that calling genTempFile like so in a loop won't ruin
>>> randomness - since it only uses random.nextInt() once, I think it's safe to
>>> add this check?
>>>
>>> Shai
>>
>>
>

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



Re: DocsEnum.freq()

2012-12-19 Thread Dawid Weiss
> BTW, I don't know if it should be like that or not, but I ran this test with
> -Dtests.iters=1000 and printed at the end of each iteration
> Codec.getDefault().
> For all iterations it printed the same Codec, but if I ran the test many
> times (no iteration), it printed different Codecs in each run.
> I thought that tests.iters should simulate the behavior of running the test
> many times? And therefore pick a new seed + Codec (among other things) for
> each iteration?

Sorry, I missed this. What you're observing is correct -- the Codec is
picked at the class level (in TestRuleSetupAndRestoreClassEnv.java).
-Dtests.iters duplicates each test much like if you put two or more
test methods in a suite class. Any static rules and static hooks
execute *once* for a class, regardless of the number of individual
tests, so the codec is picked once, depending on the master seed (not
the per-method derivative seed).

This would be of course solved if tests.dups could run each class with
a different seed. I wish I had more time to look into this; I kind of
know how to solve this I think but it'll require a significant effort
put into rewriting core parts of the runner. Also, this will only work
if you run it from command-line (an ant task); there is no way to
duplicate an entire test suite in Eclipse or other IDEs.

I'll try to look into this, be patient. Apologies for the
inconvenience it causes.

Dawid

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



[jira] [Commented] (LUCENE-4609) Write a PackedIntsEncoder/Decoder for facets

2012-12-19 Thread Gilad Barkai (JIRA)

[ 
https://issues.apache.org/jira/browse/LUCENE-4609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13536347#comment-13536347
 ] 

Gilad Barkai commented on LUCENE-4609:
--

bq. Do you encode the gaps or the straight up ords?

Well, It's a 'end point' encoder, meaning it encodes whatever values are 
received directly to the output.
One could create an encoder as: {{new SortingIntEncoder(new 
UniqueValuesIntEncoder(new DGapIntEncoder(new PackedEncoder(}}, so the 
values the packed encoder would receive are already after sort, unique and 
dgap. 

{quote}
This is PForDelta compression (the outliers are encoded separately) I think? We 
can test it and see if it helps ... but we weren't so happy with it for 
encoding postings (it adds complexity, slows down decode, and didn't seem to 
help that much in reducing the size).
{quote}

PForDelta is indeed slower. But we've met scenarios in which most dgaps are 
small - hence the NOnes, and the Four/Eight Flag encoders. If indeed most 
values are small, say, could fit in 4 bits, but there's also one or two larger 
values which would require 12 or 14 bits, we could benefit hear greatly.
This is all relevant only where there are large amount of categories per 
document.

bq. it seems like you are writing the full header per field
That is right. To be frank, I'm not 100% sure what {{PackedInts}} does.. nor 
how large its header is.. 
But I think perhaps some header per doc is required anyway? For bits-per-value 
smaller than the size of a byte, there's a need to know how many bits should be 
left out from the last read byte. 

I started writing my own version as a first step toward the 'mixed' version, in 
which a 1 byte header is written, that contained both the the 'bits per value' 
as the first 5 bits, and the amount of extra bits in the last 3 bits. I'm still 
playing with it, hope to share it soon.

> Write a PackedIntsEncoder/Decoder for facets
> 
>
> Key: LUCENE-4609
> URL: https://issues.apache.org/jira/browse/LUCENE-4609
> Project: Lucene - Core
>  Issue Type: New Feature
>  Components: modules/facet
>Reporter: Shai Erera
>Priority: Minor
> Attachments: LUCENE-4609.patch
>
>
> Today the facets API lets you write IntEncoder/Decoder to encode/decode the 
> category ordinals. We have several such encoders, including VInt (default), 
> and block encoders.
> It would be interesting to implement and benchmark a 
> PackedIntsEncoder/Decoder, with potentially two variants: (1) receives 
> bitsPerValue up front, when you e.g. know that you have a small taxonomy and 
> the max value you can see and (2) one that decides for each doc on the 
> optimal bitsPerValue, writes it as a header in the byte[] or something.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



[jira] [Commented] (LUCENE-4246) Fix IndexWriter.close() to not commit or wait for pending merges

2012-12-19 Thread Shai Erera (JIRA)

[ 
https://issues.apache.org/jira/browse/LUCENE-4246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13536341#comment-13536341
 ] 

Shai Erera commented on LUCENE-4246:


Close() throws an exception -- that's just for back-compat right? I.e., the app 
can't do anything with that exception except either calling commit() (that's 
the back-compat part) or rollback() (if it needs to close fast). And 
abortRunningMerges is just like close(false) today (without the commit())?

I mean, if an app wants to commit + wait + close, the 1 line close() becomes 
wait(), commit() and close(). And just like I made a mistake saying you should 
call commit() then wait(), I'm sure others will make that mistake too.

So again, why go to great lengths to complicate the lives of the innocent 
developer? Can't we just decide (I think that's the 3rd time I'm proposing it) 
to never wait for merges on close(), and keep close() committing changes? Would 
close() be really simplified if it will need to detect running merges and 
uncommitted changes?

Maybe someone puts up a patch with the changes to IW only, so we can review? I 
personally don't see the gain in changing the behavior of close(), but maybe a 
patch will clarify the gains ...

> Fix IndexWriter.close() to not commit or wait for pending merges
> 
>
> Key: LUCENE-4246
> URL: https://issues.apache.org/jira/browse/LUCENE-4246
> Project: Lucene - Core
>  Issue Type: Bug
>Reporter: Robert Muir
>Assignee: Robert Muir
> Fix For: 4.1
>
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



[jira] [Commented] (SOLR-4220) Move RequestHandler for global Information out of core-scope

2012-12-19 Thread Mark Miller (JIRA)

[ 
https://issues.apache.org/jira/browse/SOLR-4220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13536331#comment-13536331
 ] 

Mark Miller commented on SOLR-4220:
---

+1 - Im very interested in this as well. CoreAdminHandler should not be 
configured and routed under core - it should be at the CoreContainer level.

This would be a huge improvement.

> Move RequestHandler for global Information out of core-scope
> 
>
> Key: SOLR-4220
> URL: https://issues.apache.org/jira/browse/SOLR-4220
> Project: Solr
>  Issue Type: Improvement
>  Components: web gui
>Reporter: Stefan Matheis (steffkes)
>Assignee: Hoss Man
>
> Okay, the title perhaps wins no price right now .. but i don't have an better 
> idea, i'm sorry - if you do, don't hesitate to update it!
> What it's all about: SOLR-3633 was created because at the moment it's not 
> possible to use the Admin UI w/o at least one core. The reason for that is, 
> that some (as you might think) "global" Information - which the UI shows on 
> the top-level (which doesn't require you selecting a core) - must be fetched 
> from a core-related url, because that's how the solr routing works right now.
> Hoss and I talked about that at the ApacheCon and he mentioned that this 
> should not be that biggest change although we need to update the tests and 
> ensure that the thing is still working, of course.
> I checked the UI for having a list of Functions and their related urls:
> * *Dashboard*
> ** solr/$first_core/admin/system?wt=json
> * *Logging*
> ** /solr/$first_core/admin/logging?wt=json&since=N
> * *Logging* / *Level*
> ** /solr/$first_core/admin/logging?wt=json
> * *Java Properties*
> ** /solr/$first_core/admin/properties?wt=json
> * *Threads* 
> ** /solr/$first_core/admin/threads?wt=json
> For the sake of simplicity, i'd suggest that we're just moving the complete 
> handler (regarding their url) on level up to something like 
> {{/solr/admin/..}} like we have it already for the zookeeper thing?
> Regarding the contained content, i think we could (or perhaps should?) stick 
> with the given information/content - only the Dashboard is not using all of 
> the provided values, but just for the fact that we have no usage for 
> prettified RAM-Usage numbers ...
> Let me know if the Issue contains all required informations, otherwise i'll 
> try to update it according to your questions :)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



[jira] [Commented] (LUCENE-4609) Write a PackedIntsEncoder/Decoder for facets

2012-12-19 Thread Michael McCandless (JIRA)

[ 
https://issues.apache.org/jira/browse/LUCENE-4609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13536321#comment-13536321
 ] 

Michael McCandless commented on LUCENE-4609:


Cool!  Do you encode the gaps or the straight up ords?  (Looks like straight up 
ords?).

bq. Started to look into a semi-packed encoder, which could encode most values 
in a packed manner, but could also add large values as, e.g., vints.

This is PForDelta compression (the outliers are encoded separately) I think?  
We can test it and see if it helps ... but we weren't so happy with it for 
encoding postings (it adds complexity, slows down decode, and didn't seem to 
help that much in reducing the size).

How much worse was compression with straight up packed ints vs vInt(gap)?  The 
byte-header-per-doc is annoying but I don't think we can do much about it today 
... really we need a DocValues that allows multiple ints per field (int[]).  It 
would write the bpv once in the header (for the entire segment) and then all 
docs would use that bpv (and we'd separately have to write doc -> offset).  But 
this is a big change (we need multi-valued DV first)...

Hmm, it seems like you are writing the full header per field (which is very 
wasteful)?  We should be able to write only one byte (the bpv), if you use 
getWriterNoHeader instead?


> Write a PackedIntsEncoder/Decoder for facets
> 
>
> Key: LUCENE-4609
> URL: https://issues.apache.org/jira/browse/LUCENE-4609
> Project: Lucene - Core
>  Issue Type: New Feature
>  Components: modules/facet
>Reporter: Shai Erera
>Priority: Minor
> Attachments: LUCENE-4609.patch
>
>
> Today the facets API lets you write IntEncoder/Decoder to encode/decode the 
> category ordinals. We have several such encoders, including VInt (default), 
> and block encoders.
> It would be interesting to implement and benchmark a 
> PackedIntsEncoder/Decoder, with potentially two variants: (1) receives 
> bitsPerValue up front, when you e.g. know that you have a small taxonomy and 
> the max value you can see and (2) one that decides for each doc on the 
> optimal bitsPerValue, writes it as a header in the byte[] or something.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



[jira] [Updated] (SOLR-4220) Move RequestHandler for global Information out of core-scope

2012-12-19 Thread Stefan Matheis (steffkes) (JIRA)

 [ 
https://issues.apache.org/jira/browse/SOLR-4220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stefan Matheis (steffkes) updated SOLR-4220:


Description: 
Okay, the title perhaps wins no price right now .. but i don't have an better 
idea, i'm sorry - if you do, don't hesitate to update it!

What it's all about: SOLR-3633 was created because at the moment it's not 
possible to use the Admin UI w/o at least one core. The reason for that is, 
that some (as you might think) "global" Information - which the UI shows on the 
top-level (which doesn't require you selecting a core) - must be fetched from a 
core-related url, because that's how the solr routing works right now.

Hoss and I talked about that at the ApacheCon and he mentioned that this should 
not be that biggest change although we need to update the tests and ensure that 
the thing is still working, of course.

I checked the UI for having a list of Functions and their related urls:
* *Dashboard*
** solr/$first_core/admin/system?wt=json

* *Logging*
** /solr/$first_core/admin/logging?wt=json&since=N

* *Logging* / *Level*
** /solr/$first_core/admin/logging?wt=json

* *Java Properties*
** /solr/$first_core/admin/properties?wt=json

* *Threads* 
** /solr/$first_core/admin/threads?wt=json

For the sake of simplicity, i'd suggest that we're just moving the complete 
handler (regarding their url) on level up to something like {{/solr/admin/..}} 
like we have it already for the zookeeper thing?

Regarding the contained content, i think we could (or perhaps should?) stick 
with the given information/content - only the Dashboard is not using all of the 
provided values, but just for the fact that we have no usage for prettified 
RAM-Usage numbers ...

Let me know if the Issue contains all required informations, otherwise i'll try 
to update it according to your questions :)

  was:
Okay, the title perhaps wins no price right now .. but i don't have an better 
idea, i'm sorry - if you do, don't hesitate to update it!

What it's all about: SOLR-3633 was created because at the moment it's not 
possible to use the Admin UI w/o at least one core. The reason for that is, 
that some (as you might think) "global" Information - which the UI shows on the 
top-level (which doesn't require you selecting a core) - must be fetched from a 
core-related url, because that's how the solr routing works right now.

Hoss and I talked about that at the ApacheCon and he mentioned that this should 
not be that biggest change although we need to update the tests and ensure that 
the thing is still working, of course.

I checked the UI for having a list of Functions and their related urls:
* *Dashboard*
** solr/$first_core/admin/system?wt=json
* *Logging*
** /solr/$first_core/admin/logging?wt=json&since=N
* *Logging* / *Level*
** /solr/$first_core/admin/logging?wt=json
* *Java Properties*
** /solr/$first_core/admin/properties?wt=json
* *Threads* 
** /solr/$first_core/admin/threads?wt=json

For the sake of simplicity, i'd suggest that we're just moving the complete 
handler (regarding their url) on level up to something like {{/solr/admin/..}} 
like we have it already for the zookeeper thing?

Regarding the contained content, i think we could (or perhaps should?) stick 
with the given information/content - only the Dashboard is not using all of the 
provided values, but just for the fact that we have no usage for prettified 
RAM-Usage numbers ...

Let me know if the Issue contains all required informations, otherwise i'll try 
to update it according to your questions :)


> Move RequestHandler for global Information out of core-scope
> 
>
> Key: SOLR-4220
> URL: https://issues.apache.org/jira/browse/SOLR-4220
> Project: Solr
>  Issue Type: Improvement
>  Components: web gui
>Reporter: Stefan Matheis (steffkes)
>Assignee: Hoss Man
>
> Okay, the title perhaps wins no price right now .. but i don't have an better 
> idea, i'm sorry - if you do, don't hesitate to update it!
> What it's all about: SOLR-3633 was created because at the moment it's not 
> possible to use the Admin UI w/o at least one core. The reason for that is, 
> that some (as you might think) "global" Information - which the UI shows on 
> the top-level (which doesn't require you selecting a core) - must be fetched 
> from a core-related url, because that's how the solr routing works right now.
> Hoss and I talked about that at the ApacheCon and he mentioned that this 
> should not be that biggest change although we need to update the tests and 
> ensure that the thing is still working, of course.
> I checked the UI for having a list of Functions and their related urls:
> * *Dashboard*
> ** solr/$first_core/admin/system?wt=json
> * *Logging*
> ** /solr/$first_core/admin/logging?wt=json&since=N
> *

[jira] [Created] (SOLR-4220) Move RequestHandler for global Information out of core-scope

2012-12-19 Thread Stefan Matheis (steffkes) (JIRA)
Stefan Matheis (steffkes) created SOLR-4220:
---

 Summary: Move RequestHandler for global Information out of 
core-scope
 Key: SOLR-4220
 URL: https://issues.apache.org/jira/browse/SOLR-4220
 Project: Solr
  Issue Type: Improvement
  Components: web gui
Reporter: Stefan Matheis (steffkes)
Assignee: Hoss Man


Okay, the title perhaps wins no price right now .. but i don't have an better 
idea, i'm sorry - if you do, don't hesitate to update it!

What it's all about: SOLR-3633 was created because at the moment it's not 
possible to use the Admin UI w/o at least one core. The reason for that is, 
that some (as you might think) "global" Information - which the UI shows on the 
top-level (which doesn't require you selecting a core) - must be fetched from a 
core-related url, because that's how the solr routing works right now.

Hoss and I talked about that at the ApacheCon and he mentioned that this should 
not be that biggest change although we need to update the tests and ensure that 
the thing is still working, of course.

I checked the UI for having a list of Functions and their related urls:
* *Dashboard*
** solr/$first_core/admin/system?wt=json
* *Logging*
** /solr/$first_core/admin/logging?wt=json&since=N
* *Logging* / *Level*
** /solr/$first_core/admin/logging?wt=json
* *Java Properties*
** /solr/$first_core/admin/properties?wt=json
* *Threads* 
** /solr/$first_core/admin/threads?wt=json

For the sake of simplicity, i'd suggest that we're just moving the complete 
handler (regarding their url) on level up to something like {{/solr/admin/..}} 
like we have it already for the zookeeper thing?

Regarding the contained content, i think we could (or perhaps should?) stick 
with the given information/content - only the Dashboard is not using all of the 
provided values, but just for the fact that we have no usage for prettified 
RAM-Usage numbers ...

Let me know if the Issue contains all required informations, otherwise i'll try 
to update it according to your questions :)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



Re: VOTE: release 3.6.2

2012-12-19 Thread Robert Muir
On Wed, Dec 19, 2012 at 1:48 PM, Michael McCandless
 wrote:
> Hmm my smoke test run was angry about this javadocs warning:
>
>   [javadoc] 
> /l/36x/tmp/unpack/apache-solr-3.6.2/lucene/contrib/facet/src/java/org/apache/lucene/facet/taxonomy/writercache/lru/NameIntCacheLRU.java:76:
> warning - @return tag has no arguments.
>
> It hit this when running javadocs with 1.7.0_07.

Thanks Mike... can you fix? I have no idea why different versions of
java7 have different levels of pickiness.

This is no issue for e.g. 4.x+, because we have the eclipse checker as
part of our build which fails on this. But for 3.x we don't have as
many tools unfortunately.

I dont think we should put a lot of effort into this: but when
backporting bugfixes to old branches like this please be really
careful about this stuff.

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



Re: _TestUtil.getTempFile may fail on "Access Denied"

2012-12-19 Thread Shai Erera
If nobody objects, I'd like to commit this simple fix. Currently, if a test
fails, I cannot  re-run it w/ same seed unless I delete the directory
manually, or in the test itself (since CloseableFile doesn't delete the dir
if the test failed).

Shai


On Wed, Dec 19, 2012 at 5:15 PM, Shai Erera  wrote:

> I patched _TestUtil like so, and the test passes
>>
>
> Sorry, I meant to say that if fails where I expect it to fail, not on
> Access is Denied.
>
> Shai
>
>
> On Wed, Dec 19, 2012 at 5:14 PM, Shai Erera  wrote:
>
>> Hi
>>
>> When a test fails, the file system directories that it created are not
>> deleted.
>>
>> At least on Windows, when you run the test with the same seed again,
>> _TestUtil.getTempDir fails on  "Access Denied". The reason is that it calls
>> file.createNewFile() on a directory. I don't know if it's specific to
>> Windows, but I tried this test with both J9 and Oracle JVMs:
>>
>>   @Test
>>   public void testAccessDenied() throws Exception {
>> _TestUtil.getTempDir("testAccessDenied").mkdirs();
>> fail("msg");
>>   }
>>
>> Because of the fail() in the end, the directory is not deleted. If you
>> run it with the same seed twice (say, -Dtests.seed=6EDF27F12DB68F8D), then
>> you'll get:
>>
>> java.lang.RuntimeException: java.io.IOException: Access is denied
>> at
>> __randomizedtesting.SeedInfo.seed([6EDF27F12DB68F8D:668298B50F63FCD2]:0)
>> at org.apache.lucene.util._TestUtil.getTempDir(_TestUtil.java:107)
>> at
>> org.apache.lucene.TestAssertions.testAccessDenied(TestAssertions.java:62)
>>
>> I patched _TestUtil like so, and the test passes:
>>
>> Index:
>> lucene/test-framework/src/java/org/apache/lucene/util/_TestUtil.java
>> ===
>> ---
>> lucene/test-framework/src/java/org/apache/lucene/util/_TestUtil.java
>> (revision 1423868)
>> +++
>> lucene/test-framework/src/java/org/apache/lucene/util/_TestUtil.java
>> (working copy)
>> @@ -750,7 +750,7 @@
>>  final Random random = new
>> Random(RandomizedContext.current().getRandom().nextLong());
>>  do {
>>result = genTempFile(random, prefix, newSuffix, directory);
>> -} while (!result.createNewFile());
>> +} while (result.exists() || !result.createNewFile());
>>  return result;
>>}
>>
>> Basically, adding an exists() check before createNewFile(). Even though
>> createNewFile() documents that it returns false if the file exists, it does
>> not specify the behavior when the file exists and is a directory, and
>> obviously fails.
>>
>> I made sure that calling genTempFile like so in a loop won't ruin
>> randomness - since it only uses random.nextInt() once, I think it's safe to
>> add this check?
>>
>> Shai
>>
>
>


[jira] [Commented] (LUCENE-4246) Fix IndexWriter.close() to not commit or wait for pending merges

2012-12-19 Thread Michael McCandless (JIRA)

[ 
https://issues.apache.org/jira/browse/LUCENE-4246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13536292#comment-13536292
 ] 

Michael McCandless commented on LUCENE-4246:


I think a good compromise here is for close to throw an exception if merges are 
still running or if there are uncommitted changes.  We'd also add an 
"abortRunningMerges()".

This way the app can be explicit about discarding changes, aborting merges, and 
separate that from close(), which becomes a fast operation.

> Fix IndexWriter.close() to not commit or wait for pending merges
> 
>
> Key: LUCENE-4246
> URL: https://issues.apache.org/jira/browse/LUCENE-4246
> Project: Lucene - Core
>  Issue Type: Bug
>Reporter: Robert Muir
>Assignee: Robert Muir
> Fix For: 4.1
>
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



Re: VOTE: release 3.6.2

2012-12-19 Thread Michael McCandless
Hmm my smoke test run was angry about this javadocs warning:

  [javadoc] 
/l/36x/tmp/unpack/apache-solr-3.6.2/lucene/contrib/facet/src/java/org/apache/lucene/facet/taxonomy/writercache/lru/NameIntCacheLRU.java:76:
warning - @return tag has no arguments.

It hit this when running javadocs with 1.7.0_07.

Mike McCandless

http://blog.mikemccandless.com

On Wed, Dec 19, 2012 at 11:37 AM, Robert Muir  wrote:
> Hello everyone, given the bug Mike fixed in
> https://issues.apache.org/jira/browse/LUCENE-4635, I think we should
> push out a 3.6.2 bugfix release with the fix.
> Tom (the reporter of the bug) confirmed his checkindex passed
> successfully last night, and besides that we have a few other nice
> bugfixes there.
>
> Artifacts are here: http://s.apache.org/lusolr362rc0
>
> If you want to use the smoketester, of course you must run it from the
> lucene_solr_3_6 branch. This passes for me on linux. I'm not sure it
> works on other platforms.
>
> Thanks,
> Robert
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



[jira] [Commented] (SOLR-3840) XML query response display is unreadable in Solr Admin Query UI

2012-12-19 Thread Ryan McKinley (JIRA)

[ 
https://issues.apache.org/jira/browse/SOLR-3840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13536286#comment-13536286
 ] 

Ryan McKinley commented on SOLR-3840:
-

+1 for XCode

as always, looks great Stefan!

> XML query response display is unreadable in Solr Admin Query UI
> ---
>
> Key: SOLR-3840
> URL: https://issues.apache.org/jira/browse/SOLR-3840
> Project: Solr
>  Issue Type: Improvement
>  Components: web gui
>Affects Versions: 4.0-BETA
> Environment: Google Chrome, Windows 7 - Firefox as well
>Reporter: Jack Krupansky
>Assignee: Stefan Matheis (steffkes)
> Attachments: Query-UI-XML-unreadable.png, SOLR-3840.patch, 
> SOLR-3840.patch, solr-admin-ui-query-highlight-json.png, 
> solr-admin-ui-query-highlight-xml.png
>
>
> If I execute a query in the Solr Admin Query UI, the default XML "writer" 
> displays only the raw text of the Solr response XML element values, but none 
> of the XML syntax itself, rendering the response display on the Query page 
> almost completely useless. JSON, Ruby, et al display very reasonably 
> formatted output, but XML does not.
> You can click on the icon next to the generated query URL to display the 
> response in a browser tab of its own and then it does display the XML very 
> reasonably, but the framed display on the query page is simply not readable.
> My recollection is that the old UI had this same problem.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



[jira] [Commented] (SOLR-3840) XML query response display is unreadable in Solr Admin Query UI

2012-12-19 Thread Stefan Matheis (steffkes) (JIRA)

[ 
https://issues.apache.org/jira/browse/SOLR-3840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13536276#comment-13536276
 ] 

Stefan Matheis (steffkes) commented on SOLR-3840:
-

Ah, well .. yes .. of course! let me know which one you like .. perhaps you can 
have a look at the given themes on the highlight.js page: 
http://softwaremaniacs.org/media/soft/highlight/test.html ?

> XML query response display is unreadable in Solr Admin Query UI
> ---
>
> Key: SOLR-3840
> URL: https://issues.apache.org/jira/browse/SOLR-3840
> Project: Solr
>  Issue Type: Improvement
>  Components: web gui
>Affects Versions: 4.0-BETA
> Environment: Google Chrome, Windows 7 - Firefox as well
>Reporter: Jack Krupansky
>Assignee: Stefan Matheis (steffkes)
> Attachments: Query-UI-XML-unreadable.png, SOLR-3840.patch, 
> SOLR-3840.patch, solr-admin-ui-query-highlight-json.png, 
> solr-admin-ui-query-highlight-xml.png
>
>
> If I execute a query in the Solr Admin Query UI, the default XML "writer" 
> displays only the raw text of the Solr response XML element values, but none 
> of the XML syntax itself, rendering the response display on the Query page 
> almost completely useless. JSON, Ruby, et al display very reasonably 
> formatted output, but XML does not.
> You can click on the icon next to the generated query URL to display the 
> response in a browser tab of its own and then it does display the XML very 
> reasonably, but the framed display on the query page is simply not readable.
> My recollection is that the old UI had this same problem.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



[jira] [Commented] (SOLR-3969) Admin dashboard -- include cache/buffer memory utilization

2012-12-19 Thread Stefan Matheis (steffkes) (JIRA)

[ 
https://issues.apache.org/jira/browse/SOLR-3969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13536266#comment-13536266
 ] 

Stefan Matheis (steffkes) commented on SOLR-3969:
-

Shawn, perhaps you can provide an patch for your favorite OS? On the 
Client-Side we could add an additional check, for showing this memory-stuff, 
only if it's provided by the backend .. and then try to port this stuff on 
other platforms too?

> Admin dashboard -- include cache/buffer memory utilization
> --
>
> Key: SOLR-3969
> URL: https://issues.apache.org/jira/browse/SOLR-3969
> Project: Solr
>  Issue Type: Improvement
>  Components: web gui
>Affects Versions: 4.0
> Environment: Linux bigindy5 2.6.32-279.9.1.el6.centos.plus.x86_64 #1 
> SMP Wed Sep 26 03:52:55 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux
> java version "1.7.0_07"
> Java(TM) SE Runtime Environment (build 1.7.0_07-b10)
> Java HotSpot(TM) 64-Bit Server VM (build 23.3-b01, mixed mode)
>Reporter: Shawn Heisey
>Priority: Minor
> Fix For: 4.1
>
>
> The admin dashboard includes memory utilization, but there is no way on that 
> screen to see how much memory is being used for cache and buffers -- two 
> additional numbers shown by top.
> The most visually appealing way to display this would be to break the 
> Physical Memory graph into multiple colors, but if that's not practical, more 
> graphs could be added.  I'm not even sure it's possible, but if it is, it 
> would make a great addition.  Hopefully there would also be a cross-platform 
> way of doing it so that the major platforms could all be supported.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



[jira] [Commented] (SOLR-3969) Admin dashboard -- include cache/buffer memory utilization

2012-12-19 Thread Stefan Matheis (steffkes) (JIRA)

[ 
https://issues.apache.org/jira/browse/SOLR-3969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13536265#comment-13536265
 ] 

Stefan Matheis (steffkes) commented on SOLR-3969:
-

Shawn, perhaps you can provide an patch for your favorite OS? On the 
Client-Side we could add an additional check, for showing this memory-stuff, 
only if it's provided by the backend .. and then try to port this stuff on 
other platforms too?

> Admin dashboard -- include cache/buffer memory utilization
> --
>
> Key: SOLR-3969
> URL: https://issues.apache.org/jira/browse/SOLR-3969
> Project: Solr
>  Issue Type: Improvement
>  Components: web gui
>Affects Versions: 4.0
> Environment: Linux bigindy5 2.6.32-279.9.1.el6.centos.plus.x86_64 #1 
> SMP Wed Sep 26 03:52:55 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux
> java version "1.7.0_07"
> Java(TM) SE Runtime Environment (build 1.7.0_07-b10)
> Java HotSpot(TM) 64-Bit Server VM (build 23.3-b01, mixed mode)
>Reporter: Shawn Heisey
>Priority: Minor
> Fix For: 4.1
>
>
> The admin dashboard includes memory utilization, but there is no way on that 
> screen to see how much memory is being used for cache and buffers -- two 
> additional numbers shown by top.
> The most visually appealing way to display this would be to break the 
> Physical Memory graph into multiple colors, but if that's not practical, more 
> graphs could be added.  I'm not even sure it's possible, but if it is, it 
> would make a great addition.  Hopefully there would also be a cross-platform 
> way of doing it so that the major platforms could all be supported.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



[jira] [Commented] (SOLR-3840) XML query response display is unreadable in Solr Admin Query UI

2012-12-19 Thread Yonik Seeley (JIRA)

[ 
https://issues.apache.org/jira/browse/SOLR-3840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13536262#comment-13536262
 ] 

Yonik Seeley commented on SOLR-3840:


Sweet!

About the JSON colors though - the red text kind of stands out a bit and makes 
you think "error".  Many syntax highlighters use green for quoted strings.  And 
perhaps we could use blue for the map keys (which currently look sort of 
darkish green/gray?)

Anyway - just my first impressions - I'd like to hear what other people think 
are good colors (esp with the JSON which I hope is the default!)

> XML query response display is unreadable in Solr Admin Query UI
> ---
>
> Key: SOLR-3840
> URL: https://issues.apache.org/jira/browse/SOLR-3840
> Project: Solr
>  Issue Type: Improvement
>  Components: web gui
>Affects Versions: 4.0-BETA
> Environment: Google Chrome, Windows 7 - Firefox as well
>Reporter: Jack Krupansky
>Assignee: Stefan Matheis (steffkes)
> Attachments: Query-UI-XML-unreadable.png, SOLR-3840.patch, 
> SOLR-3840.patch, solr-admin-ui-query-highlight-json.png, 
> solr-admin-ui-query-highlight-xml.png
>
>
> If I execute a query in the Solr Admin Query UI, the default XML "writer" 
> displays only the raw text of the Solr response XML element values, but none 
> of the XML syntax itself, rendering the response display on the Query page 
> almost completely useless. JSON, Ruby, et al display very reasonably 
> formatted output, but XML does not.
> You can click on the icon next to the generated query URL to display the 
> response in a browser tab of its own and then it does display the XML very 
> reasonably, but the framed display on the query page is simply not readable.
> My recollection is that the old UI had this same problem.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



[jira] [Commented] (SOLR-4208) Refactor edismax query parser

2012-12-19 Thread JIRA

[ 
https://issues.apache.org/jira/browse/SOLR-4208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13536254#comment-13536254
 ] 

Tomás Fernández Löbbe commented on SOLR-4208:
-

Markus I have run the test on different environments and it keeps passing. 
Could you give me more details about your environment? 
If the same tests fails without the patch, maybe we should open a new Jira 
issue for it.

> Refactor edismax query parser
> -
>
> Key: SOLR-4208
> URL: https://issues.apache.org/jira/browse/SOLR-4208
> Project: Solr
>  Issue Type: Improvement
>Reporter: Tomás Fernández Löbbe
>Priority: Minor
> Fix For: 4.1, 5.0
>
> Attachments: SOLR-4208.patch
>
>
> With successive changes, the edismax query parser has become more complex. It 
> would be nice to refactor it to reduce code complexity, also to allow better 
> extension and code reuse.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



[jira] [Assigned] (SOLR-4176) analysis ui: javascript not properly handling URL decoding of input

2012-12-19 Thread Stefan Matheis (steffkes) (JIRA)

 [ 
https://issues.apache.org/jira/browse/SOLR-4176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stefan Matheis (steffkes) reassigned SOLR-4176:
---

Assignee: Stefan Matheis (steffkes)

> analysis ui: javascript not properly handling URL decoding of input
> ---
>
> Key: SOLR-4176
> URL: https://issues.apache.org/jira/browse/SOLR-4176
> Project: Solr
>  Issue Type: Bug
>  Components: web gui
>Affects Versions: 4.0
>Reporter: Hoss Man
>Assignee: Stefan Matheis (steffkes)
> Attachments: SOLR-4176.patch
>
>
> attempting to input values containing "%" in the analysis UI causes errors.
> Example of the bug, using the solr example configs...
> 1) load http://localhost:8983/solr/#/collection1/analysis in a browser
> 2) select field type "text_general"
> 3) enter into either text box: {{{foo%bar}}
> 4) click the "Analyze Values" button.
> results...
> * Window location is updated to be: 
> http://localhost:8983/solr/#/collection1/analysis?analysis.fieldvalue=foo%25bar&analysis.query=&analysis.fieldtype=text_general&verbose_output=1
> ** Note: "%" has been properly encoded in URL
> * page does not display any analyis, and text areas are now empty (although 
> text_general field type is still selected)
> * web dev error console indicates...{noformat}Error: URIError: malformed URI 
> sequence
> Source File: http://localhost:8983/solr/js/scripts/analysis.js
> Line: 132
> {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



[jira] [Updated] (SOLR-4176) analysis ui: javascript not properly handling URL decoding of input

2012-12-19 Thread Stefan Matheis (steffkes) (JIRA)

 [ 
https://issues.apache.org/jira/browse/SOLR-4176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stefan Matheis (steffkes) updated SOLR-4176:


Attachment: SOLR-4176.patch

that is really a small patch .. but i'm still not really sure why this 
url-decoding was in place .. before committing that one, i'll check that, just 
to be sure that there is no regression

> analysis ui: javascript not properly handling URL decoding of input
> ---
>
> Key: SOLR-4176
> URL: https://issues.apache.org/jira/browse/SOLR-4176
> Project: Solr
>  Issue Type: Bug
>  Components: web gui
>Affects Versions: 4.0
>Reporter: Hoss Man
> Attachments: SOLR-4176.patch
>
>
> attempting to input values containing "%" in the analysis UI causes errors.
> Example of the bug, using the solr example configs...
> 1) load http://localhost:8983/solr/#/collection1/analysis in a browser
> 2) select field type "text_general"
> 3) enter into either text box: {{{foo%bar}}
> 4) click the "Analyze Values" button.
> results...
> * Window location is updated to be: 
> http://localhost:8983/solr/#/collection1/analysis?analysis.fieldvalue=foo%25bar&analysis.query=&analysis.fieldtype=text_general&verbose_output=1
> ** Note: "%" has been properly encoded in URL
> * page does not display any analyis, and text areas are now empty (although 
> text_general field type is still selected)
> * web dev error console indicates...{noformat}Error: URIError: malformed URI 
> sequence
> Source File: http://localhost:8983/solr/js/scripts/analysis.js
> Line: 132
> {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



[jira] [Commented] (LUCENE-4246) Fix IndexWriter.close() to not commit or wait for pending merges

2012-12-19 Thread Mark Miller (JIRA)

[ 
https://issues.apache.org/jira/browse/LUCENE-4246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13536232#comment-13536232
 ] 

Mark Miller commented on LUCENE-4246:
-

bq.  you really believe there is anyone out there intending to close IW without 
committing the changes?

I do it, but I use rollback for this...

> Fix IndexWriter.close() to not commit or wait for pending merges
> 
>
> Key: LUCENE-4246
> URL: https://issues.apache.org/jira/browse/LUCENE-4246
> Project: Lucene - Core
>  Issue Type: Bug
>Reporter: Robert Muir
>Assignee: Robert Muir
> Fix For: 4.1
>
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



[jira] [Updated] (SOLR-4079) Long core names break web gui appearance and functionality

2012-12-19 Thread Stefan Matheis (steffkes) (JIRA)

 [ 
https://issues.apache.org/jira/browse/SOLR-4079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stefan Matheis (steffkes) updated SOLR-4079:


Attachment: SOLR-4079.patch

Janko, i'm not really sure about the change of the width? 250px looks a bit 
like a specific solution for your core name, doesn't it? 

the updated patch adds a {{title}}-attribut to the core-link which will show 
the complete core-name while mouseover .. would that also be useful, wdyt?

> Long core names break web gui appearance and functionality
> --
>
> Key: SOLR-4079
> URL: https://issues.apache.org/jira/browse/SOLR-4079
> Project: Solr
>  Issue Type: Bug
>  Components: web gui
>Affects Versions: 4.0
> Environment: Reproduced in Chrome (23.0.1271.64)
>Reporter: Janko Luin
>Priority: Trivial
> Attachments: proposed_patch.css, Screen Shot 2012-11-15 at 
> 10.40.29.png, SOLR-4079.patch
>
>
> Our data is split up over multiple cores with automatic naming, typically of 
> the form "articles_20080101154402_20080412181644". With long names like that, 
> the sidebar will massively overshoot its boundaries and intrude so far into 
> the main pane that it interferes with UI inputs and controls.
> Fixing this issue is trivial even within the browser, by introducing a custom 
> stylesheet.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



[jira] [Updated] (SOLR-4045) SOLR admin page returns HTTP 404 on core names containing a '.' (dot)

2012-12-19 Thread Stefan Matheis (steffkes) (JIRA)

 [ 
https://issues.apache.org/jira/browse/SOLR-4045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stefan Matheis (steffkes) updated SOLR-4045:


Attachment: SOLR-4045.patch

Updated Patch includes a more generic solution

> SOLR admin page returns HTTP 404 on core names containing a '.' (dot)
> -
>
> Key: SOLR-4045
> URL: https://issues.apache.org/jira/browse/SOLR-4045
> Project: Solr
>  Issue Type: Bug
>  Components: web gui
>Affects Versions: 4.0
> Environment: Linux, Ubuntu 12.04
>Reporter: Alessandro Tommasi
>Assignee: Stefan Matheis (steffkes)
>Priority: Minor
>  Labels: admin, solr, webgui
> Fix For: 5.0
>
> Attachments: SOLR-4045.patch, SOLR-4045.patch
>
>
> When SOLR is configured in multicore mode, cores with '.' (dot) in their 
> names are inaccessible via the admin web guy. (localhost:8983/solr). The page 
> shows an alert with the message (test.test was my core name):
> 404 Not Found get #/test.test
> To replicate: start solr in multicore mode, go to "localhost:8983/solr", via 
> core admin create a new core "test.test", then refresh the page. "test.test" 
> will show under the menu at the bottom left. Clicking on it causes the 
> message, while no core menu appears.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



[jira] [Updated] (SOLR-3851) create a new core/delete an existing core should also update the main/left list of cores

2012-12-19 Thread Stefan Matheis (steffkes) (JIRA)

 [ 
https://issues.apache.org/jira/browse/SOLR-3851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stefan Matheis (steffkes) updated SOLR-3851:


Attachment: SOLR-3851.patch

> create a new core/delete an existing core should also update the main/left 
> list of cores
> 
>
> Key: SOLR-3851
> URL: https://issues.apache.org/jira/browse/SOLR-3851
> Project: Solr
>  Issue Type: Improvement
>  Components: web gui
>Affects Versions: 4.0-BETA
>Reporter: Stefan Matheis (steffkes)
>Assignee: Stefan Matheis (steffkes)
>Priority: Minor
> Fix For: 4.1
>
> Attachments: SOLR-3851.patch, SOLR-3851.patch
>
>
> While working on SOLR-3788, i realized that the main/left list of cores 
> needs/should also be updated when a new core is created / an existing core is 
> deleted, which is right now not the fact.
> On a first quick look that should not be that hard, hopefully we can reuse 
> existing functionality from app.js, which already generates a list of cores 
> when the UI is initialized.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



[jira] [Commented] (SOLR-4063) CoreContainer should be able to load multiple SolrCore's in parallel rather than just serially.

2012-12-19 Thread Commit Tag Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/SOLR-4063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13536221#comment-13536221
 ] 

Commit Tag Bot commented on SOLR-4063:
--

[branch_4x commit] Yonik Seeley
http://svn.apache.org/viewvc?view=revision&revision=1423969

SOLR-4063: simplify core loading executor


> CoreContainer should be able to load multiple SolrCore's in parallel rather 
> than just serially.
> ---
>
> Key: SOLR-4063
> URL: https://issues.apache.org/jira/browse/SOLR-4063
> Project: Solr
>  Issue Type: Improvement
>Reporter: Mark Miller
>Assignee: Mark Miller
> Fix For: 4.1, 5.0
>
> Attachments: SOLR-4063_executorSimplify.patch, SOLR-4063.patch
>
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



[jira] [Assigned] (SOLR-3840) XML query response display is unreadable in Solr Admin Query UI

2012-12-19 Thread Stefan Matheis (steffkes) (JIRA)

 [ 
https://issues.apache.org/jira/browse/SOLR-3840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stefan Matheis (steffkes) reassigned SOLR-3840:
---

Assignee: Stefan Matheis (steffkes)

> XML query response display is unreadable in Solr Admin Query UI
> ---
>
> Key: SOLR-3840
> URL: https://issues.apache.org/jira/browse/SOLR-3840
> Project: Solr
>  Issue Type: Improvement
>  Components: web gui
>Affects Versions: 4.0-BETA
> Environment: Google Chrome, Windows 7 - Firefox as well
>Reporter: Jack Krupansky
>Assignee: Stefan Matheis (steffkes)
> Attachments: Query-UI-XML-unreadable.png, SOLR-3840.patch, 
> SOLR-3840.patch, solr-admin-ui-query-highlight-json.png, 
> solr-admin-ui-query-highlight-xml.png
>
>
> If I execute a query in the Solr Admin Query UI, the default XML "writer" 
> displays only the raw text of the Solr response XML element values, but none 
> of the XML syntax itself, rendering the response display on the Query page 
> almost completely useless. JSON, Ruby, et al display very reasonably 
> formatted output, but XML does not.
> You can click on the icon next to the generated query URL to display the 
> response in a browser tab of its own and then it does display the XML very 
> reasonably, but the framed display on the query page is simply not readable.
> My recollection is that the old UI had this same problem.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



[jira] [Updated] (SOLR-3840) XML query response display is unreadable in Solr Admin Query UI

2012-12-19 Thread Stefan Matheis (steffkes) (JIRA)

 [ 
https://issues.apache.org/jira/browse/SOLR-3840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stefan Matheis (steffkes) updated SOLR-3840:


Attachment: solr-admin-ui-query-highlight-xml.png
solr-admin-ui-query-highlight-json.png
SOLR-3840.patch

Updated Patch, includes Highlighting for XML as well as JSON (see attached 
screenshots for samples)

> XML query response display is unreadable in Solr Admin Query UI
> ---
>
> Key: SOLR-3840
> URL: https://issues.apache.org/jira/browse/SOLR-3840
> Project: Solr
>  Issue Type: Improvement
>  Components: web gui
>Affects Versions: 4.0-BETA
> Environment: Google Chrome, Windows 7 - Firefox as well
>Reporter: Jack Krupansky
> Attachments: Query-UI-XML-unreadable.png, SOLR-3840.patch, 
> SOLR-3840.patch, solr-admin-ui-query-highlight-json.png, 
> solr-admin-ui-query-highlight-xml.png
>
>
> If I execute a query in the Solr Admin Query UI, the default XML "writer" 
> displays only the raw text of the Solr response XML element values, but none 
> of the XML syntax itself, rendering the response display on the Query page 
> almost completely useless. JSON, Ruby, et al display very reasonably 
> formatted output, but XML does not.
> You can click on the icon next to the generated query URL to display the 
> response in a browser tab of its own and then it does display the XML very 
> reasonably, but the framed display on the query page is simply not readable.
> My recollection is that the old UI had this same problem.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



[jira] [Commented] (SOLR-3829) Admin UI Logging events broken if schema.xml defines a catch-all dynamicField with type ignored

2012-12-19 Thread Stefan Matheis (steffkes) (JIRA)

[ 
https://issues.apache.org/jira/browse/SOLR-3829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13536214#comment-13536214
 ] 

Stefan Matheis (steffkes) commented on SOLR-3829:
-

If no one objects i'd like to commit my hackish patch, so at least we have it 
still working. whoever is able to figure that out, no worries to drop out my 
temporary solution ;>

> Admin UI Logging events broken if schema.xml defines a catch-all dynamicField 
> with type ignored
> ---
>
> Key: SOLR-3829
> URL: https://issues.apache.org/jira/browse/SOLR-3829
> Project: Solr
>  Issue Type: Bug
>  Components: web gui
>Affects Versions: 4.0-BETA
>Reporter: Andreas Hubold
> Attachments: SOLR-3829.patch
>
>
> The Solr Admin page does not show any log events. There are Javascript errors
> {noformat}
> TypeError: doc.logger.esc is not a function
> ... '' + doc.logger.split( '.' 
> ).pop().esc()...
> {noformat}
> This is because the response of the LoggingHandler added unexpected {{[ ... 
> ]}} characters around the values for time, level, logger and message:
> {noformat}
> ...
> "history":{"numFound":2,"start":0,"docs":[{"time":["2012-09-11T15:07:05.453Z"],"level":["WARNING"],"logger":["org.apache.solr.core.SolrCore"],"message":["New
>  index directory detected: ...
> {noformat}
> This is caused by the way the JSON is created. 
> org.apache.solr.logging.LogWatcher#toSolrDocument creates a SolrDocument 
> which is then formatted with a org.apache.solr.response.JSONResponseWriter.
> But the JSONResponseWriter uses the index schema to decide how to format the 
> JSON. We have the following field declaration in schema.xml:
> {noformat}
> 
> {noformat}
> The field type "ignored" has the attribute multiValued set to true. Because 
> of this JSONResponseWriter adds [] characters in 
> org.apache.solr.response.JSONWriter#writeSolrDocument
> The formatting should be independent from schema.xml

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



RE: [JENKINS-MAVEN] Lucene-Solr-Maven-4.x #186: POMs out of sync

2012-12-19 Thread Uwe Schindler
Yeah, so we use the "special IPv6" addresses here for 2 reasons (also since my 
fix in DIH's test):
- If the machine does not support IPv6 you get the "protocol not supported" 
exception (IOException)
- If the machine has IPv6 enabled, you will get a "no route to host" (also 
IOException), because those addresses used in the URL are special private test 
addresses, which no router would ever forward.
In both cases you get an IOException when trying to connect to such a "dead 
server", which is all the test wants to check.

Previously those "dead servers" array was filled with 127.0.0.1 and a random, 
but unbound port, triggering the blackhole. 

-
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: u...@thetaphi.de


> -Original Message-
> From: Mark Miller [mailto:markrmil...@gmail.com]
> Sent: Wednesday, December 19, 2012 6:54 PM
> To: dev@lucene.apache.org
> Subject: Re: [JENKINS-MAVEN] Lucene-Solr-Maven-4.x #186: POMs out of
> sync
> 
> Ah, okay - if that is the expected exception then, the real problem is
> something else. I had not seen it before.
> 
> I'll try and figure out why it can't talk to any of the nodes then. This test 
> does
> not kill node or anything…
> 
> - Mark
> 
> On Dec 19, 2012, at 12:46 PM, "Uwe Schindler"  wrote:
> 
> > But this is exactly what we wanted! Those IPv6 server addresses are
> exactly to "emulate" servers that are down (see the array of those addresses
> in BaseDistributedTestCase)? So if you enable or disable IPv6 should not
> make a difference, the thing should only throw *any* exception and not
> make the blackhole happy.
> >
> > -
> > Uwe Schindler
> > H.-H.-Meier-Allee 63, D-28213 Bremen
> > http://www.thetaphi.de
> > eMail: u...@thetaphi.de
> >
> >
> >> -Original Message-
> >> From: Mark Miller [mailto:markrmil...@gmail.com]
> >> Sent: Wednesday, December 19, 2012 6:40 PM
> >> To: dev@lucene.apache.org
> >> Subject: Re: [JENKINS-MAVEN] Lucene-Solr-Maven-4.x #186: POMs out
> of
> >> sync
> >>
> >> So one of the tests that fails is the distrib mlt - I need to look at
> >> that test closer in general, but it's an odd fail - I see the same
> >> thing happen on my freebsd vm. Key is Caused by:
> >> java.net.SocketException: Protocol family unavailable.
> >>
> >> I tried what I could to enable ipv6 on my freebsd vm, but no luck on this
> one.
> >> Googling, everything I've seen suggests IPv6 as the possible problem.
> >> I don't know why other tests don't show the problem though...
> >>
> >> [junit4:junit4]   2>   Caused by:
> >> org.apache.solr.client.solrj.SolrServerException: No live SolrServers
> >> available to handle this request:[http://[ff01::083]:2,
> >> http://[ff01::114]:2, http://[ff01::213]:2, http://127.0.0.1:60340]
> >> [junit4:junit4]   2>   at
> >> org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolr
> >> Server.j
> >> ava:325)
> >> [junit4:junit4]   2>   at
> >>
> org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHa
> >> nd
> >> ler.java:171)
> >> [junit4:junit4]   2>   at
> >>
> org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHa
> >> nd
> >> ler.java:135)
> >> [junit4:junit4]   2>   at
> >> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> >> [junit4:junit4]   2>   at
> >> java.util.concurrent.FutureTask.run(FutureTask.java:166)
> >> [junit4:junit4]   2>   at
> >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> >> [junit4:junit4]   2>   at
> >> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> >> [junit4:junit4]   2>   at
> >> java.util.concurrent.FutureTask.run(FutureTask.java:166)
> >> [junit4:junit4]   2>   at
> >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.
> >> jav
> >> a:1110)
> >> [junit4:junit4]   2>   at
> >>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor
> >> .ja
> >> va:603)
> >> [junit4:junit4]   2>   ... 1 more
> >> [junit4:junit4]   2>   Caused by:
> >> org.apache.solr.client.solrj.SolrServerException: IOException occured
> >> when talking to server at: http://[ff01::083]:2
> >> [junit4:junit4]   2>   at
> >>
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:
> >> 413)
> >> [junit4:junit4]   2>   at
> >>
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:
> >> 181)
> >> [junit4:junit4]   2>   at
> >> org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolr
> >> Server.j
> >> ava:290)
> >> [junit4:junit4]   2>   ... 10 more
> >> [junit4:junit4]   2>   Caused by: java.net.SocketException: Protocol 
> >> family
> >> unavailable
> >>
> >> On Dec 19, 2012, at 9:35 AM, Uwe Schindler  wrote:
> >>
> >>> Juh! Thanks you.
> >>> The same applies to Mav

[jira] [Updated] (LUCENE-4609) Write a PackedIntsEncoder/Decoder for facets

2012-12-19 Thread Gilad Barkai (JIRA)

 [ 
https://issues.apache.org/jira/browse/LUCENE-4609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gilad Barkai updated LUCENE-4609:
-

Attachment: LUCENE-4609.patch

Attached a PackedEncoder, which is based on {{PackedInts}}. Currently only the 
approach of a 'per-document' bits-per-value is implemented.

I'm not convinced the header could be spared, as at the very least, the number 
of bits to neglect at the end of the stream should be written. E.g if there are 
2 bits per value, and there are 17 values, there's a need for 34 bits, but 
everything is written in (at least) bytes, so 6 bits should be neglected.

Updated EncodingTest and EncodingSpeed, and found out that the compression 
factor is not that good, probably due to large numbers which bumps the amount 
of required bits to higher value. 

Started to look into a semi-packed encoder, which could encode most values in a 
packed manner, but could also add large values as, e.g., vints.
Example: for 6 bits per value, all values 0-62 are packed, while a packed value 
of 63 (packed all 1' s) is a marker that the next value is written in a 
non-packed manner (say vint, Elias delta, whole 32 bits.. ). 
This should improve the compression factor when most ints are small, and only a 
few are large. 
Impact on encoding/decoding speed remains to be seen..

> Write a PackedIntsEncoder/Decoder for facets
> 
>
> Key: LUCENE-4609
> URL: https://issues.apache.org/jira/browse/LUCENE-4609
> Project: Lucene - Core
>  Issue Type: New Feature
>  Components: modules/facet
>Reporter: Shai Erera
>Priority: Minor
> Attachments: LUCENE-4609.patch
>
>
> Today the facets API lets you write IntEncoder/Decoder to encode/decode the 
> category ordinals. We have several such encoders, including VInt (default), 
> and block encoders.
> It would be interesting to implement and benchmark a 
> PackedIntsEncoder/Decoder, with potentially two variants: (1) receives 
> bitsPerValue up front, when you e.g. know that you have a small taxonomy and 
> the max value you can see and (2) one that decides for each doc on the 
> optimal bitsPerValue, writes it as a header in the byte[] or something.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



[jira] [Commented] (LUCENE-4638) If IndexWriter is interrupted on close and is using a channel (mmap/nio), it can throw a ClosedByInterruptException and prevent you from opening a new IndexWriter in t

2012-12-19 Thread Mark Miller (JIRA)

[ 
https://issues.apache.org/jira/browse/LUCENE-4638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13536207#comment-13536207
 ] 

Mark Miller commented on LUCENE-4638:
-

I should be able to find what it was hanging on, but a lot of logs to look back 
through. I can probably reproduce more easily instead when I get home tonight. 
If I remember right, it was trying to open an mmap input or something and if I 
remember right, it was just blocking. I'll reproduce and report the exact 
details.

> If IndexWriter is interrupted on close and is using a channel (mmap/nio), it 
> can throw a ClosedByInterruptException and prevent you from opening a new 
> IndexWriter in the same proceses if you are using Native locks.
> --
>
> Key: LUCENE-4638
> URL: https://issues.apache.org/jira/browse/LUCENE-4638
> Project: Lucene - Core
>  Issue Type: Bug
>Reporter: Mark Miller
>Priority: Minor
> Fix For: 4.1, 5.0
>
>
> The ClosedByInterruptException will prevent the index from being unlocked in 
> close. If you try and close again, the call will hang. If you are using 
> native locks and try to open a new IndexWriter, it will fail to get the lock. 
> If you try IW#forceUnlock, it wont work because the not fully closed IW will 
> still have the lock.
> ideas:
> * On ClosedByInterruptException, IW should continue trying to close what it 
> can and unlock the index? Generally I have see the exception trigger in 
> commitInternal.
> * We should add a non static forceUnlock to IW that lets you remove the lock 
> and start a new IW?
> * We should make the lock protected so IW sub classes could unlock the index 
> in advanced use cases?
> * others?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



[jira] [Commented] (LUCENE-4638) If IndexWriter is interrupted on close and is using a channel (mmap/nio), it can throw a ClosedByInterruptException and prevent you from opening a new IndexWriter in t

2012-12-19 Thread Michael McCandless (JIRA)

[ 
https://issues.apache.org/jira/browse/LUCENE-4638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13536196#comment-13536196
 ] 

Michael McCandless commented on LUCENE-4638:


Hmm why does the 2nd call to close hang?  Do you have the original exc?

IW.rollback() should do a "better job" closing and releasing the lock, and in 
general on getting an exception from IW.close I think it's the only real 
recourse you have (ie, it's hard to know what docs you lost due to that 
exception).

Also, I think LUCENE-4246 (IW.close should "just close", not wait for merges, 
commit, etc.) would improve this situation because then close would reliably 
release the lock.

> If IndexWriter is interrupted on close and is using a channel (mmap/nio), it 
> can throw a ClosedByInterruptException and prevent you from opening a new 
> IndexWriter in the same proceses if you are using Native locks.
> --
>
> Key: LUCENE-4638
> URL: https://issues.apache.org/jira/browse/LUCENE-4638
> Project: Lucene - Core
>  Issue Type: Bug
>Reporter: Mark Miller
>Priority: Minor
> Fix For: 4.1, 5.0
>
>
> The ClosedByInterruptException will prevent the index from being unlocked in 
> close. If you try and close again, the call will hang. If you are using 
> native locks and try to open a new IndexWriter, it will fail to get the lock. 
> If you try IW#forceUnlock, it wont work because the not fully closed IW will 
> still have the lock.
> ideas:
> * On ClosedByInterruptException, IW should continue trying to close what it 
> can and unlock the index? Generally I have see the exception trigger in 
> commitInternal.
> * We should add a non static forceUnlock to IW that lets you remove the lock 
> and start a new IW?
> * We should make the lock protected so IW sub classes could unlock the index 
> in advanced use cases?
> * others?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



Re: [JENKINS-MAVEN] Lucene-Solr-Maven-4.x #186: POMs out of sync

2012-12-19 Thread Mark Miller
Ah, okay - if that is the expected exception then, the real problem is 
something else. I had not seen it before.

I'll try and figure out why it can't talk to any of the nodes then. This test 
does not kill node or anything…

- Mark

On Dec 19, 2012, at 12:46 PM, "Uwe Schindler"  wrote:

> But this is exactly what we wanted! Those IPv6 server addresses are exactly 
> to "emulate" servers that are down (see the array of those addresses in 
> BaseDistributedTestCase)? So if you enable or disable IPv6 should not make a 
> difference, the thing should only throw *any* exception and not make the 
> blackhole happy.
> 
> -
> Uwe Schindler
> H.-H.-Meier-Allee 63, D-28213 Bremen
> http://www.thetaphi.de
> eMail: u...@thetaphi.de
> 
> 
>> -Original Message-
>> From: Mark Miller [mailto:markrmil...@gmail.com]
>> Sent: Wednesday, December 19, 2012 6:40 PM
>> To: dev@lucene.apache.org
>> Subject: Re: [JENKINS-MAVEN] Lucene-Solr-Maven-4.x #186: POMs out of
>> sync
>> 
>> So one of the tests that fails is the distrib mlt - I need to look at that 
>> test
>> closer in general, but it's an odd fail - I see the same thing happen on my
>> freebsd vm. Key is Caused by: java.net.SocketException: Protocol family
>> unavailable.
>> 
>> I tried what I could to enable ipv6 on my freebsd vm, but no luck on this 
>> one.
>> Googling, everything I've seen suggests IPv6 as the possible problem. I don't
>> know why other tests don't show the problem though...
>> 
>> [junit4:junit4]   2> Caused by:
>> org.apache.solr.client.solrj.SolrServerException: No live SolrServers 
>> available
>> to handle this request:[http://[ff01::083]:2, http://[ff01::114]:2,
>> http://[ff01::213]:2, http://127.0.0.1:60340]
>> [junit4:junit4]   2> at
>> org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.j
>> ava:325)
>> [junit4:junit4]   2> at
>> org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHand
>> ler.java:171)
>> [junit4:junit4]   2> at
>> org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHand
>> ler.java:135)
>> [junit4:junit4]   2> at
>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
>> [junit4:junit4]   2> at
>> java.util.concurrent.FutureTask.run(FutureTask.java:166)
>> [junit4:junit4]   2> at
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>> [junit4:junit4]   2> at
>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
>> [junit4:junit4]   2> at
>> java.util.concurrent.FutureTask.run(FutureTask.java:166)
>> [junit4:junit4]   2> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
>> a:1110)
>> [junit4:junit4]   2> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
>> va:603)
>> [junit4:junit4]   2> ... 1 more
>> [junit4:junit4]   2> Caused by:
>> org.apache.solr.client.solrj.SolrServerException: IOException occured when
>> talking to server at: http://[ff01::083]:2
>> [junit4:junit4]   2> at
>> org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:
>> 413)
>> [junit4:junit4]   2> at
>> org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:
>> 181)
>> [junit4:junit4]   2> at
>> org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.j
>> ava:290)
>> [junit4:junit4]   2> ... 10 more
>> [junit4:junit4]   2> Caused by: java.net.SocketException: Protocol 
>> family
>> unavailable
>> 
>> On Dec 19, 2012, at 9:35 AM, Uwe Schindler  wrote:
>> 
>>> Juh! Thanks you.
>>> The same applies to Maven ANT builds, they are the same tests failing.
>>> The nightly builds also seem to pass. :-)
>>> 
>>> -
>>> Uwe Schindler
>>> H.-H.-Meier-Allee 63, D-28213 Bremen
>>> http://www.thetaphi.de
>>> eMail: u...@thetaphi.de
>>> 
>>> 
 -Original Message-
 From: Mark Miller [mailto:markrmil...@gmail.com]
 Sent: Wednesday, December 19, 2012 3:21 PM
 To: dev@lucene.apache.org
 Subject: Re: [JENKINS-MAVEN] Lucene-Solr-Maven-4.x #186: POMs out
>> of
 sync
 
 Nice - down to 2 tests failed on the freebsd maven build. We will get
 this passing soon.
 
 - Mark
 
 On Dec 19, 2012, at 9:16 AM, Apache Jenkins Server
  wrote:
 
> Build: https://builds.apache.org/job/Lucene-Solr-Maven-4.x/186/
> 
> 2 tests failed.
> REGRESSION:  org.apache.solr.cloud.RecoveryZkTest.testDistribSearch
> 
> Error Message:
> expected:<280> but was:<42>
> 
> Stack Trace:
> java.lang.AssertionError: expected:<280> but was:<42>
>   at
 
>> __randomizedtesting.SeedInfo.seed([A36CEAF4C1EE24D2:228A64ECB6B144E
 E]:0)
>   at org.junit.Assert.fail(Assert.java:93)

RE: [JENKINS-MAVEN] Lucene-Solr-Maven-4.x #186: POMs out of sync

2012-12-19 Thread Uwe Schindler
But this is exactly what we wanted! Those IPv6 server addresses are exactly to 
"emulate" servers that are down (see the array of those addresses in 
BaseDistributedTestCase)? So if you enable or disable IPv6 should not make a 
difference, the thing should only throw *any* exception and not make the 
blackhole happy.

-
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: u...@thetaphi.de


> -Original Message-
> From: Mark Miller [mailto:markrmil...@gmail.com]
> Sent: Wednesday, December 19, 2012 6:40 PM
> To: dev@lucene.apache.org
> Subject: Re: [JENKINS-MAVEN] Lucene-Solr-Maven-4.x #186: POMs out of
> sync
> 
> So one of the tests that fails is the distrib mlt - I need to look at that 
> test
> closer in general, but it's an odd fail - I see the same thing happen on my
> freebsd vm. Key is Caused by: java.net.SocketException: Protocol family
> unavailable.
> 
> I tried what I could to enable ipv6 on my freebsd vm, but no luck on this one.
> Googling, everything I've seen suggests IPv6 as the possible problem. I don't
> know why other tests don't show the problem though...
> 
> [junit4:junit4]   2>  Caused by:
> org.apache.solr.client.solrj.SolrServerException: No live SolrServers 
> available
> to handle this request:[http://[ff01::083]:2, http://[ff01::114]:2,
> http://[ff01::213]:2, http://127.0.0.1:60340]
> [junit4:junit4]   2>  at
> org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.j
> ava:325)
> [junit4:junit4]   2>  at
> org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHand
> ler.java:171)
> [junit4:junit4]   2>  at
> org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHand
> ler.java:135)
> [junit4:junit4]   2>  at
> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> [junit4:junit4]   2>  at
> java.util.concurrent.FutureTask.run(FutureTask.java:166)
> [junit4:junit4]   2>  at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> [junit4:junit4]   2>  at
> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> [junit4:junit4]   2>  at
> java.util.concurrent.FutureTask.run(FutureTask.java:166)
> [junit4:junit4]   2>  at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
> a:1110)
> [junit4:junit4]   2>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
> va:603)
> [junit4:junit4]   2>  ... 1 more
> [junit4:junit4]   2>  Caused by:
> org.apache.solr.client.solrj.SolrServerException: IOException occured when
> talking to server at: http://[ff01::083]:2
> [junit4:junit4]   2>  at
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:
> 413)
> [junit4:junit4]   2>  at
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:
> 181)
> [junit4:junit4]   2>  at
> org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.j
> ava:290)
> [junit4:junit4]   2>  ... 10 more
> [junit4:junit4]   2>  Caused by: java.net.SocketException: Protocol family
> unavailable
> 
> On Dec 19, 2012, at 9:35 AM, Uwe Schindler  wrote:
> 
> > Juh! Thanks you.
> > The same applies to Maven ANT builds, they are the same tests failing.
> > The nightly builds also seem to pass. :-)
> >
> > -
> > Uwe Schindler
> > H.-H.-Meier-Allee 63, D-28213 Bremen
> > http://www.thetaphi.de
> > eMail: u...@thetaphi.de
> >
> >
> >> -Original Message-
> >> From: Mark Miller [mailto:markrmil...@gmail.com]
> >> Sent: Wednesday, December 19, 2012 3:21 PM
> >> To: dev@lucene.apache.org
> >> Subject: Re: [JENKINS-MAVEN] Lucene-Solr-Maven-4.x #186: POMs out
> of
> >> sync
> >>
> >> Nice - down to 2 tests failed on the freebsd maven build. We will get
> >> this passing soon.
> >>
> >> - Mark
> >>
> >> On Dec 19, 2012, at 9:16 AM, Apache Jenkins Server
> >>  wrote:
> >>
> >>> Build: https://builds.apache.org/job/Lucene-Solr-Maven-4.x/186/
> >>>
> >>> 2 tests failed.
> >>> REGRESSION:  org.apache.solr.cloud.RecoveryZkTest.testDistribSearch
> >>>
> >>> Error Message:
> >>> expected:<280> but was:<42>
> >>>
> >>> Stack Trace:
> >>> java.lang.AssertionError: expected:<280> but was:<42>
> >>>   at
> >>
> __randomizedtesting.SeedInfo.seed([A36CEAF4C1EE24D2:228A64ECB6B144E
> >> E]:0)
> >>>   at org.junit.Assert.fail(Assert.java:93)
> >>>   at org.junit.Assert.failNotEquals(Assert.java:647)
> >>>   at org.junit.Assert.assertEquals(Assert.java:128)
> >>>   at org.junit.Assert.assertEquals(Assert.java:472)
> >>>   at org.junit.Assert.assertEquals(Assert.java:456)
> >>>   at
> >> org.apache.solr.cloud.RecoveryZkTest.doTest(RecoveryZkTest.java:99)
> >>>   at
> >> org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseD
> >> istri
> >> butedSearchTestCase.java:794)
> >>>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >>>   

[jira] [Resolved] (SOLR-4218) SolrTestCaseJ4 throws NPE when closing the core (on the afterClass method)

2012-12-19 Thread Mark Miller (JIRA)

 [ 
https://issues.apache.org/jira/browse/SOLR-4218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mark Miller resolved SOLR-4218.
---

Resolution: Fixed

Thanks Tomas!

> SolrTestCaseJ4 throws NPE when closing the core (on the afterClass method)
> --
>
> Key: SOLR-4218
> URL: https://issues.apache.org/jira/browse/SOLR-4218
> Project: Solr
>  Issue Type: Bug
>Reporter: Tomás Fernández Löbbe
>Assignee: Mark Miller
> Fix For: 4.1, 5.0
>
> Attachments: SOLR-4218.patch
>
>
> When running a specific test like:
> ant test -Dtestcase=BasicFunctionalityTest
> at the end of the test there is a NPE.
> {code}
> [junit4:junit4]   2> 13384 T10 oasc.SolrException.log SEVERE 
> java.lang.NullPointerException
> [junit4:junit4]   2>  at 
> org.apache.solr.handler.component.HttpShardHandlerFactory.close(HttpShardHandlerFactory.java:165)
> [junit4:junit4]   2>  at 
> org.apache.solr.core.CoreContainer.shutdown(CoreContainer.java:786)
> [junit4:junit4]   2>  at 
> org.apache.solr.util.TestHarness.close(TestHarness.java:449)
> [junit4:junit4]   2>  at 
> org.apache.solr.SolrTestCaseJ4.deleteCore(SolrTestCaseJ4.java:415)
> [junit4:junit4]   2>  at 
> org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:95)
> [junit4:junit4]   2>  at 
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> [junit4:junit4]   2>  at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> [junit4:junit4]   2>  at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> [junit4:junit4]   2>  at 
> java.lang.reflect.Method.invoke(Method.java:601)
> [junit4:junit4]   2>  at 
> com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
> [junit4:junit4]   2>  at 
> com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
> [junit4:junit4]   2>  at 
> com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:700)
> [junit4:junit4]   2>  at 
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> [junit4:junit4]   2>  at 
> com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
> [junit4:junit4]   2>  at 
> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
> [junit4:junit4]   2>  at 
> org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
> [junit4:junit4]   2>  at 
> com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
> [junit4:junit4]   2>  at 
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
> [junit4:junit4]   2>  at 
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
> [junit4:junit4]   2>  at 
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> [junit4:junit4]   2>  at 
> org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
> [junit4:junit4]   2>  at 
> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
> [junit4:junit4]   2>  at 
> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
> [junit4:junit4]   2>  at 
> org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
> [junit4:junit4]   2>  at 
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> [junit4:junit4]   2>  at 
> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
> [junit4:junit4]   2>  at java.lang.Thread.run(Thread.java:722)
> [junit4:junit4]   2>  
> [junit4:junit4]   2> 13389 T10 oasc.SolrException.log SEVERE 
> java.lang.NullPointerException
> [junit4:junit4]   2>  at 
> org.apache.solr.handler.component.HttpShardHandlerFactory.close(HttpShardHandlerFactory.java:170)
> [junit4:junit4]   2>  at 
> org.apache.solr.core.CoreContainer.shutdown(CoreContainer.java:786)
> [junit4:junit4]   2>  at 
> org.apache.solr.util.TestHarness.close(TestHarness.java:449)
> [junit4:junit4]   2>  at 
> org.apache.solr.SolrTestCaseJ4.deleteCore(SolrTestCaseJ4.java:415)
> [junit4:junit4]   2>  at 
> org.apache.solr.SolrTestCaseJ4.afterClass(So

[jira] [Commented] (SOLR-4086) Refactor DIH - VariableResolver & Evaluator

2012-12-19 Thread James Dyer (JIRA)

[ 
https://issues.apache.org/jira/browse/SOLR-4086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13536180#comment-13536180
 ] 

James Dyer commented on SOLR-4086:
--

Dominik,

Thank you for reporting this!  Could you post your data-config.xml so I can get 
an idea of which features you're using?  Are these full or delta imports?  Are 
you using DateFormatEvaluator ( ${dataimporter.functions.formatDate(...) ) ? 

This issue involved extensive rework to the VariableResolver(Impl) class, and 
to the Evaluator framework, with the aim on making the code easier to 
understand and to maintain.  Whenever this type of change is made, it is always 
possible that the new implementation will suffer performance-wise.  I will look 
with you on this: We certainly do not want massive performance decreases to get 
released. :)  

One thing that jumps out at me is the old version stored a cache of parsed-out 
template strings (ex: ${foo.bar} ) in a HashMap.  I was worried that this could 
potentially consume too much memory and changed this to a WeakHashMap.  But if 
your JVM is clearing out these WeakReferences frequently it might require a lot 
more work to keep re-parsing these strings.  

Another problem could be with DateFormatEvaluator.  It used to keep a single 
instance of SimpleDateFormat (per-thread) and always use that as the 
"from-date-format" but now that Locales are involved this doesn't work.  To 
alleviate this it only creates one Dateformat per pattern/locale combination 
and caches these, also in a WeakHashMap.  This also might suffer from 
WeakReferences going away, but more seriously, I think having this map as an 
instance variable here entirely defeats its purpose if the VariableResolver is 
creating a new instance each time it is being used.  So I need to look into 
that.

probably the biggest change is DateFormatEvaluator

> Refactor DIH - VariableResolver & Evaluator
> ---
>
> Key: SOLR-4086
> URL: https://issues.apache.org/jira/browse/SOLR-4086
> Project: Solr
>  Issue Type: Improvement
>  Components: contrib - DataImportHandler
>Affects Versions: 4.0
>Reporter: James Dyer
>Assignee: James Dyer
>Priority: Minor
> Fix For: 4.1, 5.0
>
> Attachments: SOLR-4086.patch
>
>
> This simplifies VariableResolver and moves each built-in Evaluator into its 
> own class.  Compiler warnings / missing generics are fixed.  Also, the Locale 
> bug with DateFormatEvaluator is solved.  Instead of using the machine 
> default, the Root Locale is used by default.  An optional 3rd parameter 
> allows users to specify whatever locale they want.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



Re: [JENKINS-MAVEN] Lucene-Solr-Maven-4.x #186: POMs out of sync

2012-12-19 Thread Mark Miller
So one of the tests that fails is the distrib mlt - I need to look at that test 
closer in general, but it's an odd fail - I see the same thing happen on my 
freebsd vm. Key is Caused by: java.net.SocketException: Protocol family 
unavailable.

I tried what I could to enable ipv6 on my freebsd vm, but no luck on this one. 
Googling, everything I've seen suggests IPv6 as the possible problem. I don't 
know why other tests don't show the problem though...

[junit4:junit4]   2>Caused by: 
org.apache.solr.client.solrj.SolrServerException: No live SolrServers available 
to handle this request:[http://[ff01::083]:2, http://[ff01::114]:2, 
http://[ff01::213]:2, http://127.0.0.1:60340]
[junit4:junit4]   2>at 
org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:325)
[junit4:junit4]   2>at 
org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:171)
[junit4:junit4]   2>at 
org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:135)
[junit4:junit4]   2>at 
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
[junit4:junit4]   2>at 
java.util.concurrent.FutureTask.run(FutureTask.java:166)
[junit4:junit4]   2>at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
[junit4:junit4]   2>at 
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
[junit4:junit4]   2>at 
java.util.concurrent.FutureTask.run(FutureTask.java:166)
[junit4:junit4]   2>at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
[junit4:junit4]   2>at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
[junit4:junit4]   2>... 1 more
[junit4:junit4]   2>Caused by: 
org.apache.solr.client.solrj.SolrServerException: IOException occured when 
talking to server at: http://[ff01::083]:2
[junit4:junit4]   2>at 
org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:413)
[junit4:junit4]   2>at 
org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:181)
[junit4:junit4]   2>at 
org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:290)
[junit4:junit4]   2>... 10 more
[junit4:junit4]   2>Caused by: java.net.SocketException: Protocol family 
unavailable

On Dec 19, 2012, at 9:35 AM, Uwe Schindler  wrote:

> Juh! Thanks you.
> The same applies to Maven ANT builds, they are the same tests failing. The 
> nightly builds also seem to pass. :-)
> 
> -
> Uwe Schindler
> H.-H.-Meier-Allee 63, D-28213 Bremen
> http://www.thetaphi.de
> eMail: u...@thetaphi.de
> 
> 
>> -Original Message-
>> From: Mark Miller [mailto:markrmil...@gmail.com]
>> Sent: Wednesday, December 19, 2012 3:21 PM
>> To: dev@lucene.apache.org
>> Subject: Re: [JENKINS-MAVEN] Lucene-Solr-Maven-4.x #186: POMs out of
>> sync
>> 
>> Nice - down to 2 tests failed on the freebsd maven build. We will get this
>> passing soon.
>> 
>> - Mark
>> 
>> On Dec 19, 2012, at 9:16 AM, Apache Jenkins Server
>>  wrote:
>> 
>>> Build: https://builds.apache.org/job/Lucene-Solr-Maven-4.x/186/
>>> 
>>> 2 tests failed.
>>> REGRESSION:  org.apache.solr.cloud.RecoveryZkTest.testDistribSearch
>>> 
>>> Error Message:
>>> expected:<280> but was:<42>
>>> 
>>> Stack Trace:
>>> java.lang.AssertionError: expected:<280> but was:<42>
>>> at
>> __randomizedtesting.SeedInfo.seed([A36CEAF4C1EE24D2:228A64ECB6B144E
>> E]:0)
>>> at org.junit.Assert.fail(Assert.java:93)
>>> at org.junit.Assert.failNotEquals(Assert.java:647)
>>> at org.junit.Assert.assertEquals(Assert.java:128)
>>> at org.junit.Assert.assertEquals(Assert.java:472)
>>> at org.junit.Assert.assertEquals(Assert.java:456)
>>> at
>> org.apache.solr.cloud.RecoveryZkTest.doTest(RecoveryZkTest.java:99)
>>> at
>> org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistri
>> butedSearchTestCase.java:794)
>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.j
>> ava:57)
>>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
>> sorImpl.java:43)
>>> at java.lang.reflect.Method.invoke(Method.java:616)
>>> at
>> com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(Randomize
>> dRunner.java:1559)
>>> at
>> com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(Rando
>> mizedRunner.java:79)
>>> at
>> com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(Rando
>> mizedRunner.java:737)
>>> at
>> com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(Rando
>> mizedRunner.java:773)
>>> at
>> com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(Rando
>> mizedRunner.java:787)
>>> at
>> com.car

[jira] [Updated] (LUCENE-4599) Compressed term vectors

2012-12-19 Thread Adrien Grand (JIRA)

 [ 
https://issues.apache.org/jira/browse/LUCENE-4599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Adrien Grand updated LUCENE-4599:
-

Attachment: LUCENE-4599.patch

New patch (still not committable yet) with better compression ratio thanks to 
the following optimizations:
 * the block of data compressed by LZ4 only contains term and payload bytes 
(without their lengths), everything else (positions, flags, term lengths, etc.) 
is stored using packed ints,
 * term freqs are encoded in a pfor-like way to save space (this was a 3x/4x 
decrease of the space needed to store freqs),
 * when all fields have the same flags (a 3-bits int that says whether 
positions/offsets/payloads are enabled), the flag is stored only once per 
distinct field,
 * when both positions and offsets are enabled, I compute average term lengths 
and only store the difference between the start offset and the expected start 
offset computed from the average term length and the position,
 * for lengths, this impl stores the difference between the indexed term length 
and the actual length (endOffset - startOffset), with an optimization when they 
are always equal to 0 (can happen with ASCII and an analyzer that does not 
perform stemming).

Depending on the size of docs, not the same data takes most space in a single 
chunk:
|| || Small docs (28 * 1K) || Large doc (1 * 750K) ||
| Total chunk size (positions and offsets enabled) | 21K | 450K |
| Term bytes | 11K (16K before compression) | 64K (84K before compression) |
| Term lengths | 2K | 8K |
| Positions | 3K | 215K |
| Offsets | 3K (4K if positions are disabled) | 150K (240K if positions are 
disabled) |
| Term freqs | 500 | 7K |
the rest is negligible

 * So with small docs, most of space is occupied by term bytes whereas with 
large docs positions and offsets can easily take 80% of the chunk size.
 * Compression might not be as good as with stored fields, especially when docs 
are large because terms have already been deduplicated.

Overall, the on-disk format is more compact than the Lucene40 term vectors 
format (positions and offsets enabled, the number of documents indexed is not 
the same for small and large docs):
|| || Small docs || Large docs ||
| Lucene40 tvx | 160033 | 1633 |
| Lucene40 tvd | 49971 | 232 |
| Lucene40 tvf | 11279483 | 56640734 |
| Compressing tvx | 1116 | 78 |
| Compressing tvd | 7589550 | 44633841 |

This impl is 34% smaller than the Lucene40 one on small docs (mainly thanks to 
compression) and 21% on large docs (mainly thanks to packed ints). If you have 
other ideas to improve this ratio, let me know!

I still have to write more tests, clean up the patch, make reading term vectors 
more memory-efficient, and implement efficient merging...

> Compressed term vectors
> ---
>
> Key: LUCENE-4599
> URL: https://issues.apache.org/jira/browse/LUCENE-4599
> Project: Lucene - Core
>  Issue Type: Task
>  Components: core/codecs, core/termvectors
>Reporter: Adrien Grand
>Assignee: Adrien Grand
>Priority: Minor
> Fix For: 4.1
>
> Attachments: LUCENE-4599.patch, LUCENE-4599.patch
>
>
> We should have codec-compressed term vectors similarly to what we have with 
> stored fields.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



[jira] [Commented] (SOLR-4063) CoreContainer should be able to load multiple SolrCore's in parallel rather than just serially.

2012-12-19 Thread Commit Tag Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/SOLR-4063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13536155#comment-13536155
 ] 

Commit Tag Bot commented on SOLR-4063:
--

[trunk commit] Yonik Seeley
http://svn.apache.org/viewvc?view=revision&revision=1423963

SOLR-4063: simplify core loading executor


> CoreContainer should be able to load multiple SolrCore's in parallel rather 
> than just serially.
> ---
>
> Key: SOLR-4063
> URL: https://issues.apache.org/jira/browse/SOLR-4063
> Project: Solr
>  Issue Type: Improvement
>Reporter: Mark Miller
>Assignee: Mark Miller
> Fix For: 4.1, 5.0
>
> Attachments: SOLR-4063_executorSimplify.patch, SOLR-4063.patch
>
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



[jira] [Resolved] (SOLR-4047) dataimporter.functions.encodeUrl throughs Unable to encode expression: field.name with value: null

2012-12-19 Thread James Dyer (JIRA)

 [ 
https://issues.apache.org/jira/browse/SOLR-4047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

James Dyer resolved SOLR-4047.
--

Resolution: Not A Problem

> dataimporter.functions.encodeUrl throughs Unable to encode expression: 
> field.name with value: null
> --
>
> Key: SOLR-4047
> URL: https://issues.apache.org/jira/browse/SOLR-4047
> Project: Solr
>  Issue Type: Bug
>  Components: contrib - DataImportHandler
>Affects Versions: 4.0
> Environment: Windows 7
>Reporter: Igor Dobritskiy
>Assignee: James Dyer
>Priority: Critical
> Attachments: db-data-config.xml, db.sql, schema.xml, solrconfig.xml
>
>
> For some reason dataimporter.functions.encoude URL stopped work after update 
> to solr 4.0 from 3.5.
> Here is the error
> {code}
> Full Import failed:java.lang.RuntimeException: java.lang.RuntimeException: 
> org.apache.solr.handler.dataimport.DataImportHandlerException: Unable to 
> encode expression: attach.name with value: null Processing Document # 1
> {code}
> Here is the data import config snippet:
> {code}
> ...
>  query="select name from accounts where account_id = 
> '${attach.account_id}'">
>  dataSource="bin"
> format="text" 
> 
> url="http://example.com/data/${account.name}/attaches/${attach.item_id}/${dataimporter.functions.encodeUrl(attach.name)}">
> 
>  
> 
> ...
> {code}
> When I'm changing it to *not* use dataimporter.functions.encodeUrl it works 
> but I need to url encode file names as they have special chars in theirs 
> names.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



[jira] [Commented] (SOLR-4218) SolrTestCaseJ4 throws NPE when closing the core (on the afterClass method)

2012-12-19 Thread Commit Tag Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/SOLR-4218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13536126#comment-13536126
 ] 

Commit Tag Bot commented on SOLR-4218:
--

[branch_4x commit] Mark Robert Miller
http://svn.apache.org/viewvc?view=revision&revision=1423934

SOLR-4218: SolrTestCaseJ4 throws NPE when closing the core (on the afterClass 
method)


> SolrTestCaseJ4 throws NPE when closing the core (on the afterClass method)
> --
>
> Key: SOLR-4218
> URL: https://issues.apache.org/jira/browse/SOLR-4218
> Project: Solr
>  Issue Type: Bug
>Reporter: Tomás Fernández Löbbe
>Assignee: Mark Miller
> Fix For: 4.1, 5.0
>
> Attachments: SOLR-4218.patch
>
>
> When running a specific test like:
> ant test -Dtestcase=BasicFunctionalityTest
> at the end of the test there is a NPE.
> {code}
> [junit4:junit4]   2> 13384 T10 oasc.SolrException.log SEVERE 
> java.lang.NullPointerException
> [junit4:junit4]   2>  at 
> org.apache.solr.handler.component.HttpShardHandlerFactory.close(HttpShardHandlerFactory.java:165)
> [junit4:junit4]   2>  at 
> org.apache.solr.core.CoreContainer.shutdown(CoreContainer.java:786)
> [junit4:junit4]   2>  at 
> org.apache.solr.util.TestHarness.close(TestHarness.java:449)
> [junit4:junit4]   2>  at 
> org.apache.solr.SolrTestCaseJ4.deleteCore(SolrTestCaseJ4.java:415)
> [junit4:junit4]   2>  at 
> org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:95)
> [junit4:junit4]   2>  at 
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> [junit4:junit4]   2>  at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> [junit4:junit4]   2>  at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> [junit4:junit4]   2>  at 
> java.lang.reflect.Method.invoke(Method.java:601)
> [junit4:junit4]   2>  at 
> com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
> [junit4:junit4]   2>  at 
> com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
> [junit4:junit4]   2>  at 
> com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:700)
> [junit4:junit4]   2>  at 
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> [junit4:junit4]   2>  at 
> com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
> [junit4:junit4]   2>  at 
> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
> [junit4:junit4]   2>  at 
> org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
> [junit4:junit4]   2>  at 
> com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
> [junit4:junit4]   2>  at 
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
> [junit4:junit4]   2>  at 
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
> [junit4:junit4]   2>  at 
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> [junit4:junit4]   2>  at 
> org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
> [junit4:junit4]   2>  at 
> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
> [junit4:junit4]   2>  at 
> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
> [junit4:junit4]   2>  at 
> org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
> [junit4:junit4]   2>  at 
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> [junit4:junit4]   2>  at 
> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
> [junit4:junit4]   2>  at java.lang.Thread.run(Thread.java:722)
> [junit4:junit4]   2>  
> [junit4:junit4]   2> 13389 T10 oasc.SolrException.log SEVERE 
> java.lang.NullPointerException
> [junit4:junit4]   2>  at 
> org.apache.solr.handler.component.HttpShardHandlerFactory.close(HttpShardHandlerFactory.java:170)
> [junit4:junit4]   2>  at 
> org.apache.solr.core.CoreContainer.shutdown(CoreContainer.java:786)
> [junit4:junit4]   2>  at 
> org.apache.solr.util.TestHarness.clos

[jira] [Updated] (SOLR-4063) CoreContainer should be able to load multiple SolrCore's in parallel rather than just serially.

2012-12-19 Thread Yonik Seeley (JIRA)

 [ 
https://issues.apache.org/jira/browse/SOLR-4063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yonik Seeley updated SOLR-4063:
---

Attachment: SOLR-4063_executorSimplify.patch

Attaching patch for simplification of the core executor, removing the use of 
semaphore.

> CoreContainer should be able to load multiple SolrCore's in parallel rather 
> than just serially.
> ---
>
> Key: SOLR-4063
> URL: https://issues.apache.org/jira/browse/SOLR-4063
> Project: Solr
>  Issue Type: Improvement
>Reporter: Mark Miller
>Assignee: Mark Miller
> Fix For: 4.1, 5.0
>
> Attachments: SOLR-4063_executorSimplify.patch, SOLR-4063.patch
>
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



[jira] [Commented] (SOLR-4218) SolrTestCaseJ4 throws NPE when closing the core (on the afterClass method)

2012-12-19 Thread Commit Tag Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/SOLR-4218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13536109#comment-13536109
 ] 

Commit Tag Bot commented on SOLR-4218:
--

[trunk commit] Mark Robert Miller
http://svn.apache.org/viewvc?view=revision&revision=1423932

SOLR-4218: SolrTestCaseJ4 throws NPE when closing the core (on the afterClass 
method)


> SolrTestCaseJ4 throws NPE when closing the core (on the afterClass method)
> --
>
> Key: SOLR-4218
> URL: https://issues.apache.org/jira/browse/SOLR-4218
> Project: Solr
>  Issue Type: Bug
>Reporter: Tomás Fernández Löbbe
>Assignee: Mark Miller
> Fix For: 4.1, 5.0
>
> Attachments: SOLR-4218.patch
>
>
> When running a specific test like:
> ant test -Dtestcase=BasicFunctionalityTest
> at the end of the test there is a NPE.
> {code}
> [junit4:junit4]   2> 13384 T10 oasc.SolrException.log SEVERE 
> java.lang.NullPointerException
> [junit4:junit4]   2>  at 
> org.apache.solr.handler.component.HttpShardHandlerFactory.close(HttpShardHandlerFactory.java:165)
> [junit4:junit4]   2>  at 
> org.apache.solr.core.CoreContainer.shutdown(CoreContainer.java:786)
> [junit4:junit4]   2>  at 
> org.apache.solr.util.TestHarness.close(TestHarness.java:449)
> [junit4:junit4]   2>  at 
> org.apache.solr.SolrTestCaseJ4.deleteCore(SolrTestCaseJ4.java:415)
> [junit4:junit4]   2>  at 
> org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:95)
> [junit4:junit4]   2>  at 
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> [junit4:junit4]   2>  at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> [junit4:junit4]   2>  at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> [junit4:junit4]   2>  at 
> java.lang.reflect.Method.invoke(Method.java:601)
> [junit4:junit4]   2>  at 
> com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
> [junit4:junit4]   2>  at 
> com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
> [junit4:junit4]   2>  at 
> com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:700)
> [junit4:junit4]   2>  at 
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> [junit4:junit4]   2>  at 
> com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
> [junit4:junit4]   2>  at 
> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
> [junit4:junit4]   2>  at 
> org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
> [junit4:junit4]   2>  at 
> com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
> [junit4:junit4]   2>  at 
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
> [junit4:junit4]   2>  at 
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
> [junit4:junit4]   2>  at 
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> [junit4:junit4]   2>  at 
> org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
> [junit4:junit4]   2>  at 
> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
> [junit4:junit4]   2>  at 
> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
> [junit4:junit4]   2>  at 
> org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
> [junit4:junit4]   2>  at 
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> [junit4:junit4]   2>  at 
> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
> [junit4:junit4]   2>  at java.lang.Thread.run(Thread.java:722)
> [junit4:junit4]   2>  
> [junit4:junit4]   2> 13389 T10 oasc.SolrException.log SEVERE 
> java.lang.NullPointerException
> [junit4:junit4]   2>  at 
> org.apache.solr.handler.component.HttpShardHandlerFactory.close(HttpShardHandlerFactory.java:170)
> [junit4:junit4]   2>  at 
> org.apache.solr.core.CoreContainer.shutdown(CoreContainer.java:786)
> [junit4:junit4]   2>  at 
> org.apache.solr.util.TestHarness.close(Te

VOTE: release 3.6.2

2012-12-19 Thread Robert Muir
Hello everyone, given the bug Mike fixed in
https://issues.apache.org/jira/browse/LUCENE-4635, I think we should
push out a 3.6.2 bugfix release with the fix.
Tom (the reporter of the bug) confirmed his checkindex passed
successfully last night, and besides that we have a few other nice
bugfixes there.

Artifacts are here: http://s.apache.org/lusolr362rc0

If you want to use the smoketester, of course you must run it from the
lucene_solr_3_6 branch. This passes for me on linux. I'm not sure it
works on other platforms.

Thanks,
Robert

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



[jira] [Assigned] (SOLR-4218) SolrTestCaseJ4 throws NPE when closing the core (on the afterClass method)

2012-12-19 Thread Mark Miller (JIRA)

 [ 
https://issues.apache.org/jira/browse/SOLR-4218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mark Miller reassigned SOLR-4218:
-

Assignee: Mark Miller

> SolrTestCaseJ4 throws NPE when closing the core (on the afterClass method)
> --
>
> Key: SOLR-4218
> URL: https://issues.apache.org/jira/browse/SOLR-4218
> Project: Solr
>  Issue Type: Bug
>Reporter: Tomás Fernández Löbbe
>Assignee: Mark Miller
> Fix For: 4.1, 5.0
>
> Attachments: SOLR-4218.patch
>
>
> When running a specific test like:
> ant test -Dtestcase=BasicFunctionalityTest
> at the end of the test there is a NPE.
> {code}
> [junit4:junit4]   2> 13384 T10 oasc.SolrException.log SEVERE 
> java.lang.NullPointerException
> [junit4:junit4]   2>  at 
> org.apache.solr.handler.component.HttpShardHandlerFactory.close(HttpShardHandlerFactory.java:165)
> [junit4:junit4]   2>  at 
> org.apache.solr.core.CoreContainer.shutdown(CoreContainer.java:786)
> [junit4:junit4]   2>  at 
> org.apache.solr.util.TestHarness.close(TestHarness.java:449)
> [junit4:junit4]   2>  at 
> org.apache.solr.SolrTestCaseJ4.deleteCore(SolrTestCaseJ4.java:415)
> [junit4:junit4]   2>  at 
> org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:95)
> [junit4:junit4]   2>  at 
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> [junit4:junit4]   2>  at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> [junit4:junit4]   2>  at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> [junit4:junit4]   2>  at 
> java.lang.reflect.Method.invoke(Method.java:601)
> [junit4:junit4]   2>  at 
> com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
> [junit4:junit4]   2>  at 
> com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
> [junit4:junit4]   2>  at 
> com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:700)
> [junit4:junit4]   2>  at 
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> [junit4:junit4]   2>  at 
> com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
> [junit4:junit4]   2>  at 
> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
> [junit4:junit4]   2>  at 
> org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
> [junit4:junit4]   2>  at 
> com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
> [junit4:junit4]   2>  at 
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
> [junit4:junit4]   2>  at 
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
> [junit4:junit4]   2>  at 
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> [junit4:junit4]   2>  at 
> org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
> [junit4:junit4]   2>  at 
> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
> [junit4:junit4]   2>  at 
> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
> [junit4:junit4]   2>  at 
> org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
> [junit4:junit4]   2>  at 
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> [junit4:junit4]   2>  at 
> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
> [junit4:junit4]   2>  at java.lang.Thread.run(Thread.java:722)
> [junit4:junit4]   2>  
> [junit4:junit4]   2> 13389 T10 oasc.SolrException.log SEVERE 
> java.lang.NullPointerException
> [junit4:junit4]   2>  at 
> org.apache.solr.handler.component.HttpShardHandlerFactory.close(HttpShardHandlerFactory.java:170)
> [junit4:junit4]   2>  at 
> org.apache.solr.core.CoreContainer.shutdown(CoreContainer.java:786)
> [junit4:junit4]   2>  at 
> org.apache.solr.util.TestHarness.close(TestHarness.java:449)
> [junit4:junit4]   2>  at 
> org.apache.solr.SolrTestCaseJ4.deleteCore(SolrTestCaseJ4.java:415)
> [junit4:junit4]   2>  at 
> org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:9

[jira] [Commented] (SOLR-4086) Refactor DIH - VariableResolver & Evaluator

2012-12-19 Thread Dominik Siebel (JIRA)

[ 
https://issues.apache.org/jira/browse/SOLR-4086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13536089#comment-13536089
 ] 

Dominik Siebel commented on SOLR-4086:
--

[~jdyer]
I noticed a massive decrease in indexing performance when trying a current 
checkout from branch_4x after this refactoring. Do you have any explanation for 
that? Looking through the code I could not find any changes that would explain 
this.


*Some numbers:*
* 1.3M documents
* 4 DIHs


*before* (with my patch from SOLR-2141):
* DIH_1: documents processed: 325130, time taken: 00:25:44.445
* DIH_2: documents processed: 207347, time taken: 01:16:04.607
* DIH_3: documents processed: 184601, time taken: 01:18:00.797
* DIH_4: documents processed: 618580, time taken: 04:17:38.414


*after*:
* DIH_1: documents processed: 324996, time taken: 01:07:47.186
* DIH_2: documents processed: 207347, time taken: 03:31:21.345
* DIH_3: documents processed: 184521, time taken: 03:13:11.313
* DIH_4: documents processed: 618491, time taken: 06:42:54.384


Any idea?

> Refactor DIH - VariableResolver & Evaluator
> ---
>
> Key: SOLR-4086
> URL: https://issues.apache.org/jira/browse/SOLR-4086
> Project: Solr
>  Issue Type: Improvement
>  Components: contrib - DataImportHandler
>Affects Versions: 4.0
>Reporter: James Dyer
>Assignee: James Dyer
>Priority: Minor
> Fix For: 4.1, 5.0
>
> Attachments: SOLR-4086.patch
>
>
> This simplifies VariableResolver and moves each built-in Evaluator into its 
> own class.  Compiler warnings / missing generics are fixed.  Also, the Locale 
> bug with DateFormatEvaluator is solved.  Instead of using the machine 
> default, the Root Locale is used by default.  An optional 3rd parameter 
> allows users to specify whatever locale they want.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



[jira] [Updated] (SOLR-4218) SolrTestCaseJ4 throws NPE when closing the core (on the afterClass method)

2012-12-19 Thread JIRA

 [ 
https://issues.apache.org/jira/browse/SOLR-4218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tomás Fernández Löbbe updated SOLR-4218:


Attachment: SOLR-4218.patch

> SolrTestCaseJ4 throws NPE when closing the core (on the afterClass method)
> --
>
> Key: SOLR-4218
> URL: https://issues.apache.org/jira/browse/SOLR-4218
> Project: Solr
>  Issue Type: Bug
>Reporter: Tomás Fernández Löbbe
> Fix For: 4.1, 5.0
>
> Attachments: SOLR-4218.patch
>
>
> When running a specific test like:
> ant test -Dtestcase=BasicFunctionalityTest
> at the end of the test there is a NPE.
> {code}
> [junit4:junit4]   2> 13384 T10 oasc.SolrException.log SEVERE 
> java.lang.NullPointerException
> [junit4:junit4]   2>  at 
> org.apache.solr.handler.component.HttpShardHandlerFactory.close(HttpShardHandlerFactory.java:165)
> [junit4:junit4]   2>  at 
> org.apache.solr.core.CoreContainer.shutdown(CoreContainer.java:786)
> [junit4:junit4]   2>  at 
> org.apache.solr.util.TestHarness.close(TestHarness.java:449)
> [junit4:junit4]   2>  at 
> org.apache.solr.SolrTestCaseJ4.deleteCore(SolrTestCaseJ4.java:415)
> [junit4:junit4]   2>  at 
> org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:95)
> [junit4:junit4]   2>  at 
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> [junit4:junit4]   2>  at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> [junit4:junit4]   2>  at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> [junit4:junit4]   2>  at 
> java.lang.reflect.Method.invoke(Method.java:601)
> [junit4:junit4]   2>  at 
> com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
> [junit4:junit4]   2>  at 
> com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
> [junit4:junit4]   2>  at 
> com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:700)
> [junit4:junit4]   2>  at 
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> [junit4:junit4]   2>  at 
> com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
> [junit4:junit4]   2>  at 
> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
> [junit4:junit4]   2>  at 
> org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
> [junit4:junit4]   2>  at 
> com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
> [junit4:junit4]   2>  at 
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
> [junit4:junit4]   2>  at 
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
> [junit4:junit4]   2>  at 
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> [junit4:junit4]   2>  at 
> org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
> [junit4:junit4]   2>  at 
> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
> [junit4:junit4]   2>  at 
> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
> [junit4:junit4]   2>  at 
> org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
> [junit4:junit4]   2>  at 
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> [junit4:junit4]   2>  at 
> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
> [junit4:junit4]   2>  at java.lang.Thread.run(Thread.java:722)
> [junit4:junit4]   2>  
> [junit4:junit4]   2> 13389 T10 oasc.SolrException.log SEVERE 
> java.lang.NullPointerException
> [junit4:junit4]   2>  at 
> org.apache.solr.handler.component.HttpShardHandlerFactory.close(HttpShardHandlerFactory.java:170)
> [junit4:junit4]   2>  at 
> org.apache.solr.core.CoreContainer.shutdown(CoreContainer.java:786)
> [junit4:junit4]   2>  at 
> org.apache.solr.util.TestHarness.close(TestHarness.java:449)
> [junit4:junit4]   2>  at 
> org.apache.solr.SolrTestCaseJ4.deleteCore(SolrTestCaseJ4.java:415)
> [junit4:junit4]   2>  at 
> org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:95)
> [junit4:ju

[jira] [Commented] (SOLR-4218) SolrTestCaseJ4 throws NPE when closing the core (on the afterClass method)

2012-12-19 Thread JIRA

[ 
https://issues.apache.org/jira/browse/SOLR-4218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13536078#comment-13536078
 ] 

Tomás Fernández Löbbe commented on SOLR-4218:
-

bq. I don't see this exception when running from Eclipse.
Forget about that, I had a slightly outdated revision in Eclipse.
Seems to me that this issue was introduced by the changes in SOLR-4204, looks 
like the ShardHandler may not be initialized under some conditions and that's 
why some of it fields are null.

> SolrTestCaseJ4 throws NPE when closing the core (on the afterClass method)
> --
>
> Key: SOLR-4218
> URL: https://issues.apache.org/jira/browse/SOLR-4218
> Project: Solr
>  Issue Type: Bug
>Reporter: Tomás Fernández Löbbe
> Fix For: 4.1, 5.0
>
>
> When running a specific test like:
> ant test -Dtestcase=BasicFunctionalityTest
> at the end of the test there is a NPE.
> {code}
> [junit4:junit4]   2> 13384 T10 oasc.SolrException.log SEVERE 
> java.lang.NullPointerException
> [junit4:junit4]   2>  at 
> org.apache.solr.handler.component.HttpShardHandlerFactory.close(HttpShardHandlerFactory.java:165)
> [junit4:junit4]   2>  at 
> org.apache.solr.core.CoreContainer.shutdown(CoreContainer.java:786)
> [junit4:junit4]   2>  at 
> org.apache.solr.util.TestHarness.close(TestHarness.java:449)
> [junit4:junit4]   2>  at 
> org.apache.solr.SolrTestCaseJ4.deleteCore(SolrTestCaseJ4.java:415)
> [junit4:junit4]   2>  at 
> org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:95)
> [junit4:junit4]   2>  at 
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> [junit4:junit4]   2>  at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> [junit4:junit4]   2>  at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> [junit4:junit4]   2>  at 
> java.lang.reflect.Method.invoke(Method.java:601)
> [junit4:junit4]   2>  at 
> com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
> [junit4:junit4]   2>  at 
> com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
> [junit4:junit4]   2>  at 
> com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:700)
> [junit4:junit4]   2>  at 
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> [junit4:junit4]   2>  at 
> com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
> [junit4:junit4]   2>  at 
> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
> [junit4:junit4]   2>  at 
> org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
> [junit4:junit4]   2>  at 
> com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
> [junit4:junit4]   2>  at 
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
> [junit4:junit4]   2>  at 
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
> [junit4:junit4]   2>  at 
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> [junit4:junit4]   2>  at 
> org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
> [junit4:junit4]   2>  at 
> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
> [junit4:junit4]   2>  at 
> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
> [junit4:junit4]   2>  at 
> org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
> [junit4:junit4]   2>  at 
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> [junit4:junit4]   2>  at 
> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
> [junit4:junit4]   2>  at java.lang.Thread.run(Thread.java:722)
> [junit4:junit4]   2>  
> [junit4:junit4]   2> 13389 T10 oasc.SolrException.log SEVERE 
> java.lang.NullPointerException
> [junit4:junit4]   2>  at 
> org.apache.solr.handler.component.HttpShardHandlerFactory.close(HttpShardHandlerFactory.java:170)
> [junit4:junit4]   2>  at 
> org.apache.solr.core.CoreContainer.shutdown(CoreContainer.java:786)
> [junit4:

[jira] [Created] (LUCENE-4638) If IndexWriter is interrupted on close and is using a channel (mmap/nio), it can throw a ClosedByInterruptException and prevent you from opening a new IndexWriter in the

2012-12-19 Thread Mark Miller (JIRA)
Mark Miller created LUCENE-4638:
---

 Summary: If IndexWriter is interrupted on close and is using a 
channel (mmap/nio), it can throw a ClosedByInterruptException and prevent you 
from opening a new IndexWriter in the same proceses if you are using Native 
locks.
 Key: LUCENE-4638
 URL: https://issues.apache.org/jira/browse/LUCENE-4638
 Project: Lucene - Core
  Issue Type: Bug
Reporter: Mark Miller
Priority: Minor
 Fix For: 4.1, 5.0


The ClosedByInterruptException will prevent the index from being unlocked in 
close. If you try and close again, the call will hang. If you are using native 
locks and try to open a new IndexWriter, it will fail to get the lock. If you 
try IW#forceUnlock, it wont work because the not fully closed IW will still 
have the lock.

ideas:
* On ClosedByInterruptException, IW should continue trying to close what it can 
and unlock the index? Generally I have see the exception trigger in 
commitInternal.
* We should add a non static forceUnlock to IW that lets you remove the lock 
and start a new IW?
* We should make the lock protected so IW sub classes could unlock the index in 
advanced use cases?
* others?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



[jira] [Created] (SOLR-4219) org.apache.solr.common.SolrException: Invalid UUID String

2012-12-19 Thread Szasz Tamas (JIRA)
Szasz Tamas created SOLR-4219:
-

 Summary: org.apache.solr.common.SolrException: Invalid UUID String
 Key: SOLR-4219
 URL: https://issues.apache.org/jira/browse/SOLR-4219
 Project: Solr
  Issue Type: Bug
Affects Versions: 4.0
Reporter: Szasz Tamas


I use the UUIDField to generate unique keys for my documents. 
In general the configuration is working but sometimes I get the following 
exception:

[#|2012-12-19T16:00:43.471+0100|SEVERE|glassfish3.1|org.apache.solr.core.SolrCore|_ThreadID=41;_ThreadName=Thread-1;|org.apache.solr.common.SolrException:
 Invalid UUID String: '.E^M!HCX'KN^URi
D^F .E^M!HCX'KN^URi
D^F'
at org.apache.solr.schema.UUIDField.toInternal(UUIDField.java:78)
at 
org.apache.solr.schema.FieldType.readableToIndexed(FieldType.java:371)
at 
org.apache.solr.schema.FieldType.readableToIndexed(FieldType.java:376)
at 
org.apache.solr.update.AddUpdateCommand.getIndexedId(AddUpdateCommand.java:94)
at 
org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:445)
at 
org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:325)
at 
org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:51)
at 
org.apache.solr.update.processor.AbstractDefaultValueUpdateProcessorFactory$DefaultValueUpdateProcessor.processAdd(AbstractDefaultValueUpdateProcessorFactory.java:94)
at 
org.apache.solr.handler.extraction.ExtractingDocumentLoader.doAdd(ExtractingDocumentLoader.java:121)
at 
org.apache.solr.handler.extraction.ExtractingDocumentLoader.addDoc(ExtractingDocumentLoader.java:126)
at 
org.apache.solr.handler.extraction.ExtractingDocumentLoader.load(ExtractingDocumentLoader.java:228)
at 
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74)
at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129)
at 
org.apache.solr.core.RequestHandlers$LazyRequestHandlerWrapper.handleRequest(RequestHandlers.java:240)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1699)
at 
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:455)
at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:276)
at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:256)
at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:215)
at 
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:279)
at 
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
at 
org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:655)
at 
org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:595)
at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:98)
at 
com.sun.enterprise.web.PESessionLockingStandardPipeline.invoke(PESessionLockingStandardPipeline.java:91)
at 
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:162)
at 
org.apache.catalina.connector.CoyoteAdapter.doService(CoyoteAdapter.java:326)
at 
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:227)
at 
com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:170)
at 
com.sun.grizzly.http.ProcessorTask.invokeAdapter(ProcessorTask.java:822)
at com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:719)
at com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:1013)
at 
com.sun.grizzly.http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:225)
at 
com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultProtocolChain.java:137)
at 
com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:104)
at 
com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:90)
at 
com.sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:79)
at 
com.sun.grizzly.ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:54)
at 
com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:59)
at com.sun.grizzly.ContextTask.run(ContextTask.java:71)
at 
com.sun.grizzly.util.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:532)
at 
com.sun.grizzly.util.AbstractThreadPool$Worker.run(AbstractThreadPool.java:513)
at java.lang.Thread.run(Thread.java:662)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, s

Re: _TestUtil.getTempFile may fail on "Access Denied"

2012-12-19 Thread Shai Erera
>
> I patched _TestUtil like so, and the test passes
>

Sorry, I meant to say that if fails where I expect it to fail, not on
Access is Denied.

Shai


On Wed, Dec 19, 2012 at 5:14 PM, Shai Erera  wrote:

> Hi
>
> When a test fails, the file system directories that it created are not
> deleted.
>
> At least on Windows, when you run the test with the same seed again,
> _TestUtil.getTempDir fails on  "Access Denied". The reason is that it calls
> file.createNewFile() on a directory. I don't know if it's specific to
> Windows, but I tried this test with both J9 and Oracle JVMs:
>
>   @Test
>   public void testAccessDenied() throws Exception {
> _TestUtil.getTempDir("testAccessDenied").mkdirs();
> fail("msg");
>   }
>
> Because of the fail() in the end, the directory is not deleted. If you run
> it with the same seed twice (say, -Dtests.seed=6EDF27F12DB68F8D), then
> you'll get:
>
> java.lang.RuntimeException: java.io.IOException: Access is denied
> at
> __randomizedtesting.SeedInfo.seed([6EDF27F12DB68F8D:668298B50F63FCD2]:0)
> at org.apache.lucene.util._TestUtil.getTempDir(_TestUtil.java:107)
> at
> org.apache.lucene.TestAssertions.testAccessDenied(TestAssertions.java:62)
>
> I patched _TestUtil like so, and the test passes:
>
> Index: lucene/test-framework/src/java/org/apache/lucene/util/_TestUtil.java
> ===
> ---
> lucene/test-framework/src/java/org/apache/lucene/util/_TestUtil.java
> (revision 1423868)
> +++
> lucene/test-framework/src/java/org/apache/lucene/util/_TestUtil.java
> (working copy)
> @@ -750,7 +750,7 @@
>  final Random random = new
> Random(RandomizedContext.current().getRandom().nextLong());
>  do {
>result = genTempFile(random, prefix, newSuffix, directory);
> -} while (!result.createNewFile());
> +} while (result.exists() || !result.createNewFile());
>  return result;
>}
>
> Basically, adding an exists() check before createNewFile(). Even though
> createNewFile() documents that it returns false if the file exists, it does
> not specify the behavior when the file exists and is a directory, and
> obviously fails.
>
> I made sure that calling genTempFile like so in a loop won't ruin
> randomness - since it only uses random.nextInt() once, I think it's safe to
> add this check?
>
> Shai
>


_TestUtil.getTempFile may fail on "Access Denied"

2012-12-19 Thread Shai Erera
Hi

When a test fails, the file system directories that it created are not
deleted.

At least on Windows, when you run the test with the same seed again,
_TestUtil.getTempDir fails on  "Access Denied". The reason is that it calls
file.createNewFile() on a directory. I don't know if it's specific to
Windows, but I tried this test with both J9 and Oracle JVMs:

  @Test
  public void testAccessDenied() throws Exception {
_TestUtil.getTempDir("testAccessDenied").mkdirs();
fail("msg");
  }

Because of the fail() in the end, the directory is not deleted. If you run
it with the same seed twice (say, -Dtests.seed=6EDF27F12DB68F8D), then
you'll get:

java.lang.RuntimeException: java.io.IOException: Access is denied
at
__randomizedtesting.SeedInfo.seed([6EDF27F12DB68F8D:668298B50F63FCD2]:0)
at org.apache.lucene.util._TestUtil.getTempDir(_TestUtil.java:107)
at
org.apache.lucene.TestAssertions.testAccessDenied(TestAssertions.java:62)

I patched _TestUtil like so, and the test passes:

Index: lucene/test-framework/src/java/org/apache/lucene/util/_TestUtil.java
===
---
lucene/test-framework/src/java/org/apache/lucene/util/_TestUtil.java
(revision 1423868)
+++
lucene/test-framework/src/java/org/apache/lucene/util/_TestUtil.java
(working copy)
@@ -750,7 +750,7 @@
 final Random random = new
Random(RandomizedContext.current().getRandom().nextLong());
 do {
   result = genTempFile(random, prefix, newSuffix, directory);
-} while (!result.createNewFile());
+} while (result.exists() || !result.createNewFile());
 return result;
   }

Basically, adding an exists() check before createNewFile(). Even though
createNewFile() documents that it returns false if the file exists, it does
not specify the behavior when the file exists and is a directory, and
obviously fails.

I made sure that calling genTempFile like so in a loop won't ruin
randomness - since it only uses random.nextInt() once, I think it's safe to
add this check?

Shai


[jira] [Commented] (SOLR-3972) Missing admin-extra files result in SEVERE log entries with giant stacktrace

2012-12-19 Thread Shawn Heisey (JIRA)

[ 
https://issues.apache.org/jira/browse/SOLR-3972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13536017#comment-13536017
 ] 

Shawn Heisey commented on SOLR-3972:


It looks like the only non-test place that these filenames are mentioned is in 
a javascript file, and the error comes from 
core/src/java/org/apache/solr/handler/admin/ShowFileRequestHandler.java.  It 
shows up twice - once when pulling from Zookeeper, and once when pulling from 
the filesystem.

I will take a deeper look and attempt a patch when I make it to work and get it 
pulled into eclipse.


> Missing admin-extra files result in SEVERE log entries with giant stacktrace
> 
>
> Key: SOLR-3972
> URL: https://issues.apache.org/jira/browse/SOLR-3972
> Project: Solr
>  Issue Type: Improvement
>  Components: web gui
>Affects Versions: 4.0, 4.1
> Environment: Linux bigindy5 2.6.32-279.9.1.el6.centos.plus.x86_64 #1 
> SMP Wed Sep 26 03:52:55 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux
> java version "1.7.0_07"
> Java(TM) SE Runtime Environment (build 1.7.0_07-b10)
> Java HotSpot(TM) 64-Bit Server VM (build 23.3-b01, mixed mode)
>Reporter: Shawn Heisey
> Fix For: 4.1
>
>
> Missing admin-extra files result in SEVERE log entries with giant stacktrace.
> If a log entry is warranted at all, it should just be a one-line warning.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



[jira] [Commented] (SOLR-3972) Missing admin-extra files result in SEVERE log entries with giant stacktrace

2012-12-19 Thread Shawn Heisey (JIRA)

[ 
https://issues.apache.org/jira/browse/SOLR-3972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13536013#comment-13536013
 ] 

Shawn Heisey commented on SOLR-3972:


I agree with Alexandre.  When I was copying from the example, which included 
copying the included jetty and creating an init script for it, I looked at the 
three extra html files, which ARE included there.

I figured since they were "-extra" they were not necessary.  I did look into 
their actual content and found it useless, so I didn't copy them to my 
installation.  That's when I ran into the stacktrace - the 20 cores I had when 
I first set it up (deriving from my 3.x config) produced *sixty* of these huge 
log entries, even when logging at WARN.  Now I have 16 cores on the server and 
my workaround has eliminated the error.

If a fix for this issue were implemented (using log.warn()) and I deleted the 
empty html files, I would end up with 48 consecutive single log line entries if 
I left logging at WARN.  If the logging were at INFO (which I believe is the 
default), they would probably be scattered throughout the log and not make much 
difference in its size.  That's actually a good argument for making the message 
log at info and not warn.


> Missing admin-extra files result in SEVERE log entries with giant stacktrace
> 
>
> Key: SOLR-3972
> URL: https://issues.apache.org/jira/browse/SOLR-3972
> Project: Solr
>  Issue Type: Improvement
>  Components: web gui
>Affects Versions: 4.0, 4.1
> Environment: Linux bigindy5 2.6.32-279.9.1.el6.centos.plus.x86_64 #1 
> SMP Wed Sep 26 03:52:55 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux
> java version "1.7.0_07"
> Java(TM) SE Runtime Environment (build 1.7.0_07-b10)
> Java HotSpot(TM) 64-Bit Server VM (build 23.3-b01, mixed mode)
>Reporter: Shawn Heisey
> Fix For: 4.1
>
>
> Missing admin-extra files result in SEVERE log entries with giant stacktrace.
> If a log entry is warranted at all, it should just be a one-line warning.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



  1   2   >