[jira] [Commented] (LUCENE-3466) Rename Analyzer.reusableTokenStream() to tokenStream()

[Simon Willnauer (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-3466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13115283#comment-13115283
 ] 

Simon Willnauer commented on LUCENE-3466:
-

awesome! This is the end of reusableTokenStream! If I remember correctly this 
work has started in late 2009 and now it seems it only one commit away! Thanks 
chris!

> Rename Analyzer.reusableTokenStream() to tokenStream()
> --
>
> Key: LUCENE-3466
> URL: https://issues.apache.org/jira/browse/LUCENE-3466
> Project: Lucene - Java
>  Issue Type: Sub-task
>  Components: modules/analysis
>Reporter: Chris Male
>Assignee: Chris Male
> Attachments: LUCENE-3466.patch
>
>
> All Analysis consumers now use reusableTokenStream().  To finally make reuse 
> mandatory, lets rename resuableTokenStream() to tokenStream() (removing the 
> old tokenStream() method).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira



-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

trunk test failure (1317107041)

[Charlie Cron]

A test failure occurred running the tests.

revert! revert! revert!

You can see the entire build log at 
http://sierranevada.servebeer.com/1317107041.log

Thanks,
Charlie Cron


-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-2372) Upgrade Solr to Tika 0.10

[Updated]

 [ 
https://issues.apache.org/jira/browse/SOLR-2372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jan Høydahl updated SOLR-2372:
--

Summary: Upgrade Solr to Tika 0.10  (was: Upgrade Solr to Tika 0.9)

Tika 0.10 is a few days away, so we'll skip 0.9 and jump directly to 0.10
Serious bugs fixed for both PDF, HTML and XLSX.
http://search-lucene.com/m/kFMGc2BwzA4

> Upgrade Solr to Tika 0.10
> -
>
> Key: SOLR-2372
> URL: https://issues.apache.org/jira/browse/SOLR-2372
> Project: Solr
>  Issue Type: Improvement
>  Components: contrib - Solr Cell (Tika extraction)
>Reporter: Grant Ingersoll
>Assignee: Jan Høydahl
> Fix For: 3.5, 4.0
>
>
> as the title says

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira



-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-1895) ManifoldCF SearchComponent plugin for enforcing ManifoldCF security at search time

[Karl Wright (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-1895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13115372#comment-13115372
 ] 

Karl Wright commented on SOLR-1895:
---

Is there any possibility that the SOLR-1895-queries patch will be committed as 
nothing more complicated than a Solr mcf contrib module?  That's how I've set 
it up, and based on the discussion for this ticket it seems there will always 
be an MCF-specific component in any case.  If agreement is eventually reached 
and there is a true Solr security infrastructure, it should be a simple matter 
to redo the MCF pieces to use it later.


> ManifoldCF SearchComponent plugin for enforcing ManifoldCF security at search 
> time
> --
>
> Key: SOLR-1895
> URL: https://issues.apache.org/jira/browse/SOLR-1895
> Project: Solr
>  Issue Type: New Feature
>  Components: SearchComponents - other
>Reporter: Karl Wright
>  Labels: document, security, solr
> Fix For: 3.5, 4.0
>
> Attachments: LCFSecurityFilter.java, LCFSecurityFilter.java, 
> LCFSecurityFilter.java, LCFSecurityFilter.java, SOLR-1895-queries.patch, 
> SOLR-1895-queries.patch, SOLR-1895-queries.patch, 
> SOLR-1895-service-plugin.patch, SOLR-1895-service-plugin.patch, 
> SOLR-1895.patch, SOLR-1895.patch, SOLR-1895.patch, SOLR-1895.patch, 
> SOLR-1895.patch, SOLR-1895.patch
>
>
> I've written an LCF SearchComponent which filters returned results based on 
> access tokens provided by LCF's authority service.  The component requires 
> you to configure the appropriate authority service URL base, e.g.:
>   
>   
>  name="AuthorityServiceBaseURL">http://localhost:8080/lcf-authority-service
>   
> Also required are the following schema.xml additions:
>
> stored="false" multiValued="true"/>
> stored="false" multiValued="true"/>
> stored="false" multiValued="true"/>
> multiValued="true"/>
> Finally, to tie it into the standard request handler, it seems to need to run 
> last:
>   
> 
>   lcfSecurity
> 
> ...
> I have not set a package for this code.  Nor have I been able to get it 
> reviewed by someone as conversant with Solr as I would prefer.  It is my 
> hope, however, that this module will become part of the standard Solr 1.5 
> suite of search components, since that would tie it in with LCF nicely.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira



-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-1895) ManifoldCF SearchComponent plugin for enforcing ManifoldCF security at search time

[Karl Wright (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-1895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Karl Wright updated SOLR-1895:
--

Attachment: SOLR-1895-queries.patch

Updated to fix an outdated class reference

> ManifoldCF SearchComponent plugin for enforcing ManifoldCF security at search 
> time
> --
>
> Key: SOLR-1895
> URL: https://issues.apache.org/jira/browse/SOLR-1895
> Project: Solr
>  Issue Type: New Feature
>  Components: SearchComponents - other
>Reporter: Karl Wright
>  Labels: document, security, solr
> Fix For: 3.5, 4.0
>
> Attachments: LCFSecurityFilter.java, LCFSecurityFilter.java, 
> LCFSecurityFilter.java, LCFSecurityFilter.java, SOLR-1895-queries.patch, 
> SOLR-1895-queries.patch, SOLR-1895-queries.patch, SOLR-1895-queries.patch, 
> SOLR-1895-service-plugin.patch, SOLR-1895-service-plugin.patch, 
> SOLR-1895.patch, SOLR-1895.patch, SOLR-1895.patch, SOLR-1895.patch, 
> SOLR-1895.patch, SOLR-1895.patch
>
>
> I've written an LCF SearchComponent which filters returned results based on 
> access tokens provided by LCF's authority service.  The component requires 
> you to configure the appropriate authority service URL base, e.g.:
>   
>   
>  name="AuthorityServiceBaseURL">http://localhost:8080/lcf-authority-service
>   
> Also required are the following schema.xml additions:
>
> stored="false" multiValued="true"/>
> stored="false" multiValued="true"/>
> stored="false" multiValued="true"/>
> multiValued="true"/>
> Finally, to tie it into the standard request handler, it seems to need to run 
> last:
>   
> 
>   lcfSecurity
> 
> ...
> I have not set a package for this code.  Nor have I been able to get it 
> reviewed by someone as conversant with Solr as I would prefer.  It is my 
> hope, however, that this module will become part of the standard Solr 1.5 
> suite of search components, since that would tie it in with LCF nicely.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira



-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-2202) Money FieldType

[Commented]

[ 
https://issues.apache.org/jira/browse/SOLR-2202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13115375#comment-13115375
 ] 

Jan Høydahl commented on SOLR-2202:
---

Greg, do you have an updatet patch for this one? We'll help you get it the last 
mile for inclusion.

PS: When uploading patches, we prefer that you name it SOLR-2202.patch *every* 
time. JIRA will automatically "grey out" the older versions for you.

> Money FieldType
> ---
>
> Key: SOLR-2202
> URL: https://issues.apache.org/jira/browse/SOLR-2202
> Project: Solr
>  Issue Type: New Feature
>  Components: Schema and Analysis
>Affects Versions: 1.5
>Reporter: Greg Fodor
> Attachments: SOLR-2022-solr-3.patch, SOLR-2202-lucene-1.patch, 
> SOLR-2202-solr-1.patch, SOLR-2202-solr-2.patch, SOLR-2202-solr-4.patch, 
> SOLR-2202-solr-5.patch, SOLR-2202-solr-6.patch, SOLR-2202-solr-7.patch, 
> SOLR-2202-solr-8.patch, SOLR-2202-solr-9.patch
>
>
> Attached please find patches to add support for monetary values to 
> Solr/Lucene with query-time currency conversion. The following features are 
> supported:
> - Point queries (ex: "price:4.00USD")
> - Range quries (ex: "price:[$5.00 TO $10.00]")
> - Sorting.
> - Currency parsing by either currency code or symbol.
> - Symmetric & Asymmetric exchange rates. (Asymmetric exchange rates are 
> useful if there are fees associated with exchanging the currency.)
> At indexing time, money fields can be indexed in a native currency. For 
> example, if a product on an e-commerce site is listed in Euros, indexing the 
> price field as "10.00EUR" will index it appropriately. By altering the 
> currency.xml file, the sorting and querying against Solr can take into 
> account fluctuations in currency exchange rates without having to re-index 
> the documents.
> The new "money" field type is a polyfield which indexes two fields, one which 
> contains the amount of the value and another which contains the currency code 
> or symbol. The currency metadata (names, symbols, codes, and exchange rates) 
> are expected to be in an xml file which is pointed to by the field type 
> declaration in the schema.xml.
> The current patch is factored such that Money utility functions and 
> configuration metadata lie in Lucene (see MoneyUtil and CurrencyConfig), 
> while the MoneyType and MoneyValueSource lie in Solr. This was meant to 
> mirror the work being done on the spacial field types.
> This patch has not yet been deployed to production but will be getting used 
> to power the international search capabilities of the search engine at Etsy.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira



-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Created] (LUCENE-3467) Cut over numeric docvalues to fixed straight bytes

[Simon Willnauer (Created) (JIRA)]

Cut over numeric docvalues to fixed straight bytes
--

 Key: LUCENE-3467
 URL: https://issues.apache.org/jira/browse/LUCENE-3467
 Project: Lucene - Java
  Issue Type: Improvement
  Components: core/index
Affects Versions: 4.0
Reporter: Simon Willnauer
 Fix For: 4.0


Currently numeric docvalues types are encoded and stored individually which 
creates massive duplication of writing / indexing code. Yet, almost all of them 
(except packed ints) are essentially a fixed straight bytes variant. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira



-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (LUCENE-3467) Cut over numeric docvalues to fixed straight bytes

[Simon Willnauer (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LUCENE-3467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Simon Willnauer updated LUCENE-3467:


Attachment: LUCENE-3467.patch

here is a patch that factors out multiple base classes shared across variant 
specific docvalues writers / readers. Numerics subclasses 
FixedStraightBytesImpl.Writer and reuses and only encodes incoming values as a 
fixed sized bytes. Other byte variants share common base classes and override 
necessary parts for their specific variant.

This removes a large amount of code from docvalues which I should have done 
earlier. However, this is a first step towards fixing the merge problem since 
we can now almost exclusively operate on bytes rather than ints, floats, 
doubles etc. on the implementation level.

> Cut over numeric docvalues to fixed straight bytes
> --
>
> Key: LUCENE-3467
> URL: https://issues.apache.org/jira/browse/LUCENE-3467
> Project: Lucene - Java
>  Issue Type: Improvement
>  Components: core/index
>Affects Versions: 4.0
>Reporter: Simon Willnauer
> Fix For: 4.0
>
> Attachments: LUCENE-3467.patch
>
>
> Currently numeric docvalues types are encoded and stored individually which 
> creates massive duplication of writing / indexing code. Yet, almost all of 
> them (except packed ints) are essentially a fixed straight bytes variant. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira



-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Created] (LUCENE-3468) FirstPassGroupingCollector should use pollLast()

[Martijn van Groningen (Created) (JIRA)]

FirstPassGroupingCollector should use pollLast()


 Key: LUCENE-3468
 URL: https://issues.apache.org/jira/browse/LUCENE-3468
 Project: Lucene - Java
  Issue Type: Improvement
  Components: modules/grouping
Affects Versions: 4.0
Reporter: Martijn van Groningen
Assignee: Martijn van Groningen
 Fix For: 4.0


Currently FirstPassGroupingCollector uses last and remove method (TreeSet) for 
replacing a more relevant grouping during grouping.
This can be replaced by pollLast since Lucene trunk is now Java 6. 
TermFirstPassGroupingCollectorJava6 in Solr can be removed as well.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira



-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (LUCENE-1536) if a filter can support random access API, we should use it

[Chris Male (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LUCENE-1536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Male updated LUCENE-1536:
---

Attachment: LUCENE-1536.patch

New patch which adds greater control over the random-access to DocIdSet.

Also fixes most of the nocommits.

> if a filter can support random access API, we should use it
> ---
>
> Key: LUCENE-1536
> URL: https://issues.apache.org/jira/browse/LUCENE-1536
> Project: Lucene - Java
>  Issue Type: Improvement
>  Components: core/search
>Affects Versions: 2.4
>Reporter: Michael McCandless
>Assignee: Michael McCandless
>Priority: Minor
>  Labels: gsoc2011, lucene-gsoc-11, mentor
> Fix For: 4.0
>
> Attachments: CachedFilterIndexReader.java, LUCENE-1536.patch, 
> LUCENE-1536.patch, LUCENE-1536.patch, LUCENE-1536.patch, LUCENE-1536.patch, 
> LUCENE-1536.patch, LUCENE-1536.patch, LUCENE-1536.patch, LUCENE-1536.patch, 
> LUCENE-1536.patch
>
>
> I ran some performance tests, comparing applying a filter via
> random-access API instead of current trunk's iterator API.
> This was inspired by LUCENE-1476, where we realized deletions should
> really be implemented just like a filter, but then in testing found
> that switching deletions to iterator was a very sizable performance
> hit.
> Some notes on the test:
>   * Index is first 2M docs of Wikipedia.  Test machine is Mac OS X
> 10.5.6, quad core Intel CPU, 6 GB RAM, java 1.6.0_07-b06-153.
>   * I test across multiple queries.  1-X means an OR query, eg 1-4
> means 1 OR 2 OR 3 OR 4, whereas +1-4 is an AND query, ie 1 AND 2
> AND 3 AND 4.  "u s" means "united states" (phrase search).
>   * I test with multiple filter densities (0, 1, 2, 5, 10, 25, 75, 90,
> 95, 98, 99, 99.9 (filter is non-null but all bits are set),
> 100 (filter=null, control)).
>   * Method high means I use random-access filter API in
> IndexSearcher's main loop.  Method low means I use random-access
> filter API down in SegmentTermDocs (just like deleted docs
> today).
>   * Baseline (QPS) is current trunk, where filter is applied as iterator up
> "high" (ie in IndexSearcher's search loop).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira



-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Created] (SOLR-2797) NoClassDefFoundError on field.main=true

[bronco (Created) (JIRA)]

NoClassDefFoundError on field.main=true
---

 Key: SOLR-2797
 URL: https://issues.apache.org/jira/browse/SOLR-2797
 Project: Solr
  Issue Type: Bug
  Components: SearchComponents - other
Affects Versions: 3.4
 Environment: ubuntu 10.04
Reporter: bronco
Priority: Trivial


If i activate the search component collpase and use it with the parameter 
group.main=true i will receive always this error.

HTTP Status 500 - org/apache/commons/lang/ArrayUtils 
java.lang.NoClassDefFoundError: org/apache/commons/lang/ArrayUtils at 
org.apache.solr.search.Grouping$Command.createSimpleResponse(Grouping.java:573) 
at org.apache.solr.search.Grouping$CommandField.finish(Grouping.java:675) at 
org.apache.solr.search.Grouping.execute(Grouping.java:339) at 
org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:240)
 at 
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:194)
 at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129)
 at org.apache.solr.core.SolrCore.execute(SolrCore.java:1368) at 
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:356) 
at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:252)
 at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
 at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
 at 
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
 at 
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
 at 
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) 
at 
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) 
at 
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
 at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298) 
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:859) 
at 
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588)
 at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489) at 
java.lang.Thread.run(Thread.java:662) Caused by: 
java.lang.ClassNotFoundException: org.apache.commons.lang.ArrayUtils at 
org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1484)
 at 
org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1329)
 ... 21 more 

I am not really a java developer but this error drives me crazy. All my small 
programms are running normal with my java installtion. In the meanwhile i 
changed the jre and try to set the CLASSPATH and JAVA_HOME tom get it working 
but without any success. What is nessesary to get solr 3.4 running? I see there 
is obiviously a class missing or not found but how is this possible if the 
other solr things are running really well. So any help is apreciated.



--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira



-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-tests-only-trunk - Build # 10670 - Failure

[Apache Jenkins Server]

Build: https://builds.apache.org/job/Lucene-Solr-tests-only-trunk/10670/

1 tests failed.
REGRESSION:  org.apache.lucene.util.fst.TestFSTs.testRealTerms

Error Message:
Java heap space

Stack Trace:
java.lang.OutOfMemoryError: Java heap space
at org.apache.lucene.util.ArrayUtil.grow(ArrayUtil.java:338)
at org.apache.lucene.util.fst.FST.addNode(FST.java:480)
at org.apache.lucene.util.fst.NodeHash.add(NodeHash.java:122)
at org.apache.lucene.util.fst.Builder.compileNode(Builder.java:180)
at org.apache.lucene.util.fst.Builder.freezeTail(Builder.java:271)
at org.apache.lucene.util.fst.Builder.add(Builder.java:414)
at org.apache.lucene.util.fst.Builder.add(Builder.java:302)
at 
org.apache.lucene.index.codecs.memory.MemoryCodec$TermsWriter.finishTerm(MemoryCodec.java:218)
at 
org.apache.lucene.index.FreqProxTermsWriterPerField.flush(FreqProxTermsWriterPerField.java:409)
at 
org.apache.lucene.index.FreqProxTermsWriter.flush(FreqProxTermsWriter.java:92)
at org.apache.lucene.index.TermsHash.flush(TermsHash.java:117)
at org.apache.lucene.index.DocInverter.flush(DocInverter.java:80)
at 
org.apache.lucene.index.DocFieldProcessor.flush(DocFieldProcessor.java:78)
at 
org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:472)
at 
org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:420)
at 
org.apache.lucene.index.DocumentsWriter.flushAllThreads(DocumentsWriter.java:568)
at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:366)
at org.apache.lucene.index.IndexReader.open(IndexReader.java:317)
at org.apache.lucene.util.fst.TestFSTs.testRealTerms(TestFSTs.java:1034)
at 
org.apache.lucene.util.LuceneTestCase$2$1.evaluate(LuceneTestCase.java:611)




Build Log (for compile errors):
[...truncated 1264 lines...]



-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-2792) Allow case insensitive Hunspell stemming

[Robert Muir (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-2792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13115423#comment-13115423
 ] 

Robert Muir commented on SOLR-2792:
---

I dont think ignoreCaseLocale is a good idea: it doesn't make sense, for the 
same reason that String.equalsIgnoreCase does not take locale.

String.toLowerCase is only for creating a lowercase presentation to the user, 
not case-insensitive matching.



> Allow case insensitive Hunspell stemming
> 
>
> Key: SOLR-2792
> URL: https://issues.apache.org/jira/browse/SOLR-2792
> Project: Solr
>  Issue Type: Improvement
>  Components: Schema and Analysis
>Affects Versions: 3.5, 4.0
>Reporter: Jan Høydahl
>Assignee: Jan Høydahl
> Attachments: SOLR-2792.patch
>
>
> Same as http://code.google.com/p/lucene-hunspell/issues/detail?id=3
> Hunspell dictionaries are by nature case sensitive. The Hunspell stemmer thus 
> needs an option to allow case insensitive matching of the dictionaries.
> Imagine a query for "microsofts". It will never be stemmed to the dictionary 
> word "Microsoft" because of the case difference. This problem cannot be fixed 
> by putting LowercaseFilter before Hunspell.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira



-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS-MAVEN] Lucene-Solr-Maven-3.x #254: POMs out of sync

[Apache Jenkins Server]

Build: https://builds.apache.org/job/Lucene-Solr-Maven-3.x/254/

No tests ran.

Build Log (for compile errors):
[...truncated 11801 lines...]



-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (LUCENE-3468) FirstPassGroupingCollector should use pollLast()

[Martijn van Groningen (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LUCENE-3468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Martijn van Groningen updated LUCENE-3468:
--

Attachment: LUCENE-3468.patch

> FirstPassGroupingCollector should use pollLast()
> 
>
> Key: LUCENE-3468
> URL: https://issues.apache.org/jira/browse/LUCENE-3468
> Project: Lucene - Java
>  Issue Type: Improvement
>  Components: modules/grouping
>Affects Versions: 4.0
>Reporter: Martijn van Groningen
>Assignee: Martijn van Groningen
> Fix For: 4.0
>
> Attachments: LUCENE-3468.patch
>
>
> Currently FirstPassGroupingCollector uses last and remove method (TreeSet) 
> for replacing a more relevant grouping during grouping.
> This can be replaced by pollLast since Lucene trunk is now Java 6. 
> TermFirstPassGroupingCollectorJava6 in Solr can be removed as well.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira



-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-1895) ManifoldCF SearchComponent plugin for enforcing ManifoldCF security at search time

[Ryan McKinley (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-1895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13115429#comment-13115429
 ] 

Ryan McKinley commented on SOLR-1895:
-

bq.  will be committed as nothing more complicated than a Solr mcf contrib 
module?

If it is MCF specific, I doubt there will be consensus to commit it -- if it 
really is MCF specific, then maintaining it in MCF makes the most sense.

however, I don't think this is (or should be) MCF specific.  The basic approach 
you take is general, and I would love to see some building blocks like this 
exist in solr.

> ManifoldCF SearchComponent plugin for enforcing ManifoldCF security at search 
> time
> --
>
> Key: SOLR-1895
> URL: https://issues.apache.org/jira/browse/SOLR-1895
> Project: Solr
>  Issue Type: New Feature
>  Components: SearchComponents - other
>Reporter: Karl Wright
>  Labels: document, security, solr
> Fix For: 3.5, 4.0
>
> Attachments: LCFSecurityFilter.java, LCFSecurityFilter.java, 
> LCFSecurityFilter.java, LCFSecurityFilter.java, SOLR-1895-queries.patch, 
> SOLR-1895-queries.patch, SOLR-1895-queries.patch, SOLR-1895-queries.patch, 
> SOLR-1895-service-plugin.patch, SOLR-1895-service-plugin.patch, 
> SOLR-1895.patch, SOLR-1895.patch, SOLR-1895.patch, SOLR-1895.patch, 
> SOLR-1895.patch, SOLR-1895.patch
>
>
> I've written an LCF SearchComponent which filters returned results based on 
> access tokens provided by LCF's authority service.  The component requires 
> you to configure the appropriate authority service URL base, e.g.:
>   
>   
>  name="AuthorityServiceBaseURL">http://localhost:8080/lcf-authority-service
>   
> Also required are the following schema.xml additions:
>
> stored="false" multiValued="true"/>
> stored="false" multiValued="true"/>
> stored="false" multiValued="true"/>
> multiValued="true"/>
> Finally, to tie it into the standard request handler, it seems to need to run 
> last:
>   
> 
>   lcfSecurity
> 
> ...
> I have not set a package for this code.  Nor have I been able to get it 
> reviewed by someone as conversant with Solr as I would prefer.  It is my 
> hope, however, that this module will become part of the standard Solr 1.5 
> suite of search components, since that would tie it in with LCF nicely.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira



-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-1895) ManifoldCF SearchComponent plugin for enforcing ManifoldCF security at search time

[Karl Wright (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-1895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13115436#comment-13115436
 ] 

Karl Wright commented on SOLR-1895:
---

bq. If it is MCF specific, I doubt there will be consensus to commit it if it 
really is MCF specific, then maintaining it in MCF makes the most sense.

Well, part of it is going to be MCF-specific - namely the part that talks to 
the MCF authority service and interprets the response.  Erik made it clear that 
that part has to be in contrib, I think. He didn't sound like he had a problem 
with it being in Solr as long as it was in contrib though.

bq. however, I don't think this is (or should be) MCF specific. The basic 
approach you take is general, and I would love to see some building blocks like 
this exist in solr.

I would too, but I'm getting the impression that insufficient consensus exists 
to go forward with the general infrastructure. So I'm looking to see if 
consensus exists for a stop-gap solution, which is a contrib module that is MCF 
specific, with no common infrastructure (yet) until consensus can be achieved.  
When it is I can refactor the contrib module to use it.  If this looks unlikely 
as well, I'll plan to build the infrastructure in the ManifoldCF world to 
release Solr contrib modules that support ManifoldCF's security model.



> ManifoldCF SearchComponent plugin for enforcing ManifoldCF security at search 
> time
> --
>
> Key: SOLR-1895
> URL: https://issues.apache.org/jira/browse/SOLR-1895
> Project: Solr
>  Issue Type: New Feature
>  Components: SearchComponents - other
>Reporter: Karl Wright
>  Labels: document, security, solr
> Fix For: 3.5, 4.0
>
> Attachments: LCFSecurityFilter.java, LCFSecurityFilter.java, 
> LCFSecurityFilter.java, LCFSecurityFilter.java, SOLR-1895-queries.patch, 
> SOLR-1895-queries.patch, SOLR-1895-queries.patch, SOLR-1895-queries.patch, 
> SOLR-1895-service-plugin.patch, SOLR-1895-service-plugin.patch, 
> SOLR-1895.patch, SOLR-1895.patch, SOLR-1895.patch, SOLR-1895.patch, 
> SOLR-1895.patch, SOLR-1895.patch
>
>
> I've written an LCF SearchComponent which filters returned results based on 
> access tokens provided by LCF's authority service.  The component requires 
> you to configure the appropriate authority service URL base, e.g.:
>   
>   
>  name="AuthorityServiceBaseURL">http://localhost:8080/lcf-authority-service
>   
> Also required are the following schema.xml additions:
>
> stored="false" multiValued="true"/>
> stored="false" multiValued="true"/>
> stored="false" multiValued="true"/>
> multiValued="true"/>
> Finally, to tie it into the standard request handler, it seems to need to run 
> last:
>   
> 
>   lcfSecurity
> 
> ...
> I have not set a package for this code.  Nor have I been able to get it 
> reviewed by someone as conversant with Solr as I would prefer.  It is my 
> hope, however, that this module will become part of the standard Solr 1.5 
> suite of search components, since that would tie it in with LCF nicely.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira



-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-3466) Rename Analyzer.reusableTokenStream() to tokenStream()

[Robert Muir (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-3466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13115443#comment-13115443
 ] 

Robert Muir commented on LUCENE-3466:
-

+1, though i have a few additional suggestions:
* TokenStream.reset() is described by the docs as optional, i think we should 
adjust the docs here.
* We might be able to re-word some of the stuff in Analyzer javadocs about 
inconsistencies (Since these are gone!)



> Rename Analyzer.reusableTokenStream() to tokenStream()
> --
>
> Key: LUCENE-3466
> URL: https://issues.apache.org/jira/browse/LUCENE-3466
> Project: Lucene - Java
>  Issue Type: Sub-task
>  Components: modules/analysis
>Reporter: Chris Male
>Assignee: Chris Male
> Attachments: LUCENE-3466.patch
>
>
> All Analysis consumers now use reusableTokenStream().  To finally make reuse 
> mandatory, lets rename resuableTokenStream() to tokenStream() (removing the 
> old tokenStream() method).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira



-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-1895) ManifoldCF SearchComponent plugin for enforcing ManifoldCF security at search time

[Ryan McKinley (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-1895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13115445#comment-13115445
 ] 

Ryan McKinley commented on SOLR-1895:
-

What about starting a solr-security project on apache-extras?

I see this patch as a starting place for a security infrastructure.  Given the 
reluctance, maybe it makes sense to let it bake elsewhere and revisit after 
more stuff exists.  This may be a better home then MCF since it would keep 
things general enough that more people could use it.  I would even suggest that 
the core of this should only depend on lucene -- not solr.  



> ManifoldCF SearchComponent plugin for enforcing ManifoldCF security at search 
> time
> --
>
> Key: SOLR-1895
> URL: https://issues.apache.org/jira/browse/SOLR-1895
> Project: Solr
>  Issue Type: New Feature
>  Components: SearchComponents - other
>Reporter: Karl Wright
>  Labels: document, security, solr
> Fix For: 3.5, 4.0
>
> Attachments: LCFSecurityFilter.java, LCFSecurityFilter.java, 
> LCFSecurityFilter.java, LCFSecurityFilter.java, SOLR-1895-queries.patch, 
> SOLR-1895-queries.patch, SOLR-1895-queries.patch, SOLR-1895-queries.patch, 
> SOLR-1895-service-plugin.patch, SOLR-1895-service-plugin.patch, 
> SOLR-1895.patch, SOLR-1895.patch, SOLR-1895.patch, SOLR-1895.patch, 
> SOLR-1895.patch, SOLR-1895.patch
>
>
> I've written an LCF SearchComponent which filters returned results based on 
> access tokens provided by LCF's authority service.  The component requires 
> you to configure the appropriate authority service URL base, e.g.:
>   
>   
>  name="AuthorityServiceBaseURL">http://localhost:8080/lcf-authority-service
>   
> Also required are the following schema.xml additions:
>
> stored="false" multiValued="true"/>
> stored="false" multiValued="true"/>
> stored="false" multiValued="true"/>
> multiValued="true"/>
> Finally, to tie it into the standard request handler, it seems to need to run 
> last:
>   
> 
>   lcfSecurity
> 
> ...
> I have not set a package for this code.  Nor have I been able to get it 
> reviewed by someone as conversant with Solr as I would prefer.  It is my 
> hope, however, that this module will become part of the standard Solr 1.5 
> suite of search components, since that would tie it in with LCF nicely.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira



-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-3468) FirstPassGroupingCollector should use pollLast()

[Michael McCandless (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-3468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13115448#comment-13115448
 ] 

Michael McCandless commented on LUCENE-3468:


+1


> FirstPassGroupingCollector should use pollLast()
> 
>
> Key: LUCENE-3468
> URL: https://issues.apache.org/jira/browse/LUCENE-3468
> Project: Lucene - Java
>  Issue Type: Improvement
>  Components: modules/grouping
>Affects Versions: 4.0
>Reporter: Martijn van Groningen
>Assignee: Martijn van Groningen
> Fix For: 4.0
>
> Attachments: LUCENE-3468.patch
>
>
> Currently FirstPassGroupingCollector uses last and remove method (TreeSet) 
> for replacing a more relevant grouping during grouping.
> This can be replaced by pollLast since Lucene trunk is now Java 6. 
> TermFirstPassGroupingCollectorJava6 in Solr can be removed as well.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira



-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: [JENKINS-MAVEN] Lucene-Solr-Maven-3.x #254: POMs out of sync

[Michael McCandless]

1.5-only javadoc problem (again!).  I committed a fix.

Mike McCandless

http://blog.mikemccandless.com

On Tue, Sep 27, 2011 at 8:03 AM, Apache Jenkins Server
 wrote:
> Build: https://builds.apache.org/job/Lucene-Solr-Maven-3.x/254/
>
> No tests ran.
>
> Build Log (for compile errors):
> [...truncated 11801 lines...]
>
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>
>

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (LUCENE-3466) Rename Analyzer.reusableTokenStream() to tokenStream()

[Chris Male (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LUCENE-3466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Male updated LUCENE-3466:
---

Attachment: LUCENE-3466.patch

New patch which improves documentation on TokenStream.reset() and Analyzer.

> Rename Analyzer.reusableTokenStream() to tokenStream()
> --
>
> Key: LUCENE-3466
> URL: https://issues.apache.org/jira/browse/LUCENE-3466
> Project: Lucene - Java
>  Issue Type: Sub-task
>  Components: modules/analysis
>Reporter: Chris Male
>Assignee: Chris Male
> Attachments: LUCENE-3466.patch, LUCENE-3466.patch
>
>
> All Analysis consumers now use reusableTokenStream().  To finally make reuse 
> mandatory, lets rename resuableTokenStream() to tokenStream() (removing the 
> old tokenStream() method).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira



-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-1895) ManifoldCF SearchComponent plugin for enforcing ManifoldCF security at search time

[Karl Wright (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-1895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13115484#comment-13115484
 ] 

Karl Wright commented on SOLR-1895:
---

bq. What about starting a solr-security project on apache-extras?

I'll have to think about this.  If it was in the same svn I'd feel better about 
doing it that way.  Nevertheless it would be straightforward to spin off 
solr-security from ManifoldCF at some point, as long as we do a good job of 
keeping stuff in contrib, seems to me.  I could imagine an eventual lucene/solr 
subproject, which would be ideal - but until then, it's going to be in the 
wilderness somewhere.


> ManifoldCF SearchComponent plugin for enforcing ManifoldCF security at search 
> time
> --
>
> Key: SOLR-1895
> URL: https://issues.apache.org/jira/browse/SOLR-1895
> Project: Solr
>  Issue Type: New Feature
>  Components: SearchComponents - other
>Reporter: Karl Wright
>  Labels: document, security, solr
> Fix For: 3.5, 4.0
>
> Attachments: LCFSecurityFilter.java, LCFSecurityFilter.java, 
> LCFSecurityFilter.java, LCFSecurityFilter.java, SOLR-1895-queries.patch, 
> SOLR-1895-queries.patch, SOLR-1895-queries.patch, SOLR-1895-queries.patch, 
> SOLR-1895-service-plugin.patch, SOLR-1895-service-plugin.patch, 
> SOLR-1895.patch, SOLR-1895.patch, SOLR-1895.patch, SOLR-1895.patch, 
> SOLR-1895.patch, SOLR-1895.patch
>
>
> I've written an LCF SearchComponent which filters returned results based on 
> access tokens provided by LCF's authority service.  The component requires 
> you to configure the appropriate authority service URL base, e.g.:
>   
>   
>  name="AuthorityServiceBaseURL">http://localhost:8080/lcf-authority-service
>   
> Also required are the following schema.xml additions:
>
> stored="false" multiValued="true"/>
> stored="false" multiValued="true"/>
> stored="false" multiValued="true"/>
> multiValued="true"/>
> Finally, to tie it into the standard request handler, it seems to need to run 
> last:
>   
> 
>   lcfSecurity
> 
> ...
> I have not set a package for this code.  Nor have I been able to get it 
> reviewed by someone as conversant with Solr as I would prefer.  It is my 
> hope, however, that this module will become part of the standard Solr 1.5 
> suite of search components, since that would tie it in with LCF nicely.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira



-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-3466) Rename Analyzer.reusableTokenStream() to tokenStream()

[Robert Muir (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-3466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13115503#comment-13115503
 ] 

Robert Muir commented on LUCENE-3466:
-

+1 to commit

> Rename Analyzer.reusableTokenStream() to tokenStream()
> --
>
> Key: LUCENE-3466
> URL: https://issues.apache.org/jira/browse/LUCENE-3466
> Project: Lucene - Java
>  Issue Type: Sub-task
>  Components: modules/analysis
>Reporter: Chris Male
>Assignee: Chris Male
> Attachments: LUCENE-3466.patch, LUCENE-3466.patch
>
>
> All Analysis consumers now use reusableTokenStream().  To finally make reuse 
> mandatory, lets rename resuableTokenStream() to tokenStream() (removing the 
> old tokenStream() method).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira



-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-1895) ManifoldCF SearchComponent plugin for enforcing ManifoldCF security at search time

[Commented]

[ 
https://issues.apache.org/jira/browse/SOLR-1895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13115524#comment-13115524
 ] 

Jan Høydahl commented on SOLR-1895:
---

To me it sounds like a good match for contrib, ensuring a clean plugin 
experience (apache-solr-mcf-security-4.0-SNAPSHOT.jar) that ships with Solr and 
is kept up-to-date. The size will be really small as well, no external jars. 
Later when we're ready for baby steps towards a generic security framework, we 
can refactor parts of this contrib into core with a single patch.

One thing, will the contrib do more than security for MCF? If not, perhaps it 
should be renamed "mcf-security"?

Anyone against mcf-security as a contrib?

> ManifoldCF SearchComponent plugin for enforcing ManifoldCF security at search 
> time
> --
>
> Key: SOLR-1895
> URL: https://issues.apache.org/jira/browse/SOLR-1895
> Project: Solr
>  Issue Type: New Feature
>  Components: SearchComponents - other
>Reporter: Karl Wright
>  Labels: document, security, solr
> Fix For: 3.5, 4.0
>
> Attachments: LCFSecurityFilter.java, LCFSecurityFilter.java, 
> LCFSecurityFilter.java, LCFSecurityFilter.java, SOLR-1895-queries.patch, 
> SOLR-1895-queries.patch, SOLR-1895-queries.patch, SOLR-1895-queries.patch, 
> SOLR-1895-service-plugin.patch, SOLR-1895-service-plugin.patch, 
> SOLR-1895.patch, SOLR-1895.patch, SOLR-1895.patch, SOLR-1895.patch, 
> SOLR-1895.patch, SOLR-1895.patch
>
>
> I've written an LCF SearchComponent which filters returned results based on 
> access tokens provided by LCF's authority service.  The component requires 
> you to configure the appropriate authority service URL base, e.g.:
>   
>   
>  name="AuthorityServiceBaseURL">http://localhost:8080/lcf-authority-service
>   
> Also required are the following schema.xml additions:
>
> stored="false" multiValued="true"/>
> stored="false" multiValued="true"/>
> stored="false" multiValued="true"/>
> multiValued="true"/>
> Finally, to tie it into the standard request handler, it seems to need to run 
> last:
>   
> 
>   lcfSecurity
> 
> ...
> I have not set a package for this code.  Nor have I been able to get it 
> reviewed by someone as conversant with Solr as I would prefer.  It is my 
> hope, however, that this module will become part of the standard Solr 1.5 
> suite of search components, since that would tie it in with LCF nicely.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira



-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-1895) ManifoldCF SearchComponent plugin for enforcing ManifoldCF security at search time

[Karl Wright (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-1895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13115525#comment-13115525
 ] 

Karl Wright commented on SOLR-1895:
---

bq. Anyone against mcf-security as a contrib?

No objection here.  Do you want a new patch, or can you just move the directory 
and change build.xml to change the jar name?



> ManifoldCF SearchComponent plugin for enforcing ManifoldCF security at search 
> time
> --
>
> Key: SOLR-1895
> URL: https://issues.apache.org/jira/browse/SOLR-1895
> Project: Solr
>  Issue Type: New Feature
>  Components: SearchComponents - other
>Reporter: Karl Wright
>  Labels: document, security, solr
> Fix For: 3.5, 4.0
>
> Attachments: LCFSecurityFilter.java, LCFSecurityFilter.java, 
> LCFSecurityFilter.java, LCFSecurityFilter.java, SOLR-1895-queries.patch, 
> SOLR-1895-queries.patch, SOLR-1895-queries.patch, SOLR-1895-queries.patch, 
> SOLR-1895-service-plugin.patch, SOLR-1895-service-plugin.patch, 
> SOLR-1895.patch, SOLR-1895.patch, SOLR-1895.patch, SOLR-1895.patch, 
> SOLR-1895.patch, SOLR-1895.patch
>
>
> I've written an LCF SearchComponent which filters returned results based on 
> access tokens provided by LCF's authority service.  The component requires 
> you to configure the appropriate authority service URL base, e.g.:
>   
>   
>  name="AuthorityServiceBaseURL">http://localhost:8080/lcf-authority-service
>   
> Also required are the following schema.xml additions:
>
> stored="false" multiValued="true"/>
> stored="false" multiValued="true"/>
> stored="false" multiValued="true"/>
> multiValued="true"/>
> Finally, to tie it into the standard request handler, it seems to need to run 
> last:
>   
> 
>   lcfSecurity
> 
> ...
> I have not set a package for this code.  Nor have I been able to get it 
> reviewed by someone as conversant with Solr as I would prefer.  It is my 
> hope, however, that this module will become part of the standard Solr 1.5 
> suite of search components, since that would tie it in with LCF nicely.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira



-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Created] (LUCENE-3469) move DocumentStoredFieldsVisitor to o.a.l.document

[Robert Muir (Created) (JIRA)]

move DocumentStoredFieldsVisitor to o.a.l.document
--

 Key: LUCENE-3469
 URL: https://issues.apache.org/jira/browse/LUCENE-3469
 Project: Lucene - Java
  Issue Type: Task
Reporter: Robert Muir
 Fix For: 4.0
 Attachments: LUCENE-3469.patch

when examining the changes to the field/document API, i noticed this class was 
in o.a.l.index

I think it should be in o.a.l.document, its more intuitive packaging

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira



-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (LUCENE-3469) move DocumentStoredFieldsVisitor to o.a.l.document

[Robert Muir (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LUCENE-3469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Muir updated LUCENE-3469:


Attachment: LUCENE-3469.patch

> move DocumentStoredFieldsVisitor to o.a.l.document
> --
>
> Key: LUCENE-3469
> URL: https://issues.apache.org/jira/browse/LUCENE-3469
> Project: Lucene - Java
>  Issue Type: Task
>Reporter: Robert Muir
> Fix For: 4.0
>
> Attachments: LUCENE-3469.patch
>
>
> when examining the changes to the field/document API, i noticed this class 
> was in o.a.l.index
> I think it should be in o.a.l.document, its more intuitive packaging

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira



-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-3469) move DocumentStoredFieldsVisitor to o.a.l.document

[Michael McCandless (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-3469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13115529#comment-13115529
 ] 

Michael McCandless commented on LUCENE-3469:


+1

> move DocumentStoredFieldsVisitor to o.a.l.document
> --
>
> Key: LUCENE-3469
> URL: https://issues.apache.org/jira/browse/LUCENE-3469
> Project: Lucene - Java
>  Issue Type: Task
>Reporter: Robert Muir
> Fix For: 4.0
>
> Attachments: LUCENE-3469.patch
>
>
> when examining the changes to the field/document API, i noticed this class 
> was in o.a.l.index
> I think it should be in o.a.l.document, its more intuitive packaging

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira



-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-3469) move DocumentStoredFieldsVisitor to o.a.l.document

[Michael McCandless (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-3469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13115530#comment-13115530
 ] 

Michael McCandless commented on LUCENE-3469:


+1

> move DocumentStoredFieldsVisitor to o.a.l.document
> --
>
> Key: LUCENE-3469
> URL: https://issues.apache.org/jira/browse/LUCENE-3469
> Project: Lucene - Java
>  Issue Type: Task
>Reporter: Robert Muir
> Fix For: 4.0
>
> Attachments: LUCENE-3469.patch
>
>
> when examining the changes to the field/document API, i noticed this class 
> was in o.a.l.index
> I think it should be in o.a.l.document, its more intuitive packaging

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira



-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: [JENKINS] Lucene-Solr-tests-only-trunk - Build # 10670 - Failure

[Michael McCandless]

I'll take this.

Mike McCandless

http://blog.mikemccandless.com

On Tue, Sep 27, 2011 at 7:56 AM, Apache Jenkins Server
 wrote:
> Build: https://builds.apache.org/job/Lucene-Solr-tests-only-trunk/10670/
>
> 1 tests failed.
> REGRESSION:  org.apache.lucene.util.fst.TestFSTs.testRealTerms
>
> Error Message:
> Java heap space
>
> Stack Trace:
> java.lang.OutOfMemoryError: Java heap space
>        at org.apache.lucene.util.ArrayUtil.grow(ArrayUtil.java:338)
>        at org.apache.lucene.util.fst.FST.addNode(FST.java:480)
>        at org.apache.lucene.util.fst.NodeHash.add(NodeHash.java:122)
>        at org.apache.lucene.util.fst.Builder.compileNode(Builder.java:180)
>        at org.apache.lucene.util.fst.Builder.freezeTail(Builder.java:271)
>        at org.apache.lucene.util.fst.Builder.add(Builder.java:414)
>        at org.apache.lucene.util.fst.Builder.add(Builder.java:302)
>        at 
> org.apache.lucene.index.codecs.memory.MemoryCodec$TermsWriter.finishTerm(MemoryCodec.java:218)
>        at 
> org.apache.lucene.index.FreqProxTermsWriterPerField.flush(FreqProxTermsWriterPerField.java:409)
>        at 
> org.apache.lucene.index.FreqProxTermsWriter.flush(FreqProxTermsWriter.java:92)
>        at org.apache.lucene.index.TermsHash.flush(TermsHash.java:117)
>        at org.apache.lucene.index.DocInverter.flush(DocInverter.java:80)
>        at 
> org.apache.lucene.index.DocFieldProcessor.flush(DocFieldProcessor.java:78)
>        at 
> org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:472)
>        at 
> org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:420)
>        at 
> org.apache.lucene.index.DocumentsWriter.flushAllThreads(DocumentsWriter.java:568)
>        at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:366)
>        at org.apache.lucene.index.IndexReader.open(IndexReader.java:317)
>        at 
> org.apache.lucene.util.fst.TestFSTs.testRealTerms(TestFSTs.java:1034)
>        at 
> org.apache.lucene.util.LuceneTestCase$2$1.evaluate(LuceneTestCase.java:611)
>
>
>
>
> Build Log (for compile errors):
> [...truncated 1264 lines...]
>
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>
>

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-1895) ManifoldCF SearchComponent plugin for enforcing ManifoldCF security at search time

[Chris Male (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-1895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13115533#comment-13115533
 ] 

Chris Male commented on SOLR-1895:
--

bq. Anyone against mcf-security as a contrib?

Yes I am (although I won't vote -1 on it).  I don't feel it's a solution here, 
its just pushing the problem into the corner and hoping that out-of-sight 
out-of-mind.  If this is going to be in the solr codebase, why not make it 
core? It has no external dependencies and is tiny.  If we feel that it 
shouldn't be in solr-core, I don't know why it belongs in contrib.  Contrib 
shouldn't be where we go when we don't agree (code then gets ignored, falls out 
of sync, and ends up being sandboxed it after 2 years).

The issue seems to be whether this should be part of the solr project, or 
ManifoldCF.

> ManifoldCF SearchComponent plugin for enforcing ManifoldCF security at search 
> time
> --
>
> Key: SOLR-1895
> URL: https://issues.apache.org/jira/browse/SOLR-1895
> Project: Solr
>  Issue Type: New Feature
>  Components: SearchComponents - other
>Reporter: Karl Wright
>  Labels: document, security, solr
> Fix For: 3.5, 4.0
>
> Attachments: LCFSecurityFilter.java, LCFSecurityFilter.java, 
> LCFSecurityFilter.java, LCFSecurityFilter.java, SOLR-1895-queries.patch, 
> SOLR-1895-queries.patch, SOLR-1895-queries.patch, SOLR-1895-queries.patch, 
> SOLR-1895-service-plugin.patch, SOLR-1895-service-plugin.patch, 
> SOLR-1895.patch, SOLR-1895.patch, SOLR-1895.patch, SOLR-1895.patch, 
> SOLR-1895.patch, SOLR-1895.patch
>
>
> I've written an LCF SearchComponent which filters returned results based on 
> access tokens provided by LCF's authority service.  The component requires 
> you to configure the appropriate authority service URL base, e.g.:
>   
>   
>  name="AuthorityServiceBaseURL">http://localhost:8080/lcf-authority-service
>   
> Also required are the following schema.xml additions:
>
> stored="false" multiValued="true"/>
> stored="false" multiValued="true"/>
> stored="false" multiValued="true"/>
> multiValued="true"/>
> Finally, to tie it into the standard request handler, it seems to need to run 
> last:
>   
> 
>   lcfSecurity
> 
> ...
> I have not set a package for this code.  Nor have I been able to get it 
> reviewed by someone as conversant with Solr as I would prefer.  It is my 
> hope, however, that this module will become part of the standard Solr 1.5 
> suite of search components, since that would tie it in with LCF nicely.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira



-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-1895) ManifoldCF SearchComponent plugin for enforcing ManifoldCF security at search time

[Erik Hatcher (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-1895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13115536#comment-13115536
 ] 

Erik Hatcher commented on SOLR-1895:


bq. Anyone against mcf-security as a contrib?

No objection here either, though because this is such a small bit of code and 
is specific to MCF it seems best placed in MCF here.  The majority of people 
using Solr are not using MCF, though I'd venture to say that the majority of 
folks using MCF are using Solr.  It's maintenance best fits under the 
committership of MCF, in my opinion.  But again, no objections from me on it 
being a Solr contrib if others feel strongly about it.

I personally don't see a distillation of this, anytime soon, into a common 
security framework within Solr so unless someone already has this generified 
and a second real-world implementation of it I don't find it a compelling 
argument to try to make this generic from the start.  Though again, knock 
yourselves out folks.  I'm glad to see this work being done, for sure, and I 
support the effort wherever it ultimately lives.

> ManifoldCF SearchComponent plugin for enforcing ManifoldCF security at search 
> time
> --
>
> Key: SOLR-1895
> URL: https://issues.apache.org/jira/browse/SOLR-1895
> Project: Solr
>  Issue Type: New Feature
>  Components: SearchComponents - other
>Reporter: Karl Wright
>  Labels: document, security, solr
> Fix For: 3.5, 4.0
>
> Attachments: LCFSecurityFilter.java, LCFSecurityFilter.java, 
> LCFSecurityFilter.java, LCFSecurityFilter.java, SOLR-1895-queries.patch, 
> SOLR-1895-queries.patch, SOLR-1895-queries.patch, SOLR-1895-queries.patch, 
> SOLR-1895-service-plugin.patch, SOLR-1895-service-plugin.patch, 
> SOLR-1895.patch, SOLR-1895.patch, SOLR-1895.patch, SOLR-1895.patch, 
> SOLR-1895.patch, SOLR-1895.patch
>
>
> I've written an LCF SearchComponent which filters returned results based on 
> access tokens provided by LCF's authority service.  The component requires 
> you to configure the appropriate authority service URL base, e.g.:
>   
>   
>  name="AuthorityServiceBaseURL">http://localhost:8080/lcf-authority-service
>   
> Also required are the following schema.xml additions:
>
> stored="false" multiValued="true"/>
> stored="false" multiValued="true"/>
> stored="false" multiValued="true"/>
> multiValued="true"/>
> Finally, to tie it into the standard request handler, it seems to need to run 
> last:
>   
> 
>   lcfSecurity
> 
> ...
> I have not set a package for this code.  Nor have I been able to get it 
> reviewed by someone as conversant with Solr as I would prefer.  It is my 
> hope, however, that this module will become part of the standard Solr 1.5 
> suite of search components, since that would tie it in with LCF nicely.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira



-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

RE: [JENKINS] Lucene-Solr-tests-only-trunk - Build # 10670 - Failure

[Uwe Schindler]

I opened an issue about this yesterday:
https://issues.apache.org/jira/browse/LUCENE-3463

-
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: u...@thetaphi.de


> -Original Message-
> From: Michael McCandless [mailto:luc...@mikemccandless.com]
> Sent: Tuesday, September 27, 2011 4:25 PM
> To: dev@lucene.apache.org
> Subject: Re: [JENKINS] Lucene-Solr-tests-only-trunk - Build # 10670 -
Failure
> 
> I'll take this.
> 
> Mike McCandless
> 
> http://blog.mikemccandless.com
> 
> On Tue, Sep 27, 2011 at 7:56 AM, Apache Jenkins Server
>  wrote:
> > Build:
> > https://builds.apache.org/job/Lucene-Solr-tests-only-trunk/10670/
> >
> > 1 tests failed.
> > REGRESSION:  org.apache.lucene.util.fst.TestFSTs.testRealTerms
> >
> > Error Message:
> > Java heap space
> >
> > Stack Trace:
> > java.lang.OutOfMemoryError: Java heap space
> >        at org.apache.lucene.util.ArrayUtil.grow(ArrayUtil.java:338)
> >        at org.apache.lucene.util.fst.FST.addNode(FST.java:480)
> >        at org.apache.lucene.util.fst.NodeHash.add(NodeHash.java:122)
> >        at
> > org.apache.lucene.util.fst.Builder.compileNode(Builder.java:180)
> >        at
> > org.apache.lucene.util.fst.Builder.freezeTail(Builder.java:271)
> >        at org.apache.lucene.util.fst.Builder.add(Builder.java:414)
> >        at org.apache.lucene.util.fst.Builder.add(Builder.java:302)
> >        at
> >
> org.apache.lucene.index.codecs.memory.MemoryCodec$TermsWriter.finishTe
> > rm(MemoryCodec.java:218)
> >        at
> > org.apache.lucene.index.FreqProxTermsWriterPerField.flush(FreqProxTerm
> > sWriterPerField.java:409)
> >        at
> > org.apache.lucene.index.FreqProxTermsWriter.flush(FreqProxTermsWriter.
> > java:92)
> >        at org.apache.lucene.index.TermsHash.flush(TermsHash.java:117)
> >        at
> > org.apache.lucene.index.DocInverter.flush(DocInverter.java:80)
> >        at
> > org.apache.lucene.index.DocFieldProcessor.flush(DocFieldProcessor.java
> > :78)
> >        at
> > org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriter
> > PerThread.java:472)
> >        at
> > org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:4
> > 20)
> >        at
> > org.apache.lucene.index.DocumentsWriter.flushAllThreads(DocumentsWrite
> > r.java:568)
> >        at
> > org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:366)
> >        at
> > org.apache.lucene.index.IndexReader.open(IndexReader.java:317)
> >        at
> > org.apache.lucene.util.fst.TestFSTs.testRealTerms(TestFSTs.java:1034)
> >        at
> > org.apache.lucene.util.LuceneTestCase$2$1.evaluate(LuceneTestCase.java
> > :611)
> >
> >
> >
> >
> > Build Log (for compile errors):
> > [...truncated 1264 lines...]
> >
> >
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For
> > additional commands, e-mail: dev-h...@lucene.apache.org
> >
> >
> 
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional
> commands, e-mail: dev-h...@lucene.apache.org


-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-3463) Jenins trunk tests (nightly only) fail quite often with OOM in Automaton/FST tests

[Michael McCandless (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-3463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13115539#comment-13115539
 ] 

Michael McCandless commented on LUCENE-3463:


{noformat}
TestFSTs.testRealTerms -seed 4857a4f3c937e12e:15e316518d8ee33:-4b9de2133ce6125c 
-mult 3 
{noformat}

That'll hit OOME.  The problem is that the test gets RandomCodecProvider which 
then assigns MemoryCodec to a bad field.

I think we need a clean way to annotate that a given test cannot handle 
Memory/SimpleText codecs and then mark the above tests as such?

> Jenins trunk tests (nightly only) fail quite often with OOM in Automaton/FST 
> tests
> --
>
> Key: LUCENE-3463
> URL: https://issues.apache.org/jira/browse/LUCENE-3463
> Project: Lucene - Java
>  Issue Type: Bug
>Reporter: Uwe Schindler
>
> The nightly Job Lucene-trunk quite often fails with OOM (in several methods, 
> not always in the same test):
> Example from last night (this time a huge Automaton):
> {noformat}
> [junit] java.lang.OutOfMemoryError: Java heap space
> [junit] Dumping heap to 
> /home/hudson/hudson-slave/workspace/Lucene-trunk/heapdumps/java_pid38855.hprof
>  ...
> [junit] Heap dump file created [86965954 bytes in 1.186 secs]
> [junit] Testsuite: org.apache.lucene.index.TestTermsEnum
> [junit] Testcase: testIntersectRandom(org.apache.lucene.index.TestTermsEnum): 
> Caused an ERROR
> [junit] Java heap space
> [junit] java.lang.OutOfMemoryError: Java heap space
> [junit]   at 
> org.apache.lucene.util.automaton.RunAutomaton.(RunAutomaton.java:128)
> [junit]   at 
> org.apache.lucene.util.automaton.ByteRunAutomaton.(ByteRunAutomaton.java:28)
> [junit]   at 
> org.apache.lucene.util.automaton.CompiledAutomaton.(CompiledAutomaton.java:134)
> [junit]   at 
> org.apache.lucene.index.TestTermsEnum.testIntersectRandom(TestTermsEnum.java:266)
> [junit]   at 
> org.apache.lucene.util.LuceneTestCase$2$1.evaluate(LuceneTestCase.java:611)
> [junit]   at 
> org.apache.lucene.util.LuceneTestCaseRunner.runChild(LuceneTestCaseRunner.java:148)
> [junit]   at 
> org.apache.lucene.util.LuceneTestCaseRunner.runChild(LuceneTestCaseRunner.java:50)
> [junit] 
> [junit] 
> [junit] Tests run: 6, Failures: 0, Errors: 1, Time elapsed: 11.699 sec
> {noformat}
> Other traces:
> {noformat}
> [junit] Testsuite: org.apache.lucene.util.fst.TestFSTs
> [junit] Testcase: testRealTerms(org.apache.lucene.util.fst.TestFSTs): Caused 
> an ERROR
> [junit] Java heap space
> [junit] java.lang.OutOfMemoryError: Java heap space
> [junit]   at org.apache.lucene.util.ArrayUtil.grow(ArrayUtil.java:338)
> [junit]   at 
> org.apache.lucene.util.fst.FST$BytesWriter.writeBytes(FST.java:927)
> [junit]   at 
> org.apache.lucene.util.fst.ByteSequenceOutputs.write(ByteSequenceOutputs.java:113)
> [junit]   at 
> org.apache.lucene.util.fst.ByteSequenceOutputs.write(ByteSequenceOutputs.java:32)
> [junit]   at org.apache.lucene.util.fst.FST.addNode(FST.java:451)
> [junit]   at org.apache.lucene.util.fst.NodeHash.add(NodeHash.java:122)
> [junit]   at 
> org.apache.lucene.util.fst.Builder.compileNode(Builder.java:180)
> [junit]   at org.apache.lucene.util.fst.Builder.finish(Builder.java:495)
> [junit]   at 
> org.apache.lucene.index.codecs.memory.MemoryCodec$TermsWriter.finish(MemoryCodec.java:232)
> [junit]   at 
> org.apache.lucene.index.FreqProxTermsWriterPerField.flush(FreqProxTermsWriterPerField.java:414)
> [junit]   at 
> org.apache.lucene.index.FreqProxTermsWriter.flush(FreqProxTermsWriter.java:92)
> [junit]   at org.apache.lucene.index.TermsHash.flush(TermsHash.java:117)
> [junit]   at 
> org.apache.lucene.index.DocInverter.flush(DocInverter.java:80)
> [junit]   at 
> org.apache.lucene.index.DocFieldProcessor.flush(DocFieldProcessor.java:78)
> [junit]   at 
> org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:472)
> [junit]   at 
> org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:420)
> [junit]   at 
> org.apache.lucene.index.DocumentsWriter.flushAllThreads(DocumentsWriter.java:568)
> [junit]   at 
> org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:366)
> [junit]   at 
> org.apache.lucene.index.IndexReader.open(IndexReader.java:317)
> [junit]   at 
> org.apache.lucene.util.fst.TestFSTs.testRealTerms(TestFSTs.java:1034)
> [junit]   at 
> org.apache.lucene.util.LuceneTestCase$2$1.evaluate(LuceneTestCase.java:611)
> {noformat}
> or:
> {noformat}
> [junit] Testsuite: org.apache.lucene.util.automaton.TestCompiledAutomaton
> [junit] Testcase: 
> testRandom(org.apache.lucene.util.automaton.TestCompiledAutomaton): Caused an 
> ERROR
> [junit] Java heap space
> [junit] java.lang.OutOfMem

[jira] [Commented] (LUCENE-3463) Jenins trunk tests (nightly only) fail quite often with OOM in Automaton/FST tests

[Michael McCandless (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-3463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13115540#comment-13115540
 ] 

Michael McCandless commented on LUCENE-3463:


{noformat}
TestFSTs.testRealTerms -seed 4857a4f3c937e12e:15e316518d8ee33:-4b9de2133ce6125c 
-mult 3 
{noformat}

That'll hit OOME.  The problem is that the test gets RandomCodecProvider which 
then assigns MemoryCodec to a bad field.

I think we need a clean way to annotate that a given test cannot handle 
Memory/SimpleText codecs and then mark the above tests as such?

> Jenins trunk tests (nightly only) fail quite often with OOM in Automaton/FST 
> tests
> --
>
> Key: LUCENE-3463
> URL: https://issues.apache.org/jira/browse/LUCENE-3463
> Project: Lucene - Java
>  Issue Type: Bug
>Reporter: Uwe Schindler
>
> The nightly Job Lucene-trunk quite often fails with OOM (in several methods, 
> not always in the same test):
> Example from last night (this time a huge Automaton):
> {noformat}
> [junit] java.lang.OutOfMemoryError: Java heap space
> [junit] Dumping heap to 
> /home/hudson/hudson-slave/workspace/Lucene-trunk/heapdumps/java_pid38855.hprof
>  ...
> [junit] Heap dump file created [86965954 bytes in 1.186 secs]
> [junit] Testsuite: org.apache.lucene.index.TestTermsEnum
> [junit] Testcase: testIntersectRandom(org.apache.lucene.index.TestTermsEnum): 
> Caused an ERROR
> [junit] Java heap space
> [junit] java.lang.OutOfMemoryError: Java heap space
> [junit]   at 
> org.apache.lucene.util.automaton.RunAutomaton.(RunAutomaton.java:128)
> [junit]   at 
> org.apache.lucene.util.automaton.ByteRunAutomaton.(ByteRunAutomaton.java:28)
> [junit]   at 
> org.apache.lucene.util.automaton.CompiledAutomaton.(CompiledAutomaton.java:134)
> [junit]   at 
> org.apache.lucene.index.TestTermsEnum.testIntersectRandom(TestTermsEnum.java:266)
> [junit]   at 
> org.apache.lucene.util.LuceneTestCase$2$1.evaluate(LuceneTestCase.java:611)
> [junit]   at 
> org.apache.lucene.util.LuceneTestCaseRunner.runChild(LuceneTestCaseRunner.java:148)
> [junit]   at 
> org.apache.lucene.util.LuceneTestCaseRunner.runChild(LuceneTestCaseRunner.java:50)
> [junit] 
> [junit] 
> [junit] Tests run: 6, Failures: 0, Errors: 1, Time elapsed: 11.699 sec
> {noformat}
> Other traces:
> {noformat}
> [junit] Testsuite: org.apache.lucene.util.fst.TestFSTs
> [junit] Testcase: testRealTerms(org.apache.lucene.util.fst.TestFSTs): Caused 
> an ERROR
> [junit] Java heap space
> [junit] java.lang.OutOfMemoryError: Java heap space
> [junit]   at org.apache.lucene.util.ArrayUtil.grow(ArrayUtil.java:338)
> [junit]   at 
> org.apache.lucene.util.fst.FST$BytesWriter.writeBytes(FST.java:927)
> [junit]   at 
> org.apache.lucene.util.fst.ByteSequenceOutputs.write(ByteSequenceOutputs.java:113)
> [junit]   at 
> org.apache.lucene.util.fst.ByteSequenceOutputs.write(ByteSequenceOutputs.java:32)
> [junit]   at org.apache.lucene.util.fst.FST.addNode(FST.java:451)
> [junit]   at org.apache.lucene.util.fst.NodeHash.add(NodeHash.java:122)
> [junit]   at 
> org.apache.lucene.util.fst.Builder.compileNode(Builder.java:180)
> [junit]   at org.apache.lucene.util.fst.Builder.finish(Builder.java:495)
> [junit]   at 
> org.apache.lucene.index.codecs.memory.MemoryCodec$TermsWriter.finish(MemoryCodec.java:232)
> [junit]   at 
> org.apache.lucene.index.FreqProxTermsWriterPerField.flush(FreqProxTermsWriterPerField.java:414)
> [junit]   at 
> org.apache.lucene.index.FreqProxTermsWriter.flush(FreqProxTermsWriter.java:92)
> [junit]   at org.apache.lucene.index.TermsHash.flush(TermsHash.java:117)
> [junit]   at 
> org.apache.lucene.index.DocInverter.flush(DocInverter.java:80)
> [junit]   at 
> org.apache.lucene.index.DocFieldProcessor.flush(DocFieldProcessor.java:78)
> [junit]   at 
> org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:472)
> [junit]   at 
> org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:420)
> [junit]   at 
> org.apache.lucene.index.DocumentsWriter.flushAllThreads(DocumentsWriter.java:568)
> [junit]   at 
> org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:366)
> [junit]   at 
> org.apache.lucene.index.IndexReader.open(IndexReader.java:317)
> [junit]   at 
> org.apache.lucene.util.fst.TestFSTs.testRealTerms(TestFSTs.java:1034)
> [junit]   at 
> org.apache.lucene.util.LuceneTestCase$2$1.evaluate(LuceneTestCase.java:611)
> {noformat}
> or:
> {noformat}
> [junit] Testsuite: org.apache.lucene.util.automaton.TestCompiledAutomaton
> [junit] Testcase: 
> testRandom(org.apache.lucene.util.automaton.TestCompiledAutomaton): Caused an 
> ERROR
> [junit] Java heap space
> [junit] java.lang.OutOfMem

[jira] [Commented] (LUCENE-3463) Jenins trunk tests (nightly only) fail quite often with OOM in Automaton/FST tests

[Michael McCandless (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-3463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13115542#comment-13115542
 ] 

Michael McCandless commented on LUCENE-3463:


Here is a comment that I will only add once.

> Jenins trunk tests (nightly only) fail quite often with OOM in Automaton/FST 
> tests
> --
>
> Key: LUCENE-3463
> URL: https://issues.apache.org/jira/browse/LUCENE-3463
> Project: Lucene - Java
>  Issue Type: Bug
>Reporter: Uwe Schindler
>
> The nightly Job Lucene-trunk quite often fails with OOM (in several methods, 
> not always in the same test):
> Example from last night (this time a huge Automaton):
> {noformat}
> [junit] java.lang.OutOfMemoryError: Java heap space
> [junit] Dumping heap to 
> /home/hudson/hudson-slave/workspace/Lucene-trunk/heapdumps/java_pid38855.hprof
>  ...
> [junit] Heap dump file created [86965954 bytes in 1.186 secs]
> [junit] Testsuite: org.apache.lucene.index.TestTermsEnum
> [junit] Testcase: testIntersectRandom(org.apache.lucene.index.TestTermsEnum): 
> Caused an ERROR
> [junit] Java heap space
> [junit] java.lang.OutOfMemoryError: Java heap space
> [junit]   at 
> org.apache.lucene.util.automaton.RunAutomaton.(RunAutomaton.java:128)
> [junit]   at 
> org.apache.lucene.util.automaton.ByteRunAutomaton.(ByteRunAutomaton.java:28)
> [junit]   at 
> org.apache.lucene.util.automaton.CompiledAutomaton.(CompiledAutomaton.java:134)
> [junit]   at 
> org.apache.lucene.index.TestTermsEnum.testIntersectRandom(TestTermsEnum.java:266)
> [junit]   at 
> org.apache.lucene.util.LuceneTestCase$2$1.evaluate(LuceneTestCase.java:611)
> [junit]   at 
> org.apache.lucene.util.LuceneTestCaseRunner.runChild(LuceneTestCaseRunner.java:148)
> [junit]   at 
> org.apache.lucene.util.LuceneTestCaseRunner.runChild(LuceneTestCaseRunner.java:50)
> [junit] 
> [junit] 
> [junit] Tests run: 6, Failures: 0, Errors: 1, Time elapsed: 11.699 sec
> {noformat}
> Other traces:
> {noformat}
> [junit] Testsuite: org.apache.lucene.util.fst.TestFSTs
> [junit] Testcase: testRealTerms(org.apache.lucene.util.fst.TestFSTs): Caused 
> an ERROR
> [junit] Java heap space
> [junit] java.lang.OutOfMemoryError: Java heap space
> [junit]   at org.apache.lucene.util.ArrayUtil.grow(ArrayUtil.java:338)
> [junit]   at 
> org.apache.lucene.util.fst.FST$BytesWriter.writeBytes(FST.java:927)
> [junit]   at 
> org.apache.lucene.util.fst.ByteSequenceOutputs.write(ByteSequenceOutputs.java:113)
> [junit]   at 
> org.apache.lucene.util.fst.ByteSequenceOutputs.write(ByteSequenceOutputs.java:32)
> [junit]   at org.apache.lucene.util.fst.FST.addNode(FST.java:451)
> [junit]   at org.apache.lucene.util.fst.NodeHash.add(NodeHash.java:122)
> [junit]   at 
> org.apache.lucene.util.fst.Builder.compileNode(Builder.java:180)
> [junit]   at org.apache.lucene.util.fst.Builder.finish(Builder.java:495)
> [junit]   at 
> org.apache.lucene.index.codecs.memory.MemoryCodec$TermsWriter.finish(MemoryCodec.java:232)
> [junit]   at 
> org.apache.lucene.index.FreqProxTermsWriterPerField.flush(FreqProxTermsWriterPerField.java:414)
> [junit]   at 
> org.apache.lucene.index.FreqProxTermsWriter.flush(FreqProxTermsWriter.java:92)
> [junit]   at org.apache.lucene.index.TermsHash.flush(TermsHash.java:117)
> [junit]   at 
> org.apache.lucene.index.DocInverter.flush(DocInverter.java:80)
> [junit]   at 
> org.apache.lucene.index.DocFieldProcessor.flush(DocFieldProcessor.java:78)
> [junit]   at 
> org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:472)
> [junit]   at 
> org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:420)
> [junit]   at 
> org.apache.lucene.index.DocumentsWriter.flushAllThreads(DocumentsWriter.java:568)
> [junit]   at 
> org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:366)
> [junit]   at 
> org.apache.lucene.index.IndexReader.open(IndexReader.java:317)
> [junit]   at 
> org.apache.lucene.util.fst.TestFSTs.testRealTerms(TestFSTs.java:1034)
> [junit]   at 
> org.apache.lucene.util.LuceneTestCase$2$1.evaluate(LuceneTestCase.java:611)
> {noformat}
> or:
> {noformat}
> [junit] Testsuite: org.apache.lucene.util.automaton.TestCompiledAutomaton
> [junit] Testcase: 
> testRandom(org.apache.lucene.util.automaton.TestCompiledAutomaton): Caused an 
> ERROR
> [junit] Java heap space
> [junit] java.lang.OutOfMemoryError: Java heap space
> [junit]   at 
> org.apache.lucene.util.automaton.RunAutomaton.(RunAutomaton.java:128)
> [junit]   at 
> org.apache.lucene.util.automaton.ByteRunAutomaton.(ByteRunAutomaton.java:28)
> [junit]   at 
> org.apache.lucene.util.automaton.CompiledAutomaton.(CompiledAutomaton.java:134)
> [junit

Re: [JENKINS] Lucene-Solr-tests-only-trunk - Build # 10670 - Failure

[Michael McCandless]

Thanks Uwe.

I just put a comment there.

Mike McCandless

http://blog.mikemccandless.com

On Tue, Sep 27, 2011 at 10:37 AM, Uwe Schindler  wrote:
> I opened an issue about this yesterday:
> https://issues.apache.org/jira/browse/LUCENE-3463
>
> -
> Uwe Schindler
> H.-H.-Meier-Allee 63, D-28213 Bremen
> http://www.thetaphi.de
> eMail: u...@thetaphi.de
>
>
>> -Original Message-
>> From: Michael McCandless [mailto:luc...@mikemccandless.com]
>> Sent: Tuesday, September 27, 2011 4:25 PM
>> To: dev@lucene.apache.org
>> Subject: Re: [JENKINS] Lucene-Solr-tests-only-trunk - Build # 10670 -
> Failure
>>
>> I'll take this.
>>
>> Mike McCandless
>>
>> http://blog.mikemccandless.com
>>
>> On Tue, Sep 27, 2011 at 7:56 AM, Apache Jenkins Server
>>  wrote:
>> > Build:
>> > https://builds.apache.org/job/Lucene-Solr-tests-only-trunk/10670/
>> >
>> > 1 tests failed.
>> > REGRESSION:  org.apache.lucene.util.fst.TestFSTs.testRealTerms
>> >
>> > Error Message:
>> > Java heap space
>> >
>> > Stack Trace:
>> > java.lang.OutOfMemoryError: Java heap space
>> >        at org.apache.lucene.util.ArrayUtil.grow(ArrayUtil.java:338)
>> >        at org.apache.lucene.util.fst.FST.addNode(FST.java:480)
>> >        at org.apache.lucene.util.fst.NodeHash.add(NodeHash.java:122)
>> >        at
>> > org.apache.lucene.util.fst.Builder.compileNode(Builder.java:180)
>> >        at
>> > org.apache.lucene.util.fst.Builder.freezeTail(Builder.java:271)
>> >        at org.apache.lucene.util.fst.Builder.add(Builder.java:414)
>> >        at org.apache.lucene.util.fst.Builder.add(Builder.java:302)
>> >        at
>> >
>> org.apache.lucene.index.codecs.memory.MemoryCodec$TermsWriter.finishTe
>> > rm(MemoryCodec.java:218)
>> >        at
>> > org.apache.lucene.index.FreqProxTermsWriterPerField.flush(FreqProxTerm
>> > sWriterPerField.java:409)
>> >        at
>> > org.apache.lucene.index.FreqProxTermsWriter.flush(FreqProxTermsWriter.
>> > java:92)
>> >        at org.apache.lucene.index.TermsHash.flush(TermsHash.java:117)
>> >        at
>> > org.apache.lucene.index.DocInverter.flush(DocInverter.java:80)
>> >        at
>> > org.apache.lucene.index.DocFieldProcessor.flush(DocFieldProcessor.java
>> > :78)
>> >        at
>> > org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriter
>> > PerThread.java:472)
>> >        at
>> > org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:4
>> > 20)
>> >        at
>> > org.apache.lucene.index.DocumentsWriter.flushAllThreads(DocumentsWrite
>> > r.java:568)
>> >        at
>> > org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:366)
>> >        at
>> > org.apache.lucene.index.IndexReader.open(IndexReader.java:317)
>> >        at
>> > org.apache.lucene.util.fst.TestFSTs.testRealTerms(TestFSTs.java:1034)
>> >        at
>> > org.apache.lucene.util.LuceneTestCase$2$1.evaluate(LuceneTestCase.java
>> > :611)
>> >
>> >
>> >
>> >
>> > Build Log (for compile errors):
>> > [...truncated 1264 lines...]
>> >
>> >
>> >
>> > -
>> > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For
>> > additional commands, e-mail: dev-h...@lucene.apache.org
>> >
>> >
>>
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional
>> commands, e-mail: dev-h...@lucene.apache.org
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>
>

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-3463) Jenins trunk tests (nightly only) fail quite often with OOM in Automaton/FST tests

[Uwe Schindler (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-3463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13115544#comment-13115544
 ] 

Uwe Schindler commented on LUCENE-3463:
---

LOL. I just wondered about it, because this was the second time today a comment 
was posted twice. I was thinking about a problem with JIRA.

> Jenins trunk tests (nightly only) fail quite often with OOM in Automaton/FST 
> tests
> --
>
> Key: LUCENE-3463
> URL: https://issues.apache.org/jira/browse/LUCENE-3463
> Project: Lucene - Java
>  Issue Type: Bug
>Reporter: Uwe Schindler
>
> The nightly Job Lucene-trunk quite often fails with OOM (in several methods, 
> not always in the same test):
> Example from last night (this time a huge Automaton):
> {noformat}
> [junit] java.lang.OutOfMemoryError: Java heap space
> [junit] Dumping heap to 
> /home/hudson/hudson-slave/workspace/Lucene-trunk/heapdumps/java_pid38855.hprof
>  ...
> [junit] Heap dump file created [86965954 bytes in 1.186 secs]
> [junit] Testsuite: org.apache.lucene.index.TestTermsEnum
> [junit] Testcase: testIntersectRandom(org.apache.lucene.index.TestTermsEnum): 
> Caused an ERROR
> [junit] Java heap space
> [junit] java.lang.OutOfMemoryError: Java heap space
> [junit]   at 
> org.apache.lucene.util.automaton.RunAutomaton.(RunAutomaton.java:128)
> [junit]   at 
> org.apache.lucene.util.automaton.ByteRunAutomaton.(ByteRunAutomaton.java:28)
> [junit]   at 
> org.apache.lucene.util.automaton.CompiledAutomaton.(CompiledAutomaton.java:134)
> [junit]   at 
> org.apache.lucene.index.TestTermsEnum.testIntersectRandom(TestTermsEnum.java:266)
> [junit]   at 
> org.apache.lucene.util.LuceneTestCase$2$1.evaluate(LuceneTestCase.java:611)
> [junit]   at 
> org.apache.lucene.util.LuceneTestCaseRunner.runChild(LuceneTestCaseRunner.java:148)
> [junit]   at 
> org.apache.lucene.util.LuceneTestCaseRunner.runChild(LuceneTestCaseRunner.java:50)
> [junit] 
> [junit] 
> [junit] Tests run: 6, Failures: 0, Errors: 1, Time elapsed: 11.699 sec
> {noformat}
> Other traces:
> {noformat}
> [junit] Testsuite: org.apache.lucene.util.fst.TestFSTs
> [junit] Testcase: testRealTerms(org.apache.lucene.util.fst.TestFSTs): Caused 
> an ERROR
> [junit] Java heap space
> [junit] java.lang.OutOfMemoryError: Java heap space
> [junit]   at org.apache.lucene.util.ArrayUtil.grow(ArrayUtil.java:338)
> [junit]   at 
> org.apache.lucene.util.fst.FST$BytesWriter.writeBytes(FST.java:927)
> [junit]   at 
> org.apache.lucene.util.fst.ByteSequenceOutputs.write(ByteSequenceOutputs.java:113)
> [junit]   at 
> org.apache.lucene.util.fst.ByteSequenceOutputs.write(ByteSequenceOutputs.java:32)
> [junit]   at org.apache.lucene.util.fst.FST.addNode(FST.java:451)
> [junit]   at org.apache.lucene.util.fst.NodeHash.add(NodeHash.java:122)
> [junit]   at 
> org.apache.lucene.util.fst.Builder.compileNode(Builder.java:180)
> [junit]   at org.apache.lucene.util.fst.Builder.finish(Builder.java:495)
> [junit]   at 
> org.apache.lucene.index.codecs.memory.MemoryCodec$TermsWriter.finish(MemoryCodec.java:232)
> [junit]   at 
> org.apache.lucene.index.FreqProxTermsWriterPerField.flush(FreqProxTermsWriterPerField.java:414)
> [junit]   at 
> org.apache.lucene.index.FreqProxTermsWriter.flush(FreqProxTermsWriter.java:92)
> [junit]   at org.apache.lucene.index.TermsHash.flush(TermsHash.java:117)
> [junit]   at 
> org.apache.lucene.index.DocInverter.flush(DocInverter.java:80)
> [junit]   at 
> org.apache.lucene.index.DocFieldProcessor.flush(DocFieldProcessor.java:78)
> [junit]   at 
> org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:472)
> [junit]   at 
> org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:420)
> [junit]   at 
> org.apache.lucene.index.DocumentsWriter.flushAllThreads(DocumentsWriter.java:568)
> [junit]   at 
> org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:366)
> [junit]   at 
> org.apache.lucene.index.IndexReader.open(IndexReader.java:317)
> [junit]   at 
> org.apache.lucene.util.fst.TestFSTs.testRealTerms(TestFSTs.java:1034)
> [junit]   at 
> org.apache.lucene.util.LuceneTestCase$2$1.evaluate(LuceneTestCase.java:611)
> {noformat}
> or:
> {noformat}
> [junit] Testsuite: org.apache.lucene.util.automaton.TestCompiledAutomaton
> [junit] Testcase: 
> testRandom(org.apache.lucene.util.automaton.TestCompiledAutomaton): Caused an 
> ERROR
> [junit] Java heap space
> [junit] java.lang.OutOfMemoryError: Java heap space
> [junit]   at 
> org.apache.lucene.util.automaton.RunAutomaton.(RunAutomaton.java:128)
> [junit]   at 
> org.apache.lucene.util.automaton.ByteRunAutomaton.(ByteRunAutomaton.java:28)
> [junit]   at 
> or

[jira] [Commented] (SOLR-1895) ManifoldCF SearchComponent plugin for enforcing ManifoldCF security at search time

[Mark Miller (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-1895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13115548#comment-13115548
 ] 

Mark Miller commented on SOLR-1895:
---

bq. mcf-security as a contrib

I'd prefer not - I think mcf-security belongs in MCF.

> ManifoldCF SearchComponent plugin for enforcing ManifoldCF security at search 
> time
> --
>
> Key: SOLR-1895
> URL: https://issues.apache.org/jira/browse/SOLR-1895
> Project: Solr
>  Issue Type: New Feature
>  Components: SearchComponents - other
>Reporter: Karl Wright
>  Labels: document, security, solr
> Fix For: 3.5, 4.0
>
> Attachments: LCFSecurityFilter.java, LCFSecurityFilter.java, 
> LCFSecurityFilter.java, LCFSecurityFilter.java, SOLR-1895-queries.patch, 
> SOLR-1895-queries.patch, SOLR-1895-queries.patch, SOLR-1895-queries.patch, 
> SOLR-1895-service-plugin.patch, SOLR-1895-service-plugin.patch, 
> SOLR-1895.patch, SOLR-1895.patch, SOLR-1895.patch, SOLR-1895.patch, 
> SOLR-1895.patch, SOLR-1895.patch
>
>
> I've written an LCF SearchComponent which filters returned results based on 
> access tokens provided by LCF's authority service.  The component requires 
> you to configure the appropriate authority service URL base, e.g.:
>   
>   
>  name="AuthorityServiceBaseURL">http://localhost:8080/lcf-authority-service
>   
> Also required are the following schema.xml additions:
>
> stored="false" multiValued="true"/>
> stored="false" multiValued="true"/>
> stored="false" multiValued="true"/>
> multiValued="true"/>
> Finally, to tie it into the standard request handler, it seems to need to run 
> last:
>   
> 
>   lcfSecurity
> 
> ...
> I have not set a package for this code.  Nor have I been able to get it 
> reviewed by someone as conversant with Solr as I would prefer.  It is my 
> hope, however, that this module will become part of the standard Solr 1.5 
> suite of search components, since that would tie it in with LCF nicely.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira



-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Replication: abortfetch is broken if replication was started by fetchindex?

[Yury Kats]

Related question/issue: what about core close hook? Does it need to abort
replication if a core is being deleted while it's replicating (started by 
fetchhindex)?


On 9/23/2011 7:39 AM, Shalin Shekhar Mangar wrote:
> You are right Yury. It is a bug. The fix is to use tempSnapPuller.abortPull 
> always because tempSnapPuller is always the snap puller doing the actual 
> work. Have you opened an issue yet?
> 
> 2011/9/23 Yury Kats mailto:yuryk...@yahoo.com>>
> 
> If I start replication by issuing a "fetchindex" command with a 
> "masterUrl"
> parameter, it seems to me (by looking at the trunk code) that 
> "abortfetch" command
> would not do anything. It appears to be aborting only when replication is
> configured with hardcoded master/slave cores.
> 
> Is this intentional? An oversight? Am I missing something in the code and
> abortfetch does work in all cases?
> 
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org 
> 
> For additional commands, e-mail: dev-h...@lucene.apache.org 
> 
> 
> 
> 
> 
> -- 
> Regards,
> Shalin Shekhar Mangar.


-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Created] (LUCENE-3470) reorder arguments of Field constructor to be more intuitive

[Robert Muir (Created) (JIRA)]

reorder arguments of Field constructor to be more intuitive
---

 Key: LUCENE-3470
 URL: https://issues.apache.org/jira/browse/LUCENE-3470
 Project: Lucene - Java
  Issue Type: Task
Reporter: Robert Muir
 Fix For: 4.0


I think Field should take (name, value, type) not (name, type, value) ?

This seems more intuitive and consistent with previous releases

Take this change to some code I had for example:
{noformat}
-d1.add(new Field("foo", "bar", Field.Store.YES, Field.Index.ANALYZED));
+d1.add(new Field("foo", TextField.TYPE_STORED, "bar"));
{noformat}

I think it would be better if it was
{noformat}
document.add(new Field("foo", "bar", TextField.TYPE_STORED));
{noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira



-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-3470) reorder arguments of Field constructor to be more intuitive

[Michael McCandless (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-3470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13115563#comment-13115563
 ] 

Michael McCandless commented on LUCENE-3470:


+1, we should minimize the change on upgrade as much as we can.

> reorder arguments of Field constructor to be more intuitive
> ---
>
> Key: LUCENE-3470
> URL: https://issues.apache.org/jira/browse/LUCENE-3470
> Project: Lucene - Java
>  Issue Type: Task
>Reporter: Robert Muir
> Fix For: 4.0
>
>
> I think Field should take (name, value, type) not (name, type, value) ?
> This seems more intuitive and consistent with previous releases
> Take this change to some code I had for example:
> {noformat}
> -d1.add(new Field("foo", "bar", Field.Store.YES, Field.Index.ANALYZED));
> +d1.add(new Field("foo", TextField.TYPE_STORED, "bar"));
> {noformat}
> I think it would be better if it was
> {noformat}
> document.add(new Field("foo", "bar", TextField.TYPE_STORED));
> {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira



-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

trunk test failure (1317134941)

[Charlie Cron]

A test failure occurred running the tests.

revert! revert! revert!

You can see the entire build log at 
http://sierranevada.servebeer.com/1317134941.log

Thanks,
Charlie Cron


-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Created] (LUCENE-3471) TestNRTManager test failure

[Robert Muir (Created) (JIRA)]

TestNRTManager test failure
---

 Key: LUCENE-3471
 URL: https://issues.apache.org/jira/browse/LUCENE-3471
 Project: Lucene - Java
  Issue Type: Bug
Reporter: Robert Muir


reproduces for me

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira



-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-3471) TestNRTManager test failure

[Robert Muir (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-3471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13115572#comment-13115572
 ] 

Robert Muir commented on LUCENE-3471:
-

{noformat}
   [junit] Testsuite: org.apache.lucene.index.TestNRTManager
[junit] Testcase: testNRTManager(org.apache.lucene.index.TestNRTManager):   
Caused an ERROR
[junit] java.lang.AssertionError: Some threads threw uncaught exceptions!
[junit] java.lang.RuntimeException: java.lang.AssertionError: Some threads 
threw uncaught exceptions!
[junit] at 
org.apache.lucene.util.LuceneTestCase.tearDown(LuceneTestCase.java:729)
[junit] at 
org.apache.lucene.util.LuceneTestCaseRunner.runChild(LuceneTestCaseRunner.java:148)
[junit] at 
org.apache.lucene.util.LuceneTestCaseRunner.runChild(LuceneTestCaseRunner.java:50)
[junit] at 
org.apache.lucene.util.LuceneTestCase.checkUncaughtExceptionsAfter(LuceneTestCase.java:757)
[junit] at 
org.apache.lucene.util.LuceneTestCase.tearDown(LuceneTestCase.java:701)
[junit] at 
org.apache.lucene.util.LuceneTestCaseRunner.runChild(LuceneTestCaseRunner.java:148)
[junit] at 
org.apache.lucene.util.LuceneTestCaseRunner.runChild(LuceneTestCaseRunner.java:50)
[junit] 
[junit] 
[junit] Tests run: 1, Failures: 0, Errors: 1, Time elapsed: 6.796 sec
[junit] 
[junit] - Standard Error -
[junit] The following exceptions were thrown by threads:
[junit] *** Thread: Lucene Merge Thread #0 ***
[junit] org.apache.lucene.index.MergePolicy$MergeException: 
java.util.concurrent.RejectedExecutionException
[junit] at 
org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:509)
[junit] at 
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:474)
[junit] Caused by: java.util.concurrent.RejectedExecutionException
[junit] at 
java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:1768)
[junit] at 
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
[junit] at 
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
[junit] at 
java.util.concurrent.ExecutorCompletionService.submit(ExecutorCompletionService.java:152)
[junit] at 
org.apache.lucene.search.IndexSearcher$ExecutionHelper.submit(IndexSearcher.java:873)
[junit] at 
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:411)
[junit] at 
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:324)
[junit] at 
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:313)
[junit] at 
org.apache.lucene.index.TestNRTManager$1.warm(TestNRTManager.java:186)
[junit] at org.apache.lucene.index.NRTManager$1.warm(NRTManager.java:99)
[junit] at 
org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3782)
[junit] at 
org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3315)
[junit] at 
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:380)
[junit] at 
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:447)
[junit] NOTE: reproduce with: ant test -Dtestcase=TestNRTManager 
-Dtestmethod=testNRTManager 
-Dtests.seed=-1516b25f35b4703c:6f8c73d8e35c63b5:2d1be8301e699d26 
-Dtests.multiplier=3
[junit] NOTE: test params are: codec=RandomCodecProvider: {extra38=Memory, 
extra11=MockVariableIntBlock(baseBlockSize=108), body=MockRandom, 
extra9=MockSep, extra27=MockRandom, extra33=MockRandom, 
extra36=Pulsing(freqCutoff=7 minBlockSize=100 maxBlockSize=261), 
extra23=Memory, extra22=MockSep, extra0=MockFixedIntBlock(blockSize=1027), 
packID=MockSep, extra19=MockFixedIntBlock(blockSize=1027), 
date=Pulsing(freqCutoff=7 minBlockSize=100 maxBlockSize=261), 
extra4=MockRandom, extra6=MockDocValuesCodec, extra7=MockRandom, docid=Memory, 
title=Standard(minBlockSize=5 maxBlockSize=46), 
titleTokenized=MockFixedIntBlock(blockSize=1027), extra30=MockSep, 
extra10=Standard(minBlockSize=5 maxBlockSize=46)}, 
sim=RandomSimilarityProvider(queryNorm=true,coord=false): {extra38=DFR 
I(n)3(800.0), extra11=DFR GL3(800.0), body=DFR BeB3(800.0), extra9=IB SPL-D2, 
extra27=DFR G1, extra33=DFR Be3(800.0), extra36=DFR I(ne)LZ(0.3), extra23=IB 
SPL-LZ(0.3), extra22=DFR I(F)LZ(0.3), extra0=DFR GB3(800.0), packID=DFR 
BeL3(800.0), extra19=DFR BeZ(0.3), date=DFR GB3(800.0), extra4=DFR Be2, 
extra6=DFR I(n)2, extra7=DFR I(n)L3(800.0), docid=DFR I(F)B3(800.0), title=IB 
SPL-D1, titleTokenized=LM Jelinek-Mercer(0.10), extra30=DFR I(F)L2, 
extra10=DFR I(F)2}, locale=hu, timezone=Etc/GMT+9
[junit] NOTE: all tests run in this JVM:
[junit] [TestNRTManager]
[junit] NOTE: Linux 2.6.32-24-g

Re: trunk test failure (1317134941)

[Robert Muir]

I opened https://issues.apache.org/jira/browse/LUCENE-3471

On Tue, Sep 27, 2011 at 11:24 AM, Charlie Cron
 wrote:
> A test failure occurred running the tests.
>
> revert! revert! revert!
>
> You can see the entire build log at 
> http://sierranevada.servebeer.com/1317134941.log
>
> Thanks,
> Charlie Cron
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>
>



-- 
lucidimagination.com

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-tests-only-trunk - Build # 10674 - Failure

[Apache Jenkins Server]

Build: https://builds.apache.org/job/Lucene-Solr-tests-only-trunk/10674/

1 tests failed.
REGRESSION:  org.apache.solr.search.TestRealTimeGet.testStressGetRealtime

Error Message:
java.lang.AssertionError: Some threads threw uncaught exceptions!

Stack Trace:
java.lang.RuntimeException: java.lang.AssertionError: Some threads threw 
uncaught exceptions!
at 
org.apache.lucene.util.LuceneTestCase.tearDown(LuceneTestCase.java:729)
at org.apache.solr.SolrTestCaseJ4.tearDown(SolrTestCaseJ4.java:89)
at 
org.apache.lucene.util.LuceneTestCaseRunner.runChild(LuceneTestCaseRunner.java:148)
at 
org.apache.lucene.util.LuceneTestCaseRunner.runChild(LuceneTestCaseRunner.java:50)
at 
org.apache.lucene.util.LuceneTestCase.checkUncaughtExceptionsAfter(LuceneTestCase.java:757)
at 
org.apache.lucene.util.LuceneTestCase.tearDown(LuceneTestCase.java:701)




Build Log (for compile errors):
[...truncated 7887 lines...]



-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-1895) ManifoldCF SearchComponent plugin for enforcing ManifoldCF security at search time

[Commented]

[ 
https://issues.apache.org/jira/browse/SOLR-1895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13115601#comment-13115601
 ] 

Jan Høydahl commented on SOLR-1895:
---

The most important to me is that great contributions like this are being 
welcomed and brought out to Solr users somehow, whether the code lives here or 
there from the start. As I have a few projects in the workings that need 
non-mcf security filtering integration with Solr I'd be contributing to this 
code base.

Another way to think of contrib is as a great way to introduce people to 
integration with components that may be of interest to their search app. A new 
Solr user would browse contrib and think "I don't need dataImportHandler, or 
clustering, but I could surely use extraction, langId and security". She may 
not even be familiar with any of Carrot2, Tika or MCF in advance. Connectors 
and security is important to more Solr users than e.g. clustering is, in my 
experience.

> ManifoldCF SearchComponent plugin for enforcing ManifoldCF security at search 
> time
> --
>
> Key: SOLR-1895
> URL: https://issues.apache.org/jira/browse/SOLR-1895
> Project: Solr
>  Issue Type: New Feature
>  Components: SearchComponents - other
>Reporter: Karl Wright
>  Labels: document, security, solr
> Fix For: 3.5, 4.0
>
> Attachments: LCFSecurityFilter.java, LCFSecurityFilter.java, 
> LCFSecurityFilter.java, LCFSecurityFilter.java, SOLR-1895-queries.patch, 
> SOLR-1895-queries.patch, SOLR-1895-queries.patch, SOLR-1895-queries.patch, 
> SOLR-1895-service-plugin.patch, SOLR-1895-service-plugin.patch, 
> SOLR-1895.patch, SOLR-1895.patch, SOLR-1895.patch, SOLR-1895.patch, 
> SOLR-1895.patch, SOLR-1895.patch
>
>
> I've written an LCF SearchComponent which filters returned results based on 
> access tokens provided by LCF's authority service.  The component requires 
> you to configure the appropriate authority service URL base, e.g.:
>   
>   
>  name="AuthorityServiceBaseURL">http://localhost:8080/lcf-authority-service
>   
> Also required are the following schema.xml additions:
>
> stored="false" multiValued="true"/>
> stored="false" multiValued="true"/>
> stored="false" multiValued="true"/>
> multiValued="true"/>
> Finally, to tie it into the standard request handler, it seems to need to run 
> last:
>   
> 
>   lcfSecurity
> 
> ...
> I have not set a package for this code.  Nor have I been able to get it 
> reviewed by someone as conversant with Solr as I would prefer.  It is my 
> hope, however, that this module will become part of the standard Solr 1.5 
> suite of search components, since that would tie it in with LCF nicely.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira



-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-tests-only-trunk-java7 - Build # 527 - Failure

[Apache Jenkins Server]

Build: https://builds.apache.org/job/Lucene-Solr-tests-only-trunk-java7/527/

2 tests failed.
REGRESSION:  org.apache.solr.cloud.ZkControllerTest.testReadShards

Error Message:
Could not connect to ZooKeeper 127.0.0.1:27320 within 3 ms

Stack Trace:
java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 
127.0.0.1:27320 within 3 ms
at 
org.apache.solr.common.cloud.ConnectionManager.waitForConnected(ConnectionManager.java:124)
at 
org.apache.solr.common.cloud.SolrZkClient.(SolrZkClient.java:121)
at 
org.apache.solr.common.cloud.SolrZkClient.(SolrZkClient.java:84)
at 
org.apache.solr.common.cloud.SolrZkClient.(SolrZkClient.java:65)
at 
org.apache.solr.cloud.AbstractZkTestCase.tryCleanPath(AbstractZkTestCase.java:137)
at 
org.apache.solr.cloud.AbstractZkTestCase.tryCleanPath(AbstractZkTestCase.java:141)
at 
org.apache.solr.cloud.AbstractZkTestCase.tryCleanPath(AbstractZkTestCase.java:141)
at 
org.apache.solr.cloud.AbstractZkTestCase.tryCleanPath(AbstractZkTestCase.java:141)
at 
org.apache.solr.cloud.AbstractZkTestCase.tryCleanSolrZkNode(AbstractZkTestCase.java:133)
at 
org.apache.solr.cloud.ZkControllerTest.testReadShards(ZkControllerTest.java:74)
at 
org.apache.lucene.util.LuceneTestCase$2$1.evaluate(LuceneTestCase.java:611)
at 
org.apache.lucene.util.LuceneTestCaseRunner.runChild(LuceneTestCaseRunner.java:148)
at 
org.apache.lucene.util.LuceneTestCaseRunner.runChild(LuceneTestCaseRunner.java:50)


REGRESSION:  org.apache.solr.update.AutoCommitTest.testSoftAndHardCommitMaxTime

Error Message:
null

Stack Trace:
junit.framework.AssertionFailedError: 
at 
org.apache.lucene.util.LuceneTestCaseRunner.runChild(LuceneTestCaseRunner.java:148)
at 
org.apache.lucene.util.LuceneTestCaseRunner.runChild(LuceneTestCaseRunner.java:50)
at 
org.apache.solr.update.AutoCommitTest.testSoftAndHardCommitMaxTime(AutoCommitTest.java:522)
at 
org.apache.lucene.util.LuceneTestCase$2$1.evaluate(LuceneTestCase.java:611)




Build Log (for compile errors):
[...truncated 11070 lines...]



-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-1895) ManifoldCF SearchComponent plugin for enforcing ManifoldCF security at search time

[Mark Miller (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-1895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13115690#comment-13115690
 ] 

Mark Miller commented on SOLR-1895:
---

I still think MCF integration with SOLR should live in MCF.

If there are more general 'filtering' components we can add to Solr, lets 
create new issues around them.

I also think that if we go down the route of working on a general "security" 
framework, that we don't actually use the word security - as it implies much 
more than we will be providing based on any of what I have seen proposed. If we 
want to add a "document filtering" framework of some kind, that's something I 
would happily endorse. I'm still don't think we want to enter the 'security' 
game - there is a reason this has generally been pushed off to users in the 
past.

> ManifoldCF SearchComponent plugin for enforcing ManifoldCF security at search 
> time
> --
>
> Key: SOLR-1895
> URL: https://issues.apache.org/jira/browse/SOLR-1895
> Project: Solr
>  Issue Type: New Feature
>  Components: SearchComponents - other
>Reporter: Karl Wright
>  Labels: document, security, solr
> Fix For: 3.5, 4.0
>
> Attachments: LCFSecurityFilter.java, LCFSecurityFilter.java, 
> LCFSecurityFilter.java, LCFSecurityFilter.java, SOLR-1895-queries.patch, 
> SOLR-1895-queries.patch, SOLR-1895-queries.patch, SOLR-1895-queries.patch, 
> SOLR-1895-service-plugin.patch, SOLR-1895-service-plugin.patch, 
> SOLR-1895.patch, SOLR-1895.patch, SOLR-1895.patch, SOLR-1895.patch, 
> SOLR-1895.patch, SOLR-1895.patch
>
>
> I've written an LCF SearchComponent which filters returned results based on 
> access tokens provided by LCF's authority service.  The component requires 
> you to configure the appropriate authority service URL base, e.g.:
>   
>   
>  name="AuthorityServiceBaseURL">http://localhost:8080/lcf-authority-service
>   
> Also required are the following schema.xml additions:
>
> stored="false" multiValued="true"/>
> stored="false" multiValued="true"/>
> stored="false" multiValued="true"/>
> multiValued="true"/>
> Finally, to tie it into the standard request handler, it seems to need to run 
> last:
>   
> 
>   lcfSecurity
> 
> ...
> I have not set a package for this code.  Nor have I been able to get it 
> reviewed by someone as conversant with Solr as I would prefer.  It is my 
> hope, however, that this module will become part of the standard Solr 1.5 
> suite of search components, since that would tie it in with LCF nicely.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira



-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Assigned] (LUCENE-3471) TestNRTManager test failure

[Michael McCandless (Assigned) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LUCENE-3471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael McCandless reassigned LUCENE-3471:
--

Assignee: Michael McCandless

> TestNRTManager test failure
> ---
>
> Key: LUCENE-3471
> URL: https://issues.apache.org/jira/browse/LUCENE-3471
> Project: Lucene - Java
>  Issue Type: Bug
>Reporter: Robert Muir
>Assignee: Michael McCandless
>
> reproduces for me

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira



-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-3471) TestNRTManager test failure

[Michael McCandless (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-3471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13115719#comment-13115719
 ] 

Michael McCandless commented on LUCENE-3471:


The test is shutting down the executor service before closing the writer... and 
the test had installed a merged segment warmer.  So the merge finished and it 
tried to warm the segment by running searches through IS that was using the ES.

> TestNRTManager test failure
> ---
>
> Key: LUCENE-3471
> URL: https://issues.apache.org/jira/browse/LUCENE-3471
> Project: Lucene - Java
>  Issue Type: Bug
>Reporter: Robert Muir
>Assignee: Michael McCandless
>
> reproduces for me

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira



-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Created] (LUCENE-3472) add back Document.getValues()

[Robert Muir (Created) (JIRA)]

add back Document.getValues()
-

 Key: LUCENE-3472
 URL: https://issues.apache.org/jira/browse/LUCENE-3472
 Project: Lucene - Java
  Issue Type: Bug
Reporter: Robert Muir
 Fix For: 4.0


I'm porting some code to trunk's new Doc/Field apis, and i keep running into 
this pattern:
{noformat}
String[] values = doc.getValues("field");
{noformat}

But with the new apis, this becomes a little too verbose:

{noformat}
IndexableField[] fields = doc.getFields("field");
String[] values = new String[fields.length];
for (int i = 0; i < values.length; i++) {
  values[i] = fields[i].stringValue();
}
{noformat}

I think we should probably add back the sugar api (with the same name) ?


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira



-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Assigned] (LUCENE-3472) add back Document.getValues()

[Michael McCandless (Assigned) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LUCENE-3472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael McCandless reassigned LUCENE-3472:
--

Assignee: Michael McCandless

> add back Document.getValues()
> -
>
> Key: LUCENE-3472
> URL: https://issues.apache.org/jira/browse/LUCENE-3472
> Project: Lucene - Java
>  Issue Type: Bug
>Reporter: Robert Muir
>Assignee: Michael McCandless
> Fix For: 4.0
>
>
> I'm porting some code to trunk's new Doc/Field apis, and i keep running into 
> this pattern:
> {noformat}
> String[] values = doc.getValues("field");
> {noformat}
> But with the new apis, this becomes a little too verbose:
> {noformat}
> IndexableField[] fields = doc.getFields("field");
> String[] values = new String[fields.length];
> for (int i = 0; i < values.length; i++) {
>   values[i] = fields[i].stringValue();
> }
> {noformat}
> I think we should probably add back the sugar api (with the same name) ?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira



-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-3472) add back Document.getValues()

[Michael McCandless (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-3472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13115729#comment-13115729
 ] 

Michael McCandless commented on LUCENE-3472:


+1, let's just add it back?  I'll do that

> add back Document.getValues()
> -
>
> Key: LUCENE-3472
> URL: https://issues.apache.org/jira/browse/LUCENE-3472
> Project: Lucene - Java
>  Issue Type: Bug
>Reporter: Robert Muir
> Fix For: 4.0
>
>
> I'm porting some code to trunk's new Doc/Field apis, and i keep running into 
> this pattern:
> {noformat}
> String[] values = doc.getValues("field");
> {noformat}
> But with the new apis, this becomes a little too verbose:
> {noformat}
> IndexableField[] fields = doc.getFields("field");
> String[] values = new String[fields.length];
> for (int i = 0; i < values.length; i++) {
>   values[i] = fields[i].stringValue();
> }
> {noformat}
> I think we should probably add back the sugar api (with the same name) ?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira



-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Resolved] (LUCENE-3471) TestNRTManager test failure

[Michael McCandless (Resolved) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LUCENE-3471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael McCandless resolved LUCENE-3471.


   Resolution: Fixed
Fix Version/s: 4.0
   3.5

Thank you Charlie!

> TestNRTManager test failure
> ---
>
> Key: LUCENE-3471
> URL: https://issues.apache.org/jira/browse/LUCENE-3471
> Project: Lucene - Java
>  Issue Type: Bug
>Reporter: Robert Muir
>Assignee: Michael McCandless
> Fix For: 3.5, 4.0
>
>
> reproduces for me

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira



-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-2205) Rework of the TermInfosReader class to remove the Terms[], TermInfos[], and the index pointer long[] and create a more memory efficient data structure.

[Michael McCandless (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-2205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13115741#comment-13115741
 ] 

Michael McCandless commented on LUCENE-2205:


bq. I have done the PagedBytes back port already. It was a simple copy of the 
class (assuming that's what you want me to do).

Excellent, I'm glad it was straightforward.  We also need a DataInput impl that 
reads from the PagedBytes... I can help on that if you want.

bq. As for the oal.util.packed package for the packed ints, I think they should 
be modified to work against the DataInput and DataOutput instead of the 
IndexInput and IndexOutput.

I agree -- I committed this to trunk.

> Rework of the TermInfosReader class to remove the Terms[], TermInfos[], and 
> the index pointer long[] and create a more memory efficient data structure.
> ---
>
> Key: LUCENE-2205
> URL: https://issues.apache.org/jira/browse/LUCENE-2205
> Project: Lucene - Java
>  Issue Type: Improvement
>  Components: core/index
> Environment: Java5
>Reporter: Aaron McCurry
>Assignee: Michael McCandless
> Fix For: 3.5
>
> Attachments: RandomAccessTest.java, TermInfosReader.java, 
> TermInfosReaderIndex.java, TermInfosReaderIndexDefault.java, 
> TermInfosReaderIndexSmall.java, lowmemory_w_utf8_encoding.patch, 
> patch-final.txt, rawoutput.txt
>
>
> Basically packing those three arrays into a byte array with an int array as 
> an index offset.  
> The performance benefits are stagering on my test index (of size 6.2 GB, with 
> ~1,000,000 documents and ~175,000,000 terms), the memory needed to load the 
> terminfos into memory were reduced to 17% of there original size.  From 291.5 
> MB to 49.7 MB.  The random access speed has been made better by 1-2%, load 
> time of the segments are ~40% faster as well, and full GC's on my JVM were 
> made 7 times faster.
> I have already performed the work and am offering this code as a patch.  
> Currently all test in the trunk pass with this new code enabled.  I did write 
> a system property switch to allow for the original implementation to be used 
> as well.
> -Dorg.apache.lucene.index.TermInfosReader=default or small
> I have also written a blog about this patch here is the link.
> http://www.nearinfinity.com/blogs/aaron_mccurry/my_first_lucene_patch.html

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira



-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: [jira] [Assigned] (LUCENE-2205) Rework of the TermInfosReader class to remove the Terms[], TermInfos[], and the index pointer long[] and create a more memory efficient data structure.

[Michael McCandless]

Hi Anand,

If you open the issue, here:

https://issues.apache.org/jira/browse/LUCENE-2205

Then click on the "All" tab, under Activity, it will show all attached
patches and comments in order.  Go the end and scroll back to the last
patch and that's the most recent one.

But note that this issue is very much in progress/flux now...

Mike McCandless

http://blog.mikemccandless.com

On Mon, Sep 26, 2011 at 10:39 PM,   wrote:
> Could someone please give me a pointer from where can I download the latest 
> patch .
>
> Thanks & Regards,
> Anand
>
>
> -Original Message-
> From: Michael McCandless [mailto:luc...@mikemccandless.com]
> Sent: 26 September 2011 22:47
> To: dev@lucene.apache.org
> Subject: Re: [jira] [Assigned] (LUCENE-2205) Rework of the TermInfosReader 
> class to remove the Terms[], TermInfos[], and the index pointer long[] and 
> create a more memory efficient data structure.
>
> Is it possible you are using an old patch on the issue?
>
> The newer patch worked very recently for me on 3.x.
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
> On Mon, Sep 26, 2011 at 3:53 AM,   wrote:
>> Hi,
>>
>> I am using solr 3.4.0 and I want to use this patch. There is a compilation 
>> error in 'TermInfosReader' class in the patch as it is not able to find 
>> following classes:
>>
>> import org.apache.lucene.util.cache.Cache;
>> import org.apache.lucene.util.cache.SimpleLRUCache;
>>
>> On google search I found that these classes were present in 
>> 'lucene-core-2.4.1' whereas solr-3.4.0 has 'lucene-core-3.4.0 included in it 
>> which does have above classes.
>>
>> Thanks & Regards,
>> Anand
>>
>>
>> Anand Nigam
>> RBS Global Banking & Markets
>> Office: +91 124 492 5506
>>
>> -Original Message-
>> From: Michael McCandless (JIRA) [mailto:j...@apache.org]
>> Sent: 16 September 2011 01:03
>> To: dev@lucene.apache.org
>> Subject: [jira] [Assigned] (LUCENE-2205) Rework of the TermInfosReader class 
>> to remove the Terms[], TermInfos[], and the index pointer long[] and create 
>> a more memory efficient data structure.
>>
>>
>>     [
>> https://issues.apache.org/jira/browse/LUCENE-2205?page=com.atlassian.j
>> ira.plugin.system.issuetabpanels:all-tabpanel ]
>>
>> Michael McCandless reassigned LUCENE-2205:
>> --
>>
>>    Assignee: Michael McCandless
>>
>>> Rework of the TermInfosReader class to remove the Terms[], TermInfos[], and 
>>> the index pointer long[] and create a more memory efficient data structure.
>>> -
>>> -
>>> -
>>> -
>>> ---
>>>
>>>                 Key: LUCENE-2205
>>>                 URL:
>>> https://issues.apache.org/jira/browse/LUCENE-2205
>>>             Project: Lucene - Java
>>>          Issue Type: Improvement
>>>          Components: core/index
>>>         Environment: Java5
>>>            Reporter: Aaron McCurry
>>>            Assignee: Michael McCandless
>>>             Fix For: 3.5
>>>
>>>         Attachments: RandomAccessTest.java, TermInfosReader.java,
>>> TermInfosReaderIndex.java, TermInfosReaderIndexDefault.java,
>>> TermInfosReaderIndexSmall.java, patch-final.txt, rawoutput.txt
>>>
>>>
>>> Basically packing those three arrays into a byte array with an int array as 
>>> an index offset.
>>> The performance benefits are stagering on my test index (of size 6.2 GB, 
>>> with ~1,000,000 documents and ~175,000,000 terms), the memory needed to 
>>> load the terminfos into memory were reduced to 17% of there original size.  
>>> From 291.5 MB to 49.7 MB.  The random access speed has been made better by 
>>> 1-2%, load time of the segments are ~40% faster as well, and full GC's on 
>>> my JVM were made 7 times faster.
>>> I have already performed the work and am offering this code as a patch.  
>>> Currently all test in the trunk pass with this new code enabled.  I did 
>>> write a system property switch to allow for the original implementation to 
>>> be used as well.
>>> -Dorg.apache.lucene.index.TermInfosReader=default or small I have
>>> also written a blog about this patch here is the link.
>>> http://www.nearinfinity.com/blogs/aaron_mccurry/my_first_lucene_patch.
>>> html
>>
>> --
>> This message is automatically generated by JIRA.
>> For more information on JIRA, see:
>> http://www.atlassian.com/software/jira
>>
>>
>>
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For
>> additional commands, e-mail: dev-h...@lucene.apache.org
>>
>>
>> **
>> * The Royal Bank of Scotland plc. Registered in Scotland
>> No 90312.
>> Registered Office: 36 St Andrew Square, Edinburgh EH2 2YB.
>> Authorised and regulated by the Financial Services Authority. The
>> Royal Bank of Scotland N.V. is authorised and regulated by the De
>> Neder

[jira] [Resolved] (LUCENE-3472) add back Document.getValues()

[Michael McCandless (Resolved) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LUCENE-3472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael McCandless resolved LUCENE-3472.


Resolution: Fixed

> add back Document.getValues()
> -
>
> Key: LUCENE-3472
> URL: https://issues.apache.org/jira/browse/LUCENE-3472
> Project: Lucene - Java
>  Issue Type: Bug
>Reporter: Robert Muir
>Assignee: Michael McCandless
> Fix For: 4.0
>
>
> I'm porting some code to trunk's new Doc/Field apis, and i keep running into 
> this pattern:
> {noformat}
> String[] values = doc.getValues("field");
> {noformat}
> But with the new apis, this becomes a little too verbose:
> {noformat}
> IndexableField[] fields = doc.getFields("field");
> String[] values = new String[fields.length];
> for (int i = 0; i < values.length; i++) {
>   values[i] = fields[i].stringValue();
> }
> {noformat}
> I think we should probably add back the sugar api (with the same name) ?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira



-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Resolved] (LUCENE-3469) move DocumentStoredFieldsVisitor to o.a.l.document

[Robert Muir (Resolved) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LUCENE-3469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Muir resolved LUCENE-3469.
-

Resolution: Fixed
  Assignee: Robert Muir

> move DocumentStoredFieldsVisitor to o.a.l.document
> --
>
> Key: LUCENE-3469
> URL: https://issues.apache.org/jira/browse/LUCENE-3469
> Project: Lucene - Java
>  Issue Type: Task
>Reporter: Robert Muir
>Assignee: Robert Muir
> Fix For: 4.0
>
> Attachments: LUCENE-3469.patch
>
>
> when examining the changes to the field/document API, i noticed this class 
> was in o.a.l.index
> I think it should be in o.a.l.document, its more intuitive packaging

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira



-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

trunk test failure (1317144301)

[Charlie Cron]

A test failure occurred running the tests.

revert! revert! revert!

You can see the entire build log at 
http://sierranevada.servebeer.com/1317144301.log

Thanks,
Charlie Cron


-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-2752) leader-per-shard

[Mark Miller (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-2752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13115771#comment-13115771
 ] 

Mark Miller commented on SOLR-2752:
---

While working on the tests for this I ran into and filed a locale bug in zk: 
https://issues.apache.org/jira/browse/ZOOKEEPER-1206

> leader-per-shard
> 
>
> Key: SOLR-2752
> URL: https://issues.apache.org/jira/browse/SOLR-2752
> Project: Solr
>  Issue Type: Sub-task
>  Components: SolrCloud
>Reporter: Yonik Seeley
>Assignee: Mark Miller
> Fix For: 4.0
>
> Attachments: SOLR-2752.patch, SOLR-2752.patch, SOLR-2752.patch, 
> SOLR-2752.patch, SOLR-2752.patch
>
>
> We need to add metadata into zookeeper about who is the leader for each 
> shard, and have some kind of leader election.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira



-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (LUCENE-1536) if a filter can support random access API, we should use it

[Michael McCandless (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LUCENE-1536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael McCandless updated LUCENE-1536:
---

Attachment: LUCENE-1536.patch

New patch, fixing a few failing tests, adding a couple comments.  I
downgraded the 2 nocommits in LTC to TODOs, and removed the other one
(false is correct default for those two?).

For DocIdSet, can we nuke getRandomAccessBits?  Ie, if
supportRandomAccess() returns true, then we can cast the instance to
Bits?  Maybe we should rename supportRandomAccess to useRandomAccess?
(Ie, it may support it, but we only want to use random access when the
filter is dense enough).

I changed MTQWF's and CachingWrapperFilter's threshold to a double
percent (of the reader's maxDoc) instead, default 1.0%.

Hmm FieldCacheRangeFilter never enables random access... not sure
how/when we should compute that.

I think we are getting close!


> if a filter can support random access API, we should use it
> ---
>
> Key: LUCENE-1536
> URL: https://issues.apache.org/jira/browse/LUCENE-1536
> Project: Lucene - Java
>  Issue Type: Improvement
>  Components: core/search
>Affects Versions: 2.4
>Reporter: Michael McCandless
>Assignee: Michael McCandless
>Priority: Minor
>  Labels: gsoc2011, lucene-gsoc-11, mentor
> Fix For: 4.0
>
> Attachments: CachedFilterIndexReader.java, LUCENE-1536.patch, 
> LUCENE-1536.patch, LUCENE-1536.patch, LUCENE-1536.patch, LUCENE-1536.patch, 
> LUCENE-1536.patch, LUCENE-1536.patch, LUCENE-1536.patch, LUCENE-1536.patch, 
> LUCENE-1536.patch, LUCENE-1536.patch
>
>
> I ran some performance tests, comparing applying a filter via
> random-access API instead of current trunk's iterator API.
> This was inspired by LUCENE-1476, where we realized deletions should
> really be implemented just like a filter, but then in testing found
> that switching deletions to iterator was a very sizable performance
> hit.
> Some notes on the test:
>   * Index is first 2M docs of Wikipedia.  Test machine is Mac OS X
> 10.5.6, quad core Intel CPU, 6 GB RAM, java 1.6.0_07-b06-153.
>   * I test across multiple queries.  1-X means an OR query, eg 1-4
> means 1 OR 2 OR 3 OR 4, whereas +1-4 is an AND query, ie 1 AND 2
> AND 3 AND 4.  "u s" means "united states" (phrase search).
>   * I test with multiple filter densities (0, 1, 2, 5, 10, 25, 75, 90,
> 95, 98, 99, 99.9 (filter is non-null but all bits are set),
> 100 (filter=null, control)).
>   * Method high means I use random-access filter API in
> IndexSearcher's main loop.  Method low means I use random-access
> filter API down in SegmentTermDocs (just like deleted docs
> today).
>   * Baseline (QPS) is current trunk, where filter is applied as iterator up
> "high" (ie in IndexSearcher's search loop).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira



-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-3467) Cut over numeric docvalues to fixed straight bytes

[Simon Willnauer (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-3467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13115796#comment-13115796
 ] 

Simon Willnauer commented on LUCENE-3467:
-

I plan to commit this tomorrow if nobody objects. Its basically an internal 
refactoring except of the additions to BytesRef

> Cut over numeric docvalues to fixed straight bytes
> --
>
> Key: LUCENE-3467
> URL: https://issues.apache.org/jira/browse/LUCENE-3467
> Project: Lucene - Java
>  Issue Type: Improvement
>  Components: core/index
>Affects Versions: 4.0
>Reporter: Simon Willnauer
> Fix For: 4.0
>
> Attachments: LUCENE-3467.patch
>
>
> Currently numeric docvalues types are encoded and stored individually which 
> creates massive duplication of writing / indexing code. Yet, almost all of 
> them (except packed ints) are essentially a fixed straight bytes variant. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira



-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-2202) Money FieldType

[Greg Fodor (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-2202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13115803#comment-13115803
 ] 

Greg Fodor commented on SOLR-2202:
--

Hey Jan, awesome! The latest patch is the implementation we have been running 
in production for some time now. (We are on Solr trunk, however, and there were 
some small tweaks necessary to get it to build there.) Once enhancement we are 
likely to make is the ability to reload the currency.xml file without reloading 
the Solr cores. Other than that, I think this should be good to go. 

It's been quite a while since I have looked at this patch -- please let me know 
if you find that it does not merge in or test properly with the current release 
of Solr. Also, let me know what I need to do documentation wise and so on. 

Thanks a bunch, excited to see this come together!

> Money FieldType
> ---
>
> Key: SOLR-2202
> URL: https://issues.apache.org/jira/browse/SOLR-2202
> Project: Solr
>  Issue Type: New Feature
>  Components: Schema and Analysis
>Affects Versions: 1.5
>Reporter: Greg Fodor
> Attachments: SOLR-2022-solr-3.patch, SOLR-2202-lucene-1.patch, 
> SOLR-2202-solr-1.patch, SOLR-2202-solr-2.patch, SOLR-2202-solr-4.patch, 
> SOLR-2202-solr-5.patch, SOLR-2202-solr-6.patch, SOLR-2202-solr-7.patch, 
> SOLR-2202-solr-8.patch, SOLR-2202-solr-9.patch
>
>
> Attached please find patches to add support for monetary values to 
> Solr/Lucene with query-time currency conversion. The following features are 
> supported:
> - Point queries (ex: "price:4.00USD")
> - Range quries (ex: "price:[$5.00 TO $10.00]")
> - Sorting.
> - Currency parsing by either currency code or symbol.
> - Symmetric & Asymmetric exchange rates. (Asymmetric exchange rates are 
> useful if there are fees associated with exchanging the currency.)
> At indexing time, money fields can be indexed in a native currency. For 
> example, if a product on an e-commerce site is listed in Euros, indexing the 
> price field as "10.00EUR" will index it appropriately. By altering the 
> currency.xml file, the sorting and querying against Solr can take into 
> account fluctuations in currency exchange rates without having to re-index 
> the documents.
> The new "money" field type is a polyfield which indexes two fields, one which 
> contains the amount of the value and another which contains the currency code 
> or symbol. The currency metadata (names, symbols, codes, and exchange rates) 
> are expected to be in an xml file which is pointed to by the field type 
> declaration in the schema.xml.
> The current patch is factored such that Money utility functions and 
> configuration metadata lie in Lucene (see MoneyUtil and CurrencyConfig), 
> while the MoneyType and MoneyValueSource lie in Solr. This was meant to 
> mirror the work being done on the spacial field types.
> This patch has not yet been deployed to production but will be getting used 
> to power the international search capabilities of the search engine at Etsy.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira



-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Issue Comment Edited] (SOLR-2202) Money FieldType

[Greg Fodor (Issue Comment Edited) (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-2202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13115803#comment-13115803
 ] 

Greg Fodor edited comment on SOLR-2202 at 9/27/11 6:59 PM:
---

Hey Jan, awesome! The latest patch is the implementation we have been running 
in production for some time now. (We are on Solr trunk, however, and there were 
some small tweaks necessary to get it to build there.) One enhancement we are 
likely to make is the ability to reload the currency.xml file without reloading 
the Solr cores. If we can live without that feature, I think this should be 
good to go. 

It's been quite a while since I have looked at this patch -- please let me know 
if you find that it does not merge in or test properly with the current release 
of Solr. Also, let me know what I need to do documentation wise and so on. 

Thanks a bunch, excited to see this come together!

  was (Author: gfodor):
Hey Jan, awesome! The latest patch is the implementation we have been 
running in production for some time now. (We are on Solr trunk, however, and 
there were some small tweaks necessary to get it to build there.) Once 
enhancement we are likely to make is the ability to reload the currency.xml 
file without reloading the Solr cores. Other than that, I think this should be 
good to go. 

It's been quite a while since I have looked at this patch -- please let me know 
if you find that it does not merge in or test properly with the current release 
of Solr. Also, let me know what I need to do documentation wise and so on. 

Thanks a bunch, excited to see this come together!
  
> Money FieldType
> ---
>
> Key: SOLR-2202
> URL: https://issues.apache.org/jira/browse/SOLR-2202
> Project: Solr
>  Issue Type: New Feature
>  Components: Schema and Analysis
>Affects Versions: 1.5
>Reporter: Greg Fodor
> Attachments: SOLR-2022-solr-3.patch, SOLR-2202-lucene-1.patch, 
> SOLR-2202-solr-1.patch, SOLR-2202-solr-2.patch, SOLR-2202-solr-4.patch, 
> SOLR-2202-solr-5.patch, SOLR-2202-solr-6.patch, SOLR-2202-solr-7.patch, 
> SOLR-2202-solr-8.patch, SOLR-2202-solr-9.patch
>
>
> Attached please find patches to add support for monetary values to 
> Solr/Lucene with query-time currency conversion. The following features are 
> supported:
> - Point queries (ex: "price:4.00USD")
> - Range quries (ex: "price:[$5.00 TO $10.00]")
> - Sorting.
> - Currency parsing by either currency code or symbol.
> - Symmetric & Asymmetric exchange rates. (Asymmetric exchange rates are 
> useful if there are fees associated with exchanging the currency.)
> At indexing time, money fields can be indexed in a native currency. For 
> example, if a product on an e-commerce site is listed in Euros, indexing the 
> price field as "10.00EUR" will index it appropriately. By altering the 
> currency.xml file, the sorting and querying against Solr can take into 
> account fluctuations in currency exchange rates without having to re-index 
> the documents.
> The new "money" field type is a polyfield which indexes two fields, one which 
> contains the amount of the value and another which contains the currency code 
> or symbol. The currency metadata (names, symbols, codes, and exchange rates) 
> are expected to be in an xml file which is pointed to by the field type 
> declaration in the schema.xml.
> The current patch is factored such that Money utility functions and 
> configuration metadata lie in Lucene (see MoneyUtil and CurrencyConfig), 
> while the MoneyType and MoneyValueSource lie in Solr. This was meant to 
> mirror the work being done on the spacial field types.
> This patch has not yet been deployed to production but will be getting used 
> to power the international search capabilities of the search engine at Etsy.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira



-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-1536) if a filter can support random access API, we should use it

[Uwe Schindler (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-1536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13115806#comment-13115806
 ] 

Uwe Schindler commented on LUCENE-1536:
---

Wouldn't it make sense for FCRF to always return random access=true? This 
filter has a very ineffective DISI, as it has to check each doc using the match 
method, so random access is much better for this one (maps to current 
matchDoc-method, which should be renamed to a simple Bits.get(int docId)). 
What's the sense to use a DISI for this one?

> if a filter can support random access API, we should use it
> ---
>
> Key: LUCENE-1536
> URL: https://issues.apache.org/jira/browse/LUCENE-1536
> Project: Lucene - Java
>  Issue Type: Improvement
>  Components: core/search
>Affects Versions: 2.4
>Reporter: Michael McCandless
>Assignee: Michael McCandless
>Priority: Minor
>  Labels: gsoc2011, lucene-gsoc-11, mentor
> Fix For: 4.0
>
> Attachments: CachedFilterIndexReader.java, LUCENE-1536.patch, 
> LUCENE-1536.patch, LUCENE-1536.patch, LUCENE-1536.patch, LUCENE-1536.patch, 
> LUCENE-1536.patch, LUCENE-1536.patch, LUCENE-1536.patch, LUCENE-1536.patch, 
> LUCENE-1536.patch, LUCENE-1536.patch
>
>
> I ran some performance tests, comparing applying a filter via
> random-access API instead of current trunk's iterator API.
> This was inspired by LUCENE-1476, where we realized deletions should
> really be implemented just like a filter, but then in testing found
> that switching deletions to iterator was a very sizable performance
> hit.
> Some notes on the test:
>   * Index is first 2M docs of Wikipedia.  Test machine is Mac OS X
> 10.5.6, quad core Intel CPU, 6 GB RAM, java 1.6.0_07-b06-153.
>   * I test across multiple queries.  1-X means an OR query, eg 1-4
> means 1 OR 2 OR 3 OR 4, whereas +1-4 is an AND query, ie 1 AND 2
> AND 3 AND 4.  "u s" means "united states" (phrase search).
>   * I test with multiple filter densities (0, 1, 2, 5, 10, 25, 75, 90,
> 95, 98, 99, 99.9 (filter is non-null but all bits are set),
> 100 (filter=null, control)).
>   * Method high means I use random-access filter API in
> IndexSearcher's main loop.  Method low means I use random-access
> filter API down in SegmentTermDocs (just like deleted docs
> today).
>   * Baseline (QPS) is current trunk, where filter is applied as iterator up
> "high" (ie in IndexSearcher's search loop).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira



-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-2202) Money FieldType

[Greg Fodor (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-2202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13115807#comment-13115807
 ] 

Greg Fodor commented on SOLR-2202:
--

I will actually take a crack at merging this into Solr's latest release today. 
My guess is since it has been so long there are surely a few issues.

> Money FieldType
> ---
>
> Key: SOLR-2202
> URL: https://issues.apache.org/jira/browse/SOLR-2202
> Project: Solr
>  Issue Type: New Feature
>  Components: Schema and Analysis
>Affects Versions: 1.5
>Reporter: Greg Fodor
> Attachments: SOLR-2022-solr-3.patch, SOLR-2202-lucene-1.patch, 
> SOLR-2202-solr-1.patch, SOLR-2202-solr-2.patch, SOLR-2202-solr-4.patch, 
> SOLR-2202-solr-5.patch, SOLR-2202-solr-6.patch, SOLR-2202-solr-7.patch, 
> SOLR-2202-solr-8.patch, SOLR-2202-solr-9.patch
>
>
> Attached please find patches to add support for monetary values to 
> Solr/Lucene with query-time currency conversion. The following features are 
> supported:
> - Point queries (ex: "price:4.00USD")
> - Range quries (ex: "price:[$5.00 TO $10.00]")
> - Sorting.
> - Currency parsing by either currency code or symbol.
> - Symmetric & Asymmetric exchange rates. (Asymmetric exchange rates are 
> useful if there are fees associated with exchanging the currency.)
> At indexing time, money fields can be indexed in a native currency. For 
> example, if a product on an e-commerce site is listed in Euros, indexing the 
> price field as "10.00EUR" will index it appropriately. By altering the 
> currency.xml file, the sorting and querying against Solr can take into 
> account fluctuations in currency exchange rates without having to re-index 
> the documents.
> The new "money" field type is a polyfield which indexes two fields, one which 
> contains the amount of the value and another which contains the currency code 
> or symbol. The currency metadata (names, symbols, codes, and exchange rates) 
> are expected to be in an xml file which is pointed to by the field type 
> declaration in the schema.xml.
> The current patch is factored such that Money utility functions and 
> configuration metadata lie in Lucene (see MoneyUtil and CurrencyConfig), 
> while the MoneyType and MoneyValueSource lie in Solr. This was meant to 
> mirror the work being done on the spacial field types.
> This patch has not yet been deployed to production but will be getting used 
> to power the international search capabilities of the search engine at Etsy.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira



-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-1536) if a filter can support random access API, we should use it

[Uwe Schindler (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-1536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13115811#comment-13115811
 ] 

Uwe Schindler commented on LUCENE-1536:
---

Why does CachingWrapperFilter refers to OpenBitSetDISI again? It should only 
use FixedBitSet as it already has al DISI methods?

> if a filter can support random access API, we should use it
> ---
>
> Key: LUCENE-1536
> URL: https://issues.apache.org/jira/browse/LUCENE-1536
> Project: Lucene - Java
>  Issue Type: Improvement
>  Components: core/search
>Affects Versions: 2.4
>Reporter: Michael McCandless
>Assignee: Michael McCandless
>Priority: Minor
>  Labels: gsoc2011, lucene-gsoc-11, mentor
> Fix For: 4.0
>
> Attachments: CachedFilterIndexReader.java, LUCENE-1536.patch, 
> LUCENE-1536.patch, LUCENE-1536.patch, LUCENE-1536.patch, LUCENE-1536.patch, 
> LUCENE-1536.patch, LUCENE-1536.patch, LUCENE-1536.patch, LUCENE-1536.patch, 
> LUCENE-1536.patch, LUCENE-1536.patch
>
>
> I ran some performance tests, comparing applying a filter via
> random-access API instead of current trunk's iterator API.
> This was inspired by LUCENE-1476, where we realized deletions should
> really be implemented just like a filter, but then in testing found
> that switching deletions to iterator was a very sizable performance
> hit.
> Some notes on the test:
>   * Index is first 2M docs of Wikipedia.  Test machine is Mac OS X
> 10.5.6, quad core Intel CPU, 6 GB RAM, java 1.6.0_07-b06-153.
>   * I test across multiple queries.  1-X means an OR query, eg 1-4
> means 1 OR 2 OR 3 OR 4, whereas +1-4 is an AND query, ie 1 AND 2
> AND 3 AND 4.  "u s" means "united states" (phrase search).
>   * I test with multiple filter densities (0, 1, 2, 5, 10, 25, 75, 90,
> 95, 98, 99, 99.9 (filter is non-null but all bits are set),
> 100 (filter=null, control)).
>   * Method high means I use random-access filter API in
> IndexSearcher's main loop.  Method low means I use random-access
> filter API down in SegmentTermDocs (just like deleted docs
> today).
>   * Baseline (QPS) is current trunk, where filter is applied as iterator up
> "high" (ie in IndexSearcher's search loop).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira



-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-2202) Money FieldType

[Steven Rowe (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-2202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13115837#comment-13115837
 ] 

Steven Rowe commented on SOLR-2202:
---

Greg, there have been structural changes since your last patch; you can read 
about it in SOLR-2452.

I committed a Perl script that can convert a patch against the old structure 
into a patch against the current structure.  You can find it in your working 
copy at: {noformat}dev-tools/scripts/SOLR-2452.patch.hack.pl{noformat}

There is an example of its use in a comment at the top of the script.




> Money FieldType
> ---
>
> Key: SOLR-2202
> URL: https://issues.apache.org/jira/browse/SOLR-2202
> Project: Solr
>  Issue Type: New Feature
>  Components: Schema and Analysis
>Affects Versions: 1.5
>Reporter: Greg Fodor
> Attachments: SOLR-2022-solr-3.patch, SOLR-2202-lucene-1.patch, 
> SOLR-2202-solr-1.patch, SOLR-2202-solr-2.patch, SOLR-2202-solr-4.patch, 
> SOLR-2202-solr-5.patch, SOLR-2202-solr-6.patch, SOLR-2202-solr-7.patch, 
> SOLR-2202-solr-8.patch, SOLR-2202-solr-9.patch
>
>
> Attached please find patches to add support for monetary values to 
> Solr/Lucene with query-time currency conversion. The following features are 
> supported:
> - Point queries (ex: "price:4.00USD")
> - Range quries (ex: "price:[$5.00 TO $10.00]")
> - Sorting.
> - Currency parsing by either currency code or symbol.
> - Symmetric & Asymmetric exchange rates. (Asymmetric exchange rates are 
> useful if there are fees associated with exchanging the currency.)
> At indexing time, money fields can be indexed in a native currency. For 
> example, if a product on an e-commerce site is listed in Euros, indexing the 
> price field as "10.00EUR" will index it appropriately. By altering the 
> currency.xml file, the sorting and querying against Solr can take into 
> account fluctuations in currency exchange rates without having to re-index 
> the documents.
> The new "money" field type is a polyfield which indexes two fields, one which 
> contains the amount of the value and another which contains the currency code 
> or symbol. The currency metadata (names, symbols, codes, and exchange rates) 
> are expected to be in an xml file which is pointed to by the field type 
> declaration in the schema.xml.
> The current patch is factored such that Money utility functions and 
> configuration metadata lie in Lucene (see MoneyUtil and CurrencyConfig), 
> while the MoneyType and MoneyValueSource lie in Solr. This was meant to 
> mirror the work being done on the spacial field types.
> This patch has not yet been deployed to production but will be getting used 
> to power the international search capabilities of the search engine at Etsy.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira



-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

trunk test failure (1317153061)

[Charlie Cron]

A test failure occurred running the tests.

revert! revert! revert!

You can see the entire build log at 
http://sierranevada.servebeer.com/1317153061.log

Thanks,
Charlie Cron


-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-3445) Add SearcherManager, to manage IndexSearcher usage across threads and reopens

[Greg Steffensen (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-3445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13115900#comment-13115900
 ] 

Greg Steffensen commented on LUCENE-3445:
-

What are the different use cases for this and NRTManager?

> Add SearcherManager, to manage IndexSearcher usage across threads and reopens
> -
>
> Key: LUCENE-3445
> URL: https://issues.apache.org/jira/browse/LUCENE-3445
> Project: Lucene - Java
>  Issue Type: Improvement
>Reporter: Michael McCandless
>Assignee: Michael McCandless
> Fix For: 3.5, 4.0
>
> Attachments: LUCENE-3445.patch, LUCENE-3445.patch
>
>
> This is a simple helper class I wrote for Lucene in Action 2nd ed.
> I'd like to commit under Lucene (contrib/misc).
> It simplifies using & reopening an IndexSearcher across multiple
> threads, by using IndexReader's ref counts to know when it's safe
> to close the reader.
> In the process I also factored out a test base class for tests that
> want to make lots of simultaneous indexing and searching threads, and
> fixed TestNRTThreads (core), TestNRTManager (contrib/misc) and the new
> TestSearcherManager (contrib/misc) to use this base class.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira



-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Created] (LUCENE-3473) CheckIndex should verify numUniqueTerms == recomputedNumUniqueTerms

[Robert Muir (Created) (JIRA)]

CheckIndex should verify numUniqueTerms == recomputedNumUniqueTerms
---

 Key: LUCENE-3473
 URL: https://issues.apache.org/jira/browse/LUCENE-3473
 Project: Lucene - Java
  Issue Type: Improvement
Affects Versions: 3.4, 4.0
Reporter: Robert Muir


Just glancing at the code it seems to sorta do this check, but only in the 
hasOrd==true case maybe (which seems to be testing something else)?

It would be nice to verify this also for terms dicts that dont support ord.

we should add explicit checks per-field in 4.x, and for-all-fields in 3.x and 
preflex

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira



-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (LUCENE-3445) Add SearcherManager, to manage IndexSearcher usage across threads and reopens

[Simon Willnauer (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LUCENE-3445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Simon Willnauer updated LUCENE-3445:


Attachment: LUCENE-3445.patch

I have a couple of smallish improvements to SearcherManager making the hottest 
method in SM non-blocking. I added a tryIncRef to index reader that makes is 
easier to work with IR refcounts in a non-blocking fashion. I also renamed 
SM#get to SM#acquire which seems to be more clear about what happens here (also 
kind of stronger than get - so you'd better release it). 



> Add SearcherManager, to manage IndexSearcher usage across threads and reopens
> -
>
> Key: LUCENE-3445
> URL: https://issues.apache.org/jira/browse/LUCENE-3445
> Project: Lucene - Java
>  Issue Type: Improvement
>Reporter: Michael McCandless
>Assignee: Michael McCandless
> Fix For: 3.5, 4.0
>
> Attachments: LUCENE-3445.patch, LUCENE-3445.patch, LUCENE-3445.patch
>
>
> This is a simple helper class I wrote for Lucene in Action 2nd ed.
> I'd like to commit under Lucene (contrib/misc).
> It simplifies using & reopening an IndexSearcher across multiple
> threads, by using IndexReader's ref counts to know when it's safe
> to close the reader.
> In the process I also factored out a test base class for tests that
> want to make lots of simultaneous indexing and searching threads, and
> fixed TestNRTThreads (core), TestNRTManager (contrib/misc) and the new
> TestSearcherManager (contrib/misc) to use this base class.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira



-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-3445) Add SearcherManager, to manage IndexSearcher usage across threads and reopens

[Michael McCandless (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-3445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13115957#comment-13115957
 ] 

Michael McCandless commented on LUCENE-3445:


+1 -- changes look great!  Nice to have .acquire be lock free.

> Add SearcherManager, to manage IndexSearcher usage across threads and reopens
> -
>
> Key: LUCENE-3445
> URL: https://issues.apache.org/jira/browse/LUCENE-3445
> Project: Lucene - Java
>  Issue Type: Improvement
>Reporter: Michael McCandless
>Assignee: Michael McCandless
> Fix For: 3.5, 4.0
>
> Attachments: LUCENE-3445.patch, LUCENE-3445.patch, LUCENE-3445.patch
>
>
> This is a simple helper class I wrote for Lucene in Action 2nd ed.
> I'd like to commit under Lucene (contrib/misc).
> It simplifies using & reopening an IndexSearcher across multiple
> threads, by using IndexReader's ref counts to know when it's safe
> to close the reader.
> In the process I also factored out a test base class for tests that
> want to make lots of simultaneous indexing and searching threads, and
> fixed TestNRTThreads (core), TestNRTManager (contrib/misc) and the new
> TestSearcherManager (contrib/misc) to use this base class.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira



-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-1895) ManifoldCF SearchComponent plugin for enforcing ManifoldCF security at search time

[Alexey Serba (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-1895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13115961#comment-13115961
 ] 

Alexey Serba commented on SOLR-1895:


Karl, could you please clarify 
* Does ManifoldCF security component support arbitrary security model or it's 
just about Windows AD?
* What's _token_share_? How is it different from _token_document_?
* It seems that under the hood it generates "deny overrides allow" type of 
query. But I'm not sure that it is always the case, because afaiu the order of 
Access Control Entries (ACE) in Access Control List (ACL) is important.

> ManifoldCF SearchComponent plugin for enforcing ManifoldCF security at search 
> time
> --
>
> Key: SOLR-1895
> URL: https://issues.apache.org/jira/browse/SOLR-1895
> Project: Solr
>  Issue Type: New Feature
>  Components: SearchComponents - other
>Reporter: Karl Wright
>  Labels: document, security, solr
> Fix For: 3.5, 4.0
>
> Attachments: LCFSecurityFilter.java, LCFSecurityFilter.java, 
> LCFSecurityFilter.java, LCFSecurityFilter.java, SOLR-1895-queries.patch, 
> SOLR-1895-queries.patch, SOLR-1895-queries.patch, SOLR-1895-queries.patch, 
> SOLR-1895-service-plugin.patch, SOLR-1895-service-plugin.patch, 
> SOLR-1895.patch, SOLR-1895.patch, SOLR-1895.patch, SOLR-1895.patch, 
> SOLR-1895.patch, SOLR-1895.patch
>
>
> I've written an LCF SearchComponent which filters returned results based on 
> access tokens provided by LCF's authority service.  The component requires 
> you to configure the appropriate authority service URL base, e.g.:
>   
>   
>  name="AuthorityServiceBaseURL">http://localhost:8080/lcf-authority-service
>   
> Also required are the following schema.xml additions:
>
> stored="false" multiValued="true"/>
> stored="false" multiValued="true"/>
> stored="false" multiValued="true"/>
> multiValued="true"/>
> Finally, to tie it into the standard request handler, it seems to need to run 
> last:
>   
> 
>   lcfSecurity
> 
> ...
> I have not set a package for this code.  Nor have I been able to get it 
> reviewed by someone as conversant with Solr as I would prefer.  It is my 
> hope, however, that this module will become part of the standard Solr 1.5 
> suite of search components, since that would tie it in with LCF nicely.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira



-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-1895) ManifoldCF SearchComponent plugin for enforcing ManifoldCF security at search time

[Karl Wright (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-1895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13115967#comment-13115967
 ] 

Karl Wright commented on SOLR-1895:
---

bq.* Does ManifoldCF security component support arbitrary security model or 
it's just about Windows AD?

The short answer is, "no"; it is not just about Windows AD.  ManifoldCF 
supports AD, FileNet, Documentum, LiveLink, and Meridio authorization as well, 
and others get added periodically.

It might help for you to read up on ManifoldCF.  There's a book 
(http://www.manning.com/wright); you'd want to read Chapters 4 and 8.  I can 
email them to you if you give me a preferred email address. I will also be 
presenting in Barcelona in a month on this topic.

bq.* What's token_share? How is it different from token_document?

In order to be able to support AD, ManifoldCF allows multiple levels of token.  
There are allow/deny tokens for "share" level (which correspond in AD to 
Windows shares), "document" level (which correspond to windows documents), and 
also N folder levels (which don't appear in Solr because the ManifoldCF Solr 
output connector won't accept documents that have security set on those).  
Share and document security operate completely independently of one another, 
but a document cannot be viewed unless it is allowed (and not denied) on BOTH 
levels.

bq.* It seems that under the hood it generates "deny overrides allow" type 
of query. But I'm not sure that it is always the case, because afaiu the order 
of Access Control Entries (ACE) in Access Control List (ACL) is important.

That is actually not the case.  This was tested extensively at MetaCarta 
in-house; there is no order dependency in AD or any other ManifoldCF-supported 
repository.


> ManifoldCF SearchComponent plugin for enforcing ManifoldCF security at search 
> time
> --
>
> Key: SOLR-1895
> URL: https://issues.apache.org/jira/browse/SOLR-1895
> Project: Solr
>  Issue Type: New Feature
>  Components: SearchComponents - other
>Reporter: Karl Wright
>  Labels: document, security, solr
> Fix For: 3.5, 4.0
>
> Attachments: LCFSecurityFilter.java, LCFSecurityFilter.java, 
> LCFSecurityFilter.java, LCFSecurityFilter.java, SOLR-1895-queries.patch, 
> SOLR-1895-queries.patch, SOLR-1895-queries.patch, SOLR-1895-queries.patch, 
> SOLR-1895-service-plugin.patch, SOLR-1895-service-plugin.patch, 
> SOLR-1895.patch, SOLR-1895.patch, SOLR-1895.patch, SOLR-1895.patch, 
> SOLR-1895.patch, SOLR-1895.patch
>
>
> I've written an LCF SearchComponent which filters returned results based on 
> access tokens provided by LCF's authority service.  The component requires 
> you to configure the appropriate authority service URL base, e.g.:
>   
>   
>  name="AuthorityServiceBaseURL">http://localhost:8080/lcf-authority-service
>   
> Also required are the following schema.xml additions:
>
> stored="false" multiValued="true"/>
> stored="false" multiValued="true"/>
> stored="false" multiValued="true"/>
> multiValued="true"/>
> Finally, to tie it into the standard request handler, it seems to need to run 
> last:
>   
> 
>   lcfSecurity
> 
> ...
> I have not set a package for this code.  Nor have I been able to get it 
> reviewed by someone as conversant with Solr as I would prefer.  It is my 
> hope, however, that this module will become part of the standard Solr 1.5 
> suite of search components, since that would tie it in with LCF nicely.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira



-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

trunk test failure (1317159301)

[Charlie Cron]

A test failure occurred running the tests.

revert! revert! revert!

You can see the entire build log at 
http://sierranevada.servebeer.com/1317159301.log

Thanks,
Charlie Cron


-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Solr should provide an option to show only most relevant facet values

[Burton-West, Tom]

Hello all,

This post is getting no replies after several days on the Solr user list, so I 
thought I would rewrite it as a question about a possible feature for Solr.

In our use case we have a large number of documents and several facets such as 
Author and Subject, that have a very large number of values.  Since we index 
the full text of nearly 10 million books, it is easy for a query to return a 
very large number of hits.

Here is the problem:

If relevance ranking is working well, in theory it doesn't matter how many hits 
the user gets as long as the best results show up in the first page of results. 
 When showing facet values, if the values for a particular facet have a large 
number of values, the general practice is to show a relatively small number of 
facet values, selected as those values with the highest counts in the entire 
result set.   However, assuming a very large result set, these facet counts 
will be affected by the large number of results that are not relevant to the 
query

As an example, if you search in our full-text collection for "jaguar" you get 
170,000 hits.  If I am looking for the car rather than the OS or the animal, I 
might expect to be able to click on a facet and limit my results to the car.  
However, facets containing the word "car" or "automobile" are not in the top 5 
facets that we show.  If you click on "more"  you will see "automobile 
periodicals" but not the rest of the facets containing the word automobile .  
This occurs because the facet counts are for all 170,000 hits.  The facet 
counts  for at least 160,000 irrelevant hits are included (assuming only the 
top 10,000 hits are relevant) .

What we would like to do is *select* which facet values to show, based on their 
counts in the *most relevant subset* of documents, but display the actual 
counts for the full set:

1)  get the facet counts for the N most relevant documents (N = 10,000 for 
example)
2)  select the 5 or 30 facet values with the highest counts for those 
relevant documents.
3)  display only the facet values for those 5 or 30 values, but display the 
counts for those values against the entire result set.

This is possible to kludge up (subject to some scaling considerations) in the 
following way:

1)  Consider only the 1000 most relevant documents for doing the 
calculation so N = 1,000
2)  do your query and get the unique document ids for the N most relevant 
documents. (i.e. set rows=N)  also get the facet values and counts for the top 
M facets, where M is some very large number and store the facet values and 
counts in some data structure.
3)  run a second query which is the same as the first, but add a filter 
query for those 1000 unique  ids, set rows =1 but get facet counts for the top 
30 facet values
4)  Grab the top 5 or 30 facet values from this second query.  These are 
your most relevant facet values
5)  Use the list of values from the previous step to retrieve the 
appropriate counts for the whole result set from the earlier stage where you 
stored the facet counts for the whole result set

It would seem that this could be done much more efficiently inside of 
Solr/Lucene, since instead of getting the unique ids for the N most relevant 
documents, and sending those back to Solr, the code actually has access to 
bitsets containing the internal Lucene index ids which get used in the filter 
queries.  Other steps in the process could probably be streamlined as well.

Is there already some faceting code work being done along this line?
Would it make sense to open a JIRA issue for this?


Tom Burton-West
http://www.hathitrust.org/blogs/large-scale-search

[jira] [Commented] (SOLR-2752) leader-per-shard

[Mark Miller (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-2752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13116009#comment-13116009
 ] 

Mark Miller commented on SOLR-2752:
---

I've committed this early work to the solrcloud branch

> leader-per-shard
> 
>
> Key: SOLR-2752
> URL: https://issues.apache.org/jira/browse/SOLR-2752
> Project: Solr
>  Issue Type: Sub-task
>  Components: SolrCloud
>Reporter: Yonik Seeley
>Assignee: Mark Miller
> Fix For: 4.0
>
> Attachments: SOLR-2752.patch, SOLR-2752.patch, SOLR-2752.patch, 
> SOLR-2752.patch, SOLR-2752.patch
>
>
> We need to add metadata into zookeeper about who is the leader for each 
> shard, and have some kind of leader election.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira



-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-2792) Allow case insensitive Hunspell stemming

[Updated]

 [ 
https://issues.apache.org/jira/browse/SOLR-2792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jan Høydahl updated SOLR-2792:
--

Attachment: SOLR-2792.patch

Ok, makes sense. New patch which removes ignoreCaseLocale, improves javadoc, 
and fixes a test bug for affix file.

I wonder why CharArrayMap#equals did only lowercase text1 and not text2 in the 
compare. Does my fix look right?

> Allow case insensitive Hunspell stemming
> 
>
> Key: SOLR-2792
> URL: https://issues.apache.org/jira/browse/SOLR-2792
> Project: Solr
>  Issue Type: Improvement
>  Components: Schema and Analysis
>Affects Versions: 3.5, 4.0
>Reporter: Jan Høydahl
>Assignee: Jan Høydahl
> Attachments: SOLR-2792.patch, SOLR-2792.patch
>
>
> Same as http://code.google.com/p/lucene-hunspell/issues/detail?id=3
> Hunspell dictionaries are by nature case sensitive. The Hunspell stemmer thus 
> needs an option to allow case insensitive matching of the dictionaries.
> Imagine a query for "microsofts". It will never be stemmed to the dictionary 
> word "Microsoft" because of the case difference. This problem cannot be fixed 
> by putting LowercaseFilter before Hunspell.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira



-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

trunk test failure (1317177541)

[Charlie Cron]

A test failure occurred running the tests.

revert! revert! revert!

You can see the entire build log at 
http://sierranevada.servebeer.com/1317177541.log

Thanks,
Charlie Cron


-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

trunk test failure (1317179941)

[Charlie Cron]

A test failure occurred running the tests.

revert! revert! revert!

You can see the entire build log at 
http://sierranevada.servebeer.com/1317179941.log

Thanks,
Charlie Cron


-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-2306) Modify default solrconfig parameters via JMX

[Amit Nithian (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-2306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13116116#comment-13116116
 ] 

Amit Nithian commented on SOLR-2306:


I have attached a new patch including unit tests and a blog post on my site 
indicating how to use it. I hope this helps!
http://hokiesuns.blogspot.com/2011/09/using-jmx-to-modify-parameters-of.html

> Modify default solrconfig parameters via JMX
> 
>
> Key: SOLR-2306
> URL: https://issues.apache.org/jira/browse/SOLR-2306
> Project: Solr
>  Issue Type: Improvement
>  Components: SearchComponents - other
>Affects Versions: 1.5
>Reporter: Amit Nithian
>Priority: Minor
> Fix For: 3.5, 4.0
>
> Attachments: tuning.patch
>
>
> Solr JMX support is great for reading the state of the search engine but it 
> should also support writing parameters that can affect the runtime 
> performance of the engine. At Zvents, our team wrote a custom web-UI in the 
> /admin folder to accomplish this but now have made a preliminary patch to 
> move this into JMX so that JConsole can be used to modify runtime parameters. 
> This is mostly used to tune ranking parameters in the configuration file 
> without passing them via the URL (to prevent changes to our front end site) 
> nor restarting the servlet container.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira



-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-2306) Modify default solrconfig parameters via JMX

[Amit Nithian (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-2306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Amit Nithian updated SOLR-2306:
---

Attachment: tuning2.patch

> Modify default solrconfig parameters via JMX
> 
>
> Key: SOLR-2306
> URL: https://issues.apache.org/jira/browse/SOLR-2306
> Project: Solr
>  Issue Type: Improvement
>  Components: SearchComponents - other
>Affects Versions: 1.5
>Reporter: Amit Nithian
>Priority: Minor
> Fix For: 3.5, 4.0
>
> Attachments: tuning.patch, tuning2.patch
>
>
> Solr JMX support is great for reading the state of the search engine but it 
> should also support writing parameters that can affect the runtime 
> performance of the engine. At Zvents, our team wrote a custom web-UI in the 
> /admin folder to accomplish this but now have made a preliminary patch to 
> move this into JMX so that JConsole can be used to modify runtime parameters. 
> This is mostly used to tune ranking parameters in the configuration file 
> without passing them via the URL (to prevent changes to our front end site) 
> nor restarting the servlet container.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira



-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-1536) if a filter can support random access API, we should use it

[Chris Male (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-1536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13116123#comment-13116123
 ] 

Chris Male commented on LUCENE-1536:


I haven't had a chance to look at the latest patch, but:

{quote}
For DocIdSet, can we nuke getRandomAccessBits? Ie, if
supportRandomAccess() returns true, then we can cast the instance to
Bits? Maybe we should rename supportRandomAccess to useRandomAccess?
(Ie, it may support it, but we only want to use random access when the
filter is dense enough).
{quote}

I'm definitely +1 to useRandomAccess() but I think there is a usability 
question mark around removing getRandomAccessBits().  If we assume that if 
DocIdSet.useRandomAccess() returns true then the DocIdSet must be also be a 
Bits implementation, then we need to prevent non-Bits implementations from 
returning true, or setting true in setUseRandomAccess.  If we don't, we're 
likely to confuse even expert users because this all comes together in a method 
deep inside IndexSearcher.

But if we're going to constrain useRandomAccess to only Bits implementations, 
then I once again feel these should be on Bits.  What if we added to Bits 
allowRandomAccessFiltering() or something like that? So even though Bits is 
inherently random-access, we control whether the Bits should be used to do 
filtering.

Alternatively we keep getRandomAccessBits() and see DocIdSet as a random-access 
Bits factory which currently just returns itself in most cases, but potentially 
might not in the future?

> if a filter can support random access API, we should use it
> ---
>
> Key: LUCENE-1536
> URL: https://issues.apache.org/jira/browse/LUCENE-1536
> Project: Lucene - Java
>  Issue Type: Improvement
>  Components: core/search
>Affects Versions: 2.4
>Reporter: Michael McCandless
>Assignee: Michael McCandless
>Priority: Minor
>  Labels: gsoc2011, lucene-gsoc-11, mentor
> Fix For: 4.0
>
> Attachments: CachedFilterIndexReader.java, LUCENE-1536.patch, 
> LUCENE-1536.patch, LUCENE-1536.patch, LUCENE-1536.patch, LUCENE-1536.patch, 
> LUCENE-1536.patch, LUCENE-1536.patch, LUCENE-1536.patch, LUCENE-1536.patch, 
> LUCENE-1536.patch, LUCENE-1536.patch
>
>
> I ran some performance tests, comparing applying a filter via
> random-access API instead of current trunk's iterator API.
> This was inspired by LUCENE-1476, where we realized deletions should
> really be implemented just like a filter, but then in testing found
> that switching deletions to iterator was a very sizable performance
> hit.
> Some notes on the test:
>   * Index is first 2M docs of Wikipedia.  Test machine is Mac OS X
> 10.5.6, quad core Intel CPU, 6 GB RAM, java 1.6.0_07-b06-153.
>   * I test across multiple queries.  1-X means an OR query, eg 1-4
> means 1 OR 2 OR 3 OR 4, whereas +1-4 is an AND query, ie 1 AND 2
> AND 3 AND 4.  "u s" means "united states" (phrase search).
>   * I test with multiple filter densities (0, 1, 2, 5, 10, 25, 75, 90,
> 95, 98, 99, 99.9 (filter is non-null but all bits are set),
> 100 (filter=null, control)).
>   * Method high means I use random-access filter API in
> IndexSearcher's main loop.  Method low means I use random-access
> filter API down in SegmentTermDocs (just like deleted docs
> today).
>   * Baseline (QPS) is current trunk, where filter is applied as iterator up
> "high" (ie in IndexSearcher's search loop).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira



-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Resolved] (LUCENE-3466) Rename Analyzer.reusableTokenStream() to tokenStream()

[Chris Male (Resolved) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LUCENE-3466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Male resolved LUCENE-3466.


   Resolution: Fixed
Fix Version/s: 4.0

Committed revision 1176728.

> Rename Analyzer.reusableTokenStream() to tokenStream()
> --
>
> Key: LUCENE-3466
> URL: https://issues.apache.org/jira/browse/LUCENE-3466
> Project: Lucene - Java
>  Issue Type: Sub-task
>  Components: modules/analysis
>Reporter: Chris Male
>Assignee: Chris Male
> Fix For: 4.0
>
> Attachments: LUCENE-3466.patch, LUCENE-3466.patch
>
>
> All Analysis consumers now use reusableTokenStream().  To finally make reuse 
> mandatory, lets rename resuableTokenStream() to tokenStream() (removing the 
> old tokenStream() method).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira



-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Assigned] (LUCENE-3470) reorder arguments of Field constructor to be more intuitive

[Chris Male (Assigned) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LUCENE-3470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Male reassigned LUCENE-3470:
--

Assignee: Chris Male

> reorder arguments of Field constructor to be more intuitive
> ---
>
> Key: LUCENE-3470
> URL: https://issues.apache.org/jira/browse/LUCENE-3470
> Project: Lucene - Java
>  Issue Type: Task
>Reporter: Robert Muir
>Assignee: Chris Male
> Fix For: 4.0
>
> Attachments: LUCENE-3470.patch
>
>
> I think Field should take (name, value, type) not (name, type, value) ?
> This seems more intuitive and consistent with previous releases
> Take this change to some code I had for example:
> {noformat}
> -d1.add(new Field("foo", "bar", Field.Store.YES, Field.Index.ANALYZED));
> +d1.add(new Field("foo", TextField.TYPE_STORED, "bar"));
> {noformat}
> I think it would be better if it was
> {noformat}
> document.add(new Field("foo", "bar", TextField.TYPE_STORED));
> {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira



-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (LUCENE-3470) reorder arguments of Field constructor to be more intuitive

[Chris Male (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LUCENE-3470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Male updated LUCENE-3470:
---

Attachment: LUCENE-3470.patch

Patch which reorders the arguments.

> reorder arguments of Field constructor to be more intuitive
> ---
>
> Key: LUCENE-3470
> URL: https://issues.apache.org/jira/browse/LUCENE-3470
> Project: Lucene - Java
>  Issue Type: Task
>Reporter: Robert Muir
> Fix For: 4.0
>
> Attachments: LUCENE-3470.patch
>
>
> I think Field should take (name, value, type) not (name, type, value) ?
> This seems more intuitive and consistent with previous releases
> Take this change to some code I had for example:
> {noformat}
> -d1.add(new Field("foo", "bar", Field.Store.YES, Field.Index.ANALYZED));
> +d1.add(new Field("foo", TextField.TYPE_STORED, "bar"));
> {noformat}
> I think it would be better if it was
> {noformat}
> document.add(new Field("foo", "bar", TextField.TYPE_STORED));
> {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira



-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-3470) reorder arguments of Field constructor to be more intuitive

[Robert Muir (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-3470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13116190#comment-13116190
 ] 

Robert Muir commented on LUCENE-3470:
-

ahhh sweet, i'm guessing you have an IDE that does this?

If the tests pass commit, this kinda crap goes out of date too fast!

> reorder arguments of Field constructor to be more intuitive
> ---
>
> Key: LUCENE-3470
> URL: https://issues.apache.org/jira/browse/LUCENE-3470
> Project: Lucene - Java
>  Issue Type: Task
>Reporter: Robert Muir
>Assignee: Chris Male
> Fix For: 4.0
>
> Attachments: LUCENE-3470.patch
>
>
> I think Field should take (name, value, type) not (name, type, value) ?
> This seems more intuitive and consistent with previous releases
> Take this change to some code I had for example:
> {noformat}
> -d1.add(new Field("foo", "bar", Field.Store.YES, Field.Index.ANALYZED));
> +d1.add(new Field("foo", TextField.TYPE_STORED, "bar"));
> {noformat}
> I think it would be better if it was
> {noformat}
> document.add(new Field("foo", "bar", TextField.TYPE_STORED));
> {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira



-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org