Re: Survey on ManagedResources feature

2020-08-20 Thread Matthias Krueger 


I don't use ManagedResourceStorage, just the StorageIO interface and its
implementations. The benefit is ZK AND filesystem support for WRITING
configuration file updates. Otherwise I would have to come up with
something like the common interface (StorageIO), its implementations for
ZK and FS and the dispatching code (ManagedResourceStorage#newStorageIO)
myself. I guess the LTR plugin had similar reasons to go that route.


On 19.08.20 08:49, Noble Paul wrote:
> So, it's not very different from directly reading a file from ZK?
>
> what benefit do you get by using the ManagedResourceStorage?
>
> On Sun, Aug 16, 2020 at 7:08 PM Matthias Krueger  wrote:
>>
>> In a custom SolrRequestHandler#handleRequest something like this:
>>
>> final ManagedResourceStorage.StorageIO storageIO =
>> ManagedResourceStorage.newStorageIO(core.getCoreDescriptor().getCollectionName(),
>> resourceLoader, new NamedList<>());
>>
>> And then using
>>
>> storageIO.openOutputStream(resourceName)
>>
>> to store some (well-known) resources.
>>
>> Matt
>>
>>
>> On 15.08.20 11:38, Noble Paul wrote:
>>>> I use MangedResource#StorageIO and its implementations as a convenient way 
>>>> to abstract away the underlying config storage when creating plugins that 
>>>> need to support both, SolrCloud and Solr Standalone.
>>> Can you give us some more details on how you use it?
>>>
>>> On Sat, Aug 15, 2020 at 7:32 PM Noble Paul  wrote:
>>>>> As authentication is plugged into the SolrDispatchFilter I would assume 
>>>>> that you would need to be authenticated to read/write Managed Resources
>>>> I'm talking about the authorization plugins
>>>>
>>>> On Fri, Aug 14, 2020 at 10:20 PM Matthias Krueger  wrote:
>>>>> As authentication is plugged into the SolrDispatchFilter I would assume 
>>>>> that you would need to be authenticated to read/write Managed Resources 
>>>>> but no authorization is checked (i.e. any authenticated user can 
>>>>> read/write them), correct?
>>>>>
>>>>> Anyway, I came across Managed Resources in at least two scenarios:
>>>>>
>>>>> The LTR plugin is using them for updating model/features.
>>>>> I use MangedResource#StorageIO and its implementations as a convenient 
>>>>> way to abstract away the underlying config storage when creating plugins 
>>>>> that need to support both, SolrCloud and Solr Standalone.
>>>>>
>>>>> IMO an abstraction that allows distributing configuration (ML models, 
>>>>> configuration snippets, external file fields...) that exceeds the typical 
>>>>> ZK size limits to SolrCloud while also supporting Solr Standalone would 
>>>>> be nice to have.
>>>>>
>>>>> Matt
>>>>>
>>>>>
>>>>> On 12.08.20 02:08, Noble Paul wrote:
>>>>>
>>>>> The end point is served by restlet. So, your rules are not going to be 
>>>>> honored. The rules work only if it is served by a Solr request handler
>>>>>
>>>>> On Wed, Aug 12, 2020, 12:46 AM Jason Gerlowski  
>>>>> wrote:
>>>>>> Hey Noble,
>>>>>>
>>>>>> Can you explain what you mean when you say it's not secured?  Just for
>>>>>> those of us who haven't been following the discussion so far?  On the
>>>>>> surface of things users taking advantage of our RuleBasedAuth plugin
>>>>>> can secure this API like they can any other HTTP API.  Or are you
>>>>>> talking about some other security aspect here?
>>>>>>
>>>>>> Jason
>>>>>>
>>>>>> On Tue, Aug 11, 2020 at 9:55 AM Noble Paul  wrote:
>>>>>>> Hi all,
>>>>>>> The end-point for Managed resources is not secured. So it needs to be
>>>>>>> fixed/eliminated.
>>>>>>>
>>>>>>> I would like to know what is the level of adoption for that feature
>>>>>>> and if it is a critical feature for users.
>>>>>>>
>>>>>>> Another possibility is to offer a replacement for the feature using a
>>>>>>> different API
>>>>>>>
>>>>>>> Your feedback will help us decide on what a potential solution should be
>>>>>>>
>>>>>>> --
>>>>>>> -
>>>>>>> Noble Paul
>>>>
>>>> --
>>>> -
>>>> Noble Paul
>>>
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
>> For additional commands, e-mail: dev-h...@lucene.apache.org
>>
>

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Survey on ManagedResources feature

2020-08-16 Thread Matthias Krueger 


In a custom SolrRequestHandler#handleRequest something like this:

    final ManagedResourceStorage.StorageIO storageIO =
ManagedResourceStorage.newStorageIO(core.getCoreDescriptor().getCollectionName(),
    resourceLoader, new NamedList<>());

And then using

    storageIO.openOutputStream(resourceName)

to store some (well-known) resources.

Matt


On 15.08.20 11:38, Noble Paul wrote:
>> I use MangedResource#StorageIO and its implementations as a convenient way 
>> to abstract away the underlying config storage when creating plugins that 
>> need to support both, SolrCloud and Solr Standalone.
> Can you give us some more details on how you use it?
>
> On Sat, Aug 15, 2020 at 7:32 PM Noble Paul  wrote:
>>> As authentication is plugged into the SolrDispatchFilter I would assume 
>>> that you would need to be authenticated to read/write Managed Resources
>> I'm talking about the authorization plugins
>>
>> On Fri, Aug 14, 2020 at 10:20 PM Matthias Krueger  wrote:
>>>
>>> As authentication is plugged into the SolrDispatchFilter I would assume 
>>> that you would need to be authenticated to read/write Managed Resources but 
>>> no authorization is checked (i.e. any authenticated user can read/write 
>>> them), correct?
>>>
>>> Anyway, I came across Managed Resources in at least two scenarios:
>>>
>>> The LTR plugin is using them for updating model/features.
>>> I use MangedResource#StorageIO and its implementations as a convenient way 
>>> to abstract away the underlying config storage when creating plugins that 
>>> need to support both, SolrCloud and Solr Standalone.
>>>
>>> IMO an abstraction that allows distributing configuration (ML models, 
>>> configuration snippets, external file fields...) that exceeds the typical 
>>> ZK size limits to SolrCloud while also supporting Solr Standalone would be 
>>> nice to have.
>>>
>>> Matt
>>>
>>>
>>> On 12.08.20 02:08, Noble Paul wrote:
>>>
>>> The end point is served by restlet. So, your rules are not going to be 
>>> honored. The rules work only if it is served by a Solr request handler
>>>
>>> On Wed, Aug 12, 2020, 12:46 AM Jason Gerlowski  
>>> wrote:
>>>> Hey Noble,
>>>>
>>>> Can you explain what you mean when you say it's not secured?  Just for
>>>> those of us who haven't been following the discussion so far?  On the
>>>> surface of things users taking advantage of our RuleBasedAuth plugin
>>>> can secure this API like they can any other HTTP API.  Or are you
>>>> talking about some other security aspect here?
>>>>
>>>> Jason
>>>>
>>>> On Tue, Aug 11, 2020 at 9:55 AM Noble Paul  wrote:
>>>>> Hi all,
>>>>> The end-point for Managed resources is not secured. So it needs to be
>>>>> fixed/eliminated.
>>>>>
>>>>> I would like to know what is the level of adoption for that feature
>>>>> and if it is a critical feature for users.
>>>>>
>>>>> Another possibility is to offer a replacement for the feature using a
>>>>> different API
>>>>>
>>>>> Your feedback will help us decide on what a potential solution should be
>>>>>
>>>>> --
>>>>> -
>>>>> Noble Paul
>>
>>
>> --
>> -
>> Noble Paul
>
>

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Survey on ManagedResources feature

2020-08-14 Thread Matthias Krueger 

As authentication is plugged into the SolrDispatchFilter I would assume
that you would need to be authenticated to read/write Managed Resources
but no authorization is checked (i.e. any authenticated user can
read/write them), correct?

Anyway, I came across Managed Resources in at least two scenarios:

  * The LTR plugin is using them for updating model/features.
  * I use MangedResource#StorageIO and its implementations as a
convenient way to abstract away the underlying config storage when
creating plugins that need to support both, SolrCloud and Solr
Standalone.

IMO an abstraction that allows distributing configuration (ML models,
configuration snippets, external file fields...) that exceeds the
typical ZK size limits to SolrCloud while also supporting Solr
Standalone would be nice to have.

Matt


On 12.08.20 02:08, Noble Paul wrote:
> The end point is served by restlet. So, your rules are not going to be
> honored. The rules work only if it is served by a Solr request handler
>
> On Wed, Aug 12, 2020, 12:46 AM Jason Gerlowski  > wrote:
>
> Hey Noble,
>
> Can you explain what you mean when you say it's not secured?  Just for
> those of us who haven't been following the discussion so far?  On the
> surface of things users taking advantage of our RuleBasedAuth plugin
> can secure this API like they can any other HTTP API.  Or are you
> talking about some other security aspect here?
>
> Jason
>
> On Tue, Aug 11, 2020 at 9:55 AM Noble Paul  > wrote:
> >
> > Hi all,
> > The end-point for Managed resources is not secured. So it needs
> to be
> > fixed/eliminated.
> >
> > I would like to know what is the level of adoption for that feature
> > and if it is a critical feature for users.
> >
> > Another possibility is to offer a replacement for the feature
> using a
> > different API
> >
> > Your feedback will help us decide on what a potential solution
> should be
> >
> > --
> > -
> > Noble Paul
>

[jira] [Updated] (LUCENE-8185) HyphenationCompoundWordTokenFilter returns terms shorter than minSubwordSize

2018-03-06 Thread Matthias Krueger (JIRA) 

 [ 
https://issues.apache.org/jira/browse/LUCENE-8185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias Krueger updated LUCENE-8185:
-
Affects Version/s: 6.6.1
   7.2.1

> HyphenationCompoundWordTokenFilter returns terms shorter than minSubwordSize
> 
>
> Key: LUCENE-8185
> URL: https://issues.apache.org/jira/browse/LUCENE-8185
> Project: Lucene - Core
>  Issue Type: Bug
>Affects Versions: 6.6.1, 7.2.1
>        Reporter: Matthias Krueger
>Priority: Minor
> Attachments: LUCENE-8185.patch
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> To account for languages which use binding characters ("fogemorphemes") for 
> composing words the {{HyphenationCompoundWordTokenFilter}} re-checks the 
> dictionary for a candidate with the last character removed when the original 
> candidate was not found. It currently does not re-check against 
> {{minSubWordSize}} in this case. Terms that are one character shorter than 
> {{minSubWordSize}} can be returned.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-8183) HyphenationCompoundWordTokenFilter creates overlapping tokens with onlyLongestMatch enabled

2018-02-26 Thread Matthias Krueger (JIRA) 

[ 
https://issues.apache.org/jira/browse/LUCENE-8183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16376826#comment-16376826
 ] 

Matthias Krueger commented on LUCENE-8183:
--

You might want to have a look at "mocking" the HyphenationTree (see my patch 
for LUCENE-8185) which simplifies writing a decompounding test.

> HyphenationCompoundWordTokenFilter creates overlapping tokens with 
> onlyLongestMatch enabled
> ---
>
> Key: LUCENE-8183
> URL: https://issues.apache.org/jira/browse/LUCENE-8183
> Project: Lucene - Core
>  Issue Type: Bug
>  Components: modules/analysis
>Affects Versions: 6.6
> Environment: Configuration of the analyzer:
> 
> 
>          hyphenator="lang/hyph_de_DR.xml" encoding="iso-8859-1"
>          dictionary="lang/wordlist_de.txt" 
>         onlyLongestMatch="true"/>
>  
>Reporter: Rupert Westenthaler
>Assignee: Uwe Schindler
>Priority: Major
> Attachments: LUCENE-8183_20180223_rwesten.diff, lucene-8183.zip
>
>
> The HyphenationCompoundWordTokenFilter creates overlapping tokens even if 
> onlyLongestMatch is enabled. 
> Example:
> Dictionary: {{gesellschaft}}, {{schaft}}
>  Hyphenator: {{de_DR.xml}} //from Apche Offo
>  onlyLongestMatch: true
>  
> |text|gesellschaft|gesellschaft|schaft|
> |raw_bytes|[67 65 73 65 6c 6c 73 63 68 61 66 74]|[67 65 73 65 6c 6c 73 63 68 
> 61 66 74]|[73 63 68 61 66 74]|
> |start|0|0|0|
> |end|12|12|12|
> |positionLength|1|1|1|
> |type|word|word|word|
> |position|1|1|1|
> IMHO this includes 2 unexpected Tokens
>  # the 2nd 'gesellschaft' as it duplicates the original token
>  # the 'schaft' as it is a sub-token 'gesellschaft' that is present in the 
> dictionary
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (LUCENE-8185) HyphenationCompoundWordTokenFilter returns terms shorter than minSubwordSize

2018-02-25 Thread Matthias Krueger (JIRA) 

 [ 
https://issues.apache.org/jira/browse/LUCENE-8185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias Krueger updated LUCENE-8185:
-
Attachment: LUCENE-8185.patch

> HyphenationCompoundWordTokenFilter returns terms shorter than minSubwordSize
> 
>
> Key: LUCENE-8185
> URL: https://issues.apache.org/jira/browse/LUCENE-8185
> Project: Lucene - Core
>  Issue Type: Bug
>    Reporter: Matthias Krueger
>Priority: Minor
> Attachments: LUCENE-8185.patch
>
>
> To account for languages which use binding characters ("fogemorphemes") for 
> composing words the {{HyphenationCompoundWordTokenFilter}} re-checks the 
> dictionary for a candidate with the last character removed when the original 
> candidate was not found. It currently does not re-check against 
> {{minSubWordSize}} in this case. Terms that are one character shorter than 
> {{minSubWordSize}} can be returned.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Created] (LUCENE-8185) HyphenationCompoundWordTokenFilter returns terms shorter than minSubwordSize

2018-02-25 Thread Matthias Krueger (JIRA) 
Matthias Krueger created LUCENE-8185:


 Summary: HyphenationCompoundWordTokenFilter returns terms shorter 
than minSubwordSize
 Key: LUCENE-8185
 URL: https://issues.apache.org/jira/browse/LUCENE-8185
 Project: Lucene - Core
  Issue Type: Bug
Reporter: Matthias Krueger


To account for languages which use binding characters ("fogemorphemes") for 
composing words the {{HyphenationCompoundWordTokenFilter}} re-checks the 
dictionary for a candidate with the last character removed when the original 
candidate was not found. It currently does not re-check against 
{{minSubWordSize}} in this case. Terms that are one character shorter than 
{{minSubWordSize}} can be returned.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-8183) HyphenationCompoundWordTokenFilter creates overlapping tokens with onlyLongestMatch enabled

2018-02-23 Thread Matthias Krueger (JIRA) 

[ 
https://issues.apache.org/jira/browse/LUCENE-8183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16374886#comment-16374886
 ] 

Matthias Krueger commented on LUCENE-8183:
--

[~rwesten] Quick question regarding your patch: What's the reasoning behind not 
decomposing terms that are part of the dictionary at all?

The {{onlyLongestMatch}} flag currently affects whether all matches or only the 
longest match should be returned *per* *start* character (in 
DictionaryCompoundWordTokenFilter) or *per* hyphenation *start* point (in 
HyphenationCompoundWordTokenFilter).

Example:
 Dictionary {{"Schaft", "Wirt", "Wirtschaft", "Wissen", "Wissenschaft"}} for 
input "Wirtschaftswissenschaft" will return the original input plus tokens 
"Wirtschaft", "schaft", "wissenschaft", "schaft" but not "Wirt" or "Wissen". 
"schaft" is still returned (even twice) because it's the longest token starting 
at the respective position.

I like the idea of restricting this further to only the longest terms that 
*touch* a certain hyphenation point. This would exclude "schaft" in the example 
above (as "Wirtschaft" and "wissenschaft" are two longer terms encompassing the 
respective hyphenation point). On the other hand, there might be examples where 
you still want to include the "overlapping" tokens. For "Fußballpumpe" and 
dictionary {{"Ball", "Ballpumpe", "Pumpe", "Fuß", "Fußball"}} you would get the 
tokens "Fußball" and "pumpe" but not "Ballpumpe" as "Ball" has already been 
considered part of Fußball. Also, not sure if your change also improves the 
situation for languages other than German.

Regarding point 1: The current algorithm always returns the term itself again 
if it's part of the dictionary. I guess, this could be changed if we don't 
check against {{this.maxSubwordSize}} but against 
{{Math.min(this.maxSubwordSize), termAtt.length()-1)}}

Perhaps these kind of adjustments should rather be done in a TokenFilter 
similar to RemoveDuplicatesTokenFilter instead of complicating the 
decompounding algorithm?

> HyphenationCompoundWordTokenFilter creates overlapping tokens with 
> onlyLongestMatch enabled
> ---
>
> Key: LUCENE-8183
> URL: https://issues.apache.org/jira/browse/LUCENE-8183
> Project: Lucene - Core
>  Issue Type: Bug
>  Components: modules/analysis
>Affects Versions: 6.6
> Environment: Configuration of the analyzer:
> 
> 
>          hyphenator="lang/hyph_de_DR.xml" encoding="iso-8859-1"
>          dictionary="lang/wordlist_de.txt" 
>         onlyLongestMatch="true"/>
>  
>Reporter: Rupert Westenthaler
>Assignee: Uwe Schindler
>Priority: Major
> Attachments: LUCENE-8183_20180223_rwesten.diff
>
>
> The HyphenationCompoundWordTokenFilter creates overlapping tokens even if 
> onlyLongestMatch is enabled. 
> Example:
> Dictionary: {{gesellschaft}}, {{schaft}}
>  Hyphenator: {{de_DR.xml}} //from Apche Offo
>  onlyLongestMatch: true
>  
> |text|gesellschaft|gesellschaft|schaft|
> |raw_bytes|[67 65 73 65 6c 6c 73 63 68 61 66 74]|[67 65 73 65 6c 6c 73 63 68 
> 61 66 74]|[73 63 68 61 66 74]|
> |start|0|0|0|
> |end|12|12|12|
> |positionLength|1|1|1|
> |type|word|word|word|
> |position|1|1|1|
> IMHO this includes 2 unexpected Tokens
>  # the 2nd 'gesellschaft' as it duplicates the original token
>  # the 'schaft' as it is a sub-token 'gesellschaft' that is present in the 
> dictionary
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-11974) TestCollapseQParserPlugin.testStringCollapse() failure

2018-02-15 Thread Matthias Krueger (JIRA) 

[ 
https://issues.apache.org/jira/browse/SOLR-11974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16365488#comment-16365488
 ] 

Matthias Krueger commented on SOLR-11974:
-

[~joel.bernstein] FYI Not sure if this is related but there are several bugs 
open due to (intentional or not) chaining of CollapsingQParserPlugins. See 
SOLR-8807, SOLR-9104, SOLR-10336, SOLR-10885, SOLR-7435.

> TestCollapseQParserPlugin.testStringCollapse() failure
> --
>
> Key: SOLR-11974
> URL: https://issues.apache.org/jira/browse/SOLR-11974
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: Tests
>Reporter: Steve Rowe
>Priority: Major
>
> Reproducing branch_7x seed from my Jenkins - reproduces for me on master too:
> {noformat}
> Checking out Revision 6ae6ce2f42b61b117c31149ac87b1cb3d6ffe440 
> (refs/remotes/origin/branch_7x)
> [...]
>[junit4]   2> NOTE: reproduce with: ant test  
> -Dtestcase=TestCollapseQParserPlugin -Dtests.method=testStringCollapse 
> -Dtests.seed=32531513E27A1FB -Dtests.slow=true -Dtests.locale=fr-LU 
> -Dtests.timezone=Europe/Warsaw -Dtests.asserts=true 
> -Dtests.file.encoding=UTF-8
>[junit4] FAILURE 1.15s J5  | TestCollapseQParserPlugin.testStringCollapse 
> <<<
>[junit4]> Throwable #1: java.lang.AssertionError
>[junit4]>  at 
> __randomizedtesting.SeedInfo.seed([32531513E27A1FB:F6AD0130F74002EF]:0)
>[junit4]>  at 
> org.apache.lucene.index.AssertingLeafReader$AssertingSortedDocValues.advanceExact(AssertingLeafReader.java:617)
>[junit4]>  at 
> org.apache.solr.search.CollapsingQParserPlugin$OrdScoreCollector.collect(CollapsingQParserPlugin.java:557)
>[junit4]>  at 
> org.apache.lucene.search.MatchAllDocsQuery$1$1.score(MatchAllDocsQuery.java:62)
>[junit4]>  at 
> org.apache.lucene.search.BulkScorer.score(BulkScorer.java:39)
>[junit4]>  at 
> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:660)
>[junit4]>  at 
> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:462)
>[junit4]>  at 
> org.apache.solr.search.Grouping.searchWithTimeLimiter(Grouping.java:447)
>[junit4]>  at 
> org.apache.solr.search.Grouping.execute(Grouping.java:395)
>[junit4]>  at 
> org.apache.solr.handler.component.QueryComponent.doProcessGroupedSearch(QueryComponent.java:1409)
>[junit4]>  at 
> org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:367)
>[junit4]>  at 
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:295)
>[junit4]>  at 
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:195)
>[junit4]>  at 
> org.apache.solr.core.SolrCore.execute(SolrCore.java:2503)
>[junit4]>  at 
> org.apache.solr.util.TestHarness.query(TestHarness.java:337)
>[junit4]>  at 
> org.apache.solr.util.TestHarness.query(TestHarness.java:319)
>[junit4]>  at 
> org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:876)
>[junit4]>  at 
> org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:862)
>[junit4]>  at 
> org.apache.solr.search.TestCollapseQParserPlugin.testCollapseQueries(TestCollapseQParserPlugin.java:768)
>[junit4]>  at 
> org.apache.solr.search.TestCollapseQParserPlugin.testStringCollapse(TestCollapseQParserPlugin.java:213)
>[junit4]>  at java.lang.Thread.run(Thread.java:748)
> [...]
>[junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): 
> {text_ws=FSTOrd50, term_s=PostingsFormat(name=Memory), 
> id=PostingsFormat(name=MockRandom), group_s=PostingsFormat(name=MockRandom), 
> group_s_dv=FSTOrd50}, docValues:{id_i=DocValuesFormat(name=Direct), 
> group_i=DocValuesFormat(name=Direct), 
> group_ti_dv=DocValuesFormat(name=Lucene70), 
> _version_=DocValuesFormat(name=Asserting), 
> test_l=DocValuesFormat(name=Direct), 
> group_tf_dv=DocValuesFormat(name=Memory), 
> test_i=DocValuesFormat(name=Lucene70), 
> group_f=DocValuesFormat(name=Asserting), test_f=DocValuesFormat(name=Memory), 
> group_s_dv=DocValuesFormat(name=Asserting)}, maxPointsInLeafNode=1995, 
> maxMBSortInHeap=7.1608710565771645, sim=RandomSimilarity(queryNorm=false): 
> {}, locale=fr-LU, timezone=Europe/Warsaw
>[junit4]   2> NOTE: Linux 4.1.0-custom2-amd64 amd64/Oracle Corp

[jira] [Commented] (SOLR-11049) Solr in cloud mode silently fails uploading a big LTR model

2017-07-31 Thread Matthias Krueger (JIRA) 

[ 
https://issues.apache.org/jira/browse/SOLR-11049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16107364#comment-16107364
 ] 

Matthias Krueger commented on SOLR-11049:
-

See https://lucene.apache.org/solr/guide/6_6/blob-store-api.html or 
https://cwiki.apache.org/confluence/display/solr/Blob+Store+API.

> Solr in cloud mode silently fails uploading a big LTR model
> ---
>
> Key: SOLR-11049
> URL: https://issues.apache.org/jira/browse/SOLR-11049
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: contrib - LTR
> Environment: tested with Solr 6.6 an integrated zookeeper
>Reporter: Stefan Langenmaier
> Attachments: SOLR-11049.patch
>
>
> Hi,
> I'm using Solr in cloud mode, I have a MultipleAdditiveTreesModel with about 
> 3MB in size. When I upload the model with
> {noformat}
> curl -v -XPUT 'http://localhost:8983/solr/tmdb/schema/model-store' 
> --data-binary @/big-tree.model -H 'Content-type:application/json'
> {noformat}
> I get the following response
> {code:html}
> {
>   "responseHeader":{
> "status":0,
> "QTime":24318}
> }
> {code}
> This looks kind of slow but without an error. When I check the config the 
> model is not visible and when I try to run a query that uses the model I get 
> the following error
> {code:html}
> "error":{
> "metadata":[
>   "error-class","org.apache.solr.common.SolrException",
>   "root-error-class","org.apache.solr.common.SolrException"],
> "msg":"cannot find model bigTreeModel",
> "code":400}
> {code}
> When I upload the model to solr where I increased the zookeeper znode size 
> limit with
> {noformat}
> -Djute.maxbuffer=0x1ff
> {noformat}
> the same model upload succeeds much faster
> {code:html}
> {
>   "responseHeader":{
> "status":0,
> "QTime":689}
> }
> {code}
> The model is visible in the configuration and queries that use it run without 
> error.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9104) NPE in CollapsingQParser when two fq={!collapse} and zero results

2017-06-02 Thread Matthias Krueger (JIRA) 

[ 
https://issues.apache.org/jira/browse/SOLR-9104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16034358#comment-16034358
 ] 

Matthias Krueger commented on SOLR-9104:


[~markus.jel...@openindex.io]: See comment from [~hossman] here or mine on 
SOLR-8807. It seems that two Collapse-Filters just don't work if there are 
documents returned from less than all segments. A quick fix seems to be to call 
{{super.doSetNextReader(context)}} as first thing in all of 
CollapsingQParserPlugin's DelegatingCollectors' {{doSetNextReaders}} but I 
don't know if this has any undesired side effects.

> NPE in CollapsingQParser when two fq={!collapse} and zero results
> -
>
> Key: SOLR-9104
> URL: https://issues.apache.org/jira/browse/SOLR-9104
> Project: Solr
>  Issue Type: Bug
>Affects Versions: 6.0
>Reporter: Markus Jelsma
> Fix For: 6.2, master (7.0)
>
>
> This is a very weird problem that is reproducible on a small production 
> server, but not on the local machine although they run the same 6.0 version., 
> and have an almost identical index. The only minor difference is that 
> production is a SolrCloud with 1 shard and two replica's, just for a bit of 
> redundancy.
> The following query yields zero results and throws the NPE:
> {code}
> select?q=query:seis={!collapse field=query_digest}={!collapse 
> field=result_digest}
> {code}
> The next query does yield results and does not throw anything, it just works 
> as it should work:
> {code}
> select?q=query:seiz={!collapse field=query_digest}={!collapse 
> field=result_digest}
> {code}
> The /select handler used does not add any fancy param other than rows.
> Here's the NPE:
> {code}
> 2016-05-11 14:10:27.666 ERROR (qtp1209271652-3338) [c:suggestions s:shard1 
> r:core_node1 x:suggestions_shard1_replica1] o.a.s.s.HttpSolrCall 
> null:java.lang.NullPointerException
> at 
> org.apache.solr.search.CollapsingQParserPlugin$IntScoreCollector.finish(CollapsingQParserPlugin.java:814)
> at 
> org.apache.solr.search.CollapsingQParserPlugin$IntScoreCollector.finish(CollapsingQParserPlugin.java:851)
> at 
> org.apache.solr.search.SolrIndexSearcher.buildAndRunCollectorChain(SolrIndexSearcher.java:272)
> at 
> org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:1794)
> at 
> org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:1611)
> at 
> org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:634)
> at 
> org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:529)
> at 
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:287)
> at 
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:155)
> {code}
> Edit: for the sake of clarity. It really needs double fq={!collapse bla bla 
> for the NPE to appear. If i remove either of the filters from the query, i 
> get a nice zero resultset back. Both fields are defined as int.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-8807) NPE during spell checking when result collapsing is activated and index got more than one segment.

2017-04-07 Thread Matthias Krueger (JIRA) 

[ 
https://issues.apache.org/jira/browse/SOLR-8807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15960948#comment-15960948
 ] 

Matthias Krueger commented on SOLR-8807:


Thanks, [~jdyer], looks good to me. This won't help the linked 
CollapsingQParser issues (which all seem to have the same root cause) but will 
solve the SpellCheckCollator's problem.

> NPE during spell checking when result collapsing is activated and index got 
> more than one segment.
> --
>
> Key: SOLR-8807
> URL: https://issues.apache.org/jira/browse/SOLR-8807
> Project: Solr
>  Issue Type: Bug
>  Components: spellchecker
>Affects Versions: 5.4
> Environment: Solr 5.4 with an index cosisting of two segments
>Reporter: Christian Danninger
>Priority: Critical
>  Labels: collapse, spellcheck
> Attachments: SOLR-8806-failing-unit-test.patch, SOLR-8807.patch, 
> SOLR-8807.patch
>
>
> When using spellchecker with collapse/expand results, I got an NPE. Only 
> happend when the index consists of more than one segment. 
> {code}
> 11:30:33,505 WARN  [org.apache.solr.spelling.SpellCheckCollator] 
> (http-/0.0.0.0:8080-2) Exception trying to re-query to check if a spell check 
> possibility would return any hits.: java.lang.NullPointerException
> at 
> org.apache.solr.search.CollapsingQParserPlugin$OrdScoreCollector.finish(CollapsingQParserPlugin.java:631)
>  [solr-core-5.4.0.jar:5.4.0 1718046 - upayavira - 2015-12-04 23:16:46]
> at 
> org.apache.solr.search.CollapsingQParserPlugin$OrdScoreCollector.finish(CollapsingQParserPlugin.java:681)
>  [solr-core-5.4.0.jar:5.4.0 1718046 - upayavira - 2015-12-04 23:16:46]
> at 
> org.apache.solr.search.SolrIndexSearcher.buildAndRunCollectorChain(SolrIndexSearcher.java:213)
>  [solr-core-5.4.0.jar:5.4.0 1718046 - upayavira - 2015-12-04 23:16:46]
> at 
> org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:1672)
>  [solr-core-5.4.0.jar:5.4.0 1718046 - upayavira - 2015-12-04 23:16:46]
> at 
> org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:1491)
>  [solr-core-5.4.0.jar:5.4.0 1718046 - upayavira - 2015-12-04 23:16:46]
> at 
> org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:557) 
> [solr-core-5.4.0.jar:5.4.0 1718046 - upayavira - 2015-12-04 23:16:46]
> at 
> org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:525)
>  [solr-core-5.4.0.jar:5.4.0 1718046 - upayavira - 2015-12-04 23:16:46]
> at 
> org.apache.solr.spelling.SpellCheckCollator.collate(SpellCheckCollator.java:147)
>  [solr-core-5.4.0.jar:5.4.0 1718046 - upayavira - 2015-12-04 23:16:46]
> at 
> org.apache.solr.handler.component.SpellCheckComponent.addCollationsToResponse(SpellCheckComponent.java:238)
>  [solr-core-5.4.0.jar:5.4.0 1718046 - upayavira - 2015-12-04 23:16:46]
> at 
> org.apache.solr.handler.component.SpellCheckComponent.process(SpellCheckComponent.java:203)
>  [solr-core-5.4.0.jar:5.4.0 1718046 - upayavira - 2015-12-04 23:16:46]
> at 
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:281)
>  [solr-core-5.4.0.jar:5.4.0 1718046 - upayavira - 2015-12-04 23:16:46]
> at 
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:156)
>  [solr-core-5.4.0.jar:5.4.0 1718046 - upayavira - 2015-12-04 23:16:46]
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2073) 
> [solr-core-5.4.0.jar:5.4.0 1718046 - upayavira - 2015-12-04 23:16:46]
> {code}
> {code}
>   
>   
> spellchecker
>   
> 
> {code}
> The query parameters are:
> {code}
>   "spellcheck.maxCollations": "5",
>   "q.op": "AND",
>   "fq": "{!collapse field=type}",
>   "spellcheck.maxCollationTries": "10",
>   "spellcheck.collateMaxCollectDocs": "10",
>   "spellcheck.alternativeTermCount": "10",
>   "spellcheck.extendedResults": "true",
>   "spellcheck.dictionary": [
> "dest_wordbreak",
> "dest_fuzzy"
>   ],
>   "q": "kosamui thailand",
>   "defType": "edismax",
>   "expand": "true",
>   "spellcheck.maxResultsForSuggest": "3",
>   "qf": "country_name region_name",
>   "spellcheck": "true",
>   "spellcheck.accuracy": "0.5",
>   "spellcheck.count": "20",
>   "spellcheck.collate": "true",
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-10395) NPE during spell check with result collation

2017-03-31 Thread Matthias Krueger (JIRA) 

[ 
https://issues.apache.org/jira/browse/SOLR-10395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15950763#comment-15950763
 ] 

Matthias Krueger commented on SOLR-10395:
-

This is an odd code location for an NPE. I cannot see any nullable 
reassignments to either {{checkResponse}}, {{checkResponse.rsp}}, or 
{{checkResponse.rsp.getToLog}} in the current code base. Could you share the 
search request you're sending? Do you use any other search components or 
plugins in that request?


> NPE during spell check with result collation
> 
>
> Key: SOLR-10395
> URL: https://issues.apache.org/jira/browse/SOLR-10395
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: spellchecker
>Affects Versions: 6.2.1
>Reporter: Frank Steggink
>
> I'm getting a NullPointerException when using collation during spell check. 
> This is the message I'm getting: Exception trying to re-query to check if a 
> spell check possibility would return any hits.
> Stack trace:
> {code}
> 2017-03-13 13:05:54.618 WARN (qtp472654579-84) [c:locatieserver s:dkk 
> r:core_node4 x:locatieserver_dkk_replica2] o.a.s.s.SpellCheckCollator 
> Exception trying to re-query to check if a spell check possibility would 
> return any hits.
> java.lang.NullPointerException
> at 
> org.apache.solr.spelling.SpellCheckCollator.collate(SpellCheckCollator.java:150)
> at 
> org.apache.solr.handler.component.SpellCheckComponent.addCollationsToResponse(SpellCheckComponent.java:297)
> at 
> org.apache.solr.handler.component.SpellCheckComponent.process(SpellCheckComponent.java:209)
> at 
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:295)
> at 
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:154)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2089)
> ...
> {code}
> My spell check config for this search handler:
> {code:xml}
> on
> suggest
> false
> 5
> 2
> 3
> true
> true
> 5
> 3
> {code}
> The field "suggest" is a field in the index, which is also (usually) queried 
> with this search handler. The field is filled with lots of data created by 
> edge ngrams, since it's intended usage is for an autocomplete field.
> This issue is very similar to SOLR-8807, but the NPE is occurring one line in 
> the code later 
> (solr\core\src\java\org\apache\solr\spelling\SpellCheckCollator.java). As a 
> result the patch attached to that issue probably won't work in my case.
> {code:java}
> try {
>   queryComponent.prepare(checkResponse);
>   if (docCollectionLimit > 0) {
> int f = checkResponse.getFieldFlags();
> checkResponse.setFieldFlags(f |= 
> SolrIndexSearcher.TERMINATE_EARLY);
>   }
>   queryComponent.process(checkResponse);
>   hits = (Integer) checkResponse.rsp.getToLog().get("hits");
> } catch (EarlyTerminatingCollectorException etce) {
> {code}
> Although I'm using Solr 6.2.1, the source file is still identical to Solr 
> 6.5.0.
> The line causing my exception is when determining the hits. I've noticed the 
> exception only occurs when no more than 2 alternatives are found, but when 3 
> alternatives (maxCollations) are found, the NPE isn't occurring.
> As a workaround I could live for now with turning collations off.
> Please let me know if you need more information. Thanks!



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Comment Edited] (SOLR-8807) NPE during spell checking when result collapsing is activated and index got more than one segment.

2017-03-21 Thread Matthias Krueger (JIRA) 

[ 
https://issues.apache.org/jira/browse/SOLR-8807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15935520#comment-15935520
 ] 

Matthias Krueger edited comment on SOLR-8807 at 3/21/17 11:35 PM:
--

This seems to be very much related to SOLR-9104, SOLR-10336, and SOLR-7435 or 
even the same cause.


was (Author: mkrio):
This seems to be very much related to SOLR-9104 and SOLR-10336 or even the same 
cause.

> NPE during spell checking when result collapsing is activated and index got 
> more than one segment.
> --
>
> Key: SOLR-8807
> URL: https://issues.apache.org/jira/browse/SOLR-8807
> Project: Solr
>  Issue Type: Bug
>  Components: spellchecker
>Affects Versions: 5.4
> Environment: Solr 5.4 with an index cosisting of two segments
>Reporter: Christian Danninger
>Priority: Critical
>  Labels: collapse, spellcheck
> Attachments: SOLR-8807.patch
>
>
> When using spellchecker with collapse/expand results, I got an NPE. Only 
> happend when the index consists of more than one segment. 
> {code}
> 11:30:33,505 WARN  [org.apache.solr.spelling.SpellCheckCollator] 
> (http-/0.0.0.0:8080-2) Exception trying to re-query to check if a spell check 
> possibility would return any hits.: java.lang.NullPointerException
> at 
> org.apache.solr.search.CollapsingQParserPlugin$OrdScoreCollector.finish(CollapsingQParserPlugin.java:631)
>  [solr-core-5.4.0.jar:5.4.0 1718046 - upayavira - 2015-12-04 23:16:46]
> at 
> org.apache.solr.search.CollapsingQParserPlugin$OrdScoreCollector.finish(CollapsingQParserPlugin.java:681)
>  [solr-core-5.4.0.jar:5.4.0 1718046 - upayavira - 2015-12-04 23:16:46]
> at 
> org.apache.solr.search.SolrIndexSearcher.buildAndRunCollectorChain(SolrIndexSearcher.java:213)
>  [solr-core-5.4.0.jar:5.4.0 1718046 - upayavira - 2015-12-04 23:16:46]
> at 
> org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:1672)
>  [solr-core-5.4.0.jar:5.4.0 1718046 - upayavira - 2015-12-04 23:16:46]
> at 
> org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:1491)
>  [solr-core-5.4.0.jar:5.4.0 1718046 - upayavira - 2015-12-04 23:16:46]
> at 
> org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:557) 
> [solr-core-5.4.0.jar:5.4.0 1718046 - upayavira - 2015-12-04 23:16:46]
> at 
> org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:525)
>  [solr-core-5.4.0.jar:5.4.0 1718046 - upayavira - 2015-12-04 23:16:46]
> at 
> org.apache.solr.spelling.SpellCheckCollator.collate(SpellCheckCollator.java:147)
>  [solr-core-5.4.0.jar:5.4.0 1718046 - upayavira - 2015-12-04 23:16:46]
> at 
> org.apache.solr.handler.component.SpellCheckComponent.addCollationsToResponse(SpellCheckComponent.java:238)
>  [solr-core-5.4.0.jar:5.4.0 1718046 - upayavira - 2015-12-04 23:16:46]
> at 
> org.apache.solr.handler.component.SpellCheckComponent.process(SpellCheckComponent.java:203)
>  [solr-core-5.4.0.jar:5.4.0 1718046 - upayavira - 2015-12-04 23:16:46]
> at 
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:281)
>  [solr-core-5.4.0.jar:5.4.0 1718046 - upayavira - 2015-12-04 23:16:46]
> at 
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:156)
>  [solr-core-5.4.0.jar:5.4.0 1718046 - upayavira - 2015-12-04 23:16:46]
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2073) 
> [solr-core-5.4.0.jar:5.4.0 1718046 - upayavira - 2015-12-04 23:16:46]
> {code}
> {code}
>   
>   
> spellchecker
>   
> 
> {code}
> The query parameters are:
> {code}
>   "spellcheck.maxCollations": "5",
>   "q.op": "AND",
>   "fq": "{!collapse field=type}",
>   "spellcheck.maxCollationTries": "10",
>   "spellcheck.collateMaxCollectDocs": "10",
>   "spellcheck.alternativeTermCount": "10",
>   "spellcheck.extendedResults": "true",
>   "spellcheck.dictionary": [
> "dest_wordbreak",
> "dest_fuzzy"
>   ],
>   "q": "kosamui thailand",
>   "defType": "edismax",
>   "expand": "true",
>   "spellcheck.maxResultsForSuggest": "3",
>   "qf": "country_name region_name",
>   "spellcheck": "true",
>   "spellcheck.accuracy": "0.5",
>   "spellcheck.count": "20",
>   "spellcheck.collate": "true",
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-8807) NPE during spell checking when result collapsing is activated and index got more than one segment.

2017-03-21 Thread Matthias Krueger (JIRA) 

 [ 
https://issues.apache.org/jira/browse/SOLR-8807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias Krueger updated SOLR-8807:
---
Attachment: SOLR-8807.patch

A simple solution would be to have CollapsingQParserPlugin behave like other 
DelegatingCollectors and call
{code}
super.doSetNextReader(context); 
{code}
in doSetNextReader. Not sure if omission was intentional though. This at least 
avoids the NPE we're seeing.

> NPE during spell checking when result collapsing is activated and index got 
> more than one segment.
> --
>
> Key: SOLR-8807
> URL: https://issues.apache.org/jira/browse/SOLR-8807
> Project: Solr
>  Issue Type: Bug
>  Components: spellchecker
>Affects Versions: 5.4
> Environment: Solr 5.4 with an index cosisting of two segments
>Reporter: Christian Danninger
>Priority: Critical
>  Labels: collapse, spellcheck
> Attachments: SOLR-8807.patch
>
>
> When using spellchecker with collapse/expand results, I got an NPE. Only 
> happend when the index consists of more than one segment. 
> {code}
> 11:30:33,505 WARN  [org.apache.solr.spelling.SpellCheckCollator] 
> (http-/0.0.0.0:8080-2) Exception trying to re-query to check if a spell check 
> possibility would return any hits.: java.lang.NullPointerException
> at 
> org.apache.solr.search.CollapsingQParserPlugin$OrdScoreCollector.finish(CollapsingQParserPlugin.java:631)
>  [solr-core-5.4.0.jar:5.4.0 1718046 - upayavira - 2015-12-04 23:16:46]
> at 
> org.apache.solr.search.CollapsingQParserPlugin$OrdScoreCollector.finish(CollapsingQParserPlugin.java:681)
>  [solr-core-5.4.0.jar:5.4.0 1718046 - upayavira - 2015-12-04 23:16:46]
> at 
> org.apache.solr.search.SolrIndexSearcher.buildAndRunCollectorChain(SolrIndexSearcher.java:213)
>  [solr-core-5.4.0.jar:5.4.0 1718046 - upayavira - 2015-12-04 23:16:46]
> at 
> org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:1672)
>  [solr-core-5.4.0.jar:5.4.0 1718046 - upayavira - 2015-12-04 23:16:46]
> at 
> org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:1491)
>  [solr-core-5.4.0.jar:5.4.0 1718046 - upayavira - 2015-12-04 23:16:46]
> at 
> org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:557) 
> [solr-core-5.4.0.jar:5.4.0 1718046 - upayavira - 2015-12-04 23:16:46]
> at 
> org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:525)
>  [solr-core-5.4.0.jar:5.4.0 1718046 - upayavira - 2015-12-04 23:16:46]
> at 
> org.apache.solr.spelling.SpellCheckCollator.collate(SpellCheckCollator.java:147)
>  [solr-core-5.4.0.jar:5.4.0 1718046 - upayavira - 2015-12-04 23:16:46]
> at 
> org.apache.solr.handler.component.SpellCheckComponent.addCollationsToResponse(SpellCheckComponent.java:238)
>  [solr-core-5.4.0.jar:5.4.0 1718046 - upayavira - 2015-12-04 23:16:46]
> at 
> org.apache.solr.handler.component.SpellCheckComponent.process(SpellCheckComponent.java:203)
>  [solr-core-5.4.0.jar:5.4.0 1718046 - upayavira - 2015-12-04 23:16:46]
> at 
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:281)
>  [solr-core-5.4.0.jar:5.4.0 1718046 - upayavira - 2015-12-04 23:16:46]
> at 
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:156)
>  [solr-core-5.4.0.jar:5.4.0 1718046 - upayavira - 2015-12-04 23:16:46]
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2073) 
> [solr-core-5.4.0.jar:5.4.0 1718046 - upayavira - 2015-12-04 23:16:46]
> {code}
> {code}
>   
>   
> spellchecker
>   
> 
> {code}
> The query parameters are:
> {code}
>   "spellcheck.maxCollations": "5",
>   "q.op": "AND",
>   "fq": "{!collapse field=type}",
>   "spellcheck.maxCollationTries": "10",
>   "spellcheck.collateMaxCollectDocs": "10",
>   "spellcheck.alternativeTermCount": "10",
>   "spellcheck.extendedResults": "true",
>   "spellcheck.dictionary": [
> "dest_wordbreak",
> "dest_fuzzy"
>   ],
>   "q": "kosamui thailand",
>   "defType": "edismax",
>   "expand": "true",
>   "spellcheck.maxResultsForSuggest": "3",
>   "qf": "country_name region_name",
>   "spellcheck": "true",
>   "spellcheck.accuracy": "0.5",
>   "spellcheck.count": "20",
>   "spellcheck.collate": "true",
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-8807) NPE during spell checking when result collapsing is activated and index got more than one segment.

2017-03-21 Thread Matthias Krueger (JIRA) 

[ 
https://issues.apache.org/jira/browse/SOLR-8807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15935520#comment-15935520
 ] 

Matthias Krueger commented on SOLR-8807:


This seems to be very much related to SOLR-9104 and SOLR-10336 or even the same 
cause.

> NPE during spell checking when result collapsing is activated and index got 
> more than one segment.
> --
>
> Key: SOLR-8807
> URL: https://issues.apache.org/jira/browse/SOLR-8807
> Project: Solr
>  Issue Type: Bug
>  Components: spellchecker
>Affects Versions: 5.4
> Environment: Solr 5.4 with an index cosisting of two segments
>Reporter: Christian Danninger
>Priority: Critical
>  Labels: collapse, spellcheck
>
> When using spellchecker with collapse/expand results, I got an NPE. Only 
> happend when the index consists of more than one segment. 
> {code}
> 11:30:33,505 WARN  [org.apache.solr.spelling.SpellCheckCollator] 
> (http-/0.0.0.0:8080-2) Exception trying to re-query to check if a spell check 
> possibility would return any hits.: java.lang.NullPointerException
> at 
> org.apache.solr.search.CollapsingQParserPlugin$OrdScoreCollector.finish(CollapsingQParserPlugin.java:631)
>  [solr-core-5.4.0.jar:5.4.0 1718046 - upayavira - 2015-12-04 23:16:46]
> at 
> org.apache.solr.search.CollapsingQParserPlugin$OrdScoreCollector.finish(CollapsingQParserPlugin.java:681)
>  [solr-core-5.4.0.jar:5.4.0 1718046 - upayavira - 2015-12-04 23:16:46]
> at 
> org.apache.solr.search.SolrIndexSearcher.buildAndRunCollectorChain(SolrIndexSearcher.java:213)
>  [solr-core-5.4.0.jar:5.4.0 1718046 - upayavira - 2015-12-04 23:16:46]
> at 
> org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:1672)
>  [solr-core-5.4.0.jar:5.4.0 1718046 - upayavira - 2015-12-04 23:16:46]
> at 
> org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:1491)
>  [solr-core-5.4.0.jar:5.4.0 1718046 - upayavira - 2015-12-04 23:16:46]
> at 
> org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:557) 
> [solr-core-5.4.0.jar:5.4.0 1718046 - upayavira - 2015-12-04 23:16:46]
> at 
> org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:525)
>  [solr-core-5.4.0.jar:5.4.0 1718046 - upayavira - 2015-12-04 23:16:46]
> at 
> org.apache.solr.spelling.SpellCheckCollator.collate(SpellCheckCollator.java:147)
>  [solr-core-5.4.0.jar:5.4.0 1718046 - upayavira - 2015-12-04 23:16:46]
> at 
> org.apache.solr.handler.component.SpellCheckComponent.addCollationsToResponse(SpellCheckComponent.java:238)
>  [solr-core-5.4.0.jar:5.4.0 1718046 - upayavira - 2015-12-04 23:16:46]
> at 
> org.apache.solr.handler.component.SpellCheckComponent.process(SpellCheckComponent.java:203)
>  [solr-core-5.4.0.jar:5.4.0 1718046 - upayavira - 2015-12-04 23:16:46]
> at 
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:281)
>  [solr-core-5.4.0.jar:5.4.0 1718046 - upayavira - 2015-12-04 23:16:46]
> at 
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:156)
>  [solr-core-5.4.0.jar:5.4.0 1718046 - upayavira - 2015-12-04 23:16:46]
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2073) 
> [solr-core-5.4.0.jar:5.4.0 1718046 - upayavira - 2015-12-04 23:16:46]
> {code}
> {code}
>   
>   
> spellchecker
>   
> 
> {code}
> The query parameters are:
> {code}
>   "spellcheck.maxCollations": "5",
>   "q.op": "AND",
>   "fq": "{!collapse field=type}",
>   "spellcheck.maxCollationTries": "10",
>   "spellcheck.collateMaxCollectDocs": "10",
>   "spellcheck.alternativeTermCount": "10",
>   "spellcheck.extendedResults": "true",
>   "spellcheck.dictionary": [
> "dest_wordbreak",
> "dest_fuzzy"
>   ],
>   "q": "kosamui thailand",
>   "defType": "edismax",
>   "expand": "true",
>   "spellcheck.maxResultsForSuggest": "3",
>   "qf": "country_name region_name",
>   "spellcheck": "true",
>   "spellcheck.accuracy": "0.5",
>   "spellcheck.count": "20",
>   "spellcheck.collate": "true",
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-8807) NPE during spell checking when result collapsing is activated and index got more than one segment.

2017-03-21 Thread Matthias Krueger (JIRA) 

[ 
https://issues.apache.org/jira/browse/SOLR-8807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15935514#comment-15935514
 ] 

Matthias Krueger commented on SOLR-8807:


My client is affected by this NPE, too. Some oddities of the SpellCheckCollator 
seem to misalign with behaviour of the CollapsingQParserPlugin's collectors.
* The {{checkResponse}} ResponseBuilder in SpellCheckCollator#collate is being 
built using the original params (including the fq=\{!collapse…\} param) *and* 
the original filters.
* The {{checkResponse}} ends up having the 
CollapsingQParserPlugin$CollapsingPostFilter twice in the {{filters}} list.
* They will end up in a DelegatingCollector chain 
CollapsingQParserPlugin$OrdScoreCollector -> 
CollapsingQParserPlugin$OrdScoreCollector.
* Unfortunately, this does not work well when there are documents returned from 
less than all segments. The two OrdScoreCollectors are instantiated with the 
number of segments and open an empty array of LeafReaderContexts but their 
values are only populated for the first OrdScoreCollector in 
IndexSearcher#search (via Collector#getLeafCollector -> 
OrdScoreCollector#doSetNextReader) calls.
* OrdScoreCollector (and the other collectors in CollapsingQParserPlugin) only 
passes the LeafReaderContexts to the delegate within finish() (not within 
#doSetNextReader) but stops when it deems all necessary segments consumed. When 
the delegate itself then performs the iteration it will NPE in
{code}
nextDocBase = currentContext+1 < contexts.length ? 
contexts[currentContext+1].docBase : maxDoc;
{code}


> NPE during spell checking when result collapsing is activated and index got 
> more than one segment.
> --
>
> Key: SOLR-8807
> URL: https://issues.apache.org/jira/browse/SOLR-8807
> Project: Solr
>  Issue Type: Bug
>  Components: spellchecker
>Affects Versions: 5.4
> Environment: Solr 5.4 with an index cosisting of two segments
>Reporter: Christian Danninger
>Priority: Critical
>  Labels: collapse, spellcheck
>
> When using spellchecker with collapse/expand results, I got an NPE. Only 
> happend when the index consists of more than one segment. 
> {code}
> 11:30:33,505 WARN  [org.apache.solr.spelling.SpellCheckCollator] 
> (http-/0.0.0.0:8080-2) Exception trying to re-query to check if a spell check 
> possibility would return any hits.: java.lang.NullPointerException
> at 
> org.apache.solr.search.CollapsingQParserPlugin$OrdScoreCollector.finish(CollapsingQParserPlugin.java:631)
>  [solr-core-5.4.0.jar:5.4.0 1718046 - upayavira - 2015-12-04 23:16:46]
> at 
> org.apache.solr.search.CollapsingQParserPlugin$OrdScoreCollector.finish(CollapsingQParserPlugin.java:681)
>  [solr-core-5.4.0.jar:5.4.0 1718046 - upayavira - 2015-12-04 23:16:46]
> at 
> org.apache.solr.search.SolrIndexSearcher.buildAndRunCollectorChain(SolrIndexSearcher.java:213)
>  [solr-core-5.4.0.jar:5.4.0 1718046 - upayavira - 2015-12-04 23:16:46]
> at 
> org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:1672)
>  [solr-core-5.4.0.jar:5.4.0 1718046 - upayavira - 2015-12-04 23:16:46]
> at 
> org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:1491)
>  [solr-core-5.4.0.jar:5.4.0 1718046 - upayavira - 2015-12-04 23:16:46]
> at 
> org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:557) 
> [solr-core-5.4.0.jar:5.4.0 1718046 - upayavira - 2015-12-04 23:16:46]
> at 
> org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:525)
>  [solr-core-5.4.0.jar:5.4.0 1718046 - upayavira - 2015-12-04 23:16:46]
> at 
> org.apache.solr.spelling.SpellCheckCollator.collate(SpellCheckCollator.java:147)
>  [solr-core-5.4.0.jar:5.4.0 1718046 - upayavira - 2015-12-04 23:16:46]
> at 
> org.apache.solr.handler.component.SpellCheckComponent.addCollationsToResponse(SpellCheckComponent.java:238)
>  [solr-core-5.4.0.jar:5.4.0 1718046 - upayavira - 2015-12-04 23:16:46]
> at 
> org.apache.solr.handler.component.SpellCheckComponent.process(SpellCheckComponent.java:203)
>  [solr-core-5.4.0.jar:5.4.0 1718046 - upayavira - 2015-12-04 23:16:46]
> at 
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:281)
>  [solr-core-5.4.0.jar:5.4.0 1718046 - upayavira - 2015-12-04 23:16:46]
> at 
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:156)
>  [solr-core-5.4.0.jar:5.4.0 1718046 - upayavira - 2015-12-04 23:16:46]
> at or