Re: [Migration Solr4 to Solr5] Collection reload error
Thanks Shawn, Missed the openSearcher=false setting. So another thing to check really is whether there are concurrent commitWithin calls ever to the same shard. 10 марта 2016 г. 4:39 PM пользователь "Shawn Heisey" написал: > On 3/10/2016 3:05 AM, Dmitry Kan wrote: > > The only thing that I spot is that you use both auto-commit with 900 sec > > frequency AND commitWithin. Solr is smart enough to skip empty commits. > But > > if auto-commit kicks in during the doc add / delete, there will be at > least > > two commits ongoing. Could you change you Full recovery case to commit > > eventually from the client code? Then you won't need the autoCommit > section. > > The autoCommit config has openSearcher=false, so I wouldn't touch it. > With version 4.0 and later, autoCommit with openSearcher=false should be > part of *every* config. The commitWithin parameter *will* open a new > searcher, so it has no conflict with an autoCommit config using > openSearcher=false. > > Thanks, > Shawn > >
Re: [Migration Solr4 to Solr5] Collection reload error
On 3/10/2016 3:05 AM, Dmitry Kan wrote: > The only thing that I spot is that you use both auto-commit with 900 sec > frequency AND commitWithin. Solr is smart enough to skip empty commits. But > if auto-commit kicks in during the doc add / delete, there will be at least > two commits ongoing. Could you change you Full recovery case to commit > eventually from the client code? Then you won't need the autoCommit section. The autoCommit config has openSearcher=false, so I wouldn't touch it. With version 4.0 and later, autoCommit with openSearcher=false should be part of *every* config. The commitWithin parameter *will* open a new searcher, so it has no conflict with an autoCommit config using openSearcher=false. Thanks, Shawn
Re: [Migration Solr4 to Solr5] Collection reload error
Hi,
The only thing that I spot is that you use both auto-commit with 900 sec
frequency AND commitWithin. Solr is smart enough to skip empty commits. But
if auto-commit kicks in during the doc add / delete, there will be at least
two commits ongoing. Could you change you Full recovery case to commit
eventually from the client code? Then you won't need the autoCommit section.
On Mon, Mar 7, 2016 at 1:11 PM, Gerald Reinhart
wrote:
>
> Hi,
>
> To give you some context, we are migrating from Solr4 and solr5,
> the client code and the configuration haven't changed but now we are
> facing this problem. We have already checked the commit behaviour
> configuration and it seems good.
>
> Here it is :
>
> Server side, we have 2 collections (main and temp with blue and green
> aliases) :
>
>solrconfig.xml:
>
>
>
> (...)
>
>90
>false
>
>
>
>
>
>
> Client side, we have 2 different modes:
>
> 1 - Full recovery :
>
> - Delete all documents from the temp collection
> solrClient.deleteByQuery("*:*")
>
> - Add all new documents in temp collection (can be more
> than 5Millions),
> solrClient.add(doc, -1) // commitWithinMs == -1
>
> - Commit when all documents are added
> solrClient.commit(false,false) // waitFlush == false ,
> waitSearcher == false
>
> - Swap blue and green using "create alias" command
>
> - Reload the temp collection to clean the cache. This is
> at this point we have the issue.
>
> 2 - Incremental :
>
> - Add or delete documents from the main collection
>solrClient.add(doc, 180) // commitWithin
> == 30 mn
>solrClient.deleteById(doc, 180) // commitWithin == 30 mn
>
> Maybe you will spot something obviously wrong ?
>
> Thanks
>
> Gérald and Elodie
>
>
>
>
> On 03/04/2016 12:41 PM, Dmitry Kan wrote:
>
>> Hi,
>>
>> Check the the autoCommit and autoSoftCommit nodes in the solrconfig.xml.
>> Set them to reasonable values. The idea is that if you commit too often,
>> searchers will be warmed up and thrown away. If at any point in time you
>> get overlapping commits, there will be several searchers sitting on the
>> deck.
>>
>> Dmitry
>>
>> On Mon, Feb 29, 2016 at 4:20 PM, Gerald Reinhart <
>> gerald.reinh...@kelkoo.com
>>
>>> wrote:
>>> Hi,
>>>
>>> We are facing an issue during a migration from Solr4 to Solr5.
>>>
>>> Given
>>> - migration from solr 4.10.4 to 5.4.1
>>> - 2 collections
>>> - cloud with one leader and several replicas
>>> - in solrconfig.xml: maxWarmingSearchers=1
>>> - no code change
>>>
>>> When collection reload using /admin/collections using solrj
>>>
>>> Then
>>>
>>> 2016-02-29 13:42:49,011 [http-8080-3] INFO
>>> org.apache.solr.core.CoreContainer:reload:848 - Reloading SolrCore
>>> 'fr_blue' using configuration from collection fr_blue
>>> 2016-02-29 13:42:45,428 [http-8080-6] INFO
>>> org.apache.solr.search.SolrIndexSearcher::237 - Opening
>>> Searcher@58b65fc[fr_blue] main
>>> (...)
>>> 2016-02-29 13:42:49,077 [http-8080-3] WARN
>>> org.apache.solr.core.SolrCore:getSearcher:1762 - [fr_blue] Error
>>> opening new searcher. exceeded limit of maxWarmingSearchers=1, try again
>>> later.
>>> 2016-02-29 13:42:49,091 [http-8080-3] ERROR
>>> org.apache.solr.handler.RequestHandlerBase:log:139 -
>>> org.apache.solr.common.SolrException: Error handling 'reload' action
>>> at
>>>
>>>
>>> org.apache.solr.handler.admin.CoreAdminHandler.handleReloadAction(CoreAdminHandler.java:770)
>>> at
>>>
>>>
>>> org.apache.solr.handler.admin.CoreAdminHandler.handleRequestInternal(CoreAdminHandler.java:230)
>>> at
>>>
>>>
>>> org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:184)
>>> at
>>>
>>>
>>> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:156)
>>> at
>>>
>>>
>>> org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:664)
>>> at
>>> org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:438)
>>> at
>>>
>>>
>>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:223)
>>> at
>>>
>>>
>>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:181)
>>> at
>>>
>>>
>>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
>>> at
>>>
>>>
>>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
>>> at
>>>
>>>
>>> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
>>> at
>>>
>>>
>>> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
>>> at
>>>
>>>
>>> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
>>> at
>>>
>>>
>>> org.apach
Re: [Migration Solr4 to Solr5] Collection reload error
Hi,
To give you some context, we are migrating from Solr4 and solr5,
the client code and the configuration haven't changed but now we are
facing this problem. We have already checked the commit behaviour
configuration and it seems good.
Here it is :
Server side, we have 2 collections (main and temp with blue and green
aliases) :
solrconfig.xml:
(...)
90
false
Client side, we have 2 different modes:
1 - Full recovery :
- Delete all documents from the temp collection
solrClient.deleteByQuery("*:*")
- Add all new documents in temp collection (can be more
than 5Millions),
solrClient.add(doc, -1) // commitWithinMs == -1
- Commit when all documents are added
solrClient.commit(false,false) // waitFlush == false ,
waitSearcher == false
- Swap blue and green using "create alias" command
- Reload the temp collection to clean the cache. This is
at this point we have the issue.
2 - Incremental :
- Add or delete documents from the main collection
solrClient.add(doc, 180) // commitWithin
== 30 mn
solrClient.deleteById(doc, 180) // commitWithin == 30 mn
Maybe you will spot something obviously wrong ?
Thanks
Gérald and Elodie
On 03/04/2016 12:41 PM, Dmitry Kan wrote:
Hi,
Check the the autoCommit and autoSoftCommit nodes in the solrconfig.xml.
Set them to reasonable values. The idea is that if you commit too often,
searchers will be warmed up and thrown away. If at any point in time you
get overlapping commits, there will be several searchers sitting on the
deck.
Dmitry
On Mon, Feb 29, 2016 at 4:20 PM, Gerald Reinhart
wrote:
Hi,
We are facing an issue during a migration from Solr4 to Solr5.
Given
- migration from solr 4.10.4 to 5.4.1
- 2 collections
- cloud with one leader and several replicas
- in solrconfig.xml: maxWarmingSearchers=1
- no code change
When collection reload using /admin/collections using solrj
Then
2016-02-29 13:42:49,011 [http-8080-3] INFO
org.apache.solr.core.CoreContainer:reload:848 - Reloading SolrCore
'fr_blue' using configuration from collection fr_blue
2016-02-29 13:42:45,428 [http-8080-6] INFO
org.apache.solr.search.SolrIndexSearcher::237 - Opening
Searcher@58b65fc[fr_blue] main
(...)
2016-02-29 13:42:49,077 [http-8080-3] WARN
org.apache.solr.core.SolrCore:getSearcher:1762 - [fr_blue] Error
opening new searcher. exceeded limit of maxWarmingSearchers=1, try again
later.
2016-02-29 13:42:49,091 [http-8080-3] ERROR
org.apache.solr.handler.RequestHandlerBase:log:139 -
org.apache.solr.common.SolrException: Error handling 'reload' action
at
org.apache.solr.handler.admin.CoreAdminHandler.handleReloadAction(CoreAdminHandler.java:770)
at
org.apache.solr.handler.admin.CoreAdminHandler.handleRequestInternal(CoreAdminHandler.java:230)
at
org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:184)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:156)
at
org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:664)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:438)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:223)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:181)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at
org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:555)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:857)
at
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588)
at
org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.solr.common.SolrException: Unable to reload core
[fr_blue]
at
org.apache.solr.core.CoreContainer.reload(CoreContainer.java:854)
at
org.apache.solr.handler.admin.CoreAdminHandler.handleReloadAction(CoreAdminHandler
Re: [Migration Solr4 to Solr5] Collection reload error
Hi, Check the the autoCommit and autoSoftCommit nodes in the solrconfig.xml. Set them to reasonable values. The idea is that if you commit too often, searchers will be warmed up and thrown away. If at any point in time you get overlapping commits, there will be several searchers sitting on the deck. Dmitry On Mon, Feb 29, 2016 at 4:20 PM, Gerald Reinhart wrote: > Hi, > >We are facing an issue during a migration from Solr4 to Solr5. > > Given >- migration from solr 4.10.4 to 5.4.1 >- 2 collections >- cloud with one leader and several replicas >- in solrconfig.xml: maxWarmingSearchers=1 >- no code change > > When collection reload using /admin/collections using solrj > > Then > > 2016-02-29 13:42:49,011 [http-8080-3] INFO > org.apache.solr.core.CoreContainer:reload:848 - Reloading SolrCore > 'fr_blue' using configuration from collection fr_blue > 2016-02-29 13:42:45,428 [http-8080-6] INFO > org.apache.solr.search.SolrIndexSearcher::237 - Opening > Searcher@58b65fc[fr_blue] main > (...) > 2016-02-29 13:42:49,077 [http-8080-3] WARN > org.apache.solr.core.SolrCore:getSearcher:1762 - [fr_blue] Error > opening new searcher. exceeded limit of maxWarmingSearchers=1, try again > later. > 2016-02-29 13:42:49,091 [http-8080-3] ERROR > org.apache.solr.handler.RequestHandlerBase:log:139 - > org.apache.solr.common.SolrException: Error handling 'reload' action > at > > org.apache.solr.handler.admin.CoreAdminHandler.handleReloadAction(CoreAdminHandler.java:770) > at > > org.apache.solr.handler.admin.CoreAdminHandler.handleRequestInternal(CoreAdminHandler.java:230) > at > > org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:184) > at > > org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:156) > at > > org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:664) > at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:438) > at > > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:223) > at > > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:181) > at > > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) > at > > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) > at > > org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233) > at > > org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) > at > > org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) > at > > org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) > at > org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:555) > at > > org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) > at > org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298) > at > org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:857) > at > > org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588) > at > org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489) > at java.lang.Thread.run(Thread.java:745) > Caused by: org.apache.solr.common.SolrException: Unable to reload core > [fr_blue] > at > org.apache.solr.core.CoreContainer.reload(CoreContainer.java:854) > at > > org.apache.solr.handler.admin.CoreAdminHandler.handleReloadAction(CoreAdminHandler.java:768) > ... 20 more > Caused by: org.apache.solr.common.SolrException: Error opening new > searcher. exceeded limit of maxWarmingSearchers=1, try again later. > at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1764) > at org.apache.solr.core.SolrCore.reload(SolrCore.java:474) > at > org.apache.solr.core.CoreContainer.reload(CoreContainer.java:849) > ... 21 more > > > Thanks > > > Gérald and Elodie > > > Kelkoo SAS > Société par Actions Simplifiée > Au capital de € 4.168.964,30 > Siège social : 158 Ter Rue du Temple 75003 Paris > 425 093 069 RCS Paris > > Ce message et les pièces jointes sont confidentiels et établis à > l'attention exclusive de leurs destinataires. Si vous n'êtes pas le > destinataire de ce message, merci de le détruire et d'en avertir > l'expéditeur. > -- Dmitry Kan Luke Toolbox: http://github.com/DmitryKey/luke Blog: http://dmitrykan.blogspot.com Twitter: http://twitter.com/dmitrykan SemanticAnalyzer: www.semanticanalyzer.info
[Migration Solr4 to Solr5] Collection reload error
Hi, We are facing an issue during a migration from Solr4 to Solr5. Given - migration from solr 4.10.4 to 5.4.1 - 2 collections - cloud with one leader and several replicas - in solrconfig.xml: maxWarmingSearchers=1 - no code change When collection reload using /admin/collections using solrj Then 2016-02-29 13:42:49,011 [http-8080-3] INFO org.apache.solr.core.CoreContainer:reload:848 - Reloading SolrCore 'fr_blue' using configuration from collection fr_blue 2016-02-29 13:42:45,428 [http-8080-6] INFO org.apache.solr.search.SolrIndexSearcher::237 - Opening Searcher@58b65fc[fr_blue] main (...) 2016-02-29 13:42:49,077 [http-8080-3] WARN org.apache.solr.core.SolrCore:getSearcher:1762 - [fr_blue] Error opening new searcher. exceeded limit of maxWarmingSearchers=1, try again later. 2016-02-29 13:42:49,091 [http-8080-3] ERROR org.apache.solr.handler.RequestHandlerBase:log:139 - org.apache.solr.common.SolrException: Error handling 'reload' action at org.apache.solr.handler.admin.CoreAdminHandler.handleReloadAction(CoreAdminHandler.java:770) at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestInternal(CoreAdminHandler.java:230) at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:184) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:156) at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:664) at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:438) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:223) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:181) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:555) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:857) at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588) at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.solr.common.SolrException: Unable to reload core [fr_blue] at org.apache.solr.core.CoreContainer.reload(CoreContainer.java:854) at org.apache.solr.handler.admin.CoreAdminHandler.handleReloadAction(CoreAdminHandler.java:768) ... 20 more Caused by: org.apache.solr.common.SolrException: Error opening new searcher. exceeded limit of maxWarmingSearchers=1, try again later. at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1764) at org.apache.solr.core.SolrCore.reload(SolrCore.java:474) at org.apache.solr.core.CoreContainer.reload(CoreContainer.java:849) ... 21 more Thanks Gérald and Elodie Kelkoo SAS Société par Actions Simplifiée Au capital de € 4.168.964,30 Siège social : 158 Ter Rue du Temple 75003 Paris 425 093 069 RCS Paris Ce message et les pièces jointes sont confidentiels et établis à l'attention exclusive de leurs destinataires. Si vous n'êtes pas le destinataire de ce message, merci de le détruire et d'en avertir l'expéditeur.
