RE: solrcloud backup null pointer exeption

[Will Martin]

can you share:
solr version?
zookeeper ensemble type
number of shards in the collection?
distribution of shard replicas in the SolrCloud? 

from there the most obvious question is whether the stack trace is from the 
shard leader for the collection or the localhost; if they are not the same? 
There should be significantly more logging associated with this operation?

What version of NFS does your pod run?

-Original Message-
From: rffleaie  
Sent: Thursday, July 25, 2019 6:03 PM
To: solr-user@lucene.apache.org
Subject: solrcloud backup null pointer exeption

I have a solrcloud cluster installed on k8s.
I have created a nfs PVC that is mounted under /backup of every pod of the solr 
cluster.

When I start the backup with 

https://nam05.safelinks.protection.outlook.com/?url=http%3A%2F%2F127.0.0.1%3A8983%2Fsolr%2Fadmin%2Fcollections%3Faction%3DBACKUP%26name%3Dtest%26collection%3Dcollection_name%26location%3D%2Fbackupdata=02%7C01%7C%7C3bc3ddc47b6b42e7677708d7114bf3b6%7C84df9e7fe9f640afb435%7C1%7C0%7C636996890213331873sdata=%2BlF%2FDerZFDaANchFbcVXtJnN98DJh4UIxIuyacYb1Vo%3Dreserved=0

I receive the follow error, someone have the same issue ???




  "Operation backup caused
exception:":"org.apache.solr.common.SolrException:org.apache.solr.common.SolrException:
Could not backup all shards",
  "exception":{
"msg":"Could not backup all shards",
"rspCode":500},
  "error":{
"metadata":[
  "error-class","org.apache.solr.common.SolrException",
  "root-error-class","org.apache.solr.common.SolrException"],
"msg":"Could not backup all shards",
"trace":"org.apache.solr.common.SolrException: Could not backup all 
shards\n\tat 
org.apache.solr.client.solrj.SolrResponse.getException(SolrResponse.java:53)\n\tat
org.apache.solr.handler.admin.CollectionsHandler.invokeAction(CollectionsHandler.java:274)\n\tat
org.apache.solr.handler.admin.CollectionsHandler.handleRequestBody(CollectionsHandler.java:246)\n\tat
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)\n\tat
org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:734)\n\tat
org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:715)\n\tat
org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:496)\n\tat
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:377)\n\tat
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:323)\n\tat
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1634)\n\tat
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)\n\tat
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)\n\tat
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)\n\tat
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)\n\tat
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1317)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)\n\tat
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)\n\tat
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)\n\tat
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1219)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)\n\tat
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:219)\n\tat
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)\n\tat
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)\n\tat
org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)\n\tat
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)\n\tat
org.eclipse.jetty.server.Server.handle(Server.java:531)\n\tat
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:352)\n\tat
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)\n\tat
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:281)\n\tat
org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)\n\tat
org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)\n\tat
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)\n\tat
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)\n\tat
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)\n\tat

Re: solrcloud backup null pointer exeption

[Erick Erickson]

Follow the instructions here: 
http://lucene.apache.org/solr/community.html#mailing-lists-irc
. You must use the _exact_ same e-mail as you used to subscribe.

If the initial try doesn't work and following the suggestions at the "problems" 
link doesn't work for you, let us know. But note you need to show us the 
_entire_ return header to allow anyone to diagnose the problem.

Best,
Erick

> On Jul 26, 2019, at 3:44 PM, Anthony Gatlin  wrote:
> 
> unsubscribe
> 
> On Thu, Jul 25, 2019 at 5:03 PM rffleaie  wrote:
> 
>> I have a solrcloud cluster installed on k8s.
>> I have created a nfs PVC that is mounted under /backup of every pod of the
>> solr cluster.
>> 
>> When I start the backup with
>> 
>> 
>> http://127.0.0.1:8983/solr/admin/collections?action=BACKUP=test=collection_name=/backup
>> 
>> I receive the follow error, someone have the same issue ???
>> 
>> 
>> 
>> 
>>  "Operation backup caused
>> 
>> exception:":"org.apache.solr.common.SolrException:org.apache.solr.common.SolrException:
>> Could not backup all shards",
>>  "exception":{
>>"msg":"Could not backup all shards",
>>"rspCode":500},
>>  "error":{
>>"metadata":[
>>  "error-class","org.apache.solr.common.SolrException",
>>  "root-error-class","org.apache.solr.common.SolrException"],
>>"msg":"Could not backup all shards",
>>"trace":"org.apache.solr.common.SolrException: Could not backup all
>> shards\n\tat
>> 
>> org.apache.solr.client.solrj.SolrResponse.getException(SolrResponse.java:53)\n\tat
>> 
>> org.apache.solr.handler.admin.CollectionsHandler.invokeAction(CollectionsHandler.java:274)\n\tat
>> 
>> org.apache.solr.handler.admin.CollectionsHandler.handleRequestBody(CollectionsHandler.java:246)\n\tat
>> 
>> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)\n\tat
>> 
>> org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:734)\n\tat
>> 
>> org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:715)\n\tat
>> org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:496)\n\tat
>> 
>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:377)\n\tat
>> 
>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:323)\n\tat
>> 
>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1634)\n\tat
>> 
>> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)\n\tat
>> 
>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)\n\tat
>> 
>> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)\n\tat
>> 
>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)\n\tat
>> 
>> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)\n\tat
>> 
>> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)\n\tat
>> 
>> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)\n\tat
>> 
>> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1317)\n\tat
>> 
>> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)\n\tat
>> 
>> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)\n\tat
>> 
>> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)\n\tat
>> 
>> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)\n\tat
>> 
>> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1219)\n\tat
>> 
>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)\n\tat
>> 
>> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:219)\n\tat
>> 
>> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)\n\tat
>> 
>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)\n\tat
>> 
>> org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)\n\tat
>> 
>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)\n\tat
>> org.eclipse.jetty.server.Server.handle(Server.java:531)\n\tat
>> org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:352)\n\tat
>> 
>> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)\n\tat
>> org.eclipse.jetty.io
>> .AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:281)\n\tat
>> org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)\n\tat
>> org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)\n\tat
>> 
>> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)\n\tat
>> 
>> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)\n\tat
>> 
>> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)\n\tat
>> 
>> 

Re: [SOLR] - Best Practices/node down

[Rodrigo Oliveira]

Hi,

So sorry, but my explanation is incompleted.

My primary database is mongodb, I am using solr only search.

Than you for tips and tricks. It,s my first time using solr.

Regards



Em sex, 26 de jul de 2019 11:36, Shawn Heisey 
escreveu:

> On 7/26/2019 7:16 AM, Rodrigo Oliveira wrote:
> > I have a cluster from Solr with Zookeeper (5 nodes - 48 Gb each node -
> > Xms:28 Gb - Xmx:32 Gb). The bigger problem is my environment, because I
> am
> > in a process of migration from MYSQL to SOLR.
>
> Solr is not intended as a primary data store.  There are things related
> to primary data store usage that MySQL can do which Solr either can't do
> at all or has a difficult time doing.  Databases and search engines are
> each optimized for entirely different tasks.
>
> It is reasonable to have your data in both a database and a search
> engine ... but to *switch* from a database to Solr sounds like a really
> bad idea.
>
> Don't get me wrong... I'm one of the biggest fans of Solr you'll come
> across... but I am aware of its limitations as well as its strengths.
>
> If the amount of data involved is small, using Solr as a primary data
> store might prove to be worthwhile ... but if I examine everything you
> have said, it doesn't sound like the amount of data is small.
>
> > Running OOM killer script for process 21252 for Solr on port 8983
> > Killed process 21252
> >
> > I got it. Out Of Memory.
> >
> > Any workaround to this case? Any suggestion? Any tips? 10% of migration
> and
> > Out Of Memory. Why the Solr it's consumed all memory?
>
> This log is generated by the OOM killer script.  It does not output any
> indication about WHY the error occurred.  It simply indicates when the
> error occurred and what it did in response -- which is to terminate Solr.
>
> There are several possible reasons for Java's OOME.  Only a couple of
> those actually involve running out of memory.  It might not be memory at
> all.  But to find out, you will need to find the actual
> OutOfMemoryException in solr.log or one of the rotated versions
> (assuming it got logged at all), which will indicate the root issue.
>
> There are precisely two solutions for OOME, and frequently only one of
> them is actually possible:  Increase the resource that ran out, or
> figure out how to change the configuration so the program requires less
> of that resource.  As already mentioned, you will need to figure out
> which resource was depleted.
>
> If you can't find the actual exception, analyzing the GC log that Solr
> writes might help determine whether the depleted resource was heap memory.
>
> Thanks,
> Shawn
>

Re: solrcloud backup null pointer exeption

[Anthony Gatlin]

unsubscribe

On Thu, Jul 25, 2019 at 5:03 PM rffleaie  wrote:

> I have a solrcloud cluster installed on k8s.
> I have created a nfs PVC that is mounted under /backup of every pod of the
> solr cluster.
>
> When I start the backup with
>
>
> http://127.0.0.1:8983/solr/admin/collections?action=BACKUP=test=collection_name=/backup
>
> I receive the follow error, someone have the same issue ???
>
>
>
>
>   "Operation backup caused
>
> exception:":"org.apache.solr.common.SolrException:org.apache.solr.common.SolrException:
> Could not backup all shards",
>   "exception":{
> "msg":"Could not backup all shards",
> "rspCode":500},
>   "error":{
> "metadata":[
>   "error-class","org.apache.solr.common.SolrException",
>   "root-error-class","org.apache.solr.common.SolrException"],
> "msg":"Could not backup all shards",
> "trace":"org.apache.solr.common.SolrException: Could not backup all
> shards\n\tat
>
> org.apache.solr.client.solrj.SolrResponse.getException(SolrResponse.java:53)\n\tat
>
> org.apache.solr.handler.admin.CollectionsHandler.invokeAction(CollectionsHandler.java:274)\n\tat
>
> org.apache.solr.handler.admin.CollectionsHandler.handleRequestBody(CollectionsHandler.java:246)\n\tat
>
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)\n\tat
>
> org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:734)\n\tat
>
> org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:715)\n\tat
> org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:496)\n\tat
>
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:377)\n\tat
>
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:323)\n\tat
>
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1634)\n\tat
>
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)\n\tat
>
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)\n\tat
>
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)\n\tat
>
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)\n\tat
>
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)\n\tat
>
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)\n\tat
>
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)\n\tat
>
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1317)\n\tat
>
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)\n\tat
>
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)\n\tat
>
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)\n\tat
>
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)\n\tat
>
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1219)\n\tat
>
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)\n\tat
>
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:219)\n\tat
>
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)\n\tat
>
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)\n\tat
>
> org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)\n\tat
>
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)\n\tat
> org.eclipse.jetty.server.Server.handle(Server.java:531)\n\tat
> org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:352)\n\tat
>
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)\n\tat
> org.eclipse.jetty.io
> .AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:281)\n\tat
> org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)\n\tat
> org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)\n\tat
>
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)\n\tat
>
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)\n\tat
>
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)\n\tat
>
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)\n\tat
>
> org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)\n\tat
>
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)\n\tat
>
> org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)\n\tat
> java.base/java.lang.Thread.run(Thread.java:834)\n",
> "code":500}}
>
>
>
> --
> Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
>

Re: [SOLR] - Best Practices/node down

[Erick Erickson]

Also, you say "In this case, I've had just 10% of my migration completed”. 
Exactly how are you migrating the data? And how much data are you moving? In 
particular, what are you commit settings?

Best,
Erick

> On Jul 26, 2019, at 9:16 AM, Rodrigo Oliveira  
> wrote:
> 
> In this case, I've had just 10% of my migration completed

Re: Quepid, the relevance testing tool for Solr, released as open source

[Doug Turnbull]

Quepid has been really powerful for helping teams bootstsrap on relevance
problems for teams just getting started and allowing the product & tech
teams to collaborate.

Often we start with a dozen or so queries, get some ratings, create a bit
of success criteria and tune in Quepid. You can do this in a 'sprint' like
structure, where the next 'sprint' you tackle the next dozen or so queries,
ensure the stuff you fixed before is still working, and make progress on
the next set of use cases. During the process, you get a lot of feedback
you can give to the product team about 'why' results are ranked the way
they work in a relevance tuning sandbox.

I would contrast this with RRE, that you might have heard of, which has
different strengths. For example, if I wanted to automate or do a bit more
"CI" kinds of relevance evaluation, without the interactive "IDE" that
Quepid provides.

Quarite is another tool in the genre from Tim Allison. I (think) it also
adds features for genetic optimization of relevance parameters

Very cool to see the ecosystem of relevance tuning tools growing!

https://github.com/SeaseLtd/rated-ranking-evaluator

https://github.com/mitre/quaerite

-Doug

On Fri, Jul 26, 2019 at 8:03 AM Charlie Hull  wrote:

> Hi all,
>
> We've finally made Quepid, the relevance testing tool, open source.
> There's also a free hosted version at www.quepid.com . Looking forward
> to contributions driving the project forward! Quepid is a way to record
> human relevance judgements, and then to experiment with query tuning and
> see the results in real time.
>
> More details at
>
> https://opensourceconnections.com/blog/2019/07/25/2019-07-22-quepid-is-now-open-source/
>
> (also particularly pleased to see Luwak, the stored query engine we
> built at Flax become part of Lucene - it's a great day for open source!)
>
> Cheers
>
> Charlie
>
> --
> Charlie Hull
> Flax - Open Source Enterprise Search
>
> tel/fax: +44 (0)8700 118334
> mobile:  +44 (0)7767 825828
> web: www.flax.co.uk
>
>

-- 
*Doug Turnbull **| CTO* | OpenSource Connections
, LLC | 240.476.9983
Author: Relevant Search 
This e-mail and all contents, including attachments, is considered to be
Company Confidential unless explicitly stated otherwise, regardless
of whether attachments are marked as such.

Re: [SOLR] - Best Practices/node down

[Shawn Heisey]

On 7/26/2019 7:16 AM, Rodrigo Oliveira wrote:

I have a cluster from Solr with Zookeeper (5 nodes - 48 Gb each node -
Xms:28 Gb - Xmx:32 Gb). The bigger problem is my environment, because I am
in a process of migration from MYSQL to SOLR.


Solr is not intended as a primary data store.  There are things related 
to primary data store usage that MySQL can do which Solr either can't do 
at all or has a difficult time doing.  Databases and search engines are 
each optimized for entirely different tasks.


It is reasonable to have your data in both a database and a search 
engine ... but to *switch* from a database to Solr sounds like a really 
bad idea.


Don't get me wrong... I'm one of the biggest fans of Solr you'll come 
across... but I am aware of its limitations as well as its strengths.


If the amount of data involved is small, using Solr as a primary data 
store might prove to be worthwhile ... but if I examine everything you 
have said, it doesn't sound like the amount of data is small.



Running OOM killer script for process 21252 for Solr on port 8983
Killed process 21252

I got it. Out Of Memory.

Any workaround to this case? Any suggestion? Any tips? 10% of migration and
Out Of Memory. Why the Solr it's consumed all memory?


This log is generated by the OOM killer script.  It does not output any 
indication about WHY the error occurred.  It simply indicates when the 
error occurred and what it did in response -- which is to terminate Solr.


There are several possible reasons for Java's OOME.  Only a couple of 
those actually involve running out of memory.  It might not be memory at 
all.  But to find out, you will need to find the actual 
OutOfMemoryException in solr.log or one of the rotated versions 
(assuming it got logged at all), which will indicate the root issue.


There are precisely two solutions for OOME, and frequently only one of 
them is actually possible:  Increase the resource that ran out, or 
figure out how to change the configuration so the program requires less 
of that resource.  As already mentioned, you will need to figure out 
which resource was depleted.


If you can't find the actual exception, analyzing the GC log that Solr 
writes might help determine whether the depleted resource was heap memory.


Thanks,
Shawn

[SOLR] - Best Practices/node down

[Rodrigo Oliveira]

Hi,

Anyone can help me?

I have a cluster from Solr with Zookeeper (5 nodes - 48 Gb each node -
Xms:28 Gb - Xmx:32 Gb). The bigger problem is my environment, because I am
in a process of migration from MYSQL to SOLR.

In this case, I've had just 10% of my migration completed, and the problem
it ocurred.

At the log, I've had the message.

*LOG1*

2399323.AUTOCREATED/lang/stopwords_sv.txt
2019-07-24 22:37:00.620 INFO
 (OverseerThreadFactory-13-thread-75-processing-n:54.54.54.152:8983_solr) [
  ] o.a.s.c.c.ZkConfigManager Copying zk node
/configs/_default/lang/stopwords_fi.txt to
/configs/list_2399323.AUTOCREATED/lang/stopwords_fi.txt
2019-07-24 22:37:00.681 INFO  (qtp1157726741-24363) [
x:list_2356456_shard2_replica_n4] o.a.s.h.a.CoreAdminOperation core create
command
qt=/admin/cores=core_node7=list_2356456.AUTOCREATED=true=list_2356456_shard2_replica_n4=CREATE=2=list_2356456=shard2=javabin=2=NRT
2019-07-24 22:37:00.760 INFO
 (OverseerThreadFactory-13-thread-75-processing-n:54.54.54.152:8983_solr) [
  ] o.a.s.c.c.ZkConfigManager Copying zk node
/configs/_default/lang/hyphenations_ga.txt to
/configs/list_2399323.AUTOCREATED/lang/hyphenations_ga.txt
2019-07-24 22:37:00.878 INFO  (qtp1157726741-24771) [c:list_2389896
s:shard1 r:core_node6 x:list_2389896_shard1_replica_n2]
o.a.s.c.ZkController Persisted config data to node
/configs/list_2389896.AUTOCREATED/managed-schema
2019-07-24 22:37:00.878 INFO  (zkCallback-5-thread-59) [   ]
o.a.s.s.ZkIndexSchemaReader A schema change: WatchedEvent
state:SyncConnected type:NodeDataChanged
path:/configs/list_2389896.AUTOCREATED/managed-schema, has occurred -
updating schema from ZooKeeper ...
2019-07-24 22:37:00.923 INFO
 (OverseerThreadFactory-13-thread-75-processing-n:54.54.54.152:8983_solr) [
  ] o.a.s.c.c.ZkConfigManager Copying zk node
/configs/_default/lang/stopwords_gl.txt to
/configs/list_2399323.AUTOCREATED/lang/stopwords_gl.txt
2019-07-24 22:37:00.923 INFO  (Thread-1476) [   ] o.a.s.c.SolrCore config
update listener called for core list_2389896_shard1_replica_n2
2019-07-24 22:37:01.033 INFO  (Thread-1476) [   ] o.a.s.c.SolrCore core
reload list_2389896_shard1_replica_n2
2019-07-24 22:37:01.068 INFO
 (OverseerThreadFactory-13-thread-75-processing-n:54.54.54.152:8983_solr) [
  ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/params.json
to /configs/list_2399323.AUTOCREATED/params.json
2019-07-24 22:37:01.134 INFO  (Thread-1471) [   ] o.a.s.c.CoreContainer
Reloading SolrCore 'list_1217965_shard2_replica_n6' using configuration
from collection list_1217965
2019-07-24 22:37:01.150 INFO  (Thread-1471) [c:list_1217965 s:shard2
r:core_node8 x:list_1217965_shard2_replica_n6] o.a.s.m.r.SolrJmxReporter
JMX monitoring for 'solr.core.list_1217965.shard2.replica_n6' (registry
'solr.core.list_1217965.shard2.replica_n6') enabled at server:
com.sun.jmx.mbeanserver.JmxMBeanServer@62bd765
2019-07-24 22:37:01.150 INFO  (Thread-1471) [c:list_1217965 s:shard2
r:core_node8 x:list_1217965_shard2_replica_n6] o.a.s.c.SolrCore
[[list_1217965_shard2_replica_n6] ] Opening new SolrCore at
[/solr/server/solr/list_1217965_shard2_replica_n6],
dataDir=[/solr/server/solr/list_1217965_shard2_replica_n6/data/]
2019-07-24 22:37:01.152 INFO  (Thread-1471) [c:list_1217965 s:shard2
r:core_node8 x:list_1217965_shard2_replica_n6] o.a.s.r.XSLTResponseWriter
xsltCacheLifetimeSeconds=5

*LOG2*

Running OOM killer script for process 21252 for Solr on port 8983
Killed process 21252

I got it. Out Of Memory.

Any workaround to this case? Any suggestion? Any tips? 10% of migration and
Out Of Memory. Why the Solr it's consumed all memory?

Regards,

Quepid, the relevance testing tool for Solr, released as open source

[Charlie Hull]

Hi all,

We've finally made Quepid, the relevance testing tool, open source. 
There's also a free hosted version at www.quepid.com . Looking forward 
to contributions driving the project forward! Quepid is a way to record 
human relevance judgements, and then to experiment with query tuning and 
see the results in real time.


More details at 
https://opensourceconnections.com/blog/2019/07/25/2019-07-22-quepid-is-now-open-source/


(also particularly pleased to see Luwak, the stored query engine we 
built at Flax become part of Lucene - it's a great day for open source!)


Cheers

Charlie

--
Charlie Hull
Flax - Open Source Enterprise Search

tel/fax: +44 (0)8700 118334
mobile:  +44 (0)7767 825828
web: www.flax.co.uk

Re: Indexig excel (xlsx) file into SOLR 8.1.1

[Charlie Hull]

Simpler possibly, but not necessarily reliable. If you do everything 
inside Solr's DIH with Tika under the hood to extract data from Excel, a 
malformed Excel file could kill Tika and bring down your entire Solr 
cluster. Far better to do it outside of Solr as this blog describes: 
https://lucidworks.com/post/indexing-with-solrj/


If you want to see what Tika does to your Excel examples this is quite a 
neat way to experiment: https://okfnlabs.org/projects/tika-server/


Cheers

Charlie

On 26/07/2019 09:44, Vipul Bahuguna wrote:

Hi Charlie,

Thanks for your suggestion,  but I will have thousands of these files
coming from different sources. It would become very tedious if I have to
first convert them to csv and then run liny by line.

I was hoping if there could be a simpker way to achieve these using DIH
which I thought can be configured to read and ingest MS Excel (xlsx)
files.

I am not too sure of how the configuration file would look like.

Any pointers are welcome. Thanks!

On Fri, 26 Jul, 2019, 1:56 PM Charlie Hull,  wrote:


Convert the Excel file to a CSV and then write a teeny script to go
through it line by line and submit to Solr over HTTP? Tika would
probably work but it's a lot of heavy lifting for what seems to me like
a simple problem.

Cheers

Charlie

On 26/07/2019 09:19, Vipul Bahuguna wrote:

Hi Guys - can anyone suggest how to achieve this?
I have understood how to insert json documents. So one alternative that
comes to my mind is that I can convert the rows in my excel to json

format

with the header of my excel file becoming the json keys (corresponding to
the fields I have defined in my managed-schema.xml). And then each cell

in

the excel file will become the value of this field.

However, I am sure there must be a better way and directly ingesting the
excel file to achieve the same. I was trying to reach about DIH and

Apache

Tika, but I am not very sure of how the configuration works.

My sample excel file has 4 columns namely -
1. First Name
2. Last Name
3. Phone
4. Website Link

I want to index these fields into SOLR in a way that all these columns
become my solr schema fields and later I can search based on these

fields.

Any suggestions please.

thanks !


--
Charlie Hull
Flax - Open Source Enterprise Search

tel/fax: +44 (0)8700 118334
mobile:  +44 (0)7767 825828
web: www.flax.co.uk




--
Charlie Hull
Flax - Open Source Enterprise Search

tel/fax: +44 (0)8700 118334
mobile:  +44 (0)7767 825828
web: www.flax.co.uk

Field value different over replicas

[Dominique Bejean]

Hi,

We have a date field with default set to “now”. For this field, some
documents of the collection don’t have the same value in all replicas. The
difference can be 3 or 4 minutes !
The collection has 1 shard and 2 NRT replicas. Solr version is 7.5.
Collection is populated with DIH.

Any ideas for this issue ?

NRT Replica type and GC issue could be a explanation, but only for a
difference of a few ms

Regards

Dominique

Re: Indexig excel (xlsx) file into SOLR 8.1.1

[Vipul Bahuguna]

Hi Charlie,

Thanks for your suggestion,  but I will have thousands of these files
coming from different sources. It would become very tedious if I have to
first convert them to csv and then run liny by line.

I was hoping if there could be a simpker way to achieve these using DIH
which I thought can be configured to read and ingest MS Excel (xlsx)
files.

I am not too sure of how the configuration file would look like.

Any pointers are welcome. Thanks!

On Fri, 26 Jul, 2019, 1:56 PM Charlie Hull,  wrote:

> Convert the Excel file to a CSV and then write a teeny script to go
> through it line by line and submit to Solr over HTTP? Tika would
> probably work but it's a lot of heavy lifting for what seems to me like
> a simple problem.
>
> Cheers
>
> Charlie
>
> On 26/07/2019 09:19, Vipul Bahuguna wrote:
> > Hi Guys - can anyone suggest how to achieve this?
> > I have understood how to insert json documents. So one alternative that
> > comes to my mind is that I can convert the rows in my excel to json
> format
> > with the header of my excel file becoming the json keys (corresponding to
> > the fields I have defined in my managed-schema.xml). And then each cell
> in
> > the excel file will become the value of this field.
> >
> > However, I am sure there must be a better way and directly ingesting the
> > excel file to achieve the same. I was trying to reach about DIH and
> Apache
> > Tika, but I am not very sure of how the configuration works.
> >
> > My sample excel file has 4 columns namely -
> > 1. First Name
> > 2. Last Name
> > 3. Phone
> > 4. Website Link
> >
> > I want to index these fields into SOLR in a way that all these columns
> > become my solr schema fields and later I can search based on these
> fields.
> >
> > Any suggestions please.
> >
> > thanks !
> >
>
> --
> Charlie Hull
> Flax - Open Source Enterprise Search
>
> tel/fax: +44 (0)8700 118334
> mobile:  +44 (0)7767 825828
> web: www.flax.co.uk
>
>

Re: Indexig excel (xlsx) file into SOLR 8.1.1

[Charlie Hull]

Convert the Excel file to a CSV and then write a teeny script to go 
through it line by line and submit to Solr over HTTP? Tika would 
probably work but it's a lot of heavy lifting for what seems to me like 
a simple problem.


Cheers

Charlie

On 26/07/2019 09:19, Vipul Bahuguna wrote:

Hi Guys - can anyone suggest how to achieve this?
I have understood how to insert json documents. So one alternative that
comes to my mind is that I can convert the rows in my excel to json format
with the header of my excel file becoming the json keys (corresponding to
the fields I have defined in my managed-schema.xml). And then each cell in
the excel file will become the value of this field.

However, I am sure there must be a better way and directly ingesting the
excel file to achieve the same. I was trying to reach about DIH and Apache
Tika, but I am not very sure of how the configuration works.

My sample excel file has 4 columns namely -
1. First Name
2. Last Name
3. Phone
4. Website Link

I want to index these fields into SOLR in a way that all these columns
become my solr schema fields and later I can search based on these fields.

Any suggestions please.

thanks !



--
Charlie Hull
Flax - Open Source Enterprise Search

tel/fax: +44 (0)8700 118334
mobile:  +44 (0)7767 825828
web: www.flax.co.uk

Indexig excel (xlsx) file into SOLR 8.1.1

[Vipul Bahuguna]

Hi Guys - can anyone suggest how to achieve this?
I have understood how to insert json documents. So one alternative that
comes to my mind is that I can convert the rows in my excel to json format
with the header of my excel file becoming the json keys (corresponding to
the fields I have defined in my managed-schema.xml). And then each cell in
the excel file will become the value of this field.

However, I am sure there must be a better way and directly ingesting the
excel file to achieve the same. I was trying to reach about DIH and Apache
Tika, but I am not very sure of how the configuration works.

My sample excel file has 4 columns namely -
1. First Name
2. Last Name
3. Phone
4. Website Link

I want to index these fields into SOLR in a way that all these columns
become my solr schema fields and later I can search based on these fields.

Any suggestions please.

thanks !

Re: Basic Query Not Working - Please Help

[Vipul Bahuguna]

Hi Furkan -

I realized that I was searching incorrectly.
I later realized that if I need to search by specific field, I need to do
as you suggested -
q=appname:App1 .

OR if need to simply search by App1, then I need to use  to
index my field appname at the time of insertion so that it can be later
search without specifying the fieldname.

thanks for your response.

On Tue, Jul 23, 2019 at 6:07 AM Furkan KAMACI 
wrote:

> Hi Vipul,
>
> Which query do you submit? Is that one:
>
> q=appname:App1
>
> Kind Regards,
> Furkan KAMACI
>
> On Mon, Jul 22, 2019 at 10:52 AM Vipul Bahuguna <
> newthings4learn...@gmail.com> wrote:
>
> > Hi,
> >
> > I have installed SOLR 8.1.1.
> > I am new and trying the very basics.
> >
> > I installed solr8.1.1 on Windows and I am using SOLR in standalone mode.
> >
> > Steps I followed -
> >
> > 1. created a core as follows:
> > solr create_core -c dox
> >
> > 2. updated the managed_schema.xml file to add few specific fields
> specific
> > to my schema as belows:
> >
> > 
> > 
> >  stored="true"/>
> >  > stored="true"/>
> >
> > 3. then i restarted SOLR
> >
> > 4. then i went to the Documents tab to enter my sample data for indexing,
> > which looks like below:
> > {
> >
> >   "id" : "1",
> >   "prjname" : "Project1",
> >   "apps" : [
> > {
> >   "appname" : "App1",
> >   "topics" : [
> > {
> >   "topicname" : "topic1",
> >   "links" : [
> > "http://www.google.com;,
> > "http://www.t6.com;
> >   ]
> > },
> > {
> >   "topicname" : "topic2",
> >   "links" : [
> > "http://www.java.com;,
> > "http://www.rediff.com;
> >   ]
> > }
> >   ]
> > },
> > {
> >   "appname" : "App2",
> >   "topics" : [
> > {
> >   "topicname" : "topic3",
> >   "links" : [
> > "http://www.t3.com;,
> > "http://www.t4.com;
> >   ]
> > },
> > {
> >   "topicname" : "topic4",
> >   "links" : [
> > "http://www.rules.com;,
> > "http://www.amazon.com;
> >   ]
> > }
> >   ]
> > }
> >   ]
> > }
> >
> > 5. Now when i go to Query tab and click Execute Search with *.*, it shows
> > my recently added document as follows:
> > {
> > "responseHeader":{ "status":0, "QTime":0, "params":{ "q":"*:*", "_":
> > "1563780352100"}}, "response":{"numFound":1,"start":0,"docs":[ {
> "id":"1",
> > "
> > prjname":["Project1"], "apps":["{appname=App1, topics=[{topicname=topic1,
> > links=[http://www.google.com, http://www.t6.com]}, {topicname=topic2,
> > links=[http://www.java.com, http://www.rediff.com]}]};, "{appname=App2,
> > topics=[{topicname=topic3, links=[http://www.t3.com, http://www.t4.com
> ]},
> > {topicname=topic4, links=[http://www.rules.com, http://www.amazon.com
> > ]}]}"],
> > "_version_":1639742305772503040}] }}
> >
> > 6. But now when I am trying to search based on field topicname or
> prjname,
> > it does not returns any document. Even if put anything in q like App1,
> zero
> > results are being returned.
> >
> >
> > Can someone help me understanding what I might have done incorrectly?
> > May be I defined my schema incorrectly.
> >
> > Thanks in advance
> >
>

[ANNOUNCE] Apache Solr 8.2.0 released

[Ignacio Vera]

## 26 July 2019, Apache Solr™ 8.2.0 available


The Lucene PMC is pleased to announce the release of Apache Solr 8.2.0.


Solr is the popular, blazing fast, open source NoSQL search platform from
the Apache Lucene project. Its major features include powerful full-text
search, hit highlighting, faceted search, dynamic clustering, database
integration, rich document handling, and geospatial search. Solr is highly
scalable, providing fault tolerant distributed search and indexing, and
powers the search and navigation features of many of the world's largest
internet sites.


Solr 8.2.0 is available for immediate download at:


  


### Solr 8.2.0 Release Highlights:


 New features


  * Add an update param failOnVersionConflicts=false to updates not fail
when there is a version conflict

  * Add facet2D Streaming Expression.

  * Preferred replicas on nodes with same system properties as the query
master

  * OpenTracing support for Solr

  * Raw index data analysis tool (extension of COLSTATUS collection
command).

  * Add recNum Stream Evaluator.

  * Allow zplot to visualize 2D clusters and convex hulls.

  * Add a field type for Estonian language to default managed_schema,
document about Estonian language analysis in Solr Ref Guide


 Bug Fixes


  * Intermittent 401's for internode requests with basicauth enabled.

  * In 8.1, Atomic Updates were broken (NPE) when the schema declared the
new _nest_path_ field even if you weren't using nested docs. In-place
updates were not affected (worked)

  * Fix atomic update encoding issue for UUID, enum, bool, and binary
fields.

  * Impossible to delete a collection with the same name as an existing
alias. This fixes also a bug inREINDEXCOLLECTION when used with
removeSource=true which could lead to a data loss.


Please read CHANGES.txt for a full list of new features and changes:


  


Solr 8.2.0 also includes features, optimizations  and bugfixes in the
corresponding Apache Lucene release:


  


Note: The Apache Software Foundation uses an extensive mirroring network for

distributing releases. It is possible that the mirror you are using may not
have

replicated the release yet. If that is the case, please try another mirror.

This also applies to Maven access.