Re: Solr cloud backup/restore not working

[yaswanth kumar]

Can anyone here help on the posted question pls??

On Fri, Jun 12, 2020 at 10:38 AM yaswanth kumar 
wrote:

> Using solr 8.2.0 and setup a cloud with 2 nodes. (2 replica's for each
> collection)
> Enabled basic authentication and gave all access to the admin user
>
> Now trying to use solr cloud backup/restore API, backup is working great,
> but when trying to invoke restore API its throwing the below error
>
> {
>   "responseHeader":{
> "status":500,
> "QTime":349},
>   "Operation restore caused
> exception:":"org.apache.solr.common.SolrException:org.apache.solr.common.SolrException:
> ADDREPLICA failed to create replica",
>   "exception":{
> "msg":"ADDREPLICA failed to create replica",
> "rspCode":500},
>   "error":{
> "metadata":[
>   "error-class","org.apache.solr.common.SolrException",
>   "root-error-class","org.apache.solr.common.SolrException"],
> "msg":"ADDREPLICA failed to create replica",
> "trace":"org.apache.solr.common.SolrException: ADDREPLICA failed to
> create replica\n\tat
> org.apache.solr.client.solrj.SolrResponse.getException(SolrResponse.java:53)\n\tat
> org.apache.solr.handler.admin.CollectionsHandler.invokeAction(CollectionsHandler.java:280)\n\tat
> org.apache.solr.handler.admin.CollectionsHandler.handleRequestBody(CollectionsHandler.java:252)\n\tat
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)\n\tat
> org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:820)\n\tat
> org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:786)\n\tat
> org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:546)\n\tat
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:423)\n\tat
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:350)\n\tat
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602)\n\tat
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)\n\tat
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)\n\tat
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)\n\tat
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)\n\tat
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)\n\tat
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1711)\n\tat
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)\n\tat
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1347)\n\tat
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)\n\tat
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)\n\tat
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1678)\n\tat
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)\n\tat
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1249)\n\tat
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)\n\tat
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220)\n\tat
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:152)\n\tat
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)\n\tat
> org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)\n\tat
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)\n\tat
> org.eclipse.jetty.server.Server.handle(Server.java:505)\n\tat
> org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:370)\n\tat
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:267)\n\tat
> org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)\n\tat
> org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)\n\tat
> org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:427)\n\tat
> org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:321)\n\tat
> org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:159)\n\tat
> org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)\n\tat
> org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)\n\tat
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)\n\tat
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)\n\tat
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)\n\tat
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)\n\tat
> org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)\n\tat
> 

Proxy Error when cluster went down

[Vishal Vaibhav]

Hello all,

I am running on solr 8.5. For some reason entire cluster went down. When i
am trying to bring up the nodes,its not coming up. My health check is
on "/solr/rules/admin/system". I tried forcing a leader election but it
dint help.
so when i run the following commands. Why is it trying to proxy when those
nodes are down. Am i missing something?


*curl "http://localhost:8983/solr/rules/admin/system
" * % Total% Received %
Xferd  Average Speed   TimeTime Time  Current
 Dload  Upload   Total   SpentLeft
 Speed
100  5582  100  55820 0   454k  0 --:--:-- --:--:-- --:--:--
 495k
{
  "error":{
"metadata":[
  "error-class","org.apache.solr.common.SolrException",
  "root-error-class","java.net.UnknownHostException"],
"msg":"Error trying to proxy request for url:
http://search-rules-solr-v1-2.search-rules-solr-v1.search-digital.svc.cluster.local:8983/solr/rules/admin/system
",
"trace":"org.apache.solr.common.SolrException: Error trying to proxy
request for url:
http://search-rules-solr-v1-2.search-rules-solr-v1.search-digital.svc.cluster.local:8983/solr/rules/admin/system\n\tat
org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:735)\n\tat
org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:562)\n\tat
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:419)\n\tat
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:351)\n\tat
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602)\n\tat
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)\n\tat
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)\n\tat
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)\n\tat
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1711)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)\n\tat
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1347)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)\n\tat
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)\n\tat
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1678)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)\n\tat
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1249)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)\n\tat
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220)\n\tat
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:152)\n\tat
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)\n\tat
org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)\n\tat
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)\n\tat
org.eclipse.jetty.server.Server.handle(Server.java:505)\n\tat
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:370)\n\tat
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:267)\n\tat
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)\n\tat
org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)\n\tat
org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)\n\tat
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)\n\tat
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)\n\tat
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)\n\tat
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)\n\tat
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)\n\tat
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:781)\n\tat
org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:917)\n\tat
java.base/java.lang.Thread.run(Unknown Source)\nCaused by:
java.net.UnknownHostException:
search-rules-solr-v1-2.search-rules-solr-v1.search-digital.svc.cluster.local:
Name or service not known\n\tat
java.base/java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method)\n\tat
java.base/java.net.InetAddress$PlatformNameService.lookupAllHostAddr(Unknown
Source)\n\tat
java.base/java.net.InetAddress.getAddressesFromNameService(Unknown
Source)\n\tat
java.base/java.net.InetAddress$NameServiceAddresses.get(Unknown
Source)\n\tat java.base/java.net.InetAddress.getAllByName0(Unknown

getting different errors from complex phrase query

[Deepu]

Hi All,

i am trying to use {!complexphrasequeryparser inOrder=true} along with
other text fields. i am using solrj client to initiate the request.

sample query is
"{!complexphrase inOrder=true}(all_text_txt_enus:\"by\\ test*\") AND
(({!terms f=product_id_l}959945,959959,959960,959961,959962,959963)
AND (date_created_at_rdt:[2020-04-07T01:23:09Z TO *} AND
date_created_at_rdt:{* TO 2020-04-07T01:24:57Z]))"

org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error
from server  https://XX.XX.XX:8983/solr/problem: undefined field text

Please let me know what is wrong with my query.

Grouping and Learning To Rank (SOLR-8776)

[Webster Homer]

My company is very interested in using Learning To Rank in our product search. 
The problem we face is that our product search groups its results and that does 
not work with LTR.
https://issues.apache.org/jira/browse/SOLR-8776

Is there any traction to getting the SOLR-8776 patch into the main branch? 
Seems like this would be useful to a lot of people



This message and any attachment are confidential and may be privileged or 
otherwise protected from disclosure. If you are not the intended recipient, you 
must not copy this message or attachment or disclose the contents to any other 
person. If you have received this transmission in error, please notify the 
sender immediately and delete the message and any attachment from your system. 
Merck KGaA, Darmstadt, Germany and any of its subsidiaries do not accept 
liability for any omissions or errors in this message which may arise as a 
result of E-Mail-transmission or for damages resulting from any unauthorized 
changes of the content of this message and any attachment thereto. Merck KGaA, 
Darmstadt, Germany and any of its subsidiaries do not guarantee that this 
message is free of viruses and does not accept liability for any damages caused 
by any virus transmitted therewith.



Click http://www.merckgroup.com/disclaimer to access the German, French, 
Spanish and Portuguese versions of this disclaimer.

Re: eDismax query syntax question

[Mikhail Khludnev]

Hello.
Not sure if it's useful or relevant, I encountered another problem with
parentheses (braces) in eDisMax recently
https://issues.apache.org/jira/browse/SOLR-14557.

On Mon, Jun 15, 2020 at 5:01 PM Webster Homer <
webster.ho...@milliporesigma.com> wrote:

> Markus,
> Thanks, for the reference, but that doesn't answer my question. If - is a
> special character, it's not consistently special. In my example
> "3-DIMETHYL" behaves quite differently than ")-PYRIMIDINE".  If I escape
> the closing parenthesis the following minus no longer behaves specially.
> The referred article does not even mention parenthesis, but it changes the
> behavior of the following "-" if it is escaped. In "3-DIMETHYL" the minus
> is not special.
>
> These all fix the problem:
> 1,3-DIMETHYL-5-(3-PHENYL-ALLYLIDENE\)-PYRIMIDINE-2,4,6-TRIONE
> 1,3-DIMETHYL-5-(3-PHENYL-ALLYLIDENE)\-PYRIMIDINE-2,4,6-TRIONE
> 1,3-DIMETHYL-5-\(3-PHENYL-ALLYLIDENE\)-PYRIMIDINE-2,4,6-TRIONE
>
> Only the minus following the parenthesis is treated as a NOT.
> Are parentheses special? They're not mentioned in the eDismax
> documentation.
>
> -Original Message-
> From: Markus Jelsma 
> Sent: Saturday, June 13, 2020 4:57 AM
> To: solr-user@lucene.apache.org
> Subject: RE: eDismax query syntax question
>
> Hello,
>
> These are special characters, if you don't need them, you must escape them.
>
> See top of the article:
>
> https://lucene.apache.org/solr/guide/8_5/the-extended-dismax-query-parser.html
>
> Markus
>
>
>
>
> -Original message-
> > From:Webster Homer 
> > Sent: Friday 12th June 2020 22:09
> > To: solr-user@lucene.apache.org
> > Subject: eDismax query syntax question
> >
> > Recently we found strange behavior in a query. We use eDismax as the
> query parser.
> >
> > This is the query term:
> > 1,3-DIMETHYL-5-(3-PHENYL-ALLYLIDENE)-PYRIMIDINE-2,4,6-TRIONE
> >
> > It should hit one document in our index. It does not. However, if you
> use the Dismax query parser it does match the record.
> >
> > The problem seems to involve the parenthesis and the dashes. If you
> > escape the dash after the parenthesis it matches
> > 1,3-DIMETHYL-5-(3-PHENYL-ALLYLIDENE)\-PYRIMIDINE-2,4,6-TRIONE
> >
> > I thought that eDismax and Dismax escaped all lucene special characters
> before passing the query to lucene. Although I also remember reading that +
> and - can have special significance in a query if preceded with white
> space. I can find very little documentation on either query parser in how
> they work.
> >
> > Is this expected behavior or is this a bug? If expected, where can I
> find documentation?
> >
> >
> >
> > This message and any attachment are confidential and may be privileged
> or otherwise protected from disclosure. If you are not the intended
> recipient, you must not copy this message or attachment or disclose the
> contents to any other person. If you have received this transmission in
> error, please notify the sender immediately and delete the message and any
> attachment from your system. Merck KGaA, Darmstadt, Germany and any of its
> subsidiaries do not accept liability for any omissions or errors in this
> message which may arise as a result of E-Mail-transmission or for damages
> resulting from any unauthorized changes of the content of this message and
> any attachment thereto. Merck KGaA, Darmstadt, Germany and any of its
> subsidiaries do not guarantee that this message is free of viruses and does
> not accept liability for any damages caused by any virus transmitted
> therewith.
> >
> >
> >
> > Click http://www.merckgroup.com/disclaimer to access the German,
> French, Spanish and Portuguese versions of this disclaimer.
> >
>
>
> This message and any attachment are confidential and may be privileged or
> otherwise protected from disclosure. If you are not the intended recipient,
> you must not copy this message or attachment or disclose the contents to
> any other person. If you have received this transmission in error, please
> notify the sender immediately and delete the message and any attachment
> from your system. Merck KGaA, Darmstadt, Germany and any of its
> subsidiaries do not accept liability for any omissions or errors in this
> message which may arise as a result of E-Mail-transmission or for damages
> resulting from any unauthorized changes of the content of this message and
> any attachment thereto. Merck KGaA, Darmstadt, Germany and any of its
> subsidiaries do not guarantee that this message is free of viruses and does
> not accept liability for any damages caused by any virus transmitted
> therewith.
>
>
>
> Click http://www.merckgroup.com/disclaimer to access the German, French,
> Spanish and Portuguese versions of this disclaimer.
>


-- 
Sincerely yours
Mikhail Khludnev

Re: How to determine why solr stops running?

[Ryan W]

On Mon, Jun 15, 2020 at 2:00 PM Ryan W  wrote:

> What is the Service definition of Solr in Redhat?
>

I think maybe you are talking about systemd.

Maybe a service definition looks something like this?
https://gist.github.com/hammady/3d7b5964c7b0f90997865ebef40bf5e1

I haven't used systemd before. I should probably look into that.  It isn't
something I am currently using, as far as I know.




> >> Thank you.  I pasted those settings at the end of my /etc/default/
>> >> solr.in.sh just now and restarted solr.  I will see if that fixes it.
>> >> Previously, I had no settings at all in solr.in.sh except for
>> SOLR_PORT.
>> >>
>> >> On Thu, Jun 11, 2020 at 1:59 PM Walter Underwood <
>> wun...@wunderwood.org>
>> >> wrote:
>> >>
>> >>> 1. You have a tiny heap. 536 Megabytes is not enough.
>> >>> 2. I stopped using the CMS GC years ago.
>> >>>
>> >>> Here is the GC config we use on every one of our 150+ Solr hosts.
>> We’re
>> >>> still on Java 8, but will be upgrading soon.
>> >>>
>> >>> SOLR_HEAP=8g
>> >>> # Use G1 GC  -- wunder 2017-01-23
>> >>> # Settings from https://wiki.apache.org/solr/ShawnHeisey
>> >>> GC_TUNE=" \
>> >>> -XX:+UseG1GC \
>> >>> -XX:+ParallelRefProcEnabled \
>> >>> -XX:G1HeapRegionSize=8m \
>> >>> -XX:MaxGCPauseMillis=200 \
>> >>> -XX:+UseLargePages \
>> >>> -XX:+AggressiveOpts \
>> >>> "
>> >>>
>> >>> wunder
>> >>> Walter Underwood
>> >>> wun...@wunderwood.org
>> >>> http://observer.wunderwood.org/  (my blog)
>> >>>
>>  On Jun 11, 2020, at 10:52 AM, Ryan W  wrote:
>> 
>>  On Wed, Jun 10, 2020 at 8:35 PM Hup Chen 
>> wrote:
>> 
>> > I will check "dmesg" first, to find out any hardware error message.
>> >
>> 
>>  Here is what I see toward the end of the output from dmesg:
>> 
>>  [1521232.781785] [118857]48 118857   108785  677 201
>>  901 0 httpd
>>  [1521232.781787] [118860]48 118860   108785  710 201
>>  881 0 httpd
>>  [1521232.781788] [118862]48 118862   113063 5256 210
>>  725 0 httpd
>>  [1521232.781790] [118864]48 118864   114085 6634 212
>>  703 0 httpd
>>  [1521232.781791] [118871]48 118871   13968732323 262
>>  620 0 httpd
>>  [1521232.781793] [118873]48 118873   108785  821 201
>>  792 0 httpd
>>  [1521232.781795] [118879]48 118879   14026332719 263
>>  621 0 httpd
>>  [1521232.781796] [118903]48 118903   108785  812 201
>>  771 0 httpd
>>  [1521232.781798] [118905]48 118905   113575 5606 211
>>  660 0 httpd
>>  [1521232.781800] [118906]48 118906   113563 5694 211
>>  626 0 httpd
>>  [1521232.781801] Out of memory: Kill process 117529 (httpd) score 9
>> or
>>  sacrifice child
>>  [1521232.782908] Killed process 117529 (httpd), UID 48,
>> >>> total-vm:675824kB,
>>  anon-rss:181844kB, file-rss:0kB, shmem-rss:0kB
>> 
>>  Is this a relevant "Out of memory" message?  Does this suggest an OOM
>>  situation is the culprit?
>> 
>>  When I grep in the solr logs for oom, I see some entries like this...
>> 
>>  ./solr_gc.log.4.current:CommandLine flags: -XX:CICompilerCount=4
>>  -XX:CMSInitiatingOccupancyFraction=50
>> >>> -XX:CMSMaxAbortablePrecleanTime=6000
>>  -XX:+CMSParallelRemarkEnabled -XX:+CMSScavengeBeforeRemark
>>  -XX:ConcGCThreads=4 -XX:GCLogFileSize=20971520
>>  -XX:InitialHeapSize=536870912 -XX:MaxHeapSize=536870912
>>  -XX:MaxNewSize=134217728 -XX:MaxTenuringThreshold=8
>>  -XX:MinHeapDeltaBytes=196608 -XX:NewRatio=3 -XX:NewSize=134217728
>>  -XX:NumberOfGCLogFiles=9 -XX:OldPLABSize=16 -XX:OldSize=402653184
>>  -XX:-OmitStackTraceInFastThrow
>>  -XX:OnOutOfMemoryError=/opt/solr/bin/oom_solr.sh 8983
>> >>> /opt/solr/server/logs
>>  -XX:ParallelGCThreads=4 -XX:+ParallelRefProcEnabled
>>  -XX:PretenureSizeThreshold=67108864 -XX:+PrintGC
>>  -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCDateStamps
>>  -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintHeapAtGC
>>  -XX:+PrintTenuringDistribution -XX:SurvivorRatio=4
>>  -XX:TargetSurvivorRatio=90 -XX:ThreadStackSize=256
>>  -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers
>>  -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC
>> -XX:+UseGCLogFileRotation
>>  -XX:+UseParNewGC
>> 
>>  Buried in there I see "OnOutOfMemoryError=/opt/solr/bin/oom_solr.sh".
>> >>> But I
>>  think this is just a setting that indicates what to do in case of an
>> >>> OOM.
>>  And if I look in that oom_solr.sh file, I see it would write an entry
>> >>> to a
>>  solr_oom_kill log. And there is no such log in the logs directory.
>> 
>>  Many thanks.
>> 
>> 
>> 
>> 
>> > Then use some system admin tools to monitor that server,
>> 

Re: [EXTERNAL] - Re: HTTP 401 when searching on alias in secured Solr

[Isabelle Giguere]

Thank you for the input, Aroop.

It is probably a red herring.  I will have to pick the configuration apart 
piece by piece.  Sigh.

It's probably not a node down issue, since I'm only setting up one node.

(Reporting an unrelated error message should probably be considered a bug 
anyways.)

Isabelle Giguère
Computational Linguist & Java Developer
Linguiste informaticienne & développeur java



De : Aroop Ganguly 
Envoyé : 14 juin 2020 17:37
À : solr-user@lucene.apache.org 
Objet : Re: [EXTERNAL] - Re: HTTP 401 when searching on alias in secured Solr

Isabele sometime 401’s are a red herring for other issues un related to auth.
We have had issues on 7.7 where an underlying transient replica recovery and/or 
leader down situation where the only message we got back from Solr was a 401.
Please see if u have any down replicas or other issues where certain nodes may 
have trouble getting more current information from zookeeper.


> On Jun 14, 2020, at 2:13 PM, Isabelle Giguere  > wrote:
>
> I have created 
> https://urldefense.com/v3/__https://issues.apache.org/jira/browse/SOLR-14569__;!!Obbck6kTJA!PBs90R0pHCmvm6hGqjUeowZNMwhTEibIfLyr8_szdm0Jh-s9okdbuGya_nBlsjED$
>   
>   >
> It includes a patch with the unit test to reproduce the issue, and a 
> simplification of our product-specific configuration, with instructions.
>
> Let's catch up on Jira.
>
> Isabelle Giguère
> Computational Linguist & Java Developer
> Linguiste informaticienne & développeur java
>
>
> 
> De : Jan Høydahl mailto:jan@cominvent.com>>
> Envoyé : 13 juin 2020 17:50
> À : solr-user  >
> Objet : Re: [EXTERNAL] - Re: HTTP 401 when searching on alias in secured Solr
>
> I did not manage to reproduce. Feel free to open the JIRA and attach the 
> failing test. In the issue description, it is great if you manage to describe 
> the reproduction steps in a clean way, so anyone can reproduce with a minimal 
> neccessary config.
>
> Jan
>
>> 13. jun. 2020 kl. 00:41 skrev Isabelle Giguere 
>> mailto:igigu...@opentext.com.INVALID>>:
>>
>> Hello again;
>>
>> I have managed to reproduce the issue in a unit test.  I should probably add 
>> a Jira ticket with a patch for the unit test On Solr 8.5.0, not master.
>>
>> Meanwhile, for your suggested queries:
>>
>> 1.  Query on the collection:
>>
>> curl -i -u admin:admin 
>> https://urldefense.com/v3/__http://10.5.106.115:8985/solr/test1/select?q=*:*=xml__;Kio!!Obbck6kTJA!LvZRdkAwPGTDqWqS-BYMmyuuwAp9coGzkDzz5BG7hTCLmCSV2bOZBM9A7JzikWgk$
>>  
>> 
>> HTTP/1.1 200 OK
>> Content-Security-Policy: default-src 'none'; base-uri 'none'; connect-src 
>> 'self'; form-action 'self'; font-src 'self'; frame-ancestors 'none'; img-src 
>> 'self'; media-src 'self'; style-src 'self' 'unsafe-inline'; script-src 
>> 'self'; worker-src 'self';
>> X-Content-Type-Options: nosniff
>> X-Frame-Options: SAMEORIGIN
>> X-XSS-Protection: 1; mode=block
>> Content-Type: application/xml; charset=UTF-8
>> Content-Length: 8214
>>
>> 
>> 
>>
>> 
>> true
>> 0
>> 2
>> 
>>   *:*
>> 
>> 
>> 
>> Response contains the Solr document, of course
>>
>>
>> 2. Query on the alias
>>
>> curl -i -u admin:admin 
>> https://urldefense.com/v3/__http://10.5.106.115:8985/solr/test/select?q=*:*=xml__;Kio!!Obbck6kTJA!LvZRdkAwPGTDqWqS-BYMmyuuwAp9coGzkDzz5BG7hTCLmCSV2bOZBM9A7PZyiHWo$
>>  
>> >  
>> 
>>  >
>> HTTP/1.1 401 Unauthorized
>> Content-Security-Policy: default-src 'none'; base-uri 'none'; connect-src 
>> 'self'; form-action 'self'; font-src 'self'; frame-ancestors 'none'; img-src 
>> 'self'; media-src 'self'; style-src 'self' 'unsafe-inline'; script-src 
>> 'self'; worker-src 'self';
>> X-Content-Type-Options: nosniff
>> X-Frame-Options: SAMEORIGIN
>> X-XSS-Protection: 1; mode=block
>> Cache-Control: no-cache, no-store
>> Pragma: no-cache
>> Expires: Sat, 01 Jan 2000 01:00:00 GMT
>> Last-Modified: Fri, 12 Jun 2020 22:30:20 GMT
>> ETag: "172aaa7c1eb"
>> Content-Type: application/xml; charset=UTF-8
>> Content-Length: 1332
>>
>> 
>> 
>>
>> 
>> true
>> 401
>> 16
>> 
>>   *:*
>> 
>> 
>> 
>> Error contains the full 

Re: eDismax query syntax question

[Andrea Gazzarini]

Hi Webster,
what does the query debug say? if you set debug=true in the request you can
have a better idea about how the two queries get interpreted

Andrea

On Mon, 15 Jun 2020 at 16:01, Webster Homer <
webster.ho...@milliporesigma.com> wrote:

> Markus,
> Thanks, for the reference, but that doesn't answer my question. If - is a
> special character, it's not consistently special. In my example
> "3-DIMETHYL" behaves quite differently than ")-PYRIMIDINE".  If I escape
> the closing parenthesis the following minus no longer behaves specially.
> The referred article does not even mention parenthesis, but it changes the
> behavior of the following "-" if it is escaped. In "3-DIMETHYL" the minus
> is not special.
>
> These all fix the problem:
> 1,3-DIMETHYL-5-(3-PHENYL-ALLYLIDENE\)-PYRIMIDINE-2,4,6-TRIONE
> 1,3-DIMETHYL-5-(3-PHENYL-ALLYLIDENE)\-PYRIMIDINE-2,4,6-TRIONE
> 1,3-DIMETHYL-5-\(3-PHENYL-ALLYLIDENE\)-PYRIMIDINE-2,4,6-TRIONE
>
> Only the minus following the parenthesis is treated as a NOT.
> Are parentheses special? They're not mentioned in the eDismax
> documentation.
>
> -Original Message-
> From: Markus Jelsma 
> Sent: Saturday, June 13, 2020 4:57 AM
> To: solr-user@lucene.apache.org
> Subject: RE: eDismax query syntax question
>
> Hello,
>
> These are special characters, if you don't need them, you must escape them.
>
> See top of the article:
>
> https://lucene.apache.org/solr/guide/8_5/the-extended-dismax-query-parser.html
>
> Markus
>
>
>
>
> -Original message-
> > From:Webster Homer 
> > Sent: Friday 12th June 2020 22:09
> > To: solr-user@lucene.apache.org
> > Subject: eDismax query syntax question
> >
> > Recently we found strange behavior in a query. We use eDismax as the
> query parser.
> >
> > This is the query term:
> > 1,3-DIMETHYL-5-(3-PHENYL-ALLYLIDENE)-PYRIMIDINE-2,4,6-TRIONE
> >
> > It should hit one document in our index. It does not. However, if you
> use the Dismax query parser it does match the record.
> >
> > The problem seems to involve the parenthesis and the dashes. If you
> > escape the dash after the parenthesis it matches
> > 1,3-DIMETHYL-5-(3-PHENYL-ALLYLIDENE)\-PYRIMIDINE-2,4,6-TRIONE
> >
> > I thought that eDismax and Dismax escaped all lucene special characters
> before passing the query to lucene. Although I also remember reading that +
> and - can have special significance in a query if preceded with white
> space. I can find very little documentation on either query parser in how
> they work.
> >
> > Is this expected behavior or is this a bug? If expected, where can I
> find documentation?
> >
> >
> >
> > This message and any attachment are confidential and may be privileged
> or otherwise protected from disclosure. If you are not the intended
> recipient, you must not copy this message or attachment or disclose the
> contents to any other person. If you have received this transmission in
> error, please notify the sender immediately and delete the message and any
> attachment from your system. Merck KGaA, Darmstadt, Germany and any of its
> subsidiaries do not accept liability for any omissions or errors in this
> message which may arise as a result of E-Mail-transmission or for damages
> resulting from any unauthorized changes of the content of this message and
> any attachment thereto. Merck KGaA, Darmstadt, Germany and any of its
> subsidiaries do not guarantee that this message is free of viruses and does
> not accept liability for any damages caused by any virus transmitted
> therewith.
> >
> >
> >
> > Click http://www.merckgroup.com/disclaimer to access the German,
> French, Spanish and Portuguese versions of this disclaimer.
> >
>
>
> This message and any attachment are confidential and may be privileged or
> otherwise protected from disclosure. If you are not the intended recipient,
> you must not copy this message or attachment or disclose the contents to
> any other person. If you have received this transmission in error, please
> notify the sender immediately and delete the message and any attachment
> from your system. Merck KGaA, Darmstadt, Germany and any of its
> subsidiaries do not accept liability for any omissions or errors in this
> message which may arise as a result of E-Mail-transmission or for damages
> resulting from any unauthorized changes of the content of this message and
> any attachment thereto. Merck KGaA, Darmstadt, Germany and any of its
> subsidiaries do not guarantee that this message is free of viruses and does
> not accept liability for any damages caused by any virus transmitted
> therewith.
>
>
>
> Click http://www.merckgroup.com/disclaimer to access the German, French,
> Spanish and Portuguese versions of this disclaimer.
>
-- 
Andrea Gazzarini
*Search Consultant, R Software Engineer*


www.sease.io

email: a.gazzar...@sease.io
cell: +39 349 513 86 25

Re: How to determine why solr stops running?

[Ryan W]

On Mon, Jun 15, 2020 at 1:56 PM Jörn Franke  wrote:

> What is the Service definition of Solr in Redhat?
>

I am not sure what that means.  What is a service definition?  I am using
Solr in conjunction with Drupal's Search API Solr module:
https://www.drupal.org/project/search_api_solr



>
> > Am 15.06.2020 um 19:46 schrieb Ryan W :
> >
> > It happened again today.  Again, no other apparent problems on the
> server.
> > Nothing else is stopping.  Nothing in the logs that strikes me as useful.
> > I'm using Red Hat Linux 7.8 and Solr 7.7.2.
> >
> > Solr is stopping a couple times per week and I don't know how to
> determine
> > why.
> >
> >> On Sun, Jun 14, 2020 at 9:41 AM Ryan W  wrote:
> >>
> >> Thank you.  I pasted those settings at the end of my /etc/default/
> >> solr.in.sh just now and restarted solr.  I will see if that fixes it.
> >> Previously, I had no settings at all in solr.in.sh except for
> SOLR_PORT.
> >>
> >> On Thu, Jun 11, 2020 at 1:59 PM Walter Underwood  >
> >> wrote:
> >>
> >>> 1. You have a tiny heap. 536 Megabytes is not enough.
> >>> 2. I stopped using the CMS GC years ago.
> >>>
> >>> Here is the GC config we use on every one of our 150+ Solr hosts. We’re
> >>> still on Java 8, but will be upgrading soon.
> >>>
> >>> SOLR_HEAP=8g
> >>> # Use G1 GC  -- wunder 2017-01-23
> >>> # Settings from https://wiki.apache.org/solr/ShawnHeisey
> >>> GC_TUNE=" \
> >>> -XX:+UseG1GC \
> >>> -XX:+ParallelRefProcEnabled \
> >>> -XX:G1HeapRegionSize=8m \
> >>> -XX:MaxGCPauseMillis=200 \
> >>> -XX:+UseLargePages \
> >>> -XX:+AggressiveOpts \
> >>> "
> >>>
> >>> wunder
> >>> Walter Underwood
> >>> wun...@wunderwood.org
> >>> http://observer.wunderwood.org/  (my blog)
> >>>
>  On Jun 11, 2020, at 10:52 AM, Ryan W  wrote:
> 
>  On Wed, Jun 10, 2020 at 8:35 PM Hup Chen  wrote:
> 
> > I will check "dmesg" first, to find out any hardware error message.
> >
> 
>  Here is what I see toward the end of the output from dmesg:
> 
>  [1521232.781785] [118857]48 118857   108785  677 201
>  901 0 httpd
>  [1521232.781787] [118860]48 118860   108785  710 201
>  881 0 httpd
>  [1521232.781788] [118862]48 118862   113063 5256 210
>  725 0 httpd
>  [1521232.781790] [118864]48 118864   114085 6634 212
>  703 0 httpd
>  [1521232.781791] [118871]48 118871   13968732323 262
>  620 0 httpd
>  [1521232.781793] [118873]48 118873   108785  821 201
>  792 0 httpd
>  [1521232.781795] [118879]48 118879   14026332719 263
>  621 0 httpd
>  [1521232.781796] [118903]48 118903   108785  812 201
>  771 0 httpd
>  [1521232.781798] [118905]48 118905   113575 5606 211
>  660 0 httpd
>  [1521232.781800] [118906]48 118906   113563 5694 211
>  626 0 httpd
>  [1521232.781801] Out of memory: Kill process 117529 (httpd) score 9 or
>  sacrifice child
>  [1521232.782908] Killed process 117529 (httpd), UID 48,
> >>> total-vm:675824kB,
>  anon-rss:181844kB, file-rss:0kB, shmem-rss:0kB
> 
>  Is this a relevant "Out of memory" message?  Does this suggest an OOM
>  situation is the culprit?
> 
>  When I grep in the solr logs for oom, I see some entries like this...
> 
>  ./solr_gc.log.4.current:CommandLine flags: -XX:CICompilerCount=4
>  -XX:CMSInitiatingOccupancyFraction=50
> >>> -XX:CMSMaxAbortablePrecleanTime=6000
>  -XX:+CMSParallelRemarkEnabled -XX:+CMSScavengeBeforeRemark
>  -XX:ConcGCThreads=4 -XX:GCLogFileSize=20971520
>  -XX:InitialHeapSize=536870912 -XX:MaxHeapSize=536870912
>  -XX:MaxNewSize=134217728 -XX:MaxTenuringThreshold=8
>  -XX:MinHeapDeltaBytes=196608 -XX:NewRatio=3 -XX:NewSize=134217728
>  -XX:NumberOfGCLogFiles=9 -XX:OldPLABSize=16 -XX:OldSize=402653184
>  -XX:-OmitStackTraceInFastThrow
>  -XX:OnOutOfMemoryError=/opt/solr/bin/oom_solr.sh 8983
> >>> /opt/solr/server/logs
>  -XX:ParallelGCThreads=4 -XX:+ParallelRefProcEnabled
>  -XX:PretenureSizeThreshold=67108864 -XX:+PrintGC
>  -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCDateStamps
>  -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintHeapAtGC
>  -XX:+PrintTenuringDistribution -XX:SurvivorRatio=4
>  -XX:TargetSurvivorRatio=90 -XX:ThreadStackSize=256
>  -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers
>  -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC
> -XX:+UseGCLogFileRotation
>  -XX:+UseParNewGC
> 
>  Buried in there I see "OnOutOfMemoryError=/opt/solr/bin/oom_solr.sh".
> >>> But I
>  think this is just a setting that indicates what to do in case of an
> >>> OOM.
>  And if I look in that oom_solr.sh file, I see it would write an entry
> >>> to a
>  solr_oom_kill log. And 

Re: How to determine why solr stops running?

[Jörn Franke]

What is the Service definition of Solr in Redhat?


> Am 15.06.2020 um 19:46 schrieb Ryan W :
> 
> It happened again today.  Again, no other apparent problems on the server.
> Nothing else is stopping.  Nothing in the logs that strikes me as useful.
> I'm using Red Hat Linux 7.8 and Solr 7.7.2.
> 
> Solr is stopping a couple times per week and I don't know how to determine
> why.
> 
>> On Sun, Jun 14, 2020 at 9:41 AM Ryan W  wrote:
>> 
>> Thank you.  I pasted those settings at the end of my /etc/default/
>> solr.in.sh just now and restarted solr.  I will see if that fixes it.
>> Previously, I had no settings at all in solr.in.sh except for SOLR_PORT.
>> 
>> On Thu, Jun 11, 2020 at 1:59 PM Walter Underwood 
>> wrote:
>> 
>>> 1. You have a tiny heap. 536 Megabytes is not enough.
>>> 2. I stopped using the CMS GC years ago.
>>> 
>>> Here is the GC config we use on every one of our 150+ Solr hosts. We’re
>>> still on Java 8, but will be upgrading soon.
>>> 
>>> SOLR_HEAP=8g
>>> # Use G1 GC  -- wunder 2017-01-23
>>> # Settings from https://wiki.apache.org/solr/ShawnHeisey
>>> GC_TUNE=" \
>>> -XX:+UseG1GC \
>>> -XX:+ParallelRefProcEnabled \
>>> -XX:G1HeapRegionSize=8m \
>>> -XX:MaxGCPauseMillis=200 \
>>> -XX:+UseLargePages \
>>> -XX:+AggressiveOpts \
>>> "
>>> 
>>> wunder
>>> Walter Underwood
>>> wun...@wunderwood.org
>>> http://observer.wunderwood.org/  (my blog)
>>> 
 On Jun 11, 2020, at 10:52 AM, Ryan W  wrote:
 
 On Wed, Jun 10, 2020 at 8:35 PM Hup Chen  wrote:
 
> I will check "dmesg" first, to find out any hardware error message.
> 
 
 Here is what I see toward the end of the output from dmesg:
 
 [1521232.781785] [118857]48 118857   108785  677 201
 901 0 httpd
 [1521232.781787] [118860]48 118860   108785  710 201
 881 0 httpd
 [1521232.781788] [118862]48 118862   113063 5256 210
 725 0 httpd
 [1521232.781790] [118864]48 118864   114085 6634 212
 703 0 httpd
 [1521232.781791] [118871]48 118871   13968732323 262
 620 0 httpd
 [1521232.781793] [118873]48 118873   108785  821 201
 792 0 httpd
 [1521232.781795] [118879]48 118879   14026332719 263
 621 0 httpd
 [1521232.781796] [118903]48 118903   108785  812 201
 771 0 httpd
 [1521232.781798] [118905]48 118905   113575 5606 211
 660 0 httpd
 [1521232.781800] [118906]48 118906   113563 5694 211
 626 0 httpd
 [1521232.781801] Out of memory: Kill process 117529 (httpd) score 9 or
 sacrifice child
 [1521232.782908] Killed process 117529 (httpd), UID 48,
>>> total-vm:675824kB,
 anon-rss:181844kB, file-rss:0kB, shmem-rss:0kB
 
 Is this a relevant "Out of memory" message?  Does this suggest an OOM
 situation is the culprit?
 
 When I grep in the solr logs for oom, I see some entries like this...
 
 ./solr_gc.log.4.current:CommandLine flags: -XX:CICompilerCount=4
 -XX:CMSInitiatingOccupancyFraction=50
>>> -XX:CMSMaxAbortablePrecleanTime=6000
 -XX:+CMSParallelRemarkEnabled -XX:+CMSScavengeBeforeRemark
 -XX:ConcGCThreads=4 -XX:GCLogFileSize=20971520
 -XX:InitialHeapSize=536870912 -XX:MaxHeapSize=536870912
 -XX:MaxNewSize=134217728 -XX:MaxTenuringThreshold=8
 -XX:MinHeapDeltaBytes=196608 -XX:NewRatio=3 -XX:NewSize=134217728
 -XX:NumberOfGCLogFiles=9 -XX:OldPLABSize=16 -XX:OldSize=402653184
 -XX:-OmitStackTraceInFastThrow
 -XX:OnOutOfMemoryError=/opt/solr/bin/oom_solr.sh 8983
>>> /opt/solr/server/logs
 -XX:ParallelGCThreads=4 -XX:+ParallelRefProcEnabled
 -XX:PretenureSizeThreshold=67108864 -XX:+PrintGC
 -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCDateStamps
 -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintHeapAtGC
 -XX:+PrintTenuringDistribution -XX:SurvivorRatio=4
 -XX:TargetSurvivorRatio=90 -XX:ThreadStackSize=256
 -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers
 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseGCLogFileRotation
 -XX:+UseParNewGC
 
 Buried in there I see "OnOutOfMemoryError=/opt/solr/bin/oom_solr.sh".
>>> But I
 think this is just a setting that indicates what to do in case of an
>>> OOM.
 And if I look in that oom_solr.sh file, I see it would write an entry
>>> to a
 solr_oom_kill log. And there is no such log in the logs directory.
 
 Many thanks.
 
 
 
 
> Then use some system admin tools to monitor that server,
> for instance, top, vmstat, lsof, iostat ... or simply install some nice
> free monitoring tool into this system, like monit, monitorix, nagios.
> Good luck!
> 
> 
> From: Ryan W 
> Sent: Thursday, June 11, 2020 2:13 AM
> To: 

Re: How to determine why solr stops running?

[Ryan W]

It happened again today.  Again, no other apparent problems on the server.
Nothing else is stopping.  Nothing in the logs that strikes me as useful.
I'm using Red Hat Linux 7.8 and Solr 7.7.2.

Solr is stopping a couple times per week and I don't know how to determine
why.

On Sun, Jun 14, 2020 at 9:41 AM Ryan W  wrote:

> Thank you.  I pasted those settings at the end of my /etc/default/
> solr.in.sh just now and restarted solr.  I will see if that fixes it.
> Previously, I had no settings at all in solr.in.sh except for SOLR_PORT.
>
> On Thu, Jun 11, 2020 at 1:59 PM Walter Underwood 
> wrote:
>
>> 1. You have a tiny heap. 536 Megabytes is not enough.
>> 2. I stopped using the CMS GC years ago.
>>
>> Here is the GC config we use on every one of our 150+ Solr hosts. We’re
>> still on Java 8, but will be upgrading soon.
>>
>> SOLR_HEAP=8g
>> # Use G1 GC  -- wunder 2017-01-23
>> # Settings from https://wiki.apache.org/solr/ShawnHeisey
>> GC_TUNE=" \
>> -XX:+UseG1GC \
>> -XX:+ParallelRefProcEnabled \
>> -XX:G1HeapRegionSize=8m \
>> -XX:MaxGCPauseMillis=200 \
>> -XX:+UseLargePages \
>> -XX:+AggressiveOpts \
>> "
>>
>> wunder
>> Walter Underwood
>> wun...@wunderwood.org
>> http://observer.wunderwood.org/  (my blog)
>>
>> > On Jun 11, 2020, at 10:52 AM, Ryan W  wrote:
>> >
>> > On Wed, Jun 10, 2020 at 8:35 PM Hup Chen  wrote:
>> >
>> >> I will check "dmesg" first, to find out any hardware error message.
>> >>
>> >
>> > Here is what I see toward the end of the output from dmesg:
>> >
>> > [1521232.781785] [118857]48 118857   108785  677 201
>> > 901 0 httpd
>> > [1521232.781787] [118860]48 118860   108785  710 201
>> > 881 0 httpd
>> > [1521232.781788] [118862]48 118862   113063 5256 210
>> > 725 0 httpd
>> > [1521232.781790] [118864]48 118864   114085 6634 212
>> > 703 0 httpd
>> > [1521232.781791] [118871]48 118871   13968732323 262
>> > 620 0 httpd
>> > [1521232.781793] [118873]48 118873   108785  821 201
>> > 792 0 httpd
>> > [1521232.781795] [118879]48 118879   14026332719 263
>> > 621 0 httpd
>> > [1521232.781796] [118903]48 118903   108785  812 201
>> > 771 0 httpd
>> > [1521232.781798] [118905]48 118905   113575 5606 211
>> > 660 0 httpd
>> > [1521232.781800] [118906]48 118906   113563 5694 211
>> > 626 0 httpd
>> > [1521232.781801] Out of memory: Kill process 117529 (httpd) score 9 or
>> > sacrifice child
>> > [1521232.782908] Killed process 117529 (httpd), UID 48,
>> total-vm:675824kB,
>> > anon-rss:181844kB, file-rss:0kB, shmem-rss:0kB
>> >
>> > Is this a relevant "Out of memory" message?  Does this suggest an OOM
>> > situation is the culprit?
>> >
>> > When I grep in the solr logs for oom, I see some entries like this...
>> >
>> > ./solr_gc.log.4.current:CommandLine flags: -XX:CICompilerCount=4
>> > -XX:CMSInitiatingOccupancyFraction=50
>> -XX:CMSMaxAbortablePrecleanTime=6000
>> > -XX:+CMSParallelRemarkEnabled -XX:+CMSScavengeBeforeRemark
>> > -XX:ConcGCThreads=4 -XX:GCLogFileSize=20971520
>> > -XX:InitialHeapSize=536870912 -XX:MaxHeapSize=536870912
>> > -XX:MaxNewSize=134217728 -XX:MaxTenuringThreshold=8
>> > -XX:MinHeapDeltaBytes=196608 -XX:NewRatio=3 -XX:NewSize=134217728
>> > -XX:NumberOfGCLogFiles=9 -XX:OldPLABSize=16 -XX:OldSize=402653184
>> > -XX:-OmitStackTraceInFastThrow
>> > -XX:OnOutOfMemoryError=/opt/solr/bin/oom_solr.sh 8983
>> /opt/solr/server/logs
>> > -XX:ParallelGCThreads=4 -XX:+ParallelRefProcEnabled
>> > -XX:PretenureSizeThreshold=67108864 -XX:+PrintGC
>> > -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCDateStamps
>> > -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintHeapAtGC
>> > -XX:+PrintTenuringDistribution -XX:SurvivorRatio=4
>> > -XX:TargetSurvivorRatio=90 -XX:ThreadStackSize=256
>> > -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers
>> > -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseGCLogFileRotation
>> > -XX:+UseParNewGC
>> >
>> > Buried in there I see "OnOutOfMemoryError=/opt/solr/bin/oom_solr.sh".
>> But I
>> > think this is just a setting that indicates what to do in case of an
>> OOM.
>> > And if I look in that oom_solr.sh file, I see it would write an entry
>> to a
>> > solr_oom_kill log. And there is no such log in the logs directory.
>> >
>> > Many thanks.
>> >
>> >
>> >
>> >
>> >> Then use some system admin tools to monitor that server,
>> >> for instance, top, vmstat, lsof, iostat ... or simply install some nice
>> >> free monitoring tool into this system, like monit, monitorix, nagios.
>> >> Good luck!
>> >>
>> >> 
>> >> From: Ryan W 
>> >> Sent: Thursday, June 11, 2020 2:13 AM
>> >> To: solr-user@lucene.apache.org 
>> >> Subject: Re: How to determine why solr stops running?
>> >>
>> >> Hi all,
>> >>
>> >> People keep suggesting I check the logs for errors.  What do 

Re: Question about Atomic Update

[david . davila]

Hi Erick,


Thank you for your answer.

Unfortunatelly our most important field is that text field, so, we need to 
index it. We will have to assume that big documents takes a long time to 
index.


Best,

David




David Dávila Atienza
AEAT - Departamento de Informática Tributaria
Subdirección de Tecnologías de Análisis de la Información e Investigación 
del Fraude
Teléfono: 915828763
Extensión: 36763



De: "Erick Erickson" 
Para:   solr-user@lucene.apache.org
Fecha:  15/06/2020 14:27
Asunto: Re: Question about Atomic Update



All Atomic Updates do is 
1> read all the stored fields from the record being updated
2> overlay your updates
3> re-index the document.

At <3> it?s exactly as though you sent the entire document
again, so your observation that the whole document is 
re-indexed is accurate.

If the fields you want to update are single-valued, docValues=true
numeric fields you can update those without the whole doc being
re-indexed. But if you need to search on those fields it?ll probably
be unacceptably slow. However, if you _do_ need to search,
sometimes you can get creative with function queries. OK, this
last is opaque but say you have a ?quantity? field and only want to
find docs that have quantity > 0. You can add a function query
to your query (either q or fq) that returns the value of that field,
which means the score is 0 for docs where quantity==0 and the
doc drops out of the result set.

It?s not clear whether you search the text field, but if not you can
store it somewhere else and only fetch it as needed.

Best,
Erick

> On Jun 15, 2020, at 7:55 AM, david.dav...@correo.aeat.es wrote:
> 
> Hi,
> 
> I have a question related with atomic update in Solr.
> 
> In our collection,  documents have a lot of fields, most of them small. 
> However, there is one of them that includes the text of the document. 
> Sometimes, not many fortunatelly, this text is very long, more than 3 or 
4 
> MB of plain text. We use different analyzers such as synonyms, etc. and 
> this causes that index time in that documents is long, about 15 seconds.
> 
> Sometimes, we should update some small fields, and it is a big problem 
for 
> us because of the time that it consumes. We have been testing with 
atomic 
> update, but time is exactly the same than sending the document again. We 

> expected that with atomic update only the updated fields were indexed 
and 
> time would reduce. But it seems that internally Solr gets the whole 
> document and reindex all the fields.
> 
> Does it works in that way? Am I worng, any advice?
> 
> We have tested with Solr 7.4 and Solr 4.10
> 
> Thanks,
> 
> David 

Este mensaje ha sido enviado desde un correo externo a la Agencia 
Tributaria. Por favor, no haga click en enlaces ni abra los documentos 
adjuntos a menos que reconozca al remitente del correo y la temática del 
mismo.




 

RE: eDismax query syntax question

[Webster Homer]

Markus,
Thanks, for the reference, but that doesn't answer my question. If - is a 
special character, it's not consistently special. In my example "3-DIMETHYL" 
behaves quite differently than ")-PYRIMIDINE".  If I escape the closing 
parenthesis the following minus no longer behaves specially. The referred 
article does not even mention parenthesis, but it changes the behavior of the 
following "-" if it is escaped. In "3-DIMETHYL" the minus is not special.

These all fix the problem:
1,3-DIMETHYL-5-(3-PHENYL-ALLYLIDENE\)-PYRIMIDINE-2,4,6-TRIONE
1,3-DIMETHYL-5-(3-PHENYL-ALLYLIDENE)\-PYRIMIDINE-2,4,6-TRIONE
1,3-DIMETHYL-5-\(3-PHENYL-ALLYLIDENE\)-PYRIMIDINE-2,4,6-TRIONE

Only the minus following the parenthesis is treated as a NOT.
Are parentheses special? They're not mentioned in the eDismax documentation.

-Original Message-
From: Markus Jelsma 
Sent: Saturday, June 13, 2020 4:57 AM
To: solr-user@lucene.apache.org
Subject: RE: eDismax query syntax question

Hello,

These are special characters, if you don't need them, you must escape them.

See top of the article:
https://lucene.apache.org/solr/guide/8_5/the-extended-dismax-query-parser.html

Markus




-Original message-
> From:Webster Homer 
> Sent: Friday 12th June 2020 22:09
> To: solr-user@lucene.apache.org
> Subject: eDismax query syntax question
>
> Recently we found strange behavior in a query. We use eDismax as the query 
> parser.
>
> This is the query term:
> 1,3-DIMETHYL-5-(3-PHENYL-ALLYLIDENE)-PYRIMIDINE-2,4,6-TRIONE
>
> It should hit one document in our index. It does not. However, if you use the 
> Dismax query parser it does match the record.
>
> The problem seems to involve the parenthesis and the dashes. If you
> escape the dash after the parenthesis it matches
> 1,3-DIMETHYL-5-(3-PHENYL-ALLYLIDENE)\-PYRIMIDINE-2,4,6-TRIONE
>
> I thought that eDismax and Dismax escaped all lucene special characters 
> before passing the query to lucene. Although I also remember reading that + 
> and - can have special significance in a query if preceded with white space. 
> I can find very little documentation on either query parser in how they work.
>
> Is this expected behavior or is this a bug? If expected, where can I find 
> documentation?
>
>
>
> This message and any attachment are confidential and may be privileged or 
> otherwise protected from disclosure. If you are not the intended recipient, 
> you must not copy this message or attachment or disclose the contents to any 
> other person. If you have received this transmission in error, please notify 
> the sender immediately and delete the message and any attachment from your 
> system. Merck KGaA, Darmstadt, Germany and any of its subsidiaries do not 
> accept liability for any omissions or errors in this message which may arise 
> as a result of E-Mail-transmission or for damages resulting from any 
> unauthorized changes of the content of this message and any attachment 
> thereto. Merck KGaA, Darmstadt, Germany and any of its subsidiaries do not 
> guarantee that this message is free of viruses and does not accept liability 
> for any damages caused by any virus transmitted therewith.
>
>
>
> Click http://www.merckgroup.com/disclaimer to access the German, French, 
> Spanish and Portuguese versions of this disclaimer.
>


This message and any attachment are confidential and may be privileged or 
otherwise protected from disclosure. If you are not the intended recipient, you 
must not copy this message or attachment or disclose the contents to any other 
person. If you have received this transmission in error, please notify the 
sender immediately and delete the message and any attachment from your system. 
Merck KGaA, Darmstadt, Germany and any of its subsidiaries do not accept 
liability for any omissions or errors in this message which may arise as a 
result of E-Mail-transmission or for damages resulting from any unauthorized 
changes of the content of this message and any attachment thereto. Merck KGaA, 
Darmstadt, Germany and any of its subsidiaries do not guarantee that this 
message is free of viruses and does not accept liability for any damages caused 
by any virus transmitted therewith.



Click http://www.merckgroup.com/disclaimer to access the German, French, 
Spanish and Portuguese versions of this disclaimer.

Re: Question about Atomic Update

[Erick Erickson]

All Atomic Updates do is 
1> read all the stored fields from the record being updated
2> overlay your updates
3> re-index the document.

At <3> it’s exactly as though you sent the entire document
again, so your observation that the whole document is 
re-indexed is accurate.

If the fields you want to update are single-valued, docValues=true
numeric fields you can update those without the whole doc being
re-indexed. But if you need to search on those fields it’ll probably
be unacceptably slow. However, if you _do_ need to search,
sometimes you can get creative with function queries. OK, this
last is opaque but say you have a “quantity” field and only want to
find docs that have quantity > 0. You can add a function query
to your query (either q or fq) that returns the value of that field,
which means the score is 0 for docs where quantity==0 and the
doc drops out of the result set.

It’s not clear whether you search the text field, but if not you can
store it somewhere else and only fetch it as needed.

Best,
Erick

> On Jun 15, 2020, at 7:55 AM, david.dav...@correo.aeat.es wrote:
> 
> Hi,
> 
> I have a question related with atomic update in Solr.
> 
> In our collection,  documents have a lot of fields, most of them small. 
> However, there is one of them that includes the text of the document. 
> Sometimes, not many fortunatelly, this text is very long, more than 3 or 4 
> MB of plain text. We use different analyzers such as synonyms, etc. and 
> this causes that index time in that documents is long, about 15 seconds.
> 
> Sometimes, we should update some small fields, and it is a big problem for 
> us because of the time that it consumes. We have been testing with atomic 
> update, but time is exactly the same than sending the document again. We 
> expected that with atomic update only the updated fields were indexed and 
> time would reduce. But it seems that internally Solr gets the whole 
> document and reindex all the fields.
> 
> Does it works in that way? Am I worng, any advice?
> 
> We have tested with Solr 7.4 and Solr 4.10
> 
> Thanks,
> 
> David 

Question about Atomic Update

[david . davila]

Hi,

I have a question related with atomic update in Solr.

In our collection,  documents have a lot of fields, most of them small. 
However, there is one of them that includes the text of the document. 
Sometimes, not many fortunatelly, this text is very long, more than 3 or 4 
MB of plain text. We use different analyzers such as synonyms, etc. and 
this causes that index time in that documents is long, about 15 seconds.

Sometimes, we should update some small fields, and it is a big problem for 
us because of the time that it consumes. We have been testing with atomic 
update, but time is exactly the same than sending the document again. We 
expected that with atomic update only the updated fields were indexed and 
time would reduce. But it seems that internally Solr gets the whole 
document and reindex all the fields.

Does it works in that way? Am I worng, any advice?

We have tested with Solr 7.4 and Solr 4.10

Thanks,

David 

Backup and Restore Collections

[sudhir kumar]

Hi All,

I have a question on Solr Backup and Restore features of Collection.

Right now, we have a use case where we are taking backup of prod collection
on a daily basis and creating the same collection on lower env.

And we are on Solr cloud 7.7

We are following the below process to backup collections using a scheduler.

1) Daily delete backed up prod collection in lower env.
2) Create prod collection in lower env using backup and restore feature of
cloud. -https://lucene.apache.org/solr/guide/7_2/collections-api.html#backup

Below are somethings that I have noticed

1) I see the daily backup scheduler is creating replicas on different
nodes. Sometimes multiple replicas are created on one node and
sometimes all replicas are created on different nodes.
2) I don't see that the restored collections configs are on zk.

Can anyone suggest to me, is this the right way of taking backups for my
use case ?

or
 Shall I create all collections once and use a standalone mode of backups
and restore api on each replica to update the documents, instead of
deleting the collection on a daily basis and what could be the downsides of
it.

Please suggest

Thanks,
Sudhir.

Can't fetch table from cassandra through jdbc connection

[Ирина Камалова]

I’m using Solr 7.7.3 and latest Cassandra jdbc driver 1.3.5

I get  *SQLFeatureNotSupportedException *


I see this error and have no idea what’s wrong (not enough verbose - table
name or field wrong/ couldn’t mapping type or driver doesn’t support?)


Full Import failed:java.lang.RuntimeException:
java.lang.RuntimeException:
org.apache.solr.handler.dataimport.DataImportHandlerException: Unable
to execute query: select test_field from test_keyspace.test_table
limit 10; Processing Document # 1
at 
org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:271)
at 
org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:424)
at 
org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:483)
at 
org.apache.solr.handler.dataimport.DataImporter.lambda$runAsync$0(DataImporter.java:466)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException:
org.apache.solr.handler.dataimport.DataImportHandlerException: Unable
to execute query: select test_field from test_keyspace.test_table
limit 10; Processing Document # 1
at 
org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:417)
at 
org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:330)
at 
org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:233)
... 4 more
Caused by: org.apache.solr.handler.dataimport.DataImportHandlerException:
Unable to execute query: select test_field from
test_keyspace.test_table limit 10; Processing Document # 1
at 
org.apache.solr.handler.dataimport.DataImportHandlerException.wrapAndThrow(DataImportHandlerException.java:69)
at 
org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.(JdbcDataSource.java:327)
at 
org.apache.solr.handler.dataimport.JdbcDataSource.createResultSetIterator(JdbcDataSource.java:288)
at 
org.apache.solr.handler.dataimport.JdbcDataSource.getData(JdbcDataSource.java:283)
at 
org.apache.solr.handler.dataimport.JdbcDataSource.getData(JdbcDataSource.java:52)
at 
org.apache.solr.handler.dataimport.SqlEntityProcessor.initQuery(SqlEntityProcessor.java:59)
at 
org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:73)
at 
org.apache.solr.handler.dataimport.EntityProcessorWrapper.nextRow(EntityProcessorWrapper.java:267)
at 
org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:476)
at 
org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:415)
... 6 more
Caused by: java.sql.SQLFeatureNotSupportedException
at 
com.dbschema.CassandraConnection.createStatement(CassandraConnection.java:75)
at 
org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.createStatement(JdbcDataSource.java:342)
at 
org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.(JdbcDataSource.java:318)
... 14 more




Best regards,
Irina Kamalova

Regarding AWS S3 bucket data to be indexed in apache solr

[Ajeet Jha]

Hi apache solr team,
This is Ajeet Jha, just want to know that,
Is their any way to index AWS S3 bucket data in apache solr directly,
Looking for something like this, Elasticsearch pulls data from S3 bucket,
using "es-amazon-s3-river" plugin. URL
https://github.com/lbroudoux/es-amazon-s3-river. If their is a way to
implement such thing please let me know.


Regards
Ajeet Jh