Re: Fetch parent and child document in solr 8.2

[Mikhail Khludnev]

There are some changes in nested docs, a kind of named scopes, or so. The
query you provided violated of one of the block join principles. There
should be caveat panel regarding this in the doc.

On Fri, Nov 15, 2019 at 9:46 PM Gajjar, Jigar  wrote:

> Hello,
>
>
>
> I am trying to fetch parent and child document together in one Solr query,
> I was able to do that in solr 7.4 but same query does not work in solr 8.2.
>
> Are there any major changes in the way that we are fetching children?
>
>
>
> My requirement is to fetch parent and children both in one call.
>
>
>
> I am trying
>
>
>
> http://localhost:8983/solr/demo/select?fl=*,[child]={!parent
> which="cat_s:sci-fi AND pubyear_i:1992"}
>
>
>
> what are the ways to retrieve parent child as nested documents?
>
>
>
>
>
>
>
> Thanks,
>
> Jigar Gajjar
>
> *OCLC* · Senior Software  Engineer
>
> 6565 Kilgour Place, Dublin, OH, USA, 43017
>
>  *M* +1-408-334-6379
>
> [image: OCLC] 
>
> OCLC.org  *·* Blog
>  *·* Facebook
>  *·* Twitter
>  *·* YouTube 
>
>
>


-- 
Sincerely yours
Mikhail Khludnev

Re: Changing the IP or a SOLR or Zookeeper host

[Houston Putman]

If you are changing the IP of a host running Zookeeper, it can be an issue.
It depends on the version of zookeeper that you are using. There was an
issue with ZK not re-resolving IP addresses on connection errors, but it
was fixed in 3.4.13 (
https://issues.apache.org/jira/plugins/servlet/mobile#issue/ZOOKEEPER-2184
). If you are using 3.5.x, you should be safe as the fix was made there as
well.

You should be fine doing this for Solr nodes, but Java has weird default
settings for caching DNS, where it can cache a DNS resolution infinitely
and never refresh it. Therefore I'd say you're safest if you do a rolling
restart of your cluster, and any clients using solrJ. (Unless you make sure
that you have the correct Java DNS cache settings)

On Tue, Nov 19, 2019, 2:52 PM dshih  wrote:

> If a SOLR or Zookeeper host is replaced such that the hostname comes back
> but
> it now resolves to a different IP, are SOLR nodes and SOLRJ clients
> expected
> to just continue working?
>
> I know that in this case replicas need to be re-created (and stale ones
> deleted).  But I'm wondering if SOLRJ clients and/or other SOLR nodes need
> to be restarted to pick up the hostname->IP change.
>
>
>
> --
> Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html
>

RE: async BACKUP under Solr8.3

[Oakley, Craig (NIH/NLM/NCBI) [C]]

In some collections I am having problems with Solr8.1.1 through 8.3; with other 
collections it is fine in Solr8.1.1 through 8.3

I'm investigating what might be wrong with the collections which have the 
problems.

Thanks

-Original Message-
From: Oakley, Craig (NIH/NLM/NCBI) [C]  
Sent: Tuesday, November 19, 2019 9:53 AM
To: solr-user@lucene.apache.org
Subject: RE: async BACKUP under Solr8.3

FYI, I DO succeed in doing an async backup in Solr8.1

-Original Message-
From: Oakley, Craig (NIH/NLM/NCBI) [C]  
Sent: Tuesday, November 19, 2019 9:03 AM
To: solr-user@lucene.apache.org
Subject: RE: async BACKUP under Solr8.3

This is on a test server: simple case: one node, one shard, one replica

In production we currently use Solr7.4 and the async BACKUP works fine. I could 
test whether I get the same symptoms on Solr8.1 and/or 8.2

Thanks

-Original Message-
From: Mikhail Khludnev  
Sent: Tuesday, November 19, 2019 12:40 AM
To: solr-user 
Subject: Re: async BACKUP under Solr8.3

Hello, Craig.
There was a significant  fix for async BACKUP in 8.1, if I remember it
correctly.
Which version you used for it before? How many nodes, shards, replicas
`bug` has?
Unfortunately this stacktrace is not really representative, it just says
that some node (ok, it's overseer) fails to wait another one.
Ideally we need a log from overseer node and subordinate node during backup
operation.
Thanks.

On Tue, Nov 19, 2019 at 2:13 AM Oakley, Craig (NIH/NLM/NCBI) [C]
 wrote:

> For Solr 8.3, when I attempt a command of the form
>
>
> host:port/solr/admin/collections?action=BACKUP=snapshot1=col1=/tmp=bug
>
> And then when I run
> /solr/admin/collections?action=REQUESTSTATUS=bug I get
> "msg":"found [bug] in failed tasks"
>
> The solr.log file has a stack trace like the following
> 2019-11-18 17:31:31.369 ERROR
> (OverseerThreadFactory-9-thread-5-processing-n:host:port_solr) [c:col1   ]
> o.a.s.c.a.c.OverseerCollectionMessageHandler Error from shard:
> http://host:port/solr =>
> org.apache.solr.client.solrj.SolrServerException: Timeout occured while
> waiting response from server at: http://host:port/solr/admin/cores
> at
> org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:408)
> org.apache.solr.client.solrj.SolrServerException: Timeout occured while
> waiting response from server at: http://host:port/solr/admin/cores
> at
> org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:408)
> ~[?:?]
> at
> org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:754)
> ~[?:?]
> at
> org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1290) ~[?:?]
> at
> org.apache.solr.handler.component.HttpShardHandler.request(HttpShardHandler.java:238)
> ~[?:?]
> at
> org.apache.solr.handler.component.HttpShardHandler.lambda$submit$0(HttpShardHandler.java:199)
> ~[?:?]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ~[?:1.8.0_232]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[?:1.8.0_232]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ~[?:1.8.0_232]
> at
> com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:181)
> ~[metrics-core-4.0.5.jar:4.0.5]
> at
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:210)
> ~[?:?]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> ~[?:1.8.0_232]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> ~[?:1.8.0_232]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_232]
> Caused by: java.util.concurrent.TimeoutException
> at
> org.eclipse.jetty.client.util.InputStreamResponseListener.get(InputStreamResponseListener.java:216)
> ~[?:?]
> at
> org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:399)
> ~[?:?]
> ... 12 more
>
> If I remove the async=bug, then it works
>
> In fact, the backup looks successful, but REQUESTSTATUS does not recognize
> it as such
>
> I notice that the 3:30am 11/4/19 Email to solr-user@lucene.apache.org
> mentions in Solr 8.3.0 Release Highlights "Fix for SPLITSHARD (async) with
> failures in underlying sub-operations can result in data loss"
>
> Did a fix to SPLITSHARD break BACKUP?
>
> Has anyone been successful running
> solr/admin/collections?action=BACKUP=requestname under Solr8.3?
>
> Thanks
>


-- 
Sincerely yours
Mikhail Khludnev

Changing the IP or a SOLR or Zookeeper host

[dshih]

If a SOLR or Zookeeper host is replaced such that the hostname comes back but
it now resolves to a different IP, are SOLR nodes and SOLRJ clients expected
to just continue working?  

I know that in this case replicas need to be re-created (and stale ones
deleted).  But I'm wondering if SOLRJ clients and/or other SOLR nodes need
to be restarted to pick up the hostname->IP change.



--
Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Fetch parent and child document in solr 8.2

[Gajjar, Jigar]

Can anyone please answer my question. Let me know if I am going in wrong 
direction.

Thanks,
Jigar Gajjar
OCLC · Senior Software  Engineer
6565 Kilgour Place, Dublin, OH, USA, 43017
 M +1-408-334-6379
[OCLC]
OCLC.org · 
Blog · 
Facebook · 
Twitter · YouTube


From: "Gajjar, Jigar" 
Date: Friday, November 15, 2019 at 1:46 PM
To: "solr-user@lucene.apache.org" 
Subject: Fetch parent and child document in solr 8.2

Hello,

I am trying to fetch parent and child document together in one Solr query, I 
was able to do that in solr 7.4 but same query does not work in solr 8.2.
Are there any major changes in the way that we are fetching children?

My requirement is to fetch parent and children both in one call.

I am trying

http://localhost:8983/solr/demo/select?fl=*,[child]={!parent 
which="cat_s:sci-fi AND pubyear_i:1992"}

what are the ways to retrieve parent child as nested documents?



Thanks,
Jigar Gajjar
OCLC · Senior Software  Engineer
6565 Kilgour Place, Dublin, OH, USA, 43017
 M +1-408-334-6379
[OCLC]
OCLC.org · 
Blog · 
Facebook · 
Twitter · YouTube

Re: Updates blocked in Tlog solr cloud?

[Wei]

Hi Erick,

I observed that the update request rate dropped from 20 per sec to 3 per
sec for about 8 minutes. After that there is a huge burst of updates. This
looks quite match the queue up behavior you mentioned. But I don't think
the time out took that long. Is there a configurable setting for the time
out?
Also the bad tlog replica is not reachable at the time, so we did a
DELETEREPLICA command with collections API to remove it from the cloud.

Thanks,
Wei


On Tue, Nov 19, 2019 at 5:52 AM Erick Erickson 
wrote:

> How long are updates blocked and how did the tlog replica on the bad
> hardware go down?
>
> Solr has to wait for an ack back from the tlog follower to be certain that
> the follower has all the documents in case it has to switch to that replica
> to become the leader. If the update to the follower times out, the leader
> will put it into a recovering state.
>
> So I’d expect the collection to queue up indexing until the request to the
> follower on the bad hardware timed out, did you wait at least that long?
>
> Best,
> Erick
>
> > On Nov 18, 2019, at 7:11 PM, Wei  wrote:
> >
> > Hi,
> >
> > I am puzzled by a problem in solr cloud with Tlog replicas and would
> > appreciate your insights.  Our solr cloud has two shards and each shard
> > have 5 tlog replicas. When one of the non-leader replica has hardware
> issue
> > and become unreachable,  updates to the whole cloud stopped.  We are on
> > solr 7.6 and use solrj client to send updates only to leaders.  To my
> > understanding,  with Tlog replica type, the leader only forward update
> > requests to replicas for transaction log update and each replica
> > periodically pulls the segment from leader.  When one replica fails to
> > respond,  why update requests to the cloud are blocked?  Does leader need
> > to wait for response from each replica to inform client that update is
> > successful?
> >
> > Best,
> > Wei
>
>

Re: How do I add my own Streaming Expressions?

[Eric Pugh]

The documentation in the StreamHandler suggests adding into Solrconfig some 
streamFunctions:

 *  
 *  org.apache.solr.client.solrj.io.stream.ReducerStream 
 *  org.apache.solr.client.solrj.io.stream.RecordCountStream 
 * 


https://github.com/apache/lucene-solr/blob/master/solr/core/src/java/org/apache/solr/handler/StreamHandler.java#L114

What is happening in StreamHandler doesn’t seem to be working, however in the 
similar GraphHandler, there is a call to “streamFunctions”:

https://github.com/apache/lucene-solr/blob/master/solr/core/src/java/org/apache/solr/handler/GraphHandler.java#L90

I’m still debugging this…

Eric



> On Nov 15, 2019, at 9:43 PM, Eric Pugh  
> wrote:
> 
> What is the process for adding new Streaming Expressions?   
> 
> It appears that the org.apache.solr.client.solrj.io.Lang method statically 
> loads all the streaming expressions?
> 
> Eric
> 
> ___
> Eric Pugh | Founder & CEO | OpenSource Connections, LLC | 434.466.1467 | 
> http://www.opensourceconnections.com  
> | My Free/Busy   
> Co-Author: Apache Solr Enterprise Search Server, 3rd Ed 
> 
>   
> This e-mail and all contents, including attachments, is considered to be 
> Company Confidential unless explicitly stated otherwise, regardless of 
> whether attachments are marked as such.
> 

___
Eric Pugh | Founder & CEO | OpenSource Connections, LLC | 434.466.1467 | 
http://www.opensourceconnections.com  | 
My Free/Busy   
Co-Author: Apache Solr Enterprise Search Server, 3rd Ed 


This e-mail and all contents, including attachments, is considered to be 
Company Confidential unless explicitly stated otherwise, regardless of whether 
attachments are marked as such.

exact matches on a join

[rhys J]

I am trying to do a join, which I have working properly on 2 cores.

One core has report_as, and the other core has debt_id.

If I enter 'report_as: "Freeman", I expect to get 272 results. But I get
557.

When I do a database search on the matched fields, it shows me that
report_as: "Freeman" is matching also on 'A-1 Freeman'.

I have tried boosting the score as report_as: "Freeman"^2, but I get the
same results from the API, and from the browser itself.

Here is my query:

{
  "responseHeader":{
"status":0,
"QTime":5,
"params":{
  "q":"( * )",
  "indent":"on",
  "fl":"debt_id, score",
  "cursorMark":"*",
  "sort":"score desc, id desc",
  "fq":"{!join from=debtor_id to=debt_id fromIndex=dbtr}(
report_as:\"Freeman\"^2)",
  "rows":"1000"}},
  "response":{"numFound":557,"start":0,"maxScore":1.0,"docs":[
  {
"debt_id":"485435",
"score":1.0},
  {
"debt_id":"485435",
"score":1.0},
  {
"debt_id":"482795",
"score":1.0},
  {
"debt_id":"482795",
"score":1.0},
  {
"debt_id":"482794",
"score":1.0},
  {
"debt_id":"482794",
"score":1.0},
  {
"debt_id":"482794",
"score":1.0},

SKIP



{
"debt_id":"396925",
"score":1.0},
  {
"debt_id":"396925",
"score":1.0},
  {
"debt_id":"396925",
"score":1.0},
  {
"debt_id":"396925",
"score":1.0},
  {
"debt_id":"396925",
"score":1.0},
  {
"debt_id":"396925",
"score":1.0},
  {
"debt_id":"396925",
"score":1.0},
  {
"debt_id":"396925",
"score":1.0},
  {
"debt_id":"396925",
"score":1.0},
  {
"debt_id":"396925",
"score":1.0},
  {
"debt_id":"396925",


These ones are the correct matches that I can verify with the
database, but their scores are the same as the ones matching on
'A1-Freeman'

Is my scoring set up wrong?

Thanks,

Rhys

Re: 2 Async exceptions during distributed update issue...

[Erick Erickson]

First, please don’t use the “schemaless” mode (add-unknown-fields-to-schema in 
your solrconfig) while load testing. There’s quite a bit of work being done 
when Solr discovers an unknown field that’ll cause some instability under heavy 
load.

Second, when you put a large batch against Solr, you have the possibility of 
the update just taking longer than the timeout. There are several timeouts you 
can increase, see the “solr.xml” section of the ref guide.

Best,
Erick

> On Nov 19, 2019, at 12:29 PM, Fiz N  wrote:
> 
> 
> Hello  Solr Experts, 
> 
> Just wanted to follow up in case my question.Would appreciate help on this.
>  
> SOLR Version : 6.6.2
> OS – Linux 3.1.2
> JDK – 1.8
>  
> Shard – 16 – All are active.
> Xms – 16 gb
> Xmx – 16 gb
> Host has 64 cores.
>  
> the update processor chain.attaching the complete updateRequestProcessorChain 
> in a file.
> Attaching physical and CPU memory screenshot.
>  
> There are multiple threads sending products to solr. tested with batch size 
> per thread 50 & 100, it worked fine without error and if the batch size is 
> 1000 the following error occurs.
>  
>  
> I am getting the following error when the batch size is 1000. Please advise.
>  
> 2019-11-14T17:45:44,646 - ERROR [qtp876213901-1169:SolrException@YYY] - 
> {collection=c:ducts, core=x:ducts_shard15_replica1, 
> node_name=n:10.YYY.40.81:8983_solr, replica=r:core_node30, shard=s:shard15} - 
> org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException:
>  2 Async exceptions during distributed update:
> 10.YYY.40.62:8983 failed to respond
> 10.YYY.40.62:8983 failed to respond
>  
> 2019-11-14T17:45:48,821 - ERROR [qtp876213901-202:SolrException@YYY] - 
> {collection=c:ducts, core=x:ducts_shard7_replica1, 
> node_name=n:10.YYY.40.81:8983_solr, replica=r:core_node29, shard=s:shard7} - 
> org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException:
>  Async exception during distributed update: 10.YYY.40.81:8983 failed to 
> respond
>  
>  
> 2019-11-14T19:36:11,599 - ERROR 
> [updateExecutor-2-thread-176-processing-x:ducts_shard3_replica2 
> http:10.YYY.40.68:8983//solr//ducts_shard11_replica2 r:core_node26 
> n:10.YYY.40.68:8983_solr c:ducts 
> s:shard3:StreamingSolrClients$ErrorReportingConcurrentUpdateSolrClient@131] - 
> {ConcurrentUpdateSolrClient.url=http://10.YYY.40.68:8983/solr/ducts_shard11_replica2,
>  collection=c:ducts, core=x:ducts_shard3_replica2, 
> node_name=n:10.YYY.40.68:8983_solr, replica=r:core_node26, shard=s:shard3} - 
> error
> org.apache.http.NoHttpResponseException: 10.YYY.40.68:8983 failed to respond
> at 
> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:143)
>  ~[httpclient-4.4.1.jar:4.4.1]
> at 
> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
>  ~[httpclient-4.4.1.jar:4.4.1]
> at 
> org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
>  ~[httpcore-4.4.1.jar:4.4.1]
>  
>  
> 2019-11-14T19:36:14,567 - ERROR 
> [updateExecutor-2-thread-189-processing-x:ducts_shard11_replica2 
> http:10.YYY.40.62:8983//solr//ducts_shard2_replica1 r:core_node25 
> n:10.YYY.40.68:8983_solr c:ducts 
> s:shard11:StreamingSolrClients$ErrorReportingConcurrentUpdateSolrClient@131] 
> - 
> {ConcurrentUpdateSolrClient.url=http://10.YYY.40.62:8983/solr/ducts_shard2_replica1,
>  collection=c:ducts, core=x:ducts_shard11_replica2, 
> node_name=n:10.YYY.40.68:8983_solr, replica=r:core_node25, shard=s:shard11} - 
> error
> java.net.SocketException: Broken pipe (Write failed)
> at java.net.SocketOutputStream.socketWrite0(Native Method) 
> ~[?:1.8.0_232]
> at 
> java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111) 
> ~[?:1.8.0_232]
> at java.net.SocketOutputStream.write(SocketOutputStream.java:155) 
> ~[?:1.8.0_232]
>  
>  
> 2019-11-14T19:36:38,851 - ERROR [qtp876213901-542:SolrException@159] - 
> {collection=c:ducts, core=x:ducts_shard11_replica2, 
> node_name=n:10.YYY.40.68:8983_solr, replica=r:core_node25, shard=s:shard11} - 
> null:org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException:
>  Async exception during distributed update: Broken pipe (Write failed)
>at 
> org.apache.solr.update.processor.DistributedUpdateProcessor.doFinish(DistributedUpdateProcessor.java:972)
> at 
> org.apache.solr.update.processor.DistributedUpdateProcessor.finish(DistributedUpdateProcessor.java:1911)
>  
>  
> Thanks 
>  
> 

2 Async exceptions during distributed update issue...

[Fiz N]

Hello  Solr Experts,


Just wanted to follow up in case my question.Would appreciate help on this.



SOLR Version : 6.6.2

OS – Linux 3.1.2

JDK – 1.8



Shard – 16 – All are active.

Xms – 16 gb

Xmx – 16 gb

Host has 64 cores.



the update processor chain.attaching the complete updateRequestProcessorChain
in a file.

Attaching physical and CPU memory screenshot.



There are multiple threads sending products to solr. tested with batch size
per thread 50 & 100, it worked fine without error and if the batch size is
1000 the following error occurs.





*I am getting the following error when the batch size is 1000. Please
advise.*



2019-11-14T17:45:44,646 - ERROR [qtp876213901-1169:SolrException@YYY] -
{collection=c:ducts, core=x:ducts_shard15_replica1,
node_name=n:10.YYY.40.81:8983_solr, replica=r:core_node30, shard=s:shard15}
-
org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException:
2 Async exceptions during distributed update:

10.YYY.40.62:8983 failed to respond

10.YYY.40.62:8983 failed to respond



2019-11-14T17:45:48,821 - ERROR [qtp876213901-202:SolrException@YYY] -
{collection=c:ducts, core=x:ducts_shard7_replica1,
node_name=n:10.YYY.40.81:8983_solr, replica=r:core_node29, shard=s:shard7}
-
org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException:
Async exception during distributed update: 10.YYY.40.81:8983 failed to
respond





2019-11-14T19:36:11,599 - ERROR
[updateExecutor-2-thread-176-processing-x:ducts_shard3_replica2
http:10.YYY.40.68:8983//solr//ducts_shard11_replica2
 r:core_node26
n:10.YYY.40.68:8983_solr c:ducts
s:shard3:StreamingSolrClients$ErrorReportingConcurrentUpdateSolrClient@131]
- {ConcurrentUpdateSolrClient.url=
http://10.YYY.40.68:8983/solr/ducts_shard11_replica2
,
collection=c:ducts, core=x:ducts_shard3_replica2,
node_name=n:10.YYY.40.68:8983_solr, replica=r:core_node26, shard=s:shard3}
- error

org.apache.http.NoHttpResponseException: 10.YYY.40.68:8983 failed to respond

at
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:143)
~[httpclient-4.4.1.jar:4.4.1]

at
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
~[httpclient-4.4.1.jar:4.4.1]

at
org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
~[httpcore-4.4.1.jar:4.4.1]





2019-11-14T19:36:14,567 - ERROR
[updateExecutor-2-thread-189-processing-x:ducts_shard11_replica2
http:10.YYY.40.62:8983//solr//ducts_shard2_replica1
 r:core_node25
n:10.YYY.40.68:8983_solr c:ducts
s:shard11:StreamingSolrClients$ErrorReportingConcurrentUpdateSolrClient@131]
- {ConcurrentUpdateSolrClient.url=
http://10.YYY.40.62:8983/solr/ducts_shard2_replica1
,
collection=c:ducts, core=x:ducts_shard11_replica2,
node_name=n:10.YYY.40.68:8983_solr, replica=r:core_node25, shard=s:shard11}
- error

java.net.SocketException: Broken pipe (Write failed)

at java.net.SocketOutputStream.socketWrite0(Native Method)
~[?:1.8.0_232]

at
java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111)
~[?:1.8.0_232]

at java.net.SocketOutputStream.write(SocketOutputStream.java:155)
~[?:1.8.0_232]





2019-11-14T19:36:38,851 - ERROR [qtp876213901-542:SolrException@159] -
{collection=c:ducts, core=x:ducts_shard11_replica2,
node_name=n:10.YYY.40.68:8983_solr, replica=r:core_node25, shard=s:shard11}
-
null:org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException:
Async exception during distributed update: Broken pipe (Write failed)

   at
org.apache.solr.update.processor.DistributedUpdateProcessor.doFinish(DistributedUpdateProcessor.java:972)

at
org.apache.solr.update.processor.DistributedUpdateProcessor.finish(DistributedUpdateProcessor.java:1911)





Thanks

  




  [^\w-\.]
  _





  
-MM-dd'T'HH:mm:ss.SSSZ
-MM-dd'T'HH:mm:ss,SSSZ
-MM-dd'T'HH:mm:ss.SSS
-MM-dd'T'HH:mm:ss,SSS
-MM-dd'T'HH:mm:ssZ
-MM-dd'T'HH:mm:ss
-MM-dd'T'HH:mmZ
-MM-dd'T'HH:mm
-MM-dd HH:mm:ss.SSSZ
-MM-dd HH:mm:ss,SSSZ
-MM-dd HH:mm:ss.SSS
-MM-dd HH:mm:ss,SSS
-MM-dd HH:mm:ssZ
-MM-dd HH:mm:ss
-MM-dd HH:mmZ
-MM-dd HH:mm
-MM-dd
  


  strings
  
java.lang.Boolean
booleans
  
  
java.util.Date
tdates
  
  
java.lang.Long
java.lang.Integer
tlongs
  
  
java.lang.Number
tdoubles
  




  
  
  
  
  
  

Re: Upgrade solr from 7.2.1 to 8.2

[Rahul Goswami]

Hello,

Just wanted to follow up in case my question fell through the cracks :)
Would appreciate help on this.

Thanks,
Rahul

On Fri, Nov 15, 2019 at 5:32 PM Rahul Goswami  wrote:

> Hello,
>
> We are planning to upgrade our SolrCloud cluster from 7.2.1 (hosted on
> Windows server) to 8.2.
> I read the documentation
> 
> which mentions that I need to be on Solr 7.3 and higher to be able to
> upgrade to 8.x. I want to know if this is a hard requirement or a
> requirement for rolling upgrades (?).
> Let's say I am fine with bringing the whole cluster down and upgrade all
> the nodes to Solr 8.2, and then bring up one node at a time. Will it be ok
> to upgrade directly from 7.2.1 to 8.2 in that case?
>
> Thanks in advance!
>
> Regards,
> Rahul
>

Metrics avgRequestsPerSecond and avgRequestsPerSecond from documentation gone?

[Koen De Groote]

Greetings,

I'm using Solr 7.6 and have enabled JMX metrics.

I ran into this page:
https://lucene.apache.org/solr/guide/7_6/performance-statistics-reference.html#commonly-used-stats-for-request-handlers

Which mentions "avgRequestsPerSecond" and "avgTimePerRequest" and some
other attributes, which do not exist anymore in this version. I have an
older version(4) I spun up to have a look and they do exist in that version.

When getting info on a QUERY or UPDATE bean with name `requestTimes`, I get
this:

# attributes
  %0   - 50thPercentile (double, r)
  %1   - 75thPercentile (double, r)
  %2   - 95thPercentile (double, r)
  %3   - 98thPercentile (double, r)
  %4   - 999thPercentile (double, r)
  %5   - 99thPercentile (double, r)
  %6   - Count (long, r)
  %7   - DurationUnit (java.lang.String, r)
  %8   - FifteenMinuteRate (double, r)
  %9   - FiveMinuteRate (double, r)
  %10  - Max (double, r)
  %11  - Mean (double, r)
  %12  - MeanRate (double, r)
  %13  - Min (double, r)
  %14  - OneMinuteRate (double, r)
  %15  - RateUnit (java.lang.String, r)
  %16  - StdDev (double, r)
  %17  - _instanceTag (java.lang.String, r)
# operations
  %0   - javax.management.ObjectName objectName()
  %1   - [J values()
#there's no notifications

And it seems that none of the current values are actually a proper
replacement for the functionality these values used to offer.

How shall I go about getting this info now? Do I need to combine several
other metrics?

For completeness sake, my solr.xml, where I enabled JMX, is just the
default example from the documentation, with JMX added:




${host:}
${jetty.port:8983}
${hostContext:solr}
${zkClientTimeout:15000}
${genericCoreNodeNames:true}


${socketTimeout:0}
${connTimeout:0}



javax.net.ssl.keyStorePassword
javax.net.ssl.trustStorePassword
basicauth
zkDigestPassword
zkDigestReadonlyPassword


very_obvious_name_for_easy_reading_${jetty.port:8983}





Kind regards,
Koen De Groote

Re: Explicit documentation regarding all JMX metrics?

[Koen De Groote]

Thanks. I also just now found
https://lucene.apache.org/solr/guide/7_6/metrics-reporting.html

I'm currently sticking to version 7.6

I've enabled JMX metrics and am accessing them via JmxTerm. There are
indeed a lot, though filtering by key=value is easy enough.

That being said, finding out what they mean is a bit harder.

Also, it seems that even the most recent documentation is out of date.

For instance here:
https://lucene.apache.org/solr/guide/8_2/performance-statistics-reference.html#commonly-used-stats-for-request-handlers

At the bottom there is mention of "avgRequestsPerSecond" and
"avgTimePerRequest", which in fact do not exist anymore. I can see them on
older versions of solr, but not on 7.6

And it seems nothing with identical meaning has taken its place.

I'll make a separate question for that.




On Tue, Nov 19, 2019 at 2:40 PM Erick Erickson 
wrote:

> Here’s something that might be useful:
> https://support.lucidworks.com/s/article/What-should-you-monitor
>
> What version of Solr? For Solr 6.4+, you may want to use the
> /admin/metrics endpoint, see the reference guide for metrics. WARNING:
> there are a ton of metrics. The ref guide has information on now to trim
> down what you’re looking at.
>
> Best,
> Erick
>
> > On Nov 19, 2019, at 8:28 AM, Koen De Groote 
> wrote:
> >
> > Greetings,
> >
> > I've enabled JMX metrics on an install of mine and am going through the
> > available beans.
> >
> > I've google for an explanation of what the beans all mean, but I'm coming
> > up empty.
> >
> > For instance, these:
> >
> > :category=ADMIN,...,name=clientErrors,scope=/admin/file
> > :category=ADMIN,...,name=clientErrors,scope=/admin/logging
> > :category=ADMIN,...,name=clientErrors,scope=/admin/luke
> > :category=ADMIN,...,name=clientErrors,scope=/admin/mbeans
> > :category=ADMIN,...,name=clientErrors,scope=/admin/ping
> > :category=ADMIN,...,name=clientErrors,scope=/admin/plugins
> > :category=ADMIN,...,name=clientErrors,scope=/admin/properties
> > :category=ADMIN,...,name=clientErrors,scope=/admin/segments
> > :category=ADMIN,...,name=clientErrors,scope=/admin/system
> > :category=ADMIN,...,name=clientErrors,scope=/admin/threads
> > :category=ADMIN,...,name=clientErrors,scope=/config
> > :category=ADMIN,...,name=clientErrors,scope=/schema
> >
> > What am I looking at here?
> >
> > I can't find a single page explaining what the significance of these
> > metrics is.
> >
> > Nor can I find a page that basically says "You'll want to monitor X, Y
> and
> > Z get a good view of your cluster".
> >
> > Can anyone provide an example?
> >
> > Kind regards,
> > Koen De Groote
>
>

Re: Zk upconfig command is appending local directory to default confdir

[Walter Underwood]

I found the zk uploading stuff to be under-documented. Plus, it requires 
installing Solr on the deployment machine.

So I used the Python kazoo package and wrote my own uploader.

wunder
Walter Underwood
wun...@wunderwood.org
http://observer.wunderwood.org/  (my blog)

> On Nov 19, 2019, at 5:42 AM, Erick Erickson  wrote:
> 
> And to make it more complicated, if you don’t have a version that has the 
> CONFIGSET API, you can use bin/solr zk upconfig/downconfig…
> 
> Best,
> Erick
> 
>> On Nov 19, 2019, at 8:18 AM, Jörn Franke  wrote:
>> 
>> You can call it via curl and it is no more complicated then zk- in the long 
>> run you will be happier 
>> 
>>> Am 19.11.2019 um 13:27 schrieb Michael Becker :
>>> 
>>> Jörn,
>>> 
>>> We're using Solr to index a website. Basically, we set up Solr and 
>>> Zookeeper once and let it run. The API is a bit too much overhead for 
>>> something we just set once.
>>> 
>>> Mike
>>> 
>>> -Original Message-
>>> From: Jörn Franke 
>>> Sent: Tuesday, November 19, 2019 2:54 AM
>>> To: solr-user@lucene.apache.org
>>> Subject: Re: Zk upconfig command is appending local directory to default 
>>> confdir
>>> 
>>> I would use the config set API - it is more clean for production 
>>> deployments and you do not have to deal with the zkCli script:
>>> 
>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__lucene.apache.org_solr_guide_7-5F4_configsets-2Dapi.html=DwIFaQ=mV61MqUbMWY-LlirxZJvJg=8W5Kk7fJg_C2taHlYyXZLinEFeEtcUcFddYrS5aUhiE=czV2n50m-v9_w3N63i-jYUsBfnCtCKeE3wNXq3KVerU=8GQix9ABHkY0tm7kPKL_GEXCi-G4PKRjJjN-FqQcRVI=
>>>  
>>> 
 Am 18.11.2019 um 15:48 schrieb Michael Becker :
 
 I’ve run into an issue when attempting to configure Zookeeper. When 
 running the zk upconfig -d command specifying a local directory where the 
 solrconfig.xml files are located, I get the following error:
 “Could not complete upconfig operation for reason: Could not find 
 solrconfig.xml at /opt/solr-6.5.1/server/solr/configsets/solrconfig.xml, 
 /opt/solr-6.5.1/server/solr/configsets/conf/solrconfig.xml or 
 /opt/solr-6.5.1/server/solr/configsets/  
 /solrconfig.xml”
 
 I’m trying to determine why the solr zk upconfig command is appending 
 my local directory to the default confdir, rather than looking for the XML 
 files in that directory, I have two other environments with Solr where 
 this does not occur. It’s just this one environment that is having this 
 issue.
 I am using Solr version 6.5.1.
 Any suggestions on how to troubleshoot this would be appreciated.
 
 Mike
> 

Possible synchronization bug in Solr reader

[Bram Biesbrouck]

Hi all,

I think I might have discovered a synchronization bug when ingesting a lot
of data into Solr, but want to check with the specialists first ;-)

I'm using a little custom written map/reduce framework that boots a
20-something threads to do some heavy processing on data-preparation. When
this processing is done, the results of these threads are gathers in a
reduce step, where they are ingested into an (embedded) Solr instance. To
maximize throughput, I'm ingesting the data in parallel in a couple of
threads of their own and this is where I run into a synchronization error.

As with all synchronization bugs, it happens "some" of the time and they're
hard to debug, but I think I managed to get my finger on the root (I'm
using Solr 8.3):

in class org.apache.lucene.index.CodecReader, throws a NPE on line 84:
getFieldsReader().visitDocument(docID, visitor);

The issue is that the getFieldsReader() getter is mapped to a ThreadLocal
(more explicitly,
org.apache.lucene.index.SegmentCoreReaders.fieldsReaderLocal) that seems to
be released (set to null) somewhere automatically, and read afterwards,
without synchronizing the two.

I don't think I should set any resource locks of my own, since I'm only
using the SolrJ API and the /update endpoint.

I know this is quite a low-level question, but could anyone point me in the
right direction to further investigate this issue? Ie, what could be the
reason the reader is released out-of-sync?

best,

b.

RE: async BACKUP under Solr8.3

[Oakley, Craig (NIH/NLM/NCBI) [C]]

FYI, I DO succeed in doing an async backup in Solr8.1

-Original Message-
From: Oakley, Craig (NIH/NLM/NCBI) [C]  
Sent: Tuesday, November 19, 2019 9:03 AM
To: solr-user@lucene.apache.org
Subject: RE: async BACKUP under Solr8.3

This is on a test server: simple case: one node, one shard, one replica

In production we currently use Solr7.4 and the async BACKUP works fine. I could 
test whether I get the same symptoms on Solr8.1 and/or 8.2

Thanks

-Original Message-
From: Mikhail Khludnev  
Sent: Tuesday, November 19, 2019 12:40 AM
To: solr-user 
Subject: Re: async BACKUP under Solr8.3

Hello, Craig.
There was a significant  fix for async BACKUP in 8.1, if I remember it
correctly.
Which version you used for it before? How many nodes, shards, replicas
`bug` has?
Unfortunately this stacktrace is not really representative, it just says
that some node (ok, it's overseer) fails to wait another one.
Ideally we need a log from overseer node and subordinate node during backup
operation.
Thanks.

On Tue, Nov 19, 2019 at 2:13 AM Oakley, Craig (NIH/NLM/NCBI) [C]
 wrote:

> For Solr 8.3, when I attempt a command of the form
>
>
> host:port/solr/admin/collections?action=BACKUP=snapshot1=col1=/tmp=bug
>
> And then when I run
> /solr/admin/collections?action=REQUESTSTATUS=bug I get
> "msg":"found [bug] in failed tasks"
>
> The solr.log file has a stack trace like the following
> 2019-11-18 17:31:31.369 ERROR
> (OverseerThreadFactory-9-thread-5-processing-n:host:port_solr) [c:col1   ]
> o.a.s.c.a.c.OverseerCollectionMessageHandler Error from shard:
> http://host:port/solr =>
> org.apache.solr.client.solrj.SolrServerException: Timeout occured while
> waiting response from server at: http://host:port/solr/admin/cores
> at
> org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:408)
> org.apache.solr.client.solrj.SolrServerException: Timeout occured while
> waiting response from server at: http://host:port/solr/admin/cores
> at
> org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:408)
> ~[?:?]
> at
> org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:754)
> ~[?:?]
> at
> org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1290) ~[?:?]
> at
> org.apache.solr.handler.component.HttpShardHandler.request(HttpShardHandler.java:238)
> ~[?:?]
> at
> org.apache.solr.handler.component.HttpShardHandler.lambda$submit$0(HttpShardHandler.java:199)
> ~[?:?]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ~[?:1.8.0_232]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[?:1.8.0_232]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ~[?:1.8.0_232]
> at
> com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:181)
> ~[metrics-core-4.0.5.jar:4.0.5]
> at
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:210)
> ~[?:?]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> ~[?:1.8.0_232]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> ~[?:1.8.0_232]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_232]
> Caused by: java.util.concurrent.TimeoutException
> at
> org.eclipse.jetty.client.util.InputStreamResponseListener.get(InputStreamResponseListener.java:216)
> ~[?:?]
> at
> org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:399)
> ~[?:?]
> ... 12 more
>
> If I remove the async=bug, then it works
>
> In fact, the backup looks successful, but REQUESTSTATUS does not recognize
> it as such
>
> I notice that the 3:30am 11/4/19 Email to solr-user@lucene.apache.org
> mentions in Solr 8.3.0 Release Highlights "Fix for SPLITSHARD (async) with
> failures in underlying sub-operations can result in data loss"
>
> Did a fix to SPLITSHARD break BACKUP?
>
> Has anyone been successful running
> solr/admin/collections?action=BACKUP=requestname under Solr8.3?
>
> Thanks
>


-- 
Sincerely yours
Mikhail Khludnev

RE: async BACKUP under Solr8.3

[Oakley, Craig (NIH/NLM/NCBI) [C]]

This is on a test server: simple case: one node, one shard, one replica

In production we currently use Solr7.4 and the async BACKUP works fine. I could 
test whether I get the same symptoms on Solr8.1 and/or 8.2

Thanks

-Original Message-
From: Mikhail Khludnev  
Sent: Tuesday, November 19, 2019 12:40 AM
To: solr-user 
Subject: Re: async BACKUP under Solr8.3

Hello, Craig.
There was a significant  fix for async BACKUP in 8.1, if I remember it
correctly.
Which version you used for it before? How many nodes, shards, replicas
`bug` has?
Unfortunately this stacktrace is not really representative, it just says
that some node (ok, it's overseer) fails to wait another one.
Ideally we need a log from overseer node and subordinate node during backup
operation.
Thanks.

On Tue, Nov 19, 2019 at 2:13 AM Oakley, Craig (NIH/NLM/NCBI) [C]
 wrote:

> For Solr 8.3, when I attempt a command of the form
>
>
> host:port/solr/admin/collections?action=BACKUP=snapshot1=col1=/tmp=bug
>
> And then when I run
> /solr/admin/collections?action=REQUESTSTATUS=bug I get
> "msg":"found [bug] in failed tasks"
>
> The solr.log file has a stack trace like the following
> 2019-11-18 17:31:31.369 ERROR
> (OverseerThreadFactory-9-thread-5-processing-n:host:port_solr) [c:col1   ]
> o.a.s.c.a.c.OverseerCollectionMessageHandler Error from shard:
> http://host:port/solr =>
> org.apache.solr.client.solrj.SolrServerException: Timeout occured while
> waiting response from server at: http://host:port/solr/admin/cores
> at
> org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:408)
> org.apache.solr.client.solrj.SolrServerException: Timeout occured while
> waiting response from server at: http://host:port/solr/admin/cores
> at
> org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:408)
> ~[?:?]
> at
> org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:754)
> ~[?:?]
> at
> org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1290) ~[?:?]
> at
> org.apache.solr.handler.component.HttpShardHandler.request(HttpShardHandler.java:238)
> ~[?:?]
> at
> org.apache.solr.handler.component.HttpShardHandler.lambda$submit$0(HttpShardHandler.java:199)
> ~[?:?]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ~[?:1.8.0_232]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[?:1.8.0_232]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ~[?:1.8.0_232]
> at
> com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:181)
> ~[metrics-core-4.0.5.jar:4.0.5]
> at
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:210)
> ~[?:?]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> ~[?:1.8.0_232]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> ~[?:1.8.0_232]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_232]
> Caused by: java.util.concurrent.TimeoutException
> at
> org.eclipse.jetty.client.util.InputStreamResponseListener.get(InputStreamResponseListener.java:216)
> ~[?:?]
> at
> org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:399)
> ~[?:?]
> ... 12 more
>
> If I remove the async=bug, then it works
>
> In fact, the backup looks successful, but REQUESTSTATUS does not recognize
> it as such
>
> I notice that the 3:30am 11/4/19 Email to solr-user@lucene.apache.org
> mentions in Solr 8.3.0 Release Highlights "Fix for SPLITSHARD (async) with
> failures in underlying sub-operations can result in data loss"
>
> Did a fix to SPLITSHARD break BACKUP?
>
> Has anyone been successful running
> solr/admin/collections?action=BACKUP=requestname under Solr8.3?
>
> Thanks
>


-- 
Sincerely yours
Mikhail Khludnev

Re: Updates blocked in Tlog solr cloud?

[Erick Erickson]

How long are updates blocked and how did the tlog replica on the bad hardware 
go down?

Solr has to wait for an ack back from the tlog follower to be certain that the 
follower has all the documents in case it has to switch to that replica to 
become the leader. If the update to the follower times out, the leader will put 
it into a recovering state.

So I’d expect the collection to queue up indexing until the request to the 
follower on the bad hardware timed out, did you wait at least that long?

Best,
Erick

> On Nov 18, 2019, at 7:11 PM, Wei  wrote:
> 
> Hi,
> 
> I am puzzled by a problem in solr cloud with Tlog replicas and would
> appreciate your insights.  Our solr cloud has two shards and each shard
> have 5 tlog replicas. When one of the non-leader replica has hardware issue
> and become unreachable,  updates to the whole cloud stopped.  We are on
> solr 7.6 and use solrj client to send updates only to leaders.  To my
> understanding,  with Tlog replica type, the leader only forward update
> requests to replicas for transaction log update and each replica
> periodically pulls the segment from leader.  When one replica fails to
> respond,  why update requests to the cloud are blocked?  Does leader need
> to wait for response from each replica to inform client that update is
> successful?
> 
> Best,
> Wei

Possible data corruption in JavaBinCodec in Solr 8.3 during distributed update?

[Colvin Cowie]

Hello

Apologies for the lack of actual detail in this, we're still digging into
it ourselves. I will provide more detail, and maybe some logs, once I have
a better idea of what is actually happening.
But I thought I might as well ask if anyone knows of changes that were made
in the Solr 8.3 release that are likely to have caused an issue like this?

We were on Solr 8.1.1 for several months and moved to 8.2.0 for about 2
weeks before moving to 8.3.0 last week.
We didn't see this issue at all on the previous releases. Since moving to
8.3 we have had a consistent (but non-deterministic) set of failing tests,
on Windows and Linux.

The issue we are seeing as that during updates, the data we have sent is
*sometimes* corrupted, as though a buffer has been used incorrectly. For
example if the well formed data went was
*'fieldName':"this is a long string"*
The error we see from Solr might be that
unknown field * 'fieldNamis a long string" *

And variations of that kind of behaviour, were part of the data is missing
or corrupted. The data we are indexing does include fields which store
(escaped) serialized JSON strings - if that might have any bearing - but
the error isn't always on those fields.
For example, given a valid document that looks like this (I've replaced the
values by hand, so if the json is messed up here, that's not relevant:)
when returned with the json response writer:




*{"id": "abcd","testField": "blah","jsonField":
"{\"thing\":{\"abcd\":\"value\",\"xyz\":[\"abc\",\"def\",\"ghi\"],\"nnn\":\"xyz\"},\"stuff\":[{\"qqq\":\"rrr\"}],\"ttt\":0,\"mmm\":\"Some
string\",\"someBool\":true}"}*
We've had errors during indexing like:
*unknown field
'testField:"value","xyz":["abc","def","ghi"],"nnn":"xyz"},"stuff":[{"qqq":"rrr"}],"ttt":0,"mmm":"Some
string","someBool":true}���'*
(those � unprintable characters are part of it)

So far we've not been able to reproduce the problem on a collection with a
single shard, so it does seem like the problem is only happening internally
when updates are distributed to the other shards... But that's not been
totally verified.

We've also only encountered the problem on one of the collections we build
(the data within each collection is generally the same though. The ids are
slightly different - but still strings. The main difference is that this
problematic index is built using an Iterator to *solrj
org.apache.solr.client.solrj.SolrClient.add(String,
Iterator)* - the *SolrInputDocument*s are not being
reused in the client, I checked that -, while the other index is built by
streaming CSVs to Solr.)


We will look into it further, but if anyone has any ideas of what might
have changed in 8.3 from 8.1 / 8.2 that could cause this, that would be
helpful.

Cheers
Colvin

Re: Use of TLog

[Erick Erickson]

There’s also the Collections API MOVEREPLICA command that does this all as a 
single command.

> On Nov 19, 2019, at 5:21 AM, Sripra deep  wrote:
> 
> Got it, Thanks for helping me Emir.
> 
> On Tue, Nov 19, 2019 at 12:36 PM Emir Arnautović <
> emir.arnauto...@sematext.com> wrote:
> 
>> Hi,
>> Copying indices will work and it is a fine approach. An alternative would
>> be to join new node to a cluster, use add replica to copy cores to this new
>> node and then remove replicas from old nodes, if you want to move cores.
>> 
>> Regards,
>> Emir
>> --
>> Monitoring - Log Management - Alerting - Anomaly Detection
>> Solr & Elasticsearch Consulting Support Training - http://sematext.com/
>> 
>> 
>> 
>>> On 18 Nov 2019, at 13:22, Sripra deep 
>> wrote:
>>> 
>>> Hi Emir,
>>> 
>>> Thank you so much. Now I got it cleared with the TLOG purpose.
>>>  I am trying to copy an index of one solr cluster to use it to build
>>> other solr cluster. I am able to make that work but Is this design okay?
>> or
>>> any other approach I can try to get a new cluster spin up with the same
>>> data as in the old one.
>>> 
>>> Thanks,
>>> Sripradeep P
>>> 
>>> 
>>> On Mon, Nov 18, 2019 at 2:12 PM Emir Arnautović <
>>> emir.arnauto...@sematext.com> wrote:
>>> 
 Hi Sripradeep,
 Simplified: TLog files are used to replay index updates from the last
 successful hard commit in case of some Solr crashes. It is used on the
>> next
 Solr startup. It does not contain all updates, otherwise, it would
 duplicate the index size.
 If you start from these premises, you will understand why it is not
>> copied
 when adding replicas and why it is not needed and why you cannot use
>> TLog
 to spin up a new cluster.
 
 HTH,
 Emir
 --
 Monitoring - Log Management - Alerting - Anomaly Detection
 Solr & Elasticsearch Consulting Support Training - http://sematext.com/
 
 
 
> On 18 Nov 2019, at 06:35, Sripra deep 
 wrote:
> 
> Hi Guys,
> 
> I observed a scenario with the tlog creation and usage and couldn't
>> find
> any usage for the tlog.
> 
> Solr version: 7.1.0
> Number of shards = 3
> Number of replica = 1
> I indexed the about 10k docs into the collection.
> 
> Scenario 1:
> Using add replica collection API, I created one more replica (tried
>> with
> both nrt and tlog) neither of the replicas doesn't pull the tlog files.
> Only the index files are pulled from master.
> * If the tlog is not present in a replica then during ungraceful
 shutdown
> of the solr server how the replicas will regain the index without tlog
> files.
> * To verify the above scenario, I killed the newly added replica server
> with kill -9  command and started back
> also stopped the leader node.
> 
> Questions:
> 1) TLog files are not used even in the case of ungraceful shutdown,
 where
> else it will be used?
> 2) Tlog files doesn't get copied to the newly added replica so adding a
> new replica to the already created collection with data/index is not
> advisable?
> 3) Is there a way to make the newly added slave node to replicate the
> tlog file as it does for the data/index files from leader?
> 4) Is it possible to use the Tlog files /index files from an existing
> solr server to spin up a new solr cluster?
> 
> 
> It would be much helpful for me to understand the core working of Solr
> server.
> 
> Thanks,
> Sripradeep P
 
 
>> 
>> 

RE: How to implement NOTIN operator with Solr

[Raboah, Avi]

It's working!!! thanks a lot :)

-Original Message-
From: Emir Arnautović [mailto:emir.arnauto...@sematext.com]
Sent: Tuesday, November 19, 2019 2:54 PM
To: solr-user@lucene.apache.org
Subject: Re: How to implement NOTIN operator with Solr

Right - didn’t read all your examples. In that case you can use span queries. 
In this case complexphrase query parser should do the trick:
{!complexphrase df=text}”credit -card”

Regards,
Emir
--
Monitoring - Log Management - Alerting - Anomaly Detection Solr & Elasticsearch 
Consulting Support Training - http://sematext.com/



> On 19 Nov 2019, at 11:08, Raboah, Avi  wrote:
>
> In that case I got only doc1
>
> -Original Message-
> From: Emir Arnautović [mailto:emir.arnauto...@sematext.com]
> Sent: Tuesday, November 19, 2019 11:51 AM
> To: solr-user@lucene.apache.org
> Subject: Re: How to implement NOTIN operator with Solr
>
> Hi Avi,
> There are span queries, but in this case you don’t need it. It is
> enough to simply filter out documents that are with “credit card”.
> Your query can be something like
> +text:credit -text:”credit card”
> If you prefer using boolean operators, you can write it as:
> text:credit AND NOT text: “credit card”
>
> HTH,
> Emir
> --
> Monitoring - Log Management - Alerting - Anomaly Detection Solr &
> Elasticsearch Consulting Support Training - http://sematext.com/
>
>
>
>> On 19 Nov 2019, at 10:30, Raboah, Avi  wrote:
>>
>> I am trying to find the documents which hit this example:
>>
>> q=text:"credit" NOTIN "credit card"
>>
>> for that query I want to get all the documents which contain the term 
>> "credit" but not as part of the phrase "credit card".
>>
>> so:
>>
>> 1. I don't want to get the documents which include just "credit card".
>>
>> 2. I want to get the documents which include just "credit".
>>
>> 3. I want to get the documents which include "credit" but not as part of 
>> credit card.
>>
>>
>>
>> for example:
>>
>> doc1 text: "I want to buy with my credit in my card"
>>
>> doc2 text: "I want to buy with my credit in my credit card"
>>
>> doc3 text: "I want to buy with my credit card"
>>
>> The documents should be returned:
>>
>> doc1, doc2
>>
>> I can't find nothing about NOTIN operator implementation in SOLR docs.
>>
>>
>>
>> This electronic message may contain proprietary and confidential information 
>> of Verint Systems Inc., its affiliates and/or subsidiaries. The information 
>> is intended to be for the use of the individual(s) or entity(ies) named 
>> above. If you are not the intended recipient (or authorized to receive this 
>> e-mail for the intended recipient), you may not use, copy, disclose or 
>> distribute to anyone this message or any information contained in this 
>> message. If you have received this electronic message in error, please 
>> notify us by replying to this e-mail.
>
>
>
> This electronic message may contain proprietary and confidential information 
> of Verint Systems Inc., its affiliates and/or subsidiaries. The information 
> is intended to be for the use of the individual(s) or entity(ies) named 
> above. If you are not the intended recipient (or authorized to receive this 
> e-mail for the intended recipient), you may not use, copy, disclose or 
> distribute to anyone this message or any information contained in this 
> message. If you have received this electronic message in error, please notify 
> us by replying to this e-mail.



This electronic message may contain proprietary and confidential information of 
Verint Systems Inc., its affiliates and/or subsidiaries. The information is 
intended to be for the use of the individual(s) or entity(ies) named above. If 
you are not the intended recipient (or authorized to receive this e-mail for 
the intended recipient), you may not use, copy, disclose or distribute to 
anyone this message or any information contained in this message. If you have 
received this electronic message in error, please notify us by replying to this 
e-mail.

Re: Need to recreate collection when version-2 folder deleted in zookeeper

[Erick Erickson]

Almost certainly. You can recreate all the “state.json” znodes “by hand”, but 
that’ll be very, very difficult to get right.

It’s possible you have  ZK snapshot laying around you can restore, you’ll have 
to look.

Best,
Erick

> On Nov 19, 2019, at 6:33 AM, vishal patel  
> wrote:
> 
> I have created 2 shards of Solr 8.3.0. After I have created 10 collections 
> and also re-indexed data.
> 
> Some fields are changed in one collection. I deleted a version-2 folder from 
> zoo_data and up config that collection.
> 
> Is it necessary to create all collections again? Also indexing data again?
> 
> Regards,
> Vishal

Re: Zk upconfig command is appending local directory to default confdir

[Erick Erickson]

And to make it more complicated, if you don’t have a version that has the 
CONFIGSET API, you can use bin/solr zk upconfig/downconfig…

Best,
Erick

> On Nov 19, 2019, at 8:18 AM, Jörn Franke  wrote:
> 
> You can call it via curl and it is no more complicated then zk- in the long 
> run you will be happier 
> 
>> Am 19.11.2019 um 13:27 schrieb Michael Becker :
>> 
>> Jörn,
>> 
>> We're using Solr to index a website. Basically, we set up Solr and Zookeeper 
>> once and let it run. The API is a bit too much overhead for something we 
>> just set once.
>> 
>> Mike
>> 
>> -Original Message-
>> From: Jörn Franke 
>> Sent: Tuesday, November 19, 2019 2:54 AM
>> To: solr-user@lucene.apache.org
>> Subject: Re: Zk upconfig command is appending local directory to default 
>> confdir
>> 
>> I would use the config set API - it is more clean for production deployments 
>> and you do not have to deal with the zkCli script:
>> 
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__lucene.apache.org_solr_guide_7-5F4_configsets-2Dapi.html=DwIFaQ=mV61MqUbMWY-LlirxZJvJg=8W5Kk7fJg_C2taHlYyXZLinEFeEtcUcFddYrS5aUhiE=czV2n50m-v9_w3N63i-jYUsBfnCtCKeE3wNXq3KVerU=8GQix9ABHkY0tm7kPKL_GEXCi-G4PKRjJjN-FqQcRVI=
>>  
>> 
>>> Am 18.11.2019 um 15:48 schrieb Michael Becker :
>>> 
>>> I’ve run into an issue when attempting to configure Zookeeper. When 
>>> running the zk upconfig -d command specifying a local directory where the 
>>> solrconfig.xml files are located, I get the following error:
>>> “Could not complete upconfig operation for reason: Could not find 
>>> solrconfig.xml at /opt/solr-6.5.1/server/solr/configsets/solrconfig.xml, 
>>> /opt/solr-6.5.1/server/solr/configsets/conf/solrconfig.xml or 
>>> /opt/solr-6.5.1/server/solr/configsets/  
>>> /solrconfig.xml”
>>> 
>>> I’m trying to determine why the solr zk upconfig command is appending 
>>> my local directory to the default confdir, rather than looking for the XML 
>>> files in that directory, I have two other environments with Solr where this 
>>> does not occur. It’s just this one environment that is having this issue.
>>> I am using Solr version 6.5.1.
>>> Any suggestions on how to troubleshoot this would be appreciated.
>>> 
>>> Mike

Re: Explicit documentation regarding all JMX metrics?

[Erick Erickson]

Here’s something that might be useful:
https://support.lucidworks.com/s/article/What-should-you-monitor

What version of Solr? For Solr 6.4+, you may want to use the /admin/metrics 
endpoint, see the reference guide for metrics. WARNING: there are a ton of 
metrics. The ref guide has information on now to trim down what you’re looking 
at.

Best,
Erick

> On Nov 19, 2019, at 8:28 AM, Koen De Groote  
> wrote:
> 
> Greetings,
> 
> I've enabled JMX metrics on an install of mine and am going through the
> available beans.
> 
> I've google for an explanation of what the beans all mean, but I'm coming
> up empty.
> 
> For instance, these:
> 
> :category=ADMIN,...,name=clientErrors,scope=/admin/file
> :category=ADMIN,...,name=clientErrors,scope=/admin/logging
> :category=ADMIN,...,name=clientErrors,scope=/admin/luke
> :category=ADMIN,...,name=clientErrors,scope=/admin/mbeans
> :category=ADMIN,...,name=clientErrors,scope=/admin/ping
> :category=ADMIN,...,name=clientErrors,scope=/admin/plugins
> :category=ADMIN,...,name=clientErrors,scope=/admin/properties
> :category=ADMIN,...,name=clientErrors,scope=/admin/segments
> :category=ADMIN,...,name=clientErrors,scope=/admin/system
> :category=ADMIN,...,name=clientErrors,scope=/admin/threads
> :category=ADMIN,...,name=clientErrors,scope=/config
> :category=ADMIN,...,name=clientErrors,scope=/schema
> 
> What am I looking at here?
> 
> I can't find a single page explaining what the significance of these
> metrics is.
> 
> Nor can I find a page that basically says "You'll want to monitor X, Y and
> Z get a good view of your cluster".
> 
> Can anyone provide an example?
> 
> Kind regards,
> Koen De Groote

Explicit documentation regarding all JMX metrics?

[Koen De Groote]

Greetings,

I've enabled JMX metrics on an install of mine and am going through the
available beans.

I've google for an explanation of what the beans all mean, but I'm coming
up empty.

For instance, these:

:category=ADMIN,...,name=clientErrors,scope=/admin/file
:category=ADMIN,...,name=clientErrors,scope=/admin/logging
:category=ADMIN,...,name=clientErrors,scope=/admin/luke
:category=ADMIN,...,name=clientErrors,scope=/admin/mbeans
:category=ADMIN,...,name=clientErrors,scope=/admin/ping
:category=ADMIN,...,name=clientErrors,scope=/admin/plugins
:category=ADMIN,...,name=clientErrors,scope=/admin/properties
:category=ADMIN,...,name=clientErrors,scope=/admin/segments
:category=ADMIN,...,name=clientErrors,scope=/admin/system
:category=ADMIN,...,name=clientErrors,scope=/admin/threads
:category=ADMIN,...,name=clientErrors,scope=/config
:category=ADMIN,...,name=clientErrors,scope=/schema

What am I looking at here?

I can't find a single page explaining what the significance of these
metrics is.

Nor can I find a page that basically says "You'll want to monitor X, Y and
Z get a good view of your cluster".

Can anyone provide an example?

Kind regards,
Koen De Groote

Re: Zk upconfig command is appending local directory to default confdir

[Jörn Franke]

You can call it via curl and it is no more complicated then zk- in the long run 
you will be happier 

> Am 19.11.2019 um 13:27 schrieb Michael Becker :
> 
> Jörn,
> 
> We're using Solr to index a website. Basically, we set up Solr and Zookeeper 
> once and let it run. The API is a bit too much overhead for something we just 
> set once.
> 
> Mike
> 
> -Original Message-
> From: Jörn Franke 
> Sent: Tuesday, November 19, 2019 2:54 AM
> To: solr-user@lucene.apache.org
> Subject: Re: Zk upconfig command is appending local directory to default 
> confdir
> 
> I would use the config set API - it is more clean for production deployments 
> and you do not have to deal with the zkCli script:
> 
> https://urldefense.proofpoint.com/v2/url?u=https-3A__lucene.apache.org_solr_guide_7-5F4_configsets-2Dapi.html=DwIFaQ=mV61MqUbMWY-LlirxZJvJg=8W5Kk7fJg_C2taHlYyXZLinEFeEtcUcFddYrS5aUhiE=czV2n50m-v9_w3N63i-jYUsBfnCtCKeE3wNXq3KVerU=8GQix9ABHkY0tm7kPKL_GEXCi-G4PKRjJjN-FqQcRVI=
>  
> 
>> Am 18.11.2019 um 15:48 schrieb Michael Becker :
>> 
>> I’ve run into an issue when attempting to configure Zookeeper. When running 
>> the zk upconfig -d command specifying a local directory where the 
>> solrconfig.xml files are located, I get the following error:
>> “Could not complete upconfig operation for reason: Could not find 
>> solrconfig.xml at /opt/solr-6.5.1/server/solr/configsets/solrconfig.xml, 
>> /opt/solr-6.5.1/server/solr/configsets/conf/solrconfig.xml or 
>> /opt/solr-6.5.1/server/solr/configsets/  
>> /solrconfig.xml”
>> 
>> I’m trying to determine why the solr zk upconfig command is appending 
>> my local directory to the default confdir, rather than looking for the XML 
>> files in that directory, I have two other environments with Solr where this 
>> does not occur. It’s just this one environment that is having this issue.
>> I am using Solr version 6.5.1.
>> Any suggestions on how to troubleshoot this would be appreciated.
>> 
>> Mike

Re: How to implement NOTIN operator with Solr

[Emir Arnautović]

Right - didn’t read all your examples. In that case you can use span queries. 
In this case complexphrase query parser should do the trick:
{!complexphrase df=text}”credit -card”

Regards,
Emir
--
Monitoring - Log Management - Alerting - Anomaly Detection
Solr & Elasticsearch Consulting Support Training - http://sematext.com/



> On 19 Nov 2019, at 11:08, Raboah, Avi  wrote:
> 
> In that case I got only doc1
> 
> -Original Message-
> From: Emir Arnautović [mailto:emir.arnauto...@sematext.com]
> Sent: Tuesday, November 19, 2019 11:51 AM
> To: solr-user@lucene.apache.org
> Subject: Re: How to implement NOTIN operator with Solr
> 
> Hi Avi,
> There are span queries, but in this case you don’t need it. It is enough to 
> simply filter out documents that are with “credit card”. Your query can be 
> something like
> +text:credit -text:”credit card”
> If you prefer using boolean operators, you can write it as:
> text:credit AND NOT text: “credit card”
> 
> HTH,
> Emir
> --
> Monitoring - Log Management - Alerting - Anomaly Detection Solr & 
> Elasticsearch Consulting Support Training - http://sematext.com/
> 
> 
> 
>> On 19 Nov 2019, at 10:30, Raboah, Avi  wrote:
>> 
>> I am trying to find the documents which hit this example:
>> 
>> q=text:"credit" NOTIN "credit card"
>> 
>> for that query I want to get all the documents which contain the term 
>> "credit" but not as part of the phrase "credit card".
>> 
>> so:
>> 
>> 1. I don't want to get the documents which include just "credit card".
>> 
>> 2. I want to get the documents which include just "credit".
>> 
>> 3. I want to get the documents which include "credit" but not as part of 
>> credit card.
>> 
>> 
>> 
>> for example:
>> 
>> doc1 text: "I want to buy with my credit in my card"
>> 
>> doc2 text: "I want to buy with my credit in my credit card"
>> 
>> doc3 text: "I want to buy with my credit card"
>> 
>> The documents should be returned:
>> 
>> doc1, doc2
>> 
>> I can't find nothing about NOTIN operator implementation in SOLR docs.
>> 
>> 
>> 
>> This electronic message may contain proprietary and confidential information 
>> of Verint Systems Inc., its affiliates and/or subsidiaries. The information 
>> is intended to be for the use of the individual(s) or entity(ies) named 
>> above. If you are not the intended recipient (or authorized to receive this 
>> e-mail for the intended recipient), you may not use, copy, disclose or 
>> distribute to anyone this message or any information contained in this 
>> message. If you have received this electronic message in error, please 
>> notify us by replying to this e-mail.
> 
> 
> 
> This electronic message may contain proprietary and confidential information 
> of Verint Systems Inc., its affiliates and/or subsidiaries. The information 
> is intended to be for the use of the individual(s) or entity(ies) named 
> above. If you are not the intended recipient (or authorized to receive this 
> e-mail for the intended recipient), you may not use, copy, disclose or 
> distribute to anyone this message or any information contained in this 
> message. If you have received this electronic message in error, please notify 
> us by replying to this e-mail.

RE: Zk upconfig command is appending local directory to default confdir

[Michael Becker]

Jörn,

We're using Solr to index a website. Basically, we set up Solr and Zookeeper 
once and let it run. The API is a bit too much overhead for something we just 
set once.

Mike

-Original Message-
From: Jörn Franke  
Sent: Tuesday, November 19, 2019 2:54 AM
To: solr-user@lucene.apache.org
Subject: Re: Zk upconfig command is appending local directory to default confdir

I would use the config set API - it is more clean for production deployments 
and you do not have to deal with the zkCli script:

https://urldefense.proofpoint.com/v2/url?u=https-3A__lucene.apache.org_solr_guide_7-5F4_configsets-2Dapi.html=DwIFaQ=mV61MqUbMWY-LlirxZJvJg=8W5Kk7fJg_C2taHlYyXZLinEFeEtcUcFddYrS5aUhiE=czV2n50m-v9_w3N63i-jYUsBfnCtCKeE3wNXq3KVerU=8GQix9ABHkY0tm7kPKL_GEXCi-G4PKRjJjN-FqQcRVI=
 

> Am 18.11.2019 um 15:48 schrieb Michael Becker :
> 
> I’ve run into an issue when attempting to configure Zookeeper. When running 
> the zk upconfig -d command specifying a local directory where the 
> solrconfig.xml files are located, I get the following error:
> “Could not complete upconfig operation for reason: Could not find 
> solrconfig.xml at /opt/solr-6.5.1/server/solr/configsets/solrconfig.xml, 
> /opt/solr-6.5.1/server/solr/configsets/conf/solrconfig.xml or 
> /opt/solr-6.5.1/server/solr/configsets/  
> /solrconfig.xml”
> 
> I’m trying to determine why the solr zk upconfig command is appending 
> my local directory to the default confdir, rather than looking for the XML 
> files in that directory, I have two other environments with Solr where this 
> does not occur. It’s just this one environment that is having this issue.
> I am using Solr version 6.5.1.
> Any suggestions on how to troubleshoot this would be appreciated.
> 
> Mike

RE: Zk upconfig command is appending local directory to default confdir

[Michael Becker]

Domimique:

That's what is odd. There IS a file named solrconfig.xml within that directory. 
The naming convention for the directory is the same in all three environments 
where we have Solr running. The command works fine in the other two 
environments. But for some reason, in this environment, Solr is reading the 
directory path as a subdirectory of /opt/solr-6.5.1/server/solr/configsets/, 
rather than as an absolute path.
So, if the full path specified in -d is 
/zkCongifgset/sitecore_index/solrconfig.xml, with /zkConfigset being the root 
directory. rather than going to that directory, Solr is looking for it in 
/opt/solr-6.5.1/server/solr/configsets/zkCongifgset/sitecore_index/. I'm not 
clear on why Solr is doing that.

Mike


-Original Message-
From: Dominique Bejean  
Sent: Tuesday, November 19, 2019 2:45 AM
To: solr-user@lucene.apache.org
Subject: Re: Zk upconfig command is appending local directory to default confdir

Hi Michael,

It seems Sorl really don't find any solrconfig.xml file or a 
conf/solrconfig.xml file in the local path you specified. The last try is to 
look in "/opt/solr-6.5.1/server/solr/configsets/", but obviously it doesn't work has you didn't specify a
confiset name.

The code is here -
https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_lucene-2Dsolr_blob_8cde1277ec7151bd6ab62950ac93cbdd6ff04d9f_solr_solrj_src_java_org_apache_solr_common_cloud_ZkConfigManager.java-23L181=DwIFaQ=mV61MqUbMWY-LlirxZJvJg=8W5Kk7fJg_C2taHlYyXZLinEFeEtcUcFddYrS5aUhiE=XQfOcrEywCrmCsBoBMPAznCdw_ZV28m7HTnR5rf1q50=bik3n9dWgRpi2fcr0t60J2NV7UzzXdg5T4VpPDt_-yE=
 


Any error in read access rights to your config directory ?

Regards

Dominique



Le lun. 18 nov. 2019 à 15:48, Michael Becker  a écrit :

> I’ve run into an issue when attempting to configure Zookeeper. When 
> running the zk upconfig -d command specifying a local directory where 
> the solrconfig.xml files are located, I get the following error:
> “Could not complete upconfig operation for reason: Could not find 
> solrconfig.xml at 
> /opt/solr-6.5.1/server/solr/configsets/solrconfig.xml,
> /opt/solr-6.5.1/server/solr/configsets/conf/solrconfig.xml or 
> /opt/solr-6.5.1/server/solr/configsets/  
> /solrconfig.xml”
>
> I’m trying to determine why the solr zk upconfig command is appending 
> my local directory to the default confdir, rather than looking for the 
> XML files in that directory, I have two other environments with Solr 
> where this does not occur. It’s just this one environment that is 
> having this issue.
> I am using Solr version 6.5.1.
> Any suggestions on how to troubleshoot this would be appreciated.
>
> Mike
>

Re: Solr 8.3.0

[vishal patel]

My autoAddReplicas is false. Is it necessary to change? Also, the replication 
factor is 1. Need to update?

Regards,
Vishal

Sent from Outlook

From: Erick Erickson 
Sent: Monday, November 18, 2019 9:47 PM
To: solr-user@lucene.apache.org 
Subject: Re: Solr 8.3.0

The Collections API ADDREPLICA command.

Best,
Erick

> On Nov 18, 2019, at 12:48 AM, vishal patel  
> wrote:
>
>
> I have created 2 shards of Solr 8.3.0. We have created 27 collections using 
> the below
> http://191.162.100.148:7971/solr/admin/collections?_=1573813004271=CREATE=false=actionscomments=1=actionscomments=2=1=compositeId=json
>
>
> After the re-indexing Data, I want to add a replica of each shard. How can I 
> add a replica without re-creating collection and re-indexing?
> Can I add one more shard dynamically without re-creating collections and 
> re-indexing?

Need to recreate collection when version-2 folder deleted in zookeeper

[vishal patel]

I have created 2 shards of Solr 8.3.0. After I have created 10 collections and 
also re-indexed data.

Some fields are changed in one collection. I deleted a version-2 folder from 
zoo_data and up config that collection.

Is it necessary to create all collections again? Also indexing data again?

Regards,
Vishal

Re: How to implement NOTIN operator with Solr

[Alexandre Rafalovitch]

I think the main question here is the compound word "credit card"
always the same? If yes, you can preprocess it during indexing to
something unique and discard (see Vincenzo's reply). You could even
copyfield and process the copy to only leave standalone word "credit"
in it, so it basically serves as a boolean presence marker.

But if it can change for every search, you have to do it during query
only. I suspect span queries can detect something like this, but don't
have a reference example. I suspect it would be either with:
*) Surround Query Parser:
https://lucene.apache.org/solr/guide/8_3/other-parsers.html#surround-query-parser
or directly with
*) XML Query Parser:
https://lucene.apache.org/solr/guide/8_3/other-parsers.html#xml-query-parser

Once you figured the syntax out, you should be able to substitute
values with variables and perhaps even push the long syntax into a
separate Query Handler, so you just pass "yes word" and "no phrase" to
Solr and have it construct longer query.

Please do let us know when you figure it out. I think other people
were interested in the similar problem before.

Regards,
   Alex.

On Tue, 19 Nov 2019 at 05:08, Raboah, Avi  wrote:
>
> In that case I got only doc1
>
> -Original Message-
> From: Emir Arnautović [mailto:emir.arnauto...@sematext.com]
> Sent: Tuesday, November 19, 2019 11:51 AM
> To: solr-user@lucene.apache.org
> Subject: Re: How to implement NOTIN operator with Solr
>
> Hi Avi,
> There are span queries, but in this case you don’t need it. It is enough to 
> simply filter out documents that are with “credit card”. Your query can be 
> something like
> +text:credit -text:”credit card”
> If you prefer using boolean operators, you can write it as:
> text:credit AND NOT text: “credit card”
>
> HTH,
> Emir
> --
> Monitoring - Log Management - Alerting - Anomaly Detection Solr & 
> Elasticsearch Consulting Support Training - http://sematext.com/
>
>
>
> > On 19 Nov 2019, at 10:30, Raboah, Avi  wrote:
> >
> > I am trying to find the documents which hit this example:
> >
> > q=text:"credit" NOTIN "credit card"
> >
> > for that query I want to get all the documents which contain the term 
> > "credit" but not as part of the phrase "credit card".
> >
> > so:
> >
> > 1. I don't want to get the documents which include just "credit card".
> >
> > 2. I want to get the documents which include just "credit".
> >
> > 3. I want to get the documents which include "credit" but not as part 
> > of credit card.
> >
> >
> >
> > for example:
> >
> > doc1 text: "I want to buy with my credit in my card"
> >
> > doc2 text: "I want to buy with my credit in my credit card"
> >
> > doc3 text: "I want to buy with my credit card"
> >
> > The documents should be returned:
> >
> > doc1, doc2
> >
> > I can't find nothing about NOTIN operator implementation in SOLR docs.
> >
> >
> >
> > This electronic message may contain proprietary and confidential 
> > information of Verint Systems Inc., its affiliates and/or subsidiaries. The 
> > information is intended to be for the use of the individual(s) or 
> > entity(ies) named above. If you are not the intended recipient (or 
> > authorized to receive this e-mail for the intended recipient), you may not 
> > use, copy, disclose or distribute to anyone this message or any information 
> > contained in this message. If you have received this electronic message in 
> > error, please notify us by replying to this e-mail.
>
>
>
> This electronic message may contain proprietary and confidential information 
> of Verint Systems Inc., its affiliates and/or subsidiaries. The information 
> is intended to be for the use of the individual(s) or entity(ies) named 
> above. If you are not the intended recipient (or authorized to receive this 
> e-mail for the intended recipient), you may not use, copy, disclose or 
> distribute to anyone this message or any information contained in this 
> message. If you have received this electronic message in error, please notify 
> us by replying to this e-mail.

Re: Use of TLog

[Sripra deep]

Got it, Thanks for helping me Emir.

On Tue, Nov 19, 2019 at 12:36 PM Emir Arnautović <
emir.arnauto...@sematext.com> wrote:

> Hi,
> Copying indices will work and it is a fine approach. An alternative would
> be to join new node to a cluster, use add replica to copy cores to this new
> node and then remove replicas from old nodes, if you want to move cores.
>
> Regards,
> Emir
> --
> Monitoring - Log Management - Alerting - Anomaly Detection
> Solr & Elasticsearch Consulting Support Training - http://sematext.com/
>
>
>
> > On 18 Nov 2019, at 13:22, Sripra deep 
> wrote:
> >
> > Hi Emir,
> >
> >  Thank you so much. Now I got it cleared with the TLOG purpose.
> >   I am trying to copy an index of one solr cluster to use it to build
> > other solr cluster. I am able to make that work but Is this design okay?
> or
> > any other approach I can try to get a new cluster spin up with the same
> > data as in the old one.
> >
> > Thanks,
> > Sripradeep P
> >
> >
> > On Mon, Nov 18, 2019 at 2:12 PM Emir Arnautović <
> > emir.arnauto...@sematext.com> wrote:
> >
> >> Hi Sripradeep,
> >> Simplified: TLog files are used to replay index updates from the last
> >> successful hard commit in case of some Solr crashes. It is used on the
> next
> >> Solr startup. It does not contain all updates, otherwise, it would
> >> duplicate the index size.
> >> If you start from these premises, you will understand why it is not
> copied
> >> when adding replicas and why it is not needed and why you cannot use
> TLog
> >> to spin up a new cluster.
> >>
> >> HTH,
> >> Emir
> >> --
> >> Monitoring - Log Management - Alerting - Anomaly Detection
> >> Solr & Elasticsearch Consulting Support Training - http://sematext.com/
> >>
> >>
> >>
> >>> On 18 Nov 2019, at 06:35, Sripra deep 
> >> wrote:
> >>>
> >>> Hi Guys,
> >>>
> >>> I observed a scenario with the tlog creation and usage and couldn't
> find
> >>> any usage for the tlog.
> >>>
> >>> Solr version: 7.1.0
> >>> Number of shards = 3
> >>> Number of replica = 1
> >>> I indexed the about 10k docs into the collection.
> >>>
> >>> Scenario 1:
> >>> Using add replica collection API, I created one more replica (tried
> with
> >>> both nrt and tlog) neither of the replicas doesn't pull the tlog files.
> >>> Only the index files are pulled from master.
> >>> * If the tlog is not present in a replica then during ungraceful
> >> shutdown
> >>> of the solr server how the replicas will regain the index without tlog
> >>> files.
> >>> * To verify the above scenario, I killed the newly added replica server
> >>> with kill -9  command and started back
> >>> also stopped the leader node.
> >>>
> >>> Questions:
> >>> 1) TLog files are not used even in the case of ungraceful shutdown,
> >> where
> >>> else it will be used?
> >>> 2) Tlog files doesn't get copied to the newly added replica so adding a
> >>> new replica to the already created collection with data/index is not
> >>> advisable?
> >>> 3) Is there a way to make the newly added slave node to replicate the
> >>> tlog file as it does for the data/index files from leader?
> >>> 4) Is it possible to use the Tlog files /index files from an existing
> >>> solr server to spin up a new solr cluster?
> >>>
> >>>
> >>> It would be much helpful for me to understand the core working of Solr
> >>> server.
> >>>
> >>> Thanks,
> >>> Sripradeep P
> >>
> >>
>
>

RE: How to implement NOTIN operator with Solr

[Raboah, Avi]

In that case I got only doc1

-Original Message-
From: Emir Arnautović [mailto:emir.arnauto...@sematext.com]
Sent: Tuesday, November 19, 2019 11:51 AM
To: solr-user@lucene.apache.org
Subject: Re: How to implement NOTIN operator with Solr

Hi Avi,
There are span queries, but in this case you don’t need it. It is enough to 
simply filter out documents that are with “credit card”. Your query can be 
something like
+text:credit -text:”credit card”
If you prefer using boolean operators, you can write it as:
text:credit AND NOT text: “credit card”

HTH,
Emir
--
Monitoring - Log Management - Alerting - Anomaly Detection Solr & Elasticsearch 
Consulting Support Training - http://sematext.com/



> On 19 Nov 2019, at 10:30, Raboah, Avi  wrote:
>
> I am trying to find the documents which hit this example:
>
> q=text:"credit" NOTIN "credit card"
>
> for that query I want to get all the documents which contain the term 
> "credit" but not as part of the phrase "credit card".
>
> so:
>
> 1. I don't want to get the documents which include just "credit card".
>
> 2. I want to get the documents which include just "credit".
>
> 3. I want to get the documents which include "credit" but not as part of 
> credit card.
>
>
>
> for example:
>
> doc1 text: "I want to buy with my credit in my card"
>
> doc2 text: "I want to buy with my credit in my credit card"
>
> doc3 text: "I want to buy with my credit card"
>
> The documents should be returned:
>
> doc1, doc2
>
> I can't find nothing about NOTIN operator implementation in SOLR docs.
>
>
>
> This electronic message may contain proprietary and confidential information 
> of Verint Systems Inc., its affiliates and/or subsidiaries. The information 
> is intended to be for the use of the individual(s) or entity(ies) named 
> above. If you are not the intended recipient (or authorized to receive this 
> e-mail for the intended recipient), you may not use, copy, disclose or 
> distribute to anyone this message or any information contained in this 
> message. If you have received this electronic message in error, please notify 
> us by replying to this e-mail.



This electronic message may contain proprietary and confidential information of 
Verint Systems Inc., its affiliates and/or subsidiaries. The information is 
intended to be for the use of the individual(s) or entity(ies) named above. If 
you are not the intended recipient (or authorized to receive this e-mail for 
the intended recipient), you may not use, copy, disclose or distribute to 
anyone this message or any information contained in this message. If you have 
received this electronic message in error, please notify us by replying to this 
e-mail.

RE: Active directory integration in Solr

[Kommu, Vinodh K.]

Thanks Charlie.

We are already using Basic authentication in our existing clusters, however 
it's getting difficult to maintain number of users as we are getting too many 
requests for readonly access from support teams. So we desperately looking for 
active directory solution. Just wondering if someone might have same 
requirement need.


Regards,
Vinodh 

-Original Message-
From: Charlie Hull  
Sent: Tuesday, November 19, 2019 2:55 PM
To: solr-user@lucene.apache.org
Subject: Re: Active directory integration in Solr

ATTENTION! This email originated outside of DTCC; exercise caution.

Not out of the box, there are a few authentication plugins bundled but not for 
AD
https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Flucene.apache.org%2Fsolr%2Fguide%2F7_2%2Fauthentication-and-authorization-plugins.htmldata=02%7C01%7Cvkommu%40dtcc.com%7C2e17e1feef78432502e008d76cd26635%7C0465519d7f554d47998b55e2a86f04a8%7C0%7C0%7C637097523245309858sdata=fkahJ62aWFYh7QxcyFQbJV9u8OsTYSWp6pv0MNdzjps%3Dreserved=0
- there's also some useful stuff in Apache ManifoldCF
https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.francelabs.com%2Fblog%2Ftutorial-on-authorizations-for-manifold-cf-and-solr%2Fdata=02%7C01%7Cvkommu%40dtcc.com%7C2e17e1feef78432502e008d76cd26635%7C0465519d7f554d47998b55e2a86f04a8%7C0%7C0%7C637097523245319858sdata=iYiKRDJKYBZaxUd%2F%2BIddFBwxB2RhSqih2KZc26aZlRU%3Dreserved=0


Best

Charlie

On 18/11/2019 15:08, Kommu, Vinodh K. wrote:
> Hi,
>
> Does anyone know that Solr has any out of the box capability to integrate 
> Active directory (using LDAP) when security is enabled? Instead of creating 
> users in security.json file, planning to use users who already exists in 
> active directory so they can use their individual credentials rather than 
> defining in Solr. Did anyone came across similar requirement? If so was there 
> any working solution?
>
>
> Thanks,
> Vinodh
>
> DTCC DISCLAIMER: This email and any files transmitted with it are 
> confidential and intended solely for the use of the individual or entity to 
> whom they are addressed. If you have received this email in error, please 
> notify us immediately and delete the email and any attachments from your 
> system. The recipient should check this email and any attachments for the 
> presence of viruses. The company accepts no liability for any damage caused 
> by any virus transmitted by this email.
>

--
Charlie Hull
Flax - Open Source Enterprise Search

tel/fax: +44 (0)8700 118334
mobile:  +44 (0)7767 825828
web: 
https://nam02.safelinks.protection.outlook.com/?url=www.flax.co.ukdata=02%7C01%7Cvkommu%40dtcc.com%7C2e17e1feef78432502e008d76cd26635%7C0465519d7f554d47998b55e2a86f04a8%7C0%7C0%7C637097523245319858sdata=YNGIg%2FVgL2w82i3JWsBkBTJeefHMjSxbjLaQyOdJVt0%3Dreserved=0

DTCC DISCLAIMER: This email and any files transmitted with it are confidential 
and intended solely for the use of the individual or entity to whom they are 
addressed. If you have received this email in error, please notify us 
immediately and delete the email and any attachments from your system. The 
recipient should check this email and any attachments for the presence of 
viruses. The company accepts no liability for any damage caused by any virus 
transmitted by this email.

NPE For Atomic Update on Solr 8.3

[sayan.das]

On upgrading to solr 8.3 also I am facing the same issue as SOLR-13523. 
I have added these on my schema





Document present in solr:
[{
   "id": "parent1",
   "isInStock": 1,
   "parent": true,
   "_childDocuments_": [
 {
 "id": "child 1",
 "isInStock": 0,
 "parent": false
 }
   ]
 }]

On doing atomic update I am getting this

 






--
Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: How to implement NOTIN operator with Solr

[Emir Arnautović]

Hi Avi,
There are span queries, but in this case you don’t need it. It is enough to 
simply filter out documents that are with “credit card”. Your query can be 
something like
+text:credit -text:”credit card”
If you prefer using boolean operators, you can write it as:
text:credit AND NOT text: “credit card”

HTH,
Emir
--
Monitoring - Log Management - Alerting - Anomaly Detection
Solr & Elasticsearch Consulting Support Training - http://sematext.com/



> On 19 Nov 2019, at 10:30, Raboah, Avi  wrote:
> 
> I am trying to find the documents which hit this example:
> 
> q=text:"credit" NOTIN "credit card"
> 
> for that query I want to get all the documents which contain the term 
> "credit" but not as part of the phrase "credit card".
> 
> so:
> 
> 1. I don't want to get the documents which include just "credit card".
> 
> 2. I want to get the documents which include just "credit".
> 
> 3. I want to get the documents which include "credit" but not as part of 
> credit card.
> 
> 
> 
> for example:
> 
> doc1 text: "I want to buy with my credit in my card"
> 
> doc2 text: "I want to buy with my credit in my credit card"
> 
> doc3 text: "I want to buy with my credit card"
> 
> The documents should be returned:
> 
> doc1, doc2
> 
> I can't find nothing about NOTIN operator implementation in SOLR docs.
> 
> 
> 
> This electronic message may contain proprietary and confidential information 
> of Verint Systems Inc., its affiliates and/or subsidiaries. The information 
> is intended to be for the use of the individual(s) or entity(ies) named 
> above. If you are not the intended recipient (or authorized to receive this 
> e-mail for the intended recipient), you may not use, copy, disclose or 
> distribute to anyone this message or any information contained in this 
> message. If you have received this electronic message in error, please notify 
> us by replying to this e-mail.

Re: How to implement NOTIN operator with Solr

[Vincenzo D'Amore]

This is a tricky problem, you’re trying to handle the meani using the words. A 
simple solution could be apply a synonym filter that convert “credit card” in 
two terms: “creditcard” and “card”. In this way searching credit will not match 
any term.

Ciao,
Vincenzo

--
mobile: 3498513251
skype: free.dev

> On 19 Nov 2019, at 10:30, Raboah, Avi  wrote:
> 
> I am trying to find the documents which hit this example:
> 
> q=text:"credit" NOTIN "credit card"
> 
> for that query I want to get all the documents which contain the term 
> "credit" but not as part of the phrase "credit card".
> 
> so:
> 
> 1. I don't want to get the documents which include just "credit card".
> 
> 2. I want to get the documents which include just "credit".
> 
> 3. I want to get the documents which include "credit" but not as part of 
> credit card.
> 
> 
> 
> for example:
> 
> doc1 text: "I want to buy with my credit in my card"
> 
> doc2 text: "I want to buy with my credit in my credit card"
> 
> doc3 text: "I want to buy with my credit card"
> 
> The documents should be returned:
> 
> doc1, doc2
> 
> I can't find nothing about NOTIN operator implementation in SOLR docs.
> 
> 
> 
> This electronic message may contain proprietary and confidential information 
> of Verint Systems Inc., its affiliates and/or subsidiaries. The information 
> is intended to be for the use of the individual(s) or entity(ies) named 
> above. If you are not the intended recipient (or authorized to receive this 
> e-mail for the intended recipient), you may not use, copy, disclose or 
> distribute to anyone this message or any information contained in this 
> message. If you have received this electronic message in error, please notify 
> us by replying to this e-mail.

How to implement NOTIN operator with Solr

[Raboah, Avi]

I am trying to find the documents which hit this example:

q=text:"credit" NOTIN "credit card"

for that query I want to get all the documents which contain the term "credit" 
but not as part of the phrase "credit card".

so:

1. I don't want to get the documents which include just "credit card".

2. I want to get the documents which include just "credit".

3. I want to get the documents which include "credit" but not as part of 
credit card.



for example:

doc1 text: "I want to buy with my credit in my card"

doc2 text: "I want to buy with my credit in my credit card"

doc3 text: "I want to buy with my credit card"

The documents should be returned:

doc1, doc2

I can't find nothing about NOTIN operator implementation in SOLR docs.



This electronic message may contain proprietary and confidential information of 
Verint Systems Inc., its affiliates and/or subsidiaries. The information is 
intended to be for the use of the individual(s) or entity(ies) named above. If 
you are not the intended recipient (or authorized to receive this e-mail for 
the intended recipient), you may not use, copy, disclose or distribute to 
anyone this message or any information contained in this message. If you have 
received this electronic message in error, please notify us by replying to this 
e-mail.

Re: Active directory integration in Solr

[Charlie Hull]

Not out of the box, there are a few authentication plugins bundled but 
not for AD 
https://lucene.apache.org/solr/guide/7_2/authentication-and-authorization-plugins.html 
- there's also some useful stuff in Apache ManifoldCF 
https://www.francelabs.com/blog/tutorial-on-authorizations-for-manifold-cf-and-solr/ 



Best

Charlie

On 18/11/2019 15:08, Kommu, Vinodh K. wrote:

Hi,

Does anyone know that Solr has any out of the box capability to integrate 
Active directory (using LDAP) when security is enabled? Instead of creating 
users in security.json file, planning to use users who already exists in active 
directory so they can use their individual credentials rather than defining in 
Solr. Did anyone came across similar requirement? If so was there any working 
solution?


Thanks,
Vinodh

DTCC DISCLAIMER: This email and any files transmitted with it are confidential 
and intended solely for the use of the individual or entity to whom they are 
addressed. If you have received this email in error, please notify us 
immediately and delete the email and any attachments from your system. The 
recipient should check this email and any attachments for the presence of 
viruses. The company accepts no liability for any damage caused by any virus 
transmitted by this email.



--
Charlie Hull
Flax - Open Source Enterprise Search

tel/fax: +44 (0)8700 118334
mobile:  +44 (0)7767 825828
web: www.flax.co.uk