Re: Exact matching without using new fields

[Doss]

Hi,

You can try search query -> "+information +retrieval"

Meaning the document should have both the keywords. Doc 5 will also be in
the results.

https://lucene.apache.org/solr/guide/8_7/the-standard-query-parser.html#the-boolean-operator

- Mohandoss.

On Wed, Jan 20, 2021 at 1:38 AM gnandre  wrote:

> Thanks for replying, Dave.
>
> I am afraid that I am looking for non-index time i.e. query time solution.
>
> Actually in my case I am expecting both documents to be returned from your
> example. I am just trying to avoid returning of documents which contain a
> tokenized versions
> of the provided search query when it is enclosed within double quotes to
> indicate exact matching expectation.
>
> e.g.
> search query -> "information retrieval"
>
> This should match documents like following:
> doc 1: "information retrieval"
> doc 2: "Advanced information retrieval with Solr"
>
> but should NOT match documents like
> doc 3: "informed retrieval"
> doc 4: "information extraction"  (considering 'extraction' was a specified
> synonym of 'retrieval' )
> doc 5: "INFORMATION RETRIEVAL"
>
> etc
>
> I am also ok with these documents showing up as long as they show up at
> bottom. Also, query time solution is a must.
>
> On Tue, Jan 19, 2021 at 12:22 PM David R  wrote:
>
> > We had the same requirement. Just to echo back your requirements, I
> > understand your case to be this. Given these 2 doc titles:
> >
> > doc 1: "information retrieval"
> > doc 2: "Advanced information retrieval with Solr"
> >
> > You want a phrase search for "information retrieval" to find both
> > documents, but an EXACT phrase search for "information retrieval" to find
> > doc #1 only.
> >
> > If that's true, and case-sensitive search isn't a requirement, I indexed
> > this in the token stream, with adjacent positions of course.
> >
> > START information retrieval END
> > START advanced information retrieval with solr END
> >
> > And with our custom query parser, when an EXACT operator is found, I
> > tokenize the query to match the first case. Otherwise pass it through.
> >
> > Needs custom analyzers on the query and index sides to generate the
> > correct token sequences.
> >
> > It's worked out well for our case.
> >
> > Dave
> >
> >
> >
> > 
> > From: gnandre 
> > Sent: Tuesday, January 19, 2021 4:07 PM
> > To: solr-user@lucene.apache.org 
> > Subject: Exact matching without using new fields
> >
> > Hi,
> >
> > I am aware that to do exact matching (only whatever is provided inside
> > double quotes should be matched) in Solr, we can copy existing fields
> with
> > the help of copyFields into new fields that have very minimal
> tokenization
> > or no tokenization (e.g. using KeywordTokenizer or using string field
> type)
> >
> > However this solution is expensive in terms of index size because it
> might
> > almost double the size of the existing index.
> >
> > Is there any inexpensive way of achieving exact matches from the query
> > side. e.g. boost the original tokens more at query time compared to their
> > tokens?
> >
>

Re: read/write on different node?

[Doss]

Hi,

Master/Slave concept is not there in SolrCloud, but similar thing we can
achive by choosing TLOG + PULL replicas, where writes will happen in TLOG
nodes and the PULL replicas gets the updated segments.

Ensure minimum 2 TLOG nodes as PULL replica notes can't became leader of a
shard.

You can contol the selects to use only PULL replicas by
passing shards.preference=replica.type:PULL

http://ip:port
/solr/indexname/select?q=*:*=replica.type:PULL

- Mohandoss.

On Thu, Jan 21, 2021 at 6:51 PM Luke Oak  wrote:

> Thanks Mohandoss,
>
> It is realtime inserting and updating if a query is executed. My
> environment is three nodes, 3 shards and 2 replicas.
>
> I noticed there was master slave mode in the old version,  but for solr
> cloud, I don’t know whether it is doable.
>
> Derrick
>
> Sent from my iPhone
>
> > On Jan 21, 2021, at 3:15 AM, Doss  wrote:
> >
> > Hi,
> >
> > You haven't shared information about your environment and how frequently
> > you are commiting the changes, whether your user searching collection
> gets
> > real time inserts / updates etc.,
> >
> > but if you are not doing any real time analysis with the user query
> > information, you can store the information in mysql table and do a batch
> > import(full or delta). If you want the current approach then try
> increasing
> > the commit interval.
> >
> > Recently I have asked the same question and get to know that we can use
> > shards.preference (Replica Types) to achieve our requirement
> >
> https://lucene.472066.n3.nabble.com/Solrcloud-Reads-on-specific-nodes-tp4467568.html
> >
> > - Mohandoss
> >
> >> On Wed, Jan 20, 2021 at 7:22 PM Luke  wrote:
> >>
> >> Hi,
> >>
> >> I have one data collection on 3 shards and 2 replicas, user searches on
> it.
> >> Also I log all user queries and save to another collection on the same
> solr
> >> cloud, but user queries are very slow when there are a lot of logs to be
> >> written to the log collection.
> >>
> >> any solution for me, please advise. or does solr support separate write
> >> operation on different node and read on other nodes?
> >>
>

Re: read/write on different node?

[Doss]

Hi,

You haven't shared information about your environment and how frequently
you are commiting the changes, whether your user searching collection gets
real time inserts / updates etc.,

but if you are not doing any real time analysis with the user query
information, you can store the information in mysql table and do a batch
import(full or delta). If you want the current approach then try increasing
the commit interval.

Recently I have asked the same question and get to know that we can use
shards.preference (Replica Types) to achieve our requirement
https://lucene.472066.n3.nabble.com/Solrcloud-Reads-on-specific-nodes-tp4467568.html

- Mohandoss

On Wed, Jan 20, 2021 at 7:22 PM Luke  wrote:

> Hi,
>
> I have one data collection on 3 shards and 2 replicas, user searches on it.
> Also I log all user queries and save to another collection on the same solr
> cloud, but user queries are very slow when there are a lot of logs to be
> written to the log collection.
>
> any solution for me, please advise. or does solr support separate write
> operation on different node and read on other nodes?
>

Re: Solrcloud - Reads on specific nodes

[Doss]

Thanks Michael Gibney , Shawn Heisey for pointing in the right direction.

1. Will there be any performance degrade if we use shards.preference?
2. How about leader election if we decided to use NRT + PULL ? TLOG has the
advantage of participating in leader election correct?
3. NRT + TLOG is there any parameter which can reduce the TLOG replication
time

Have a greate week ahead!

Regards,
Mohandoss.

On Fri, Jan 15, 2021 at 9:20 PM Shawn Heisey  wrote:

> On 1/15/2021 7:56 AM, Doss wrote:
> > 1. Suppose we have 10 node SOLR Cloud setup, is it possible to dedicate 4
> > nodes for writes and 6 nodes for selects?
> >
> > 2. We have a SOLR cloud setup for our customer facing applications, and
> we
> > would like to have two more SOLR nodes for some backend jobs. Is it good
> > idea to form these nodes as slave nodes and making one node in the cloud
> as
> > Master?
>
> SolrCloud does not have masters or slaves.
>
> One thing you could do is set the replica types on four of those nodes
> to one type, and on the other nodes, use a different replica type.  For
> instance, the four nodes could be TLOG and the six nodes could be PULL.
>
> Then you can use the shards.preference parameter on your queries to only
> query the type of replica that you want.
>
>
> https://lucene.apache.org/solr/guide/8_7/distributed-requests.html#shards-preference-parameter
>
> Thanks,
> Shawn
>

Solrcloud - Reads on specific nodes

[Doss]

Dear All,

1. Suppose we have 10 node SOLR Cloud setup, is it possible to dedicate 4
nodes for writes and 6 nodes for selects?

2. We have a SOLR cloud setup for our customer facing applications, and we
would like to have two more SOLR nodes for some backend jobs. Is it good
idea to form these nodes as slave nodes and making one node in the cloud as
Master?

Thanks!
Mohandoss.

Remote error message: empty String & null:java.lang.NumberFormatException: empty String

[Doss]

We have 12 node SOLR cloud with 3 zookeeper ensemble
RAM: 80 CPU:40 Heap:16GB Records: 4 Million

We do real time update and deletes (by ID), and we do us Inplace updates
for 4 fields

We have one index with 4 shards: 1 shard in 3 nodes

Often we are getting the following errors

1. *2021-01-08 17:11:14.305 ERROR (qtp1720891078-7429) [c:profilesindex
s:shard4 r:core_node42 x:profilesindex_shard4_replica_n41]
o.a.s.s.HttpSolrCall
null:org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException:
Async exception during distributed update: Error from server at
http://171.0.0.145:8983/solr/profilesindex_shard3_replica_n49/
: null*

request: http://171.0.0.145:8983/solr/profilesindex_shard3_replica_n49/
Remote error message: empty String
at
org.apache.solr.update.processor.DistributedZkUpdateProcessor.doDistribFinish(DistributedZkUpdateProcessor.java:1193)
at
org.apache.solr.update.processor.DistributedUpdateProcessor.finish(DistributedUpdateProcessor.java:1125)
at
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:78)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:214)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:2606)
at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:812)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:588)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:415)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:345)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1596)
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:545)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:590)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1610)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1300)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:485)
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1580)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1215)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:221)
at
org.eclipse.jetty.server.handler.InetAccessHandler.handle(InetAccessHandler.java:177)
at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at
org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:322)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.Server.handle(Server.java:500)
at
org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:547)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)
at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273)
at
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
at
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
at java.base/java.lang.Thread.run(Thread.java:832)
=
2.  *2021-01-08 17:15:17.849 ERROR (qtp1720891078-7120) [c:profilesindex
s:shard4 r:core_node42 x:profilesindex_shard4_replica_n41]
o.a.s.s.HttpSolrCall null:java.lang.NumberFormatException: empty String*
at

SOLR 8.6.1 - Request Log - timeZone & Response Time

[Doss]

Hi,

We have upgraded our SOLR instances to 8.6.1, we are trying to change the
timeZone and enabling Response Time, below is our configuration, but the
change is not reflecting. true adding this
parameter throws exception.

File: server/etc/jetty_requestlog_xml


  

  
  

  /_mm_dd.request.log
  _MM_dd
  90
  true
  IST

  

  
  

  


Sample Logs
0:0:0:0:0:0:0:1 - - [18/Sep/2020:07:53:11 +] "GET
/solr/admin/info/logging?_=1600414902762=0=json HTTP/1.1" 200 802
0:0:0:0:0:0:0:1 - - [18/Sep/2020:07:53:22 +] "GET
/solr/admin/info/logging?_=1600414902762=0=json HTTP/1.1" 200 802
0:0:0:0:0:0:0:1 - - [18/Sep/2020:07:53:33 +] "GET
/solr/admin/info/logging?_=1600414902762=0=json HTTP/1.1" 200 802
0:0:0:0:0:0:0:1 - - [18/Sep/2020:07:53:44 +] "GET
/solr/admin/info/logging?_=1600414902762=0=json HTTP/1.1" 200 802
0:0:0:0:0:0:0:1 - - [18/Sep/2020:07:53:55 +] "GET
/solr/admin/info/logging?_=1600414902762=0=json HTTP/1.1" 200 802
0:0:0:0:0:0:0:1 - - [18/Sep/2020:07:54:06 +] "GET
/solr/admin/info/logging?_=1600414902762=0=json HTTP/1.1" 200 802
0:0:0:0:0:0:0:1 - - [18/Sep/2020:07:54:17 +] "GET
/solr/admin/info/logging?_=1600414902762=0=json HTTP/1.1" 200 802
0:0:0:0:0:0:0:1 - - [18/Sep/2020:07:54:28 +] "GET
/solr/admin/info/logging?_=1600414902762=0=json HTTP/1.1" 200 802

Thanks.
Doss.

Re: Production Issue: TIMED_WAITING - Will net.ipv4.tcp_tw_reuse=1 help?

[Doss]

Hi Dominique,

Our issues are similar to the one discussed here.
https://github.com/eclipse/jetty.project/issues/4105

Your views on this.

Thanks,
Mohandoss.

On Tue, Aug 11, 2020 at 7:06 AM Doss  wrote:

> Hi Dominique,
>
> Thanks for the response.
>
> I don't think I would use a JVM version 14. OpenJDK 11 in my opinion is
> the best choice for LTS version.
>
> >> We will try changing it.
>
> You change a lot of default values. Any specific raisons ? Il seems very
> aggressive !
>
> >> Our product team wants data to be reflected in Near Real Time.
>  mergePolicyFactory, mergeScheduler - This is based on our oldest SOLR
> cluster where these parameter tweaking gave good results.
>
> You have to analyze GC on all nodes !
>
> >> I checked other nodes GC, found no issues. I shared the node's GC which
> gets into trouble very frequently.
>
> Your heap is very big. According to full GC frequency, I don't think you
> really need such a big heap for only indexing. May be when you will perform
> queries.
>
> >> Heap Sizing is based on the select requests we are expecting. We expect
> it would be around 10 to 15 million per day. We have plans to increase CPU
> before routing select traffics.
>
> Did you check your network performances ?
>
> >> We do checked in sar reports, but unable to figure out an issue, we use
> 10 GBPS connection. Is there any SOLR metric API which will give network
> related information? Please suggest other ways to dig this further.
>
> Did you check Zookeeper logs ?
>
> >> We never looked at the Zookeeper logs, will check and share, is there
> any kind of information to watch out for?
>
> Regards,
> Doss
>
>
> On Monday, August 10, 2020, Dominique Bejean 
> wrote:
>
>> Doss,
>>
>> See below.
>>
>> Dominique
>>
>>
>> Le lun. 10 août 2020 à 17:41, Doss  a écrit :
>>
>>> Hi Dominique,
>>>
>>> Thanks for your response. Find below the details, please do let me know
>>> if anything I missed.
>>>
>>>
>>> *- hardware architecture and sizing*
>>> >> Centos 7, VMs,4CPUs, 66GB RAM, 16GB Heap, 250GB SSD
>>>
>>>
>>> *- JVM version / settings*
>>> >> Red Hat, Inc. OpenJDK 64-Bit Server VM, version:"14.0.1 14.0.1+7" -
>>> Default Settings including GC
>>>
>>
>> I don't think I would use a JVM version 14. OpenJDK 11 in my opinion is
>> the best choice for LTS version.
>>
>>
>>>
>>> *- Solr settings*
>>> >> softCommit: 15000 (15 sec), autoCommit: 30 (5 mins)
>>> >> class="org.apache.solr.index.TieredMergePolicyFactory">>> name="maxMergeAtOnce">30 100
>>> 30.0 
>>>
>>>   >> class="org.apache.lucene.index.ConcurrentMergeScheduler">>> name="maxMergeCount">18>> name="maxThreadCount">6
>>>
>>
>> You change a lot of default values. Any specific raisons ? Il seems very
>> aggressive !
>>
>>
>>>
>>>
>>> *- collections and queries information   *
>>> >> One Collection, with 4 shards , 3 replicas , 3.5 Million Records, 150
>>> columns, mostly integer fields, Average doc size is 350kb. Insert / Updates
>>> 0.5 Million Span across the whole day (peak time being 6PM to 10PM) ,
>>> selects not yet started. Daily once we do delta import of cetrain fields of
>>> type multivalued with some good amount of data.
>>>
>>> *- gc logs or gceasy results*
>>>
>>> Easy GC Report says GC health is good, one server's gc report:
>>> https://drive.google.com/file/d/1C2SqEn0iMbUOXnTNlYi46Gq9kF_CmWss/view?usp=sharing
>>> CPU Load Pattern:
>>> https://drive.google.com/file/d/1rjRMWv5ritf5QxgbFxDa0kPzVlXdbySe/view?usp=sharing
>>>
>>>
>> You have to analyze GC on all nodes !
>> Your heap is very big. According to full GC frequency, I don't think you
>> really need such a big heap for only indexing. May be when you will perform
>> queries.
>>
>> Did you check your network performances ?
>> Did you check Zookeeper logs ?
>>
>>
>>>
>>> Thanks,
>>> Doss.
>>>
>>>
>>>
>>> On Mon, Aug 10, 2020 at 7:39 PM Dominique Bejean <
>>> dominique.bej...@eolya.fr> wrote:
>>>
>>>> Hi Doss,
>>>>
>>>> See a lot of TIMED_WATING connection occurs with high tcp traffic
>>>> infrastructure as in a 

Re: Production Issue: TIMED_WAITING - Will net.ipv4.tcp_tw_reuse=1 help?

[Doss]

Hi Dominique,

Thanks for the response.

I don't think I would use a JVM version 14. OpenJDK 11 in my opinion is the
best choice for LTS version.

>> We will try changing it.

You change a lot of default values. Any specific raisons ? Il seems very
aggressive !

>> Our product team wants data to be reflected in Near Real Time.
 mergePolicyFactory, mergeScheduler - This is based on our oldest SOLR
cluster where these parameter tweaking gave good results.

You have to analyze GC on all nodes !

>> I checked other nodes GC, found no issues. I shared the node's GC which
gets into trouble very frequently.

Your heap is very big. According to full GC frequency, I don't think you
really need such a big heap for only indexing. May be when you will perform
queries.

>> Heap Sizing is based on the select requests we are expecting. We expect
it would be around 10 to 15 million per day. We have plans to increase CPU
before routing select traffics.

Did you check your network performances ?

>> We do checked in sar reports, but unable to figure out an issue, we use
10 GBPS connection. Is there any SOLR metric API which will give network
related information? Please suggest other ways to dig this further.

Did you check Zookeeper logs ?

>> We never looked at the Zookeeper logs, will check and share, is there
any kind of information to watch out for?

Regards,
Doss


On Monday, August 10, 2020, Dominique Bejean 
wrote:

> Doss,
>
> See below.
>
> Dominique
>
>
> Le lun. 10 août 2020 à 17:41, Doss  a écrit :
>
>> Hi Dominique,
>>
>> Thanks for your response. Find below the details, please do let me know
>> if anything I missed.
>>
>>
>> *- hardware architecture and sizing*
>> >> Centos 7, VMs,4CPUs, 66GB RAM, 16GB Heap, 250GB SSD
>>
>>
>> *- JVM version / settings*
>> >> Red Hat, Inc. OpenJDK 64-Bit Server VM, version:"14.0.1 14.0.1+7" -
>> Default Settings including GC
>>
>
> I don't think I would use a JVM version 14. OpenJDK 11 in my opinion is
> the best choice for LTS version.
>
>
>>
>> *- Solr settings*
>> >> softCommit: 15000 (15 sec), autoCommit: 30 (5 mins)
>> > class="org.apache.solr.index.TieredMergePolicyFactory">> name="maxMergeAtOnce">30 100
>> 30.0 
>>
>>   > class="org.apache.lucene.index.ConcurrentMergeScheduler">> name="maxMergeCount">186<
>> /mergeScheduler>
>>
>
> You change a lot of default values. Any specific raisons ? Il seems very
> aggressive !
>
>
>>
>>
>> *- collections and queries information   *
>> >> One Collection, with 4 shards , 3 replicas , 3.5 Million Records, 150
>> columns, mostly integer fields, Average doc size is 350kb. Insert / Updates
>> 0.5 Million Span across the whole day (peak time being 6PM to 10PM) ,
>> selects not yet started. Daily once we do delta import of cetrain fields of
>> type multivalued with some good amount of data.
>>
>> *- gc logs or gceasy results*
>>
>> Easy GC Report says GC health is good, one server's gc report:
>> https://drive.google.com/file/d/1C2SqEn0iMbUOXnTNlYi46Gq9kF_
>> CmWss/view?usp=sharing
>> CPU Load Pattern: https://drive.google.com/file/d/
>> 1rjRMWv5ritf5QxgbFxDa0kPzVlXdbySe/view?usp=sharing
>>
>>
> You have to analyze GC on all nodes !
> Your heap is very big. According to full GC frequency, I don't think you
> really need such a big heap for only indexing. May be when you will perform
> queries.
>
> Did you check your network performances ?
> Did you check Zookeeper logs ?
>
>
>>
>> Thanks,
>> Doss.
>>
>>
>>
>> On Mon, Aug 10, 2020 at 7:39 PM Dominique Bejean <
>> dominique.bej...@eolya.fr> wrote:
>>
>>> Hi Doss,
>>>
>>> See a lot of TIMED_WATING connection occurs with high tcp traffic
>>> infrastructure as in a LAMP solution when the Apache server can't
>>> anymore connect to the MySQL/MariaDB database.
>>> In this case, tweak net.ipv4.tcp_tw_reuse is a possible solution (but
>>> never net.ipv4.tcp_tw_recycle as you suggested in your previous post).
>>> This
>>> is well explained in this great article
>>> https://vincent.bernat.ch/en/blog/2014-tcp-time-wait-state-linux
>>>
>>> However, in general and more specifically in your case, I would
>>> investigate
>>> the root cause of your issue and do not try to find a workaround.
>>>
>>> Can you provide more information about your use case (we know : 3 node
>>> SOLR
>>> (8.3.1 NRT) + 3 Node 

Re: Production Issue: TIMED_WAITING - Will net.ipv4.tcp_tw_reuse=1 help?

[Doss]

Hi Dominique,

Thanks for your response. Find below the details, please do let me know if
anything I missed.


*- hardware architecture and sizing*
>> Centos 7, VMs,4CPUs, 66GB RAM, 16GB Heap, 250GB SSD


*- JVM version / settings*
>> Red Hat, Inc. OpenJDK 64-Bit Server VM, version:"14.0.1 14.0.1+7" -
Default Settings including GC


*- Solr settings*
>> softCommit: 15000 (15 sec), autoCommit: 30 (5 mins)
30 100
30.0 

  186


*- collections and queries information   *
>> One Collection, with 4 shards , 3 replicas , 3.5 Million Records, 150
columns, mostly integer fields, Average doc size is 350kb. Insert / Updates
0.5 Million Span across the whole day (peak time being 6PM to 10PM) ,
selects not yet started. Daily once we do delta import of cetrain fields of
type multivalued with some good amount of data.

*- gc logs or gceasy results*

Easy GC Report says GC health is good, one server's gc report:
https://drive.google.com/file/d/1C2SqEn0iMbUOXnTNlYi46Gq9kF_CmWss/view?usp=sharing
CPU Load Pattern:
https://drive.google.com/file/d/1rjRMWv5ritf5QxgbFxDa0kPzVlXdbySe/view?usp=sharing



Thanks,
Doss.



On Mon, Aug 10, 2020 at 7:39 PM Dominique Bejean 
wrote:

> Hi Doss,
>
> See a lot of TIMED_WATING connection occurs with high tcp traffic
> infrastructure as in a LAMP solution when the Apache server can't
> anymore connect to the MySQL/MariaDB database.
> In this case, tweak net.ipv4.tcp_tw_reuse is a possible solution (but
> never net.ipv4.tcp_tw_recycle as you suggested in your previous post). This
> is well explained in this great article
> https://vincent.bernat.ch/en/blog/2014-tcp-time-wait-state-linux
>
> However, in general and more specifically in your case, I would investigate
> the root cause of your issue and do not try to find a workaround.
>
> Can you provide more information about your use case (we know : 3 node SOLR
> (8.3.1 NRT) + 3 Node Zookeeper Ensemble) ?
>
>- hardware architecture and sizing
>- JVM version / settings
>- Solr settings
>- collections and queries information
>    - gc logs or gceasy results
>
> Regards
>
> Dominique
>
>
>
> Le lun. 10 août 2020 à 15:43, Doss  a écrit :
>
> > Hi,
> >
> > In solr 8.3.1 source, I see the following , which I assume could be the
> > reason for the issue "Max requests queued per destination 3000 exceeded
> for
> > HttpDestination",
> >
> >
> solr/solrj/src/java/org/apache/solr/client/solrj/impl/Http2SolrClient.java:
> >private static final int MAX_OUTSTANDING_REQUESTS = 1000;
> >
> solr/solrj/src/java/org/apache/solr/client/solrj/impl/Http2SolrClient.java:
> >  available = new Semaphore(MAX_OUTSTANDING_REQUESTS, false);
> >
> solr/solrj/src/java/org/apache/solr/client/solrj/impl/Http2SolrClient.java:
> >  return MAX_OUTSTANDING_REQUESTS * 3;
> >
> > how can I increase this?
> >
> > On Mon, Aug 10, 2020 at 12:01 AM Doss  wrote:
> >
> > > Hi,
> > >
> > > We are having 3 node SOLR (8.3.1 NRT) + 3 Node Zookeeper Ensemble now
> and
> > > then we are facing "Max requests queued per destination 3000 exceeded
> for
> > > HttpDestination"
> > >
> > > After restart evering thing starts working fine until another problem.
> > > Once a problem occurred we are seeing soo many TIMED_WAITING threads
> > >
> > > Server 1:
> > >*7722*  Threads are in TIMED_WATING
> > >
> >
> ("lock":"java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@151d5f2f
> > > ")
> > > Server 2:
> > >*4046*   Threads are in TIMED_WATING
> > >
> >
> ("lock":"java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1e0205c3
> > > ")
> > > Server 3:
> > >*4210*   Threads are in TIMED_WATING
> > >
> >
> ("lock":"java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5ee792c0
> > > ")
> > >
> > > Please suggest whether net.ipv4.tcp_tw_reuse=1 will help ? or how can
> we
> > > increase the 3000 limit?
> > >
> > > Sorry, since I haven't got any response to my previous query,  I am
> > > creating this as new,
> > >
> > > Thanks,
> > > Mohandoss.
> > >
> >
>

Re: Production Issue: TIMED_WAITING - Will net.ipv4.tcp_tw_reuse=1 help?

[Doss]

Hi,

In solr 8.3.1 source, I see the following , which I assume could be the
reason for the issue "Max requests queued per destination 3000 exceeded for
HttpDestination",

solr/solrj/src/java/org/apache/solr/client/solrj/impl/Http2SolrClient.java:
   private static final int MAX_OUTSTANDING_REQUESTS = 1000;
solr/solrj/src/java/org/apache/solr/client/solrj/impl/Http2SolrClient.java:
 available = new Semaphore(MAX_OUTSTANDING_REQUESTS, false);
solr/solrj/src/java/org/apache/solr/client/solrj/impl/Http2SolrClient.java:
 return MAX_OUTSTANDING_REQUESTS * 3;

how can I increase this?

On Mon, Aug 10, 2020 at 12:01 AM Doss  wrote:

> Hi,
>
> We are having 3 node SOLR (8.3.1 NRT) + 3 Node Zookeeper Ensemble now and
> then we are facing "Max requests queued per destination 3000 exceeded for
> HttpDestination"
>
> After restart evering thing starts working fine until another problem.
> Once a problem occurred we are seeing soo many TIMED_WAITING threads
>
> Server 1:
>*7722*  Threads are in TIMED_WATING
> ("lock":"java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@151d5f2f
> ")
> Server 2:
>*4046*   Threads are in TIMED_WATING
> ("lock":"java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1e0205c3
> ")
> Server 3:
>*4210*   Threads are in TIMED_WATING
> ("lock":"java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5ee792c0
> ")
>
> Please suggest whether net.ipv4.tcp_tw_reuse=1 will help ? or how can we
> increase the 3000 limit?
>
> Sorry, since I haven't got any response to my previous query,  I am
> creating this as new,
>
> Thanks,
> Mohandoss.
>

Production Issue: TIMED_WAITING - Will net.ipv4.tcp_tw_reuse=1 help?

[Doss]

Hi,

We are having 3 node SOLR (8.3.1 NRT) + 3 Node Zookeeper Ensemble now and
then we are facing "Max requests queued per destination 3000 exceeded for
HttpDestination"

After restart evering thing starts working fine until another problem. Once
a problem occurred we are seeing soo many TIMED_WAITING threads

Server 1:
   *7722*  Threads are in TIMED_WATING
("lock":"java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@151d5f2f
")
Server 2:
   *4046*   Threads are in TIMED_WATING
("lock":"java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1e0205c3
")
Server 3:
   *4210*   Threads are in TIMED_WATING
("lock":"java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5ee792c0
")

Please suggest whether net.ipv4.tcp_tw_reuse=1 will help ? or how can we
increase the 3000 limit?

Sorry, since I haven't got any response to my previous query,  I am
creating this as new,

Thanks,
Mohandoss.

Re: java.util.concurrent.RejectedExecutionException: Max requests queued per destination 3000 exceeded for HttpDestination

[Doss]

Hi,

Sorry, actually the issue was not fixed, on that day the service
restart cleared the TIMED_WAITNG threads. The problem re-occured recently,
everything works fine until a disturbance happens to the cluster due to
high load or a node responds slow.

We have 3 servers, the following TIMED_WAITING threads are reducing

Server 1:
   *7722*  Threads are in TIMED_WATING
("lock":"java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@151d5f2f
")
Server 2:
   *4046*   Threads are in TIMED_WATING
("lock":"java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1e0205c3
")
Server 3:
   *4210*   Threads are in TIMED_WATING
("lock":"java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5ee792c0
")

Please help in resolving this issue.

Thanks,
Doss

On Wed, Aug 5, 2020 at 11:30 PM Doss  wrote:

> Hi All,
>
> We have tried the system variable recommend here
>
> https://www.eclipse.org/jetty/documentation/current/high-load.html
>
> The errors stopped now, and system is running stable.
>
> Is there any disadvantage using the following?
>
> sysctl -w net.ipv4.tcp_tw_recycle=1
>
>
> Somewhere I read tw_reuse also will help, can we use both ? please suggest.
>
> Thanks,
> Doss.
>
> On Sunday, August 2, 2020, Doss  wrote:
>
>> Hi All,
>>
>> We are having SOLR (8.3.1) CLOUD (NRT) with Zookeeper Ensemble , 3 nodes
>> each on Centos VMs
>>
>> SOLR Nodes has 66GB RAM, 15GB HEAP MEM, 4 CPUs.
>> Record Count: 33L. Avg Doc Size is 350Kb.
>>
>> In recent times while doing a full import (Few bulk data fields computed
>> on a daily basis) we are getting this error, what could be the problem? how
>> to increase the Queue Size? Please help.
>>
>> 2020-08-02 08:10:30.847 ERROR
>> (updateExecutor-5-thread-190288-processing-x:userinfoindex_6jul20_shard4_replica_n13
>> r:core_node24 null n:172.29.3.23:8983_solr c:userinfoindex_6jul20
>> s:shard4) [c:userinfoindex_6jul20 s:shard4 r:core_node24
>> x:userinfoindex_6jul20_shard4_replica_n13]
>> o.a.s.u.ErrorReportingConcurrentUpdateSolrClient Error when calling
>> SolrCmdDistributor$Req: cmd=add{,id=1081904963}; node=ForwardNode:
>> http://172.29.3.23:8983/solr/userinfoindex_6jul20_shard3_replica_n10/ to
>> http://172.29.3.23:8983/solr/userinfoindex_6jul20_shard3_replica_n10/ =>
>> java.io.IOException: java.util.concurrent.RejectedExecutionException: Max
>> requests queued per destination 3000 exceeded for HttpDestination[
>> http://172.29
>> .3.23:8983]@5a9a216b,queue=3000,pool=MultiplexConnectionPool@545dc448
>> [c=4/4,b=4,m=0,i=0]
>> java.io.IOException: java.util.concurrent.RejectedExecutionException: Max
>> requests queued per destination 3000 exceeded for HttpDestination[
>> http://172.29
>> .3.23:8983]@5a9a216b,queue=3000,pool=MultiplexConnectionPool@545dc448
>> [c=4/4,b=4,m=0,i=0]
>> Suppressed: java.io.IOException:
>> java.util.concurrent.RejectedExecutionException: Max requests queued per
>> destination 3000 exceeded for HttpDestination[http://172.29
>> .3.23:8983]@5a9a216b,queue=3000,pool=MultiplexConnectionPool@545dc448
>> [c=4/4,b=4,m=0,i=0]
>> Caused by: java.util.concurrent.RejectedExecutionException: Max requests
>> queued per destination 3000 exceeded for HttpDestination[http://172.29
>> .3.23:8983]@5a9a216b,queue=3000,pool=MultiplexConnectionPool@545dc448
>> [c=4/4,b=4,m=0,i=0]
>> Caused by: java.util.concurrent.RejectedExecutionException: Max requests
>> queued per destination 3000 exceeded for HttpDestination[http://172.29
>> .3.23:8983]@5a9a216b,queue=3000,pool=MultiplexConnectionPool@545dc448
>> [c=4/4,b=4,m=0,i=0]
>>
>> 2020-08-02 08:10:30.911 ERROR
>> (updateExecutor-5-thread-190288-processing-x:userinfoindex_6jul20_shard4_replica_n13
>> r:core_node24 null n:172.29.3.23:8983_solr c:userinfoindex_6jul20
>> s:shard4) [c:userinfoindex_6jul20 s:shard4 r:core_node24
>> x:userinfoindex_6jul20_shard4_replica_n13]
>> o.a.s.u.ErrorReportingConcurrentUpdateSolrClient Error when calling
>> SolrCmdDistributor$Req: cmd=add{,id=1034918151}; node=ForwardNode:
>> http://172.29.3.23:8983/solr/userinfoindex_6jul20_shard1_replica_n3/ to
>> http://172.29.3.23:8983/solr/userinfoindex_6jul20_shard1_replica_n3/ =>
>> java.io.IOException: java.util.concurrent.RejectedExecutionException: Max
>> requests queued per destination 3000 exceeded for HttpDestination[
>> http://172.29
>> .3.23:8983]@5a9a216b,queue=3000,pool=MultiplexConnectionPool@545dc448
>> [c=4/4,b=4,m=0,i=0]
>>
>>
>> Thanks,
>> Doss.
>>
>

Re: java.util.concurrent.RejectedExecutionException: Max requests queued per destination 3000 exceeded for HttpDestination

[Doss]

Hi All,

We have tried the system variable recommend here

https://www.eclipse.org/jetty/documentation/current/high-load.html

The errors stopped now, and system is running stable.

Is there any disadvantage using the following?

sysctl -w net.ipv4.tcp_tw_recycle=1


Somewhere I read tw_reuse also will help, can we use both ? please suggest.

Thanks,
Doss.

On Sunday, August 2, 2020, Doss  wrote:

> Hi All,
>
> We are having SOLR (8.3.1) CLOUD (NRT) with Zookeeper Ensemble , 3 nodes
> each on Centos VMs
>
> SOLR Nodes has 66GB RAM, 15GB HEAP MEM, 4 CPUs.
> Record Count: 33L. Avg Doc Size is 350Kb.
>
> In recent times while doing a full import (Few bulk data fields computed
> on a daily basis) we are getting this error, what could be the problem? how
> to increase the Queue Size? Please help.
>
> 2020-08-02 08:10:30.847 ERROR (updateExecutor-5-thread-
> 190288-processing-x:userinfoindex_6jul20_shard4_replica_n13 r:core_node24
> null n:172.29.3.23:8983_solr c:userinfoindex_6jul20 s:shard4)
> [c:userinfoindex_6jul20 s:shard4 r:core_node24
> x:userinfoindex_6jul20_shard4_replica_n13] o.a.s.u.
> ErrorReportingConcurrentUpdateSolrClient Error when calling
> SolrCmdDistributor$Req: cmd=add{,id=1081904963}; node=ForwardNode:
> http://172.29.3.23:8983/solr/userinfoindex_6jul20_shard3_replica_n10/ to
> http://172.29.3.23:8983/solr/userinfoindex_6jul20_shard3_replica_n10/ =>
> java.io.IOException: java.util.concurrent.RejectedExecutionException: Max
> requests queued per destination 3000 exceeded for HttpDestination[
> http://172.29.3.23:8983]@5a9a216b,queue=3000,pool=MultiplexConnectionPool@
> 545dc448[c=4/4,b=4,m=0,i=0]
> java.io.IOException: java.util.concurrent.RejectedExecutionException: Max
> requests queued per destination 3000 exceeded for HttpDestination[
> http://172.29.3.23:8983]@5a9a216b,queue=3000,pool=MultiplexConnectionPool@
> 545dc448[c=4/4,b=4,m=0,i=0]
> Suppressed: java.io.IOException: 
> java.util.concurrent.RejectedExecutionException:
> Max requests queued per destination 3000 exceeded for HttpDestination[
> http://172.29.3.23:8983]@5a9a216b,queue=3000,pool=MultiplexConnectionPool@
> 545dc448[c=4/4,b=4,m=0,i=0]
> Caused by: java.util.concurrent.RejectedExecutionException: Max requests
> queued per destination 3000 exceeded for HttpDestination[http://172.29.
> 3.23:8983]@5a9a216b,queue=3000,pool=MultiplexConnectionPool@
> 545dc448[c=4/4,b=4,m=0,i=0]
> Caused by: java.util.concurrent.RejectedExecutionException: Max requests
> queued per destination 3000 exceeded for HttpDestination[http://172.29.
> 3.23:8983]@5a9a216b,queue=3000,pool=MultiplexConnectionPool@
> 545dc448[c=4/4,b=4,m=0,i=0]
>
> 2020-08-02 08:10:30.911 ERROR (updateExecutor-5-thread-
> 190288-processing-x:userinfoindex_6jul20_shard4_replica_n13 r:core_node24
> null n:172.29.3.23:8983_solr c:userinfoindex_6jul20 s:shard4)
> [c:userinfoindex_6jul20 s:shard4 r:core_node24
> x:userinfoindex_6jul20_shard4_replica_n13] o.a.s.u.
> ErrorReportingConcurrentUpdateSolrClient Error when calling
> SolrCmdDistributor$Req: cmd=add{,id=1034918151}; node=ForwardNode:
> http://172.29.3.23:8983/solr/userinfoindex_6jul20_shard1_replica_n3/ to
> http://172.29.3.23:8983/solr/userinfoindex_6jul20_shard1_replica_n3/ =>
> java.io.IOException: java.util.concurrent.RejectedExecutionException: Max
> requests queued per destination 3000 exceeded for HttpDestination[
> http://172.29.3.23:8983]@5a9a216b,queue=3000,pool=MultiplexConnectionPool@
> 545dc448[c=4/4,b=4,m=0,i=0]
>
>
> Thanks,
> Doss.
>

java.util.concurrent.RejectedExecutionException: Max requests queued per destination 3000 exceeded for HttpDestination

[Doss]

Hi All,

We are having SOLR (8.3.1) CLOUD (NRT) with Zookeeper Ensemble , 3 nodes
each on Centos VMs

SOLR Nodes has 66GB RAM, 15GB HEAP MEM, 4 CPUs.
Record Count: 33L. Avg Doc Size is 350Kb.

In recent times while doing a full import (Few bulk data fields computed on
a daily basis) we are getting this error, what could be the problem? how to
increase the Queue Size? Please help.

2020-08-02 08:10:30.847 ERROR
(updateExecutor-5-thread-190288-processing-x:userinfoindex_6jul20_shard4_replica_n13
r:core_node24 null n:172.29.3.23:8983_solr c:userinfoindex_6jul20 s:shard4)
[c:userinfoindex_6jul20 s:shard4 r:core_node24
x:userinfoindex_6jul20_shard4_replica_n13]
o.a.s.u.ErrorReportingConcurrentUpdateSolrClient Error when calling
SolrCmdDistributor$Req: cmd=add{,id=1081904963}; node=ForwardNode:
http://172.29.3.23:8983/solr/userinfoindex_6jul20_shard3_replica_n10/ to
http://172.29.3.23:8983/solr/userinfoindex_6jul20_shard3_replica_n10/ =>
java.io.IOException: java.util.concurrent.RejectedExecutionException: Max
requests queued per destination 3000 exceeded for
HttpDestination[http://172.29.3.23:8983
]@5a9a216b,queue=3000,pool=MultiplexConnectionPool@545dc448
[c=4/4,b=4,m=0,i=0]
java.io.IOException: java.util.concurrent.RejectedExecutionException: Max
requests queued per destination 3000 exceeded for
HttpDestination[http://172.29.3.23:8983
]@5a9a216b,queue=3000,pool=MultiplexConnectionPool@545dc448
[c=4/4,b=4,m=0,i=0]
Suppressed: java.io.IOException:
java.util.concurrent.RejectedExecutionException: Max requests queued per
destination 3000 exceeded for HttpDestination[http://172.29.3.23:8983
]@5a9a216b,queue=3000,pool=MultiplexConnectionPool@545dc448
[c=4/4,b=4,m=0,i=0]
Caused by: java.util.concurrent.RejectedExecutionException: Max requests
queued per destination 3000 exceeded for
HttpDestination[http://172.29.3.23:8983
]@5a9a216b,queue=3000,pool=MultiplexConnectionPool@545dc448
[c=4/4,b=4,m=0,i=0]
Caused by: java.util.concurrent.RejectedExecutionException: Max requests
queued per destination 3000 exceeded for
HttpDestination[http://172.29.3.23:8983
]@5a9a216b,queue=3000,pool=MultiplexConnectionPool@545dc448
[c=4/4,b=4,m=0,i=0]

2020-08-02 08:10:30.911 ERROR
(updateExecutor-5-thread-190288-processing-x:userinfoindex_6jul20_shard4_replica_n13
r:core_node24 null n:172.29.3.23:8983_solr c:userinfoindex_6jul20 s:shard4)
[c:userinfoindex_6jul20 s:shard4 r:core_node24
x:userinfoindex_6jul20_shard4_replica_n13]
o.a.s.u.ErrorReportingConcurrentUpdateSolrClient Error when calling
SolrCmdDistributor$Req: cmd=add{,id=1034918151}; node=ForwardNode:
http://172.29.3.23:8983/solr/userinfoindex_6jul20_shard1_replica_n3/ to
http://172.29.3.23:8983/solr/userinfoindex_6jul20_shard1_replica_n3/ =>
java.io.IOException: java.util.concurrent.RejectedExecutionException: Max
requests queued per destination 3000 exceeded for
HttpDestination[http://172.29.3.23:8983
]@5a9a216b,queue=3000,pool=MultiplexConnectionPool@545dc448
[c=4/4,b=4,m=0,i=0]


Thanks,
Doss.

SOLR Data Import Handler : A command is still running...

[Doss]

We are doing hourly data import to our index, per day one or two requests
are getting failed with the message "A command is still running...".

1. Does it mean, the data import not happened for the last hour?
2. If you look at the "Full Dump Started" time has an older data, in the
below log all most 13 days, why is that so?

userinfoindex start - Wed Jan 22 05:12:01 IST 2020 {
"responseHeader":{ "status":0, "QTime":0},   "initArgs":[
"defaults",[   "config","data-import.xml"]],
"command":"full-import",   "status":"busy",   "importResponse":"A command
is still running...",   "statusMessages":{ "Time
Elapsed":"298:1:59.986", "Total Requests made to DataSource":"1",
"Total Rows Fetched":"17426", "Total Documents Processed":"17425",
"Total Documents Skipped":"0", "Full Dump Started":"2020-01-09
19:10:02"}}

Thanks,
Doss.

Re: Query Regarding SOLR cross collection join

[Doss]

@ Alessandro Benedetti , Thanks for your input!

@ Mikhail Khludnev , I made docValues="true" for from & to and did a index
rotation, now the score join works perfectly!  Saw 7x performance increase.
Thanks!


On Thu, Jan 23, 2020 at 9:53 PM Mikhail Khludnev  wrote:

> On Wed, Jan 22, 2020 at 4:27 PM Doss  wrote:
>
> > HI,
> >
> > SOLR version 8.3.1 (10 nodes), zookeeper ensemble (3 nodes)
> >
> > Read somewhere that the score join parser will be faster, but for me it
> > produces no results. I am using string type fields for from and to.
> >
>
> That's odd. Can you try to enable docValues on from side and reindex small
> portion of data just to check if it works.
>
>
> >
> >
> > Thanks!
> >
>
>
> --
> Sincerely yours
> Mikhail Khludnev
>

Query Regarding SOLR cross collection join

[Doss]

HI,

SOLR version 8.3.1 (10 nodes), zookeeper ensemble (3 nodes)

One of our use cases requires joins, we are joining 2 large indexes. As
required by SOLR one index (2GB) has one shared and 10 replicas and the
other has 10 shard (40GB / Shard).

The query takes too much time, some times in minutes how can we improve
this?

Debug query produces one or more based on the number of shards (i believe)

"time":303442,
"fromSetSize":0,
"toSetSize":81653955,
"fromTermCount":0,
"fromTermTotalDf":0,
"fromTermDirectCount":0,
"fromTermHits":0,
"fromTermHitsTotalDf":0,
"toTermHits":0,
"toTermHitsTotalDf":0,
"toTermDirectCount":0,
"smallSetsDeferred":0,
"toSetDocsAdded":0},

here what is the  toSetSize  mean? does it read 81MB of data from the
index? how can we reduce this?

Read somewhere that the score join parser will be faster, but for me it
produces no results. I am using string type fields for from and to.


Thanks!

In Place Updates: Can we filter on fields with only docValues="true"

[Doss]

Hi,

4 to 5 million documents.

For an NTR index, we need a field to be updated very frequently and filter
results based on it. Will In-Place updates help us?




Thanks,
Doss.

Re: Suggestion Needed: Exclude documents that are already served / viewed by a customer

[Doss]

Hi Experts,

We are migrating our entire search platform from SPHINX to SOLR, we wanted
to do this without any flaw so any suggestion would be greatly appreciated.

Thanks!


On Fri, Sep 6, 2019 at 11:13 AM Doss  wrote:

> Dear Experts,
>
> For a matchmaking portal, we have one requirement where in, if a customer
> viewed complete details of a bride or groom then we have to exclude that
> profile id from further search results. Currently, along with other details
> we are storing the viewed profile ids in a field (multivalued field)
> against that bride or groom's details.
>
> Eg., if A viewed B, then in B's document under the field saw_me we will
> add A's id
>
> while searching, lets say, the currently searching members id is 123456
> then we will fire a query like
>
> fq=-saw_me:(123456)
>
> Problem #1: The saw_me field value is growing like anything.
> Problem #2: Removal of ids which are deleted from the base. Right now we
> are doing this job as follows
>Query #1: fq=saw_me:(123456)=DocId //Get all document ids
> which has the deleted id as part of saw_me field.
>Query #2: {"DociId":"234567","saw_me":{"remove":"123456"}
> //loop through the results got through the 1st query and fire the update
> query one by one
>
> We feel that this method of handling is not that optimum, so we need
> expert advice. Please guide.
>

Re: Production Issue: SOLR node goes to non responsive , restart not helping at peak hours

[Doss]

Jorn we have add additional zookeeper nodes, now it is a 3 node quorum.

Does all nodes in a quorum sends heart beat request to all cores and shards
?

If zookeeper node 1 unable to communicate with a shard and it declares that
shard as dead, now this state can be changed by zookeeper node 2 if it got
a successful response from that particular shard?

On Thu, Sep 5, 2019 at 4:53 PM Jörn Franke  wrote:

> 1 Node zookeeper ensemble does not sound very healthy
>
> > Am 05.09.2019 um 13:07 schrieb Doss :
> >
> > Hi,
> >
> > We are using 3 node SOLR (7.0.1) cloud setup 1 node zookeeper ensemble.
> > Each system has 16CPUs, 90GB RAM (14GB HEAP), 130 cores (3 replicas NRT)
> > with index size ranging from 700MB to 20GB.
> >
> > autoCommit - 10 minutes once
> > softCommit - 30 Sec Once
> >
> > At peak time if a shard goes to recovery mode many other shards also
> going
> > to recovery mode in few minutes, which creates huge load (200+ load
> > average) and SOLR becomes non responsive. To fix this we are restarting
> the
> > node, again leader tries to correct the index by initiating replication,
> > which causes load again, and the node goes to non responsive state.
> >
> > As soon as a node starts the replication process initiated for all 130
> > cores, is there any we control it, like one after the other?
> >
> > Thanks,
> > Doss.
>

Re: Suggestion Needed: Exclude documents that are already served / viewed by a customer

[Doss]

Jorn Thanks for the input, I learned something new today!
https://cwiki.apache.org/confluence/display/solr/BloomIndexComponent this
works per segment level, but our requirement is per document level.

Thanks,
Mohandoss.

On Fri, Sep 6, 2019 at 11:41 AM Jörn Franke  wrote:

> I am not 100% sure if Solr has something out of the box, but you could
> implement a bloom filter https://en.wikipedia.org/wiki/Bloom_filter and
> store it in Solr. It is a probabilistic data structure, which is not
> growing, but can achieve your use case.
> However it has a caveat: it can, for example in your case, only say for
> sure if a person A has NOT visited person B. If you want to know if Person
> A has visited person B then there might be (with a known probability) false
> positives.
>
> Nevertheless, it still seems to address your use case as you want to show
> only not visited profiles.
>
> > Am 06.09.2019 um 07:43 schrieb Doss :
> >
> > Dear Experts,
> >
> > For a matchmaking portal, we have one requirement where in, if a customer
> > viewed complete details of a bride or groom then we have to exclude that
> > profile id from further search results. Currently, along with other
> details
> > we are storing the viewed profile ids in a field (multivalued field)
> > against that bride or groom's details.
> >
> > Eg., if A viewed B, then in B's document under the field saw_me we will
> add
> > A's id
> >
> > while searching, lets say, the currently searching members id is 123456
> > then we will fire a query like
> >
> > fq=-saw_me:(123456)
> >
> > Problem #1: The saw_me field value is growing like anything.
> > Problem #2: Removal of ids which are deleted from the base. Right now we
> > are doing this job as follows
> >   Query #1: fq=saw_me:(123456)=DocId //Get all document ids
> > which has the deleted id as part of saw_me field.
> >   Query #2: {"DociId":"234567","saw_me":{"remove":"123456"}
> //loop
> > through the results got through the 1st query and fire the update query
> one
> > by one
> >
> > We feel that this method of handling is not that optimum, so we need
> expert
> > advice. Please guide.
>

Suggestion Needed: Exclude documents that are already served / viewed by a customer

[Doss]

Dear Experts,

For a matchmaking portal, we have one requirement where in, if a customer
viewed complete details of a bride or groom then we have to exclude that
profile id from further search results. Currently, along with other details
we are storing the viewed profile ids in a field (multivalued field)
against that bride or groom's details.

Eg., if A viewed B, then in B's document under the field saw_me we will add
A's id

while searching, lets say, the currently searching members id is 123456
then we will fire a query like

fq=-saw_me:(123456)

Problem #1: The saw_me field value is growing like anything.
Problem #2: Removal of ids which are deleted from the base. Right now we
are doing this job as follows
   Query #1: fq=saw_me:(123456)=DocId //Get all document ids
which has the deleted id as part of saw_me field.
   Query #2: {"DociId":"234567","saw_me":{"remove":"123456"} //loop
through the results got through the 1st query and fire the update query one
by one

We feel that this method of handling is not that optimum, so we need expert
advice. Please guide.

Re: Production Issue: SOLR node goes to non responsive , restart not helping at peak hours

[Doss]

Dear Jack,

Thanks for your input. Non of our cores were created with autoAddReplicas.
The problem we are facing is, upon rebooting leader tries to sync the data
with other nodes which are part of the cluster.

Thanks,
Doss.

On Thu, Sep 5, 2019 at 9:46 PM Jack Schlederer 
wrote:

> My mistake on the link, which should be this:
>
> https://lucene.apache.org/solr/guide/7_1/solrcloud-autoscaling-auto-add-replicas.html#implementation-using-autoaddreplicas-trigger
>
> --Jack
>
> On Thu, Sep 5, 2019 at 11:02 AM Jack Schlederer 
> wrote:
>
> > I'd defer to the committers if they have any further advice, but you
> might
> > have to suspend the autoAddReplicas trigger through the autoscaling API (
> >
> https://solr.stage.ecommerce.sandbox.directsupply-sandbox.cloud:8985/solr/
> )
> > if you set up your collections with autoAddReplicas enabled. Then, the
> > system will not try to re-create missing replicas.
> >
> > Just another note on your setup-- It seems to me like using only 3 nodes
> > for 168 GB worth of indices isn't making the most of SolrCloud, which
> > provides the capabilities for sharding indices across a high number of
> > nodes. Just a data point for you to consider when considering your
> cluster
> > sizing, my org is running only about 50GB of indices, but we run it over
> 35
> > nodes with 8GB of heap apiece, each collection with 2+ shards.
> >
> > --Jack
> >
> > On Thu, Sep 5, 2019 at 8:47 AM Doss  wrote:
> >
> >> Thanks Eric for the explanation. Sum of all our index size is about 138
> >> GB,
> >> only 2 indexes are > 19 GB, time to scale up :-). Adding new hardware
> will
> >> require at least couple of days, till that time is there any option to
> >> control the replication method?
> >>
> >> Thanks,
> >> Doss.
> >>
> >> On Thu, Sep 5, 2019 at 6:12 PM Erick Erickson 
> >> wrote:
> >>
> >> > You say you have three nodes, 130 replicas and a replication factor of
> >> 3,
> >> > so
> >> > you have 130 cores/node. At least one of those cores has a 20G index,
> >> > right?
> >> >
> >> > What is the sum of all the indexes on a single physical machine?
> >> >
> >> > I think your system is under-provisioned and that you’ve been riding
> at
> >> > the edge
> >> > of instability for quite some time and have added enough more docs
> that
> >> > you finally reached a tipping point. But that’s largely speculation.
> >> >
> >> > So adding more heap may help. But Real Soon Now you need to think
> about
> >> > adding
> >> > more hardware and moving some of your replicas to that new hardware.
> >> >
> >> > Again, this is speculation. But when systems are running with an
> >> > _aggregate_
> >> > index size that is many multiples of the available memory (total
> >> phycisal
> >> > memory)
> >> > it’s a red flag. I’m guessing a bit since I don’t know the aggregate
> for
> >> > all replicas…
> >> >
> >> > Best,
> >> > Erick
> >> >
> >> > > On Sep 5, 2019, at 8:08 AM, Doss  wrote:
> >> > >
> >> > > @Jorn We are adding few more zookeeper nodes soon. Thanks.
> >> > >
> >> > > @ Erick, sorry I couldn't understand it clearly, we have 90GB RAM
> per
> >> > node,
> >> > > out of which 14 GB assigned for HEAP, you mean to say we have to
> >> allocate
> >> > > more HEAP? or we need add more Physical RAM?
> >> > >
> >> > > This system ran for 8 to 9 months without any major issues, in
> recent
> >> > times
> >> > > only we are facing too many such incidents.
> >> > >
> >> > > On Thu, Sep 5, 2019 at 5:20 PM Erick Erickson <
> >> erickerick...@gmail.com>
> >> > > wrote:
> >> > >
> >> > >> If I'm reading this correctly, you have a huge amount of index in
> not
> >> > much
> >> > >> memory. You only have 14g allocated across 130 replicas, at least
> >> one of
> >> > >> which has a 20g index. You don't need as much memory as your
> >> aggregate
> >> > >> index size, but this system feels severely under provisioned. I
> >> suspect
> >> > >> that's the root of your instability
> >> > >>
> >> > >> Best,
> &

Re: Production Issue: SOLR node goes to non responsive , restart not helping at peak hours

[Doss]

Thanks Eric for the explanation. Sum of all our index size is about 138 GB,
only 2 indexes are > 19 GB, time to scale up :-). Adding new hardware will
require at least couple of days, till that time is there any option to
control the replication method?

Thanks,
Doss.

On Thu, Sep 5, 2019 at 6:12 PM Erick Erickson 
wrote:

> You say you have three nodes, 130 replicas and a replication factor of 3,
> so
> you have 130 cores/node. At least one of those cores has a 20G index,
> right?
>
> What is the sum of all the indexes on a single physical machine?
>
> I think your system is under-provisioned and that you’ve been riding at
> the edge
> of instability for quite some time and have added enough more docs that
> you finally reached a tipping point. But that’s largely speculation.
>
> So adding more heap may help. But Real Soon Now you need to think about
> adding
> more hardware and moving some of your replicas to that new hardware.
>
> Again, this is speculation. But when systems are running with an
> _aggregate_
> index size that is many multiples of the available memory (total phycisal
> memory)
> it’s a red flag. I’m guessing a bit since I don’t know the aggregate for
> all replicas…
>
> Best,
> Erick
>
> > On Sep 5, 2019, at 8:08 AM, Doss  wrote:
> >
> > @Jorn We are adding few more zookeeper nodes soon. Thanks.
> >
> > @ Erick, sorry I couldn't understand it clearly, we have 90GB RAM per
> node,
> > out of which 14 GB assigned for HEAP, you mean to say we have to allocate
> > more HEAP? or we need add more Physical RAM?
> >
> > This system ran for 8 to 9 months without any major issues, in recent
> times
> > only we are facing too many such incidents.
> >
> > On Thu, Sep 5, 2019 at 5:20 PM Erick Erickson 
> > wrote:
> >
> >> If I'm reading this correctly, you have a huge amount of index in not
> much
> >> memory. You only have 14g allocated across 130 replicas, at least one of
> >> which has a 20g index. You don't need as much memory as your aggregate
> >> index size, but this system feels severely under provisioned. I suspect
> >> that's the root of your instability
> >>
> >> Best,
> >> Erick
> >>
> >> On Thu, Sep 5, 2019, 07:08 Doss  wrote:
> >>
> >>> Hi,
> >>>
> >>> We are using 3 node SOLR (7.0.1) cloud setup 1 node zookeeper ensemble.
> >>> Each system has 16CPUs, 90GB RAM (14GB HEAP), 130 cores (3 replicas
> NRT)
> >>> with index size ranging from 700MB to 20GB.
> >>>
> >>> autoCommit - 10 minutes once
> >>> softCommit - 30 Sec Once
> >>>
> >>> At peak time if a shard goes to recovery mode many other shards also
> >> going
> >>> to recovery mode in few minutes, which creates huge load (200+ load
> >>> average) and SOLR becomes non responsive. To fix this we are restarting
> >> the
> >>> node, again leader tries to correct the index by initiating
> replication,
> >>> which causes load again, and the node goes to non responsive state.
> >>>
> >>> As soon as a node starts the replication process initiated for all 130
> >>> cores, is there any we control it, like one after the other?
> >>>
> >>> Thanks,
> >>> Doss.
> >>>
> >>
>
>

Re: Production Issue: SOLR node goes to non responsive , restart not helping at peak hours

[Doss]

@Jorn We are adding few more zookeeper nodes soon. Thanks.

@ Erick, sorry I couldn't understand it clearly, we have 90GB RAM per node,
out of which 14 GB assigned for HEAP, you mean to say we have to allocate
more HEAP? or we need add more Physical RAM?

This system ran for 8 to 9 months without any major issues, in recent times
only we are facing too many such incidents.

On Thu, Sep 5, 2019 at 5:20 PM Erick Erickson 
wrote:

> If I'm reading this correctly, you have a huge amount of index in not much
> memory. You only have 14g allocated across 130 replicas, at least one of
> which has a 20g index. You don't need as much memory as your aggregate
> index size, but this system feels severely under provisioned. I suspect
> that's the root of your instability
>
> Best,
> Erick
>
> On Thu, Sep 5, 2019, 07:08 Doss  wrote:
>
> > Hi,
> >
> > We are using 3 node SOLR (7.0.1) cloud setup 1 node zookeeper ensemble.
> > Each system has 16CPUs, 90GB RAM (14GB HEAP), 130 cores (3 replicas NRT)
> > with index size ranging from 700MB to 20GB.
> >
> > autoCommit - 10 minutes once
> > softCommit - 30 Sec Once
> >
> > At peak time if a shard goes to recovery mode many other shards also
> going
> > to recovery mode in few minutes, which creates huge load (200+ load
> > average) and SOLR becomes non responsive. To fix this we are restarting
> the
> > node, again leader tries to correct the index by initiating replication,
> > which causes load again, and the node goes to non responsive state.
> >
> > As soon as a node starts the replication process initiated for all 130
> > cores, is there any we control it, like one after the other?
> >
> > Thanks,
> > Doss.
> >
>

Production Issue: SOLR node goes to non responsive , restart not helping at peak hours

[Doss]

Hi,

We are using 3 node SOLR (7.0.1) cloud setup 1 node zookeeper ensemble.
Each system has 16CPUs, 90GB RAM (14GB HEAP), 130 cores (3 replicas NRT)
with index size ranging from 700MB to 20GB.

autoCommit - 10 minutes once
softCommit - 30 Sec Once

At peak time if a shard goes to recovery mode many other shards also going
to recovery mode in few minutes, which creates huge load (200+ load
average) and SOLR becomes non responsive. To fix this we are restarting the
node, again leader tries to correct the index by initiating replication,
which causes load again, and the node goes to non responsive state.

As soon as a node starts the replication process initiated for all 130
cores, is there any we control it, like one after the other?

Thanks,
Doss.

Re: SOLR Cloud: Few cores goes to recovery mode all of a sudden

[Doss]

Thanks Erick,

ulimit in all three lodes are more than 65K including max process list.  If
you look at the timestamp the core down error happened ahead of unable to
create thread error, and also core down error took place in node1 and
unable to create thread error took place in node3.

BTW we are running the SOLR instances in VMs. We are maintaining only last
three days zookeeper logs.

Thanks,
Doss.

On Tue, Sep 3, 2019 at 8:21 PM Erick Erickson 
wrote:

> The “unable to create new thread” is where I’d focus first. It means
> you’re running out of some system resources and it’s quite possible that
> your other problems are arising from that root cause.
>
> What are you “ulimit” settings? the number of file handles and processes
> should be set to 65k at least.
>
> Best,
> Erick
>
> > On Sep 3, 2019, at 9:38 AM, Doss  wrote:
> >
> > Hi,
> >
> > We are using 3 node SOLR (7.0.1) cloud setup 1 node zookeeper ensemble.
> > Each system has 16CPUs, 90GB RAM (14GB HEAP), 130 cores (3 replicas NRT)
> > with index size ranging from 700MB to 20GB.
> >
> > autoCommit - 10 minutes once
> > softCommit - 30 Sec Once
> >
> > We are facing the following problems in recent times
> >
> >
> 
> >
> >
> > 1. Few Indexes goes to recovery mode without any prior error or warnings
> >
> > 2019-09-03 08:42:41.650 ERROR (qtp959447386-38425) [c:viewsindex s:shard1
> > r:core_node3 x:viewsindex] o.a.s.h.RequestHandlerBase or
> > g.apache.solr.common.SolrException: SolrCloud join: memberdetails has a
> > local replica (memberdetails) on 12.1.2.10:8983_solr, but it is down
> >at
> >
> org.apache.solr.search.join.ScoreJoinQParserPlugin.findLocalReplicaForFromIndex(ScoreJoinQParserPlugin.java:325)
> >
> >at
> >
> org.apache.solr.search.join.ScoreJoinQParserPlugin.getCoreName(ScoreJoinQParserPlugin.java:285)
> >
> >at
> >
> org.apache.solr.search.JoinQParserPlugin$1.parseJoin(JoinQParserPlugin.java:92)
> >
> >at
> >
> org.apache.solr.search.JoinQParserPlugin$1.parse(JoinQParserPlugin.java:74)
> >at org.apache.solr.search.QParser.getQuery(QParser.java:168)
> >at
> >
> org.apache.solr.handler.component.QueryComponent.prepare(QueryComponent.java:207)
> >
> >at
> >
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:269)
> >
> >at
> >
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:177)
> >
> >at org.apache.solr.core.SolrCore.execute(SolrCore.java:2474)
> >at
> > org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:720)
> >at
> org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:526)
> >at
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:378)
> >
> >at
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:322)
> >
> >at
> >
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1691)
> >
> >at
> >
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
> >at
> >
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> >
> >at
> >
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
> >at
> >
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
> >
> >at
> >
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
> >
> >at
> > org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
> >at
> >
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
> >
> >at
> >
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
> >
> >at
> >
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> >
> >at
> >
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
> >
> >at
> >
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
> >
> >at
> >
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
> >
> >at
>

SOLR Cloud: Few cores goes to recovery mode all of a sudden

[Doss]

y.java:337)

at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:349)
at
org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1948)




3. Unable to create new native thread

2019-09-03 08:42:36.460 WARN  (qtp1348949648-20980) [c:viewsindex s:shard1
r:core_node5 x:viewsindex] o.e.j.u.t.QueuedThreadPool
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:714)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.startThreads(QueuedThreadPool.java:475)

at
org.eclipse.jetty.util.thread.QueuedThreadPool.access$200(QueuedThreadPool.java:48)

at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:634)

at java.lang.Thread.run(Thread.java:745)




What could be the reason how can we avoid such failures in future?

How can we fix "AlreadyClosedException" without restarting the service.


Thanks,
Doss.

SOLR Atomic Update - String multiValued Field

[Doss]

HI,

I have a multiValued field of type String.



I want to keep this list unique, so I am using atomic updates with
"add-distinct"

{"docid":123456,"namelist":{"add-distinct":["Adam","Jane"]}}

but this is not maintaining the expected uniqueness, am I doing something
wrong? Guide me please.

Thanks,
Doss.

RE: [solr-solrcloud] How does DIH work when there are multiple nodes?

[Doss]

Hi,

The data import process will not happen automatically, we have to do it
manually through the admin interface or by calling the URL

https://lucene.apache.org/solr/guide/7_5/uploading-structured-data-store-data-with-the-data-import-handler.html

Full Import:

http://node1ip:8983/solr/yourindexname/dataimport?command=full-import=true

Delta Import:

http://node1ip:8983/solr/yourindexname/dataimport?command=delta-import=true


If you want to do the delta import automatically you can setup a cron
(linux) which can call the URL periodically.

Best,
Doss.




--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: [solr-solrcloud] How does DIH work when there are multiple nodes?

[Doss]

Hi,

I am assuming you are having the same index replicated in all 3 nodes, then
doing a full index/ delta index using DIH in one node will replicate the
data to other nodes, so no need to do it in all 3 nodes. Hope this helps!

Best,
Doss.



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Regarding Shards - Composite / Implicit , Replica Type - NRT / TLOG

[Doss]

Hi,

We are planning to setup a SOLR cloud with 6 nodes for 3 million records
(expected to grow to 5 million in a year), with 150 fields and over all
index would come around 120GB.

We plan to use NRT with 5 sec soft commit and 1 min hard commit.

Expected query volume would be 5000 select hits per second and 7000 inserts
/ updates per second.

Our records can be classified under 15 categories, but they will not have
even number of records, few categories will have more number of records.

Queries will also come in the same pattern, that is., categories with high
number of records will get high volume of select / updates.

For this situation we are confused in choosing what type of sharding would
help us in better performance in both select and updates?

Composite / implicit - Composite with 15 shards or implicit based on 15
categories.

Our select queries will have minimum 15 filters in fq, with extensive
function queries used in sort.

Updates will have 6 integer fields, 5 string fields and 4 string/integer
fields with multi valued.

If we choose implicit to boost select performance, our updates will be
heavy on few shards (major category shards), will this be a problem?

For our kind of situation which replica Type can we choose? All NRT or NRT
with TLOG ?

Thanks in advance!

Best,
Doss.

Re: SOLR Cloud - Full index replication

[Doss]

Thanks Erick!

We are using SOLR version 7.0.1.

is there any disadvantages if we increase  peer sync size to 1000 ?

We have analysed the GC logs but we have not seen long GC pauses so far.

We tried to find the reason for the full sync, but noting more informative,
but we have seen too many logs which reads "No registered leader was found
after waiting for 4000ms" followed by this full index.

Thanks,
Doss.


On Sun, Dec 30, 2018 at 8:49 AM Erick Erickson 
wrote:

> No. There's a "peer sync" that will try to update from the leader's
> transaction log if (and only if) the replica has fallen behind. By
> "fallen behind" I mean it was unable to accept any updates for
> some period of time. The default peer sync size is 100 docs,
> you can make it larger see numRecordsToKeep here:
> http://lucene.apache.org/solr/guide/7_6/updatehandlers-in-solrconfig.html
>
> Some observations though:
> 12G heap for 250G of index on disk _may_ work, but I'd be looking at
> the GC characteristics, particularly stop-the-world pauses.
>
> Your hard commit interval looks too long. I'd shorten it to < 1 minute
> with openSearcher=false. See:
>
> https://lucidworks.com/2013/08/23/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/
>
> I'd concentrate on _why_ the replica goes into recovery in the first
> place. You say you're on 7x, which one? Starting in 7.3 the recovery
> logic was pretty thoroughly reworked, so _which_ 7x version is
> important to know.
>
> The Solr logs should give you some idea of _why_ the replica
> goes into recovery, concentrate on the replica that goes into
> recovery and the corresponding leader's log.
>
> Best,
> Erick
>
> On Sat, Dec 29, 2018 at 6:23 PM Doss  wrote:
> >
> > we are using 3 node solr (64GB ram/8cpu/12GB heap)cloud setup with
> version
> > 7.X. we have 3 indexes/collection on each node. index size were about
> > 250GB. NRT with 5sec soft /10min hard commit. Sometimes in any one node
> we
> > are seeing full index replication started running..  is there any
> > configuration which forces solr to replicate full , like 100/200 updates
> > difference if a node sees with the leader ? - Thanks.
>

SOLR Cloud - Full index replication

[Doss]

we are using 3 node solr (64GB ram/8cpu/12GB heap)cloud setup with version
7.X. we have 3 indexes/collection on each node. index size were about
250GB. NRT with 5sec soft /10min hard commit. Sometimes in any one node we
are seeing full index replication started running..  is there any
configuration which forces solr to replicate full , like 100/200 updates
difference if a node sees with the leader ? - Thanks.

Re: SOLR: Array Key to Value on Result

[Doss]

Hi,

I found a work around for our requirement with the help of Transforming
Result Documents.

https://lucene.apache.org/solr/guide/7_3/transforming-result-documents.html#subquery

I need insights about the performance impact (if any) this is going to
create. I assume this transformation is happening after the results being
obtained by the parent query, so there won't be much performance impact it
will create, but 

we are going to use this functionality for a large and busy index, so before
taking further steps I need expert opinion. 

Thanks,
Doss.



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

SOLR: Array Key to Value on Result

[Doss]

Hi,

I am looking for a solution. Lets say we have complete City, State, Country
values in an array (with Multi language), based on user selection I will be
storing only the corresponding keys in the index.

If I place the complete  array inside solr conf or inside the data
directory, is there any component  which can map the key and adds the
corresponding value in the result set?

Eg.

Array: {"1":{"EN":"India","TAM":"\u0b87\u0ba8\u0bcd\u0ba4\u0bbf\u0baf\u0bbe"}}

while getting the result if I pass

country key as 1 & language as EN then the result should have the country
value as India.

Please help to crack this.


Thanks,
Doss.

JVM Heap Memory Increase (SOLR CLOUD)

[Doss]

We have SOLR(7.0.1) cloud 3 VM Linux instances wit 4 CPU, 90 GB RAM with
zookeeper (3.4.11) ensemble running on the same machines. We have 130 cores of
overall size of 45GB. No Sharding, almost all VMs has the same copy of data.
These nodes are under LB.

http://lucene.472066.n3.nabble.com/SOLR-Cloud-1500-threads-are-in-TIMED-WAITING-status-td4383636.html
- Refer this for Merge and commit configs

JVM Heap Size : 15GB

Optimize : Daily Once

GC Config: Default

-XX:+PrintGCApplicationStoppedTime
-XX:+PrintGCDateStamps
-XX:+PrintGCDetails
-XX:+PrintGCTimeStamps
-XX:+PrintHeapAtGC
-XX:+PrintTenuringDistribution
-XX:+UseCMSInitiatingOccupancyOnly
-XX:+UseConcMarkSweepGC
-XX:+UseGCLogFileRotation
-XX:+UseParNewGC
-XX:-OmitStackTraceInFastThrow
-XX:CMSInitiatingOccupancyFraction=50
-XX:CMSMaxAbortablePrecleanTime=6000
-XX:ConcGCThreads=4
-XX:GCLogFileSize=20M
-XX:MaxTenuringThreshold=8
-XX:NewRatio=3
-XX:NumberOfGCLogFiles=9
-XX:OnOutOfMemoryError=/home/solr/bin/oom_solr.sh 8980
/data/solr/server/logs
-XX:ParallelGCThreads=4

ISSUE: After running 8 to 9 hours the JVM heap memory keeps on increasing,
at that time if we did optimize then I am seeing 3 to 3.5 gb reduction, but
doing optimize at day time will be a problem, on the other side if the heap
got full then OOM exception is happening and the cloud crashes.

I read somewhere that G1GC will give better results, but SOLR experts
doesn't encourage to use it

what else we can do to resolve this issue?

Thanks,
Doss.

Re: SOLR Cloud: 1500+ threads are in TIMED_WAITING status

[Doss]

Hi,

Is there any network parameter that we need to fine tune? Is there any
specific tweaking needed to deploy solr in virtual machines? We use VMware. 

Thanks.



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: SOLR Cloud: 1500+ threads are in TIMED_WAITING status

[Doss]

Hi Emir,

Just realised DBQ = Delete by Query,  we are not using that, we are deleting
documents using the document id / unique id.

Thanks,
Mohandoss.





--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: SOLR Cloud: 1500+ threads are in TIMED_WAITING status

[Doss]

Hi Emir,

We do fire delete queries but that is very very minimal.

Thanks!
Mohandoss



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: SOLR Cloud: 1500+ threads are in TIMED_WAITING status

[Doss]

@wunder

Are you sending updates in batches? Are you doing a commit after every
update? 

>> We want the system to be near real time, so we are not doing updates in
>> batches and also we are not doing commit after every update.
>> autoSoftCommit once in every minute, and autoCommit once in every  10
>> minutes.

This thread increase is not happening in all the time, on our peak hours
where used we to get more user interactions the system works absolutely
fine, suddenly this problem creeps up and system gets into trouble.

nproc value increased 18000. 

Did jetty related linux fine tuning  as described in the below link

http://www.eclipse.org/jetty/documentation/current/high-load.html

Thanks.



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

SOLR Cloud: 1500+ threads are in TIMED_WAITING status

[Doss]

We have SOLR(7.0.1) cloud 3 VM Linux instances wit 4 CPU, 90 GB RAM with
zookeeper (3.4.11) ensemble running on the same machines. We have 130 cores
of overall size of 45GB. No Sharding, almost all VMs has the same copy of
data. These nodes are under LB.

Index Config:
=

300

   30
   100
   30.0



   18
   6


Commit Configs:
===

   ${solr.autoCommit.maxTime:60}
   false



   ${solr.autoSoftCommit.maxTime:6}



We do 3500 Insert / Updates per second spread across all 130 cores, We yet
to start using selects effectively.

The problem what we are facing is at times suddenly the thread count
increase heavily which results SOLR non responsive or throwing 503 response
for client (PHP HTTP CURL) requests.

Today 04-04-2018 the thread dump shows that the peak went upto 13000+

Please hlep me in fixing this issue. Thanks!


Sample Threads:
===

1.updateExecutor-2-thread-25746-processing-http:
172.10.2.19:8983//solr//profileviews x:profileviews r:core_node2
n:172.10.2.18:8983_solr s:shard1 c:profileviews", "state":"TIMED_WAITING",
"lock":"java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@297be1d5",
"cpuTime":"162.4371ms", "userTime":"120.ms",
"stackTrace":["sun.misc.Unsafe.park(Native Method)",
"java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)",
"java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)",

2. ERROR true HttpSolrCall
null:org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException:
Async exception during distributed update: Error from server at
172.10.2.18:8983/solr/profileviews: Server Error request:
http://172.10.2.18:8983/solr/profileviews/update?update.distrib=TOLEADER=http%3A%2F%2F172.10.2.19%3A8983%2Fsolr%2Fprofileviews%2F=javabin=2
Remote error message: empty String

3. So Many Threads like:
"name":"qtp959447386-21",
"state":"TIMED_WAITING",

"lock":"java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6a1a2bf4
",
"cpuTime":"4522.0837ms",
"userTime":"3770.ms",
"stackTrace":["sun.misc.Unsafe.park(Native Method)",

"java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)",

"java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)",

"org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)",

"org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)",

"org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)",

"org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)",
  "java.lang.Thread.run(Thread.java:748)"

Re: SOLR not starting after restart 2 node cloud setup

[Doss]

Dear Erick,

Thanks for your thoughts, it helped me a lot. In my instances no solr logs
are appended in to catalina.out.

Now I placed the log4j.properties file. Solr logs are captured in solr.log
file with the help of it I found the reason for the issue.

I am starting tomcat with the option -Dbootstrap_conf=true which made solr
to look for core configuration files in a wrong directory, after removing
this it started without any issues.

I also commented suggester component which made solr to load fast.

Thanks,
Doss.




On Thu, Nov 20, 2014 at 9:47 PM, Erick Erickson erickerick...@gmail.com
wrote:

 Doss:

 Tomcat often puts things in catalina.out, you might check there,
 I've often seen logging information from Solr go there by
 default.

 Without having some idea what kinds of problems Solr is
 reporting when you see this situation, it's really hard to say.

 Some things I'd check first though, in order of what
 I _guess_ is most likely.

  There have been anecdotal reports (in fact, I'm trying
 to understand the why of it right now) of the suggester
 taking a long time to initialize, even if you don't use it!
 So if you're not using the suggest component, try
 commenting out those sections in solrconfig.xml for
 the cores in question. I like this explanation since it
 fits with your symptoms, but I don't like it since the
 index you are using isn't all that big. So it's something
 of a shot in the dark. I expect that the core will
 _eventually_ come up, but I've seen reports of 10-15
 minutes being required, far beyond my patience! That
 said, this would also explain why deleting the index
 works.

  OutOfMemory errors. You might be able to attach
 jConsole (part of the standard Java stuff) to the process
 and monitor the memory usage. If it's being pushed near
 the 5G limit that's the first thing I'd suspect.

  If you're using the default setups, then the Zookeeper
 timeout may be too low, I think the default (not sure about
 whether it's been changed in 4.9) is 15 seconds, 30-60
 is usually much better.

 Best,
 Erick


 On Thu, Nov 20, 2014 at 3:47 AM, Doss itsmed...@gmail.com wrote:
  Dear Erick,
 
  Forgive my ignorance.
 
  Please find some of the details you required.
 
  *have you looked at the solr logs?*
 
Sorry I haven't defined the log4j.properties file, so I don't have
 solr
  logs. Since it requires tomcat restart I am planning to do it in next
  restart.
 
  But found the following in tomcat log
 
  18-Nov-2014 11:27:29.028 WARNING [localhost-startStop-2]
  org.apache.catalina.loader.WebappClassLoader.clearReferencesThreads The
 web
  application [/mima] appears to have started a thread named
  [localhost-startStop-1-SendThread(10.236.149.28:2181)] but has failed to
  stop it. This is very likely to create a memory leak. Stack trace of
 thread:
   sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
   sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
   sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
   sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
   sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
 
 org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
   org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
 
 
  *How big are the cores?*
 
  We have 16 cores, out of it only 5 are big ones. Total size of all 16
  cores is 10+ GB
 
  *How many docs in the cores when the problem happens?*
 
  1 core with 163 fields and 33,00,000 documents (Index size 2+ GB)
   4 cores with 3 fields and has 150,00,000 (approx) documents (1.2 to 1.5
 GB)
  remaining cores are 1,00,000 to 40,00,000 documents
 
  *How much memory are you allocating the JVM? *
 
  5GB for JVM, Total RAM available in the systems is 30 GB
 
  *can you restart Tomcat without a problem?*
 
  This problem is occurring in production, I never tried.
 
 
  Thanks,
  Doss.
 
 
  On Wed, Nov 19, 2014 at 7:55 PM, Erick Erickson erickerick...@gmail.com
 
  wrote:
 
  You've really got to provide details for us to say much
  of anything. There are about a zillion things that it could be.
 
  In particular, have you looked at the solr logs? Are there
  any interesting things in them? How big are the cores?
  How much memory are you allocating the JVM? How
  many docs in the cores when the problem happens?
  Before the nodes stop responding, can you restart
  Tomcat without a problem?
 
  You might review:
  http://wiki.apache.org/solr/UsingMailingLists
 
  Best,
  Erick
 
 
  On Wed, Nov 19, 2014 at 1:04 AM, Doss itsmed...@gmail.com wrote:
   I have two node SOLR (4.9.0) cloud with Tomcat (8), Zookeeper. At
 times
   SOLR in Node 1 stops responding, to fix the issue I am restarting
 tomcat
  in
   Node 1, but SOLR not starting up, but if I remove the solr cores in
 both
   nodes and try restarting it starts working, and then I have to reindex
  the
   whole data again. We are using this setup in production because of
 this
   issue we

Re: SOLR not starting after restart 2 node cloud setup

[Doss]

Dear Erick,

Forgive my ignorance.

Please find some of the details you required.

*have you looked at the solr logs?*

  Sorry I haven't defined the log4j.properties file, so I don't have solr
logs. Since it requires tomcat restart I am planning to do it in next
restart.

But found the following in tomcat log

18-Nov-2014 11:27:29.028 WARNING [localhost-startStop-2]
org.apache.catalina.loader.WebappClassLoader.clearReferencesThreads The web
application [/mima] appears to have started a thread named
[localhost-startStop-1-SendThread(10.236.149.28:2181)] but has failed to
stop it. This is very likely to create a memory leak. Stack trace of thread:
 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
 sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
 sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
 sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
 sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
 
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
 org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)


*How big are the cores?*

 We have 16 cores, out of it only 5 are big ones. Total size of all 16
cores is 10+ GB

*How many docs in the cores when the problem happens?*

1 core with 163 fields and 33,00,000 documents (Index size 2+ GB)
 4 cores with 3 fields and has 150,00,000 (approx) documents (1.2 to 1.5 GB)
remaining cores are 1,00,000 to 40,00,000 documents

*How much memory are you allocating the JVM? *

5GB for JVM, Total RAM available in the systems is 30 GB

*can you restart Tomcat without a problem?*

This problem is occurring in production, I never tried.


Thanks,
Doss.


On Wed, Nov 19, 2014 at 7:55 PM, Erick Erickson erickerick...@gmail.com
wrote:

 You've really got to provide details for us to say much
 of anything. There are about a zillion things that it could be.

 In particular, have you looked at the solr logs? Are there
 any interesting things in them? How big are the cores?
 How much memory are you allocating the JVM? How
 many docs in the cores when the problem happens?
 Before the nodes stop responding, can you restart
 Tomcat without a problem?

 You might review:
 http://wiki.apache.org/solr/UsingMailingLists

 Best,
 Erick


 On Wed, Nov 19, 2014 at 1:04 AM, Doss itsmed...@gmail.com wrote:
  I have two node SOLR (4.9.0) cloud with Tomcat (8), Zookeeper. At times
  SOLR in Node 1 stops responding, to fix the issue I am restarting tomcat
 in
  Node 1, but SOLR not starting up, but if I remove the solr cores in both
  nodes and try restarting it starts working, and then I have to reindex
 the
  whole data again. We are using this setup in production because of this
  issue we are having 1 to 1.30 hours of service down time. Any suggestions
  would be greatly appreciated.
 
  Thanks,
  Doss.

SOLR not starting after restart 2 node cloud setup

[Doss]

I have two node SOLR (4.9.0) cloud with Tomcat (8), Zookeeper. At times
SOLR in Node 1 stops responding, to fix the issue I am restarting tomcat in
Node 1, but SOLR not starting up, but if I remove the solr cores in both
nodes and try restarting it starts working, and then I have to reindex the
whole data again. We are using this setup in production because of this
issue we are having 1 to 1.30 hours of service down time. Any suggestions
would be greatly appreciated.

Thanks,
Doss.

Re: Tagging using SOLR

[Doss]

Dear Thorsten, Erik,

Thanks for the guidelines, but basically our idea is to build a system like
http://del.icio.us/tag/, is it possible to take counts of similar words from
a solr indexed field?

Thanks,
Mohandoss


On 9/6/07, Erik Hatcher [EMAIL PROTECTED] wrote:


 On Sep 6, 2007, at 3:29 AM, Doss wrote:
  We are running an appalication built using SOLR, now we are trying
  to build
  a tagging system using the existing SOLR indexed field called
  tag_keywords, this field has different keywords seperated by
  comma, please
  give suggestions on how can we build tagging system using this field?

 There is also a wiki page on some brainstorming on how to implement
 tagging within Solr: http://wiki.apache.org/solr/UserTagDesign

 It's easy enough to have a tag_keywords field, but updating a single
 tag_keywords field is not so straightforward without sending the
 entire document to Solr every time it is tagged.  See SOLR-139's
 extensive comments and patches to see what you're getting into.

Erik

Tagging using SOLR

[Doss]

Dear all,

We are running an appalication built using SOLR, now we are trying to build
a tagging system using the existing SOLR indexed field called
tag_keywords, this field has different keywords seperated by comma, please
give suggestions on how can we build tagging system using this field?

Thanks,
Mohandoss.

Re: Doubt in using synonyms.txt

[Doss]

Thanks Yonik.

Regards,
Doss.

On 5/25/07, Yonik Seeley [EMAIL PROTECTED] wrote:


On 5/24/07, Doss [EMAIL PROTECTED] wrote:
 Is it advisable to maintain a large amount of data in synonyms.txt file?

It's read into an in-memory map, so the only real impact is increased
RAM usage.  There really shouldn't be a performance impact.

-Yonik

Doubt in using synonyms.txt

[Doss]

Dear all,

Is it advisable to maintain a large amount of data in synonyms.txt file?

Thanks,
Doss.

help need on words with special characters

[Doss]

Hi,

I am new to solr(and 0 in lucene), my doubt is how can i protect  words with 
special characters from tokenizing, sat for example A+, A1+ etc. because when i 
searched for group A i am getting results with A+ aswell as A1+ and so on, is 
there any special way to index these type of words? 

Thanks,
Doss.