RE: Index fetch failed

[Akreeti Agarwal]

Hello,

Please help me with the solution for below error.

Memory details of slave server:
total   used   free sharedbuffers cached
Mem: 15947  15460487 62144   6007
-/+ buffers/cache:   9308   6639
Swap:0  0  0


Thanks & Regards,
Akreeti Agarwal

-Original Message-
From: Akreeti Agarwal  
Sent: Wednesday, August 28, 2019 2:45 PM
To: solr-user@lucene.apache.org
Subject: RE: Index fetch failed

Yes I am using solr-5.5.5.
This error is intermittent. I don't think there must be any issue with master 
connection limits. This error is accompanied by this on master side:

ERROR (qtp1450821318-60072) [   x:sitecore_web_index] 
o.a.s.h.ReplicationHandler Unable to get file names for indexCommit generation: 
1558637
java.nio.file.NoSuchFileException: 
/solrm-efs/solr-m/server/solr/sitecore_web_index/data/index/_12i9p_1.liv
   at 
sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
   at 
sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
   at 
sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
   at 
sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55)
   at 
sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:144)
   at 
sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99)
   at java.nio.file.Files.readAttributes(Files.java:1737)
   at java.nio.file.Files.size(Files.java:2332)
   at 
org.apache.lucene.store.FSDirectory.fileLength(FSDirectory.java:210)
   at 
org.apache.lucene.store.NRTCachingDirectory.fileLength(NRTCachingDirectory.java:124)
   at 
org.apache.solr.handler.ReplicationHandler.getFileList(ReplicationHandler.java:563)
   at 
org.apache.solr.handler.ReplicationHandler.handleRequestBody(ReplicationHandler.java:253)
   at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:155)
   at org.apache.solr.core.SolrCore.execute(SolrCore.java:2102)
   at 
org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:654)
   at 
org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:460)
   at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:257)
   at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:208)
   at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
   at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
   at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
   at 
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
   at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
   at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
   at 
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
   at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
   at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
   at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
   at 
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
   at 
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
   at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
   at org.eclipse.jetty.server.Server.handle(Server.java:499)
   at 
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
   at 
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
   at 
org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
   at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
   at 
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
   at java.lang.Thread.run(Thread.java:748)

Thanks & Regards,
Akreeti Agarwal

-Original Message-
From: Atita Arora 
Sent: Wednesday, August 28, 2019 2:23 PM
To: solr-user@lucene.apache.org
Subject: Re: Index fetch failed

This looks like ample memory to get the index chunk.
Also, I looked at the IndexFetcher code, I remember you were using Solr
5.5.5 and the only reason in my view, this would happen is when the index chunk 
is not downloaded as can also be seen in the error (Downloaded
0!=123) which clearly states that 

Re: ZooKeeper error in solr.log

[Shawn Heisey]

On 9/2/2019 6:04 AM, Gell-Holleron, Daniel wrote:

The error I get in the log is below:

2019-09-02 09:17:22.812 ERROR (qtp1192171522-16) [   ] 
o.a.s.h.RequestHandlerBase java.lang.ArrayIndexOutOfBoundsException: 1
at 
org.apache.solr.handler.admin.ZookeeperStatusHandler.monitorZookeeper(ZookeeperStatusHandler.java:185)





To confirm is this the setting 4lw.commands.whitelist=mntr,conf,ruok?

I am running ZooKeeper version 3.5.5





https://issues.apache.org/jira/browse/SOLR-13672


Yes, you will need the 4lw whitelist.

But since your config lists two servers, you have an ensemble, and that 
means you're going to run into SOLR-13672 after adding the whitelist. 
Solr 8.1 should work properly with ZK 3.5.5, but you won't be able to 
get ZK status in the admin UI until Solr 8.3.0, which hasn't been 
released yet.  I do not know when that will happen.


Thanks,
Shawn

Re: SolrNet...multiValued.

[Shawn Heisey]

On 9/2/2019 8:58 AM, Britto Raj wrote:

I am working MNC and doing Prototype using SolrNet and Solr. I have
few questions and got stuck not able to move forward..





4. When i try to access using
SolrQueryResults results = solr.Query(new
SolrQuery("title:\"changeme4\""));
It throws error as
could not convert value 'system.collections.arraylist' to
property 'title' of document type solr rest client.program+product'


SolrNet is third party software.  It was not created by the Solr 
project.  I have absolutely no idea what that error message means.  With 
no experience at all using .net, I can't even tell you whether that 
error message comes from the SolrNet library or from the language itself.


I found this:

https://groups.google.com/forum/embed/#!forum/solrnet

There is also an "issues" tab on the project page:

https://github.com/SolrNet/SolrNet

Thanks,
Shawn

Re: ZooKeeper error in solr.log

[David M Giannone]

Get Outlook for Android


From: Gell-Holleron, Daniel 
Sent: Monday, September 2, 2019 8:04:42 AM
To: solr-user@lucene.apache.org 
Subject: [EXTERNAL] RE: ZooKeeper error in solr.log

Hello,

The error I get in the log is below:

2019-09-02 09:17:22.812 ERROR (qtp1192171522-16) [   ] 
o.a.s.h.RequestHandlerBase java.lang.ArrayIndexOutOfBoundsException: 1
at 
org.apache.solr.handler.admin.ZookeeperStatusHandler.monitorZookeeper(ZookeeperStatusHandler.java:185)
at 
org.apache.solr.handler.admin.ZookeeperStatusHandler.getZkStatus(ZookeeperStatusHandler.java:99)
at 
org.apache.solr.handler.admin.ZookeeperStatusHandler.handleRequestBody(ZookeeperStatusHandler.java:77)
at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)
at 
org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:796)
at 
org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:762)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:522)
at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:397)
at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:343)
at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602)
at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
at 
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at 
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1588)
at 
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)
at 
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
at 
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1557)
at 
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)
at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
at 
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220)
at 
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)
at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at 
org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at org.eclipse.jetty.server.Server.handle(Server.java:502)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364)
at 
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
at 
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
at 
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
at 
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
at java.lang.Thread.run(Unknown Source)

To confirm is this the setting 4lw.commands.whitelist=mntr,conf,ruok?

I am running ZooKeeper version 3.5.5

Thanks,

Daniel


-Original Message-
From: Shawn Heisey 
Sent: 02 September 2019 12:25
To: solr-user@lucene.apache.org
Subject: Re: ZooKeeper error in solr.log

On 9/2/2019 4:03 AM, Gell-Holleron, Daniel wrote:
> I'm having trouble retrieving ZK status in SolrCloud.
>
> As far as I know, I've followed all configuration instructions.
>
> The ZK_HOST, ZK_CLIENT_TIMEOUT and SOLR_HOST settings are configured
> along with a myid file in the data directory. The zoo.cfg contains the

SolrNet...multiValued.

[Britto Raj]

Hello All,
I am working MNC and doing Prototype using SolrNet and Solr. I have
few questions and got stuck not able to move forward..

1. I have created a collection e.g Product without any field.
2. Using SolrNet.. created one Field like below.
public class Product
{

[SolrField("title")]
public string title { get; set; }
}
3. And below code is executed.. title is created as MultiValued as true.
   Startup.Init("http://localhost:8983/solr/product;);

ISolrOperations solr =
ServiceLocator.Current.GetInstance>();
// Product test = new Product() {title = new
List() { "changeme4" } };
Product test = new Product() { title =  "changeme4" };
solr.Add(test);
solr.Commit();
4. When i try to access using
   SolrQueryResults results = solr.Query(new
SolrQuery("title:\"changeme4\""));
It throws error as
   could not convert value 'system.collections.arraylist' to
property 'title' of document type solr rest client.program+product'
So i need to convert public string title { get; set; } as public
ICollection title { get; set; }
But i want to create a set of Properties in my document. I really
don't need to use ICollection and MultiValued as false. Please help me
to resolve the issue to move forward.

Re: solr.HTMLStripCharFilterFactory issue

[Big Gosh]

Thank you for your answer, very clear and precise.



On Mon, 2 Sep 2019 at 15:52, Erick Erickson  wrote:

> This is expected behavior, assuming you’re asking for your stored field as
> part of the “fl” list.
>
> The default behavior is to store the raw input and return it unaltered.
> The stored data is recorded before _any_ analysis, including charFilters.
> Otherwise it’d be surprising to see, say, the original text with all the
> accents removed (to use another CharFilter as an example).
>
> If you want the returned text to not include the markup, use an
> UpdateProcessorFactory in your update chain. These modify the input before
> the data is stored. For instance:
>
>
> https://lucene.apache.org/solr/7_6_0//solr-core/org/apache/solr/update/processor/HTMLStripFieldUpdateProcessorFactory.html
>
> It’s not obvious from the desctiption unless you follow the link to the
> superclass that you can specify one or more fields too, see:
>
> https://lucene.apache.org/solr/7_6_0//solr-core/org/apache/solr/update/processor/FieldMutatingUpdateProcessorFactory.html
>
> Best,
> Erick
>
> > On Sep 2, 2019, at 9:30 AM, Big Gosh  wrote:
> >
> > Hi,
> >
> > I've configured in solr 8.2.0 a field type as follows:
> >
> >  > positionIncrementGap="100" multiValued="true">
> >  
> >
> >
> > > words="stopwords.txt" />
> >
> >
> >  
> >  
> >
> > > words="stopwords.txt" />
> > > synonyms="synonyms.txt" ignoreCase="true" expand="true"/>
> >
> >  
> >
> >
> > I expected that the search returns the field stripped, instead HTML tags
> > are still in the field.
> >
> > Is this correct or I made a mistake in configuration
> >
> > I'm quite sure in the past I used this approach to strip html from the
> text
> >
> > Thanks in advance
>
>

Re: solr.HTMLStripCharFilterFactory issue

[Erick Erickson]

This is expected behavior, assuming you’re asking for your stored field as part 
of the “fl” list.

The default behavior is to store the raw input and return it unaltered. The 
stored data is recorded before _any_ analysis, including charFilters. Otherwise 
it’d be surprising to see, say, the original text with all the accents removed 
(to use another CharFilter as an example).

If you want the returned text to not include the markup, use an 
UpdateProcessorFactory in your update chain. These modify the input before the 
data is stored. For instance: 

https://lucene.apache.org/solr/7_6_0//solr-core/org/apache/solr/update/processor/HTMLStripFieldUpdateProcessorFactory.html

It’s not obvious from the desctiption unless you follow the link to the 
superclass that you can specify one or more fields too, see:
https://lucene.apache.org/solr/7_6_0//solr-core/org/apache/solr/update/processor/FieldMutatingUpdateProcessorFactory.html

Best,
Erick

> On Sep 2, 2019, at 9:30 AM, Big Gosh  wrote:
> 
> Hi,
> 
> I've configured in solr 8.2.0 a field type as follows:
> 
>  positionIncrementGap="100" multiValued="true">
>  
>
>
> words="stopwords.txt" />
>
>
>  
>  
>
> words="stopwords.txt" />
> synonyms="synonyms.txt" ignoreCase="true" expand="true"/>
>
>  
>
> 
> I expected that the search returns the field stripped, instead HTML tags
> are still in the field.
> 
> Is this correct or I made a mistake in configuration
> 
> I'm quite sure in the past I used this approach to strip html from the text
> 
> Thanks in advance

Re: solr.HTMLStripCharFilterFactory issue

[Erik Hatcher]

Analysis has no effect on the stored (what you get back from fl) value.   The 
html stripping is happening behind the scenes on the indexed/searchable terms. 

 Erik

> On Sep 2, 2019, at 09:30, Big Gosh  wrote:
> 
> Hi,
> 
> I've configured in solr 8.2.0 a field type as follows:
> 
>  positionIncrementGap="100" multiValued="true">
>  
>
>
> words="stopwords.txt" />
>
>
>  
>  
>
> words="stopwords.txt" />
> synonyms="synonyms.txt" ignoreCase="true" expand="true"/>
>
>  
>
> 
> I expected that the search returns the field stripped, instead HTML tags
> are still in the field.
> 
> Is this correct or I made a mistake in configuration
> 
> I'm quite sure in the past I used this approach to strip html from the text
> 
> Thanks in advance

solr.HTMLStripCharFilterFactory issue

[Big Gosh]

Hi,

I've configured in solr 8.2.0 a field type as follows:


  





  
  




  


I expected that the search returns the field stripped, instead HTML tags
are still in the field.

Is this correct or I made a mistake in configuration

I'm quite sure in the past I used this approach to strip html from the text

Thanks in advance

Re: Question: Solr perform well with thousands of replicas?

[Erick Erickson]

> why so many collection/replica: it's our customer needs, for example: each 
> database table mappings a collection.

I always cringe when I see statements like this. What this means is that your 
customer doesn’t understand search and needs guidance in the proper use of any 
search technology, Solr included.

Solr is _not_ an RDBMS. Simply mapping the DB tables onto collections will 
almost certainly result in a poor experience. Next the customer will want to 
ask Solr to do the same thing a DB does, i.e. run a join across 10 tables etc., 
which will be abysmal. Solr isn’t designed for that. Some brilliant RDBMS 
people have spent many years making DBs to what they do and do it well. 

That said, RDBMSs have poor search capabilities, they aren’t built to solve the 
search problem.

I suspect the time you spend making Solr load a thousand cores will be wasted. 
Once you do get them loaded, performance will be horrible. IMO you’d be far 
better off helping the customer define their problem so they properly model 
their search problem. This may mean that the result will be a hybrid where Solr 
is used for the free-text search and the RDBMS uses the results of the search 
to do something. Or vice versa.

FWIW
Erick

> On Sep 2, 2019, at 5:55 AM, Hongxu Ma  wrote:
> 
> Thanks @Jörn and @Erick
> I enlarged my JVM memory, so far it's stable (but used many memory).
> And I will check lower-level errors according to your suggestion if error 
> happens.
> 
> About my scenario:
> 
>  *   why so many collection/replica: it's our customer needs, for example: 
> each database table mappings a collection.
>  *   this env is just a test cluster: I want to verify the max collection 
> number solr can support stably.
> 
> 
> 
> From: Erick Erickson 
> Sent: Friday, August 30, 2019 20:05
> To: solr-user@lucene.apache.org 
> Subject: Re: Question: Solr perform well with thousands of replicas?
> 
> “no registered leader” is the effect of some problem usually, not the root 
> cause. In this case, for instance, you could be running out of file handles 
> and see other errors like “too many open files”. That’s just one example.
> 
> One common problem is that Solr needs a lot of file handles and the system 
> defaults are too low. We usually recommend you start with 65K file handles 
> (ulimit) and bump up the number of processes to 65K too.
> 
> So to throw some numbers out. With 1,000 replicas, and let’s say you have 50 
> segments in the index in each replica. Each segment consists of multiple 
> files (I’m skipping “compound files” here as an advanced topic), so each 
> segment has, let’s say, 10 segments. 1,000 * 50 * 10 would require 500,000 
> file handles on your system.
> 
> Bottom line: look for other, lower-level errors in the log to try to 
> understand what limit you’re running into.
> 
> All that said, there’ll be a number of “gotchas” when running that many 
> replicas on a particular node, I second Jörn;’s question...
> 
> Best,
> Erick
> 
>> On Aug 30, 2019, at 3:18 AM, Jörn Franke  wrote:
>> 
>> What is the reason for this number of replicas? Solr should work fine, but 
>> maybe it is worth to consolidate some collections to avoid also 
>> administrative overhead.
>> 
>>> Am 29.08.2019 um 05:27 schrieb Hongxu Ma :
>>> 
>>> Hi
>>> I have a solr-cloud cluster, but it's unstable when collection number is 
>>> big: 1000 replica/core per solr node.
>>> 
>>> To solve this issue, I have read the performance guide:
>>> https://cwiki.apache.org/confluence/display/SOLR/SolrPerformanceProblems
>>> 
>>> I noted there is a sentence on solr-cloud section:
>>> "Recent Solr versions perform well with thousands of replicas."
>>> 
>>> I want to know does it mean a single solr node can handle thousands of 
>>> replicas? or a solr cluster can (if so, what's the size of the cluster?)
>>> 
>>> My solr version is 7.3.1 and 6.6.2 (looks they are the same in performance)
>>> 
>>> Thanks for you help.
>>> 
> 

RE: ZooKeeper error in solr.log

[Gell-Holleron, Daniel]

Hello, 

The error I get in the log is below: 

2019-09-02 09:17:22.812 ERROR (qtp1192171522-16) [   ] 
o.a.s.h.RequestHandlerBase java.lang.ArrayIndexOutOfBoundsException: 1
at 
org.apache.solr.handler.admin.ZookeeperStatusHandler.monitorZookeeper(ZookeeperStatusHandler.java:185)
at 
org.apache.solr.handler.admin.ZookeeperStatusHandler.getZkStatus(ZookeeperStatusHandler.java:99)
at 
org.apache.solr.handler.admin.ZookeeperStatusHandler.handleRequestBody(ZookeeperStatusHandler.java:77)
at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)
at 
org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:796)
at 
org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:762)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:522)
at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:397)
at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:343)
at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602)
at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
at 
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at 
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1588)
at 
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)
at 
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
at 
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1557)
at 
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)
at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
at 
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220)
at 
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)
at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at 
org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at org.eclipse.jetty.server.Server.handle(Server.java:502)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364)
at 
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
at 
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
at 
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
at 
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
at java.lang.Thread.run(Unknown Source)

To confirm is this the setting 4lw.commands.whitelist=mntr,conf,ruok?

I am running ZooKeeper version 3.5.5

Thanks, 

Daniel
   

-Original Message-
From: Shawn Heisey  
Sent: 02 September 2019 12:25
To: solr-user@lucene.apache.org
Subject: Re: ZooKeeper error in solr.log

On 9/2/2019 4:03 AM, Gell-Holleron, Daniel wrote:
> I'm having trouble retrieving ZK status in SolrCloud.
> 
> As far as I know, I've followed all configuration instructions.
> 
> The ZK_HOST, ZK_CLIENT_TIMEOUT and SOLR_HOST settings are configured 
> along with a myid file in the data directory. The zoo.cfg contains the 
> following:
> 
> tickTime=2000
> dataDir=C:\\apps\\zookeeper\\data
> clientPort=2181
> initLimit=5
> syncLimit=2
> server.1=server1:2888:3888
> server.2=server2:2888:3888
> autopurge.snapRetainCount=3
> autopurge.purgeInterval=1

Re: ZooKeeper error in solr.log

[Shawn Heisey]

On 9/2/2019 4:03 AM, Gell-Holleron, Daniel wrote:

I’m having trouble retrieving ZK status in SolrCloud.

As far as I know, I’ve followed all configuration instructions.

The ZK_HOST, ZK_CLIENT_TIMEOUT and SOLR_HOST settings are configured 
along with a myid file in the data directory. The zoo.cfg contains the 
following:


tickTime=2000
dataDir=C:\\apps\\zookeeper\\data
clientPort=2181
initLimit=5
syncLimit=2
server.1=server1:2888:3888
server.2=server2:2888:3888
autopurge.snapRetainCount=3
autopurge.purgeInterval=1

I’m using Solr 8.1 for Windows, which is running on Java 8. Log file 
attached.


The log file is NOT attached.  The mailing list eats almost all 
attachments.  The ZK version of your ZK servers is not here.


If the ZK version is 3.5.x, you will need a new option in the server 
config.  But you'll still have a problem, because you have a 
multi-server ensemble, and ZK 3.5 changes the output for that in a way 
that Solr can't handle.  The second problem will be fixed in Solr 8.3 
when it is released.


https://issues.apache.org/jira/browse/SOLR-8346?focusedCommentId=1677=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-1677

https://issues.apache.org/jira/browse/SOLR-13672

Side note:  A 2-server ZK ensemble is actually LESS fault tolerant than 
a single server.  For fault tolerance, you need three ZK servers.


Thanks,
Shawn

ZooKeeper error in solr.log

[Gell-Holleron, Daniel]

Hello,

I'm having trouble retrieving ZK status in SolrCloud.

[cid:image008.png@01D5617D.76F32470]

As far as I know, I've followed all configuration instructions.

The ZK_HOST, ZK_CLIENT_TIMEOUT and SOLR_HOST settings are configured along with 
a myid file in the data directory. The zoo.cfg contains the following:

tickTime=2000
dataDir=C:\\apps\\zookeeper\\data
clientPort=2181
initLimit=5
syncLimit=2
server.1=server1:2888:3888
server.2=server2:2888:3888
autopurge.snapRetainCount=3
autopurge.purgeInterval=1

I'm using Solr 8.1 for Windows, which is running on Java 8. Log file attached.

Thanks,

Daniel

Re: Question: Solr perform well with thousands of replicas?

[Hongxu Ma]

Thanks @Jörn and @Erick
I enlarged my JVM memory, so far it's stable (but used many memory).
And I will check lower-level errors according to your suggestion if error 
happens.

About my scenario:

  *   why so many collection/replica: it's our customer needs, for example: 
each database table mappings a collection.
  *   this env is just a test cluster: I want to verify the max collection 
number solr can support stably.



From: Erick Erickson 
Sent: Friday, August 30, 2019 20:05
To: solr-user@lucene.apache.org 
Subject: Re: Question: Solr perform well with thousands of replicas?

“no registered leader” is the effect of some problem usually, not the root 
cause. In this case, for instance, you could be running out of file handles and 
see other errors like “too many open files”. That’s just one example.

One common problem is that Solr needs a lot of file handles and the system 
defaults are too low. We usually recommend you start with 65K file handles 
(ulimit) and bump up the number of processes to 65K too.

So to throw some numbers out. With 1,000 replicas, and let’s say you have 50 
segments in the index in each replica. Each segment consists of multiple files 
(I’m skipping “compound files” here as an advanced topic), so each segment has, 
let’s say, 10 segments. 1,000 * 50 * 10 would require 500,000 file handles on 
your system.

Bottom line: look for other, lower-level errors in the log to try to understand 
what limit you’re running into.

All that said, there’ll be a number of “gotchas” when running that many 
replicas on a particular node, I second Jörn;’s question...

Best,
Erick

> On Aug 30, 2019, at 3:18 AM, Jörn Franke  wrote:
>
> What is the reason for this number of replicas? Solr should work fine, but 
> maybe it is worth to consolidate some collections to avoid also 
> administrative overhead.
>
>> Am 29.08.2019 um 05:27 schrieb Hongxu Ma :
>>
>> Hi
>> I have a solr-cloud cluster, but it's unstable when collection number is 
>> big: 1000 replica/core per solr node.
>>
>> To solve this issue, I have read the performance guide:
>> https://cwiki.apache.org/confluence/display/SOLR/SolrPerformanceProblems
>>
>> I noted there is a sentence on solr-cloud section:
>> "Recent Solr versions perform well with thousands of replicas."
>>
>> I want to know does it mean a single solr node can handle thousands of 
>> replicas? or a solr cluster can (if so, what's the size of the cluster?)
>>
>> My solr version is 7.3.1 and 6.6.2 (looks they are the same in performance)
>>
>> Thanks for you help.
>>

RE: Idle Timeout while DIH indexing and implicit sharding in 7.4

[Vadim Ivanov]

Timeout causes DIH to finish with error message. So, If I check DIH response to 
be sure 
that DIH session have finished without any mistakes it causes some trouble 
:)
I haven't check yet whether all records successfully imported to solr. 
Supposed that after timeout shard does not accept records from DIH.  Am I wrong?
-- 
Vadim


> -Original Message-
> From: Mikhail Khludnev [mailto:m...@apache.org]
> Sent: Monday, September 02, 2019 12:23 PM
> To: Vadim Ivanov; solr-user
> Subject: Re: Idle Timeout while DIH indexing and implicit sharding in 7.4
> 
> It seems like reasonable behavior. SolrWriter hangs while import is
> running, it holds DistributedZkUpdateProcessor, which holds
> SolrCmdDistributor, which keep client connection to shards open, which
> causes timeout.
> It might be worked around by supplying custom SolrWriter which finishes
> UpdateProcessor's chain from time to time and recreate it. However, that
> exception shouldn't cause any problem or it does?
> Also, it's worth to track as a jira, or mentioned in the ticket regarding
> adjusting DIH for Cloud.
> 
> On Mon, Sep 2, 2019 at 9:44 AM Vadim Ivanov <
> vadim.iva...@spb.ntk-intourist.ru> wrote:
> 
> > I’ve raised that timeout from 5 min to 40 min.
> >
> > It somehow mitigated the issue in my use case.
> >
> > Problem occurs when some updates goes to one shard in the beginning of
> > long DIH session, then all updates goes to other shards for more than 300
> > sec.
> >
> > And when ( after 300 sec) some updates goes again to the first shard it
> > faces closed connection problem.
> >
> >
> >
> > Either it’s right behavior or not – it’s hard to say….
> >
> >
> >
> >
> >
> > --
> >
> > Vadim
> >
> > *From:* Mikhail Khludnev [mailto:m...@apache.org]
> > *Sent:* Monday, September 02, 2019 1:31 AM
> > *To:* solr-user
> > *Cc:* vadim.iva...@spb.ntk-intourist.ru
> > *Subject:* Re: Idle Timeout while DIH indexing and implicit sharding in
> > 7.4
> >
> >
> >
> > Giving that
> >
> > org.apache.solr.common.util.FastInputStream.peek(FastInputStream.java:60)
> > at
> >
> >
> org.apache.solr.handler.loader.JavabinLoader.parseAndLoadDocs(JavabinLoad
> er.java:107)
> >
> >
> >
> >
> > JavabinLoader hangs on Stream.peek(), awaiting -1, and hit timeout. I
> > guess it's might be related with "closing sockets". I've looked through
> > CHANGES after 6.5.1, here's what might be examined for impacting this:
> >
> >
> >
> > * SOLR-10779: JavaBinCodec should use close consistently
> >
> > * SOLR-12290: Do not close any servlet streams and improve our servlet
> > stream
> >
> > Description Resource Path Location
> > * SOLR-12477: An update would return a client error(400) if it hit a
> > AlreadyClos
> >
> > * SOLR-12897: Introduce AlreadyClosedException to clean up silly close /
> > shutd
> >
> >
> >
> > Have nothing more than that so far.
> >
> >
> >
> > On Sun, Sep 1, 2019 at 8:50 PM swapna.minnaka...@copart.com <
> > swapna.minnaka...@copart.com> wrote:
> >
> > I am facing same exact issue. We never had any issue with 6.5.1 when doing
> > full index (initial bulk load)
> > After upgrading to 7.5.0, getting below exception and indexing is taking a
> > very long time
> >
> > 2019-09-01 10:11:27.436 ERROR (qtp1650813924-22) [c:c_collection s:shard1
> > r:core_node3 x:c_collection_shard1_replica_n1] o.a.s.h.RequestHandlerBase
> > java.io.IOException: java.util.concurrent.TimeoutException: Idle timeout
> > expired: 30/30 ms
> > at
> >
> >
> org.eclipse.jetty.server.HttpInput$ErrorState.noContent(HttpInput.java:1080)
> > at org.eclipse.jetty.server.HttpInput.read(HttpInput.java:313)
> > at
> >
> >
> org.apache.solr.servlet.ServletInputStreamWrapper.read(ServletInputStream
> Wrapper.java:74)
> > at
> >
> >
> org.apache.commons.io.input.ProxyInputStream.read(ProxyInputStream.java:
> 100)
> > at
> >
> >
> org.apache.solr.common.util.FastInputStream.readWrappedStream(FastInputS
> tream.java:79)
> > at
> > org.apache.solr.common.util.FastInputStream.refill(FastInputStream.java:88)
> > at
> > org.apache.solr.common.util.FastInputStream.peek(FastInputStream.java:60)
> > at
> >
> >
> org.apache.solr.handler.loader.JavabinLoader.parseAndLoadDocs(JavabinLoad
> er.java:107)
> > at
> > org.apache.solr.handler.loader.JavabinLoader.load(JavabinLoader.java:55)
> > at
> >
> >
> org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandl
> er.java:97)
> > at
> >
> >
> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(Cont
> entStreamHandlerBase.java:68)
> > at
> >
> >
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandler
> Base.java:199)
> > at org.apache.solr.core.SolrCore.execute(SolrCore.java:2541)
> > at
> > org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:709)
> > at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:515)
> > at
> >
> >
> 

Re: Idle Timeout while DIH indexing and implicit sharding in 7.4

[Mikhail Khludnev]

It seems like reasonable behavior. SolrWriter hangs while import is
running, it holds DistributedZkUpdateProcessor, which holds
SolrCmdDistributor, which keep client connection to shards open, which
causes timeout.
It might be worked around by supplying custom SolrWriter which finishes
UpdateProcessor's chain from time to time and recreate it. However, that
exception shouldn't cause any problem or it does?
Also, it's worth to track as a jira, or mentioned in the ticket regarding
adjusting DIH for Cloud.

On Mon, Sep 2, 2019 at 9:44 AM Vadim Ivanov <
vadim.iva...@spb.ntk-intourist.ru> wrote:

> I’ve raised that timeout from 5 min to 40 min.
>
> It somehow mitigated the issue in my use case.
>
> Problem occurs when some updates goes to one shard in the beginning of
> long DIH session, then all updates goes to other shards for more than 300
> sec.
>
> And when ( after 300 sec) some updates goes again to the first shard it
> faces closed connection problem.
>
>
>
> Either it’s right behavior or not – it’s hard to say….
>
>
>
>
>
> --
>
> Vadim
>
> *From:* Mikhail Khludnev [mailto:m...@apache.org]
> *Sent:* Monday, September 02, 2019 1:31 AM
> *To:* solr-user
> *Cc:* vadim.iva...@spb.ntk-intourist.ru
> *Subject:* Re: Idle Timeout while DIH indexing and implicit sharding in
> 7.4
>
>
>
> Giving that
>
> org.apache.solr.common.util.FastInputStream.peek(FastInputStream.java:60)
> at
>
> org.apache.solr.handler.loader.JavabinLoader.parseAndLoadDocs(JavabinLoader.java:107)
>
>
>
>
> JavabinLoader hangs on Stream.peek(), awaiting -1, and hit timeout. I
> guess it's might be related with "closing sockets". I've looked through
> CHANGES after 6.5.1, here's what might be examined for impacting this:
>
>
>
> * SOLR-10779: JavaBinCodec should use close consistently
>
> * SOLR-12290: Do not close any servlet streams and improve our servlet
> stream
>
> Description Resource Path Location
> * SOLR-12477: An update would return a client error(400) if it hit a
> AlreadyClos
>
> * SOLR-12897: Introduce AlreadyClosedException to clean up silly close /
> shutd
>
>
>
> Have nothing more than that so far.
>
>
>
> On Sun, Sep 1, 2019 at 8:50 PM swapna.minnaka...@copart.com <
> swapna.minnaka...@copart.com> wrote:
>
> I am facing same exact issue. We never had any issue with 6.5.1 when doing
> full index (initial bulk load)
> After upgrading to 7.5.0, getting below exception and indexing is taking a
> very long time
>
> 2019-09-01 10:11:27.436 ERROR (qtp1650813924-22) [c:c_collection s:shard1
> r:core_node3 x:c_collection_shard1_replica_n1] o.a.s.h.RequestHandlerBase
> java.io.IOException: java.util.concurrent.TimeoutException: Idle timeout
> expired: 30/30 ms
> at
>
> org.eclipse.jetty.server.HttpInput$ErrorState.noContent(HttpInput.java:1080)
> at org.eclipse.jetty.server.HttpInput.read(HttpInput.java:313)
> at
>
> org.apache.solr.servlet.ServletInputStreamWrapper.read(ServletInputStreamWrapper.java:74)
> at
>
> org.apache.commons.io.input.ProxyInputStream.read(ProxyInputStream.java:100)
> at
>
> org.apache.solr.common.util.FastInputStream.readWrappedStream(FastInputStream.java:79)
> at
> org.apache.solr.common.util.FastInputStream.refill(FastInputStream.java:88)
> at
> org.apache.solr.common.util.FastInputStream.peek(FastInputStream.java:60)
> at
>
> org.apache.solr.handler.loader.JavabinLoader.parseAndLoadDocs(JavabinLoader.java:107)
> at
> org.apache.solr.handler.loader.JavabinLoader.load(JavabinLoader.java:55)
> at
>
> org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:97)
> at
>
> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:68)
> at
>
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2541)
> at
> org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:709)
> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:515)
> at
>
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:377)
> at
>
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:323)
> at
>
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1634)
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
> at
>
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
> at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
> at
>
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
> at
>
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
> at
>
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
> at
>
>