Re: Solr Cloud freezes during scheduled backup

2021-02-02 Thread Jason Gerlowski 
Hi Pawel,

This definitely sounds like garbage collection biting you.

Backups themselves aren't usually memory intensive, but if indexing is
going on at the same time you should expect elevated memory usage.
Essentially this is because for each core being backed up, Solr needs
to hold pieces of two different "versions" of the index in memory: the
commit-point being backed up, and the current state of the index with
the new documents.

If disabling indexing during backups is feasible that's where I'd
start in your shoes.  If it's not you might need to consider tweaks to
your heap and JVM GC settings to shorten the long individual GC pauses
you're reporting.

Good luck,

Jason

On Wed, Jan 20, 2021 at 7:00 AM Paweł Róg  wrote:
>
> Hello everyone,
> I have a nasty problem with the scheduled Solr collections backup. From
> time to time when a scheduled backup is triggered (backup operation takes
> around 10 minutes) Solr freezes for 20-30 seconds. The freeze happens on
> one Solr instance at time but this affects all queries latency (because of
> distributed queries on 6 shards). I can reproduce the problem only when
> updates in the Solr cluster are enabled. When I disable updates, the
> problem is gone.
>
> Lucene index is not big and fits into OS cache. I am wondering if taking a
> backup can be the culprit of the problem. I'm wondering if the process
> messes up operating system caches. Maybe all the files which are copied to
> NFS are eating up the OS cache and when the OS reaches high memory usage it
> starts cleaning up memory and making Solr to freeze.
>
> During the time of freeze monitoring charts are showing higher IO wait
> times. In addition to that Solr nodes which seem to be affected are
> reaching 95-100% total memory usage (used + buffers + caches).
>
> I cannot see anything valuable in GC logs apart from a message which
> suggests that the application was stopped for 20-30 seconds (Application
> time).
>
> The cluster consists of 12 machines. Each Solr is running on Ubuntu 16.04.
> All the servers are running in AWS EC2. Each Solr node is running inside
> Docker. EC2 instances have local SSD disks (but the same problem appeared
> with EBS).
>
> Does anyone have a similar problem and can share some thoughts? I'll
> appreciate all help.
>
> --
> Pawel Rog

Re: Apache Solr Reference Guide isn't accessible

2021-02-02 Thread Cassandra Targett 
Did you file an issue for this error?
On Feb 2, 2021, 1:31 AM -0600, Bernd Fehling , 
wrote:
> Yeah, but guide 8.8 is still buggy.
>
> As I reported a month ago, "ICU Normalizer 2 Filter" states:
> - NFC: ... Normalization Form C, canonical decomposition
> - NFD: ... Normalization Form D, canonical decomposition, followed by 
> canonical composition
> - NFKC: ... Normalization Form KC, compatibility decomposition
> - NFKD: ... Normalization Form KD, compatibility decomposition, followed by 
> canonical composition
>
> But the link to "Unicode Standard Annex #15" right above says:
> - NFC: ... Normalization Form C, Canonical Decomposition, followed by 
> Canonical Composition
> - NFD: ... Normalization Form D, Canonical Decomposition
> - NFKC: ... Normalization Form KC, Compatibility Decomposition, followed by 
> Canonical Composition
> - NFKD: ... Normalization Form KD, Compatibility Decomposition
>
> But, well who cares.
>
> Have a nice day.
>
>
> Am 01.02.21 um 23:04 schrieb Cassandra Targett:
> > The problem causing this has been fixed and the docs should be available 
> > again.
> > On Feb 1, 2021, 2:15 PM -0600, Alexandre Rafalovitch , 
> > wrote:
> > > And if you need something more recent while this is being fixed, you
> > > can look right at the source in GitHub, though a navigation, etc is
> > > missing:
> > > https://github.com/apache/lucene-solr/blob/master/solr/solr-ref-guide/src/analyzers.adoc
> > >
> > > Open Source :-)
> > >
> > > Regards,
> > > Alex.
> > >
> > > On Mon, 1 Feb 2021 at 15:04, Mike Drob  wrote:
> > > >
> > > > Hi Dorion,
> > > >
> > > > We are currently working with our infra team to get these restored. In 
> > > > the
> > > > meantime, the 8.4 guide is still available at
> > > > https://lucene.apache.org/solr/guide/8_4/ and are hopeful that the 8.8
> > > > guide will be back up soon. Thank you for your patience.
> > > >
> > > > Mike
> > > >
> > > > On Mon, Feb 1, 2021 at 1:58 PM Dorion Caroline 
> > > > 
> > > > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > I can't access to Apache Solr Reference Guide since few days.
> > > > > Example:
> > > > > URL
> > > > >
> > > > > * https://lucene.apache.org/solr/guide/8_8/
> > > > > * https://lucene.apache.org/solr/guide/8_7/
> > > > > Result:
> > > > > Not Found
> > > > > The requested URL was not found on this server.
> > > > >
> > > > > Do you know what going on?
> > > > >
> > > > > Thanks
> > > > > Caroline Dorion
> > > > >
> >

If replication on Master disabled, the Slaves drop ther indexes [Solr 7.7.2]

2021-02-02 Thread Mahmut Izci 

Hello,

we are using Solr 7.7.2 and sometimes we are performing a full reindex 
of a core. Therefor we stop the replication on the master 
(solr//replication?command=disablereplication),
we backup and delete the index, finally we rebuild the index and enable 
the replication again.


However, the moment we stop the replication on the master, the 
Slave-Solr's delete their index.
This behavior is also described in the issue 
https://issues.apache.org/jira/browse/SOLR-11938, but without any 
comments yet.


This seems to be a result of fixing the issue 
https://issues.apache.org/jira/browse/SOLR-11293

Because if I rollback the line in IndexFetcher.java
from "if (commit.getGeneration() != 0)"
to "if (forceReplication && commit.getGeneration() != 0)"
back, then everything works fine, it means that the slaves stop deleting 
her indexes after turning off the replication.


Is this behavior with stopping replication and loosing the index a known 
problem?

Has anyone else had the problem and knows how to solve it?
I am not sure, what will be fail, if i revert the line in 
IndexFetcher.java.


Thanks and regards
Mahmut

Re: How to get case-sensitive Terms?

2021-02-02 Thread Alexandre Rafalovitch 
It is documented in the reference guide:
https://lucene.apache.org/solr/guide/8_8/analysis-screen.html

Hope it helps,
   Alex.

On Tue, 2 Feb 2021 at 00:57, elivis  wrote:
>
> Alexandre Rafalovitch wrote
> > Admin UI also allows you to run text string against a field definition to
> > see what each stage of analyzer chain does.
>
> Thank you. Could please let me know how to do this (see what each stage of
> analyzer chain does)?
>
>
>
>
> --
> Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Solr Slack Workspace

2021-02-02 Thread Ishan Chattopadhyaya 
Hi all,
I've created an invite link for the Slack workspace:
https://s.apache.org/solr-slack.
Please test it out. I'll send a broader notification once this is tested
out to be working well.
Thanks and regards,
Ishan

On Thu, Jan 28, 2021 at 12:26 AM Justin Sweeney 
wrote:

> Thanks, I joined the Relevance Slack:
> https://opensourceconnections.com/slack, I definitely think a dedicated
> Solr workspace would also be good allowing for channels to get involved
> with development as well as user based questions.
>
> It does seem like slack has made it increasingly difficult to create open
> workspaces and not force someone to approve or only allow specific email
> domains. Has anyone tried to do that recently? I tried for an hour or so
> last weekend and it seemed to not be very straightforward anymore.
>
> On Tue, Jan 26, 2021 at 12:57 PM Houston Putman 
> wrote:
>
> > There is https://solr-dev.slack.com
> >
> > It's not really used, but it's there and we can open it up for people to
> > join and start using.
> >
> > On Tue, Jan 26, 2021 at 5:38 AM Ishan Chattopadhyaya <
> > ichattopadhy...@gmail.com> wrote:
> >
> > > Thanks ufuk. I'll take a look.
> > >
> > > On Tue, 26 Jan, 2021, 4:05 pm ufuk yılmaz,  >
> > > wrote:
> > >
> > > > It’s asking for a searchscale.com email address?
> > > >
> > > > Sent from Mail for Windows 10
> > > >
> > > > From: Ishan Chattopadhyaya
> > > > Sent: 26 January 2021 13:33
> > > > To: solr-user
> > > > Subject: Re: Solr Slack Workspace
> > > >
> > > > There is a Slack backed by official IRC support. Please see
> > > > https://lucene.472066.n3.nabble.com/Solr-Users-Slack-td4466856.html
> > for
> > > > details on how to join it.
> > > >
> > > > On Tue, 19 Jan, 2021, 2:54 pm Charlie Hull, <
> > > > ch...@opensourceconnections.com>
> > > > wrote:
> > > >
> > > > > Relevance Slack is open to anyone working on search & relevance -
> > #solr
> > > > is
> > > > > only one of the channels, there's lots more! Hope to see you there.
> > > > >
> > > > > Cheers
> > > > >
> > > > > Charlie
> > > > > https://opensourceconnections.com/slack
> > > > >
> > > > >
> > > > > On 16/01/2021 02:18, matthew sporleder wrote:
> > > > > > IRC has kind of died off,
> > > > > > https://lucene.apache.org/solr/community.html has a slack
> > mentioned,
> > > > > > I'm on https://opensourceconnections.com/slack after taking
> their
> > > solr
> > > > > > training class and assume it's mostly open to solr community.
> > > > > >
> > > > > > On Fri, Jan 15, 2021 at 8:10 PM Justin Sweeney
> > > > > >  wrote:
> > > > > >> Hi all,
> > > > > >>
> > > > > >> I did some googling and didn't find anything, but is there a
> Slack
> > > > > >> workspace for Solr? I think this could be useful to expand
> > > interaction
> > > > > >> within the community of Solr users and connect people solving
> > > similar
> > > > > >> problems.
> > > > > >>
> > > > > >> I'd be happy to get this setup if it does not exist already.
> > > > > >>
> > > > > >> Justin
> > > > >
> > > > >
> > > > > --
> > > > > Charlie Hull - Managing Consultant at OpenSource Connections
> Limited
> > > > > 
> > > > > Founding member of The Search Network <
> https://thesearchnetwork.com/
> > >
> > > > > and co-author of Searching the Enterprise
> > > > > 
> > > > > tel/fax: +44 (0)8700 118334
> > > > > mobile: +44 (0)7767 825828
> > > > >
> > > >
> > > >
> > >
> >
>

Re: NRT - Indexing

2021-02-02 Thread Dominique Bejean 
Hi,

The issue was buildOnCommit=true on a SuggestComponent.

Dominique

Le mar. 2 févr. 2021 à 00:54, Shawn Heisey  a écrit :

> On 2/1/2021 12:08 AM, haris.k...@vnc.biz wrote:
> > Hope you're doing good. I am trying to configure NRT - Indexing in my
> > project. For this reason, I have configured *autoSoftCommit* to execute
> > every second and *autoCommit* to execute every 5 minutes. Everything
> > works as expected on the dev and test server. But on the production
> > server, there are more than 6 million documents indexed in Solr, so
> > whenever a new document is indexed it takes 2-3 minutes before appearing
> > in the search despite the setting I have described above. Since the
> > target is to develop a real-time system, this delay of 2-3 minutes is
> > not acceptable. How can I reduce this time window?
>
> Setting autoSoftCommit with a max time of 1000 (one second) does not
> mean you will see changes within one second.  It means that one second
> after indexing begins, Solr will start a soft commit operation.  That
> commit operation must fully complete and the new searcher must come
> online before changes are visible.  Those steps may take much longer
> than one second, which seems to be happening on your system.
>
> With the information available, I cannot tell you why your commits are
> taking so long.  One of the most common reasons for poor Solr
> performance is a lack of free memory on the system for caching purposes.
>
> Thanks,
> Shawn
>