Re: Any blog or url that explain step by step configure grafana dashboard to monitor solr metrics

[Rajdeep Sahoo]

You can refer the below url

https://lucene.apache.org/solr/guide/7_3/monitoring-solr-with-prometheus-and-grafana.html

On Thu, 24 Sep, 2020, 8:25 am yaswanth kumar,  wrote:

> Can some one post here any blogs or url where I can get the detailed steps
> involved in configuring grafana dashboard for monitoring solr metrics??
>
> Sent from my iPhone

Re: How to remove duplicate tokens from solr

[Rajdeep Sahoo]

Hi all,
 I have found the below details in stackoverflow but not sure how to
include the jar. Can any one help with this?


I've created a new filter class from "FilteringTokenFilter". The task is
pretty simple. I would check before adding into the list.

I have created a simple plugin Eliminate duplicate words
<https://github.com/volkan/lucene-solr-filter-eliminateduplicate>

To load the plugins, JAR files (along with EliminateDuplicate-*.jar, which
can be created by executing mvn package command or
https://github.com/volkan/lucene-solr-filter-eliminateduplicate/tree/master/solr/lib)
in a lib directory in the Solr Home directory. The location for the lib
directory is near the solr.xml file.

On Fri, 18 Sep, 2020, 1:04 am Rajdeep Sahoo, 
wrote:

> But not sure why these type of search string is causing high cpu
> utilization.
>
> On Fri, 18 Sep, 2020, 12:49 am Rahul Goswami, 
> wrote:
>
>> Is this for a phrase search? If yes then the position of the token would
>> matter too and not sure which token would you want to remove. "eg
>> "tshirt hat tshirt".
>> Also, are you looking to save space and want this at index time? Or just
>> want to remove duplicates from the search string?
>>
>> If this is at search time AND is not a phrase search, there are a couple
>> approaches I could think of :
>>
>> 1) You could either handle this in the application layer to only pass the
>> deduplicated string before it hits solr
>> 2) You can write a custom search component and configure it in the
>>   list to process the search string and remove
>> duplicates
>> before it hits the default search components. See here (
>>
>> https://lucene.apache.org/solr/guide/7_7/requesthandlers-and-searchcomponents-in-solrconfig.html#first-components-and-last-components
>> ).
>>
>> However if for search, I would still evaluate if writing those extra lines
>> of code is worth the investment. I say so since my assumption is that for
>> duplicated tokens in search string, lucene would have the intelligence to
>> not fetch the doc ids again, so you should not be worried about spending
>> computation resources to reevaluate the same tokens (Someone correct me if
>> I am wrong!)
>>
>> -Rahul
>>
>> On Thu, Sep 17, 2020 at 2:56 PM Rajdeep Sahoo > >
>> wrote:
>>
>> > If someone is searching with " tshirt tshirt tshirt tshirt tshirt
>> tshirt"
>> > we need to remove the duplicates and search with tshirt.
>> >
>> >
>> > On Fri, 18 Sep, 2020, 12:19 am Alexandre Rafalovitch, <
>> arafa...@gmail.com>
>> > wrote:
>> >
>> > > This is not quite enough information.
>> > > There is
>> > >
>> >
>> https://lucene.apache.org/solr/guide/8_6/filter-descriptions.html#remove-duplicates-token-filter
>> > > but it has specific limitations.
>> > >
>> > > What is the problem that you are trying to solve that you feel is due
>> > > to duplicate tokens? Why are they duplicates? Is it about storage or
>> > > relevancy?
>> > >
>> > > Regards,
>> > >Alex.
>> > >
>> > > On Thu, 17 Sep 2020 at 14:35, Rajdeep Sahoo <
>> rajdeepsahoo2...@gmail.com>
>> > > wrote:
>> > > >
>> > > > Hi team,
>> > > >  Is there any way to remove duplicate tokens from solr. Is there any
>> > > filter
>> > > > for this.
>> > >
>> >
>>
>

Re: How to remove duplicate tokens from solr

[Rajdeep Sahoo]

But not sure why these type of search string is causing high cpu
utilization.

On Fri, 18 Sep, 2020, 12:49 am Rahul Goswami,  wrote:

> Is this for a phrase search? If yes then the position of the token would
> matter too and not sure which token would you want to remove. "eg
> "tshirt hat tshirt".
> Also, are you looking to save space and want this at index time? Or just
> want to remove duplicates from the search string?
>
> If this is at search time AND is not a phrase search, there are a couple
> approaches I could think of :
>
> 1) You could either handle this in the application layer to only pass the
> deduplicated string before it hits solr
> 2) You can write a custom search component and configure it in the
>   list to process the search string and remove duplicates
> before it hits the default search components. See here (
>
> https://lucene.apache.org/solr/guide/7_7/requesthandlers-and-searchcomponents-in-solrconfig.html#first-components-and-last-components
> ).
>
> However if for search, I would still evaluate if writing those extra lines
> of code is worth the investment. I say so since my assumption is that for
> duplicated tokens in search string, lucene would have the intelligence to
> not fetch the doc ids again, so you should not be worried about spending
> computation resources to reevaluate the same tokens (Someone correct me if
> I am wrong!)
>
> -Rahul
>
> On Thu, Sep 17, 2020 at 2:56 PM Rajdeep Sahoo 
> wrote:
>
> > If someone is searching with " tshirt tshirt tshirt tshirt tshirt tshirt"
> > we need to remove the duplicates and search with tshirt.
> >
> >
> > On Fri, 18 Sep, 2020, 12:19 am Alexandre Rafalovitch, <
> arafa...@gmail.com>
> > wrote:
> >
> > > This is not quite enough information.
> > > There is
> > >
> >
> https://lucene.apache.org/solr/guide/8_6/filter-descriptions.html#remove-duplicates-token-filter
> > > but it has specific limitations.
> > >
> > > What is the problem that you are trying to solve that you feel is due
> > > to duplicate tokens? Why are they duplicates? Is it about storage or
> > > relevancy?
> > >
> > > Regards,
> > >Alex.
> > >
> > > On Thu, 17 Sep 2020 at 14:35, Rajdeep Sahoo <
> rajdeepsahoo2...@gmail.com>
> > > wrote:
> > > >
> > > > Hi team,
> > > >  Is there any way to remove duplicate tokens from solr. Is there any
> > > filter
> > > > for this.
> > >
> >
>

Re: How to remove duplicate tokens from solr

[Rajdeep Sahoo]

If someone is searching with " tshirt tshirt tshirt tshirt tshirt tshirt"
we need to remove the duplicates and search with tshirt.


On Fri, 18 Sep, 2020, 12:19 am Alexandre Rafalovitch, 
wrote:

> This is not quite enough information.
> There is
> https://lucene.apache.org/solr/guide/8_6/filter-descriptions.html#remove-duplicates-token-filter
> but it has specific limitations.
>
> What is the problem that you are trying to solve that you feel is due
> to duplicate tokens? Why are they duplicates? Is it about storage or
> relevancy?
>
> Regards,
>    Alex.
>
> On Thu, 17 Sep 2020 at 14:35, Rajdeep Sahoo 
> wrote:
> >
> > Hi team,
> >  Is there any way to remove duplicate tokens from solr. Is there any
> filter
> > for this.
>

How to remove duplicate tokens from solr

[Rajdeep Sahoo]

Hi team,
 Is there any way to remove duplicate tokens from solr. Is there any filter
for this.

Re: Max docs and num docs are not matching after optimization

[Rajdeep Sahoo]

Please help

On Mon, 11 May, 2020, 11:51 AM Rajdeep Sahoo, 
wrote:

> Hi all,
> We are using solr 7.7.2 . After optimization the deleted docs count is
> still showing as part of max docs.
>   As per my knowledge after optimization max docs and num docs count
> should match. It is not happening here.. Is there any way to troubleshoot
> this.
>

Max docs and num docs are not matching after optimization

[Rajdeep Sahoo]

Hi all,
We are using solr 7.7.2 . After optimization the deleted docs count is
still showing as part of max docs.
  As per my knowledge after optimization max docs and num docs count should
match. It is not happening here.. Is there any way to troubleshoot this.

Solr index size has increased in solr 7.7.2

[Rajdeep Sahoo]

Hi all
We are migrating from solr 4.6 to solr 7.7.2.
In solr 4.6 the size was 2.5 gb but here in solr 7.7.2 the solr index size
is showing 6.8 gb with the same no of documents. Is it expected behavior or
any suggestions how to optimize the size.

Re: Replica is going into recovery in Solr 6.1.0

[Rajdeep Sahoo]

What is your memory configuration

On Thu, 13 Feb, 2020, 9:46 AM vishal patel, 
wrote:

> Is there anyone looking at this?
>
> Sent from Outlook
> 
> From: vishal patel 
> Sent: Wednesday, February 12, 2020 3:45 PM
> To: solr-user@lucene.apache.org 
> Subject: Replica is going into recovery in Solr 6.1.0
>
> I am using solr version 6.1.0, Java 8 version and G1gc on production. We
> have 2 shards and each shard has 1 replica. Suddenly one replica is going
> into recovery mode and Requests become slow in our production.
> I have analyzed that minor GC max pause time was 1 min 6 sec 800 ms on
> that time and also multiple times minor GC pauses.
>
> My logs :
>
> https://drive.google.com/file/d/158z3nzLsnHGouyRnXgfzCjwD4iadgKSp/view?usp=sharing
>
> https://drive.google.com/file/d/1E4jyffvIWVJB7EeEMXBXyqaK2ZfAA8kk/view?usp=sharing
>
> I do not know why long GC pause time happened. In our platform heavy
> searching and indexing is performed.
> long GC pause times happen due to searching or indexing?
> If GC pause time long then why replica is going into recovery? can we set
> the waiting time of update request?
> what is the minimum GC pause time for going into recovery mode?
>
> It is useful for my problem? :
> https://issues.apache.org/jira/browse/SOLR-9310
>
> Regards,
> Vishal Patel
>
> Sent from Outlook
>

Re: Solr 7.7 heap space is getting full

[Rajdeep Sahoo]

If we reduce the no of threads then is it going to help.
  Is there any other way to debug this.


On Mon, 3 Feb, 2020, 2:52 AM Walter Underwood, 
wrote:

> The only time I’ve ever had an OOM is when Solr gets a huge load
> spike and fires up 2000 threads. Then it runs out of space for stacks.
>
> I’ve never run anything other than an 8GB heap, starting with Solr 1.3
> at Netflix.
>
> Agreed about filter cache, though I’d expect heavy use of that to most
> often be part of a faceted search system.
>
> wunder
> Walter Underwood
> wun...@wunderwood.org
> http://observer.wunderwood.org/  (my blog)
>
> > On Feb 2, 2020, at 12:36 PM, Erick Erickson 
> wrote:
> >
> > Mostly I was reacting to the statement that the number
> > of docs increased by over 4x and then there were
> > memory problems.
> >
> > Hmmm, that said, what does “heap space is getting full”
> > mean anyway? If you’re hitting OOMs, that’s one thing. If
> > you’re measuring the amount of heap consumed and
> > noticing that it fills up, that’s totally normal. Java will
> > collect garbage when it needs to. If you attach something
> > like jconsole to Solr you’ll see memory grow and shrink
> > quite regularly. Take a look at your garbage collection logs
> > with something like GCViewer to see how much memory is
> > still required after a GC cycle. If that number is reasonable
> > then there’s no problem.
> >
> > Walter:
> >
> > Well, the expectation that one can keep adding docs without
> > considering heap size is simply naive. The filterCache
> > for instance grows linearly with the number of documents
> > (OK, if it it stores the full bitset). Real Time Get requires
> > on-heap structures to keep track of changed docs between
> > commits. Etc.
> >
> > The OP hasn’t even told us whether docValues are enabled
> > appropriately, which if not set for fields needing it will also
> > grow heap requirements linearly with the number of docs.
> >
> > I’ll totally agree that the relationship between the size of
> > the index on disk and heap is iffy at best. But if more heap is
> > _not_ needed for bigger indexes then we’d never hit OOMs
> > no matter how many docs we put in 4G.
> >
> > Best,
> > Erick
> >
> >
> >
> >> On Feb 2, 2020, at 11:18 AM, Walter Underwood 
> wrote:
> >>
> >> We CANNOT diagnose anything until you tell us the error message!
> >>
> >> Erick, I strongly disagree that more heap is needed for bigger indexes.
> >> Except for faceting, Lucene was designed to stream index data and
> >> work regardless of the size of the index. Indexing is in RAM buffer
> >> sized chunks, so large updates also don’t need extra RAM.
> >>
> >> wunder
> >> Walter Underwood
> >> wun...@wunderwood.org
> >> http://observer.wunderwood.org/  (my blog)
> >>
> >>> On Feb 2, 2020, at 7:52 AM, Rajdeep Sahoo 
> wrote:
> >>>
> >>> We have allocated 16 gb of heap space  out of 24 g.
> >>> There are 3 solr cores here, for one core when the no of documents are
> >>> getting increased i.e. around 4.5 lakhs,then this scenario is
> happening.
> >>>
> >>>
> >>> On Sun, 2 Feb, 2020, 9:02 PM Erick Erickson, 
> >>> wrote:
> >>>
> >>>> Allocate more heap and possibly add more RAM.
> >>>>
> >>>> What are you expectations? You can't continue to
> >>>> add documents to your Solr instance without regard to
> >>>> how much heap you’ve allocated. You’ve put over 4x
> >>>> the number of docs on the node. There’s no magic here.
> >>>> You can’t continue to add docs to a Solr instance without
> >>>> increasing the heap at some point.
> >>>>
> >>>> And as far as I know, you’ve never told us how much heap yo
> >>>> _are_ allocating. The default for Java processes is 512M, which
> >>>> is quite small. so perhaps it’s a simple matter of starting Solr
> >>>> with the -XmX parameter set to something larger.
> >>>>
> >>>> Best,
> >>>> Erick
> >>>>
> >>>>> On Feb 2, 2020, at 10:19 AM, Rajdeep Sahoo <
> rajdeepsahoo2...@gmail.com>
> >>>> wrote:
> >>>>>
> >>>>> What can we do in this scenario as the solr master node is going
> down and
> >>>>> the indexing is failing.
> >>>>> Please provide some wo

Re: Solr 7.7 heap space is getting full

[Rajdeep Sahoo]

We have allocated 16 gb of heap space  out of 24 g.
   There are 3 solr cores here, for one core when the no of documents are
getting increased i.e. around 4.5 lakhs,then this scenario is happening.


On Sun, 2 Feb, 2020, 9:02 PM Erick Erickson, 
wrote:

> Allocate more heap and possibly add more RAM.
>
> What are you expectations? You can't continue to
> add documents to your Solr instance without regard to
> how much heap you’ve allocated. You’ve put over 4x
> the number of docs on the node. There’s no magic here.
> You can’t continue to add docs to a Solr instance without
> increasing the heap at some point.
>
> And as far as I know, you’ve never told us how much heap yo
>  _are_ allocating. The default for Java processes is 512M, which
> is quite small. so perhaps it’s a simple matter of starting Solr
> with the -XmX parameter set to something larger.
>
> Best,
> Erick
>
> > On Feb 2, 2020, at 10:19 AM, Rajdeep Sahoo 
> wrote:
> >
> > What can we do in this scenario as the solr master node is going down and
> > the indexing is failing.
> > Please provide some workaround for this issue.
> >
> > On Sat, 1 Feb, 2020, 11:51 PM Walter Underwood, 
> > wrote:
> >
> >> What message do you get about the heap space.
> >>
> >> It is completely normal for Java to use all of heap before running a
> major
> >> GC. That
> >> is how the JVM works.
> >>
> >> wunder
> >> Walter Underwood
> >> wun...@wunderwood.org
> >> http://observer.wunderwood.org/  (my blog)
> >>
> >>> On Feb 1, 2020, at 6:35 AM, Rajdeep Sahoo 
> >> wrote:
> >>>
> >>> Please reply anyone
> >>>
> >>> On Fri, 31 Jan, 2020, 11:37 PM Rajdeep Sahoo, <
> >> rajdeepsahoo2...@gmail.com>
> >>> wrote:
> >>>
> >>>> This is happening when the no of indexed document count is increasing.
> >>>>  With 1 million docs it's working fine but when it's crossing 4.5
> >>>> million it's heap space is getting full.
> >>>>
> >>>>
> >>>> On Wed, 22 Jan, 2020, 7:05 PM Michael Gibney, <
> >> mich...@michaelgibney.net>
> >>>> wrote:
> >>>>
> >>>>> Rajdeep, you say that "suddenly" heap space is getting full ... does
> >>>>> this mean that some variant of this configuration was working for you
> >>>>> at some point, or just that the failure happens quickly?
> >>>>>
> >>>>> If heap space and faceting are indeed the bottleneck, you might make
> >>>>> sure that you have docValues enabled for your facet field fieldTypes,
> >>>>> and perhaps set uninvertible=false.
> >>>>>
> >>>>> I'm not seeing where large numbers of facets initially came from in
> >>>>> this thread? But on that topic this is perhaps relevant, regarding
> the
> >>>>> potential utility of a facet cache:
> >>>>> https://issues.apache.org/jira/browse/SOLR-13807
> >>>>>
> >>>>> Michael
> >>>>>
> >>>>> On Wed, Jan 22, 2020 at 7:16 AM Toke Eskildsen  wrote:
> >>>>>>
> >>>>>> On Sun, 2020-01-19 at 21:19 -0500, Mehai, Lotfi wrote:
> >>>>>>> I  had a similar issue with a large number of facets. There is no
> way
> >>>>>>> (At least I know) your can get an acceptable response time from
> >>>>>>> search engine with high number of facets.
> >>>>>>
> >>>>>> Just for the record then it is doable under specific circumstances
> >>>>>> (static single-shard index, only String fields, Solr 4 with patch,
> >>>>>> fixed list of facet fields):
> >>>>>> https://sbdevel.wordpress.com/2013/03/20/over-9000-facet-fields/
> >>>>>>
> >>>>>> More usable for the current case would be to play with facet.threads
> >>>>>> and throw hardware with many CPU-cores after the problem.
> >>>>>>
> >>>>>> - Toke Eskildsen, Royal Danish Library
> >>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>
> >>
>
>

Re: Solr 7.7 heap space is getting full

[Rajdeep Sahoo]

What can we do in this scenario as the solr master node is going down and
the indexing is failing.
 Please provide some workaround for this issue.

On Sat, 1 Feb, 2020, 11:51 PM Walter Underwood, 
wrote:

> What message do you get about the heap space.
>
> It is completely normal for Java to use all of heap before running a major
> GC. That
> is how the JVM works.
>
> wunder
> Walter Underwood
> wun...@wunderwood.org
> http://observer.wunderwood.org/  (my blog)
>
> > On Feb 1, 2020, at 6:35 AM, Rajdeep Sahoo 
> wrote:
> >
> > Please reply anyone
> >
> > On Fri, 31 Jan, 2020, 11:37 PM Rajdeep Sahoo, <
> rajdeepsahoo2...@gmail.com>
> > wrote:
> >
> >> This is happening when the no of indexed document count is increasing.
> >>   With 1 million docs it's working fine but when it's crossing 4.5
> >> million it's heap space is getting full.
> >>
> >>
> >> On Wed, 22 Jan, 2020, 7:05 PM Michael Gibney, <
> mich...@michaelgibney.net>
> >> wrote:
> >>
> >>> Rajdeep, you say that "suddenly" heap space is getting full ... does
> >>> this mean that some variant of this configuration was working for you
> >>> at some point, or just that the failure happens quickly?
> >>>
> >>> If heap space and faceting are indeed the bottleneck, you might make
> >>> sure that you have docValues enabled for your facet field fieldTypes,
> >>> and perhaps set uninvertible=false.
> >>>
> >>> I'm not seeing where large numbers of facets initially came from in
> >>> this thread? But on that topic this is perhaps relevant, regarding the
> >>> potential utility of a facet cache:
> >>> https://issues.apache.org/jira/browse/SOLR-13807
> >>>
> >>> Michael
> >>>
> >>> On Wed, Jan 22, 2020 at 7:16 AM Toke Eskildsen  wrote:
> >>>>
> >>>> On Sun, 2020-01-19 at 21:19 -0500, Mehai, Lotfi wrote:
> >>>>> I  had a similar issue with a large number of facets. There is no way
> >>>>> (At least I know) your can get an acceptable response time from
> >>>>> search engine with high number of facets.
> >>>>
> >>>> Just for the record then it is doable under specific circumstances
> >>>> (static single-shard index, only String fields, Solr 4 with patch,
> >>>> fixed list of facet fields):
> >>>> https://sbdevel.wordpress.com/2013/03/20/over-9000-facet-fields/
> >>>>
> >>>> More usable for the current case would be to play with facet.threads
> >>>> and throw hardware with many CPU-cores after the problem.
> >>>>
> >>>> - Toke Eskildsen, Royal Danish Library
> >>>>
> >>>>
> >>>
> >>
>
>

Re: Solr 7.7 heap space is getting full

[Rajdeep Sahoo]

Please reply anyone

On Fri, 31 Jan, 2020, 11:37 PM Rajdeep Sahoo, 
wrote:

> This is happening when the no of indexed document count is increasing.
>With 1 million docs it's working fine but when it's crossing 4.5
> million it's heap space is getting full.
>
>
> On Wed, 22 Jan, 2020, 7:05 PM Michael Gibney, 
> wrote:
>
>> Rajdeep, you say that "suddenly" heap space is getting full ... does
>> this mean that some variant of this configuration was working for you
>> at some point, or just that the failure happens quickly?
>>
>> If heap space and faceting are indeed the bottleneck, you might make
>> sure that you have docValues enabled for your facet field fieldTypes,
>> and perhaps set uninvertible=false.
>>
>> I'm not seeing where large numbers of facets initially came from in
>> this thread? But on that topic this is perhaps relevant, regarding the
>> potential utility of a facet cache:
>> https://issues.apache.org/jira/browse/SOLR-13807
>>
>> Michael
>>
>> On Wed, Jan 22, 2020 at 7:16 AM Toke Eskildsen  wrote:
>> >
>> > On Sun, 2020-01-19 at 21:19 -0500, Mehai, Lotfi wrote:
>> > > I  had a similar issue with a large number of facets. There is no way
>> > > (At least I know) your can get an acceptable response time from
>> > > search engine with high number of facets.
>> >
>> > Just for the record then it is doable under specific circumstances
>> > (static single-shard index, only String fields, Solr 4 with patch,
>> > fixed list of facet fields):
>> > https://sbdevel.wordpress.com/2013/03/20/over-9000-facet-fields/
>> >
>> > More usable for the current case would be to play with facet.threads
>> > and throw hardware with many CPU-cores after the problem.
>> >
>> > - Toke Eskildsen, Royal Danish Library
>> >
>> >
>>
>

Re: Solr 7.7 heap space is getting full

[Rajdeep Sahoo]

This is happening when the no of indexed document count is increasing.
   With 1 million docs it's working fine but when it's crossing 4.5 million
it's heap space is getting full.


On Wed, 22 Jan, 2020, 7:05 PM Michael Gibney, 
wrote:

> Rajdeep, you say that "suddenly" heap space is getting full ... does
> this mean that some variant of this configuration was working for you
> at some point, or just that the failure happens quickly?
>
> If heap space and faceting are indeed the bottleneck, you might make
> sure that you have docValues enabled for your facet field fieldTypes,
> and perhaps set uninvertible=false.
>
> I'm not seeing where large numbers of facets initially came from in
> this thread? But on that topic this is perhaps relevant, regarding the
> potential utility of a facet cache:
> https://issues.apache.org/jira/browse/SOLR-13807
>
> Michael
>
> On Wed, Jan 22, 2020 at 7:16 AM Toke Eskildsen  wrote:
> >
> > On Sun, 2020-01-19 at 21:19 -0500, Mehai, Lotfi wrote:
> > > I  had a similar issue with a large number of facets. There is no way
> > > (At least I know) your can get an acceptable response time from
> > > search engine with high number of facets.
> >
> > Just for the record then it is doable under specific circumstances
> > (static single-shard index, only String fields, Solr 4 with patch,
> > fixed list of facet fields):
> > https://sbdevel.wordpress.com/2013/03/20/over-9000-facet-fields/
> >
> > More usable for the current case would be to play with facet.threads
> > and throw hardware with many CPU-cores after the problem.
> >
> > - Toke Eskildsen, Royal Danish Library
> >
> >
>

Re: Solr cloud production set up

[Rajdeep Sahoo]

Please suggest anyone

On Sun, 19 Jan, 2020, 9:43 AM Rajdeep Sahoo, 
wrote:

> Apart from reducing no of facets in the query, is there any other query
> params or gc params or heap space or anything else that we need to tweak
> for improving search response time.
>
> On Sun, 19 Jan, 2020, 3:15 AM Erick Erickson, 
> wrote:
>
>> Add &debug=timing to the query and it’ll show you the time each component
>> takes.
>>
>> > On Jan 18, 2020, at 1:50 PM, Rajdeep Sahoo 
>> wrote:
>> >
>> > Thanks for the suggestion,
>> >
>> > Is there any way to get the info which operation or which query params
>> are
>> > increasing the response time.
>> >
>> >
>> > On Sat, 18 Jan, 2020, 11:59 PM Dave, 
>> wrote:
>> >
>> >> If you’re not getting values, don’t ask for the facet. Facets are
>> >> expensive as hell, maybe you should think more about your query’s than
>> your
>> >> infrastructure, solr cloud won’t help you at all especially if your
>> asking
>> >> for things you don’t need
>> >>
>> >>> On Jan 18, 2020, at 1:25 PM, Rajdeep Sahoo <
>> rajdeepsahoo2...@gmail.com>
>> >> wrote:
>> >>>
>> >>> We have assigned 16 gb out of 24gb for heap .
>> >>> No other process is running on that node.
>> >>>
>> >>> 200 facets fields are there in the query but we will not be getting
>> the
>> >>> values for each facets for every search.
>> >>> There can be max of 50-60 facets for which we will be getting values.
>> >>>
>> >>> We are using caching,is it not going to help.
>> >>>
>> >>>
>> >>>
>> >>>> On Sat, 18 Jan, 2020, 11:36 PM Shawn Heisey, 
>> >> wrote:
>> >>>>
>> >>>>> On 1/18/2020 10:09 AM, Rajdeep Sahoo wrote:
>> >>>>> We are having 2.3 million documents and size is 2.5 gb.
>> >>>>>  10 core cpu and 24 gb ram . 16 slave nodes.
>> >>>>>
>> >>>>>  Still some of the queries are taking 50 sec at solr end.
>> >>>>> As we are using solr 4.6 .
>> >>>>>  Other thing is we are having 200 (avg) facet fields  in a query.
>> >>>>> And 30 searchable fields.
>> >>>>> Is there any way to identify why it is taking 50 sec for a query.
>> >>>>>Multiple concurrent requests are there.
>> >>>>
>> >>>> Searching 30 fields and computing 200 facets is never going to be
>> super
>> >>>> fast.  Switching to cloud will not help, and might make it slower.
>> >>>>
>> >>>> Your index is pretty small to a lot of us.  There are people running
>> >>>> indexes with billions of documents that take terabytes of disk space.
>> >>>>
>> >>>> As Walter mentioned, computing 200 facets is going to require a fair
>> >>>> amount of heap memory.  One *possible* problem here is that the Solr
>> >>>> heap size is too small, so a lot of GC is required.  How much of the
>> >>>> 24GB have you assigned to the heap?  Is there any software other than
>> >>>> Solr running on these nodes?
>> >>>>
>> >>>> Thanks,
>> >>>> Shawn
>> >>>>
>> >>
>>
>>

Re: Solr 7.7 heap space is getting full

[Rajdeep Sahoo]

The problem is happening for one index, for other two indexes it is working
fine.
  For other two indexes indexing and search both are working fine.
  But for one index after indexing completion the heap space is getting
full and solr is not responding at all.
  Index sizes are almost same,its around 1gb each.
  When loading the core by selecting it,the jvm space us becoming full.


On Mon, 20 Jan, 2020, 9:26 AM Rajdeep Sahoo, 
wrote:

> Anything else regarding gc tuning.
>
> On Mon, 20 Jan, 2020, 8:08 AM Rajdeep Sahoo, 
> wrote:
>
>> Initially we were getting the warning message as  ulimit is low i.e. 1024
>> so we changed it to 65000
>> Using ulimit -u 65000.
>>
>> Then the error was failed to reserve shared memory error =1
>>  Because of this we removed
>>-xx : +uselargepages
>>
>> Now in console log it is showing
>> Could not find or load main class \
>>
>> And solr is not starting up
>>
>>
>> On Mon, 20 Jan, 2020, 7:50 AM Mehai, Lotfi, 
>> wrote:
>>
>>> I  had a similar issue with a large number of facets. There is no way (At
>>> least I know) your can get an acceptable response time from search engine
>>> with high number of facets.
>>> The way we solved the issue was to cache shallow Facets data structure in
>>> the web services. Facts structures are refreshed periodically. We don't
>>> have near real time indexation requirements. Page response time is under
>>> 5s.
>>>
>>> Here the URLs for our worst use case:
>>> https://www.govinfo.gov/app/collection/cfr
>>> https://www.govinfo.gov/app/cfrparts/month
>>>
>>> I hope that helps.
>>>
>>> Lotfi Mehai
>>> https://www.linkedin.com/in/lmehai/
>>>
>>>
>>>
>>>
>>>
>>> On Sun, Jan 19, 2020 at 9:05 PM Rajdeep Sahoo <
>>> rajdeepsahoo2...@gmail.com>
>>> wrote:
>>>
>>> > Initially we were getting the warning message as  ulimit is low i.e.
>>> 1024
>>> > so we changed it to 65000
>>> > Using ulimit -u 65000.
>>> >
>>> > Then the error was failed to reserve shared memory error =1
>>> >  Because of this we removed
>>> >-xx : +uselargepages
>>> >
>>> > Now in console log it is showing
>>> > Could not find or load main class \
>>> >
>>> > And solr is not starting up
>>> >
>>> >
>>> >
>>> > On Mon, 20 Jan, 2020, 7:20 AM Walter Underwood, >> >
>>> > wrote:
>>> >
>>> > > What message do you get that means the heap space is full?
>>> > >
>>> > > Java will always use all of the heap, either as live data or
>>> > > not-yet-collected garbage.
>>> > >
>>> > > wunder
>>> > > Walter Underwood
>>> > > wun...@wunderwood.org
>>> > > http://observer.wunderwood.org/  (my blog)
>>> > >
>>> > > > On Jan 19, 2020, at 5:47 PM, Rajdeep Sahoo <
>>> rajdeepsahoo2...@gmail.com
>>> > >
>>> > > wrote:
>>> > > >
>>> > > > Hi,
>>> > > > Currently there is no request or indexing is happening.
>>> > > >  It's just start up
>>> > > > And during that time heap is getting full.
>>> > > > Index size is approx 1 g.
>>> > > >
>>> > > >
>>> > > > On Mon, 20 Jan, 2020, 7:01 AM Walter Underwood, <
>>> wun...@wunderwood.org
>>> > >
>>> > > > wrote:
>>> > > >
>>> > > >> A new garbage collector won’t fix it, but it might help a bit.
>>> > > >>
>>> > > >> Requesting 200 facet fields and having 50-60 of them with results
>>> is a
>>> > > >> huge amount of work for Solr. A typical faceting implementation
>>> might
>>> > > have
>>> > > >> three to five facets. Your requests will be at least 10X to 20X
>>> > slower.
>>> > > >>
>>> > > >> Check the CPU during one request. It should use nearly 100% of a
>>> > single
>>> > > >> CPU. If it a lot lower than 100%, you have another bottleneck.
>>> That
>>> > > might
>>> > > >> be insufficient heap or accessing disk during query requests (not

Solr console showing error in 7 .7

[Rajdeep Sahoo]

When reloading the solr console,it is showing some error in the console
itself for some small amount of time.
The error is error reloading/initialising the core.

Re: Solr 7.7 heap space is getting full

[Rajdeep Sahoo]

Anything else regarding gc tuning.

On Mon, 20 Jan, 2020, 8:08 AM Rajdeep Sahoo, 
wrote:

> Initially we were getting the warning message as  ulimit is low i.e. 1024
> so we changed it to 65000
> Using ulimit -u 65000.
>
> Then the error was failed to reserve shared memory error =1
>  Because of this we removed
>-xx : +uselargepages
>
> Now in console log it is showing
> Could not find or load main class \
>
> And solr is not starting up
>
>
> On Mon, 20 Jan, 2020, 7:50 AM Mehai, Lotfi, 
> wrote:
>
>> I  had a similar issue with a large number of facets. There is no way (At
>> least I know) your can get an acceptable response time from search engine
>> with high number of facets.
>> The way we solved the issue was to cache shallow Facets data structure in
>> the web services. Facts structures are refreshed periodically. We don't
>> have near real time indexation requirements. Page response time is under
>> 5s.
>>
>> Here the URLs for our worst use case:
>> https://www.govinfo.gov/app/collection/cfr
>> https://www.govinfo.gov/app/cfrparts/month
>>
>> I hope that helps.
>>
>> Lotfi Mehai
>> https://www.linkedin.com/in/lmehai/
>>
>>
>>
>>
>>
>> On Sun, Jan 19, 2020 at 9:05 PM Rajdeep Sahoo > >
>> wrote:
>>
>> > Initially we were getting the warning message as  ulimit is low i.e.
>> 1024
>> > so we changed it to 65000
>> > Using ulimit -u 65000.
>> >
>> > Then the error was failed to reserve shared memory error =1
>> >  Because of this we removed
>> >-xx : +uselargepages
>> >
>> > Now in console log it is showing
>> > Could not find or load main class \
>> >
>> > And solr is not starting up
>> >
>> >
>> >
>> > On Mon, 20 Jan, 2020, 7:20 AM Walter Underwood, 
>> > wrote:
>> >
>> > > What message do you get that means the heap space is full?
>> > >
>> > > Java will always use all of the heap, either as live data or
>> > > not-yet-collected garbage.
>> > >
>> > > wunder
>> > > Walter Underwood
>> > > wun...@wunderwood.org
>> > > http://observer.wunderwood.org/  (my blog)
>> > >
>> > > > On Jan 19, 2020, at 5:47 PM, Rajdeep Sahoo <
>> rajdeepsahoo2...@gmail.com
>> > >
>> > > wrote:
>> > > >
>> > > > Hi,
>> > > > Currently there is no request or indexing is happening.
>> > > >  It's just start up
>> > > > And during that time heap is getting full.
>> > > > Index size is approx 1 g.
>> > > >
>> > > >
>> > > > On Mon, 20 Jan, 2020, 7:01 AM Walter Underwood, <
>> wun...@wunderwood.org
>> > >
>> > > > wrote:
>> > > >
>> > > >> A new garbage collector won’t fix it, but it might help a bit.
>> > > >>
>> > > >> Requesting 200 facet fields and having 50-60 of them with results
>> is a
>> > > >> huge amount of work for Solr. A typical faceting implementation
>> might
>> > > have
>> > > >> three to five facets. Your requests will be at least 10X to 20X
>> > slower.
>> > > >>
>> > > >> Check the CPU during one request. It should use nearly 100% of a
>> > single
>> > > >> CPU. If it a lot lower than 100%, you have another bottleneck. That
>> > > might
>> > > >> be insufficient heap or accessing disk during query requests (not
>> > enough
>> > > >> RAM). If it is near 100%, the only thing you can do is get a faster
>> > CPU.
>> > > >>
>> > > >> One other question, how frequently is the index updated?
>> > > >>
>> > > >> wunder
>> > > >> Walter Underwood
>> > > >> wun...@wunderwood.org
>> > > >> http://observer.wunderwood.org/  (my blog)
>> > > >>
>> > > >>> On Jan 19, 2020, at 4:49 PM, Rajdeep Sahoo <
>> > rajdeepsahoo2...@gmail.com
>> > > >
>> > > >> wrote:
>> > > >>>
>> > > >>> Hi,
>> > > >>> Still facing the same issue...
>> > > >>> Anything else that we need to check.
>> > > >>>
>> > > >>>
>> > > >&

Re: Solr 7.7 heap space is getting full

[Rajdeep Sahoo]

Initially we were getting the warning message as  ulimit is low i.e. 1024
so we changed it to 65000
Using ulimit -u 65000.

Then the error was failed to reserve shared memory error =1
 Because of this we removed
   -xx : +uselargepages

Now in console log it is showing
Could not find or load main class \

And solr is not starting up


On Mon, 20 Jan, 2020, 7:50 AM Mehai, Lotfi,  wrote:

> I  had a similar issue with a large number of facets. There is no way (At
> least I know) your can get an acceptable response time from search engine
> with high number of facets.
> The way we solved the issue was to cache shallow Facets data structure in
> the web services. Facts structures are refreshed periodically. We don't
> have near real time indexation requirements. Page response time is under
> 5s.
>
> Here the URLs for our worst use case:
> https://www.govinfo.gov/app/collection/cfr
> https://www.govinfo.gov/app/cfrparts/month
>
> I hope that helps.
>
> Lotfi Mehai
> https://www.linkedin.com/in/lmehai/
>
>
>
>
>
> On Sun, Jan 19, 2020 at 9:05 PM Rajdeep Sahoo 
> wrote:
>
> > Initially we were getting the warning message as  ulimit is low i.e. 1024
> > so we changed it to 65000
> > Using ulimit -u 65000.
> >
> > Then the error was failed to reserve shared memory error =1
> >  Because of this we removed
> >-xx : +uselargepages
> >
> > Now in console log it is showing
> > Could not find or load main class \
> >
> > And solr is not starting up
> >
> >
> >
> > On Mon, 20 Jan, 2020, 7:20 AM Walter Underwood, 
> > wrote:
> >
> > > What message do you get that means the heap space is full?
> > >
> > > Java will always use all of the heap, either as live data or
> > > not-yet-collected garbage.
> > >
> > > wunder
> > > Walter Underwood
> > > wun...@wunderwood.org
> > > http://observer.wunderwood.org/  (my blog)
> > >
> > > > On Jan 19, 2020, at 5:47 PM, Rajdeep Sahoo <
> rajdeepsahoo2...@gmail.com
> > >
> > > wrote:
> > > >
> > > > Hi,
> > > > Currently there is no request or indexing is happening.
> > > >  It's just start up
> > > > And during that time heap is getting full.
> > > > Index size is approx 1 g.
> > > >
> > > >
> > > > On Mon, 20 Jan, 2020, 7:01 AM Walter Underwood, <
> wun...@wunderwood.org
> > >
> > > > wrote:
> > > >
> > > >> A new garbage collector won’t fix it, but it might help a bit.
> > > >>
> > > >> Requesting 200 facet fields and having 50-60 of them with results
> is a
> > > >> huge amount of work for Solr. A typical faceting implementation
> might
> > > have
> > > >> three to five facets. Your requests will be at least 10X to 20X
> > slower.
> > > >>
> > > >> Check the CPU during one request. It should use nearly 100% of a
> > single
> > > >> CPU. If it a lot lower than 100%, you have another bottleneck. That
> > > might
> > > >> be insufficient heap or accessing disk during query requests (not
> > enough
> > > >> RAM). If it is near 100%, the only thing you can do is get a faster
> > CPU.
> > > >>
> > > >> One other question, how frequently is the index updated?
> > > >>
> > > >> wunder
> > > >> Walter Underwood
> > > >> wun...@wunderwood.org
> > > >> http://observer.wunderwood.org/  (my blog)
> > > >>
> > > >>> On Jan 19, 2020, at 4:49 PM, Rajdeep Sahoo <
> > rajdeepsahoo2...@gmail.com
> > > >
> > > >> wrote:
> > > >>>
> > > >>> Hi,
> > > >>> Still facing the same issue...
> > > >>> Anything else that we need to check.
> > > >>>
> > > >>>
> > > >>> On Mon, 20 Jan, 2020, 4:07 AM Walter Underwood, <
> > wun...@wunderwood.org
> > > >
> > > >>> wrote:
> > > >>>
> > > >>>> With Java 1.8, I would use the G1 garbage collector. We’ve been
> > > running
> > > >>>> that combination in prod for three years with no problems.
> > > >>>>
> > > >>>> SOLR_HEAP=8g
> > > >>>> # Use G1 GC  -- wunder 2017-01-23
> > > >>>> # Settings from https://wiki.apache.org/solr/ShawnHeisey

Re: Solr 7.7 heap space is getting full

[Rajdeep Sahoo]

Initially we were getting the warning message as  ulimit is low i.e. 1024
so we changed it to 65000
Using ulimit -u 65000.

Then the error was failed to reserve shared memory error =1
 Because of this we removed
   -xx : +uselargepages

Now in console log it is showing
Could not find or load main class \

And solr is not starting up



On Mon, 20 Jan, 2020, 7:20 AM Walter Underwood, 
wrote:

> What message do you get that means the heap space is full?
>
> Java will always use all of the heap, either as live data or
> not-yet-collected garbage.
>
> wunder
> Walter Underwood
> wun...@wunderwood.org
> http://observer.wunderwood.org/  (my blog)
>
> > On Jan 19, 2020, at 5:47 PM, Rajdeep Sahoo 
> wrote:
> >
> > Hi,
> > Currently there is no request or indexing is happening.
> >  It's just start up
> > And during that time heap is getting full.
> > Index size is approx 1 g.
> >
> >
> > On Mon, 20 Jan, 2020, 7:01 AM Walter Underwood, 
> > wrote:
> >
> >> A new garbage collector won’t fix it, but it might help a bit.
> >>
> >> Requesting 200 facet fields and having 50-60 of them with results is a
> >> huge amount of work for Solr. A typical faceting implementation might
> have
> >> three to five facets. Your requests will be at least 10X to 20X slower.
> >>
> >> Check the CPU during one request. It should use nearly 100% of a single
> >> CPU. If it a lot lower than 100%, you have another bottleneck. That
> might
> >> be insufficient heap or accessing disk during query requests (not enough
> >> RAM). If it is near 100%, the only thing you can do is get a faster CPU.
> >>
> >> One other question, how frequently is the index updated?
> >>
> >> wunder
> >> Walter Underwood
> >> wun...@wunderwood.org
> >> http://observer.wunderwood.org/  (my blog)
> >>
> >>> On Jan 19, 2020, at 4:49 PM, Rajdeep Sahoo  >
> >> wrote:
> >>>
> >>> Hi,
> >>> Still facing the same issue...
> >>> Anything else that we need to check.
> >>>
> >>>
> >>> On Mon, 20 Jan, 2020, 4:07 AM Walter Underwood,  >
> >>> wrote:
> >>>
> >>>> With Java 1.8, I would use the G1 garbage collector. We’ve been
> running
> >>>> that combination in prod for three years with no problems.
> >>>>
> >>>> SOLR_HEAP=8g
> >>>> # Use G1 GC  -- wunder 2017-01-23
> >>>> # Settings from https://wiki.apache.org/solr/ShawnHeisey
> >>>> GC_TUNE=" \
> >>>> -XX:+UseG1GC \
> >>>> -XX:+ParallelRefProcEnabled \
> >>>> -XX:G1HeapRegionSize=8m \
> >>>> -XX:MaxGCPauseMillis=200 \
> >>>> -XX:+UseLargePages \
> >>>> -XX:+AggressiveOpts \
> >>>> “
> >>>>
> >>>> wunder
> >>>> Walter Underwood
> >>>> wun...@wunderwood.org
> >>>> http://observer.wunderwood.org/  (my blog)
> >>>>
> >>>>> On Jan 19, 2020, at 9:25 AM, Rajdeep Sahoo <
> rajdeepsahoo2...@gmail.com
> >>>
> >>>> wrote:
> >>>>>
> >>>>> We are using solr 7.7 . Ram size is 24 gb and allocated space is 12
> gb.
> >>>> We
> >>>>> have completed indexing after starting the server suddenly heap space
> >> is
> >>>>> getting full.
> >>>>> Added gc params  , still not working and jdk version is 1.8 .
> >>>>> Please find the below gc  params
> >>>>> -XX:NewRatio=2
> >>>>> -XX:SurvivorRatio=3
> >>>>> -XX:TargetSurvivorRatio=90 \
> >>>>> -XX:MaxTenuringThreshold=8 \
> >>>>> -XX:+UseConcMarkSweepGC \
> >>>>> -XX:+CMSScavengeBeforeRemark \
> >>>>> -XX:ConcGCThreads=4 -XX:ParallelGCThreads=4 \
> >>>>> -XX:PretenureSizeThreshold=512m \
> >>>>> -XX:CMSFullGCsBeforeCompaction=1 \
> >>>>> -XX:+UseCMSInitiatingOccupancyOnly \
> >>>>> -XX:CMSInitiatingOccupancyFraction=70 \
> >>>>> -XX:CMSMaxAbortablePrecleanTime=6000 \
> >>>>> -XX:+CMSParallelRemarkEnabled
> >>>>> -XX:+ParallelRefProcEnabled
> >>>>> -XX:+UseLargePages \
> >>>>> -XX:+AggressiveOpts \
> >>>>
> >>>>
> >>
> >>
>
>

Re: Solr 7.7 heap space is getting full

[Rajdeep Sahoo]

Hi,
 Currently there is no request or indexing is happening.
  It's just start up
 And during that time heap is getting full.
 Index size is approx 1 g.


On Mon, 20 Jan, 2020, 7:01 AM Walter Underwood, 
wrote:

> A new garbage collector won’t fix it, but it might help a bit.
>
> Requesting 200 facet fields and having 50-60 of them with results is a
> huge amount of work for Solr. A typical faceting implementation might have
> three to five facets. Your requests will be at least 10X to 20X slower.
>
> Check the CPU during one request. It should use nearly 100% of a single
> CPU. If it a lot lower than 100%, you have another bottleneck. That might
> be insufficient heap or accessing disk during query requests (not enough
> RAM). If it is near 100%, the only thing you can do is get a faster CPU.
>
> One other question, how frequently is the index updated?
>
> wunder
> Walter Underwood
> wun...@wunderwood.org
> http://observer.wunderwood.org/  (my blog)
>
> > On Jan 19, 2020, at 4:49 PM, Rajdeep Sahoo 
> wrote:
> >
> > Hi,
> > Still facing the same issue...
> > Anything else that we need to check.
> >
> >
> > On Mon, 20 Jan, 2020, 4:07 AM Walter Underwood, 
> > wrote:
> >
> >> With Java 1.8, I would use the G1 garbage collector. We’ve been running
> >> that combination in prod for three years with no problems.
> >>
> >> SOLR_HEAP=8g
> >> # Use G1 GC  -- wunder 2017-01-23
> >> # Settings from https://wiki.apache.org/solr/ShawnHeisey
> >> GC_TUNE=" \
> >> -XX:+UseG1GC \
> >> -XX:+ParallelRefProcEnabled \
> >> -XX:G1HeapRegionSize=8m \
> >> -XX:MaxGCPauseMillis=200 \
> >> -XX:+UseLargePages \
> >> -XX:+AggressiveOpts \
> >> “
> >>
> >> wunder
> >> Walter Underwood
> >> wun...@wunderwood.org
> >> http://observer.wunderwood.org/  (my blog)
> >>
> >>> On Jan 19, 2020, at 9:25 AM, Rajdeep Sahoo  >
> >> wrote:
> >>>
> >>> We are using solr 7.7 . Ram size is 24 gb and allocated space is 12 gb.
> >> We
> >>> have completed indexing after starting the server suddenly heap space
> is
> >>> getting full.
> >>>  Added gc params  , still not working and jdk version is 1.8 .
> >>> Please find the below gc  params
> >>> -XX:NewRatio=2
> >>> -XX:SurvivorRatio=3
> >>> -XX:TargetSurvivorRatio=90 \
> >>> -XX:MaxTenuringThreshold=8 \
> >>> -XX:+UseConcMarkSweepGC \
> >>> -XX:+CMSScavengeBeforeRemark \
> >>> -XX:ConcGCThreads=4 -XX:ParallelGCThreads=4 \
> >>> -XX:PretenureSizeThreshold=512m \
> >>> -XX:CMSFullGCsBeforeCompaction=1 \
> >>> -XX:+UseCMSInitiatingOccupancyOnly \
> >>> -XX:CMSInitiatingOccupancyFraction=70 \
> >>> -XX:CMSMaxAbortablePrecleanTime=6000 \
> >>> -XX:+CMSParallelRemarkEnabled
> >>> -XX:+ParallelRefProcEnabled
> >>> -XX:+UseLargePages \
> >>> -XX:+AggressiveOpts \
> >>
> >>
>
>

Re: Solr 7.7 heap space is getting full

[Rajdeep Sahoo]

Hi,
Still facing the same issue...
Anything else that we need to check.


On Mon, 20 Jan, 2020, 4:07 AM Walter Underwood, 
wrote:

> With Java 1.8, I would use the G1 garbage collector. We’ve been running
> that combination in prod for three years with no problems.
>
> SOLR_HEAP=8g
> # Use G1 GC  -- wunder 2017-01-23
> # Settings from https://wiki.apache.org/solr/ShawnHeisey
> GC_TUNE=" \
> -XX:+UseG1GC \
> -XX:+ParallelRefProcEnabled \
> -XX:G1HeapRegionSize=8m \
> -XX:MaxGCPauseMillis=200 \
> -XX:+UseLargePages \
> -XX:+AggressiveOpts \
> “
>
> wunder
> Walter Underwood
> wun...@wunderwood.org
> http://observer.wunderwood.org/  (my blog)
>
> > On Jan 19, 2020, at 9:25 AM, Rajdeep Sahoo 
> wrote:
> >
> > We are using solr 7.7 . Ram size is 24 gb and allocated space is 12 gb.
> We
> > have completed indexing after starting the server suddenly heap space is
> > getting full.
> >   Added gc params  , still not working and jdk version is 1.8 .
> > Please find the below gc  params
> > -XX:NewRatio=2
> > -XX:SurvivorRatio=3
> > -XX:TargetSurvivorRatio=90 \
> > -XX:MaxTenuringThreshold=8 \
> > -XX:+UseConcMarkSweepGC \
> > -XX:+CMSScavengeBeforeRemark \
> > -XX:ConcGCThreads=4 -XX:ParallelGCThreads=4 \
> > -XX:PretenureSizeThreshold=512m \
> > -XX:CMSFullGCsBeforeCompaction=1 \
> > -XX:+UseCMSInitiatingOccupancyOnly \
> > -XX:CMSInitiatingOccupancyFraction=70 \
> > -XX:CMSMaxAbortablePrecleanTime=6000 \
> > -XX:+CMSParallelRemarkEnabled
> > -XX:+ParallelRefProcEnabled
> > -XX:+UseLargePages \
> > -XX:+AggressiveOpts \
>
>

Re: Solr 7.7 heap space is getting full

[Rajdeep Sahoo]

Please reply anyone

On Sun, 19 Jan, 2020, 10:55 PM Rajdeep Sahoo, 
wrote:

> We are using solr 7.7 . Ram size is 24 gb and allocated space is 12 gb. We
> have completed indexing after starting the server suddenly heap space is
> getting full.
>Added gc params  , still not working and jdk version is 1.8 .
> Please find the below gc  params
> -XX:NewRatio=2
> -XX:SurvivorRatio=3
> -XX:TargetSurvivorRatio=90 \
> -XX:MaxTenuringThreshold=8 \
> -XX:+UseConcMarkSweepGC \
> -XX:+CMSScavengeBeforeRemark \
> -XX:ConcGCThreads=4 -XX:ParallelGCThreads=4 \
> -XX:PretenureSizeThreshold=512m \
> -XX:CMSFullGCsBeforeCompaction=1 \
> -XX:+UseCMSInitiatingOccupancyOnly \
> -XX:CMSInitiatingOccupancyFraction=70 \
> -XX:CMSMaxAbortablePrecleanTime=6000 \
> -XX:+CMSParallelRemarkEnabled
> -XX:+ParallelRefProcEnabled
> -XX:+UseLargePages \
> -XX:+AggressiveOpts \
>

Solr 7.7 heap space is getting full

[Rajdeep Sahoo]

We are using solr 7.7 . Ram size is 24 gb and allocated space is 12 gb. We
have completed indexing after starting the server suddenly heap space is
getting full.
   Added gc params  , still not working and jdk version is 1.8 .
Please find the below gc  params
-XX:NewRatio=2
-XX:SurvivorRatio=3
-XX:TargetSurvivorRatio=90 \
-XX:MaxTenuringThreshold=8 \
-XX:+UseConcMarkSweepGC \
-XX:+CMSScavengeBeforeRemark \
-XX:ConcGCThreads=4 -XX:ParallelGCThreads=4 \
-XX:PretenureSizeThreshold=512m \
-XX:CMSFullGCsBeforeCompaction=1 \
-XX:+UseCMSInitiatingOccupancyOnly \
-XX:CMSInitiatingOccupancyFraction=70 \
-XX:CMSMaxAbortablePrecleanTime=6000 \
-XX:+CMSParallelRemarkEnabled
-XX:+ParallelRefProcEnabled
-XX:+UseLargePages \
-XX:+AggressiveOpts \

Re: Solr cloud production set up

[Rajdeep Sahoo]

Apart from reducing no of facets in the query, is there any other query
params or gc params or heap space or anything else that we need to tweak
for improving search response time.

On Sun, 19 Jan, 2020, 3:15 AM Erick Erickson, 
wrote:

> Add &debug=timing to the query and it’ll show you the time each component
> takes.
>
> > On Jan 18, 2020, at 1:50 PM, Rajdeep Sahoo 
> wrote:
> >
> > Thanks for the suggestion,
> >
> > Is there any way to get the info which operation or which query params
> are
> > increasing the response time.
> >
> >
> > On Sat, 18 Jan, 2020, 11:59 PM Dave, 
> wrote:
> >
> >> If you’re not getting values, don’t ask for the facet. Facets are
> >> expensive as hell, maybe you should think more about your query’s than
> your
> >> infrastructure, solr cloud won’t help you at all especially if your
> asking
> >> for things you don’t need
> >>
> >>> On Jan 18, 2020, at 1:25 PM, Rajdeep Sahoo  >
> >> wrote:
> >>>
> >>> We have assigned 16 gb out of 24gb for heap .
> >>> No other process is running on that node.
> >>>
> >>> 200 facets fields are there in the query but we will not be getting the
> >>> values for each facets for every search.
> >>> There can be max of 50-60 facets for which we will be getting values.
> >>>
> >>> We are using caching,is it not going to help.
> >>>
> >>>
> >>>
> >>>> On Sat, 18 Jan, 2020, 11:36 PM Shawn Heisey, 
> >> wrote:
> >>>>
> >>>>> On 1/18/2020 10:09 AM, Rajdeep Sahoo wrote:
> >>>>> We are having 2.3 million documents and size is 2.5 gb.
> >>>>>  10 core cpu and 24 gb ram . 16 slave nodes.
> >>>>>
> >>>>>  Still some of the queries are taking 50 sec at solr end.
> >>>>> As we are using solr 4.6 .
> >>>>>  Other thing is we are having 200 (avg) facet fields  in a query.
> >>>>> And 30 searchable fields.
> >>>>> Is there any way to identify why it is taking 50 sec for a query.
> >>>>>Multiple concurrent requests are there.
> >>>>
> >>>> Searching 30 fields and computing 200 facets is never going to be
> super
> >>>> fast.  Switching to cloud will not help, and might make it slower.
> >>>>
> >>>> Your index is pretty small to a lot of us.  There are people running
> >>>> indexes with billions of documents that take terabytes of disk space.
> >>>>
> >>>> As Walter mentioned, computing 200 facets is going to require a fair
> >>>> amount of heap memory.  One *possible* problem here is that the Solr
> >>>> heap size is too small, so a lot of GC is required.  How much of the
> >>>> 24GB have you assigned to the heap?  Is there any software other than
> >>>> Solr running on these nodes?
> >>>>
> >>>> Thanks,
> >>>> Shawn
> >>>>
> >>
>
>

Re: Solr cloud production set up

[Rajdeep Sahoo]

Thanks for the suggestion,

 Is there any way to get the info which operation or which query params are
increasing the response time.


On Sat, 18 Jan, 2020, 11:59 PM Dave,  wrote:

> If you’re not getting values, don’t ask for the facet. Facets are
> expensive as hell, maybe you should think more about your query’s than your
> infrastructure, solr cloud won’t help you at all especially if your asking
> for things you don’t need
>
> > On Jan 18, 2020, at 1:25 PM, Rajdeep Sahoo 
> wrote:
> >
> > We have assigned 16 gb out of 24gb for heap .
> > No other process is running on that node.
> >
> > 200 facets fields are there in the query but we will not be getting the
> > values for each facets for every search.
> > There can be max of 50-60 facets for which we will be getting values.
> >
> > We are using caching,is it not going to help.
> >
> >
> >
> >> On Sat, 18 Jan, 2020, 11:36 PM Shawn Heisey, 
> wrote:
> >>
> >>> On 1/18/2020 10:09 AM, Rajdeep Sahoo wrote:
> >>> We are having 2.3 million documents and size is 2.5 gb.
> >>>   10 core cpu and 24 gb ram . 16 slave nodes.
> >>>
> >>>   Still some of the queries are taking 50 sec at solr end.
> >>> As we are using solr 4.6 .
> >>>   Other thing is we are having 200 (avg) facet fields  in a query.
> >>>  And 30 searchable fields.
> >>>  Is there any way to identify why it is taking 50 sec for a query.
> >>> Multiple concurrent requests are there.
> >>
> >> Searching 30 fields and computing 200 facets is never going to be super
> >> fast.  Switching to cloud will not help, and might make it slower.
> >>
> >> Your index is pretty small to a lot of us.  There are people running
> >> indexes with billions of documents that take terabytes of disk space.
> >>
> >> As Walter mentioned, computing 200 facets is going to require a fair
> >> amount of heap memory.  One *possible* problem here is that the Solr
> >> heap size is too small, so a lot of GC is required.  How much of the
> >> 24GB have you assigned to the heap?  Is there any software other than
> >> Solr running on these nodes?
> >>
> >> Thanks,
> >> Shawn
> >>
>

Re: Solr cloud production set up

[Rajdeep Sahoo]

We have assigned 16 gb out of 24gb for heap .
 No other process is running on that node.

200 facets fields are there in the query but we will not be getting the
values for each facets for every search.
There can be max of 50-60 facets for which we will be getting values.

 We are using caching,is it not going to help.



On Sat, 18 Jan, 2020, 11:36 PM Shawn Heisey,  wrote:

> On 1/18/2020 10:09 AM, Rajdeep Sahoo wrote:
> > We are having 2.3 million documents and size is 2.5 gb.
> >10 core cpu and 24 gb ram . 16 slave nodes.
> >
> >Still some of the queries are taking 50 sec at solr end.
> > As we are using solr 4.6 .
> >Other thing is we are having 200 (avg) facet fields  in a query.
> >   And 30 searchable fields.
> >   Is there any way to identify why it is taking 50 sec for a query.
> >  Multiple concurrent requests are there.
>
> Searching 30 fields and computing 200 facets is never going to be super
> fast.  Switching to cloud will not help, and might make it slower.
>
> Your index is pretty small to a lot of us.  There are people running
> indexes with billions of documents that take terabytes of disk space.
>
> As Walter mentioned, computing 200 facets is going to require a fair
> amount of heap memory.  One *possible* problem here is that the Solr
> heap size is too small, so a lot of GC is required.  How much of the
> 24GB have you assigned to the heap?  Is there any software other than
> Solr running on these nodes?
>
> Thanks,
> Shawn
>

Re: Solr cloud production set up

[Rajdeep Sahoo]

Although we are having a avg of 200 facet fields in the search request all
of them will not be having values in each request.
Max of 50-60 facet fields will be having some value.
  And we are using function query,is it having some performance impact.


On Sat, 18 Jan, 2020, 11:10 PM Walter Underwood, 
wrote:

> For indexing, is the master node CPU around 90%? If not, you aren’t
> sending requests fast enough or your disk is slow.
>
> For querying, 200 facet fields is HUGE. That will take a lot of Java heap
> memory and will be slow. Each facet fields requires large in-memory arrays
> and sorting.
>
> wunder
> Walter Underwood
> wun...@wunderwood.org
> http://observer.wunderwood.org/  (my blog)
>
> > On Jan 18, 2020, at 9:29 AM, Rajdeep Sahoo 
> wrote:
> >
> > Hi shawn,
> >  Thanks for this info,
> > Could you Please address my below query,
> >
> >
> > We are having 2.3 million documents and size is 2.5 gb.
> > With this data do we need solr cloud.
> >
> >  10 core cpu and 24 gb ram . 16 slave nodes.
> >
> >  Still some of the queries are taking 50 sec at solr end.
> > As we are using solr 4.6 .
> >  Other thing is we are having 200 (avg) facet fields  in a query.
> > And 30 searchable fields.
> > Is there any way to identify why it is taking 50 sec for a query.
> >Multiple concurrent requests are there.
> >
> > And how to optimize the search response time as it is almost 1 mins for
> > some request.
> >
> >
> > On Sat, 18 Jan, 2020, 10:52 PM Shawn Heisey, 
> wrote:
> >
> >> On 1/18/2020 9:55 AM, Rajdeep Sahoo wrote:
> >>> We do parallel indexing in production,
> >>>
> >>>  What about search performance in solr cloud in comparison with master
> >>> slave.
> >>>And what about  block join performance in solr cloud.
> >>>Do we need to increase the infra for solr cloud as we would be
> >>> maintaining multiple shard and replica.
> >>>   Is there any co relation with master slave set up.
> >>
> >> As I said before, SolrCloud is not a magic bullet that solves
> >> performance issues.  If the index characteristics are the same (number
> >> of docs, total size), performance in SolrCloud will be nearly identical
> >> to non-cloud.
> >>
> >> Thanks,
> >> Shawn
> >>
>
>

Re: Solr cloud production set up

[Rajdeep Sahoo]

Hi shawn,
  Thanks for this info,
Could you Please address my below query,


We are having 2.3 million documents and size is 2.5 gb.
 With this data do we need solr cloud.

  10 core cpu and 24 gb ram . 16 slave nodes.

  Still some of the queries are taking 50 sec at solr end.
As we are using solr 4.6 .
  Other thing is we are having 200 (avg) facet fields  in a query.
 And 30 searchable fields.
 Is there any way to identify why it is taking 50 sec for a query.
Multiple concurrent requests are there.

And how to optimize the search response time as it is almost 1 mins for
some request.


On Sat, 18 Jan, 2020, 10:52 PM Shawn Heisey,  wrote:

> On 1/18/2020 9:55 AM, Rajdeep Sahoo wrote:
> > We do parallel indexing in production,
> >
> >   What about search performance in solr cloud in comparison with master
> > slave.
> > And what about  block join performance in solr cloud.
> > Do we need to increase the infra for solr cloud as we would be
> > maintaining multiple shard and replica.
> >Is there any co relation with master slave set up.
>
> As I said before, SolrCloud is not a magic bullet that solves
> performance issues.  If the index characteristics are the same (number
> of docs, total size), performance in SolrCloud will be nearly identical
> to non-cloud.
>
> Thanks,
> Shawn
>

Re: Solr cloud production set up

[Rajdeep Sahoo]

We are having 2.3 million documents and size is 2.5 gb.
  10 core cpu and 24 gb ram . 16 slave nodes.

  Still some of the queries are taking 50 sec at solr end.
As we are using solr 4.6 .
  Other thing is we are having 200 (avg) facet fields  in a query.
 And 30 searchable fields.
 Is there any way to identify why it is taking 50 sec for a query.
Multiple concurrent requests are there.



On Sat, 18 Jan, 2020, 10:32 PM Dave,  wrote:

> Agreed with the above. what’s your idea of “huge”? I have 600 ish gb in
> one core plus another 250x2 in two more on the same standalone solr
> instance and it runs more than fine
>
> > On Jan 18, 2020, at 11:31 AM, Shawn Heisey  wrote:
> >
> > On 1/18/2020 1:05 AM, Rajdeep Sahoo wrote:
> >> Our Index size is huge and in master slave the full indexing time is
> almost
> >> 24 hrs.
> >>In future the no of documents will increase.
> >> So,please some one recommend about the no of nodes and configuration
> like
> >> ram and cpu core for solr cloud.
> >
> > Indexing is not going to be any faster in SolrCloud.  It would probably
> be a little bit slower.  The best way to speed up indexing, whether running
> SolrCloud or not, is to make your indexing processes run in parallel, so
> that multiple batches of documents are being indexed at the same time.
> >
> > SolrCloud is not a magic bullet that solves all problems.  It's just a
> different way of managing indexes that has more automation, and makes
> initial setup of a distributed index a lot easier.  It doesn't do the job
> any faster than running without SolrCloud.  The legacy master/slave mode is
> likely to be a little bit faster.
> >
> > You haven't provided any of the information required for us to guess
> about the system requirements.  And it will be a guess ... we could be
> completely wrong.
> >
> >
> https://lucidworks.com/post/sizing-hardware-in-the-abstract-why-we-dont-have-a-definitive-answer/
> >
> > Thanks,
> > Shawn
>

Re: Solr cloud production set up

[Rajdeep Sahoo]

Hi shawn,
 Thanks for your reply

We do parallel indexing in production,

 What about search performance in solr cloud in comparison with master
slave.
   And what about  block join performance in solr cloud.
   Do we need to increase the infra for solr cloud as we would be
maintaining multiple shard and replica.
  Is there any co relation with master slave set up.




On Sat, 18 Jan, 2020, 10:01 PM Shawn Heisey,  wrote:

> On 1/18/2020 1:05 AM, Rajdeep Sahoo wrote:
> > Our Index size is huge and in master slave the full indexing time is
> almost
> > 24 hrs.
> > In future the no of documents will increase.
> > So,please some one recommend about the no of nodes and configuration like
> > ram and cpu core for solr cloud.
>
> Indexing is not going to be any faster in SolrCloud.  It would probably
> be a little bit slower.  The best way to speed up indexing, whether
> running SolrCloud or not, is to make your indexing processes run in
> parallel, so that multiple batches of documents are being indexed at the
> same time.
>
> SolrCloud is not a magic bullet that solves all problems.  It's just a
> different way of managing indexes that has more automation, and makes
> initial setup of a distributed index a lot easier.  It doesn't do the
> job any faster than running without SolrCloud.  The legacy master/slave
> mode is likely to be a little bit faster.
>
> You haven't provided any of the information required for us to guess
> about the system requirements.  And it will be a guess ... we could be
> completely wrong.
>
>
> https://lucidworks.com/post/sizing-hardware-in-the-abstract-why-we-dont-have-a-definitive-answer/
>
> Thanks,
> Shawn
>

Re: Solr cloud production set up

[Rajdeep Sahoo]

Got your point.
  If we think about the infra, then in cloud do we need more infra in
comparison to master slave.



On Sat, 18 Jan, 2020, 2:24 PM Jörn Franke,  wrote:

> I think you should do your own measurements. This is very document and
> processing specific.
> You can run a test with a simple setup for let’s say 1 mio document and
> interpolate from this. It could be also that your ETL is the bottleneck and
> not Solr.
> At the same time you can simulate user queries using Jmeter or similar.
>
> > Am 18.01.2020 um 09:05 schrieb Rajdeep Sahoo  >:
> >
> > Our Index size is huge and in master slave the full indexing time is
> almost
> > 24 hrs.
> >   In future the no of documents will increase.
> > So,please some one recommend about the no of nodes and configuration like
> > ram and cpu core for solr cloud.
> >
> >> On Sat, 18 Jan, 2020, 8:05 AM Walter Underwood, 
> >> wrote:
> >>
> >> Why do you want to change to Solr Cloud? Master/slave is a great, stable
> >> cluster architecture.
> >>
> >> wunder
> >> Walter Underwood
> >> wun...@wunderwood.org
> >> http://observer.wunderwood.org/  (my blog)
> >>
> >>> On Jan 17, 2020, at 6:19 PM, Rajdeep Sahoo  >
> >> wrote:
> >>>
> >>> Please reply anyone
> >>>
> >>> On Sat, 18 Jan, 2020, 12:13 AM Rajdeep Sahoo, <
> >> rajdeepsahoo2...@gmail.com>
> >>> wrote:
> >>>
> >>>> Hi all,
> >>>> We are using solr cloud 7.7.1
> >>>> In a live production environment how many solr cloud server do we
> need,
> >>>> Currently ,we are using master slave set up with 16 slave server with
> >>>> solr 4.6.
> >>>> In solr cloud do we need to scale it up or 16 server will suffice the
> >>>> purpose.
> >>>>
> >>>>
> >>
> >>
>

Re: Solr cloud production set up

[Rajdeep Sahoo]

Our Index size is huge and in master slave the full indexing time is almost
24 hrs.
   In future the no of documents will increase.
So,please some one recommend about the no of nodes and configuration like
ram and cpu core for solr cloud.

On Sat, 18 Jan, 2020, 8:05 AM Walter Underwood, 
wrote:

> Why do you want to change to Solr Cloud? Master/slave is a great, stable
> cluster architecture.
>
> wunder
> Walter Underwood
> wun...@wunderwood.org
> http://observer.wunderwood.org/  (my blog)
>
> > On Jan 17, 2020, at 6:19 PM, Rajdeep Sahoo 
> wrote:
> >
> > Please reply anyone
> >
> > On Sat, 18 Jan, 2020, 12:13 AM Rajdeep Sahoo, <
> rajdeepsahoo2...@gmail.com>
> > wrote:
> >
> >> Hi all,
> >> We are using solr cloud 7.7.1
> >> In a live production environment how many solr cloud server do we need,
> >> Currently ,we are using master slave set up with 16 slave server with
> >> solr 4.6.
> >> In solr cloud do we need to scale it up or 16 server will suffice the
> >> purpose.
> >>
> >>
>
>

Re: Solr cloud production set up

[Rajdeep Sahoo]

Please reply anyone

On Sat, 18 Jan, 2020, 12:13 AM Rajdeep Sahoo, 
wrote:

> Hi all,
>  We are using solr cloud 7.7.1
> In a live production environment how many solr cloud server do we need,
>  Currently ,we are using master slave set up with 16 slave server with
> solr 4.6.
> In solr cloud do we need to scale it up or 16 server will suffice the
> purpose.
>
>

Solr cloud production set up

[Rajdeep Sahoo]

Hi all,
 We are using solr cloud 7.7.1
In a live production environment how many solr cloud server do we need,
 Currently ,we are using master slave set up with 16 slave server with solr
4.6.
In solr cloud do we need to scale it up or 16 server will suffice the
purpose.

Re: In production solr cloud 4.6 is not working

[Rajdeep Sahoo]

Nodes were busy with full Gc over 15 sec.
In solr console it was showing no server hosting shard,status code 500.
In solr log the same error was there i.e. solrexception: No server hosting
shard.

Apart from this,in the zoo console it was showing recovery failing status
of the nodes.

Thanks in advance

On Sat, 11 Jan, 2020, 7:15 PM Erick Erickson, 
wrote:

> You’ve provided no details, nor relayed any findings from looking
> at the Solr logs. In short, there’s not enough information here to
> provide any helpful response.
>
> Full GCs are normal, but if they’re long enough
> to exceed certain timeouts, they can trigger recoveries. Solr 4.6 had
> a number of conditions that can lead to this this, but Solr 4.6 is
> over 6 years old. There’s going to be little help available at this point.
>
> You might want to review: https://wiki.apache.org/solr/UsingMailingLists
>
>
> > On Jan 11, 2020, at 2:47 AM, Rajdeep Sahoo 
> wrote:
> >
> > Hi ,
> >
> > Anyone Please suggest.
> >
> > On Sat, 11 Jan, 2020, 12:33 AM Rajdeep Sahoo, <
> rajdeepsahoo2...@gmail.com>
> > wrote:
> >
> >> Hi all,
> >>  In production we have done the set up of solr cloud with solr version
> >> 4.6 , the set up contains  3 zookeeper and 4 shards each having three
> >> replicas, a total of 12 solr nodes.
> >> Active indexing was going on , after switching on we are experiencing a
> >> lot of issues , all the nodes stopped serving the search requests and in
> >> log it is showing recovery failing , and can see Full GC for 2 shards.
> >>  After graceful restart , the recovery failing issue got resolved but
> >> currently we are using master slave with 16 slave nodes and 1 leader ,
> it
> >> is working fine.
> >>  Do we need to scale it up in solr cloud? Please suggest as it is a
> >> production env. , I guess all you can understand the impact of it.
> >>
> >> Thanks in advance
> >>
>
>

Re: In production solr cloud 4.6 is not working

[Rajdeep Sahoo]

Hi ,

Anyone Please suggest.

On Sat, 11 Jan, 2020, 12:33 AM Rajdeep Sahoo, 
wrote:

> Hi all,
>   In production we have done the set up of solr cloud with solr version
> 4.6 , the set up contains  3 zookeeper and 4 shards each having three
> replicas, a total of 12 solr nodes.
> Active indexing was going on , after switching on we are experiencing a
> lot of issues , all the nodes stopped serving the search requests and in
> log it is showing recovery failing , and can see Full GC for 2 shards.
>   After graceful restart , the recovery failing issue got resolved but
> currently we are using master slave with 16 slave nodes and 1 leader , it
> is working fine.
>   Do we need to scale it up in solr cloud? Please suggest as it is a
> production env. , I guess all you can understand the impact of it.
>
> Thanks in advance
>

In production solr cloud 4.6 is not working

[Rajdeep Sahoo]

Hi all,
  In production we have done the set up of solr cloud with solr version 4.6
, the set up contains  3 zookeeper and 4 shards each having three replicas,
a total of 12 solr nodes.
Active indexing was going on , after switching on we are experiencing a lot
of issues , all the nodes stopped serving the search requests and in log it
is showing recovery failing , and can see Full GC for 2 shards.
  After graceful restart , the recovery failing issue got resolved but
currently we are using master slave with 16 slave nodes and 1 leader , it
is working fine.
  Do we need to scale it up in solr cloud? Please suggest as it is a
production env. , I guess all you can understand the impact of it.

Thanks in advance

Re: Solr 7.5 speed up, accuracy details

[Rajdeep Sahoo]

Hi all,
  How can I get the performance improvement features in indexing and search
in solr 7.5...

On Sat, 28 Dec, 2019, 9:18 PM Rajdeep Sahoo, 
wrote:

> Thank you for the information
>   Why you are recommending to use the schema api instead of schema xml?
>
>
> On Sat, 28 Dec, 2019, 8:01 PM Jörn Franke,  wrote:
>
>> This highly depends on how you designed your collections etc. - there is
>> no general answer. You have to do a performance test based on your
>> configuration and documents.
>>
>> I also recommend to check the Solr documentation on how to design a
>> collection for 7.x and maybe start even from scratch defining it with a new
>> fresh schema (using the schema api instead of schema.xml and solrconfig.xml
>> etc). You will have anyway to reindex everything so it is a also a good
>> opportunity to look at your existing processes and optimize them.
>>
>> > Am 28.12.2019 um 15:19 schrieb Rajdeep Sahoo <
>> rajdeepsahoo2...@gmail.com>:
>> >
>> > Hi all,
>> > Is there any way I can get the speed up,accuracy details i.e.
>> performance
>> > improvements of solr 7.5 in comparison with solr 4.6
>> >  Currently,we are using solr 4.6 and we are in a process to upgrade to
>> > solr 7.5. Need these details.
>> >
>> > Thanks in advance
>>
>

Re: Solr 7.5 seed up, accuracy details

[Rajdeep Sahoo]

Thank you for the information
  Why you are recommending to use the schema api instead of schema xml?


On Sat, 28 Dec, 2019, 8:01 PM Jörn Franke,  wrote:

> This highly depends on how you designed your collections etc. - there is
> no general answer. You have to do a performance test based on your
> configuration and documents.
>
> I also recommend to check the Solr documentation on how to design a
> collection for 7.x and maybe start even from scratch defining it with a new
> fresh schema (using the schema api instead of schema.xml and solrconfig.xml
> etc). You will have anyway to reindex everything so it is a also a good
> opportunity to look at your existing processes and optimize them.
>
> > Am 28.12.2019 um 15:19 schrieb Rajdeep Sahoo  >:
> >
> > Hi all,
> > Is there any way I can get the speed up,accuracy details i.e. performance
> > improvements of solr 7.5 in comparison with solr 4.6
> >  Currently,we are using solr 4.6 and we are in a process to upgrade to
> > solr 7.5. Need these details.
> >
> > Thanks in advance
>

Solr 7.5 seed up, accuracy details

[Rajdeep Sahoo]

Hi all,
Is there any way I can get the speed up,accuracy details i.e. performance
improvements of solr 7.5 in comparison with solr 4.6
  Currently,we are using solr 4.6 and we are in a process to upgrade to
solr 7.5. Need these details.

Thanks in advance

Solr join query

[Rajdeep Sahoo]

Can we join two solr collection based on multivalued field.

Solr query high response time

[Rajdeep Sahoo]

Hi all,
  My solr query sometime taking more than 60 sec to return the response .
Is there any way I can check why it is taking so much time .
  Please let me know if there is any way to analyse this issue(high
response time ) .Thanks

Re: Solr block join

[Rajdeep Sahoo]

But how can I use it. In what context block join is required. If I use it,
what are the benefits I will be getting.
On Jan 27, 2019 6:49 PM, "Mikhail Khludnev"  wrote:

> Rajdeep, it's already implemented in Lucene and Solr.
>
>
> On Sun, Jan 27, 2019 at 1:14 PM Rajdeep Sahoo 
> wrote:
>
> > Please help in understanding how to implement block join query
> > On Jan 27, 2019 1:43 AM, "Mikhail Khludnev"  wrote:
> >
> > > https://lucene.apache.org/solr/guide/7_6/other-parsers.
> > > html#block-join-parent-query-parser
> > >
> > > On Sat, Jan 26, 2019 at 10:30 PM Rajdeep Sahoo <
> > rajdeepsahoo2...@gmail.com
> > > >
> > > wrote:
> > >
> > > > Please suggest anyone
> > > > On Jan 8, 2019 11:07 PM, "Rajdeep Sahoo"  >
> > > > wrote:
> > > >
> > > > > What is the use of block join in solr
> > > > >
> > > >
> > >
> > >
> > > --
> > > Sincerely yours
> > > Mikhail Khludnev
> > >
> >
>
>
> --
> Sincerely yours
> Mikhail Khludnev
>

Re: Solr block join

[Rajdeep Sahoo]

Please help in understanding how to implement block join query
On Jan 27, 2019 1:43 AM, "Mikhail Khludnev"  wrote:

> https://lucene.apache.org/solr/guide/7_6/other-parsers.
> html#block-join-parent-query-parser
>
> On Sat, Jan 26, 2019 at 10:30 PM Rajdeep Sahoo  >
> wrote:
>
> > Please suggest anyone
> > On Jan 8, 2019 11:07 PM, "Rajdeep Sahoo" 
> > wrote:
> >
> > > What is the use of block join in solr
> > >
> >
>
>
> --
> Sincerely yours
> Mikhail Khludnev
>

Re: Solr block join

[Rajdeep Sahoo]

Please suggest anyone
On Jan 8, 2019 11:07 PM, "Rajdeep Sahoo"  wrote:

> What is the use of block join in solr
>

Solr thread dump analysis

[Rajdeep Sahoo]

Hi all,
How can I analyse solr thread dump and how the thread dump analysis can be
helpful for improving solr performance. Please suggest

Solr block join

[Rajdeep Sahoo]

What is the use of block join in solr

Solr code base setup

[Rajdeep Sahoo]

How can I set up code Base and do the changes.

Inverted index and forward index in Solr shrading

[Rajdeep Sahoo]

Can any explain me what is inverted index and forward index used in solr
shrading . Is there any resource from where I can find some knowledge about
this.

Re: Solr cache clear

[Rajdeep Sahoo]

Hi all,
The problem is that we are using master slave solr configuration and for
similar type of query  sometime it is taking 512 ms and sometime it is 29
ms . we are observing this issue since the query modification. As part of
modification we have reduced a large no of facet.field param. In
solrconfig.xml file we have changed the cache size and reloaded the the
cores as part of this modification.
 Reloading cores is not clearing the cache. Is there any reason why we
are getting this much of difference in response time for similar type of
queries.
On Nov 21, 2018 4:16 AM, "Edward Ribeiro"  wrote:

> Disabling or reducing autowarming can help too, in addition to cache size
> reduction.
>
> Edward
>
> Em ter, 20 de nov de 2018 17:29, Erick Erickson  escreveu:
>
> > Why would you want to? This  sounds like an XY problem, there's some
> > problem you think would be cured by clearing the cache. What is
> > that problem?
> >
> > Because I doubt this would do anything useful, pretty soon the caches
> > would be filled up again and you'd be right  back where you started and
> > the real solution is to stop doing whatever you're doing that leads to
> > whatever the real problem is. Maybe reducing the cache sizes.
> >
> > Best,
> > Erick
> > On Tue, Nov 20, 2018 at 9:05 AM Shawn Heisey 
> wrote:
> > >
> > > On 11/20/2018 9:25 AM, Rajdeep Sahoo wrote:
> > > > Hi all,
> > > > Without restarting is it possible to clear the cache?
> > >
> > > You'll need to clarify what cache you're talking about, but I think for
> > > the most part that if you reload the core (or collection if running
> > > SolrCloud) that all caches should be rebuilt empty.
> > >
> > > Thanks,
> > > Shawn
> > >
> >
>

Solr cache clear

[Rajdeep Sahoo]

Hi all,
Without restarting is it possible to clear the cache?

Solr query response time analysis

[Rajdeep Sahoo]

Hi all,
 Please suggest, how can I analyze the time taken by a solr query?
Is there any tool for analyzing the query response time.I f there is any
way to do this please suggest.

Re: Solr File based spellchecker is not returning expected result

[Rajdeep Sahoo]

can anyone please help me regarding spell checking issue what I have
mentioned?

On Thu, Sep 13, 2018 at 12:31 AM Rajdeep Sahoo 
wrote:

> Hi ,
> I am using solr 4.6 version.
> My document is having a "iphone 7" but when I am searching with with
> "iphone7" I am getting the result because here worddelimiterfilterfactory
> is taking care by slipt on numerics functionality.
>   (Iphone7-->iphone 7)
> But I want solr to return a spellcheck suggestion as "iphone 7" .
> When I am configuring "Iphone 7" in the spellings.txt file it is not
> returning expected result which i slike
> iphone7--->"iphone 7" 7
>
> Another problem is how can I use filebasedspellchecke ,   wordbreak
> spellchecke and directsolrspellchecker at the same time.Here getting error
> for distanceMeasure param
>
>   please help.Thanks in advance
>

Re: Need Support - Apache Solr - 20180915

[Rajdeep Sahoo]

You can go for solr cloud if you have billions of data and in future if you
want to increase the volume.
Solr is a file based database for retriving data and not for complex
oprations like joining multiple tables.If your requirement is only storing
and fast retrieving then solr is the best option in comparison to
conventional relational  db.
You can configurer the no of  threads from jetty server  configuration .


On Sat, Sep 15, 2018 at 8:54 PM senthil  wrote:

> Dear Team,
>
>
>
> We are beginners to Apache Solr and its implementations. We need the
> following basic clarifications regarding Apache Solr usage and implementing
> with MS-SQL server database.
>
>
>
> 1. Our MS-SQL server database having the data table which contains 20
> columns with billions of data.
>
>
>
> 2. How to implement Apache Solr in the particular above table to increase
> search capability?
>
>
>
> 3. Is there any way to call the data which is distributed across 2
> shards/node of Apache Solr at a time?
>
>
>
> 4. Is there any performance difference between search the data in a single
> shard/node and multiple shard/node?
>
>
>
>
>
>
>
> Thanks & Regards
>
>
>
> *SENTHIL KUMAR P*
>
> Team Leader
>
>
>
> Office: +91-4362-243433
>
> Skype: syxsenthilp
>
>
>
> *Sardonyx Technologies Pvt. Ltd.,*
>
> Thanjavur - 613007
>
> www.sardonyx.in
>
>
>
> [image: sardonyx_logo]
>
>
>
> [image: signature-1]
>
>
>
> *From:* senthil [mailto:senthilkum...@sardonyx.in]
> *Sent:* Friday, September 14, 2018 12:40 PM
> *To:* 'gene...@lucene.apache.org'; 'solr-user@lucene.apache.org'; '
> d...@lucene.apache.org'
> *Cc:* kulothungand; 'karthic...@sardonyx.in'
> *Subject:* Need Support - Apache Solr - 20180914
>
>
>
> Dear Team,
>
>
>
> We need the below clarifications and doubts in the “*Apache Solr*” and
> please give us the solution.
>
>
>
> 1.  Apache Solr is the Database or not ?
>
>
>
> 2.  The limitation for Apache Solr is 2 billion records and how can
> we increase it (unlimitted) ?
>
>
>
> 3.  How many users can use / access Apache Solr through web
> application at a time ?
>
>
>
>
>
>
>
> Thanks & Regards
>
>
>
> *SENTHIL KUMAR P*
>
> Team Leader
>
>
>
> Office: +91-4362-243433
>
> Skype: syxsenthilp
>
>
>
> *Sardonyx Technologies Pvt. Ltd.,*
>
> Thanjavur - 613007
>
> www.sardonyx.in
>
>
>
> [image: sardonyx_logo]
>
>
>
> [image: signature-1]
>
>
>

Re: Solr File based spellchecker is not returning expected result

[Rajdeep Sahoo]

Another ask is , how can  I use multiple spellchecker at the same time
based on condition
Currently we are using two spellchecker   [spellcheck.dictionary=
wordbreak  , spellcheck.dictionary=en ]
 If wordbreak dictionary is having suggesion it will make a second call for
fethching the result and in the same call we are using direct solr
spellchecker
  If we are not getitng some result in the second call we are using
directsolr spellchecker.

How can I write a function query for getting suggestion against multiple
spellchecker?




On Thu, Sep 13, 2018 at 12:31 AM Rajdeep Sahoo 
wrote:

> Hi ,
> I am using solr 4.6 version.
> My document is having a "iphone 7" but when I am searching with with
> "iphone7" I am getting the result because here worddelimiterfilterfactory
> is taking care by slipt on numerics functionality.
>   (Iphone7-->iphone 7)
> But I want solr to return a spellcheck suggestion as "iphone 7" .
> When I am configuring "Iphone 7" in the spellings.txt file it is not
> returning expected result which i slike
> iphone7--->"iphone 7" 7
>
> Another problem is how can I use filebasedspellchecke ,   wordbreak
> spellchecke and directsolrspellchecker at the same time.Here getting error
> for distanceMeasure param
>
>   please help.Thanks in advance
>

Solr File based spellchecker is not returning expected result

[Rajdeep Sahoo]

Hi ,
I am using solr 4.6 version.
My document is having a "iphone 7" but when I am searching with with
"iphone7" I am getting the result because here worddelimiterfilterfactory
is taking care by slipt on numerics functionality.
  (Iphone7-->iphone 7)
But I want solr to return a spellcheck suggestion as "iphone 7" .
When I am configuring "Iphone 7" in the spellings.txt file it is not
returning expected result which i slike
iphone7--->"iphone 7" 7

Another problem is how can I use filebasedspellchecke ,   wordbreak
spellchecke and directsolrspellchecker at the same time.Here getting error
for distanceMeasure param

  please help.Thanks in advance