Upayavira:
I am using solr 4.7 and yes I am using TieredMergePolicy
Erick:
All my boxes have SSD’s and there isn’t a big disparity between qTime and
response time.
The performance hit on my end is because of the fragmented index files causing
more disk seeks are you mentioned.
And I tried reques
On 6/30/2015 6:23 AM, Erick Erickson wrote:
> I've actually seen this happen right in front of my eyes "in the
> field". However, that was a very high-performance environment. My
> assumption was that fragmented index files were causing more disk
> seeks especially for the first-pass query response
ly to match mongodb (indexing
speed is not its strong point) ... :-)
-Original Message-
From: Upayavira [mailto:u...@odoko.co.uk]
Sent: Tuesday, June 30, 2015 2:46 PM
To: solr-user@lucene.apache.org
Subject: Re: optimize status
On Tue, Jun 30, 2015, at 04:42 PM, Shawn Heisey wrote:
>
On Tue, Jun 30, 2015, at 04:42 PM, Shawn Heisey wrote:
> On 6/29/2015 2:48 PM, Reitzel, Charles wrote:
> > I take your point about shards and segments being different things. I
> > understand that the hash ranges per segment are not kept in ZK. I guess I
> > wish they were.
> >
> > In this r
On 6/29/2015 2:48 PM, Reitzel, Charles wrote:
> I take your point about shards and segments being different things. I
> understand that the hash ranges per segment are not kept in ZK. I guess I
> wish they were.
>
> In this regard, I liked Mongodb, uses a 2-level sharding scheme. Each shard
I've actually seen this happen right in front of my eyes "in the
field". However, that was a very high-performance environment. My
assumption was that fragmented index files were causing more disk
seeks especially for the first-pass query response in distributed
mode. So, if the problem is similar,
We need to work out why your performance is bad without optimise. What
version of Solr are you using? Can you confirm that your config is using
the TieredMergePolicy?
Upayavira
Oe, Jun 30, 2015, at 04:48 AM, Summer Shire wrote:
> Hi Upayavira and Erick,
>
> There are two things we are talking a
Hi Upayavira and Erick,
There are two things we are talking about here.
First: Why am I optimizing? If I don’t our SEARCH (NOT INDEXING) performance is
100% worst.
The problem lies in the number of total segments. We have to have max segments
1 or 2.
I have done intensive performance related
I see what you mean. Many thanks for the details.
-Original Message-
From: Toke Eskildsen [mailto:t...@statsbiblioteket.dk]
Sent: Monday, June 29, 2015 6:36 PM
To: solr-user@lucene.apache.org
Subject: Re: optimize status
Reitzel, Charles wrote:
> Question, Toke: in your "i
Reitzel, Charles wrote:
> Question, Toke: in your "immutable" cases, don't the benefits of
> optimizing come mostly from eliminating deleted records?
Not for us. We have about 1 deleted document for every 1000 or 10.000 standard
documents.
> Is there any material difference in heap, CPU, etc. b
say
2.5?) be a good best practice?
-Original Message-
From: Toke Eskildsen [mailto:t...@statsbiblioteket.dk]
Sent: Monday, June 29, 2015 3:56 PM
To: solr-user@lucene.apache.org
Subject: Re: optimize status
Reitzel, Charles wrote:
> Is there really a good reason to consolidate down t
:15 PM
To: solr-user@lucene.apache.org
Subject: RE: optimize status
" Is there really a good reason to consolidate down to a single segment?"
Archiving (as one example). Come July 1, the collection for log
entries/transactions in June will never be changed, so optimizing is actually a
good thin
For the sake of history, somewhere around Solr/Lucene 3.2 a new
"MergePolicy" was introduced. The old one merged simply based upon age,
or "index generation", meaning the older the segment, the less likely it
would get merged, hence needing optimize to clear out deletes from your
older segments.
T
Reitzel, Charles wrote:
> Is there really a good reason to consolidate down to a single segment?
In the scenario spawning this thread it does not seem to be the best choice.
Speaking more broadly there are Solr setups out there that deals with immutable
data, often tied to a point in time, e.g
ed properly, and I don't think the value used
for routing has anything to do with what segment they happen to be stored
into.
-Original Message-
From: Reitzel, Charles [mailto:charles.reit...@tiaa-cref.org]
Sent: Monday, June 29, 2015 11:38 AM
To: solr-user@lucene.apache.org
Subject: R
here
> ...Or am I all wet (again)?
>
> -Original Message-
> From: Walter Underwood [mailto:wun...@wunderwood.org]
> Sent: Monday, June 29, 2015 10:39 AM
> To: solr-user@lucene.apache.org
> Subject: Re: optimize status
>
> "Optimize" is a manual full mer
-Original Message-
From: Walter Underwood [mailto:wun...@wunderwood.org]
Sent: Monday, June 29, 2015 10:39 AM
To: solr-user@lucene.apache.org
Subject: Re: optimize status
"Optimize" is a manual full merge.
Solr automatically merges segments as needed. This also expunges deleted
“Optimize” is a manual full merge.
Solr automatically merges segments as needed. This also expunges deleted
documents.
We really need to rename “optimize” to “force merge”. Is there a Jira for that?
wunder
Walter Underwood
wun...@wunderwood.org
http://observer.wunderwood.org/ (my blog)
On Jun
Steven:
Yes, but
First, here's Mike McCandles' excellent blog on segment merging:
http://blog.mikemccandless.com/2011/02/visualizing-lucenes-segment-merges.html
I think the third animation is the TieredMergePolicy. In short, yes an
optimize will reclaim disk space. But as you update, this is
Hi Upayavira,
This is news to me that we should not optimize and index.
What about disk space saving, isn't optimization to reclaim disk space or
is Solr somehow does that? Where can I read more about this?
I'm on Solr 5.1.0 (may switch to 5.2.1)
Thanks
Steve
On Mon, Jun 29, 2015 at 4:16 AM,
I'm afraid I don't understand. You're saying that optimising is causing
performance issues?
Simple solution: DO NOT OPTIMIZE!
Optimisation is very badly named. What it does is squashes all segments
in your index into one segment, removing all deleted documents. It is
good to get rid of deletes -
Have to cause of performance issues.
Just want to know if there is a way to tap into the status.
> On Jun 28, 2015, at 11:37 PM, Upayavira wrote:
>
> Bigger question, why are you optimizing? Since 3.6 or so, it generally
> hasn't been requires, even, is a bad thing.
>
> Upayavira
>
>> On Su
Bigger question, why are you optimizing? Since 3.6 or so, it generally
hasn't been requires, even, is a bad thing.
Upayavira
On Sun, Jun 28, 2015, at 09:37 PM, Summer Shire wrote:
> Hi All,
>
> I have two indexers (Independent processes ) writing to a common solr
> core.
> If One indexer process
Hi All,
I have two indexers (Independent processes ) writing to a common solr core.
If One indexer process issued an optimize on the core
I want the second indexer to wait adding docs until the optimize has finished.
Are there ways I can do this programmatically?
pinging the core when the optimi
Hi All,
I have two indexers (Independent processes ) writing to a common solr core.
If One indexer process issued an optimize on the core
I want the second indexer to wait adding docs until the optimize has finished.
Are there ways I can do this programmatically?
pinging the core when the optimi
From: Alexander Rothenberg [mailto:a.rothenb...@fotofinder.net]
Sent: terça-feira, 29 de Junho de 2010 12:20
To: solr-user@lucene.apache.org
Subject: Re: Where to check optimize status
To determine if the optimize is still in progress, you can look at the
admin-frontend on the page "T
To determine if the optimize is still in progress, you can look at the
admin-frontend on the page "THREAD DUMP" for something like "Lucene Merge
Thread". If its there, then optimize is still running. Also, index-filesize
and filenames in your index-dir are changing a lot...
On Tuesday 29 June
Hi,
I'm using solr1.4.0 default installation.
Is there a place where I can find the optimization status.
I sent a optimize http request and it should had finish by now, but I
still see the lock file on index folder.
Can I see somewhere if the optimization is still running?
Thanks,
28 matches
Mail list logo
