Toke Eskildsen [t...@statsbiblioteket.dk] wrote:
[Toke: SSDs with 2.7TB of index on a 256GB machine]
tl;dr: for small result sets ( 1M hits) on unwarmed searches with
simple queries, response time is below 100ms. If we enable faceting with
plain Solr, this jumps to about 1 second.
I did a
Hi Shawn,
Thanks for your response, wanted to clarify a few things.
*Does that mean for querying smoothly we need to have memory atleast equal
or greater to the size of index? As in my case the index size will be very
heavy(~2TB) and practically speaking that amount of memory is not possible.
Mishra
Sent: Wednesday, June 4, 2014 2:45 AM
To: solr-user@lucene.apache.org
Subject: Re: Solr maximum Optimal Index Size per Shard
Thanks all for your response.
I presume this conversation concludes that indexing around 1Billion
documents per shard won't be a problem, as I have 10 Billion docs
On Fri, 2014-06-06 at 12:32 +0200, Vineet Mishra wrote:
*Does that mean for querying smoothly we need to have memory atleast equal
or greater to the size of index?
If you absolutely, positively have to reduce latency as much as
possible, then yes. With an estimated index size of 2TB, I would
Hi Toke,
That was Spectacular, really great to hear that you have already indexed
2.7TB+ data to your server and still the query response time is under ms or
a few seconds for such a huge dataset.
Could you state what indexing mechanism are you using, as I started with
EmbeddedSolrServer but it
On Fri, 2014-06-06 at 14:05 +0200, Vineet Mishra wrote:
Could you state what indexing mechanism are you using, as I started
with EmbeddedSolrServer but it was pretty slow after a few GB(~30+) of
indexing.
I suspect that is due to too-frequent commits, too small heap or
something third,
Earlier I used to index with HtttpPost Mechanism only, making each post
size specific to 2Mb to 20Mb that was going fine, but we had a suspect that
instead of indexing through network call(which ofcourse results in latency
due to network delays and http protocol) if we can index Offline by just
Thanks all for your response.
I presume this conversation concludes that indexing around 1Billion
documents per shard won't be a problem, as I have 10 Billion docs to index,
so approx 10 shards with 1 Billion each should be fine with it and how
about Memory, what size of RAM should be fine for
On 6/4/2014 12:45 AM, Vineet Mishra wrote:
Thanks all for your response.
I presume this conversation concludes that indexing around 1Billion
documents per shard won't be a problem, as I have 10 Billion docs to index,
so approx 10 shards with 1 Billion each should be fine with it and how
about
need to know.
-- Jack Krupansky
-Original Message-
From: Vineet Mishra
Sent: Wednesday, June 4, 2014 2:45 AM
To: solr-user@lucene.apache.org
Subject: Re: Solr maximum Optimal Index Size per Shard
Thanks all for your response.
I presume this conversation concludes that indexing around
How much free system memory do you have for the OS to cache file system
data? If your entire index fits in system memory operations will be fast,
but as your index grows beyond the space the OS can use to cache the data,
performance will decline.
But there's no hard limit in Solr per se.
--
On 6/3/2014 12:54 PM, Jack Krupansky wrote:
How much free system memory do you have for the OS to cache file
system data? If your entire index fits in system memory operations
will be fast, but as your index grows beyond the space the OS can use
to cache the data, performance will decline.
: Tuesday, June 3, 2014 3:34 PM
To: solr-user@lucene.apache.org
Subject: Re: Solr maximum Optimal Index Size per Shard
On 6/3/2014 12:54 PM, Jack Krupansky wrote:
How much free system memory do you have for the OS to cache file
system data? If your entire index fits in system memory operations
On 6/3/2014 1:47 PM, Jack Krupansky wrote:
Anybody care to forecast when hardware will catch up with Solr and we
can routinely look forward to newbies complaining that they indexed
some data and after only 10 minutes they hit this weird 2G document
count limit?
I would speculate that Lucene
mark.
--
View this message in context:
http://lucene.472066.n3.nabble.com/Solr-maximum-Optimal-Index-Size-per-Shard-tp4139565p4139698.html
Sent from the Solr - User mailing list archive at Nabble.com.
15 matches
Mail list logo
