date:20140606

Date number to text

2014-06-06 Thread O. Klein

Is there a way to tokenize 12-6-2014 as 12 june 2014 and the other way around
as well?

Making it possible for people to search on text and numbers for dates?



--
View this message in context: 
http://lucene.472066.n3.nabble.com/Date-number-to-text-tp4140323.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Solr maximum Optimal Index Size per Shard

2014-06-06 Thread Vineet Mishra

Hi Shawn,

Thanks for your response, wanted to clarify a few things.

*Does that mean for querying smoothly we need to have memory atleast equal
or greater to the size of index? As in my case the index size will be very
heavy(~2TB) and practically speaking that amount of memory is not possible.
Even If it goes to multiple shards, say around 10 Shards then also 200GB of
RAM will not be an feasible option.

*With CloudSolrServer can we specify which Shard the particular index
should go and reside, which I can do with EmbeddedSolrServer by indexing in
different directories and moving them to appropriate shard directories.

Thanks!

On Wed, Jun 4, 2014 at 12:43 PM, Shawn Heisey s...@elyograg.org wrote:

On 6/4/2014 12:45 AM, Vineet Mishra wrote:
Thanks all for your response.
I presume this conversation concludes that indexing around 1Billion
documents per shard won't be a problem, as I have 10 Billion docs to
index,
so approx 10 shards with 1 Billion each should be fine with it and how
about Memory, what size of RAM should be fine for this amount of data?

Figure out the heap requirements of the operating system and every
program on the machine (Solr especially). Then you would add that
number to the total size of the index data on the machine. That is the
ideal minimum RAM.

http://wiki.apache.org/solr/SolrPerformanceProblems

Unfortunately, if you are dealing with a huge index with billions of
documents, it is likely to be prohibitively expensive to buy that much
RAM. If you are running Solr on Amazon's cloud, the cost for that much
RAM would be astronomical.

Exactly how much RAM would actually be required is very difficult to
predict. If you had only 25% of the ideal, your index might have
perfectly acceptable performance, or it might not. It might do fine
under a light query load, but if you increase to 50 queries per second,
performance may drop significantly ... or it might be good. It's
generally not possible to know how your hardware will perform until you
actually build and use your index.

http://searchhub.org/2012/07/23/sizing-hardware-in-the-abstract-why-we-dont-have-a-definitive-answer/

A general rule of thumb for RAM that I have found to be useful is that
if you've got less than half of the ideal memory size, you might have
performance problems.

Moreover what should be the indexing technique for this huge data set, as
currently I am indexing with EmbeddedSolrServer but its going
pathetically
slow after some 20Gb of indexing. Comparatively SolrHttpPost was slow due
to network delays and response but after this long running the indexing
with EmbeddedSolrServer I am getting a different notion.
Any good indexing technique for this huge dataset would be highly
appreciated.

EmbeddedSolrServer is not recommended. Run Solr in the traditional way
with HTTP connectivity. HTTP overhead on a LAN is usually quite small.
Solr is fully thread-safe, so you can have several indexing threads all
going at the same time.

Indexes at this scale should normally be built with SolrCloud, with
enough servers so that each machine is only handling one shard replica.
The ideal indexing program would be written in Java, using
CloudSolrServer.

59 matches

Mail list logo