Re: how many shards required to search data
On 2/21/2014 1:39 AM, search engn dev wrote: > As you suggestedI have indexed 12million sample records in solr on hardware > of 8gb ram. Size of index is 3gb. > can i extrapolate this to predict actual size of index.? If the sizes of those records are about the same size as the records in the system as a whole, you can probably use that to extrapolate. Based on that, I would guess that the index is probably going to be about 85GB. That's a lot less than I would have guessed, so perhaps there's a lot of extra stuff in that 250GB that doesn't actually get sent to Solr. Even though they are small, the number of documents will probably require a larger Java heap than the relatively small index size would normally require. Do you have any kind of notion as to what kind of query volume you're going to have? If it's low, you can put multiple shards on your multi-cpu machines and take advantage of parallel processing. If the query volume is high, you'll need all those cpus to handle the load of one shard, and you might need more than two machines for each shard. You'll want to shard your index even though it's relatively small in terms of disk space, because a billion documents is a LOT. If you're just starting out, SolrCloud is probably a good way to go. It handles document routing across shards for you. You didn't say whether that was your plan or not. Thanks, Shawn
Re: how many shards required to search data
As you suggestedI have indexed 12million sample records in solr on hardware of 8gb ram. Size of index is 3gb. can i extrapolate this to predict actual size of index.? -- View this message in context: http://lucene.472066.n3.nabble.com/how-many-shards-required-to-search-data-tp4118715p4118753.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: how many shards required to search data
On 2/20/2014 9:24 PM, search engn dev wrote: > Data size is 250 GB of small records. each record is of around 0.3kb size. It > consists around 1 billion records. my index has 20 different fields. . > Majorly queries will be very simple or spacial queries mainly on on 2-3 > fields. > all 20 fields will be stored. Any suggestions on how many shards will I need > to search data? Your question is impossible to answer. I will tell you that this is a very big index, and it's going to take a lot of hardware. It's not the biggest I've heard of, but it is quite large. Any situation that would result in a performance issue on a small index is going to be far worse on a large index. http://searchhub.org/2012/07/23/sizing-hardware-in-the-abstract-why-we-ont-have-a-definitive-answer/ Two machines with 32GB of RAM each are not going to be anywhere near enough. If you can't get more than 32GB of RAM in each server, you're probably going to need a lot of them. Since all your fields will be stored, the *minimum* size of your index will be approximately equal to the original data size after compression -- assuming you're using 4.1.0 or later, where compression was introduced. That will not be the end, though -- it doesn't take into account the size of the *indexed* data. Although it is theoretically possible to look at a schema and the original data to calculate the size of the indexed data, in reality the only way to be SURE is to actually index a significant percentage of your real data with the same schema you would use in production. Once you know how big your index is actually going to be, you can begin to figure out how much total RAM you'll need across all the servers for a single copy of the index (no redundancy). If you want redundancy, the requirements will be at least twice what you calculate. http://wiki.apache.org/solr/SolrPerformanceProblems The number of shards and replicas that you're going to need is going to depend on the query volume, the nature of the queries, and the nature of the data. Just like with index size, the only way to know is to try it with all your real data. If your query volume is large, you'll need multiple copies of the complete index, which means more servers. If you don't care how long each query takes and your query volume will be low, then your server requirements will be a LOT smaller. Thanks, Shawn
Re: how many shards required to search data
Shard1 config : Config: 32GB RAM, 4 core Shard2 config : Config: 32GB RAM, 4 core -- View this message in context: http://lucene.472066.n3.nabble.com/how-many-shards-required-to-search-data-tp4118715p4118717.html Sent from the Solr - User mailing list archive at Nabble.com.
how many shards required to search data
Data size is 250 GB of small records. each record is of around 0.3kb size. It consists around 1 billion records. my index has 20 different fields. . Majorly queries will be very simple or spacial queries mainly on on 2-3 fields. all 20 fields will be stored. Any suggestions on how many shards will I need to search data? -- View this message in context: http://lucene.472066.n3.nabble.com/how-many-shards-required-to-search-data-tp4118715.html Sent from the Solr - User mailing list archive at Nabble.com.