We run a central database of 14M (and growing) photos with dates, captions, keywords, etc.
We currently upgrading from old Lucene Servers to latest Solr running with a couple of dedicated servers (6 core, 36GB, 500SSD). Planning on using Solr Cloud. We take in thousands of changes each day (big and small) so indexing may be a bigger problem than searching. My question is an architecture one. These photos are currently indexed and searched in three ways. 1: The 14M pictures from above are split into a few hundred indexes that feed a single website. This means index sizes of between 100 and 500,000 entries each. 2: 95% of these same photos are also wanted for searching on a global site. Index size of 12M plus. 3: 80% of these same photos are also required for smaller group sites. Index sizes of between 400K and 4M. We currently make changes the single indexes and then merge into groups and global. Due to the size of the numbers, is it worth changing or not. Is it quicker/better to just have one big 14M index and filter the complexities for each website or is it better to still maintain hundreds of indexes so we are searching smaller one. Bear in mind, we get thousands of changes a day PLUS very busy search servers. Thanks Col -- View this message in context: http://lucene.472066.n3.nabble.com/Newbie-Question-Master-Index-or-100s-Small-Index-tp4125407.html Sent from the Solr - User mailing list archive at Nabble.com.
