Wilburn, Scott [scott.wilb...@verizonwireless.com.INVALID] wrote: > Thanks for your suggestion to look into MapReduceIndexerTool, I'm looking > into that now. > I agree what I am trying to do is a tall order, and the more I hear from all > of your > comments, the more I am convinced that lack of memory is my biggest problem. > I'm going to work on increasing the memory now, but was wondering if there are > any configuration or other techniques that could also increase ingest > performance?
More RAM basically compensates for slow storage, so the obvious trick is to increase your I/O performance. If your index is placed on network storage, then put it on local storage. If you are using spinning drives, then change to SSDs. If you are using SSDs then RAID them. Way cheaper than trying to match your RAM with your projected index size. > Does anyone know if a cloud of this size( hundreds of billions ) with an > ingest rate of 5 billion new each day, has ever been attempted before? Sorry, my experience is primarily with maximizing search performance. - Toke Eskildsen
