Hi, Solr is fine out of RAM if you don't change it (build and then let it cache what it needs). The RAM is needed when you constantly pepper it with updates and commits. If you can have the logs update certain shards and then merge those indexes periodically to machines you can leave alone - this might be a way to use fewer machines and still get acceptable performance.
If you were going to be happy with Hadoop, then you probably are happy with queries that take longer than a few seconds to complete. You might be happy enough just spreading it across a few machines and write a script to merge indexes when the update machines run out of RAM. You will see a performance change when you run out of RAM but you won't know until you test it to see how bad it is. For me, going from 8GB index to 16GB index was about 10 times slower on a machine with 8GB RAM (around 100ms to just over a second query time). Regards, Ian Connor http://pubget.com On Thu, Aug 6, 2009 at 1:27 AM, Silent Surfer <silentsurfe...@yahoo.com>wrote: > > Hi, > > We initially went with Hadoop path, but as it is one more software based > file system on top of the OS file system, we didn't get a buy in from our > system Engineers. i.e In case if we run into any HDFS issues, SEs won't be > supporting us :( > > Regards, > sS > > --- On Thu, 8/6/09, Walter Underwood <wun...@wunderwood.org> wrote: > > > From: Walter Underwood <wun...@wunderwood.org> > > Subject: Re: Limit of Index size per machine.. > > To: solr-user@lucene.apache.org > > Date: Thursday, August 6, 2009, 5:12 AM > > That is why people don't use search > > engines to manage logs. Look at a > > Hadoop cluster. > > > > wunder > > > > On Aug 5, 2009, at 10:08 PM, Silent Surfer wrote: > > > > > > > > Hi, > > > > > > That means we need approximately 3000 GB (Index > > Size)/24 GB (RAM) = > > > 125 servers. > > > > > > It would be very hard to convince my org to go for 125 > > servers for > > > log management of 3 Terabytes of indexes. > > > > > > Has any one used, solr for processing and handling of > > the indexes of > > > the order of 3 TB ? If so how many servers were used > > for indexing > > > alone. > > > > > > Thanks, > > > sS > > > > > > > > > --- On Wed, 8/5/09, Ian Connor <ian.con...@gmail.com> > > wrote: > > > > > >> From: Ian Connor <ian.con...@gmail.com> > > >> Subject: Re: Limit of Index size per machine.. > > >> To: solr-user@lucene.apache.org > > >> Date: Wednesday, August 5, 2009, 9:38 PM > > >> I try to keep the index directory > > >> size less than the amount of RAM and rely > > >> on the OS to cache as it needs. Linux does a > > pretty good > > >> job here and I am > > >> sure OS X will do a good job also. > > >> > > >> Distributed search here will be your friend so you > > can > > >> chunk it up to a > > >> number of servers to keep your cost down (2GB RAM > > sticks > > >> are much cheaper > > >> than 4GB RAM sticks $20 < $100). > > >> > > >> Ian. > > >> > > >> On Wed, Aug 5, 2009 at 1:44 PM, Silent Surfer > > <silentsurfe...@yahoo.com > > > > >> >wrote: > > >> > > >>> > > >>> Hi , > > >>> > > >>> We are planning to use Solr for indexing the > > server > > >> log contents. > > >>> The expected processed log file size per day: > > 100 GB > > >>> We are expecting to retain these indexes for > > 30 days > > >> (100*30 ~ 3 TB). > > >>> > > >>> Can any one provide what would be the optimal > > size of > > >> the index that I can > > >>> store on a single server, without hampering > > the search > > >> performance etc. > > >>> > > >>> We are planning to use OSX server with a > > configuration > > >> of 16 GB (Can go to > > >>> 24 GB). > > >>> > > >>> We need to figure out how many servers are > > required to > > >> handle such amount > > >>> of data.. > > >>> > > >>> Any help would be greatly appreciated. > > >>> > > >>> Thanks > > >>> SilentSurfer > > >>> > > >>> > > >>> > > >>> > > >>> > > >> > > >> > > >> -- > > >> Regards, > > >> > > >> Ian Connor > > >> 1 Leighton St #723 > > >> Cambridge, MA 02141 > > >> Call Center Phone: +1 (714) 239 3875 (24 hrs) > > >> Fax: +1(770) 818 5697 > > >> Skype: ian.connor > > >> > > > > > > > > > > > > > > > > > > > > > > -- Regards, Ian Connor 1 Leighton St #723 Cambridge, MA 02141 Call Center Phone: +1 (714) 239 3875 (24 hrs) Fax: +1(770) 818 5697 Skype: ian.connor
