Unfortunately I don't have lots of solutions for you, because I'm still not having a so big index! But it sounds like the weak point is disk access during the copy? Try to cache the index in memory? (needs a lot of ram!) Or having two HDD on your searcher, one for current index, the other for 'incoming index' (so during the copy the other drive can still have good access time) Or having a really big and powerful RAID system with SCSI 15k disks?
I'm really interested to heard more details about your automated configuration, what kind of copy you use? (crontab? batch?) how do you detect the new index on the searcher? what are your merge tactics?, etc ... (If you can share I'm sure it will be useful lots of nutch beginners like me! thanks) thanks Chun Wei Ho wrote: > > We are running a search service on the internet using two machines. We > have a crawler machine which crawls the web and merges new documents > found into the Lucene index. We have a searcher machine which allows > users to perform searches on the Lucene index. > > Periodically, we would copy the newest version of the index from the > crawler machine over to the searcher machine (via copy over a NFS > mount). The searcher would then detect the new version, close the old > index, open the new index and resume the search service. > > As the index have been growing in size, we have been noticing that the > search response time on the searcher machine increases drastically > when an index (about 15GB) is being copied from the crawler to the > searcher. Both machines run Fedora Core 4 and are on a gbps lan. > > We've tried a number of ways to reduce the impact of the copy over NFS > on searching performance, such as "nice"ing the copy process, but to > no avail. I wonder if anyone is running a lucene search service over a > similar architecture and how you are managing the updates to the > lucene index. > > Thanks! > > Regards, > CW > > -- View this message in context: http://www.nabble.com/Index-updates-between-machines-tf3514574.html#a9816799 Sent from the Nutch - User mailing list archive at Nabble.com. ------------------------------------------------------------------------- Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT & business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV _______________________________________________ Nutch-general mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/nutch-general
