Re: Distributed Indexes, Searches and HDFS

2006-09-22 Thread Chris D
Afternoon (here anyway), I think understanding Solr's overall approach (whose design I believe came out of the thread you've referenced) is also a good step here. Even if you can't re-use the hard links trick, you might be able to reuse its snapshotting & index distribution protocol. I'll ha

Distributed Indexes, Searches and HDFS

2006-09-21 Thread Chris D
Hi List, As a bit of an experiment I'm redoing some of our indexing and searching code to try to make it easier to manage and distributed. The system has to modify its indexes frequently, sometimes in huge batches, and the documents in the indexes are frequently modified (deleted, modified and re