---------- Forwarded message ---------- From: Steven A Rowe <sar...@syr.edu> Date: 2011/3/11 Subject: RE: Problem of Replication Reservation Duration To: "solr-...@lucene.apache.org" <solr-...@lucene.apache.org>
Hi Li Li, Please do not use the solr-dev mailing list - Solr and Lucene development both use the <dev at lucene.apache.org> list. Steve > -----Original Message----- > From: Li Li [mailto:fancye...@gmail.com] > Sent: Friday, March 11, 2011 8:41 AM > To: solr-...@lucene.apache.org > Subject: Problem of Replication Reservation Duration > > hi all, > The replication handler in solr 1.4 which we used seems to be a > little problematic in some extreme situation. > The default reserve duration is 10s and can't modified by any method. > private Integer reserveCommitDuration = > SnapPuller.readInterval("00:00:10"); > The current implementation is: slave send a http > request(CMD_GET_FILE_LIST) to ask server list current index files. > In the response codes of master, it will reserve this commit for 10s. > // reserve the indexcommit for sometime > core.getDeletionPolicy().setReserveDuration(version, > reserveCommitDuration); > If the master's indexes are changed within 10s, the old version > will not be deleted. Otherwise, the old version will be deleted. > slave then get the files in the list one by one. > considering the following situation. > Every mid-night we optimize the whole indexes into one single > index, and every 15 minutes, we add new segments to it. > e.g. when the slave copy the large optimized indexes, it will cost > more than 15 minutes. So it will fail to copy all files and > retry 5 minutes later. But each time it will re-copy all the files > into a new tmp directory. it will fail again and again as long as > we update indexes within 15 minutes. > we can tack this problem by setting reserveCommitDuration to 20 > minutes. But then because we update small number of > documents very frequently, many useless indexes will be reserved and > it's a waste of disk space. > Any one confronted the problem before and is there any solution for > it? > We comes up a ugly solution like this: slave fetches files using > multithreads. each file a thread. Thus master will open all the > files that slave needs. As long as the file is opened. when master > want to delete them, these files will be deleted. But the inode > reference count is larger than 0. Because reading too many files by > master will decrease the ability of master. we want to use > some synchronization mechanism to allow only 1 or 2 ReplicationHandler > threads are doing CMD_GET_FILE command. > Is that solution feasible? > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org > For additional commands, e-mail: dev-h...@lucene.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
