James thanks .
If this is true the place to fix this is in
ReplicationHandler#getFileList(). patch is attached.
On Wed, Dec 24, 2008 at 4:04 PM, James Grant <james.gr...@semantico.com> wrote:
> I had the same problem. It turned out that the list of files from the master
> included duplicates. When the slave completes the download and tries to move
> the files into the index it comes across a file that does not exist because
> it has already been moved so it backs out the whole operation.
>
> My solution for now was to patch the copyindexFiles method of
> org.apache.solr.handler.SnapPuller so that it normalises the list before
> moving the files. This isn't the best solution since it will still download
> the file twice but it was the easiest and smallest change to make. The patch
> is below
>
> Regards
>
> James
>
> --- src/java/org/apache/solr/handler/SnapPuller.java (revision 727347)
> +++ src/java/org/apache/solr/handler/SnapPuller.java (working copy)
> @@ -470,7 +470,7 @@
> */
> private boolean copyIndexFiles(File snapDir, File indexDir) {
> String segmentsFile = null;
> - List<String> copiedfiles = new ArrayList<String>();
> + Set<String> filesToCopy = new HashSet<String>();
> for (Map<String, Object> f : filesDownloaded) {
> String fname = (String) f.get(NAME);
> // the segments file must be copied last
> @@ -482,6 +482,10 @@
> segmentsFile = fname;
> continue;
> }
> + filesToCopy.add(fname);
> + }
> + List<String> copiedfiles = new ArrayList<String>();
> + for (String fname: filesToCopy) {
> if (!copyAFile(snapDir, indexDir, fname, copiedfiles)) return false;
> copiedfiles.add(fname);
> }
>
>
> Jaco wrote:
>>
>> Hello,
>>
>> While testing out the new replication features, I'm running into some
>> strange problem. On the slave, I keep getting an error like this after all
>> files have been copied from the master to the temporary index.xxxxxxxxx
>> directory:
>>
>> SEVERE: Unable to move index file from:
>> D:\Data\solr\Slave\data\index.20081224110855\_21e.tvx to:
>> D:\Data\Solr\Slave\data\index\_21e.tvx
>>
>> The replication then stops, index remains in original state, so the
>> updates
>> are not available at the slave.
>>
>> This is my replication config at the master:
>>
>> <requestHandler name="/replication" class="solr.ReplicationHandler" >
>> <lst name="master">
>> <!--Replicate on 'optimize' it can also be 'commit' -->
>> <str name="replicateAfter">commit</str>
>> <str name="confFiles">schema.xml</str>
>> </lst>
>> </requestHandler>
>>
>> This is the replication config at the slave:
>>
>> <requestHandler name="/replication" class="solr.ReplicationHandler" >
>> <lst name="slave">
>> <str name="masterUrl">
>> http://hostnamemaster:8080/solr/Master/replication</str>
>> <str name="pollInterval">00:10:00</str>
>> <str name="zip">true</str>
>> </lst>
>> </requestHandler>
>>
>> I'm running a Solr nightly build of 21.12.2008 in Tomcat 6 on Windows
>> 2003.
>> Initially I thought there was some problem with disk space, but this is
>> not
>> the case. Replication did run fine for intial version of index, but after
>> that at some point it didn't work anymore. Any ideas what could be wrong
>> here?
>>
>> Thanks very much in advance, bye,
>>
>> Jaco.
>>
>>
>
>
--
--Noble Paul
Index: src/java/org/apache/solr/handler/ReplicationHandler.java
===================================================================
--- src/java/org/apache/solr/handler/ReplicationHandler.java (revision 729282)
+++ src/java/org/apache/solr/handler/ReplicationHandler.java (working copy)
@@ -268,7 +268,7 @@
List<Map<String, Object>> result = new ArrayList<Map<String, Object>>();
try {
//get all the files in the commit
- Collection<String> files = commit.getFileNames();
+ Collection<String> files = new HashSet<String>(commit.getFileNames());
for (String fileName : files) {
File file = new File(core.getIndexDir(), fileName);
Map<String, Object> fileMeta = getFileInfo(file);
