Re: Solr Replication: disk space consumed on slave much higher than on master

2009-01-25 Thread Jaco
Hi Noble, Great stuff, no problem, I really think the Solr development team is excellent and takes pride in delivering high quality software! And we're going into production with a brand new Solr based system in a few weeks as well, so I'm really happy that this is fixed now. Bye, Jaco.

Re: Solr Replication: disk space consumed on slave much higher than on master

2009-01-23 Thread Jaco
Hi, I applied the patch and did some more tests - also adding some LOG.info() calls in delTree to see if it actually gets invoked (LOG.info(START: delTree: +dir.getName()); at the start of that method). I don't see any entries of this showing up in the log file at all, so it looks like delTree

Re: Solr Replication: disk space consumed on slave much higher than on master

2009-01-23 Thread Shalin Shekhar Mangar
On Fri, Jan 23, 2009 at 2:12 PM, Jaco jdevr...@gmail.com wrote: Hi, I applied the patch and did some more tests - also adding some LOG.info() calls in delTree to see if it actually gets invoked (LOG.info(START: delTree: +dir.getName()); at the start of that method). I don't see any entries

Re: Solr Replication: disk space consumed on slave much higher than on master

2009-01-23 Thread Noble Paul നോബിള്‍ नोब्ळ्
I tested with the patch it has solved both the issues On Fri, Jan 23, 2009 at 5:00 PM, Shalin Shekhar Mangar shalinman...@gmail.com wrote: On Fri, Jan 23, 2009 at 2:12 PM, Jaco jdevr...@gmail.com wrote: Hi, I applied the patch and did some more tests - also adding some LOG.info() calls in

Re: Solr Replication: disk space consumed on slave much higher than on master

2009-01-23 Thread Noble Paul നോബിള്‍ नोब्ळ्
I have opened an issue to track this https://issues.apache.org/jira/browse/SOLR-978 On Fri, Jan 23, 2009 at 5:22 PM, Noble Paul നോബിള്‍ नोब्ळ् noble.p...@gmail.com wrote: I tested with the patch it has solved both the issues On Fri, Jan 23, 2009 at 5:00 PM, Shalin Shekhar Mangar

Re: Solr Replication: disk space consumed on slave much higher than on master

2009-01-23 Thread Jaco
Hi, I have tested this as well, looking fine! Both issues are indeed fixed, and the index directory of the slaves gets cleaned up nicely. I will apply the changes to all systems I've got running and report back in this thread in case any issues are found. Thanks for the very fast help! I usually

Re: Solr Replication: disk space consumed on slave much higher than on master

2009-01-22 Thread Jaco
Hm, I don't know what to do anymore. I tried this: - Run Tomcat service as local administrator to overcome any permissioning issues - Installed latest nightly build (I noticed that item I mentioned before ( http://markmail.org/message/yq2ram4f3jblermd) had been committed which is good - Build a

Re: Solr Replication: disk space consumed on slave much higher than on master

2009-01-22 Thread Shalin Shekhar Mangar
On Thu, Jan 22, 2009 at 10:18 PM, Jaco jdevr...@gmail.com wrote: Hm, I don't know what to do anymore. I tried this: - Run Tomcat service as local administrator to overcome any permissioning issues - Installed latest nightly build (I noticed that item I mentioned before (

Re: Solr Replication: disk space consumed on slave much higher than on master

2009-01-22 Thread Jeff Newburn
We are seeing something very similar. Ours is intermittent and usually happens a great deal on random days. Often it seems to occur during large index updates on the master. On 1/22/09 8:58 AM, Shalin Shekhar Mangar shalinman...@gmail.com wrote: On Thu, Jan 22, 2009 at 10:18 PM, Jaco

Re: Solr Replication: disk space consumed on slave much higher than on master

2009-01-22 Thread Shalin Shekhar Mangar
On Thu, Jan 22, 2009 at 10:37 PM, Jeff Newburn jnewb...@zappos.com wrote: We are seeing something very similar. Ours is intermittent and usually happens a great deal on random days. Often it seems to occur during large index updates on the master. Jeff, is this also on a Windows box? --

Re: Solr Replication: disk space consumed on slave much higher than on master

2009-01-22 Thread Jeff Newburn
My apologies. No we are using linux, tomcat setup. On 1/22/09 9:15 AM, Shalin Shekhar Mangar shalinman...@gmail.com wrote: On Thu, Jan 22, 2009 at 10:37 PM, Jeff Newburn jnewb...@zappos.com wrote: We are seeing something very similar. Ours is intermittent and usually happens a great deal

Re: Solr Replication: disk space consumed on slave much higher than on master

2009-01-22 Thread Noble Paul നോബിള്‍ नोब्ळ्
Jeff , Do you see both the empty index. dirs as well as the extra files in the index? --Noble On Thu, Jan 22, 2009 at 10:37 PM, Jeff Newburn jnewb...@zappos.com wrote: We are seeing something very similar. Ours is intermittent and usually happens a great deal on random days. Often it seems

Re: Solr Replication: disk space consumed on slave much higher than on master

2009-01-22 Thread Jeff Newburn
We have both. A majority of them are just empty but others have almost a full index worth of files. I have also noticed that during a lengthy index update the system will throw errors about how it cannot move one of the index files. Essentially on reindex the system does not replicate until an

Re: Solr Replication: disk space consumed on slave much higher than on master

2009-01-22 Thread Noble Paul നോബിള്‍ नोब्ळ्
This was reported by another user and was fixed recently.Are you using a recent version? --Noble On Fri, Jan 23, 2009 at 12:00 AM, Jeff Newburn jnewb...@zappos.com wrote: We have both. A majority of them are just empty but others have almost a full index worth of files. I have also noticed

Re: Solr Replication: disk space consumed on slave much higher than on master

2009-01-22 Thread Jeff Newburn
Few weeks ago is our version. Does this contribute to the directory issues and extra files that are left? On 1/22/09 10:33 AM, Noble Paul നോബിള്‍ नोब्ळ् noble.p...@gmail.com wrote: This was reported by another user and was fixed recently.Are you using a recent version? --Noble On Fri,

Re: Solr Replication: disk space consumed on slave much higher than on master

2009-01-22 Thread Noble Paul നോബിള്‍ नोब्ळ्
I am not sure if it was completely fixed. (This was related to a Lucene bug) But you can try w/ a recent build and confirm it for us. I have never encountered these during our tests in windows XP/Linux I have attached a patch which logs the names of the files which could not get deleted (which

Re: Solr Replication: disk space consumed on slave much higher than on master

2009-01-22 Thread Shalin Shekhar Mangar
On Fri, Jan 23, 2009 at 12:15 AM, Noble Paul നോബിള്‍ नोब्ळ् noble.p...@gmail.com wrote: I have attached a patch which logs the names of the files which could not get deleted (which may help us diagnose the problem). If you are comfortable applying a patch you may try it out. I've committed

Solr Replication: disk space consumed on slave much higher than on master

2009-01-21 Thread Jaco
Hi, I'm running Solr nightly build of 20.12.2008, with patch as discussed on http://markmail.org/message/yq2ram4f3jblermd, using Solr replication. On various systems running, I see that the disk space consumed on the slave is much higher than on the master. One example: - Master: 30 GB in 138

Re: Solr Replication: disk space consumed on slave much higher than on master

2009-01-21 Thread Rafał Kuć
Hello, Hi, I'm running Solr nightly build of 20.12.2008, with patch as discussed on http://markmail.org/message/yq2ram4f3jblermd, using Solr replication. On various systems running, I see that the disk space consumed on the slave is much higher than on the master. One example: - Master:

Re: Solr Replication: disk space consumed on slave much higher than on master

2009-01-21 Thread Noble Paul നോബിള്‍ नोब्ळ्
the index.xxx directories are supposed to be deleted (automatically). you can safely delete them. But, I am wondering why the index files in the slave did not get deleted. By default the deletionPolicy is KeepOnlyLastCommit. On Wed, Jan 21, 2009 at 2:15 PM, Jaco jdevr...@gmail.com wrote: Hi,

Re: Solr Replication: disk space consumed on slave much higher than on master

2009-01-21 Thread Shalin Shekhar Mangar
Hi, There shouldn't be so many files on the slave. Since the empty index.x folders are not getting deleted, is it possible that Solr process user does not enough privileges to delete files/folders? Also, have you made any changes to the IndexDeletionPolicy configuration? On Wed, Jan 21,

Re: Solr Replication: disk space consumed on slave much higher than on master

2009-01-21 Thread Jaco
Thanks for the fast replies! It appears that I made a (probably classical) error... I didnt' make the change to solrconfig.xml to include the deletionPolicy when applying the upgrade. I include this now, but the slave is not cleaning up. Will this be done at some point automatically? Can I

Re: Solr Replication: disk space consumed on slave much higher than on master

2009-01-21 Thread Noble Paul നോബിള്‍ नोब्ळ्
On Wed, Jan 21, 2009 at 3:42 PM, Jaco jdevr...@gmail.com wrote: Thanks for the fast replies! It appears that I made a (probably classical) error... I didnt' make the change to solrconfig.xml to include the deletionPolicy when applying the upgrade. I include this now, but the slave is not