On 10/20/06, Jeff Gortatowsky <[EMAIL PROTECTED]> wrote: > David Balmain wrote: > > On 10/13/06, Jeff Gortatowsky <[EMAIL PROTECTED]> wrote: > > > Hi Jeff, this doesn't sound right at all. Could send a partial listing > > of the directory so I can see what files are in it? Do `ls -l` so I > > > Below is a very very very partial listing. My env is Windows XP Pro. THe > verions of gems is listed below as well. Basically I accessed the first > model object and said model.save! to kick off the indexing. Which it > did. BTW: this is SQLServer if it matters. BTW: The searching the index > works. well...
Ahhh. I've had this problem in Windows before but I thought it was fixed. For some reason the operating system musn't be allowing Ferret to delete the index files when it is finished with them. I'm not sure why this would be happening though. This would gives us approximately 25_000 + 2500 + 250 + 25 + 2 = 27777 files after merging. This is still short of the 28300 files you have though. :-( > I found out when I asked to highlight() that I never get anything back. > Looking at the soruce code and my fields I find I must have had (or > defaulted) to :store=>no so I have to retrieve the row, iterate myself > over the fields to find out which field matched, and then display the > results. That is not pretty but I have to admit, it's painless. This is one of the reasons I want to implement a database based on Ferret, so that operations like this will be very simple. I could add a highlighting method for externally stored fields but you need to store term vectors for the highlighting to work exactly (ie for stemmed terms and matching sloppy phrases exactly) so if you are storing term_vectors, you may as well store the field as well. For externally stored fields the highlighting method you are using is best. > Still > 25,000 records made 28,000+ files. Can you imagine all 8.1 million > records!! Is it because one of the fields being indexed is always unique > (think User ID/Primary key)? No, I think the majority of those files are obselete. In fact I'm not sure if Windows would even allow you to open that many files at once (and Ferret does open all of the files in the index directory.) If you open up the segments file you'll see a list of the segments that are actually still being used by Ferret (along with a bunch of binary data). Given that your segments file is only 29 bytes, I'm guessing that you have optimized your index and you only have one valid index segment. The rest is junk. For the record I indexed 2,000,000 records the other day (approximately 4000kb each) in 2 1/2 hours and I had at most 120 files in my index directory. > I was going to trying it in Lucene and see what happens. I figure if it > is different, the must be doing something odd in Ferret/AaF. Plus I can > try native ferret to create the index and forego AaF for the initial > index creation (assuming that is a 'fix'). Lucene actually records a list of files it fails to delete and continues to try and delete those files. It's a bit of a hack and I was hoping to get away with not doing that in Ferret. Looks like I was wrong. I wonder why it works for me and not for you. I have XP Home edition so it should be the same. > Thank you for any time and effort. I am becoming quite a > Ruby/Rails/Ferret fan for prototyping. I can say as I am ready for Rails > on my production envronoment hosting 40k logged in users a night, but > it's wonderful for concept exploration. > > Here is the partial listing (they are representational of all the other > files except for the last two of which they are the only ones). After > the listing is my gems versions > > 10/11/2006 08:23 PM 1,300 _z.cfs > 10/11/2006 08:25 PM 1,314 _z0.cfs > 10/11/2006 08:25 PM 1,705 _z1.cfs > 10/11/2006 08:26 PM 3,039 _z2.cfs > 10/11/2006 08:26 PM 970 _z3.cfs > 10/11/2006 08:26 PM 3,015 _z4.cfs > 10/11/2006 08:26 PM 14,266 _z5.cfs > 10/11/2006 08:26 PM 770 _z6.cfs > 10/11/2006 08:26 PM 815 _z7.cfs > 10/11/2006 08:26 PM 1,150 _z8.cfs > 10/11/2006 08:26 PM 1,564 _z9.cfs > 10/11/2006 08:26 PM 2,283 _za.cfs > 10/11/2006 08:26 PM 1,259 _zb.cfs > 10/11/2006 08:26 PM 1,598 _zc.cfs > 10/11/2006 08:26 PM 1,655 _zd.cfs > 10/11/2006 08:26 PM 5,466 _ze.cfs > 10/11/2006 08:26 PM 1,242 _zf.cfs > 10/11/2006 08:26 PM 13,609 _zg.cfs > 10/11/2006 08:26 PM 2,081 _zh.cfs > 10/11/2006 08:26 PM 1,101 _zi.cfs > 10/11/2006 08:26 PM 1,053 _zj.cfs > 10/11/2006 08:26 PM 2,208 _zk.cfs > 10/11/2006 08:26 PM 920 _zl.cfs > 10/11/2006 08:26 PM 3,003 _zm.cfs > 10/11/2006 08:26 PM 2,148 _zn.cfs > 10/11/2006 08:26 PM 1,195 _zo.cfs > 10/11/2006 08:26 PM 1,707 _zp.cfs > 10/11/2006 08:26 PM 1,747 _zq.cfs > 10/11/2006 08:26 PM 12,889 _zr.cfs > 10/11/2006 08:26 PM 2,531 _zs.cfs > 10/11/2006 08:26 PM 1,359 _zt.cfs > 10/11/2006 08:26 PM 2,330 _zu.cfs > 10/11/2006 08:26 PM 1,793 _zv.cfs > 10/11/2006 08:26 PM 1,788 _zw.cfs > 10/11/2006 08:26 PM 3,135 _zx.cfs > 10/11/2006 08:26 PM 2,603 _zy.cfs > 10/11/2006 08:26 PM 2,210 _zz.cfs > 10/12/2006 08:39 AM 213 fields > 10/12/2006 08:40 AM 29 segments > 28381 File(s) 261,021,758 bytes > 2 Dir(s) 35,192,127,488 bytes fre > > > > actionmailer (1.2.5), actionpack (1.12.5), actionwebservice (1.1.6) > activerecord (1.14.4), activesupport (1.3.1), ferret (0.10.9), > fxri (0.3.3), fxruby (1.6.2, 1.6.1, 1.6.0, 1.2.6), gem_plugin (0.2.1) > log4r (1.0.5), mongrel (0.3.13.3) > rails (1.1.6), rake (0.7.1) > sources (0.0.1), win32-clipboard (0.4.1, 0.4.0) > win32-dir (0.3.0) > win32-eventlog (0.4.2, 0.4.1) > win32-file (0.5.2) > win32-file-stat (1.2.2) > win32-process (0.5.1, 0.4.2) > win32-sapi (0.1.3) > win32-service (0.5.0) > win32-sound (0.4.0) > windows-pr (0.5.4, 0.5.1) > > -- > Posted via http://www.ruby-forum.com/. > _______________________________________________ > Ferret-talk mailing list > [email protected] > http://rubyforge.org/mailman/listinfo/ferret-talk > _______________________________________________ Ferret-talk mailing list [email protected] http://rubyforge.org/mailman/listinfo/ferret-talk
