>Mike, I'm reindexing 9890 sites a day (those days that everything >works allright). The first run took much more, but it seems that yours is >taking too much time to complete.
Ah, finally, someone who's running a similar task. I can see that the system is constantly connecting to a good dozen systems or so and pulling down what ever it's indexing. I know it's alive and I can also see that the drive space is constantly on the rise so it's not being slow about it that I can tell? >Now I'm running on a RedHat 7.3, but before I was using FreeBSD on a >fairly similar configuration and it was many times slower than now. I'm running on RedHat 7.2 myself, with the dual 350Mhz CPU's and 8GB's of memory as I mentioned. The system seems to be handing everything ok, the load is not very high or anything but it's been running non stop for over two weeks now. >Some times seems that stoping and restarting the indexer is faster than >simply wait the end of the run. I think I started the process from the console and left it there. I've been monitoring the system from an SSH login. My point is, I can't quit the session unless I hit CTL-C or is there some other way, from another login? >The disk space depends on the content of the pages, but on my case >it is using 1GB for the cache and 2GB for the database. I know it's into the millions of pages because there is constantly a queue of between 1.2 and 1.5 million pages. It's already into 45GB's of drive space usage now. I'm indexing 10028 sites, not pages but I notice you also said you are indexing 9000+ *sites*. I had assumed that the drive usage would be pretty normal for the number of sites that I'm indexing. >By the way, I use the following mysql command to report top 10 sites: >sometimes there's a huge site that shouldn't be indexed, which is >consuming lots of resources. I'll check this out. >Hope some of this may help you. Invaluable, thanks very much for your help. Mike
