I'll tell you what: here's _my_ personal experience.
I'm indexing many many gigabytes of data. Currently it's at ~20, but it will
go up in the near[ish] future. When I get a cluster to run it on, and the
correct access to the other files, etc.
I'm using crc-multi and most of my queries are coming back in under a
second.
The average is only quite high because I happen to know that if I search for
"index || perl", I absolutely trounce the system because thos're about the
two most popular words.
Anyway.
I'm using CRC-multi, and I've added a few features [eg the query trakcing,
more useful server tables, and some other stuff], and it's still fast enough
to not worry about.
I haven't done any proper stress,testing yet, because the server _cannot_ go
down. But from the behavior I'm seeing, it shouldn't be too bad.
Gary (-;
> -----Original Message-----
> From: Caffeinate The World [SMTP:[EMAIL PROTECTED]]
> Sent: Tuesday, March 13, 2001 4:41 PM
> To: [EMAIL PROTECTED]
> Subject: crc-multi and millions of urls
>
> is there any reason why you can't index millions of urls using the DB
> crc-multi mode? is it speed? when i first started using mnogosearch, i
> was under the assumption that if you were to index millions of urls,
> you should use cachemode. now that i've run into several limitations of
> cachemode itself:
>
> 1. limited depth for categories,
> 2. unreliable -- i've yet to have a fully indexed
> service -- seems like i've been debugging for months
> and indexer, cachelogd, splitter still core dumps, some
> were fixed and new ones showed up.
>
> so now i'm back to investigating the use of sql db instead. i do like
> the speed i see in cachemode, but the unreliabililty doesn't make it
> usable.
>
> __________________________________________________
> Do You Yahoo!?
> Yahoo! Auctions - Buy the things you want at great prices.
> http://auctions.yahoo.com/
> ___________________________________________
> If you want to unsubscribe send "unsubscribe general"
> to [EMAIL PROTECTED]
>
___________________________________________
If you want to unsubscribe send "unsubscribe general"
to [EMAIL PROTECTED]
