hi,
i can't tell you the exact limits but a few hints:
1) check the size of your /tmp dir. while the merging runs (after the
digging)
the tmp-dir will grow (i am not sure but at least to the size of the
normal
htdig-database size)
when i first indexed for the search in http://www.wuppertal-navigator.de
i got the same error (found it somewhere in the logfiles), after setting
the TMPDIR
> In addition to the command line options, the environment
> variable TMPDIR will be used to designate the directory
> where intermediate files are stored during the sorting process.
2) we use htdig to create a searchindex for sites which are about wuppertal
or being made by people from wuppertal. at the moment we index at about
700
different servers/urls with 56907 documents.
so you can imagine that your database get's very big.
as you can see:
123.158.528 Apr 9 10:56 db.docdb
6.147.072 Apr 9 10:56 db.docs.index
157.28.271 Apr 9 10:07 db.images
32.548.231 Apr 9 10:07 db.urls
197.932.056 Apr 9 10:36 db.wordlist
163.500.032 Apr 9 10:36 db.words.db
but i have to say i am intrested in the limits of htdig, too.
the answering time is ok, even it will take a few seconds if you search for
a
very common word like "wuppertal" (
does anyone have more informations?
mfg
Markus Rietzler
* Wuppertal-Forum | Wuppertal-Navigator
* http://www.wuppertal-forum.de
* http://www.wuppertal-navigator.de
* eMail: [EMAIL PROTECTED]
Ronsdorfer Strasse 14 Fon: 0202.42 08 30
42119 Wuppertal Fax: 0202.242 24 66
Mobil: 0171.480 77 82
> -----Original Message-----
> From: Jos Vos [mailto:[EMAIL PROTECTED]]
> Sent: Sunday, April 11, 1999 8:31 PM
> To: [EMAIL PROTECTED]
> Subject: [htdig] DB size boundaries ('new' failed)
>
>
>
> Hi,
>
> What are the practical and/or theoretical limits of ht://Dig?
> I'm trying to index a large site and after some time I always
> get something like "out of memory, 'new' failed" or so.
>
> Is there a workaround for this, what are the real limits here?
>
> Thanks,
>
> --
> -- Jos Vos <[EMAIL PROTECTED]>
> -- X/OS Experts in Open Systems BV | Phone: +31 20 6938364
> -- Amsterdam, The Netherlands | Fax: +31 20 6948204
> ------------------------------------
> To unsubscribe from the htdig mailing list, send a message to
> [EMAIL PROTECTED] containing the single word "unsubscribe" in
> the SUBJECT of the message.
>
------------------------------------
To unsubscribe from the htdig mailing list, send a message to
[EMAIL PROTECTED] containing the single word "unsubscribe" in
the SUBJECT of the message.
