Re: [htdig] Quick Question about using htdig

2000-01-07 Thread Bill Carlson
On Fri, 7 Jan 2000, Deepak Vaidya wrote: > On Fri, 7 Jan 2000, Torsten Neuer wrote: > > > Geoff Hutchison wrote: > > > > > > At 11:05 AM -0500 1/7/00, Deepak Vaidya wrote: > > > >The problem I am having is the list archives are password proctected and > > > >using authorization: username:passwo

Re: [htdig] indexing

2000-01-07 Thread Max de Mendizábal
dou u hav simbolic links? Saludos. To unsubscribe from the htdig mailing list, send a message to [EMAIL PROTECTED] You will receive a message to confirm this.

Re: [htdig] Quick Question about using htdig

2000-01-07 Thread Geoff Hutchison
At 8:47 PM +0100 1/7/00, Torsten Neuer wrote: >Hmm.. wasn't it that Ht://Dig was only capable of doing Basic >authentication >so far? If the Digest authentication method is used, it might be the >reason >for barfing on that. You are correct. I made the assumption that it wasn't Digest since tha

Re: [htdig] Quick Question about using htdig

2000-01-07 Thread Deepak Vaidya
On Fri, 7 Jan 2000, Torsten Neuer wrote: > Geoff Hutchison wrote: > > > > At 11:05 AM -0500 1/7/00, Deepak Vaidya wrote: > > >The problem I am having is the list archives are password proctected and > > >using authorization: username:password did not work was planned. What > > >could I be doing

Re: [htdig] Quick Question about using htdig

2000-01-07 Thread Torsten Neuer
Geoff Hutchison wrote: > > At 11:05 AM -0500 1/7/00, Deepak Vaidya wrote: > >The problem I am having is the list archives are password proctected and > >using authorization: username:password did not work was planned. What > >could I be doing wrong. > > How is the password protection implemente

RE: [htdig] indexing

2000-01-07 Thread David Schwartz
> At 11:01 AM -0800 1/7/00, David Schwartz wrote: > > The 'htdig' process consumes more and more memory as it runs. > >This might be > >due to memory leaks, or it might be legimitately due to it > > keeping track of > >all the URLs it has to process. I tried htdigging 250,000 > > documents and

RE: [htdig] Quick Question about using htdig

2000-01-07 Thread David Schwartz
> How is the password protection implemented? If it's via server-side > passwords (e.g. the browser throws up a dialog box) this should work. > If it's via some sort of CGI, it's not easy to do--you'll have to > figure out some way to authenticate the htdig robot to the CGI. Easiest way i

RE: [htdig] indexing

2000-01-07 Thread Geoff Hutchison
At 11:01 AM -0800 1/7/00, David Schwartz wrote: > The 'htdig' process consumes more and more memory as it runs. >This might be >due to memory leaks, or it might be legimitately due to it keeping track of >all the URLs it has to process. I tried htdigging 250,000 documents and hit >about 180

Re: [htdig] multiple SGML tags for noindex_start noindex_end

2000-01-07 Thread Geoff Hutchison
At 10:33 AM -0600 1/7/00, Glenn Nielsen wrote: >But we are indexing 10 servers with close to 200 virtual hosts plus a >dozen or so external servers we don't administer. We have no way to >force the 100's of people/organizations that publish content to do >this. I am looking at this from a server

Re: [htdig] Quick Question about using htdig

2000-01-07 Thread Geoff Hutchison
At 11:05 AM -0500 1/7/00, Deepak Vaidya wrote: >The problem I am having is the list archives are password proctected and >using authorization: username:password did not work was planned. What >could I be doing wrong. How is the password protection implemented? If it's via server-side passwords

Re: [htdig] Indexing a whole Intranet

2000-01-07 Thread Geoff Hutchison
At 2:35 PM +0100 1/7/00, Paul COURBIS wrote: >Does everyone use htdig to index a whole Intranet (about 1000+ servers) ? Probably not "everyone," but this is fairly common. >Any advices on that subject ? How should I run htdig to be able to >update database & to add new servers when requested in

Re: [htdig] indexing

2000-01-07 Thread Geoff Hutchison
At 11:58 AM +0100 1/7/00, Rickard Lundgren wrote: >I run htidg och MacOS X Server with a G3 with 384 MB of physical memory, >when htdig hangs the server the database is about 800 MB large, is it >possible that the site and hence the databse is to large to handle for htdig >and the hardware? Someo

RE: [htdig] indexing

2000-01-07 Thread David Schwartz
> > I run htidg och MacOS X Server with a G3 with 384 MB of physical memory, > > when htdig hangs the server the database is about 800 MB large, is it > > possible that the site and hence the databse is to large to > > handle for htdig > > and the hardware? > > I would say that you have ans

Re: [htdig] indexing

2000-01-07 Thread Doug Barton
Rickard Lundgren wrote: > > Hi > > I'm having a problem when I'm indexing my site, when i start htidg it goes > smooth at first but later it take up more and more memory space (after a few > hours) and finally the server hangs. Even though i set the priority to htdig > to 20 i still does the sam

Re: [htdig] Quick Question about using htdig

2000-01-07 Thread Glenn Nielsen
Are you doing anything special when indexing the majordomo list archives, such as using an external parser? We also have some majordomo lists we currently search using glimpse that we want to convert over to HtDig. Regards, Glenn Deepak Vaidya wrote: > > Hello, > > I just started playing wit

Re: [htdig] multiple SGML tags for noindex_start noindex_end

2000-01-07 Thread Glenn Nielsen
Thanks for the info. I know that this can be overcome by structuring the HTML correctly. But we are indexing 10 servers with close to 200 virtual hosts plus a dozen or so external servers we don't administer. We have no way to force the 100's of people/organizations that publish content to do t

Re: [htdig] multiple SGML tags for noindex_start noindex_end

2000-01-07 Thread Torsten Neuer
Glenn Nielsen wrote: > > Configuring a single SGML tag for noindex_start and noindex_end works. > But I have not been able to find a way to get multiple tags to work. > I would like to configure HtDig so that it ignores content inside > both and the default SGML tags. Curre

[htdig] Quick Question about using htdig

2000-01-07 Thread Deepak Vaidya
Hello, I just started playing with htdig to replace glimpse with. I will be using the software mostly to provide searching mailing list archive, which currently use mhonarc, glimpse and wilma. The mailing list are hosted using majordomo. The problem I am having is the list archives are passwo

[htdig] multiple SGML tags for noindex_start noindex_end

2000-01-07 Thread Glenn Nielsen
Configuring a single SGML tag for noindex_start and noindex_end works. But I have not been able to find a way to get multiple tags to work. I would like to configure HtDig so that it ignores content inside both

[htdig] Indexing a whole Intranet

2000-01-07 Thread Paul COURBIS
Hi ! Does everyone use htdig to index a whole Intranet (about 1000+ servers) ? Any advices on that subject ? How should I run htdig to be able to update database & to add new servers when requested in an easy way ? Paul To unsubscribe from the htdi

[htdig] indexing

2000-01-07 Thread Rickard Lundgren
Hi I'm having a problem when I'm indexing my site, when i start htidg it goes smooth at first but later it take up more and more memory space (after a few hours) and finally the server hangs. Even though i set the priority to htdig to 20 i still does the same thing. But when i run htmerge och the