What do you mean under "discard" ? Not index it again ?
Alexander. Gregory Kozlovsky wrote: > What I mean is to discard the URL with the end slash *after* the downloading > and finding out that this is, in fact, an incorrectly specified directory. > Is there > anything wrong with it? > > Gregory > > -----Original Message----- > From: Alexander F Avdonkin [mailto:[EMAIL PROTECTED]] > Sent: Freitag, 14. Juni 2002 09:26 > To: [EMAIL PROTECTED] > Subject: Re: [aseek-users] Indexing with MaxHops > > This is not good idea. How are you going to find out if this URL is > directory > or just a file without extension before downloading it ? > > Alexander. > > Gregory Kozlovsky wrote: > > > The solution of Matt is a very good one. However, I wonder > > whether the automatic redirection when a folder name is > > specified without the slash at the end should be put into > > a category of its own. This case is far too common to treat > > it as normal redirection. May be in this case, the URL without > > the end slash should be just discarded and replaced with one > > with the end slash? The fields "referrer" and "redir" in the > > urlword set as if the URL with the end slash was used from the > > start? Are there any pitfalls in this approach? What do the > > authors of ASPSeek think? > > > > Gregory > > > > -----Original Message----- > > From: Matt Sullivan [mailto:[EMAIL PROTECTED]] > > Sent: Donnerstag, 13. Juni 2002 23:05 > > To: [EMAIL PROTECTED] > > Subject: Re: [aseek-users] Indexing with MaxHops > > > > Actually I have a patch for this problem which adds two new config > > parameters > > (config excerpt below): > > > > ####################################################################### > > #IncrementHopsOnRedirect yes/no > > # Allow/disallow index to increment hops value when redirects are > > # encountered. Applies only to redirects generated by Location headers. > > # ***** SURGEON GENERALS WARNING ***** > > # This option can be harmful as it negates the indexers built in ability > > # to be self limiting in the case where a redirect loop is encountered. > > # Please ensure that RedirectLoopLimit is set to a resonable value to > > # enable recovery from entry into a redirect loop. > > # ***** SURGEON GENERALS WARNING ***** > > # This option does however allow a greater number of documents to be > > # indexed for sites that redirect frequently (e.g. for cookie testing, > > # typically on each page). Test results (with MaxHops 4) on such a site > > # increased actual documents indexed from 34 to 756. > > # Can be set multiple times before "Server" command and takes effect till > > # the end of config file or till next IncrementHopsOnRedirect command. > > # Default value is "yes". > > IncrementHopsOnRedirect no > > > > ####################################################################### > > #RedirectLoopLimit <number> > > # Maximum allowable contiguous redirects. > > # Default value is 8. > > # Can be set multiple times before "Server" command and takes effect till > > # the end of config file or till next RedirectLoopLimit command. > > RedirectLoopLimit 16 > > > > If there is interest I'll send it through. > > > > Matt. > > > > > > > > > > Gregory > > > > > > > > -----Original Message----- > > > > From: Alexander F Avdonkin [mailto:[EMAIL PROTECTED]] > > > > Sent: Donnerstag, 13. Juni 2002 16:34 > > > > To: [EMAIL PROTECTED] > > > > Subject: Re: [aseek-users] Indexing with MaxHops > > > > > > > > You told that page is reacheable by 2 clicks. > > > > Check if all intermediate pages are indexed and which hop values are > > > > assigned > > > > to them. > > > > > > > > Gregory Kozlovsky wrote: > > > > > > > > > No, it was not indexed. I checked logging into mysql and using a > > SELECT > > > > > statement. > > > > > > > > > > -----Original Message----- > > > > > From: Alexander F Avdonkin [mailto:[EMAIL PROTECTED]] > > > > > Sent: Donnerstag, 13. Juni 2002 16:17 > > > > > To: [EMAIL PROTECTED] > > > > > Subject: Re: [aseek-users] Indexing with MaxHops > > > > > > > > > > Hello Gregory, > > > > > > > > > > As for first problem, check if page referring to absent URL is > indexed > > and > > > > > what > > > > > hop value is assigned to it. > > > > > > > > > > Alexander. > > >
