What I mean is to discard the URL with the end slash *after* the downloading
and finding out that this is, in fact, an incorrectly specified directory.
Is there
anything wrong with it?
Gregory
-----Original Message-----
From: Alexander F Avdonkin [mailto:[EMAIL PROTECTED]]
Sent: Freitag, 14. Juni 2002 09:26
To: [EMAIL PROTECTED]
Subject: Re: [aseek-users] Indexing with MaxHops
This is not good idea. How are you going to find out if this URL is
directory
or just a file without extension before downloading it ?
Alexander.
Gregory Kozlovsky wrote:
> The solution of Matt is a very good one. However, I wonder
> whether the automatic redirection when a folder name is
> specified without the slash at the end should be put into
> a category of its own. This case is far too common to treat
> it as normal redirection. May be in this case, the URL without
> the end slash should be just discarded and replaced with one
> with the end slash? The fields "referrer" and "redir" in the
> urlword set as if the URL with the end slash was used from the
> start? Are there any pitfalls in this approach? What do the
> authors of ASPSeek think?
>
> Gregory
>
> -----Original Message-----
> From: Matt Sullivan [mailto:[EMAIL PROTECTED]]
> Sent: Donnerstag, 13. Juni 2002 23:05
> To: [EMAIL PROTECTED]
> Subject: Re: [aseek-users] Indexing with MaxHops
>
> Actually I have a patch for this problem which adds two new config
> parameters
> (config excerpt below):
>
> #######################################################################
> #IncrementHopsOnRedirect yes/no
> # Allow/disallow index to increment hops value when redirects are
> # encountered. Applies only to redirects generated by Location headers.
> # ***** SURGEON GENERALS WARNING *****
> # This option can be harmful as it negates the indexers built in ability
> # to be self limiting in the case where a redirect loop is encountered.
> # Please ensure that RedirectLoopLimit is set to a resonable value to
> # enable recovery from entry into a redirect loop.
> # ***** SURGEON GENERALS WARNING *****
> # This option does however allow a greater number of documents to be
> # indexed for sites that redirect frequently (e.g. for cookie testing,
> # typically on each page). Test results (with MaxHops 4) on such a site
> # increased actual documents indexed from 34 to 756.
> # Can be set multiple times before "Server" command and takes effect till
> # the end of config file or till next IncrementHopsOnRedirect command.
> # Default value is "yes".
> IncrementHopsOnRedirect no
>
> #######################################################################
> #RedirectLoopLimit <number>
> # Maximum allowable contiguous redirects.
> # Default value is 8.
> # Can be set multiple times before "Server" command and takes effect till
> # the end of config file or till next RedirectLoopLimit command.
> RedirectLoopLimit 16
>
> If there is interest I'll send it through.
>
> Matt.
>
> > >
> > > Gregory
> > >
> > > -----Original Message-----
> > > From: Alexander F Avdonkin [mailto:[EMAIL PROTECTED]]
> > > Sent: Donnerstag, 13. Juni 2002 16:34
> > > To: [EMAIL PROTECTED]
> > > Subject: Re: [aseek-users] Indexing with MaxHops
> > >
> > > You told that page is reacheable by 2 clicks.
> > > Check if all intermediate pages are indexed and which hop values are
> > > assigned
> > > to them.
> > >
> > > Gregory Kozlovsky wrote:
> > >
> > > > No, it was not indexed. I checked logging into mysql and using a
> SELECT
> > > > statement.
> > > >
> > > > -----Original Message-----
> > > > From: Alexander F Avdonkin [mailto:[EMAIL PROTECTED]]
> > > > Sent: Donnerstag, 13. Juni 2002 16:17
> > > > To: [EMAIL PROTECTED]
> > > > Subject: Re: [aseek-users] Indexing with MaxHops
> > > >
> > > > Hello Gregory,
> > > >
> > > > As for first problem, check if page referring to absent URL is
indexed
> and
> > > > what
> > > > hop value is assigned to it.
> > > >
> > > > Alexander.
> >
