ManifoldCF reads all the URLs on its queue. If it's a 301, it detects this and pushes the new URL onto the document queue. When it gets to the new URL, it processes it like any other.
Karl On Wed, May 19, 2021 at 8:32 AM ritika jain <ritikajain5...@gmail.com> wrote: > Hi > > I want to understand the process of "How does manifold.cf handles > redirection of URL." in case of web crawler connector > > If there is a page redirect (through a 301) to another URL, then the next > crawl will detect the redirect and index the new (final) URL and display it > in the search results. (instead of the old URL that redirects). Just as is > also done by search engines like Google / Bing. > > Is that true, , what manifold is capable of avoiding the URL that is 301 > and pick the URL to which it is redirected? and ingest that URL . > > If not , what process Manifoldcf follows to inges redirection of URL's > > Thanks > Ritika > > >
