I noticed the same thing that the outlinks are fetched during subsequent runs even though you have URLfilters in place.
-byron --- carmmello <[EMAIL PROTECTED]> wrote: > When someone uses the crawl method with, lets say > 100 hundred sites, you > establish your url filters to allow only those > sites. In the first run, > just those 100 sites are indexed, but in subsequent > runs, the outlinks > are indexed too, together with other hops of the > seeds sites. This is > fine, as someone gets some really good related > sites, but if those > sites do not comply with the url filter, how come > are they indexed? > > >
