We've got a site that is causing our crawl to slow dramatically, from
20mbits down to about 3 or 4. The basic problem is that the site seems
to consist of huge numbers of pages that aren't responding. We can
remove the site from the index, but it seems like a problem to remove
this site permanently from the webdb so that we never fetch it again.
Is there an easy way in 0.71 to remove a site from the webdb, and then
keep it permanently removed?
- removing site from webdb Insurance Squared Inc.
- Re: removing site from webdb Rod Taylor
- Re: removing site from webdb Matt Kangas
