[analog-help] Identifying Known Spiders?

I'd like to know the success of my efforts to submit a new site to all the search engines; some spiders won't visit a site until it's been online for a while, and some will only visit the home page. I can see some of the spiders in the BROWSERREP and BROWSERSUM, but it's missing some because it's

Re: [analog-help] Identifying Known Spiders?

On 7/3/2008 3:48 AM, Michael Crawford wrote: I'd like to know the success of my efforts to submit a new site to all the search engines; some spiders won't visit a site until it's been online for a while, and some will only visit the home page. I can see some of the spiders in the BROWSERREP and

RE: [analog-help] Identifying Known Spiders?

The robots list from which that page was built no longer exists. The group that was maintaining it decided that it didn't make sense to maintain a database of "known robots" any more as anyone can make a robot. However, some quick review looks like there's a new user-submitted list at http://ww

Re: [analog-help] Identifying Known Spiders?

On Thu, Jul 3, 2008 at 8:37 AM, Jeremy Wadsack <[EMAIL PROTECTED]> wrote: > The robots list from which that page was built no longer exists. The group > that was maintaining it decided that it didn't make sense to maintain a > database of "known robots" any more as anyone can make a robot. In my