Paul Stewart wrote:
That is the right behaviour, ASPseek never trust "old" robots.txt and always reloads it when starting a new indexing session. In other words,When we restart indexer (runs 8 hours daily) it always reloads the robots.txt files from what appears to be every site when it first starts up...
"Period" directive doesn't apply to robots.txt.
--
== kir_at_asplinux.ru == 7551596_at_ICQ == 6722750_at_sms.beemail.ru ==
Dream like you'll live forever...Love like you've never been hurt...
Work like you don't need the money...and Dance like nobody is watching!
-- Satchel Paige
