I imagine you're aware that it sounds like while this is what Should happen, it isn't what Is happening.
Agreed, so that needs to be addressed.
Yes, yes, of course. The point appears to be that it Isn't happening.
So let's pinpoint it and fix it! ;)
Interesting. I don't suppose "pounding" means "approximately once a day"?
I count 513 hits so far.. that would be a bit more than one hit per-day. Its not a major issue, but spidering plkr.org with the Plucker spider once a day (every link), is probably not a great idea. You'll proably want to be checking whether or not the content has changed first, before re-spidering it.
Once the error in their script has been corrected, I will remove the block.
Please define "error". :)
Spidering the site repeatedly (every link, several hundred) without checking whether the content has changed or not. If every user of Plucker did that, the site would be so inundated with hits from spiders, that no actual valid (human) users would be able to visit the site to take advanrage of it.
I've removed the block for now, please adhere to robots.txt and increase the delay between your spidering hits, so you don't end up blocking users who want to use the site.
David A. Desrosiers [EMAIL PROTECTED] http://gnu-designs.com _______________________________________________ plucker-list mailing list [email protected] http://lists.rubberchicken.org/mailman/listinfo/plucker-list
