When I started indexer with '-v 5' option and in just one process, the
last messages has been these:
Indexer[5553]: [1] http://www.banky.sk/robots.txt
I have the same problem. Everytime my indexer finds a robots.txt, it
hangs. Although, I have robots set to no in my indexer.conf.
also sprach lang:
When I started indexer with '-v 5' option and in just one process, the
last messages has been these:
Indexer[5553]: [1] http://www.banky.sk/robots.txt
I have the same problem. Everytime my indexer finds a robots.txt, it
hangs. Although, I have robots set to no in
