Webboard: robots.txt disallowed

Author: costas Email: [EMAIL PROTECTED] Message: Something is definetly not functioning properly in indexer. If I try indexing the below url, evrtything works fine and the .asp files in the directory are indexed perfect Follow path Server http://http://www.deliaonline.com/recipesandfood/ When

Webboard: robots.txt disallowed

Author: costas Email: [EMAIL PROTECTED] Message: robots table is completely empty when i try to index "http://www.wwf.org.uk/news/"; but it still refuses to pick up the asp pages inside this folder. Reply:

Webboard: robots.txt disallowed

Author: Alexander Email: Barkov Message: 3.1.x caches robots.txt information in SQL table robots. Remove www.wwf.org.uk entries from this table. > I am trying to index this address but it contains a robot.txt which prevents me from >indexing this directory > > Server http://www.wwf.org.uk/news/

Webboard: robots.txt disallowed

Author: costas Email: [EMAIL PROTECTED] Message: I am trying to index this address but it contains a robot.txt which prevents me from indexing this directory Server http://www.wwf.org.uk/news/ when i add "Robots no" to indexer i get the message indexer[1744]: [1] robots.txt support is disallow