- - - - - - - - - - - - - - - - - - - - - - - - - - - - Name: alex Subject: bugs in indexer (importing)
hi (maxime) I'm trying to feed some urls to indexer via (-i -f) and if I have the Robots On it will take some time to load them, but if I put Robots Off it will do it quite quickly. with Robots on it's fetching the robots.txt for every new url; why is it doing so when -n 0 (to stop at 0 indexed pages) 10x - - - - - - - - - - - - - - - - - - - - - - - - - - - - Read the full topic here: http://www.dataparksearch.org./cgi-bin/simpleforum.cgi?fid=03;post=
