problem is that server command only sets the starting page to begin with index but is will index the
whole server (if there are links from the subpage to the root), so
will start in /bar to index. but if there is a page let's say http://www.foo.com/bar/links.htm with links to http://www.foo.com/fred/ then also the subdir fred will be indexed (and so on). with sites like xoom.com or geocities you can run in the problem to index all the pages those services host... and that can be a lot... ;-)
i do something like
Server http://www.foo.com/bar
Server (...)
Allow http://www.foo.com/bar
Allow (...)
Disallow .*
so i set Server and a corresponding Allow Command, with this i can limit the index to the real subdirs of servers...
Markus Rietzler
* kommunikation & online service
* RZF NRW
* Tel: 0211.4572-130
-----Urspr�ngliche Nachricht-----
Von: fabrice valere [mailto:[EMAIL PROTECTED]]
Gesendet am: Dienstag, 24. Juli 2001 09:45
An: [EMAIL PROTECTED]
Betreff: [aseek-users] follow
hi all
I have a little question for you
I want index sites hosted freely, generaly the url is something like
this
www.xoom.com/yourname/
Why in the database the site is www.xoom.com and not
www.xoom.com/yourname/.
Does it can be a problem for the option follow and followoutside.
In my case sometimes i find urls from site I don't index; the first
part of the
url is allways a part of url I ask to index before.
example
I want to index www.xoom.com/yourname/
I can find sometimes an url like www.xoom.com/anothername/
I have a config file with option
follow yes
followoutside no
when I index a site I do
server www.foo.com/
Disallow .*
Allow www.foo.com/
