[aseek-users] allow/disallow rules

[Paul Stewart]

Title: Message

Hi there...

I'm trying to index everything I have a server statement for but *also* permit everything that Aspseek finds with .CA after it. The country I am trying to index has all kinds of tld's besides just .ca (canada) but I"d like aspseek to discover all .ca sites without having to put a server entry in the config for each one.

Previous posts were answered with "use allow/disallow"... well I've tried and can't get it figured out.

Here's a rundown of my config:

DeleteBad yes
DeleteNoServer no
FollowOutside yes
Server http://www.nexicom.net/
Server http://www.cbc.ca/
Server http://www.canoe.ca/
Server http://www.nexicom.com/
DisallowNoMatch \/$|\.htm$|\.html$|\.shtml$|\.phtml$|\.php$|\.txt$|\.asp$
Disallow /cgi-bin/ \.cgi /nph
Disallow \?
Disallow \&


I've tried putting DisallowNoMatch ://[^/]*\.ca/
into the config but then it won't index .com .net etc.

I'm sure this is just an order issue but I can't seem to figure it out.. am I trying something that can't be done?

Can someone take the above example and rework it for my purpose??

Thanks very very much!

Paul