<Jim Cole wrote: snipped>...
I took out .cgi and cgi-bin out of the excludes as suggested and this is
what I got. Any ideas? Thanks!
-bash-2.05b$ ./htdig -vvvvv 0:1:http://www.cyberpump.com/members/
redirect:
http://www.cyberpump.com/cgi-bin/sentry/sentry.cgi? setup+searchengine+69.93.62.98+/members/%3f
Rejected: URL not in the limits!pick: www.cyberpump.com, # servers = 1
It sounds like you are using the default limit_urls_to attribute, which simply takes on the value of whatever you provide for start_url. If this is the case, the pattern "http://www.cyberpump.com/members/"; is used to determine whether a candidate URL is within the specified limits. The URL you are being redirected to would not be within these limits since it does not include the "/members/" piece. The solution would be to customize your limit_urls_to attributes to match your specific needs. If you are trying to limit indexing to the /members/ section, you might try something like the following.
limit_urls_to: http://www.cyberpump.com/members/ http://www.cyberpump.com/cgi-bin/
If that seems to work, you might want to make the second pattern a bit more specific to avoid anything else in /cgi-bin/ being inadvertently picked up.
Jim
------------------------------------------------------- This SF.Net email is sponsored by: IBM Linux Tutorials Free Linux tutorial presented by Daniel Robbins, President and CEO of GenToo technologies. Learn everything from fundamentals to system administration.http://ads.osdn.com/?ad_id=1470&alloc_id=3638&op=click _______________________________________________ ht://Dig general mailing list: <[EMAIL PROTECTED]> ht://Dig FAQ: http://htdig.sourceforge.net/FAQ.html List information (subscribe/unsubscribe, etc.) https://lists.sourceforge.net/lists/listinfo/htdig-general
