If your config file has not changed, then you probably need to look for changes involving the site listed in your start_url. According to the output you sent, the first URL checked (after the normal robots.txt check) resulted in a redirect (302) to http://www.cyberpump.com/cgi-bin/sentry/sentry.cgi. With your current exclude_url settings, this URL would be excluded for both /cgi-bin/ and .cgi. Since the first page is excluded out of hand, there are no URLs parsed from the page and therefore no further URLs to try to index. At least that is the way I read it given the available information. If this were an authorization problem, I would have expected indexing to stop with a 401 code. But perhaps I am missing something. If you provide start_url with an absolute URL that does not include /cgi-bin/ or .cgi, is htdig able to at least index that page? That might be a quick way to rule out issues involving basic authorization. You might also try temporarily removing /cgi-bin/ and .cgi from exclude_urls to see how that affects things.
Jim
On Feb 28, 2004, at 9:00 PM, Cyberpump! wrote:
Yes, it does. It always has. My files and conf have not changed for a long
time. The only thing that's change is the server move. I've done server
moves before with no issues using the same conf file.
exclude_urls: /cgi-bin/ .cgi .pdf .php /data4996/ /db/ /entrophybanner/ /bin/ /backups/ /qna/forums/ /grippage/ /totalcoaching/ /garagegym/
This has not changed for a few years. :)
----- Original Message ----- From: "Jim Cole" <[EMAIL PROTECTED]> To: "Cyberpump!" <[EMAIL PROTECTED]> Sent: Saturday, February 28, 2004 9:56 PM Subject: Re: [htdig] Problem Digging - Not Had Before
On Feb 28, 2004, at 7:30 PM, Cyberpump! wrote:
I recently moved to a server with Red Hat Enterprise 3.0...
It's not digging my password protected members area anymore. The verbose output is below.
Even though in the config file I used the same "authorization:" in the
config file.http://www.cyberpump.com/cgi-bin/sentry/sentry.cgi? setup+searchengine+69.93.62.98+/members/%3f
Rejected: Item in the exclude list: item # 1 length: 9 pick: www.cyberpump.com, # servers = 1
My guess is that this doesn't have anything to do with authorization. It looks like you are simply hitting a URL that htdig is configured to exclude. What does your 'exclude_urls' attribute look like? Does it currently include /cgi-bin/ and/or .cgi ?
Jim
------------------------------------------------------- SF.Net is sponsored by: Speed Start Your Linux Apps Now. Build and deploy apps & Web services for Linux with a free DVD software kit from IBM. Click Now! http://ads.osdn.com/?ad_id=1356&alloc_id=3438&op=click _______________________________________________ ht://Dig general mailing list: <[EMAIL PROTECTED]> ht://Dig FAQ: http://htdig.sourceforge.net/FAQ.html List information (subscribe/unsubscribe, etc.) https://lists.sourceforge.net/lists/listinfo/htdig-general
