In regard to my difficulties with recursively retrieving
http://www.iana.org/assignments/index.html: I discovered that one URL
(http://www.iana.org/assignments/forces/forces.xhtml) is pointed to by
no less than three different URLs:
http://www.iana.org/assignments/forces/forces.xhtml
http://www.iana.org/assignments/forces-parameters/forces-parameters.xhtml
http://www.iana.org/assignments/forces
The first is the proper URL for it, and the second two are redirected to
the first URL.
There are several other occurrences of this situation.
And I discovered that if I specify --trust-server-names, then wget will
realize that the redirection URL can be retrieved once, and links to the
other two URLs can be directed to that one file. Without
--trust-server-names, wget considers all three URLs to be different,
despite that they are redirected to the same URL, and dutifully stores
essentially the same content three times. With --trust-server-names,
wget understands that all three URLs are the same.
It turns out that this provides me with a much better mirror of the web
site.
I've attached a patch that improves the documentation of
--trust-server-names, to clarify that if -nd is not in effect, then the
file name is constructed from the entire redirection URL, not just the
last component.
(--trust-server-names is also mentioned in doc/metalink-standard.txt,
but that text does not seem to me to have the problem the patch
corrects.)
Dale
>From 740c68d4d820334362dc93ce5c31b9d742f12558 Mon Sep 17 00:00:00 2001
From: "Dale R. Worley" <wor...@ariadne.com>
Date: Wed, 2 Nov 2016 12:14:46 -0400
Subject: [PATCH] Improve documentation of --trust-server-names.
---
doc/wget.texi | 12 +++++++-----
1 file changed, 7 insertions(+), 5 deletions(-)
diff --git a/doc/wget.texi b/doc/wget.texi
index 91219e5..3632fd1 100644
--- a/doc/wget.texi
+++ b/doc/wget.texi
@@ -1700,9 +1700,11 @@ with a http status code that indicates error.
@cindex Trust server names
@item --trust-server-names
-If this is set to on, on a redirect the last component of the
-redirection URL will be used as the local file name. By default it is
-used the last component in the original URL.
+If this is set, on a redirect, the local file name will be based
+on the redirection URL. By default the local file name is is based on
+the original URL. When doing recursive retrieving this can be helpful
+because in many web sites redirected URLs correspond to an underlying
+file structure, while link URLs do not.
@cindex authentication
@item --auth-no-challenge
@@ -3261,8 +3263,8 @@ Turn on recognition of the (non-standard) @samp{Content-Disposition}
HTTP header---if set to @samp{on}, the same as @samp{--content-disposition}.
@item trust_server_names = on/off
-If set to on, use the last component of a redirection URL for the local
-file name.
+If set to on, construct the local file name from redirection URLs
+rather than original URLs.
@item continue = on/off
If set to on, force continuation of preexistent partially retrieved
--
1.8.3.1
