Re: wget ignores --user and --password if you have a .netrc
On 9/3/07, Andreas Kohlbach <[EMAIL PROTECTED]> wrote:
> Hi,
>
> though the man page of wget mentions .netrc, I assume this is a bug.
>
> For my understanding if you provide a --user=user and --password=password
> at the command line this should overwrite any setting elsewhere, as in
> the .netrc. It doesn't. And it took me quite some time and bothering
> other guys to realise that it seems wget is ignoring --user and
> --password at the command line if a .netrc exists with the matching
> content.
Indeed. Whether this is a bug or not needs some discussion, I think,
but here's a patch to fix your problem.
Index: src/netrc.c
===
--- src/netrc.c (revision 2376)
+++ src/netrc.c (working copy)
@@ -59,6 +59,7 @@
search_netrc (const char *host, const char **acc, const char **passwd,
int slack_default)
{
+ if (strlen(opt.user) && strlen(opt.passwd)) return;
acc_t *l;
static int processed_netrc;
wget ignores --user and --password if you have a .netrc
Hi, though the man page of wget mentions .netrc, I assume this is a bug. For my understanding if you provide a --user=user and --password=password at the command line this should overwrite any setting elsewhere, as in the .netrc. It doesn't. And it took me quite some time and bothering other guys to realise that it seems wget is ignoring --user and --password at the command line if a .netrc exists with the matching content. -- Andreas (PGP Key available on public key servers) 23. What happens to a Hard Disk when you drop it? --Top 100 things you don't want the sysadmin to say
Re: Myriad merges
And now, finally, the ultimate real-life test with proxy-cache, timestamping and contentdisposition, where HEAD and GET have different timestamps. And this is perfectly correct now ! So it looks now to me, that the new error (local timestamp not set to remote) only occurs in the cases when no HEAD is used. Best regards, J.Roderburg HEAD -p http://wwwcache.uni-koeln.de:8080 http://download.lavasoft.com/public/core.zip 200 OK Date: Thu, 30 Aug 2007 09:31:35 GMT Accept-Ranges: bytes Age: 361684 ETag: "3014d-233e3c-cbcb000" Server: Apache/2.0.55 (Ubuntu) mod_ssl/2.0.55 OpenSSL/0.9.8a Content-Length: 2309692 Content-Type: application/zip Last-Modified: Mon, 27 Aug 2007 13:08:16 GMT Client-Date: Mon, 03 Sep 2007 13:59:39 GMT Client-Response-Num: 1 Proxy-Connection: close X-Cache: HIT from wwwcache.uni-koeln.de HEAD http://download.lavasoft.com/public/core.zip 200 OK Connection: close Date: Mon, 03 Sep 2007 14:00:48 GMT Accept-Ranges: bytes ETag: "3016f-275cfc-f35fc640" Server: Apache/2.0.55 (Ubuntu) mod_ssl/2.0.55 OpenSSL/0.9.8a Content-Length: 2579708 Content-Type: application/zip Last-Modified: Mon, 03 Sep 2007 08:28:01 GMT Client-Date: Mon, 03 Sep 2007 14:00:48 GMT Client-Response-Num: 1 wget.111-svn-0709 --debug http://download.lavasoft.com/public/core.zip DEBUG output created by Wget 1.10+devel on linux-gnu. --16:04:16-- http://download.lavasoft.com/public/core.zip Resolving wwwcache.uni-koeln.de... 134.95.19.61 Caching wwwcache.uni-koeln.de => 134.95.19.61 Connecting to wwwcache.uni-koeln.de|134.95.19.61|:8080... connected. Created socket 3. Releasing 0x080889b8 (new refcount 1). ---request begin--- HEAD http://download.lavasoft.com/public/core.zip HTTP/1.0 User-Agent: Wget/1.10+devel Accept: */* Host: download.lavasoft.com ---request end--- Proxy request sent, awaiting response... ---response begin--- HTTP/1.0 200 OK Date: Thu, 30 Aug 2007 09:31:35 GMT Server: Apache/2.0.55 (Ubuntu) mod_ssl/2.0.55 OpenSSL/0.9.8a Last-Modified: Mon, 27 Aug 2007 13:08:16 GMT ETag: "3014d-233e3c-cbcb000" Accept-Ranges: bytes Content-Length: 2309692 Content-Type: application/zip Age: 361961 X-Cache: HIT from wwwcache.uni-koeln.de Proxy-Connection: close ---response end--- 200 OK Length: 2309692 (2.2M) [application/zip] Closed fd 3 --16:04:16-- http://download.lavasoft.com/public/core.zip Found wwwcache.uni-koeln.de in host_name_addresses_map (0x80889b8) Connecting to wwwcache.uni-koeln.de|134.95.19.61|:8080... connected. Created socket 3. Releasing 0x080889b8 (new refcount 1). ---request begin--- GET http://download.lavasoft.com/public/core.zip HTTP/1.0 User-Agent: Wget/1.10+devel Accept: */* Host: download.lavasoft.com ---request end--- Proxy request sent, awaiting response... ---response begin--- HTTP/1.0 200 OK Date: Mon, 03 Sep 2007 14:04:16 GMT Server: Apache/2.0.55 (Ubuntu) mod_ssl/2.0.55 OpenSSL/0.9.8a Last-Modified: Mon, 03 Sep 2007 08:28:01 GMT ETag: "2370002-275cfc-f35fc640" Accept-Ranges: bytes Content-Length: 2579708 Content-Type: application/zip X-Cache: MISS from wwwcache.uni-koeln.de Proxy-Connection: close ---response end--- 200 OK Length: 2579708 (2.5M) [application/zip] Saving to: `core.zip' 100%[=>] 2,579,708155K/s in 15s Closed fd 3 16:04:31 (169 KB/s) - `core.zip' saved [2579708/2579708] ls -l core.zip -rw-r- 1 a0045 RRZK 2579708 03.09.2007 10:28 core.zip
Re: Myriad merges
And now, for a change, a case, that works now (better) ;-) This is an example where a HEAD request gets a 500 Error response. Wget default options again, but contentdisposition=yes to "force" a HEAD. wget.111-svn-0709 --debug -e "contentdisposition = yes" http://www.eudora.com/cgi-bin/export.cgi?productid=EUDORA_win_7109 Setting contentdisposition (contentdisposition) to yes DEBUG output created by Wget 1.10+devel on linux-gnu. --15:26:54-- http://www.eudora.com/cgi-bin/export.cgi?productid=EUDORA_win_7109 Resolving www.eudora.com... 199.106.114.30 Caching www.eudora.com => 199.106.114.30 Connecting to www.eudora.com|199.106.114.30|:80... connected. Created socket 3. Releasing 0x080888d8 (new refcount 1). ---request begin--- HEAD /cgi-bin/export.cgi?productid=EUDORA_win_7109 HTTP/1.0 User-Agent: Wget/1.10+devel Accept: */* Host: www.eudora.com Connection: Keep-Alive ---request end--- HTTP request sent, awaiting response... ---response begin--- HTTP/1.1 500 Server Error Server: Netscape-Enterprise/6.0 Date: Mon, 03 Sep 2007 13:26:54 GMT Content-length: 305 Content-type: text/html Connection: keep-alive ---response end--- 500 Server Error Registered socket 3 for persistent reuse. --15:26:56-- (try: 2) http://www.eudora.com/cgi-bin/export.cgi?productid=EUDORA_win_7109 Disabling further reuse of socket 3. Closed fd 3 Found www.eudora.com in host_name_addresses_map (0x80888d8) Connecting to www.eudora.com|199.106.114.30|:80... connected. Created socket 3. Releasing 0x080888d8 (new refcount 1). ---request begin--- GET /cgi-bin/export.cgi?productid=EUDORA_win_7109 HTTP/1.0 User-Agent: Wget/1.10+devel Accept: */* Host: www.eudora.com Connection: Keep-Alive ---request end--- HTTP request sent, awaiting response... ---response begin--- HTTP/1.1 302 Moved Temporarily Server: Netscape-Enterprise/6.0 Date: Mon, 03 Sep 2007 13:26:55 GMT Location: http://www.eudora.com/download/eudora/windows/7.1/Eudora_7.1.0.9.exe Content-length: 0 Connection: keep-alive ---response end--- 302 Moved Temporarily Registered socket 3 for persistent reuse. Location: http://www.eudora.com/download/eudora/windows/7.1/Eudora_7.1.0.9.exe [following] Skipping 0 bytes of body: [] done. --15:26:56-- http://www.eudora.com/download/eudora/windows/7.1/Eudora_7.1.0.9.exe Reusing existing connection to www.eudora.com:80. Reusing fd 3. ---request begin--- HEAD /download/eudora/windows/7.1/Eudora_7.1.0.9.exe HTTP/1.0 User-Agent: Wget/1.10+devel Accept: */* Host: www.eudora.com Connection: Keep-Alive ---request end--- HTTP request sent, awaiting response... ---response begin--- HTTP/1.1 200 OK Server: Netscape-Enterprise/6.0 Date: Mon, 03 Sep 2007 13:26:56 GMT Content-type: application/octet-stream Last-modified: Thu, 05 Oct 2006 18:45:18 GMT Content-length: 17416184 Accept-ranges: bytes Connection: keep-alive ---response end--- 200 OK Length: 17416184 (17M) [application/octet-stream] --15:26:56-- http://www.eudora.com/download/eudora/windows/7.1/Eudora_7.1.0.9.exe Reusing existing connection to www.eudora.com:80. Reusing fd 3. ---request begin--- GET /download/eudora/windows/7.1/Eudora_7.1.0.9.exe HTTP/1.0 User-Agent: Wget/1.10+devel Accept: */* Host: www.eudora.com Connection: Keep-Alive ---request end--- HTTP request sent, awaiting response... ---response begin--- HTTP/1.1 200 OK Server: Netscape-Enterprise/6.0 Date: Mon, 03 Sep 2007 13:26:56 GMT Content-type: application/octet-stream Last-modified: Thu, 05 Oct 2006 18:45:18 GMT Content-length: 17416184 Accept-ranges: bytes Connection: keep-alive ---response end--- 200 OK Length: 17416184 (17M) [application/octet-stream] Saving to: `Eudora_7.1.0.9.exe' 100%[=>] 17,416,184 397K/s in 44s 15:27:40 (386 KB/s) - `Eudora_7.1.0.9.exe' saved [17416184/17416184] ls -l Eudora_7.1.0.9.exe -rw-r- 1 a0045 RRZK 17416184 05.10.2006 20:45 Eudora_7.1.0.9.exe This seems also to use the only available source for the timestamp, the response to the GET request. Best regards, J.Roderburg
Re: --post-data encoding
"control H" <[EMAIL PROTECTED]> writes: > After a few hours of headache I found out my --post-data option > didn't work as I expected because the data I send has to be > URL-escaped. This is not mentioned both in the manpage and inline > help. A remark would be helpful. Note that, in general, it doesn't. POST requests are a generic mechanism for transferring data, and a valid POST request can contain an (unencoded) XML document or even binary data in its body. URL encoding is only necessary when transferring HTML form data from a client to the server. Wget doesn't assume that this is the case -- the POST options are designed as a low-level tool which the user is expected to understand how to use. I now see that this is not the most useful design for most people; for one, the manual could at least document the typical usage. It is also inconsistent because Wget automatically sends "Content-Type: application/x-www-form-urlencoded" when one of the POST options is in use, which indicates that the primary usage for POST was uploading form data. One way to solve this is to introduce higher-level functionality, such as --form-data and --form-attach (for uploading files) which construct a POST request suitable for sending form data, so the user doesn't have to. In that case --post-data and --post-file would no longer need to set content-type to application/x-www-form-urlencoded.
Re: Myriad merges
Zitat von Micah Cowan <[EMAIL PROTECTED]>: > Hm, that should not be. It should definitely set the timestamp if it > gets downloaded... I'll investigate. > > OOC, was there a specific resource you tested against (just in case I > have difficulty reproducing)? > Not a very specific one, just used our university homepage for this test ;-) Here a full protocol: ls -l index.html ls: cannot access index.html: No such file or directory HEAD http://www.uni-koeln.de/index.html 200 OK Connection: close Date: Mon, 03 Sep 2007 11:44:59 GMT Accept-Ranges: bytes Server: Apache/2.0.59 Content-Language: de Content-Type: text/html Last-Modified: Mon, 03 Sep 2007 11:04:09 GMT Client-Date: Mon, 03 Sep 2007 11:44:59 GMT Client-Response-Num: 1 wget.111-svn-0709 --debug http://www.uni-koeln.de/index.html DEBUG output created by Wget 1.10+devel on linux-gnu. --13:45:12-- http://www.uni-koeln.de/index.html Resolving www.uni-koeln.de... 134.95.19.39 Caching www.uni-koeln.de => 134.95.19.39 Connecting to www.uni-koeln.de|134.95.19.39|:80... connected. Created socket 3. Releasing 0x08088820 (new refcount 1). ---request begin--- GET /index.html HTTP/1.0 User-Agent: Wget/1.10+devel Accept: */* Host: www.uni-koeln.de Connection: Keep-Alive ---request end--- HTTP request sent, awaiting response... ---response begin--- HTTP/1.1 200 OK Date: Mon, 03 Sep 2007 11:45:12 GMT Server: Apache/2.0.59 Last-Modified: Mon, 03 Sep 2007 11:04:09 GMT Accept-Ranges: bytes Content-Type: text/html Content-Language: de Connection: close ---response end--- 200 OK Length: unspecified [text/html] Saving to: `index.html' [ <=> ] 9,131 --.-K/s in 0s Closed fd 3 13:45:12 (207 MB/s) - `index.html' saved [9131] ls -l index.html -rw-r- 1 a0045 RRZK 9131 03.09.2007 13:45 index.html date Mon Sep 3 13:45:24 CEST 2007 Jochen Roderburg ZAIK/RRZK University of Cologne Robert-Koch-Str. 10 Tel.: +49-221/478-7024 D-50931 Koeln E-Mail: [EMAIL PROTECTED] Germany
--post-data encoding
Hi, I'm not sure wether I'm experiencing a bug or not, so just to let you know. After a few hours of headache I found out my --post-data option didn't work as I expected because the data I send has to be URL-escaped. This is not mentioned both in the manpage and inline help. A remark would be helpful. Since I don't know what should be expected behaviour I can't decide wether this satisfies the bug criteria. As you have guessed by now, I would mark this as a bug. There's not much use in URL-escaping data yourself. Or maybe it could be an extra option to specify if your suplied data is to be URL encoded or not. A quick search in the bugarchive about this subject didn't yield any results. Tested versions: 1.10.2 (Red Hat modified) But I know for sure older version act the same. In fact, I guess all versions act the same. Regards, CH
