Hi, Great thanks
On Mon, Aug 24, 2026, at 11:49 AM, Tim Rühsen wrote: > Hi, > > > > curl.htm is the desired content, whereas wget.htm contains a > generic 404 > > error page "Page cannot be found", suggesting wget is not redirecting to > > the correct URL. > > Wget2 received couple of fixes in the past weeks. With the current code > (not yet released), wget2 downloads the page correctly. > > The README.md contains recipies to build Wget2 with the help of docker. > No need to install dev tools. > > Regards, Tim > > On 8/23/26 13:00, [email protected] wrote: > > Hi, > > > > Thanks for looking into this, it seems though that wget fails to use the > > correct URL anyway, as when using "--content-on-error=on", a generic 404 > > error page is returned by the server instead of the blog post. > > > > Here is an example using a better URL without the encoding. This > > produces curl.htm and wget.htm: > > > >> curl -L 'https://www.bbc.co.uk/blogs/adamcurtis/entries/ > >> a2094c9d-9864-348e-a241-7aa93adf0c09 <https://www.bbc.co.uk/blogs/ > >> adamcurtis/entries/a2094c9d-9864-348e-a241-7aa93adf0c09>' > curl.htm > >> > >> wget --content-on-error=on 'https://www.bbc.co.uk/blogs/adamcurtis/ > >> entries/a2094c9d-9864-348e-a241-7aa93adf0c09 <https://www.bbc.co.uk/ > >> blogs/adamcurtis/entries/a2094c9d-9864-348e-a241-7aa93adf0c09>' -O > >> wget.htm > > > > curl.htm is the desired content, whereas wget.htm contains a generic 404 > > error page "Page cannot be found", suggesting wget is not redirecting to > > the correct URL. > > > > Thanks > > > > On Sun, Aug 23, 2026, at 11:45 AM, Tim Rühsen wrote: > >> Hi, > >> > >> > Wget fails to download pages from Adam Curtis' archived blog, such as > >> this: > >> > >> This is expected since the server answers with 404 instead of 200 and > >> wget dicards any content in this case (by default). > >> > >> To do store the content on error, you have to add `--content-on-error` > >> on the command line. > >> > >> Cheers, Tim > >> > >> On 8/22/26 13:24, [email protected] > >> <mailto:[email protected]> wrote: > >> > Wget fails to download pages from Adam Curtis' archived blog, such > >> as this: > >> > > >> > https://www.bbc.co.uk/webarchive/ > >> https%253A%252F%252Fwww.bbc.co.uk%252Fblogs%252Fadamcurtis%252Fentries%252Fa2094c9d-9864-348e-a241-7aa9 > >> > >> <https://www.bbc.co.uk/webarchive/https%253A%252F%252Fwww.bbc.co.uk%252Fblogs%252Fadamcurtis%252Fentries%252Fa2094c9d-9864-348e-a241-7aa9> > >> > > >> > But Curl succeeds. This may be because it is re-encoding the % symbols. > >> > > >> > output of wget -d: > >> > > >> >> 22.122345.635 Local URI encoding = 'UTF-8' > >> >> 22.122345.635 Input URI encoding = 'UTF-8' > >> >> 22.122345.635 Fetched HSTS data from '/home/miles/.local/share/ > >> wget/.wget-hsts' > >> >> 22.122345.635 Fetched HPKP data from '/home/miles/.local/share/ > >> wget/.wget-hpkp' > >> >> 22.122345.635 set_exit_status(0) > >> >> 22.122345.635 path webarchive/ > >> https%3A%2F%2Fwww.bbc.co.uk%2Fblogs%2Fadamcurtis%2Fentries%2Fa2094c9d-9864-348e-a241-7aa9 > >> -> > >> >> 22.122345.635 webarchive/ > >> https%3A%2F%2Fwww.bbc.co.uk%2Fblogs%2Fadamcurtis%2Fentries%2Fa2094c9d-9864-348e-a241-7aa9 > >> >> 22.122345.635 local filename = > >> 'https%3A%2F%2Fwww.bbc.co.uk%2Fblogs%2Fadamcurtis%2Fentries%2Fa2094c9d-9864-348e-a241-7aa9' > >> >> 22.122345.635 host_add_job: job fname > >> https%3A%2F%2Fwww.bbc.co.uk%2Fblogs%2Fadamcurtis%2Fentries%2Fa2094c9d-9864-348e-a241-7aa9 > >> >> 22.122345.635 host_add_job: 0x5622fb64ec70 https://www.bbc.co.uk/ > >> webarchive/ > >> https%253A%252F%252Fwww.bbc.co.uk%252Fblogs%252Fadamcurtis%252Fentries%252Fa2094c9d-9864-348e-a241-7aa9 > >> > >> <https://www.bbc.co.uk/webarchive/https%253A%252F%252Fwww.bbc.co.uk%252Fblogs%252Fadamcurtis%252Fentries%252Fa2094c9d-9864-348e-a241-7aa9> > >> >> 22.122345.635 host_add_job: qsize 1 host-qsize=1 > >> >> 22.122345.635 queue_size: qsize=1 > >> >> 22.122345.635 queue_size: qsize=1 > >> >> 22.122345.635 queue_size: qsize=1 > >> >> 22.122345.635 [0] action=1 pending=0 host=0x0 goaway=n/a > >> >> 22.122345.635 dequeue job https://www.bbc.co.uk/webarchive/ > >> https%253A%252F%252Fwww.bbc.co.uk%252Fblogs%252Fadamcurtis%252Fentries%252Fa2094c9d-9864-348e-a241-7aa9 > >> > >> <https://www.bbc.co.uk/webarchive/https%253A%252F%252Fwww.bbc.co.uk%252Fblogs%252Fadamcurtis%252Fentries%252Fa2094c9d-9864-348e-a241-7aa9> > >> >> 22.122345.635 resolving www.bbc.co.uk:443 <http://www.bbc.co.uk:443>... > >> >> 22.122345.643 has 212.58.237.1:443 > >> >> 22.122345.643 has 212.58.236.129:443 > >> >> 22.122345.643 has 212.58.237.129:443 > >> >> 22.122345.643 has 212.58.235.1:443 > >> >> 22.122345.643 has 212.58.235.129:443 > >> >> 22.122345.643 has 212.58.236.1:443 > >> >> 22.122345.643 has 212.58.237.129:443 > >> >> 22.122345.643 has 212.58.236.1:443 > >> >> 22.122345.643 has 212.58.235.1:443 > >> >> 22.122345.643 has 212.58.237.1:443 > >> >> 22.122345.643 has 212.58.235.129:443 > >> >> 22.122345.643 has 212.58.236.129:443 > >> >> 22.122345.643 trying 212.58.237.1:443... > >> >> 22.122345.659 GnuTLS init > >> >> 22.122345.835 GnuTLS system certificate store is empty > >> >> 22.122345.835 Certificates loaded: 388 > >> >> 22.122345.835 GnuTLS init done > >> >> 22.122345.836 TLS False Start requested > >> >> 22.122345.836 SNI www.bbc.co.uk <http://www.bbc.co.uk> > >> >> 22.122345.836 OCSP stapling requested for www.bbc.co.uk <http:// > >> www.bbc.co.uk> > >> >> 22.122345.836 ALPN offering h2 > >> >> 22.122345.836 ALPN offering http/1.1 > >> >> 22.122345.894 OCSP stapling is not supported by 'www.bbc.co.uk' > >> >> 22.122345.894 host has no pubkey pinnings stored in hpkp db > >> >> 22.122345.894 host has no pubkey pinnings stored in hpkp db > >> >> 22.122345.894 host has no pubkey pinnings stored in hpkp db > >> >> 22.122345.894 host has no pubkey pinnings stored in hpkp db > >> >> 22.122345.894 host has no pubkey pinnings stored in hpkp db > >> >> 22.122345.894 host has no pubkey pinnings stored in hpkp db > >> >> 22.122345.894 host has no pubkey pinnings stored in hpkp db > >> >> 22.122345.894 host has no pubkey pinnings stored in hpkp db > >> >> 22.122345.894 TLS False Start: off > >> >> 22.122345.894 ALPN: Server accepted protocol 'h2' > >> >> 22.122345.895 Handshake completed > >> >> 22.122345.895 established connection www.bbc.co.uk <http:// > >> www.bbc.co.uk> > >> >> 22.122345.895 cookie_create_request_header for host=www.bbc.co.uk > >> <http://www.bbc.co.uk> path=webarchive/ > >> https%3A%2F%2Fwww.bbc.co.uk%2Fblogs%2Fadamcurtis%2Fentries%2Fa2094c9d-9864-348e-a241-7aa9 > >> >> 22.122345.895 HTTP2 stream id 1 > >> >> 22.122345.895 [0] action=1 pending=1 host=0x5622fb64e6b0 goaway=false > >> >> 22.122345.895 [0] action=2 pending=1 host=0x5622fb64e6b0 goaway=false > >> >> 22.122345.895 ## pending_requests = 1 > >> >> 22.122345.895 [FRAME 0] > SETTINGS > >> >> 22.122345.895 [FRAME 0] > WINDOW_UPDATE > >> >> 22.122345.895 [FRAME 1] > HEADERS > >> >> 22.122345.895 [FRAME 1] > :method: GET > >> >> 22.122345.895 [FRAME 1] > :path: /webarchive/ > >> https%253A%252F%252Fwww.bbc.co.uk%252Fblogs%252Fadamcurtis%252Fentries%252Fa2094c9d-9864-348e-a241-7aa9 > >> >> 22.122345.895 [FRAME 1] > :scheme: https > >> >> 22.122345.895 [FRAME 1] > :authority: www.bbc.co.uk <http:// > >> www.bbc.co.uk> > >> >> 22.122345.895 [FRAME 1] > accept-encoding: gzip, deflate, br, zstd > >> >> 22.122345.895 [FRAME 1] > accept: */* > >> >> 22.122345.895 [FRAME 1] > user-agent: Wget/2.2.1 > >> >> 22.122345.909 [FRAME 0] < SETTINGS > >> >> 22.122345.909 [FRAME 0] < WINDOW_UPDATE > >> >> 22.122345.909 [FRAME 0] > SETTINGS > >> >> 22.122345.910 [FRAME 0] < SETTINGS > >> >> 22.122346.113 :status: 404 > >> >> 22.122346.113 date: Sat, 22 Aug 2026 11:23:46 GMT > >> >> 22.122346.113 content-type: text/html;charset=utf-8 > >> >> 22.122346.113 content-length: 9687 > >> >> 22.122346.113 belfrage-cache-status: MISS > >> >> 22.122346.113 bid: bruce > >> >> 22.122346.113 brequestid: 268b641cf47a4cd78df3312be105da1c > >> >> 22.122346.113 bsig: b24bbece8fc068ce8b9dfd6803e8e57e > >> >> 22.122346.113 cache-control: public, stale-if-error=90, stale- > >> while-revalidate=30, max-age=5 > >> >> 22.122346.113 content-encoding: gzip > >> >> 22.122346.113 req-svc-chain: GTM,BELFRAGE,MOZART > >> >> 22.122346.113 vary: accept-encoding > >> >> 22.122346.114 via: 2.0 BBC-GTM, 1.1 Belfrage > >> >> 22.122346.114 x-content-type-options: nosniff > >> >> 22.122346.114 x-mrid: n1 > >> >> 22.122346.114 x-xss-protection: 1; mode=block > >> >> 22.122346.114 origin-agent-cluster: ?0 > >> >> 22.122346.114 x-lb-nocache: true > >> >> 22.122346.114 nel: > >> {"report_to":"default","max_age":2592000,"include_subdomains":true,"failure_fraction":0.25} > >> >> 22.122346.114 x-bbc-edge-cache-status: MISS > >> >> 22.122346.114 x-bbc-origin-response-status: 404 > >> >> 22.122346.114 report-to: > >> {"group":"default","max_age":2592000,"endpoints":[{"url":"https:// > >> default.bbc-reporting-api.app/report-endpoint <https://default.bbc- > >> reporting-api.app/report- > >> endpoint>","priority":1}],"include_subdomains":true <https:// > >> default.bbc-reporting-api.app/report-endpoint%22,%22priority%22:1%7D > >> <https://default.bbc-reporting-api.app/report-endpoint%22, > >> %22priority%22:1%7D>],%22include_subdomains%22:true>} > >> >> 22.122346.114 server: BBC-GTM > >> >> 22.122346.114 strict-transport-security: max-age=31536000; preload > >> >> 22.122346.114 permissions-policy: browsing-topics=(), join-ad- > >> interest-group=(), run-ad-auction=() > >> >> 22.122346.114 x-robots-tag: bingbot: noarchive > >> >> 22.122346.114 [FRAME 1] < HEADERS > >> >> 22.122346.114 closing stream 1 > >> >> 22.122346.114 ## response status 404 > >> >> 22.122346.114 set_exit_status(8) > >> >> 22.122346.114 keep_alive=1 > >> >> 22.122346.114 update HSTS www.bbc.co.uk:443 <http:// > >> www.bbc.co.uk:443> (maxage=31536000, includeSubDomains=0) > >> >> 22.122346.114 _host_remove_job: 0x5622fb64ec70 > >> >> 22.122346.114 host_remove_job: qsize=0 host->qsize=0 > >> >> 22.122346.114 [0] action=1 pending=0 host=0x5622fb64e6b0 goaway=false > >> >> 22.122346.114 closing connection > >> >> 22.122346.115 [0] action=1 pending=0 host=0x0 goaway=n/a > >> >> 22.122346.115 main: wake up > >> >> 22.122346.115 main: done > >> >> 22.122346.115 Successfully updated '/home/miles/.local/share/ > >> wget/.wget-hsts'. > >> >> 22.122346.115 Saved 1 HSTS entry into '/home/miles/.local/share/ > >> wget/.wget-hsts' > >> >> 22.122346.115 blacklist https://www.bbc.co.uk/webarchive/ > >> https%253A%252F%252Fwww.bbc.co.uk%252Fblogs%252Fadamcurtis%252Fentries%252Fa2094c9d-9864-348e-a241-7aa9 > >> > >> <https://www.bbc.co.uk/webarchive/https%253A%252F%252Fwww.bbc.co.uk%252Fblogs%252Fadamcurtis%252Fentries%252Fa2094c9d-9864-348e-a241-7aa9> > >> >> ---- > >> >> Certificate info [0]: > >> >> Valid since: Mon 13 Jul 2026 14:21:36 BST > >> >> Expires: Thu 28 Jan 2027 13:21:36 GMT > >> >> Fingerprint: e3756b8bc1f345cc3018a2b146b88ca5 > >> >> Serial number: 518996aacb43bd1f78166375 > >> >> Public key: EC/ECDSA, High (256 bits) > >> >> Version: #3 > >> >> DN: C=GB,ST=London,L=London,O=BRITISH BROADCASTING > >> CORPORATION,CN=www.bbc.co.uk <http://www.bbc.co.uk> > >> >> Issuer's DN: C=BE,O=GlobalSign nv-sa,CN=GlobalSign ECC OV SSL CA > >> 2018 > >> >> Issuer's OID: 2.5.4.6 > >> >> Issuer's UID: 2.5.4.6 > >> >> Certificate info [1]: > >> >> Valid since: Wed 21 Nov 2018 00:00:00 GMT > >> >> Expires: Tue 21 Nov 2028 00:00:00 GMT > >> >> Fingerprint: 9635de986bc7f9b19f09aec50e3a3909 > >> >> Public key: EC/ECDSA, Ultra (384 bits) > >> >> Version: #3 > >> >> DN: C=BE,O=GlobalSign nv-sa,CN=GlobalSign ECC OV SSL CA 2018 > >> >> Issuer's DN: OU=GlobalSign ECC Root CA - > >> R5,O=GlobalSign,CN=GlobalSign > >> >> Issuer's OID: 2.5.4.11 > >> >> Issuer's UID: 2.5.4.11 > >> >> Certificate info [2]: > >> >> Valid since: Wed 21 Nov 2018 00:00:00 GMT > >> >> Expires: Sun 18 Mar 2029 10:00:00 GMT > >> >> Fingerprint: 8f665697f9d7f2c6c6b96a99259b7ffc > >> >> Serial number: 01ee5f223e3d355b992bc82ab4 > >> >> Public key: EC/ECDSA, Ultra (384 bits) > >> >> Version: #3 > >> >> DN: OU=GlobalSign ECC Root CA - R5,O=GlobalSign,CN=GlobalSign > >> >> Issuer's DN: OU=GlobalSign Root CA - R3,O=GlobalSign,CN=GlobalSign > >> >> Issuer's OID: 2.5.4.11 > >> >> Issuer's UID: 2.5.4.11 > >> >> Certificate info [3]: > >> >> Valid since: Wed 19 Sep 2018 01:00:00 BST > >> >> Expires: Fri 28 Jan 2028 12:00:00 GMT > >> >> Fingerprint: 58bfb0eacf17e2d0a65a4d061042c3bf > >> >> Serial number: 01ee5f169dff97352b6465d66a > >> >> Public key: RSA, Medium (2048 bits) > >> >> Version: #3 > >> >> DN: OU=GlobalSign Root CA - R3,O=GlobalSign,CN=GlobalSign > >> >> Issuer's DN: C=BE,O=GlobalSign nv-sa,OU=Root CA,CN=GlobalSign > >> Root CA > >> >> Issuer's OID: 2.5.4.6 > >> >> Issuer's UID: 2.5.4.6 > >> >> ---- > >> >> Ephemeral ECDH using curve (null) > >> >> Key Exchange: ECDHE-RSA > >> >> Protocol: TLS1.3 > >> >> Certificate Type: X.509 > >> >> Cipher: AES-256-GCM > >> >> MAC: AEAD > >> >> ---- > >> >> [0] Downloading 'https://www.bbc.co.uk/webarchive/ > >> https%253A%252F%252Fwww.bbc.co.uk%252Fblogs%252Fadamcurtis%252Fentries%252Fa2094c9d-9864-348e-a241-7aa9 > >> > >> <https://www.bbc.co.uk/webarchive/https%253A%252F%252Fwww.bbc.co.uk%252Fblogs%252Fadamcurtis%252Fentries%252Fa2094c9d-9864-348e-a241-7aa9>' > >> ... > >> >> HTTP ERROR response 404 [https://www.bbc.co.uk/webarchive/ > >> https%253A%252F%252Fwww.bbc.co.uk%252Fblogs%252Fadamcurtis%252Fentries%252Fa2094c9d-9864-348e-a241-7aa9 > >> > >> <https://www.bbc.co.uk/webarchive/https%253A%252F%252Fwww.bbc.co.uk%252Fblogs%252Fadamcurtis%252Fentries%252Fa2094c9d-9864-348e-a241-7aa9>] > >> >> > >> > >> > >> > >> *Attachments:* > >> > >> * OpenPGP_signature.asc > > > > > > *Attachments:* > • OpenPGP_signature.asc
