Re: [Bug-wget] Wget Suggestion
On 08/11/12 10:30, horse6 wrote:
> Dear Sir:
>
> how use wget.exe to Memory-Resident?
>
> my mean is ( add parameter -MR:n )
>
> as: "wget.exe https://any.com/my.htm --no-check-certificate -nc -b -MR:15
> -Omy.htm"
> if not file "my.htm", auto download & after download ,every 15 seconds loop
> to check isfile("my.htm") or not
>
> ** when i get & read "my.htm", i will auto delete it & wait to next new
> my.htm
>
> thank you very much!
>
> 2012-11-08
>
> horse6
If you want your program to delete the file after reading it, and wget
to automatically redownload it.
Why don't you just ask wget to download it just before you go to read it?
Re: [Bug-wget] Supercookie issues
On 09/11/12 20:17, Tim Rühsen wrote: > Am Freitag, 9. November 2012 schrieb Ángel González: >> I see little reason for concern about supercookies on wget given that it >> is unlikely >> to use it for different "tasks" in the same invocation, and cookies are not >> automatically loaded/saved accross invocations. >> And for having a supercookie passed in the same run (eg. one website >> redirected >> to the other), they are probably cooperating domains, so the supercookie >> doesn't >> add much information. >> You would need to be using --load-cookies and --save-cookies to allow such >> supercookie spying. > That's what i use wget. Logging in on a website and accessing my private data. > One could do that even with one call to wget, so --load/save-cookies is not > even needed. > >> The worst case is probably if the cookie file was shared with a browser, >> or it was >> taken from a browser (with many cookies unrelated for what is intended) and >> passed to wget with --load-cookies and wget sent more cookies than >> expected . > That is one possibility, but as i said, you won't need --load/save-cookies to > be vulnerable. Can you provide an example on how you are using wget, that leads you to think you would be vulnerable? I think you may be misunderstanding something (but perhaps it turns out it's me who is wrong!). Maybe it is most dangerous if you have load_cookies and save_cookies set in your wgetrc, instead of passing an appropiate one for each task. Even then, the case you mention of stealing your private data is quite hard, as the to-be-stolen website is very unlikely to use a "vulnerable" domain (ie. not caught by the general rules used by wget). >> Although not too important, it should be fixed, of course. The Mozilla >> Public Suffix >> List isn't very simple for reuse, its format is designed for how they >> use it internally. > No, the format is easy, clear, understandable und well documented. > > Maybe I didn't say understandable in my first post: > The code is there and tested > You just have to call cookie_load_public_suffixes(filename), call repeatedly > cookie_suffix_match(domain). Very easy. I answered without checking, from what I remembered. Reading it now, it does seem quite straightforward to interpret the list. Although perhaps a bit more complex to do that efficiently for multiple domains. I don't know where's that function you mention, I don't see it in the website. Perhaps it belongs to your mget?
Re: [Bug-wget] Supercookie issues
Am Freitag, 9. November 2012 schrieb Ángel González: > On 09/11/12 16:27, Tim Ruehsen wrote: > > While implementing cookies for Mget (https://github.com/rockdaboot/mget) > > conforming to RFC 6265, I stubled over http://publicsuffix.org/ (Mozilla > > Public Suffix List). > > > > Looking at Wget sources discovers, that there is just a very incomplete check > > for public suffixes. That implies a very severe vulnerability to "supercookie" > > attacks when cookies are switched on (they are by default). > > > > Since Mget was ment as a Wget2 candidate (all or parts of the sources), please > > feel free to copy the needed sourcecode from it (see cookie.c/cookie.h and > > tests/test.c for test routines). Right now, I just don't have the time to do > > the work, but of course I will answer your questions. > > > > ShouldN't there be a warning within the docs / man pages. > > What do you think ? > > > > Regards, Tim > I see little reason for concern about supercookies on wget given that it > is unlikely > to use it for different "tasks" in the same invocation, and cookies are not > automatically loaded/saved accross invocations. > And for having a supercookie passed in the same run (eg. one website > redirected > to the other), they are probably cooperating domains, so the supercookie > doesn't > add much information. > You would need to be using --load-cookies and --save-cookies to allow such > supercookie spying. That's what i use wget. Logging in on a website and accessing my private data. One could do that even with one call to wget, so --load/save-cookies is not even needed. > The worst case is probably if the cookie file was shared with a browser, > or it was > taken from a browser (with many cookies unrelated for what is intended) and > passed to wget with --load-cookies and wget sent more cookies than > expected . That is one possibility, but as i said, you won't need --load/save-cookies to be vulnerable. > Although not too important, it should be fixed, of course. The Mozilla > Public Suffix > List isn't very simple for reuse, its format is designed for how they > use it internally. No, the format is easy, clear, understandable und well documented. Maybe I didn't say understandable in my first post: The code is there and tested You just have to call cookie_load_public_suffixes(filename), call repeatedly cookie_suffix_match(domain). Very easy. Regards, Tim
Re: [Bug-wget] Supercookie issues
On 09/11/12 16:27, Tim Ruehsen wrote: > While implementing cookies for Mget (https://github.com/rockdaboot/mget) > conforming to RFC 6265, I stubled over http://publicsuffix.org/ (Mozilla > Public Suffix List). > > Looking at Wget sources discovers, that there is just a very incomplete check > for public suffixes. That implies a very severe vulnerability to > "supercookie" > attacks when cookies are switched on (they are by default). > > Since Mget was ment as a Wget2 candidate (all or parts of the sources), > please > feel free to copy the needed sourcecode from it (see cookie.c/cookie.h and > tests/test.c for test routines). Right now, I just don't have the time to do > the work, but of course I will answer your questions. > > ShouldN't there be a warning within the docs / man pages. > What do you think ? > > Regards, Tim I see little reason for concern about supercookies on wget given that it is unlikely to use it for different "tasks" in the same invocation, and cookies are not automatically loaded/saved accross invocations. And for having a supercookie passed in the same run (eg. one website redirected to the other), they are probably cooperating domains, so the supercookie doesn't add much information. You would need to be using --load-cookies and --save-cookies to allow such supercookie spying. The worst case is probably if the cookie file was shared with a browser, or it was taken from a browser (with many cookies unrelated for what is intended) and passed to wget with --load-cookies and wget sent more cookies than expected . Although not too important, it should be fixed, of course. The Mozilla Public Suffix List isn't very simple for reuse, its format is designed for how they use it internally.
[Bug-wget] Supercookie issues
While implementing cookies for Mget (https://github.com/rockdaboot/mget) conforming to RFC 6265, I stubled over http://publicsuffix.org/ (Mozilla Public Suffix List). Looking at Wget sources discovers, that there is just a very incomplete check for public suffixes. That implies a very severe vulnerability to "supercookie" attacks when cookies are switched on (they are by default). Since Mget was ment as a Wget2 candidate (all or parts of the sources), please feel free to copy the needed sourcecode from it (see cookie.c/cookie.h and tests/test.c for test routines). Right now, I just don't have the time to do the work, but of course I will answer your questions. ShouldN't there be a warning within the docs / man pages. What do you think ? Regards, Tim
[Bug-wget] [PATCH] Test-cookies.px
To check wget for NOT sending a request header, I patched HTTPServer.pm.
Here is an example taken from the patched Test-cookies.px (see below):
...
request_headers => {
"!Cookie" => qr|foo=bar|,
},
...
That means: return an error, if there is a Cookie request header that contains
foo=bar. (I hope, I understood that perl stuff correctly).
I am not perl programmer, so maybe my changes are suboptimal. Please someone
check that.
Then, I expanded Test-cookies.px to delete the session cookie by using an
outdated 'Expires' tag. (works fine).
Also included is a test for testing the ignorance of a non-matching domain in
the Set-Cookie header. (wget fails, but shouldn't due to RFC 6265 5.3.6).
wget just replaces that foreign domain with the domain from the request-uri.
Should things like that go to the bug tracker ? (There are so many open issues
for so long time that I am unshure about that.)
Regards, Tim
From 52433c758c416e1f448ffe743c6cf502b4be4c5a Mon Sep 17 00:00:00 2001
From: Tim Ruehsen
Date: Fri, 9 Nov 2012 15:51:23 +0100
Subject: [PATCH 2/2] check cookie deletion and cookie domain matching
---
tests/Test-cookies.px | 57 -
1 file changed, 56 insertions(+), 1 deletion(-)
diff --git a/tests/Test-cookies.px b/tests/Test-cookies.px
index a4b9125..ac7da56 100755
--- a/tests/Test-cookies.px
+++ b/tests/Test-cookies.px
@@ -10,6 +10,10 @@ use HTTPTest;
my $page1 = "Hello, world!\n";
my $page2 = "Goodbye, Sam.\n";
+my $page3 = "Page three.\n";
+my $page4 = "Page four.\n";
+my $page5 = "Page five.\n";
+my $page6 = "Page six.\n";
# code, msg, headers, content
my %urls = (
@@ -30,10 +34,49 @@ my %urls = (
"Cookie" => qr|foo=bar|,
},
},
+# remove the cookie 'foo'
+'/three.txt' => {
+code => "200",
+msg => "Ok",
+headers => {
+"Content-type" => "text/plain",
+"Set-Cookie" => "foo=; Expires=Sun, 06 Nov 1994 08:49:37 GMT",
+},
+content => $page3,
+},
+'/four.txt' => {
+code => "200",
+msg => "Ok",
+content => $page4,
+request_headers => {
+"!Cookie" => qr|foo=|,
+},
+},
+# try to set a cookie 'foo' with mismatching domain
+# see RFC 6265 5.3.6: ignore the cookie if it doesn't domain-match
+'/five.txt' => {
+code => "200",
+msg => "Ok",
+headers => {
+"Content-type" => "text/plain",
+"Set-Cookie" => "foo=bar; domain=.example.com",
+},
+content => $page5,
+},
+'/six.txt' => {
+code => "200",
+msg => "Ok",
+content => $page6,
+request_headers => {
+"!Cookie" => qr|foo=bar|,
+},
+},
);
my $cmdline = $WgetTest::WGETPATH . " http://localhost:{{port}}/one.txt";
-. " http://localhost:{{port}}/two.txt";;
+. " http://localhost:{{port}}/two.txt"; . " http://localhost:{{port}}/three.txt";
+. " http://localhost:{{port}}/four.txt"; . " http://localhost:{{port}}/five.txt";
+ . " http://localhost:{{port}}/six.txt";;
my $expected_error_code = 0;
@@ -44,6 +87,18 @@ my %expected_downloaded_files = (
'two.txt' => {
content => $page2,
},
+'three.txt' => {
+content => $page3,
+},
+'four.txt' => {
+content => $page4,
+},
+'five.txt' => {
+content => $page5,
+},
+'six.txt' => {
+content => $page6,
+},
);
###
--
1.7.10.4
From 6df4298e93f54ef6a284f5e58666c6158f472fed Mon Sep 17 00:00:00 2001
From: Tim Ruehsen
Date: Fri, 9 Nov 2012 15:50:03 +0100
Subject: [PATCH 1/2] added check for must-not-match request-header
---
tests/HTTPServer.pm | 22 +-
1 file changed, 17 insertions(+), 5 deletions(-)
diff --git a/tests/HTTPServer.pm b/tests/HTTPServer.pm
index 627c102..065ea1e 100644
--- a/tests/HTTPServer.pm
+++ b/tests/HTTPServer.pm
@@ -218,12 +218,24 @@ sub verify_request_headers {
return 1 unless exists $url_rec->{'request_headers'};
for my $hdrname (keys %{$url_rec->{'request_headers'}}) {
-my $rhdr = $req->header ($hdrname);
+my $must_not_match;
my $ehdr = $url_rec->{'request_headers'}{$hdrname};
-unless (defined $rhdr && $rhdr =~ $ehdr) {
-$rhdr = '' unless defined $rhdr;
-print STDERR "\n*** Mismatch on $hdrname: $rhdr =~ $ehdr\n";
-return undef;
+if ($must_not_match = ($hdrname =~ /^!(\w+)/)) {
+$hdrname = $1;
+}
+my $rhdr = $req->header ($hdrname);
+if ($must_not_match) {
+if (defined $rhdr && $rhdr =~ $ehdr) {
+$rhdr = '' unless defined $rhdr;
+print STDERR "\n*** Match forbidden $hdrname: $rhdr =~ $ehdr\n";
+return undef;
+}
+} else {
+u
[Bug-wget] [PATCH] Test-cookies.px
To check wget for NOT sending a request header, I patched HTTPServer.pm.
Here is an example taken from the patched Test-cookies.px (see below):
...
request_headers => {
"!Cookie" => qr|foo=bar|,
},
...
That means: return an error, if there is a Cookie request header that contains
foo=bar. (I hope, I understood that perl stuff correctly).
I am not perl programmer, so maybe my changes are suboptimal. Please someone
check that.
Then, I expanded Test-cookies.px to delete the session cookie by using an
outdated 'Expires' tag. (works fine).
Also included is a test for testing the ignorance of a non-matching domain in
the Set-Cookie header. (wget fails, but shouldn't due to RFC 6265 5.3.6).
wget just replaces that foreign domain with the domain from the request-uri.
Should things like that go to the bug tracker ? (There are so many open issues
for so long time that I am unshure about that.)
Regards, Tim
Re: [Bug-wget] Wget Suggestion
Hello horse6,
why not simply using a shell script ?
I don't exactly get what you need, but it should look something like:
for ((;;)); do
wget -nc -O my.htm www.example.com
test -e my.htm && exit 0 # exit successful download
sleep 15 # wait 15sm than try again
done
You could also check wget's return code.
Tim
Am Thursday 08 November 2012 schrieb horse6:
> Dear Sir:
>
> how use wget.exe to Memory-Resident?
>
> my mean is ( add parameter -MR:n )
>
> as: "wget.exe https://any.com/my.htm --no-check-certificate -nc -b -MR:15
> -Omy.htm" if not file "my.htm", auto download & after download ,every 15
> seconds loop to check isfile("my.htm") or not
>
> ** when i get & read "my.htm", i will auto delete it & wait to next new
> my.htm
>
> thank you very much!
>
> 2012-11-08
>
>
>
> horse6
