[Bug-wget] [PATCH 1/3] Added tests for HTTP authentication using credentials from .netrc

2017-05-12 Thread Tomas Hozza 
Getting credentials from .netrc has been broken from time to time, thus
adding a test coverage to prevent regressions.

Also added setting of "HOME" environment variable when executing wget,
to make sure LocalFiles like .netrc, which are created just for the
test, are actually used.

Signed-off-by: Tomas Hozza 
---
 testenv/Makefile.am |  3 ++
 testenv/Test-auth-basic-netrc-pass-given.py | 68 +
 testenv/Test-auth-basic-netrc-user-given.py | 68 +
 testenv/Test-auth-basic-netrc.py| 66 
 testenv/test/base_test.py   |  2 +-
 5 files changed, 206 insertions(+), 1 deletion(-)
 create mode 100755 testenv/Test-auth-basic-netrc-pass-given.py
 create mode 100755 testenv/Test-auth-basic-netrc-user-given.py
 create mode 100755 testenv/Test-auth-basic-netrc.py

diff --git a/testenv/Makefile.am b/testenv/Makefile.am
index 3febec7..7104314 100644
--- a/testenv/Makefile.am
+++ b/testenv/Makefile.am
@@ -75,6 +75,9 @@ if HAVE_PYTHON3
   TESTS = Test-504.py   \
 Test-auth-basic-fail.py \
 Test-auth-basic.py  \
+Test-auth-basic-netrc.py\
+Test-auth-basic-netrc-user-given.py \
+Test-auth-basic-netrc-pass-given.py \
 Test-auth-both.py   \
 Test-auth-digest.py \
 Test-auth-no-challenge.py   \
diff --git a/testenv/Test-auth-basic-netrc-pass-given.py 
b/testenv/Test-auth-basic-netrc-pass-given.py
new file mode 100755
index 000..43dfe34
--- /dev/null
+++ b/testenv/Test-auth-basic-netrc-pass-given.py
@@ -0,0 +1,68 @@
+#!/usr/bin/env python3
+from sys import exit
+from test.http_test import HTTPTest
+from misc.wget_file import WgetFile
+
+"""
+This test ensures Wget uses credentials from .netrc for Basic 
Authorization Negotiation.
+In this case we test that .netrc credentials are used in case only
+password is given on the command line.
+Also, we ensure that Wget saves the host after a successful auth and
+doesn't wait for a challenge the second time.
+"""
+# File Definitions ###
+File1 = "I am an invisble man."
+File2 = "I too am an invisible man."
+
+User = "Sauron"
+Password = "TheEye"
+
+File1_rules = {
+"Authentication": {
+"Type"  : "Basic",
+"User"  : User,
+"Pass"  : Password
+}
+}
+File2_rules = {
+"ExpectHeader"  : {
+"Authorization" : "Basic U2F1cm9uOlRoZUV5ZQ=="
+}
+}
+
+Netrc = "machine 127.0.0.1\n\tlogin {0}".format(User)
+
+A_File = WgetFile ("File1", File1, rules=File1_rules)
+B_File = WgetFile ("File2", File2, rules=File2_rules)
+Netrc_File = WgetFile (".netrc", Netrc)
+
+WGET_OPTIONS = "--password={0}".format(Password)
+WGET_URLS = [["File1", "File2"]]
+
+Files = [[A_File, B_File]]
+LocalFiles = [Netrc_File]
+
+ExpectedReturnCode = 0
+ExpectedDownloadedFiles = [A_File, B_File, Netrc_File]
+
+ Pre and Post Test Hooks #
+pre_test = {
+"ServerFiles"   : Files,
+"LocalFiles": LocalFiles
+}
+test_options = {
+"WgetCommands"  : WGET_OPTIONS,
+"Urls"  : WGET_URLS
+}
+post_test = {
+"ExpectedFiles" : ExpectedDownloadedFiles,
+"ExpectedRetcode"   : ExpectedReturnCode
+}
+
+err = HTTPTest (
+pre_hook=pre_test,
+test_params=test_options,
+post_hook=post_test
+).begin ()
+
+exit (err)
diff --git a/testenv/Test-auth-basic-netrc-user-given.py 
b/testenv/Test-auth-basic-netrc-user-given.py
new file mode 100755
index 000..57b6148
--- /dev/null
+++ b/testenv/Test-auth-basic-netrc-user-given.py
@@ -0,0 +1,68 @@
+#!/usr/bin/env python3
+from sys import exit
+from test.http_test import HTTPTest
+from misc.wget_file import WgetFile
+
+"""
+This test ensures Wget uses credentials from .netrc for Basic 
Authorization Negotiation.
+In this case we test that .netrc credentials are used in case only
+user login is given on the command line.
+Also, we ensure that Wget saves the host after a successful auth and
+doesn't wait for a challenge the second time.
+"""
+# File Definitions ###
+File1 = "I am an invisble man."
+File2 = "I too am an invisible man."
+
+User = "Sauron"
+Password = "TheEye"
+
+File1_rules = {
+"Authentication": {
+"Type"  : "Basic",
+"User"  : User,
+"Pass"  : Password
+}
+}
+File2_rules = {
+"ExpectHeader"  : {
+"Authorization" : "Basic U2F1cm9uOlRoZUV5ZQ=="
+}
+}
+
+Netrc = "machine 127.0.0.1\n\tlogin {0}\n\tpassword {1}".format(User, Password)
+
+A_File = WgetFile ("File1", File1, rules=

Re: [Bug-wget] [PATCH 1/3] Added tests for HTTP authentication using credentials from .netrc

2017-05-15 Thread Tim Rühsen 
Hi Tomas,

thank you very much for the fixes and the new tests !

Currently, patch 1/3 breaks two Metalink tests:

FAIL: Test-metalink-http.py
FAIL: Test-metalink-http-quoted.py

I am still investigating, but it has something to do with the
environment that you reset in testenv/test/base_test.py:

ret_code = call(params, env={"HOME": os.getcwd()})

Maybe a bug that just comes out with a different locale/charset encoding.


With Best Regards, Tim



On 05/12/2017 07:17 PM, Tomas Hozza wrote:
> Getting credentials from .netrc has been broken from time to time, thus
> adding a test coverage to prevent regressions.
> 
> Also added setting of "HOME" environment variable when executing wget,
> to make sure LocalFiles like .netrc, which are created just for the
> test, are actually used.
> 
> Signed-off-by: Tomas Hozza 
> ---
>  testenv/Makefile.am |  3 ++
>  testenv/Test-auth-basic-netrc-pass-given.py | 68 
> +
>  testenv/Test-auth-basic-netrc-user-given.py | 68 
> +
>  testenv/Test-auth-basic-netrc.py| 66 
>  testenv/test/base_test.py   |  2 +-
>  5 files changed, 206 insertions(+), 1 deletion(-)
>  create mode 100755 testenv/Test-auth-basic-netrc-pass-given.py
>  create mode 100755 testenv/Test-auth-basic-netrc-user-given.py
>  create mode 100755 testenv/Test-auth-basic-netrc.py
> 
> diff --git a/testenv/Makefile.am b/testenv/Makefile.am
> index 3febec7..7104314 100644
> --- a/testenv/Makefile.am
> +++ b/testenv/Makefile.am
> @@ -75,6 +75,9 @@ if HAVE_PYTHON3
>TESTS = Test-504.py   \
>  Test-auth-basic-fail.py \
>  Test-auth-basic.py  \
> +Test-auth-basic-netrc.py\
> +Test-auth-basic-netrc-user-given.py \
> +Test-auth-basic-netrc-pass-given.py \
>  Test-auth-both.py   \
>  Test-auth-digest.py \
>  Test-auth-no-challenge.py   \
> diff --git a/testenv/Test-auth-basic-netrc-pass-given.py 
> b/testenv/Test-auth-basic-netrc-pass-given.py
> new file mode 100755
> index 000..43dfe34
> --- /dev/null
> +++ b/testenv/Test-auth-basic-netrc-pass-given.py
> @@ -0,0 +1,68 @@
> +#!/usr/bin/env python3
> +from sys import exit
> +from test.http_test import HTTPTest
> +from misc.wget_file import WgetFile
> +
> +"""
> +This test ensures Wget uses credentials from .netrc for Basic 
> Authorization Negotiation.
> +In this case we test that .netrc credentials are used in case only
> +password is given on the command line.
> +Also, we ensure that Wget saves the host after a successful auth and
> +doesn't wait for a challenge the second time.
> +"""
> +# File Definitions 
> ###
> +File1 = "I am an invisble man."
> +File2 = "I too am an invisible man."
> +
> +User = "Sauron"
> +Password = "TheEye"
> +
> +File1_rules = {
> +"Authentication": {
> +"Type"  : "Basic",
> +"User"  : User,
> +"Pass"  : Password
> +}
> +}
> +File2_rules = {
> +"ExpectHeader"  : {
> +"Authorization" : "Basic U2F1cm9uOlRoZUV5ZQ=="
> +}
> +}
> +
> +Netrc = "machine 127.0.0.1\n\tlogin {0}".format(User)
> +
> +A_File = WgetFile ("File1", File1, rules=File1_rules)
> +B_File = WgetFile ("File2", File2, rules=File2_rules)
> +Netrc_File = WgetFile (".netrc", Netrc)
> +
> +WGET_OPTIONS = "--password={0}".format(Password)
> +WGET_URLS = [["File1", "File2"]]
> +
> +Files = [[A_File, B_File]]
> +LocalFiles = [Netrc_File]
> +
> +ExpectedReturnCode = 0
> +ExpectedDownloadedFiles = [A_File, B_File, Netrc_File]
> +
> + Pre and Post Test Hooks 
> #
> +pre_test = {
> +"ServerFiles"   : Files,
> +"LocalFiles": LocalFiles
> +}
> +test_options = {
> +"WgetCommands"  : WGET_OPTIONS,
> +"Urls"  : WGET_URLS
> +}
> +post_test = {
> +"ExpectedFiles" : ExpectedDownloadedFiles,
> +"ExpectedRetcode"   : ExpectedReturnCode
> +}
> +
> +err = HTTPTest (
> +pre_hook=pre_test,
> +test_params=test_options,
> +post_hook=post_test
> +).begin ()
> +
> +exit (err)
> diff --git a/testenv/Test-auth-basic-netrc-user-given.py 
> b/testenv/Test-auth-basic-netrc-user-given.py
> new file mode 100755
> index 000..57b6148
> --- /dev/null
> +++ b/testenv/Test-auth-basic-netrc-user-given.py
> @@ -0,0 +1,68 @@
> +#!/usr/bin/env python3
> +from sys import exit
> +from test.http_test import HTTPTest
> +from misc.wget_file import WgetFile
> +
> +"""
> +This test ensures Wget uses credentials from .netrc for Basic 
> Authorization Negotiation.
> +In this case we test that .netrc credentials are used in case only
> +

Re: [Bug-wget] [PATCH 1/3] Added tests for HTTP authentication using credentials from .netrc

2017-05-15 Thread Tomas Hozza 
Hello Tim.

I run all the tests before sending patches and they all passed. If you can send 
me more info about the failures, I could have a look at it.

My locale is:
$ locale
LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC=sk_SK.UTF-8
LC_TIME=sk_SK.UTF-8
LC_COLLATE="en_US.UTF-8"
LC_MONETARY=sk_SK.UTF-8
LC_MESSAGES="en_US.UTF-8"
LC_PAPER=sk_SK.UTF-8
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT=sk_SK.UTF-8
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

Thanks.

Regards,
Tomas

On 15.05.2017 15:52, Tim Rühsen wrote:
> Hi Tomas,
>
> thank you very much for the fixes and the new tests !
>
> Currently, patch 1/3 breaks two Metalink tests:
>
> FAIL: Test-metalink-http.py
> FAIL: Test-metalink-http-quoted.py
>
> I am still investigating, but it has something to do with the
> environment that you reset in testenv/test/base_test.py:
>
> ret_code = call(params, env={"HOME": os.getcwd()})
>
> Maybe a bug that just comes out with a different locale/charset encoding.
>
>
> With Best Regards, Tim
>
>
>
> On 05/12/2017 07:17 PM, Tomas Hozza wrote:
> > Getting credentials from .netrc has been broken from time to time, thus
> > adding a test coverage to prevent regressions.
> >
> > Also added setting of "HOME" environment variable when executing wget,
> > to make sure LocalFiles like .netrc, which are created just for the
> > test, are actually used.
> >
> > Signed-off-by: Tomas Hozza 
> > ---
> >  testenv/Makefile.am |  3 ++
> >  testenv/Test-auth-basic-netrc-pass-given.py | 68 
> > +
> >  testenv/Test-auth-basic-netrc-user-given.py | 68 
> > +
> >  testenv/Test-auth-basic-netrc.py| 66 
> > 
> >  testenv/test/base_test.py   |  2 +-
> >  5 files changed, 206 insertions(+), 1 deletion(-)
> >  create mode 100755 testenv/Test-auth-basic-netrc-pass-given.py
> >  create mode 100755 testenv/Test-auth-basic-netrc-user-given.py
> >  create mode 100755 testenv/Test-auth-basic-netrc.py
> >
> > diff --git a/testenv/Makefile.am b/testenv/Makefile.am
> > index 3febec7..7104314 100644
> > --- a/testenv/Makefile.am
> > +++ b/testenv/Makefile.am
> > @@ -75,6 +75,9 @@ if HAVE_PYTHON3
> >TESTS = Test-504.py   \
> >  Test-auth-basic-fail.py \
> >  Test-auth-basic.py  \
> > +Test-auth-basic-netrc.py\
> > +Test-auth-basic-netrc-user-given.py \
> > +Test-auth-basic-netrc-pass-given.py \
> >  Test-auth-both.py   \
> >  Test-auth-digest.py \
> >  Test-auth-no-challenge.py   \
> > diff --git a/testenv/Test-auth-basic-netrc-pass-given.py 
> > b/testenv/Test-auth-basic-netrc-pass-given.py
> > new file mode 100755
> > index 000..43dfe34
> > --- /dev/null
> > +++ b/testenv/Test-auth-basic-netrc-pass-given.py
> > @@ -0,0 +1,68 @@
> > +#!/usr/bin/env python3
> > +from sys import exit
> > +from test.http_test import HTTPTest
> > +from misc.wget_file import WgetFile
> > +
> > +"""
> > +This test ensures Wget uses credentials from .netrc for Basic 
> > Authorization Negotiation.
> > +In this case we test that .netrc credentials are used in case only
> > +password is given on the command line.
> > +Also, we ensure that Wget saves the host after a successful auth and
> > +doesn't wait for a challenge the second time.
> > +"""
> > +# File Definitions 
> > ###
> > +File1 = "I am an invisble man."
> > +File2 = "I too am an invisible man."
> > +
> > +User = "Sauron"
> > +Password = "TheEye"
> > +
> > +File1_rules = {
> > +"Authentication": {
> > +"Type"  : "Basic",
> > +"User"  : User,
> > +"Pass"  : Password
> > +}
> > +}
> > +File2_rules = {
> > +"ExpectHeader"  : {
> > +"Authorization" : "Basic U2F1cm9uOlRoZUV5ZQ=="
> > +}
> > +}
> > +
> > +Netrc = "machine 127.0.0.1\n\tlogin {0}".format(User)
> > +
> > +A_File = WgetFile ("File1", File1, rules=File1_rules)
> > +B_File = WgetFile ("File2", File2, rules=File2_rules)
> > +Netrc_File = WgetFile (".netrc", Netrc)
> > +
> > +WGET_OPTIONS = "--password={0}".format(Password)
> > +WGET_URLS = [["File1", "File2"]]
> > +
> > +Files = [[A_File, B_File]]
> > +LocalFiles = [Netrc_File]
> > +
> > +ExpectedReturnCode = 0
> > +ExpectedDownloadedFiles = [A_File, B_File, Netrc_File]
> > +
> > + Pre and Post Test Hooks 
> > #
> > +pre_test = {
> > +"ServerFiles"   : Files,
> > +"LocalFiles": LocalFiles
> > +}
> > +test_options = {
> > +"WgetCommands"  : WGET_OPTIONS,
> > +"Urls"  : WGET_URLS
> > +}
> > +post_test = {
> > +"ExpectedFiles" : ExpectedDownloa

Re: [Bug-wget] [PATCH 1/3] Added tests for HTTP authentication using credentials from .netrc

2017-05-15 Thread Tim Rühsen 
P.S.:

Just saw that the byte seems a bit random, on a second run I got:

'utf-8' codec can't decode byte 0xc3 in position 18: invalid
continuation byte


With Best Regards, Tim



On 05/15/2017 05:04 PM, Tim Rühsen wrote:
> Hi Tomas,
> 
> my locale looks similar:
> 
> $ locale
> LANG=en_US.UTF-8
> LANGUAGE=en_US.UTF-8
> LC_CTYPE="en_US.UTF-8"
> LC_NUMERIC="en_US.UTF-8"
> LC_TIME="en_US.UTF-8"
> LC_COLLATE="en_US.UTF-8"
> LC_MONETARY="en_US.UTF-8"
> LC_MESSAGES="en_US.UTF-8"
> LC_PAPER="en_US.UTF-8"
> LC_NAME="en_US.UTF-8"
> LC_ADDRESS="en_US.UTF-8"
> LC_TELEPHONE="en_US.UTF-8"
> LC_MEASUREMENT="en_US.UTF-8"
> LC_IDENTIFICATION="en_US.UTF-8"
> LC_ALL=en_US.UTF-8
> 
> 
> Reproducable here *without* your patches:
> 
> $ TESTS_ENVIRONMENT="env -i; " make check
> 
> 
> The system is Debian unstable amd64.
> I asked Darshit, the author of the python script, if he could take a
> look... maybe this is something trivial for a Pythonista.
> 
> The traceback is:
> 
> Unhandled exception caught.
> 'ascii' codec can't decode byte 0xc1 in position 18: ordinal not in
> range(128)
> Traceback (most recent call last):
>   File "./Test-metalink-http.py", line 124, in 
> err = http_test.begin ()
>   File "/usr/oms/src/wget1.x/testenv/test/http_test.py", line 41, in begin
> self.do_test()
>   File "/usr/oms/src/wget1.x/testenv/test/base_test.py", line 187, in
> do_test
> self.post_hook_call()
>   File "/usr/oms/src/wget1.x/testenv/test/base_test.py", line 206, in
> post_hook_call
> self.hook_call(self.post_configs, 'Post Test Function')
>   File "/usr/oms/src/wget1.x/testenv/test/base_test.py", line 196, in
> hook_call
> conf.find_conf(conf_name)(conf_arg)(self)
>   File "/usr/oms/src/wget1.x/testenv/conf/expected_files.py", line 36,
> in __call__
> local_fs = self.gen_local_fs_snapshot()
>   File "/usr/oms/src/wget1.x/testenv/conf/expected_files.py", line 30,
> in gen_local_fs_snapshot
> f['content'] = fp.read()
>   File "/usr/lib/python3.5/encodings/ascii.py", line 26, in decode
> return codecs.ascii_decode(input, self.errors)[0]
> UnicodeDecodeError: 'ascii' codec can't decode byte 0xc1 in position 18:
> ordinal not in range(128)
> 
> 
> With Best Regards, Tim
> 
> 
> 
> On 05/15/2017 04:53 PM, Tomas Hozza wrote:
>> Hello Tim.
>>
>> I run all the tests before sending patches and they all passed. If you can 
>> send me more info about the failures, I could have a look at it.
>>
>> My locale is:
>> $ locale
>> LANG=en_US.UTF-8
>> LC_CTYPE="en_US.UTF-8"
>> LC_NUMERIC=sk_SK.UTF-8
>> LC_TIME=sk_SK.UTF-8
>> LC_COLLATE="en_US.UTF-8"
>> LC_MONETARY=sk_SK.UTF-8
>> LC_MESSAGES="en_US.UTF-8"
>> LC_PAPER=sk_SK.UTF-8
>> LC_NAME="en_US.UTF-8"
>> LC_ADDRESS="en_US.UTF-8"
>> LC_TELEPHONE="en_US.UTF-8"
>> LC_MEASUREMENT=sk_SK.UTF-8
>> LC_IDENTIFICATION="en_US.UTF-8"
>> LC_ALL=
>>
>> Thanks.
>>
>> Regards,
>> Tomas
>>
>> On 15.05.2017 15:52, Tim Rühsen wrote:
>>> Hi Tomas,
>>>
>>> thank you very much for the fixes and the new tests !
>>>
>>> Currently, patch 1/3 breaks two Metalink tests:
>>>
>>> FAIL: Test-metalink-http.py
>>> FAIL: Test-metalink-http-quoted.py
>>>
>>> I am still investigating, but it has something to do with the
>>> environment that you reset in testenv/test/base_test.py:
>>>
>>> ret_code = call(params, env={"HOME": os.getcwd()})
>>>
>>> Maybe a bug that just comes out with a different locale/charset encoding.
>>>
>>>
>>> With Best Regards, Tim
>>>
>>>
>>>
>>> On 05/12/2017 07:17 PM, Tomas Hozza wrote:
 Getting credentials from .netrc has been broken from time to time, thus
 adding a test coverage to prevent regressions.

 Also added setting of "HOME" environment variable when executing wget,
 to make sure LocalFiles like .netrc, which are created just for the
 test, are actually used.

 Signed-off-by: Tomas Hozza 
 ---
  testenv/Makefile.am |  3 ++
  testenv/Test-auth-basic-netrc-pass-given.py | 68 
 +
  testenv/Test-auth-basic-netrc-user-given.py | 68 
 +
  testenv/Test-auth-basic-netrc.py| 66 
 
  testenv/test/base_test.py   |  2 +-
  5 files changed, 206 insertions(+), 1 deletion(-)
  create mode 100755 testenv/Test-auth-basic-netrc-pass-given.py
  create mode 100755 testenv/Test-auth-basic-netrc-user-given.py
  create mode 100755 testenv/Test-auth-basic-netrc.py

 diff --git a/testenv/Makefile.am b/testenv/Makefile.am
 index 3febec7..7104314 100644
 --- a/testenv/Makefile.am
 +++ b/testenv/Makefile.am
 @@ -75,6 +75,9 @@ if HAVE_PYTHON3
TESTS = Test-504.py   \
  Test-auth-basic-fail.py \
  Test-auth-basic.py  \
 +Test-auth-basic-netrc.py\
 +Test-auth-basic-netrc-user-given.py \
 +T

Re: [Bug-wget] [PATCH 1/3] Added tests for HTTP authentication using credentials from .netrc

2017-05-15 Thread Tim Rühsen 
Hi Tomas,

my locale looks similar:

$ locale
LANG=en_US.UTF-8
LANGUAGE=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=en_US.UTF-8


Reproducable here *without* your patches:

$ TESTS_ENVIRONMENT="env -i; " make check


The system is Debian unstable amd64.
I asked Darshit, the author of the python script, if he could take a
look... maybe this is something trivial for a Pythonista.

The traceback is:

Unhandled exception caught.
'ascii' codec can't decode byte 0xc1 in position 18: ordinal not in
range(128)
Traceback (most recent call last):
  File "./Test-metalink-http.py", line 124, in 
err = http_test.begin ()
  File "/usr/oms/src/wget1.x/testenv/test/http_test.py", line 41, in begin
self.do_test()
  File "/usr/oms/src/wget1.x/testenv/test/base_test.py", line 187, in
do_test
self.post_hook_call()
  File "/usr/oms/src/wget1.x/testenv/test/base_test.py", line 206, in
post_hook_call
self.hook_call(self.post_configs, 'Post Test Function')
  File "/usr/oms/src/wget1.x/testenv/test/base_test.py", line 196, in
hook_call
conf.find_conf(conf_name)(conf_arg)(self)
  File "/usr/oms/src/wget1.x/testenv/conf/expected_files.py", line 36,
in __call__
local_fs = self.gen_local_fs_snapshot()
  File "/usr/oms/src/wget1.x/testenv/conf/expected_files.py", line 30,
in gen_local_fs_snapshot
f['content'] = fp.read()
  File "/usr/lib/python3.5/encodings/ascii.py", line 26, in decode
return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc1 in position 18:
ordinal not in range(128)


With Best Regards, Tim



On 05/15/2017 04:53 PM, Tomas Hozza wrote:
> Hello Tim.
> 
> I run all the tests before sending patches and they all passed. If you can 
> send me more info about the failures, I could have a look at it.
> 
> My locale is:
> $ locale
> LANG=en_US.UTF-8
> LC_CTYPE="en_US.UTF-8"
> LC_NUMERIC=sk_SK.UTF-8
> LC_TIME=sk_SK.UTF-8
> LC_COLLATE="en_US.UTF-8"
> LC_MONETARY=sk_SK.UTF-8
> LC_MESSAGES="en_US.UTF-8"
> LC_PAPER=sk_SK.UTF-8
> LC_NAME="en_US.UTF-8"
> LC_ADDRESS="en_US.UTF-8"
> LC_TELEPHONE="en_US.UTF-8"
> LC_MEASUREMENT=sk_SK.UTF-8
> LC_IDENTIFICATION="en_US.UTF-8"
> LC_ALL=
> 
> Thanks.
> 
> Regards,
> Tomas
> 
> On 15.05.2017 15:52, Tim Rühsen wrote:
>> Hi Tomas,
>>
>> thank you very much for the fixes and the new tests !
>>
>> Currently, patch 1/3 breaks two Metalink tests:
>>
>> FAIL: Test-metalink-http.py
>> FAIL: Test-metalink-http-quoted.py
>>
>> I am still investigating, but it has something to do with the
>> environment that you reset in testenv/test/base_test.py:
>>
>> ret_code = call(params, env={"HOME": os.getcwd()})
>>
>> Maybe a bug that just comes out with a different locale/charset encoding.
>>
>>
>> With Best Regards, Tim
>>
>>
>>
>> On 05/12/2017 07:17 PM, Tomas Hozza wrote:
>>> Getting credentials from .netrc has been broken from time to time, thus
>>> adding a test coverage to prevent regressions.
>>>
>>> Also added setting of "HOME" environment variable when executing wget,
>>> to make sure LocalFiles like .netrc, which are created just for the
>>> test, are actually used.
>>>
>>> Signed-off-by: Tomas Hozza 
>>> ---
>>>  testenv/Makefile.am |  3 ++
>>>  testenv/Test-auth-basic-netrc-pass-given.py | 68 
>>> +
>>>  testenv/Test-auth-basic-netrc-user-given.py | 68 
>>> +
>>>  testenv/Test-auth-basic-netrc.py| 66 
>>> 
>>>  testenv/test/base_test.py   |  2 +-
>>>  5 files changed, 206 insertions(+), 1 deletion(-)
>>>  create mode 100755 testenv/Test-auth-basic-netrc-pass-given.py
>>>  create mode 100755 testenv/Test-auth-basic-netrc-user-given.py
>>>  create mode 100755 testenv/Test-auth-basic-netrc.py
>>>
>>> diff --git a/testenv/Makefile.am b/testenv/Makefile.am
>>> index 3febec7..7104314 100644
>>> --- a/testenv/Makefile.am
>>> +++ b/testenv/Makefile.am
>>> @@ -75,6 +75,9 @@ if HAVE_PYTHON3
>>>TESTS = Test-504.py   \
>>>  Test-auth-basic-fail.py \
>>>  Test-auth-basic.py  \
>>> +Test-auth-basic-netrc.py\
>>> +Test-auth-basic-netrc-user-given.py \
>>> +Test-auth-basic-netrc-pass-given.py \
>>>  Test-auth-both.py   \
>>>  Test-auth-digest.py \
>>>  Test-auth-no-challenge.py   \
>>> diff --git a/testenv/Test-auth-basic-netrc-pass-given.py 
>>> b/testenv/Test-auth-basic-netrc-pass-given.py
>>> new file mode 100755
>>> index 000..43dfe34
>>> --- /dev/null
>>> +++ b/testenv/Test-auth

Re: [Bug-wget] [PATCH 1/3] Added tests for HTTP authentication using credentials from .netrc

2017-05-16 Thread Tim Rühsen 
Hi Tomas,

fixed the problem by skipping the processing of 'pubring.kbx'. This file
is created by libgpgme if we change $HOME.

Your changes have been pushed now. Thanks for your work.

The charset/locale issue is a general problem when the test suite reads
and compares binary data. Currently, we don't have to fix that.



With Best Regards, Tim



On 05/15/2017 04:53 PM, Tomas Hozza wrote:
> Hello Tim.
> 
> I run all the tests before sending patches and they all passed. If you can 
> send me more info about the failures, I could have a look at it.
> 
> My locale is:
> $ locale
> LANG=en_US.UTF-8
> LC_CTYPE="en_US.UTF-8"
> LC_NUMERIC=sk_SK.UTF-8
> LC_TIME=sk_SK.UTF-8
> LC_COLLATE="en_US.UTF-8"
> LC_MONETARY=sk_SK.UTF-8
> LC_MESSAGES="en_US.UTF-8"
> LC_PAPER=sk_SK.UTF-8
> LC_NAME="en_US.UTF-8"
> LC_ADDRESS="en_US.UTF-8"
> LC_TELEPHONE="en_US.UTF-8"
> LC_MEASUREMENT=sk_SK.UTF-8
> LC_IDENTIFICATION="en_US.UTF-8"
> LC_ALL=
> 
> Thanks.
> 
> Regards,
> Tomas
> 
> On 15.05.2017 15:52, Tim Rühsen wrote:
>> Hi Tomas,
>>
>> thank you very much for the fixes and the new tests !
>>
>> Currently, patch 1/3 breaks two Metalink tests:
>>
>> FAIL: Test-metalink-http.py
>> FAIL: Test-metalink-http-quoted.py
>>
>> I am still investigating, but it has something to do with the
>> environment that you reset in testenv/test/base_test.py:
>>
>> ret_code = call(params, env={"HOME": os.getcwd()})
>>
>> Maybe a bug that just comes out with a different locale/charset encoding.
>>
>>
>> With Best Regards, Tim
>>
>>
>>
>> On 05/12/2017 07:17 PM, Tomas Hozza wrote:
>>> Getting credentials from .netrc has been broken from time to time, thus
>>> adding a test coverage to prevent regressions.
>>>
>>> Also added setting of "HOME" environment variable when executing wget,
>>> to make sure LocalFiles like .netrc, which are created just for the
>>> test, are actually used.
>>>
>>> Signed-off-by: Tomas Hozza 
>>> ---
>>>  testenv/Makefile.am |  3 ++
>>>  testenv/Test-auth-basic-netrc-pass-given.py | 68 
>>> +
>>>  testenv/Test-auth-basic-netrc-user-given.py | 68 
>>> +
>>>  testenv/Test-auth-basic-netrc.py| 66 
>>> 
>>>  testenv/test/base_test.py   |  2 +-
>>>  5 files changed, 206 insertions(+), 1 deletion(-)
>>>  create mode 100755 testenv/Test-auth-basic-netrc-pass-given.py
>>>  create mode 100755 testenv/Test-auth-basic-netrc-user-given.py
>>>  create mode 100755 testenv/Test-auth-basic-netrc.py
>>>
>>> diff --git a/testenv/Makefile.am b/testenv/Makefile.am
>>> index 3febec7..7104314 100644
>>> --- a/testenv/Makefile.am
>>> +++ b/testenv/Makefile.am
>>> @@ -75,6 +75,9 @@ if HAVE_PYTHON3
>>>TESTS = Test-504.py   \
>>>  Test-auth-basic-fail.py \
>>>  Test-auth-basic.py  \
>>> +Test-auth-basic-netrc.py\
>>> +Test-auth-basic-netrc-user-given.py \
>>> +Test-auth-basic-netrc-pass-given.py \
>>>  Test-auth-both.py   \
>>>  Test-auth-digest.py \
>>>  Test-auth-no-challenge.py   \
>>> diff --git a/testenv/Test-auth-basic-netrc-pass-given.py 
>>> b/testenv/Test-auth-basic-netrc-pass-given.py
>>> new file mode 100755
>>> index 000..43dfe34
>>> --- /dev/null
>>> +++ b/testenv/Test-auth-basic-netrc-pass-given.py
>>> @@ -0,0 +1,68 @@
>>> +#!/usr/bin/env python3
>>> +from sys import exit
>>> +from test.http_test import HTTPTest
>>> +from misc.wget_file import WgetFile
>>> +
>>> +"""
>>> +This test ensures Wget uses credentials from .netrc for Basic 
>>> Authorization Negotiation.
>>> +In this case we test that .netrc credentials are used in case only
>>> +password is given on the command line.
>>> +Also, we ensure that Wget saves the host after a successful auth and
>>> +doesn't wait for a challenge the second time.
>>> +"""
>>> +# File Definitions 
>>> ###
>>> +File1 = "I am an invisble man."
>>> +File2 = "I too am an invisible man."
>>> +
>>> +User = "Sauron"
>>> +Password = "TheEye"
>>> +
>>> +File1_rules = {
>>> +"Authentication": {
>>> +"Type"  : "Basic",
>>> +"User"  : User,
>>> +"Pass"  : Password
>>> +}
>>> +}
>>> +File2_rules = {
>>> +"ExpectHeader"  : {
>>> +"Authorization" : "Basic U2F1cm9uOlRoZUV5ZQ=="
>>> +}
>>> +}
>>> +
>>> +Netrc = "machine 127.0.0.1\n\tlogin {0}".format(User)
>>> +
>>> +A_File = WgetFile ("File1", File1, rules=File1_rules)
>>> +B_File = WgetFile ("File2", File2, rules=File2_rules)
>>> +Netrc_File = WgetFile (".netrc", Netrc)
>>> +
>>> +WGET_OPTIONS = "--password={0}".format(Password)
>>> +WGET_URLS = [["File1", "File2"]]
>>> +
>>> +Files = [[A_File, B_File]]
>>> +LocalFiles = [Netrc_File]
>>

Re: [Bug-wget] [PATCH 1/3] Added tests for HTTP authentication using credentials from .netrc

2017-05-16 Thread Tomas Hozza 
Hello Tim.

I'm pleased to hear that you figured it out before I had the chance to look at 
it thoroughly. Thanks for merging my changes.

Regards,
Tomas

On 16.05.2017 10:32, Tim Rühsen wrote:
> Hi Tomas,
> 
> fixed the problem by skipping the processing of 'pubring.kbx'. This file
> is created by libgpgme if we change $HOME.
> 
> Your changes have been pushed now. Thanks for your work.
> 
> The charset/locale issue is a general problem when the test suite reads
> and compares binary data. Currently, we don't have to fix that.
> 
> 
> 
> With Best Regards, Tim
> 
> 
> 
> On 05/15/2017 04:53 PM, Tomas Hozza wrote:
>> Hello Tim.
>>
>> I run all the tests before sending patches and they all passed. If you can 
>> send me more info about the failures, I could have a look at it.
>>
>> My locale is:
>> $ locale
>> LANG=en_US.UTF-8
>> LC_CTYPE="en_US.UTF-8"
>> LC_NUMERIC=sk_SK.UTF-8
>> LC_TIME=sk_SK.UTF-8
>> LC_COLLATE="en_US.UTF-8"
>> LC_MONETARY=sk_SK.UTF-8
>> LC_MESSAGES="en_US.UTF-8"
>> LC_PAPER=sk_SK.UTF-8
>> LC_NAME="en_US.UTF-8"
>> LC_ADDRESS="en_US.UTF-8"
>> LC_TELEPHONE="en_US.UTF-8"
>> LC_MEASUREMENT=sk_SK.UTF-8
>> LC_IDENTIFICATION="en_US.UTF-8"
>> LC_ALL=
>>
>> Thanks.
>>
>> Regards,
>> Tomas
>>
>> On 15.05.2017 15:52, Tim Rühsen wrote:
>>> Hi Tomas,
>>>
>>> thank you very much for the fixes and the new tests !
>>>
>>> Currently, patch 1/3 breaks two Metalink tests:
>>>
>>> FAIL: Test-metalink-http.py
>>> FAIL: Test-metalink-http-quoted.py
>>>
>>> I am still investigating, but it has something to do with the
>>> environment that you reset in testenv/test/base_test.py:
>>>
>>> ret_code = call(params, env={"HOME": os.getcwd()})
>>>
>>> Maybe a bug that just comes out with a different locale/charset encoding.
>>>
>>>
>>> With Best Regards, Tim
>>>
>>>
>>>
>>> On 05/12/2017 07:17 PM, Tomas Hozza wrote:
 Getting credentials from .netrc has been broken from time to time, thus
 adding a test coverage to prevent regressions.

 Also added setting of "HOME" environment variable when executing wget,
 to make sure LocalFiles like .netrc, which are created just for the
 test, are actually used.

 Signed-off-by: Tomas Hozza 
 ---
  testenv/Makefile.am |  3 ++
  testenv/Test-auth-basic-netrc-pass-given.py | 68 
 +
  testenv/Test-auth-basic-netrc-user-given.py | 68 
 +
  testenv/Test-auth-basic-netrc.py| 66 
 
  testenv/test/base_test.py   |  2 +-
  5 files changed, 206 insertions(+), 1 deletion(-)
  create mode 100755 testenv/Test-auth-basic-netrc-pass-given.py
  create mode 100755 testenv/Test-auth-basic-netrc-user-given.py
  create mode 100755 testenv/Test-auth-basic-netrc.py

 diff --git a/testenv/Makefile.am b/testenv/Makefile.am
 index 3febec7..7104314 100644
 --- a/testenv/Makefile.am
 +++ b/testenv/Makefile.am
 @@ -75,6 +75,9 @@ if HAVE_PYTHON3
TESTS = Test-504.py   \
  Test-auth-basic-fail.py \
  Test-auth-basic.py  \
 +Test-auth-basic-netrc.py\
 +Test-auth-basic-netrc-user-given.py \
 +Test-auth-basic-netrc-pass-given.py \
  Test-auth-both.py   \
  Test-auth-digest.py \
  Test-auth-no-challenge.py   \
 diff --git a/testenv/Test-auth-basic-netrc-pass-given.py 
 b/testenv/Test-auth-basic-netrc-pass-given.py
 new file mode 100755
 index 000..43dfe34
 --- /dev/null
 +++ b/testenv/Test-auth-basic-netrc-pass-given.py
 @@ -0,0 +1,68 @@
 +#!/usr/bin/env python3
 +from sys import exit
 +from test.http_test import HTTPTest
 +from misc.wget_file import WgetFile
 +
 +"""
 +This test ensures Wget uses credentials from .netrc for Basic 
 Authorization Negotiation.
 +In this case we test that .netrc credentials are used in case only
 +password is given on the command line.
 +Also, we ensure that Wget saves the host after a successful auth and
 +doesn't wait for a challenge the second time.
 +"""
 +# File Definitions 
 ###
 +File1 = "I am an invisble man."
 +File2 = "I too am an invisible man."
 +
 +User = "Sauron"
 +Password = "TheEye"
 +
 +File1_rules = {
 +"Authentication": {
 +"Type"  : "Basic",
 +"User"  : User,
 +"Pass"  : Password
 +}
 +}
 +File2_rules = {
 +"ExpectHeader"  : {
 +"Authorization" : "Basic U2F1cm9uOlRoZUV5ZQ=="
 +}
 +}
 +
 +Netrc = "machine 127.0.0.1\n\tlogin {0}".