Re: Preventing network access during nose doctest ?
Nicolas Dandrimont writes:
> * Olivier Berger [2014-05-12 14:36:17
> +0200]:
>
>> Hi.
>>
>> I'm trying to fix #739222 where tests fail (-> FTBFS) during execution
>> of nose's doctest plugin on something like :
>> >>> import rdflib
>>
>> >>> g = rdflib.Graph()
>> >>> result =
>> g.parse("http://www.w3.org/2000/10/swap/test/meet/white.rdf";)
>>
>> >>> print("graph has %s statements." % len(g))
>> graph has 19 statements.
>>
>> I'm puzzled : I'm invoking the tests run with :
>> PYBUILD_SYSTEM=custom \
>> PYBUILD_TEST_ARGS="{interpreter} run_tests.py" \
>> http_proxy= https_proxy= \
>> dh_auto_test --buildsystem=pybuild
>>
>> where run_tests.py will invoke nose with --with-doctest, but even though
>> the HTTP proxy variables being set, they don't seem to be preventing
>> urllib2 to try to access the file.
Thanks everyone for the hints.
>
> Hi,
>
> Here, you're setting an empty http_proxy variable, which means "don't use a
> proxy". What you really want is to set the proxy to something that errors out,
> e.g. http://127.0.0.1:9/ (the discard port on localhost).
>
> And then, you'll need to disable the tests that need internet access, or
> modify
> them so that they stop needing it.
>
Indeed I was overly confused : I had previously manually set them to
empty values to workaround the proxy setting made by pybuild internally,
so that the build would effectively access the network... and then I
forgot why I had done that... and of course I hadn't documented that ;)
Sorry about the bothering.
But, then there's still the option of maybe mocking the tests so that we
can proxy locally a copy of the document being fetched... but I'll let
that for another day ;)
Best regards,
--
Olivier BERGER
http://www-public.telecom-sudparis.eu/~berger_o/ - OpenPGP-Id: 2048R/5819D7E8
Ingenieur Recherche - Dept INF
Institut Mines-Telecom, Telecom SudParis, Evry (France)
--
To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/87ppjjqf8k@inf-8660.int-evry.fr
Re: Preventing network access during nose doctest ?
* Barry Warsaw , 2014-05-12, 09:41: Note that some test suites do legitimate http/https access during their tests. By "legitimate" I mean, they set up their own localhost special-port service and vend files out of them. AFAICS urllib honours the no_proxy environment variable. -- Jakub Wilk -- To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: https://lists.debian.org/20140512135255.ga...@jwilk.net
Re: Preventing network access during nose doctest ?
On May 12, 2014, at 02:57 PM, Nicolas Dandrimont wrote: >Here, you're setting an empty http_proxy variable, which means "don't use a >proxy". What you really want is to set the proxy to something that errors out, >e.g. http://127.0.0.1:9/ (the discard port on localhost). > >And then, you'll need to disable the tests that need internet access, or >modify them so that they stop needing it. Note that some test suites do legitimate http/https access during their tests. By "legitimate" I mean, they set up their own localhost special-port service and vend files out of them. You could call these integration tests, but still, test suites do this and setting the proxies to the discard port will break them. I agree that *external* access must be blocked, but cases such as the above must also be allowed. Unfortunately I don't know of a better way then unsetting the proxy envars for the test suite. -Barry signature.asc Description: PGP signature
Re: Preventing network access during nose doctest ?
* Olivier Berger [2014-05-12 14:36:17
+0200]:
> Hi.
>
> I'm trying to fix #739222 where tests fail (-> FTBFS) during execution
> of nose's doctest plugin on something like :
> >>> import rdflib
>
> >>> g = rdflib.Graph()
> >>> result = g.parse("http://www.w3.org/2000/10/swap/test/meet/white.rdf";)
>
> >>> print("graph has %s statements." % len(g))
> graph has 19 statements.
>
> I'm puzzled : I'm invoking the tests run with :
> PYBUILD_SYSTEM=custom \
> PYBUILD_TEST_ARGS="{interpreter} run_tests.py" \
> http_proxy= https_proxy= \
> dh_auto_test --buildsystem=pybuild
>
> where run_tests.py will invoke nose with --with-doctest, but even though
> the HTTP proxy variables being set, they don't seem to be preventing
> urllib2 to try to access the file.
Hi,
Here, you're setting an empty http_proxy variable, which means "don't use a
proxy". What you really want is to set the proxy to something that errors out,
e.g. http://127.0.0.1:9/ (the discard port on localhost).
And then, you'll need to disable the tests that need internet access, or modify
them so that they stop needing it.
HTH,
--
Nicolas Dandrimont
BOFH excuse #446:
Mailer-daemon is busy burning your message in hell.
signature.asc
Description: Digital signature
Re: Preventing network access during nose doctest ?
[Olivier Berger, 2014-05-12]
> I'm trying to fix #739222 where tests fail (-> FTBFS) during execution
> of nose's doctest plugin on something like :
> >>> import rdflib
>
> >>> g = rdflib.Graph()
> >>> result = g.parse("http://www.w3.org/2000/10/swap/test/meet/white.rdf";)
>
> >>> print("graph has %s statements." % len(g))
> graph has 19 statements.
>
> I'm puzzled : I'm invoking the tests run with :
> PYBUILD_SYSTEM=custom \
> PYBUILD_TEST_ARGS="{interpreter} run_tests.py" \
> http_proxy= https_proxy= \
> dh_auto_test --buildsystem=pybuild
>
> where run_tests.py will invoke nose with --with-doctest, but even though
> the HTTP proxy variables being set, they don't seem to be preventing
> urllib2 to try to access the file.
remove "http_proxy= https_proxy=" line and pybuild will try to block
internet connection (and you will be able to figure out which test
tries to access network and fix/disable it)
--
Piotr Ożarowski Debian GNU/Linux Developer
www.ozarowski.pl www.griffith.cc www.debian.org
GPG Fingerprint: 1D2F A898 58DA AF62 1786 2DF7 AEF6 F1A2 A745 7645
--
To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/20140512125355.ga4...@sts0.p1otr.com
Re: Preventing network access during nose doctest ?
On Mon, 12 May 2014, Olivier Berger wrote:
> Hi.
> I'm trying to fix #739222 where tests fail (-> FTBFS) during execution
> of nose's doctest plugin on something like :
> >>> import rdflib
> >>> g = rdflib.Graph()
> >>> result = g.parse("http://www.w3.org/2000/10/swap/test/meet/white.rdf";)
> >>> print("graph has %s statements." % len(g))
> graph has 19 statements.
> I'm puzzled : I'm invoking the tests run with :
> PYBUILD_SYSTEM=custom \
> PYBUILD_TEST_ARGS="{interpreter} run_tests.py" \
> http_proxy= https_proxy= \
> dh_auto_test --buildsystem=pybuild
> where run_tests.py will invoke nose with --with-doctest, but even though
> the HTTP proxy variables being set, they don't seem to be preventing
> urllib2 to try to access the file.
should work AFAICT:
$> python -c 'import rdflib; g = rdflib.Graph(); result =
g.parse("http://www.w3.org/2000/10/swap/test/meet/white.rdf";); print("graph has
%s statements." % len(g))'
graph has 19 statements.
$> http_proxy=http://127.0.0.1:9/ python -c 'import rdflib; g = rdflib.Graph();
result = g.parse("http://www.w3.org/2000/10/swap/test/meet/white.rdf";);
print("graph has %s statements." % len(g))'
Traceback (most recent call last):
File "", line 1, in
File "/usr/lib/python2.7/dist-packages/rdflib/Graph.py", line 886, in parse
source = self.prepare_input_source(source, publicID)
File "/usr/lib/python2.7/dist-packages/rdflib/Graph.py", line 692, in
prepare_input_source
input_source = URLInputSource(location)
File "/usr/lib/python2.7/dist-packages/rdflib/URLInputSource.py", line 21, in
__init__
file = urlopen(req)
File "/usr/lib/python2.7/urllib2.py", line 127, in urlopen
return _opener.open(url, data, timeout)
File "/usr/lib/python2.7/urllib2.py", line 404, in open
response = self._open(req, data)
File "/usr/lib/python2.7/urllib2.py", line 422, in _open
'_open', req)
File "/usr/lib/python2.7/urllib2.py", line 382, in _call_chain
result = func(*args)
File "/usr/lib/python2.7/urllib2.py", line 1214, in http_open
return self.do_open(httplib.HTTPConnection, req)
File "/usr/lib/python2.7/urllib2.py", line 1184, in do_open
raise URLError(err)
urllib2.URLError:
make sure you exported them in debian/rules:
export http_proxy=http://127.0.0.1:9/
--
Yaroslav O. Halchenko, Ph.D.
http://neuro.debian.net http://www.pymvpa.org http://www.fail2ban.org
Research Scientist,Psychological and Brain Sciences Dept.
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834 Fax: +1 (603) 646-1419
WWW: http://www.linkedin.com/in/yarik
--
To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/20140512124702.gh8...@onerussian.com
