[issue28022] SSL releated deprecation for 3.6
Nick Guenther added the comment:
Hello everyone, and thank you as usual for all your hard work keeping the
python ecosystem going.
I saw that the start of this thread said it was going to
> - make ftplib, imaplib, nntplib, pop3lib, smtplib etc. validate certs by
> default.
but this hasn't been done, at least not for imaplib. imaplib is still calling
the undocumented "ssl._create_stdlib_context":
https://github.com/python/cpython/blob/2b496e79293a8b80e8ba0e514e186b3b1467b64b/Lib/imaplib.py#L1320
which is actually "ssl._create_unverified_context":
https://github.com/python/cpython/blob/2b496e79293a8b80e8ba0e514e186b3b1467b64b/Lib/ssl.py#L842
which is indeed unverified: despite defaulting to PROTOCOL_TLS_CLIENT, which
"enables CERT_REQUIRED and check_hostname by default.", it overrides that by
setting check_hostname=False:
https://github.com/python/cpython/blob/2b496e79293a8b80e8ba0e514e186b3b1467b64b/Lib/ssl.py#L811
To demonstrate, check out this tester script:
$ cat a.py
import os, imaplib
with imaplib.IMAP4_SSL(os.environ.get('HOSTNAME')) as S:
print(S.login(os.environ.get('USERNAME'), os.environ.get('PASSWORD')))
$ HOSTNAME=46.23.90.174 USERNAME=test1 PASSWORD=test1test1 python3 a.py
('OK', [b'Logged in'])
I don't have a cert for 46.23.90.174 (no one will give out certs for IPs!), so
this is wrong!
In order to actually enable verification you need to know the incantation. It's
not that long but it is subtle and frighteningly easy to get wrong. Here it is:
$ cat a.py
import os, ssl, imaplib
ctx = ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT)
ctx.load_default_certs()
with imaplib.IMAP4_SSL(os.environ.get('HOSTNAME'), ssl_context=ctx) as S:
print(S.login(os.environ.get('USERNAME'), os.environ.get('PASSWORD')))
$ HOSTNAME=46.23.90.174 USERNAME=test1 PASSWORD=test1test1 python3 a.py
Traceback (most recent call last):
File "a.py", line 6, in
with imaplib.IMAP4_SSL(os.environ.get('HOSTNAME'), ssl_context=ctx) as S:
File "/usr/lib/python3.6/imaplib.py", line 1288, in __init__
IMAP4.__init__(self, host, port)
File "/usr/lib/python3.6/imaplib.py", line 198, in __init__
self.open(host, port)
File "/usr/lib/python3.6/imaplib.py", line 1301, in open
IMAP4.open(self, host, port)
File "/usr/lib/python3.6/imaplib.py", line 299, in open
self.sock = self._create_socket()
File "/usr/lib/python3.6/imaplib.py", line 1293, in _create_socket
server_hostname=self.host)
File "/usr/lib/python3.6/ssl.py", line 407, in wrap_socket
_context=self, _session=session)
File "/usr/lib/python3.6/ssl.py", line 817, in __init__
self.do_handshake()
File "/usr/lib/python3.6/ssl.py", line 1077, in do_handshake
self._sslobj.do_handshake()
File "/usr/lib/python3.6/ssl.py", line 694, in do_handshake
match_hostname(self.getpeercert(), self.server_hostname)
File "/usr/lib/python3.6/ssl.py", line 327, in match_hostname
% (hostname, ', '.join(map(repr, dnsnames
ssl.CertificateError: hostname '46.23.90.174' doesn't match either of
'comms.kousu.ca', 'comms3.kousu.ca'
I can see from this thread there were some concerns about breaking people's
self-signed certs back in 2016. But it's five years later now and letsencrypt
is super common now, and most servers and clients are enforcing TLS, especially
when credentials are involved.
Could this be revisited?
Thanks for any attention you have gifted to this :)
--
nosy: +kousu
___
Python tracker
<https://bugs.python.org/issue28022>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue40110] multiprocessing.Pool.imap() should be lazy
Nick Guenther added the comment:
Thank you for taking the time to consider my points! Yes, I think you
understood exactly what I was getting at.
I slept on it and thought about what I'd posted the day after and realized most
of the points you raise, especially that serialized next() would mean
serialized processing. So the naive approach is out.
I am motivated by trying to introduce backpressure to my pipelines. The example
you gave has potentially infinite memory usage; if I simply slow it down with
sleep() I get a memory leak and the main python proc pinning my CPU, even
though it "isn't" doing anything:
with multiprocessing.Pool(4) as pool:
for i, v in enumerate(pool.imap(worker, itertools.count(1)), 1):
time.sleep(7)
print(f"At {i}: {v}, memory usage is {sys.getallocatedblocks()}")
At 1->1, memory usage is 230617
At 2->8, memory usage is 411053
At 3->27, memory usage is 581439
At 4->64, memory usage is 748584
At 5->125, memory usage is 909694
At 6->216, memory usage is 1074304
At 7->343, memory usage is 1238106
At 8->512, memory usage is 1389162
At 9->729, memory usage is 1537830
At 10->1000, memory usage is 1648502
At 11->1331, memory usage is 1759864
At 12->1728, memory usage is 1909807
At 13->2197, memory usage is 2005700
At 14->2744, memory usage is 2067407
At 15->3375, memory usage is 2156479
At 16->4096, memory usage is 2240936
At 17->4913, memory usage is 2328123
At 18->5832, memory usage is 2456865
At 19->6859, memory usage is 2614602
At 20->8000, memory usage is 2803736
At 21->9261, memory usage is 2999129
PID USER PR NIVIRTRESSHR S %CPU %MEM TIME+ COMMAND
11314 kousu 20 0 303308 40996 6340 S 91.4 2.1 0:21.68 python3.8
11317 kousu 20 0 54208 10264 4352 S 16.2 0.5 0:03.76 python3.8
11315 kousu 20 0 54208 10260 4352 S 15.8 0.5 0:03.74 python3.8
11316 kousu 20 0 54208 10260 4352 S 15.8 0.5 0:03.74 python3.8
11318 kousu 20 0 54208 10264 4352 S 15.5 0.5 0:03.72 python3.8
It seems to me like any usage of Pool.imap() either has the same issue lurking
or is run on a small finite data set where you are better off using Pool.map().
I like generators because they keep constant-time constant-memory work, which
seems like a natural fit for stream processing situations. Unix pipelines have
backpressure built-in, because write() blocks when the pipe buffer is full.
I did come up with one possibility after sleeping on it again: run the final
iteration in parallel, perhaps by a special plist() method that makes as many
parallel next() calls as possible. There's definitely details to work out but I
plan to prototype when I have spare time in the next couple weeks.
You're entirely right that it's a risky change to suggest, so maybe it would be
best expressed as a package if I get it working. Can I keep this issue open to
see if it draws in insights from anyone else in the meantime?
Thanks again for taking a look! That's really cool of you!
--
___
Python tracker
<https://bugs.python.org/issue40110>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue40110] multiprocessing.Pool.imap() should be lazy
New submission from Nick Guenther : multiprocessing.Pool.imap() is supposed to be a lazy version of map. But it's not: it submits work to its workers eagerly. As a consequence, in a pipeline, all the work from earlier steps is queued, performed, and finished first, before starting later steps. If you use python3's built-in map() -- aka the old itertools.imap() -- the operations are on-demand, so it surprised me that Pool.imap() doesn't. It's basically no better than using Pool.map(). Maybe it saves memory by not materializing large iterables in every worker process? But it still materializes the CPU time from the iterables even if unneeded. This can be partially worked around by giving each step of the pipeline its own Pool -- then, at least the earlier steps of the pipeline don't block the later steps -- but the jobs are still done eagerly instead of waiting for their results to actually be requested. -- files: multiprocessing-eager-imap.py messages: 365295 nosy: kousu priority: normal severity: normal status: open title: multiprocessing.Pool.imap() should be lazy versions: Python 2.7, Python 3.5, Python 3.6, Python 3.7, Python 3.8 Added file: https://bugs.python.org/file49010/multiprocessing-eager-imap.py ___ Python tracker <https://bugs.python.org/issue40110> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10976] json.loads() raises TypeError on bytes object
Changes by Nick Guenther n...@kousu.ca: -- nosy: +kousu ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10976 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19282] dbm is not a context manager
New submission from Nick Guenther: This code doesn't work. I think it should. import dbm with dbm.open(what is box.db, c) as db: db[Bpoind] = Boing Indeed, there is nothing supporting PEP 343 for dbm on my system: [kousu@galleon ~]$ grep -r __exit__ /usr/lib/python3.3/dbm [kousu@galleon ~]$ -- components: Library (Lib) messages: 200191 nosy: kousu priority: normal severity: normal status: open title: dbm is not a context manager versions: Python 3.1, Python 3.2, Python 3.3, Python 3.4, Python 3.5 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19282 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19248] sphinx is not py3k compatible
New submission from Nick Guenther: I'm running Arch. I just checked out python's hg and tried to build the docs, and found that I couldn't: $ cd cpython/doc $ make update [lots of output stripped] $ make pydoc-topics mkdir -p build/pydoc-topics build/doctrees python tools/sphinx-build.py -b pydoc-topics -d build/doctrees -D latex_paper_size= . build/pydoc-topics Traceback (most recent call last): File tools/sphinx-build.py, line 27, in module from sphinx import main File /home/kousu/pro/cpython/Doc/tools/sphinx/__init__.py, line 44 except ImportError, err: ^ SyntaxError: invalid syntax make: *** [build] Error 1 I edited the Makefile to say PYTHON = python2 and then it worked. Is the assumption that python2 is still supposed to be the default python everywhere? Is python 3 not able to build itself yet? [kousu@galleon Doc]$ pacman -Qi python Name : python Version: 3.3.2-2 Description: Next generation of the python high-level scripting language Architecture : x86_64 URL: http://www.python.org/ Licenses : custom Groups : None Provides : python3 Depends On : expat bzip2 gdbm openssl libffi zlib Optional Deps : tk: for tkinter [installed] sqlite [installed] Required By: cython ipython ktoblzcheck libreoffice-common python-cairo python-dateutil python-dbus python-markupsafe python-numpy python-pyparsing python-pytz python-pyzmq python-setuptools python-sip python-six python-sympy python-tornado python-xdg ranger youtube-dl Optional For : systemd Conflicts With : None Replaces : python3 Installed Size : 86980.00 KiB Packager : Bart Build Date : Fri Sep 6 03:31:57 2013 Install Date : Tue Sep 17 10:29:36 2013 Install Reason : Installed as a dependency for another package Install Script : No Validated By : Signature -- assignee: docs@python components: Documentation messages: 199764 nosy: docs@python, kousu priority: normal severity: normal status: open title: sphinx is not py3k compatible versions: Python 3.1, Python 3.2, Python 3.3, Python 3.4, Python 3.5 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19248 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19248] sphinx is not py3k compatible
Nick Guenther added the comment: I see that in Doc/tools/sphinx-build.py there is this check: if sys.version_info[:3] (2, 4, 0): sys.stderr.write(\ Error: Sphinx needs to be executed with Python 2.4 or newer (not 3.0 though). (If you run this from the Makefile, you can set the PYTHON variable to the path of an alternative interpreter executable, e.g., ``make html PYTHON=python2.5``). ) But I never saw this and had to debug it myself. I submit the attached patch as a solution. -- keywords: +patch resolution: duplicate - status: closed - open type: enhancement - Added file: http://bugs.python.org/file32093/sphinx_version_check.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19248 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19250] Duplicate import documentation in py3k
New submission from Nick Guenther: Python3's docs given by help(import) duplicate these two paragraphs: The *public names* defined by a module are determined by checking the module's namespace for a variable named ``__all__``; if defined, it must be a sequence of strings which are names defined or imported by that module. The names given in ``__all__`` are all considered public and are required to exist. If ``__all__`` is not defined, the set of public names includes all names found in the module's namespace which do not begin with an underscore character (``'_'``). ``__all__`` should contain the entire public API. It is intended to avoid accidentally exporting items that are not part of the API (such as library modules which were imported and used within the module). The ``from`` form with ``*`` may only occur in a module scope. Attempting to use it in class or function definitions will raise a ``SyntaxError``. (full output attached for proof) Digging through the sources, I can't find where this is defined. The python2 on my system has the same two paragraphs but they are *not* duplicated there, and in the sources, which admittedly I don't know my way around, Doc/reference/simple_stmts.rst contains these paragraphs but only once each, and grep seems to imply this is the source of them. Did sphinx screw up somewhere between your hg and my distro mirror? Here's my version info (arch linux, 64 bit) [kousu@galleon ~]$ pacman -Qi python Name : python Version: 3.3.2-2 Description: Next generation of the python high-level scripting language Architecture : x86_64 URL: http://www.python.org/ Licenses : custom Groups : None Provides : python3 Depends On : expat bzip2 gdbm openssl libffi zlib Optional Deps : tk: for tkinter [installed] sqlite [installed] Required By: cython ipython ktoblzcheck libreoffice-common python-cairo python-dateutil python-dbus python-markupsafe python-numpy python-pyparsing python-pytz python-pyzmq python-setuptools python-sip python-six python-sympy python-tornado python-xdg ranger youtube-dl Optional For : systemd Conflicts With : None Replaces : python3 Installed Size : 86980.00 KiB Packager : Bart Build Date : Fri Sep 6 03:31:57 2013 Install Date : Tue Sep 17 10:29:36 2013 Install Reason : Installed as a dependency for another package Install Script : No Validated By : Signature -- assignee: docs@python components: Documentation files: The ``import`` statement - (2013-10-13 15:36:07) messages: 199775 nosy: docs@python, kousu priority: normal severity: normal status: open title: Duplicate import documentation in py3k versions: Python 3.3 Added file: http://bugs.python.org/file32096/The ``import`` statement - (2013-10-13 15:36:07) ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19250 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19250] Duplicate import documentation in py3k
Nick Guenther added the comment: Thank you. Sorry for the duplicates. I tried to search the bug tracker but nothing stood out to me. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19250 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2529] list/generator comprehension parser doesn't match spec
Nick Guenther [EMAIL PROTECTED] added the comment: Oh, okay. That's really confusing because I expect in to always return a bool, but in the spirit of python there's no reason it should I guess. __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue2529 __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2529] list/generator comprehension parser doesn't match spec
New submission from Nick Guenther [EMAIL PROTECTED]: I think I've found a bug in python's list comprehension parser. Observe: [e for i in j in ['a','b','c']] Traceback (most recent call last): File stdin, line 1, in module NameError: name 'j' is not defined Now, according to the grammar at http://docs.python.org/ref/lists.html, a list comprehension is (condensed for clarity): list_comprehension ::= expression list_for list_for::= for target_list in old_expression_list [list_for] So a list comprehension should always be [ for ... in for ... in ... for ... in ...] (that is, alternating 'for's and 'in's) but here I have a test case that python happily tries to run that looks like [... for ... in ... in ] -- components: Interpreter Core messages: 64818 nosy: kousu severity: normal status: open title: list/generator comprehension parser doesn't match spec type: behavior versions: Python 2.5 __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue2529 __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
