[issue29539] [smtplib] collect response data for all recipients
David Ford (FirefighterBlu3) added the comment: i have a fully built patch and personally tested (i use it 24/7) but haven't done test_* yet as was requested On Thu, Feb 28, 2019 at 9:16 PM Windson Yang wrote: > > Windson Yang added the comment: > > sls, are you working on this feature now? > > -- > nosy: +Windson Yang > > ___ > Python tracker > <https://bugs.python.org/issue29539> > ___ > -- ___ Python tracker <https://bugs.python.org/issue29539> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue29539] [smtplib] collect response data for all recipients
David Ford (FirefighterBlu3) added the comment: Yes, it is distinctly intended to collect the results for every recipient as in modern MTAs, there's more than just success/fail results. This is to collect data such as queue ID and the MTA's programmatic response for each individual recipient. I have several needs of knowing if the message was immediately relayed, queued for later because of a remote issue, queued because of MTA issue, graylisted or blocked because of milter reasons, or ... any of a list of failure reasons. This data can be collected if there is only one recipient per message, but that is considerably resource expensive. Without this patch, when sending to say 100 recipients, the only response returned to the caller is the very last (100th) result. A 250 then assumes that all 100 were transmitted successfully when in truth, the first 99 could have failed. Yes, I'll make a PR, do the CLA, and add some tests. -- ___ Python tracker <https://bugs.python.org/issue29539> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue29539] [smtplib] collect response data for all recipients
David Ford (FirefighterBlu3) added the comment: 3.7 release candidates are in the queue, any thoughts on this simple enhancement? -- ___ Python tracker <https://bugs.python.org/issue29539> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue29470] ssl: SNI callbacks should not modify context objects
David Ford (FirefighterBlu3) added the comment: yes, after i've modified my tool and ensured i know the correct way of doing it. one of the problems i'm dealing with is not carting around a global variable holding prebuilt contexts. i find it unappealing. -- ___ Python tracker <http://bugs.python.org/issue29470> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue29470] ssl: SNI callbacks should not modify context objects
David Ford (FirefighterBlu3) added the comment: @Christian, that was years ago and there were few examples available of people that had tried to make an SNI capable server. in several cases, people were monkey patching to make a callback. .set_servername_callback() didn't formally show up in ssl mod until 3.4. i disagree that the documentation is very clear about this. there's one direct reference to usage which is worded as "A typical use of this callback is to...". that doesn't imply 'very clear' :} i cannot recall anything anywhere that says one must not modify the context after it is used. 18.2.3 implies that a context is a long lived object, longer lived than a socket. it's clear that several people including myself, have mistakenly tried modifying the context after it was built. until now, it worked, probably purely by coincidence. as a matter of fact, i used to change more than just the key loaded. i changed the ciphers, options, and verify_flags when i knew the incoming IP was incapable of supporting modern ciphers and options. i also modified the verify paths to support a messed up corporate environment. i suggest the 18.2.3 documentation be modified to be clear about: a) if, when, and how a context can be modified after creation b) an SNI example that changes contexts and explicitly indicates (a) i don't see any value to not being clear about it, and it's disingenuous to leave nooses hiding in shadows for devs to hang themselves with :-] -- ___ Python tracker <http://bugs.python.org/issue29470> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue29470] [python] Error in `/usr/bin/python': free(): invalid size: 0x00007f628400d0e0; ssl SNI callbacks should not modify context objects
Changes by David Ford (FirefighterBlu3) : -- title: [python] Error in `/usr/bin/python': free(): invalid size: 0x7f628400d0e0 -> [python] Error in `/usr/bin/python': free(): invalid size: 0x7f628400d0e0; ssl SNI callbacks should not modify context objects ___ Python tracker <http://bugs.python.org/issue29470> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue29470] [python] Error in `/usr/bin/python': free(): invalid size: 0x00007f628400d0e0
David Ford (FirefighterBlu3) added the comment: yes, i create a context and then when a request comes in, i load the appropriate key. this is how most [all?] SNI capable python server examples show how to do it, or did, at the time I wrote this module a few years ago. from that time, ~py 3.2 until now, it worked perfectly. thank you for this information, i'll rewrite my callback accordingly. perhaps 18.2.5.3 in https://docs.python.org/3/library/ssl.html can be updated to inform developers how to change their socket.context in an SNI callback -- resolution: -> not a bug ___ Python tracker <http://bugs.python.org/issue29470> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue29564] ResourceWarning tracking is attaching incorrect file:position to unclosed socket
David Ford (FirefighterBlu3) added the comment: @stinner your traceback suggestion was highly useful and much appreciated. i have spent several dozen hours researching ResourceWarnings in the past and tearing my hair out because the emitted warning was very much useless. i couldn't even make educated guesses about the code as socket pairs were used randomly. i strongly approve of mentioning the tracemalloc / traceback in the warning, and i would also very much encourage saying "this filename:position could be completely wrong" or do away with it all together. -- ___ Python tracker <http://bugs.python.org/issue29564> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue29564] ResourceWarning tracking is attaching incorrect file:position to unclosed socket
David Ford (FirefighterBlu3) added the comment: i quite understand the way objects can be associated with reference counts by using a, a.b, c=a.b, d=a, etc etc, and how they increase/decrease and when the GC can prune the objects. yes, the root issue is the unclosed web socket and i've already filed a bug with Twilio. @Martin, /usr/lib/python3.6/site-packages/psycopg2/extras.py:314: ResourceWarning: unclosed /usr/lib/python3.6/site-packages/psycopg2/extras.py:314 is unrelated to the resource in question. there is nothing in psycopg2 that can have any direct or indirect reference to the tcp/443 socket used by the twilio by any path in the code all the way back to glibc under python. nothing in psycopg2 and twilio down to _ssl/glibc have any associated references. the reference link between the two is at best as vague as connecting the water in the garden hose to the condensation on the hot shower mirror. the pg socket is created and used which will fetch a result that has no reference to the socket used by that query method. regardless, the socket is a completely different IP/port pair. there is nothing in the pool, connection, cursor, or query that will ever be used by twilio. psycopg2 has nothing to do with any of the http or httplib2 modules. after the pgsql query is completed, the twilio module gets used. a new https socket is created. the only data every seen by the twilio module that has a reference to any objected created by the psycopg2 module is the r.delivery_id which is a UUID value derived from the pgsql database. it has no references to its parent chain of objects and the twilio module doesn't derive any objects from it. to be more obtuse about it, if i replace the r.delivery_id with a quoted string i manually insert, the ResourceWarning is now logged against asyncio/events.py. the traceback is still the same. /usr/lib/python3.6/asyncio/events.py:126: ResourceWarning: unclosed self._callback(*self._args) this one happens to exist in the stack trace albeit even further up the call chain. if the ResourceWarning is unable to emit an accurate file:pos, then it shouldn't print it at all and instead, use the stack print as mentioned by @STINNER. at least at this point, the developer has a legitimate stack to wander through and isn't going on a wild goose chase in an unrelated file. i argue that printing a completely unrelated file:pos is definitely a bug. if it is likely to emit a wholly unrelated file:pos, it shouldn't be printed at all. unreliable information is just useless. -- ___ Python tracker <http://bugs.python.org/issue29564> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue29564] ResourceWarning tracking is attaching incorrect file:position to unclosed socket
David Ford (FirefighterBlu3) added the comment:
i do know what's causing it, it's the twilio module not cleanly finishing up. i
pointed out my call to this in the initial description.
/usr/lib/python3.6/site-packages/httplib2/__init__.py:857: DeprecationWarning:
key_file, cert_file and check_hostname are deprecated, use a custom context
instead.
check_hostname=disable_ssl_certificate_validation ^ True)
/usr/lib/python3.6/site-packages/psycopg2/extras.py:314: ResourceWarning:
unclosed
ts = super(NamedTupleCursor, self).fetchall()
Object allocated at (most recent call first):
File "/usr/lib/python3.6/ssl.py", lineno 401
_context=self, _session=session)
File "/usr/lib/python3.6/http/client.py", lineno 1400
server_hostname=server_hostname)
File "/usr/lib/python3.6/site-packages/httplib2/__init__.py", lineno 994
conn.connect()
File "/usr/lib/python3.6/site-packages/httplib2/__init__.py", lineno 1071
(response, content) = self._conn_request(conn, request_uri, method, body,
headers)
File "/usr/lib/python3.6/site-packages/httplib2/__init__.py", lineno 1321
(response, content) = self._request(conn, authority, uri, request_uri,
method, body, headers, redirections, cachekey)
File "/usr/lib/python3.6/site-packages/twilio/rest/resources/base.py", lineno
117
resp, content = http.request(url, method, headers=headers, body=data)
File "/usr/lib/python3.6/site-packages/twilio/rest/resources/base.py", lineno
152
resp = make_request(method, uri, **kwargs)
File "/usr/lib/python3.6/site-packages/twilio/rest/resources/base.py", lineno
200
resp = make_twilio_request(method, uri, auth=self.auth, **kwargs)
File "/usr/lib/python3.6/site-packages/twilio/rest/resources/base.py", lineno
333
resp, item = self.request("GET", uri)
File "/usr/lib/python3.6/site-packages/twilio/rest/resources/base.py", lineno
328
return self.get_instance(sid)
File "provider.py", lineno 524
msg = self._twilio.messages.get(r.delivery_id)
File "/usr/lib/python3.6/asyncio/events.py", lineno 126
self._callback(*self._args)
File "/usr/lib/python3.6/asyncio/base_events.py", lineno 1425
handle._run()
File "/usr/lib/python3.6/asyncio/base_events.py", lineno 421
self._run_once()
File "/usr/lib/python3.6/site-packages/autobahn/asyncio/wamp.py", lineno 168
loop.run_forever()
File "provider.py", lineno 595
runner.run(make)
the socket is referenced by ip:port correctly and is coherent with the stack
trace. nothing about the IP, socket, or stack trace, is related to the
referenced file:position that is reported:
/usr/lib/python3.6/site-packages/psycopg2/extras.py:314: ResourceWarning:
unclosed
ts = super(NamedTupleCursor, self).fetchall()
the psycopg2 connection is against a completely different IP and on the
standard pgsql port of 5432.
here's the snippet of psycopg2/extras.py:
313 def fetchall(self):
314 ts = super(NamedTupleCursor, self).fetchall()
315 nt = self.Record
316 if nt is None:
317 nt = self.Record = self._make_nt()
318 return list(map(nt._make, ts))
nothing at all between my pgsql database and twilio share any form of
connection or socket and there are no similarly named variables, complex code
paths, or global variables. the only object data which has any shared scope is
the twilio message fetch using the ID fetched from the pgsql database. if this
section of code is commented out, the resource warning vanishes.
msg = self._twilio.messages.get(r.delivery_id)
if i patch the involved calls in twilio so i can touch the socket object and
close it such as:
class Response(object):
"""
Take a httplib2 response and turn it into a requests response
"""
def __init__(self, httplib_resp, content, url, http_obj):
self.content = content
self.cached = False
self.status_code = int(httplib_resp.status)
self.ok = self.status_code < 400
self.url = url
self.connections = http_obj.connections
(http_obj added which is an instance of httplib2.Http which has a property of
.connections wherein sockets are cached)
then after modifying the twilio module to pass the Response object to me as
well as the message resource, i can eventually in my code:
for r in rows:
resp, msg = self._twilio.messages.get(r.delivery_id)
errors[r.delivery_id] = r.recipient, msg.error_code,
msg.error_message
for k in resp.connections:
resp.connections[k].close()
this closes all the cached sockets and ensures the ResourceWarning is never
emitted. this messiness is another conversation however, the incorrect file:pos
annotation for the Re
[issue29470] [python] Error in `/usr/bin/python': free(): invalid size: 0x00007f628400d0e0
David Ford (FirefighterBlu3) added the comment: Arch Linux (updated ~weekly but not often rebooted) on a Digital Ocean droplet. Kernel 4.5.1-1-ARCH, x86_64, OpenSSL 1.0.2j, Python 3.6.0 -- ___ Python tracker <http://bugs.python.org/issue29470> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue29470] [python] Error in `/usr/bin/python': free(): invalid size: 0x00007f628400d0e0
David Ford (FirefighterBlu3) added the comment: my SNI callback is a function that chooses which SSL certificate to load and summarized is: 6 def sni_callback(sock, servername, context): [...] 44 # load a specific crt/key 45 store = crtstore[servername] 46 47 context.load_cert_chain( certfile = store['certfile'], 48 keyfile = store['keyfile'], 49 password = store['password'] ) -- ___ Python tracker <http://bugs.python.org/issue29470> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue29470] [python] Error in `/usr/bin/python': free(): invalid size: 0x00007f628400d0e0
David Ford (FirefighterBlu3) added the comment: in process of creating a non-stripped python install. the default Arch Linux PKGBUILD uses xvfb-run at the end of the build process which is crashing. -- ___ Python tracker <http://bugs.python.org/issue29470> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue29564] ResourceWarning tracking is attaching incorrect file:position to unclosed socket
New submission from David Ford (FirefighterBlu3): the following warning is incorrectly reported as initiated at psycopg2/extras.py:314, it actually occurs inside the Twilio python module. /usr/lib/python3.6/site-packages/psycopg2/extras.py:314: ResourceWarning: unclosed ts = super(NamedTupleCursor, self).fetchall() namedtuple_Record:19: ResourceWarning: unclosed This is the snippet of my code which makes the Twilio API call over https (noted by the >>> line). # get failed/undelivered with (yield from self.aiopg_pool.cursor(cursor_factory=psycopg2.extras.NamedTupleCursor)) as c: yield from c.execute(stmt_bad, (id,)) rows = yield from c.fetchall() # make calls to twilio and fetch failure reasons for r in rows: >>> msg = self._twilio.messages.get(r.delivery_id) errors[r.delivery_id] = r.recipient, msg.error_code, msg.error_message my tool uses crossbar.io's python-autobahn asyncio classes which do not use threading to my knowledge. psycopg2/pool.py does use threading. python-twilio does not use threading either. -- messages: 287810 nosy: David Ford (FirefighterBlu3) priority: normal severity: normal status: open title: ResourceWarning tracking is attaching incorrect file:position to unclosed socket type: resource usage versions: Python 3.6 ___ Python tracker <http://bugs.python.org/issue29564> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue29470] [python] Error in `/usr/bin/python': free(): invalid size: 0x00007f628400d0e0
David Ford (FirefighterBlu3) added the comment: 2017-02-14 06:58:29 INFO connected: mustang.blue-labs.org/107.170.82.162:37188 2017-02-14 06:58:29 INFO No cert required for this host *** Error in `/usr/bin/python': double free or corruption (out): 0x7fb0f8003400 *** 2017-02-14 06:58:29 WARNING SSL client handshake has failed: EOF occurred in violation of protocol (_ssl.c:749) Fatal Python error: Aborted Thread 0x7fb0e700 (most recent call first): File "/var/bluelabs/python/_cams_threaded_server.py", line 116 in handle File "/usr/lib/python3.6/socketserver.py", line 696 in __init__ File "/usr/lib/python3.6/socketserver.py", line 361 in finish_request File "/usr/lib/python3.6/socketserver.py", line 639 in process_request_thread File "/usr/lib/python3.6/threading.py", line 864 in run File "/usr/lib/python3.6/threading.py", line 916 in _bootstrap_inner File "/usr/lib/pytho2017-02-14 06:58:29 INFO goodbye3 107.170.82.162 2017-02-14 06:58:29 WARNING SSL client handshake has failed: EOF occurred in violation of protocol (_ssl.c:749) 2017-02-14 06:58:29 WARNING SSL client handshake has failed: EOF occurred in violation of protocol (_ssl.c:749) 2017-02-14 06:58:29 WARNING SSL client handshake has failed: EOF occurred in violation of protocol (_ssl.c:749) n3.6/threading.py", line 884 in _bootstrap Thread 0x7fb100de2700 (most recent call first): File "/var/bluelabs/python/_cams_thread2017-02-14 06:58:29 INFO goodbye3 107.170.82.162 2017-02-14 06:58:29 INFO disconnect: Thread-6 2017-02-14 06:58:29 INFO goodbye3 107.170.82.162 2017-02-14 06:58:29 INFO disconnect: Thread-5 ed_server.py", line 116 in handle File "/usr/lib/python3.6/socketserver.py", line 696 in __init__ File "2017-02-14 06:58:29 INFO goodbye3 107.170.82.162 2017-02-14 06:58:29 INFO disconnect: Thread-10 /usr/lib/python3.6/socketserver.py", line 361 in 2017-02-14 06:58:29 INFO disconnect: Thread-11 finish_request File "/usr/lib/python3.6/socketserver.py", line 639 in process_request_thread File "/usr/lib/python3.6/threading.py", line 864 in run File "/usr/lib/python3.6/threading.py", line 916 in _bootstrap_inner File "/usr/lib/python3.6/threading.py", line 884 in _bootstrap Thread 0x7fb1025e5700 (most recent call first): File "/var/bluelabs/python/_cams_threaded_server.py", line 137 in handle File "/usr/lib/python3.6/socketserver.py", line 696 in __init__ File "/usr/lib/python3.6/socketserver.py", line 361 in finish_request File "/usr/lib/python3.6/socketserver.py", line 639 in process_request_thread File "/usr/lib/python3.6/threading.py", line 864 in run File "/usr/lib/python3.6/threading.py", line 916 in _bootstrap_inner File "/usr/lib/python3.6/threading.py", line 884 in _bootstrap Thread 0x7fb102de6700 (most recent call first): File "/var/bluelabs/python/_cams_threaded_server.py", line 137 in handle File "/usr/lib/python3.6/socketserver.py", line 696 in __init__ File "/usr/lib/python3.6/socketserver.py", line 361 in finish_request File "/usr/lib/python3.6/socketserver.py", line 639 in process_request_thread File "/usr/lib/python3.6/threading.py", line 864 in run File "/usr/lib/python3.6/threading.py", line 916 in _bootstrap_inner File "/usr/lib/python3.6/threading.py", line 884 in _bootstrap Current thread 0x7fb1035e7700 (most recent call first): File "/var/bluelabs/python/_cams_threaded_server.py", line 49 in sni_callback File "/usr/lib/python3.6/ssl.py", line 683 in do_handshake File "/usr/lib/python3.6/ssl.py", line 1061 in do_handshake File "/var/bluelabs/python/_cams_threaded_server.py", line 152 in handle File "/usr/lib/python3.6/socketserver.py", line 696 in __init__ File "/usr/lib/python3.6/socketserver.py", line 361 in finish_request File "/usr/lib/python3.6/socketserver.py", line 639 in process_request_thread File "/usr/lib/python3.6/threading.py", line 864 in run File "/usr/lib/python3.6/threading.py", line 916 in _bootstrap_inner File "/usr/lib/python3.6/threading.py", line 884 in _bootstrap Thread 0x7fb11131a700 (most recent call first): File "/usr/lib/python3.6/threading.py", line 299 in wait File "/usr/lib/python3.6/threading.py", line 551 in wait File "/var/bluelabs/bots/cams/camsbot.py", line 753 in start_message_queue_handler File "/usr/lib/python3.6/threading.py", line 864 in run File "/usr/lib/python3.6/threading.py", line 916 in _bootstrap_inner File "/usr/lib/python3.6/threading.py", line 884 in _bootstrap Thread 0x7fb111b1b700 (most recent call
[issue29539] [smtplib] collect response data for all recipients
New submission from David Ford (FirefighterBlu3): Feature request; collect SMTP response results for all recipients so data likely including the queue ID or SMTP delay expectation can be collected I propose the keyword "keep_results=False" be added to smtplib.sendmail() to accomplish this. The default value of False maintains the legacy functionality of prior versions. No other changes have been done to smtplib.send_message() pending discussion of the following. @@ -785,7 +785,7 @@ return (resp, reply) def sendmail(self, from_addr, to_addrs, msg, mail_options=[], - rcpt_options=[]): + rcpt_options=[], keep_results=False): """This command performs an entire mail transaction. The arguments are: @@ -797,6 +797,8 @@ mail command. - rcpt_options : List of ESMTP options (such as DSN commands) for all the rcpt commands. +- keep_results : If True, return a dictionary of recipients and the + SMTP result for each. msg may be a string containing characters in the ASCII range, or a byte string. A string is encoded to bytes using the ascii codec, and lone @@ -807,17 +809,20 @@ and each of the specified options will be passed to it. If EHLO fails, HELO will be tried and ESMTP options suppressed. -This method will return normally if the mail is accepted for at least -one recipient. It returns a dictionary, with one entry for each -recipient that was refused. Each entry contains a tuple of the SMTP -error code and the accompanying error message sent by the server. +If keep_results is False, this method will return normally if the mail +is accepted for at least one recipient. It returns a dictionary, with +one entry for each recipient that was refused. Each entry contains a +tuple of the SMTP error code and the accompanying error message sent by +the server. If keep_results is True, this method returns a dictionary +of all recipients and the SMTP result whether refused or accepted +unless all are refused then the normal exception is raised. This method may raise the following exceptions: SMTPHeloError The server didn't reply properly to the helo greeting. - SMTPRecipientsRefused The server rejected ALL recipients -(no mail was sent). + SMTPRecipientsRefused The server rejected ALL recipients (no mail +was sent). SMTPSenderRefused The server didn't accept the from_addr. SMTPDataError The server replied with an unexpected error code (other than a refusal of @@ -879,6 +884,10 @@ self._rset() raise SMTPRecipientsRefused(senderrs) (code, resp) = self.data(msg) +if keep_results: +for each in to_addrs: +if not each in senderrs: +senderrs[each]=(code, resp) if code != 250: if code == 421: self.close() -- components: email messages: 287662 nosy: David Ford (FirefighterBlu3), barry, r.david.murray priority: normal severity: normal status: open title: [smtplib] collect response data for all recipients type: enhancement versions: Python 3.7 ___ Python tracker <http://bugs.python.org/issue29539> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue29470] [python] Error in `/usr/bin/python': free(): invalid size: 0x00007f628400d0e0
David Ford (FirefighterBlu3) added the comment: yes, it's running now, hasn't crashed yet -- ___ Python tracker <http://bugs.python.org/issue29470> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue29470] [python] Error in `/usr/bin/python': free(): invalid size: 0x00007f628400d0e0
David Ford (FirefighterBlu3) added the comment: does the builtin memory debugger handle things well such as when involving external libraries like ssl/libcrypto? -- ___ Python tracker <http://bugs.python.org/issue29470> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue29470] [python] Error in `/usr/bin/python': free(): invalid size: 0x00007f628400d0e0
Changes by David Ford (FirefighterBlu3) : -- title: Error in `/usr/bin/python': free(): invalid size: 0x7f628400d0e0 -> [python] Error in `/usr/bin/python': free(): invalid size: 0x7f628400d0e0 ___ Python tracker <http://bugs.python.org/issue29470> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue29470] Error in `/usr/bin/python': free(): invalid size: 0x00007f628400d0e0
New submission from David Ford (FirefighterBlu3): (Only 3.[56] tested) my application listens on an SSL wrapped port, accepts socket and gets an incoming set of messages. connection time frame is several seconds but due to bad Comcast service frequently we have outages which hang connections. almost every day python crashes as follows: Feb 07 02:36:11 ranger.blue-labs.org python[11816]: *** Error in `/usr/bin/python': free(): invalid size: 0x7f628400d0e0 *** Feb 07 02:36:11 ranger.blue-labs.org python[11816]: *** Error in `/usr/bin/python': free(): invalid size: 0x7f628400d0e0 *** Feb 07 02:36:11 ranger.blue-labs.org python[11816]: 2017-02-07 02:36:11 WARNING SSL client handshake has failed: EOF occurred in viola Feb 07 02:36:11 ranger.blue-labs.org python[11816]: === Backtrace: = Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libc.so.6(+0x70c4b)[0x7f62a84cac4b] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libc.so.6(+0x76fe6)[0x7f62a84d0fe6] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libc.so.6(+0x777de)[0x7f62a84d17de] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libcrypto.so.1.0.0(CRYPTO_free+0x1d)[0x7f62a5ceaa6d] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libcrypto.so.1.0.0(BN_clear_free+0x58)[0x7f62a5d4bbc8] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libcrypto.so.1.0.0(RSA_free+0x6f)[0x7f62a5d99a2f] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libcrypto.so.1.0.0(EVP_PKEY_free+0x4a)[0x7f62a5dcdcea] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libcrypto.so.1.0.0(+0x163780)[0x7f62a5de5780] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libcrypto.so.1.0.0(ASN1_template_free+0x450)[0x7f62a5debf00] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libcrypto.so.1.0.0(ASN1_template_free+0x567)[0x7f62a5dec017] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libcrypto.so.1.0.0(ASN1_item_free+0x1fb)[0x7f62a5dec2fb] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libssl.so.1.0.0(SSL_CTX_use_certificate+0xc4)[0x7f62a4ee3d14] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libssl.so.1.0.0(SSL_CTX_use_certificate_chain_file+0x7f)[0x7f62a4ee490f] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/python3.5/lib-dynload/_ssl.cpython-35m-x86_64-linux-gnu.so(+0xe77d)[0x7f6 Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyCFunction_Call+0xe9)[0x7f62a8ac4be9] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyEval_EvalFrameEx+0x78a1)[0x7f62a8b3b941] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(+0x1285e3)[0x7f62a8b3d5e3] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyEval_EvalCodeEx+0x23)[0x7f62a8b3d6c3] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(+0x92fd8)[0x7f62a8aa7fd8] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyObject_Call+0x47)[0x7f62a8a7d2b7] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyObject_CallFunctionObjArgs+0xc2)[0x7f62a8a7dec2] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/python3.5/lib-dynload/_ssl.cpython-35m-x86_64-linux-gnu.so(+0x9c46)[0x7f6 Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libssl.so.1.0.0(ssl_parse_clienthello_tlsext+0x648)[0x7f62a4ec9838] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libssl.so.1.0.0(ssl3_get_client_hello+0x6fa)[0x7f62a4eadeca] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libssl.so.1.0.0(ssl3_accept+0x66f)[0x7f62a4eb27af] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/python3.5/lib-dynload/_ssl.cpython-35m-x86_64-linux-gnu.so(+0xc3a9)[0x7f6 Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyEval_EvalFrameEx+0x843c)[0x7f62a8b3c4dc] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyEval_EvalFrameEx+0x88ca)[0x7f62a8b3c96a] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(+0x1285e3)[0x7f62a8b3d5e3] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: === Backtrace: = Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libc.so.6(+0x70c4b)[0x7f62a84cac4b] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libc.so.6(+0x76fe6)[0x7f62a84d0fe6] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libc.so.6(+0x777de)[0x7f62a84d17de] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libcrypto.so.1.0.0(CRYPTO_free+0x1d)[0x7f62a5ceaa6d] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libcrypto.so.1.0.0(BN_clear_free+0x58)[0x7f62a5d4bbc8] Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libcrypto.so.1.0.0(RSA
[issue18543] urllib.parse.urlopen shouldn't ignore installed opener when called with any ca* argument
David Ford (FirefighterBlu3) added the comment: can we get a review on this please? i'd love to move forward and get the fixes in place. -- versions: +Python 3.5, Python 3.6 ___ Python tracker <http://bugs.python.org/issue18543> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18543] urllib.parse.urlopen shouldn't ignore installed opener when called with any ca* argument
David Ford (FirefighterBlu3) added the comment: In my quest for completeness, I discovered a lack of handling given HTTP->HTTPS redirect. So I've attached another version of this patch which ensures an HTTPS handler is installed if such a redirect is found. -- Added file: http://bugs.python.org/file39863/urllib.request.py-do-not-overwrite-existing-opener.diff ___ Python tracker <http://bugs.python.org/issue18543> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23166] urllib2 ignores opener configuration under certain circumstances
David Ford (FirefighterBlu3) added the comment: I've made a patch for 3.4 that addresses this issue. See issue 18543, latest patch, and test file -- nosy: +David Ford (FirefighterBlu3) ___ Python tracker <http://bugs.python.org/issue23166> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18543] urllib.parse.urlopen shouldn't ignore installed opener when called with any ca* argument
David Ford (FirefighterBlu3) added the comment: Test cases for SSL _opener, contexts and HTTPSHandlers in this file -- Added file: http://bugs.python.org/file39849/ssl_context_tests.py ___ Python tracker <http://bugs.python.org/issue18543> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18543] urllib.parse.urlopen shouldn't ignore installed opener when called with any ca* argument
David Ford (FirefighterBlu3) added the comment: Third version of this patch and a short test suite specifically for this problem. per awareness of :issue:`23166` I rewrote my patch to handle subclassed HTTPS handlers. There are also exceptions raised if an attempt to install more than one HTTPS handler is done. HTTPS does not play well with multiple handlers trying to process data and will immediately fail on a second attempt to negotiate a handshake on an established socket. Additionally, if (a) an HTTPSHandler is already in the chain and (b) doesn't have an SSL context, then we create an appropriate context and attach it to the existing handler -- Added file: http://bugs.python.org/file39848/urllib.request.py-do-not-overwrite-existing-opener.diff ___ Python tracker <http://bugs.python.org/issue18543> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18543] urllib.parse.urlopen shouldn't ignore installed opener when called with any ca* argument
David Ford (FirefighterBlu3) added the comment: perhaps an HTTPSHandler() should only merged into the handler chain if an https URI is found and no existing handler is found that has an https_open() defined -- ___ Python tracker <http://bugs.python.org/issue18543> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18543] urllib.parse.urlopen shouldn't ignore installed opener when called with any ca* argument
David Ford (FirefighterBlu3) added the comment:
Unfortunately more breakage exists within urllib.request. A context supplied to
urlopen() is useless in the following pseudo code:
build_some_openers()
context = ssl.foo()
urlopen('foo.com', context=context)
When urlopen() runs, it does indeed (with my earlier patch) add another
HTTPSHandler(context). However, the default added HTTPSHandler is called first
in the chain (see the bisect.insort) and will cause the urlopen attempt to fail
if the SSL connection does not work with a default or void context.
The end-user specified context will never be reached whether they attempt to
install their own HTTPSHandler or not since the default installed HTTPSHandler
will raise an exception.
Therefore, I've attached another patch to urllib.request which ensures that (a)
existing opener chain is not discarded and (b) a default opener chain is not
made with an HTTPSHandler in it, only adding the HTTPSHandler at urlopen() time
if 'https' is found in the URL.
--
Added file:
http://bugs.python.org/file39847/urllib.request.py-do-not-overwrite-existing-opener.diff
___
Python tracker
<http://bugs.python.org/issue18543>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18543] urllib.parse.urlopen shouldn't ignore installed opener when called with any ca* argument
David Ford (FirefighterBlu3) added the comment: the attached diff (for py3) fixes the (badly) broken urlopen :} previously, if SSL args were detected, all installed handlers via _opener got wiped out. now, we check if _opener already exists and if so we just make sure the HTTPSHandler is added thus preserving installed handlers. -- keywords: +patch nosy: +David Ford (FirefighterBlu3) Added file: http://bugs.python.org/file39846/urllib.request.py-do-not-overwrite-existing-opener.diff ___ Python tracker <http://bugs.python.org/issue18543> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
