David Ford (FirefighterBlu3) added the comment: i do know what's causing it, it's the twilio module not cleanly finishing up. i pointed out my call to this in the initial description.
/usr/lib/python3.6/site-packages/httplib2/__init__.py:857: DeprecationWarning: key_file, cert_file and check_hostname are deprecated, use a custom context instead. check_hostname=disable_ssl_certificate_validation ^ True) /usr/lib/python3.6/site-packages/psycopg2/extras.py:314: ResourceWarning: unclosed <ssl.SSLSocket fd=10, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('173.12.76.132', 42016), raddr=('52.7.124.13', 443)> ts = super(NamedTupleCursor, self).fetchall() Object allocated at (most recent call first): File "/usr/lib/python3.6/ssl.py", lineno 401 _context=self, _session=session) File "/usr/lib/python3.6/http/client.py", lineno 1400 server_hostname=server_hostname) File "/usr/lib/python3.6/site-packages/httplib2/__init__.py", lineno 994 conn.connect() File "/usr/lib/python3.6/site-packages/httplib2/__init__.py", lineno 1071 (response, content) = self._conn_request(conn, request_uri, method, body, headers) File "/usr/lib/python3.6/site-packages/httplib2/__init__.py", lineno 1321 (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey) File "/usr/lib/python3.6/site-packages/twilio/rest/resources/base.py", lineno 117 resp, content = http.request(url, method, headers=headers, body=data) File "/usr/lib/python3.6/site-packages/twilio/rest/resources/base.py", lineno 152 resp = make_request(method, uri, **kwargs) File "/usr/lib/python3.6/site-packages/twilio/rest/resources/base.py", lineno 200 resp = make_twilio_request(method, uri, auth=self.auth, **kwargs) File "/usr/lib/python3.6/site-packages/twilio/rest/resources/base.py", lineno 333 resp, item = self.request("GET", uri) File "/usr/lib/python3.6/site-packages/twilio/rest/resources/base.py", lineno 328 return self.get_instance(sid) File "provider.py", lineno 524 msg = self._twilio.messages.get(r.delivery_id) File "/usr/lib/python3.6/asyncio/events.py", lineno 126 self._callback(*self._args) File "/usr/lib/python3.6/asyncio/base_events.py", lineno 1425 handle._run() File "/usr/lib/python3.6/asyncio/base_events.py", lineno 421 self._run_once() File "/usr/lib/python3.6/site-packages/autobahn/asyncio/wamp.py", lineno 168 loop.run_forever() File "provider.py", lineno 595 runner.run(make) the socket is referenced by ip:port correctly and is coherent with the stack trace. nothing about the IP, socket, or stack trace, is related to the referenced file:position that is reported: /usr/lib/python3.6/site-packages/psycopg2/extras.py:314: ResourceWarning: unclosed <ssl.SSLSocket fd=10, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('173.12.76.132', 42016), raddr=('52.7.124.13', 443)> ts = super(NamedTupleCursor, self).fetchall() the psycopg2 connection is against a completely different IP and on the standard pgsql port of 5432. here's the snippet of psycopg2/extras.py: 313 def fetchall(self): 314 ts = super(NamedTupleCursor, self).fetchall() 315 nt = self.Record 316 if nt is None: 317 nt = self.Record = self._make_nt() 318 return list(map(nt._make, ts)) nothing at all between my pgsql database and twilio share any form of connection or socket and there are no similarly named variables, complex code paths, or global variables. the only object data which has any shared scope is the twilio message fetch using the ID fetched from the pgsql database. if this section of code is commented out, the resource warning vanishes. msg = self._twilio.messages.get(r.delivery_id) if i patch the involved calls in twilio so i can touch the socket object and close it such as: class Response(object): """ Take a httplib2 response and turn it into a requests response """ def __init__(self, httplib_resp, content, url, http_obj): self.content = content self.cached = False self.status_code = int(httplib_resp.status) self.ok = self.status_code < 400 self.url = url self.connections = http_obj.connections (http_obj added which is an instance of httplib2.Http which has a property of .connections wherein sockets are cached) then after modifying the twilio module to pass the Response object to me as well as the message resource, i can eventually in my code: for r in rows: resp, msg = self._twilio.messages.get(r.delivery_id) errors[r.delivery_id] = r.recipient, msg.error_code, msg.error_message for k in resp.connections: resp.connections[k].close() this closes all the cached sockets and ensures the ResourceWarning is never emitted. this messiness is another conversation however, the incorrect file:pos annotation for the ResourceWarning is the purpose of this bug. ======== the only way i can see this inaccurate message being presented as above, is if the resource warning code is not correctly tracking state inside GC runs. supposition: * psycopg2 opened the socket to the PG database (threading is involved when the pool is used) * psycopg2 fetches the rows * psycopg2 closed socket, releasing object * twilio opens a socket which coincidently uses the same object resource just closed but not purged in GC * twilio doesn't close it the SSL socket so it is left hanging * method exits, GC runs but RW state wasn't updated somewhere to say that the twilio/httplib2/http/ssl series of calls now owns that object memory * RW code emits an unclosed socket warning against the outdated psycopg2/extras.py:314 information i'm definitely not intimate with GC code so the above is purely speculation. i've seen people post some vague references to confusing resource warnings when threading is involved. as nothing appears to be affected with long running instances, the actual object handling seems solid, it's just the state tracking for the resource warning which is not correct ---------- _______________________________________ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue29564> _______________________________________ _______________________________________________ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com