On 3/12/2015 5:18 PM, John Nagle wrote: > On 3/12/2015 2:56 PM, Cameron Simpson wrote: >> On 12Mar2015 12:55, John Nagle <na...@animats.com> wrote: >>> I have working code from Python 2 which uses "pickle" to talk to a >>> subprocess via stdin/stdio. I'm trying to make that work in Python >>> 3.
I'm starting to think that the "cpickle" module, which Python 3 uses by default, has a problem. After the program has been running for a while, I start seeing errors such as File "C:\projects\sitetruth\InfoSiteRating.py", line 200, in scansite if len(self.badbusinessinfo) > 0 : # if bad stuff NameError: name 'len' is not defined which ought to be impossible in Python, and File "C:\projects\sitetruth\subprocesscall.py", line 129, in send self.writer.dump(args) # send data OSError: [Errno 22] Invalid argument from somewhere deep inside CPickle. I got File "C:\projects\sitetruth\InfoSiteRating.py", line 223, in get_rating_text (ratingsmalliconurl, ratinglargiconurl, ratingalttext) = DetailsPageBuilder.getratingiconinfo(rating) NameError: name 'DetailsPageBuilder' is not defined (That's an imported module. It worked earlier in the run.) and finally, even after I deleted all .pyc files and all Python cache directories: Fatal Python error: GC object already tracked Current thread 0x00001a14 (most recent call first): File "C:\python34\lib\site-packages\pymysql\connections.py", line 411 in description File "C:\python34\lib\site-packages\pymysql\connections.py", line 1248 in _get_descriptions File "C:\python34\lib\site-packages\pymysql\connections.py", line 1182 in _read_result_packet File "C:\python34\lib\site-packages\pymysql\connections.py", line 1132 in read File "C:\python34\lib\site-packages\pymysql\connections.py", line 929 in _read_query_result File "C:\python34\lib\site-packages\pymysql\connections.py", line 768 in query File "C:\python34\lib\site-packages\pymysql\cursors.py", line 282 in _query File "C:\python34\lib\site-packages\pymysql\cursors.py", line 134 in execute File "C:\projects\sitetruth\domaincacheitem.py", line 128 in select File "C:\projects\sitetruth\domaincache.py", line 30 in search File "C:\projects\sitetruth\ratesite.py", line 31 in ratedomain File "C:\projects\sitetruth\RatingProcess.py", line 68 in call File "C:\projects\sitetruth\subprocesscall.py", line 140 in docall File "C:\projects\sitetruth\subprocesscall.py", line 158 in run File "C:\projects\sitetruth\RatingProcess.py", line 89 in main File "C:\projects\sitetruth\RatingProcess.py", line 95 in <module> That's a definite memory error. So something is corrupting memory. Probably CPickle. All my code is in Python. Every library module came in via "pip", into a clean Python 3.4.3 (32 bit) installation on Win7/x86-64. Currently installed packages: beautifulsoup4 (4.3.2) dnspython3 (1.12.0) html5lib (0.999) pip (6.0.8) PyMySQL (0.6.6) pyparsing (2.0.3) setuptools (12.0.5) six (1.9.0) And it works fine with Python 2.7.9. Is there some way to force the use of the pure Python pickle module? My guess is that there's something about reusing "pickle" instances that botches memory uses in CPython 3's C code for "cpickle". John Nagle -- https://mail.python.org/mailman/listinfo/python-list