[issue25698] test regressions introduced with the fix for #22995

[Barry A. Warsaw]

Barry A. Warsaw added the comment:

I set a breakpoint where the error occurs.  I'm in test_cpickle.py dumps().  
Doing cPickle.dumps(arg)  in the bp, I get the error.  arg is:

(Pdb) arg
{<__main__.H object at 0x7ff8e2a870d0>: None}
(Pdb) type(arg)


Look again at the AttributeError:

(Pdb) cPickle.dumps(arg)
*** AttributeError: 'module' object has no attribute '_reduce_ex'

First, why is that trying to find the attribute of a module object instead of 
the MyDict instance?  I even tried this just to discount the effects of 
__main__.H:

(Pdb) cPickle.dumps(type(arg)())
*** AttributeError: 'module' object has no attribute '_reduce_ex'

(as you'd expect, type(arg)() is just an instance of a MyDict.

Two questions: why is it trying to get the attribute on a module object, and 
why is the attribute it's trying to get '_reduce_ex' instead of '__reduce_ex__'?

The attribute to get is taken from the __reduce_ex___str in cPickle.c, which 
gets defined as:

#define INIT_STR(S) if (!( S ## _str=PyString_InternFromString(#S)))  return -1;
...
INIT_STR(__reduce_ex__);

This is a compile-time definition, so it makes no sense that it would be trying 
to find the wrong attribute in some cases (e.g. when running the full 
regrtest.py) but not in others (e.g. running just test_cpickle.py).  But I can 
find no other references to a string containing "reduce_ex" so the cPickle 
version has to be it.  And that gets initialized in init_stuff(), called from 
initcPickle() so I don't see any other way for it to get corrupted.

cPickle.dumps({}) works just fine, so it's an artifact of the MyDict.  That's 
all I've figured out for now.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25698] test regressions introduced with the fix for #22995

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Could you please add debugging output to check that the copy_reg module is 
empty if tests failed?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25698] test regressions introduced with the fix for #22995

[Barry A. Warsaw]

Barry A. Warsaw added the comment:

This seems entirely reproducible in a local Xenial sbuild using the source 
packages in https://launchpad.net/~doko/+archive/ubuntu/toolchain/+packages

Interestingly, inspection of the built artifacts in the schroot after all is 
done yields:

# find . -name copy_reg.py? | xargs ls -l
-rw-rw-r-- 1 barry barry 5597 Nov 30 16:52 ./Lib/copy_reg.pyc
-rw-rw-r-- 1 barry barry 5553 Nov 30 16:53 ./Lib/copy_reg.pyo
-rw-rw-r-- 1 barry barry 5091 Nov 30 17:03 
./debian/tmp-dbg/usr/lib/python2.7/copy_reg.pyc
-rw-rw-r-- 1 barry barry 5047 Nov 30 17:03 
./debian/tmp-dbg/usr/lib/python2.7/copy_reg.pyo
-rw-rw-r-- 1 barry barry 5091 Nov 30 17:02 
./debian/tmp-shdbg/usr/lib/python2.7/copy_reg.pyc
-rw-rw-r-- 1 barry barry 5047 Nov 30 17:02 
./debian/tmp-shdbg/usr/lib/python2.7/copy_reg.pyo

Those are debug builds, so it seems reasonable they are different sizes than 
the non-debug artifacts in Lib, right?  And since they are self-consistent in 
size, it seems like it wouldn't be caused by a pyc/pyo write bug.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25698] test regressions introduced with the fix for #22995

[Barry A. Warsaw]

Barry A. Warsaw added the comment:

But if I just run test_cpickle.py, it succeeds!

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25698] test regressions introduced with the fix for #22995

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Default __reduce_ex__ and __reduce__ implementations call copy_reg._reduce_ex. 
cPickle.dumps({}) works because __reduce_ex__ is not called, dict is supported 
by pickle directly.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25698] test regressions introduced with the fix for #22995

[Barry A. Warsaw]

Barry A. Warsaw added the comment:

First failure in test run afaict:

==
ERROR: test_recursive_dict_subclass_and_inst 
(test.test_cpickle.cStringIOCPicklerFastTests)
--
Traceback (most recent call last):
  File "/build/python2.7-96oGYh/python2.7-2.7.11~rc1/Lib/test/test_cpickle.py", 
line 175, in wrapper
func(self)
  File "/build/python2.7-96oGYh/python2.7-2.7.11~rc1/Lib/test/pickletester.py", 
line 819, in test_recursive_dict_subclass_and_inst
self.check_recursive_collection_and_inst(MyDict.fromkeys)
  File "/build/python2.7-96oGYh/python2.7-2.7.11~rc1/Lib/test/pickletester.py", 
line 790, in check_recursive_collection_and_inst
s = self.dumps(y, proto)
  File "/build/python2.7-96oGYh/python2.7-2.7.11~rc1/Lib/test/test_cpickle.py", 
line 141, in dumps
p.dump(arg)
AttributeError: 'module' object has no attribute '_reduce_ex'

==

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25698] test regressions introduced with the fix for #22995

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

... for protocols < 2.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25698] test regressions introduced with the fix for #22995

[Barry A. Warsaw]

Changes by Barry A. Warsaw :


--
nosy: +barry

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25698] test regressions introduced with the fix for #22995

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

This can be related to issue25083. Can you look at the size of copy_reg.pyc and 
copy_reg.pyo files?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25698] test regressions introduced with the fix for #22995

[Martin Panter]

Martin Panter added the comment:

Duplicate or related to Issue 25601?

--
nosy: +martin.panter

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25698] test regressions introduced with the fix for #22995

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

In your report test_cpickle has number 93. What other 92 tests were ran before? 
Could you run tests few times and find minimal list of tests that precede 
failing test_cpickle?

This failure is not related to issue22995 because the same failure was reported 
in issue25601 5 commits before first commit in issue22995.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25698] test regressions introduced with the fix for #22995

[Matthias Klose]

Matthias Klose added the comment:

looks like re-running the tests in verbose mode using -w is affected by this 
issue.

the first cpickle subtest failing is:

==
ERROR: test_recursive_dict_subclass_and_inst 
(test.test_cpickle.cStringIOCPicklerFastTests)
--
Traceback (most recent call last):
  File 
"/home/packages/python/2.7/python2.7-2.7.11~rc1/Lib/test/test_cpickle.py", line 
175, in wrapper
func(self)
  File 
"/home/packages/python/2.7/python2.7-2.7.11~rc1/Lib/test/pickletester.py", line 
819, in test_recursive_dict_subclass_a
nd_inst
self.check_recursive_collection_and_inst(MyDict.fromkeys)
  File 
"/home/packages/python/2.7/python2.7-2.7.11~rc1/Lib/test/pickletester.py", line 
790, in check_recursive_collection_and
_inst
s = self.dumps(y, proto)
  File 
"/home/packages/python/2.7/python2.7-2.7.11~rc1/Lib/test/test_cpickle.py", line 
141, in dumps
p.dump(arg)
AttributeError: 'module' object has no attribute '_reduce_ex'

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25698] test regressions introduced with the fix for #22995

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Yes, it is a duplicate of issue25601. And I can't reproduce it locally.

Matthias, can you reproduce test failure every time when run tests? If yes, 
could you please run tests in verbose mode and found what tests in test_cpickle 
were failed first?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25698] test regressions introduced with the fix for #22995

[Matthias Klose]

New submission from Matthias Klose:

seen with the 2.7.11 release candidate; I'm not yet 100% if this is the fix for 
#22995 causing these test regressions.

[ 93/395] test_cpickle
test test_cpickle failed -- multiple errors occurred; run in verbose mode for 
details
[101/395/2] test_decimal
test test_decimal failed -- Traceback (most recent call last):
  File 
"/home/packages/python/2.7/python2.7-2.7.11~rc1/Lib/test/test_decimal.py", line 
1774, in test_pickle
e = pickle.loads(pickle.dumps(c, proto))
  File "/home/packages/python/2.7/python2.7-2.7.11~rc1/Lib/pickle.py", line 
1380, in dumps
Pickler(file, protocol).dump(obj)
  File "/home/packages/python/2.7/python2.7-2.7.11~rc1/Lib/pickle.py", line 
224, in dump
self.save(obj)
  File "/home/packages/python/2.7/python2.7-2.7.11~rc1/Lib/pickle.py", line 
306, in save
rv = reduce(self.proto)
AttributeError: 'module' object has no attribute '_reduce_ex'

[108/395/4] test_dictviews
test test_dictviews failed -- Traceback (most recent call last):
  File 
"/home/packages/python/2.7/python2.7-2.7.11~rc1/Lib/test/test_dictviews.py", 
line 201, in test_pickle
pickle.dumps, d.viewkeys(), proto)
  File "/home/packages/python/2.7/python2.7-2.7.11~rc1/Lib/unittest/case.py", 
line 473, in assertRaises
callableObj(*args, **kwargs)
  File "/home/packages/python/2.7/python2.7-2.7.11~rc1/Lib/pickle.py", line 
1380, in dumps
Pickler(file, protocol).dump(obj)
  File "/home/packages/python/2.7/python2.7-2.7.11~rc1/Lib/pickle.py", line 
224, in dump
self.save(obj)
  File "/home/packages/python/2.7/python2.7-2.7.11~rc1/Lib/pickle.py", line 
306, in save
rv = reduce(self.proto)
AttributeError: 'module' object has no attribute '_reduce_ex'

[177/395/7] test_imp
test test_imp failed -- Traceback (most recent call last):
  File "/home/packages/python/2.7/python2.7-2.7.11~rc1/Lib/test/test_imp.py", 
line 60, in test_source
imp.reload(os)
  File "/home/packages/python/2.7/python2.7-2.7.11~rc1/Lib/os.py", line 727, in 

_copy_reg.pickle(stat_result, _pickle_stat_result, _make_stat_result)
AttributeError: 'module' object has no attribute 'pickle'

[184/395/10] test_int_literal
[185/395/10] test_ioctl
test test_ioctl crashed -- : 'module' object 
has no attribute 'pickle'
[186/395/11] test_isinstance
[187/395/11] test_iter
[188/395/11] test_iterlen
[189/395/11] test_itertools
[190/395/11] test_json
test test_json crashed -- : 'module' object 
has no attribute 'pickle'
[191/395/12] test_kqueue
test_kqueue skipped -- test works only on BSD
[192/395/12] test_largefile
test test_largefile crashed -- : 'module' 
object has no attribute 'pickle'
[193/395/13] test_lib2to3
test test_lib2to3 crashed -- : 'module' 
object has no attribute 'pickle'
[194/395/14] test_linecache
test test_linecache crashed -- : 'module' 
object has no attribute 'pickle'
[195/395/15] test_list
test test_list crashed -- : 'module' object 
has no attribute 'pickle'
[196/395/16] test_locale
[197/395/16] test_logging
test test_logging crashed -- : 'module' 
object has no attribute 'pickle'
[198/395/17] test_long
[199/395/17] test_long_future
[200/395/17] test_longexp
[201/395/17] test_macos
test test_macos crashed -- : 'module' object 
has no attribute 'pickle'
[202/395/18] test_macostools
test test_macostools crashed -- : 'module' 
object has no attribute 'pickle'
[203/395/19] test_macpath
[204/395/19] test_macurl2path
[205/395/19] test_mailbox
test test_mailbox crashed -- : 'module' 
object has no attribute 'pickle'
[206/395/20] test_marshal
test test_marshal crashed -- : 'module' 
object has no attribute 'pickle'
[207/395/21] test_math
test test_math crashed -- : 'module' object 
has no attribute 'pickle'
[208/395/22] test_md5
[209/395/22] test_memoryio
test test_memoryio failed -- multiple errors occurred; run in verbose mode for 
details
[210/395/23] test_memoryview
[211/395/23] test_mhlib
test test_mhlib crashed -- : 'module' object 
has no attribute 'pickle'

--
components: Extension Modules
messages: 255109
nosy: benjamin.peterson, doko, serhiy.storchaka
priority: release blocker
severity: normal
status: open
title: test regressions introduced with the fix for #22995
versions: Python 2.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com