[issue39672] SIGSEGV crash on shutdown with shelve & c pickle

[zd nex]

zd nex  added the comment:

So I want to properly debug this? How I can debug that call dump() for pickle? 
It does not seem to be possible. I guess I need to make some custom build?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39672] SIGSEGV crash on shutdown with shelve & c pickle

[zd nex]

zd nex  added the comment:

Hello,

so I was trying to figure out where actually is problem is. As I do not think 
it is in shelve itself. So I have made different method for __setitem__ on 
shelve and I have found that it is actually in pickle.dump >

Here is code which I have used

def __setitem__(self, key, value):
if self.writeback:
self.cache[key] = value
f = BytesIO()
print("set")
p = pickle.Pickler(f, self._protocol)
try: 
print("before dumps - > crash",value)
p.dump(value)
print("after dump > will not be printed on crash")
except Exception as e:
print("error set",e)
print("after dump",key)
self.dict[key.encode(self.keyencoding)] = f.getvalue()
print("saved")

When in this code user changes p.dump to another method cpython crash does not 
happen. Can you please try to see if it is like that?

--
components: +Interpreter Core, ctypes

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39672] SIGSEGV crash on shutdown with shelve & c pickle

[zd nex]

zd nex  added the comment:

Hi,

i was looking on failing tests on attached pull request and it seems to me that 
it intentionally should create new entry for saving.

Maybe that save should actually happen, but it should be fixed in different 
way? From my small tests I was thinking that problem was actually in pickle, 
but maybe it is connected to shelve itself.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39672] SIGSEGV crash on shutdown with shelve & c pickle

[Andrei Daraschenka]

Change by Andrei Daraschenka :


--
pull_requests: +18016
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/18655

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39672] SIGSEGV crash on shutdown with shelve & c pickle

[Andrei Daraschenka]

Andrei Daraschenka  added the comment:

Hello
I was finally reproduce your problem.
Probles was in Lib/shelve.py module in method Shelf.sync. When python is shut 
down in classes calls __exit__ methods, in our issues method __exti__ called 
method close() which called method sync(). Method sync() tried sync data 
between disk and memory storage. But if key didn't exists on disk - python has 
error segfault.
I attach path for this problem and I will prepare PR soon.

--
keywords: +patch
Added file: https://bugs.python.org/file48913/shelve.py.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39672] SIGSEGV crash on shutdown with shelve & c pickle

[zd nex]

zd nex  added the comment:

So I was trying it again in Python 3.6.9 and 3.8.1 directly in REPL. And it 
behaves same.  I have tried it on two different linux boxes (both 64bit) where 
I have diffrent versions. In both of them it crashes in same way .. destroy 
_ast and then it crashes and faulthandler again shows shelve (pickle)


So I am attaching new crash reports directly from REPL where I just call 
list(data.items()) and then exit()

Btw it seems to me that when PDB is active crash does not occurs until exit() 
is called.

--
Added file: https://bugs.python.org/file48912/crash.txt

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39672] SIGSEGV crash on shutdown with shelve & c pickle

[zd nex]

zd nex  added the comment:

Okay I have managed to crash it when exit() was called and also I am attaching 
output with -v

python3.8 -X faulthandler ce_test_2.py 
start
end
--Return--
> /home/fractal/workspace/test_py_crash/ce_test_2.py(19)()->None
-> breakpoint()
(Pdb) exit()
Traceback (most recent call last):
  File "ce_test_2.py", line 19, in 
breakpoint()
  File "/usr/lib/python3.8/bdb.py", line 92, in trace_dispatch
return self.dispatch_return(frame, arg)
  File "/usr/lib/python3.8/bdb.py", line 154, in dispatch_return
if self.quitting: raise BdbQuit
bdb.BdbQuit
Fatal Python error: Segmentation fault

Current thread 0x7f976975f740 (most recent call first):
  File "/usr/lib/python3.8/shelve.py", line 124 in __setitem__
  File "/usr/lib/python3.8/shelve.py", line 168 in sync
  File "/usr/lib/python3.8/shelve.py", line 144 in close
  File "/usr/lib/python3.8/shelve.py", line 162 in __del__
Neoprávněný přístup do paměti (SIGSEGV)
$ 


# destroy string
# cleanup[2] removing threading
# cleanup[2] removing atexit
# cleanup[2] removing logging
# cleanup[2] removing material
# cleanup[2] removing _dbm
# cleanup[2] removing dbm.ndbm
# cleanup[2] removing dbm
# destroy dbm
# cleanup[2] removing _gdbm
# cleanup[2] removing dbm.gnu
# cleanup[2] removing _ast
# cleanup[2] removing ast
# cleanup[2] removing dbm.dumb
# destroy _ast
Fatal Python error: Segmentation fault

Current thread 0x7fd5477cb740 (most recent call first):
  File "/usr/lib/python3.8/shelve.py", line 124 in __setitem__
  File "/usr/lib/python3.8/shelve.py", line 168 in sync
  File "/usr/lib/python3.8/shelve.py", line 144 in close
  File "/usr/lib/python3.8/shelve.py", line 162 in __del__
Neoprávněný přístup do paměti (SIGSEGV)

--
Added file: https://bugs.python.org/file48907/crash_pdb.txt

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39672] SIGSEGV crash on shutdown with shelve & c pickle

[zd nex]

zd nex  added the comment:

Okay I have tried to run it with breakpoint() but it does not crash on 3.8

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39672] SIGSEGV crash on shutdown with shelve & c pickle

[Andrei Daraschenka]

Andrei Daraschenka  added the comment:

Yes, It's work for me

$ uname -a
Linux laptop 5.3.0-40-generic #32~18.04.1-Ubuntu SMP Mon Feb 3 14:05:59 UTC 
2020 x86_64 x86_64 x86_64 GNU/Linux
$ python3.8 --version
Python 3.8.0
$ python3.8 ce_test_2.py 
start
end

Could you please write about the results of the research with pdb.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39672] SIGSEGV crash on shutdown with shelve & c pickle

[zd nex]

zd nex  added the comment:

Ok I will try pdb

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39672] SIGSEGV crash on shutdown with shelve & c pickle

[zd nex]

zd nex  added the comment:

Hello,

well and in 3.8 it does not crash for you? Is there some devel build of 3.9 for 
ubuntu which I can try?

I have tested it on 3.7,3.8 and 3.6 and it crashed always when close was not 
present or when list was called in another function.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39672] SIGSEGV crash on shutdown with shelve & c pickle

[Andrei Daraschenka]

Andrei Daraschenka  added the comment:

Could you give more details for reproduce it because on the latest version 
cpython from master branch it's work
You can debug it with help pdb. Just set breakpoint:

...
test_shelve(data)
breakpoint()
data.close()
...

And try run it step-by-step (press 'S' to go to the next step and press 'll' to 
know where you are now)
https://docs.python.org/3/library/pdb.html#debugger-commands

--
nosy: +dorosch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39672] SIGSEGV crash on shutdown with shelve & c pickle

[zd nex]

Change by zd nex :


--
versions: +Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39672] SIGSEGV crash on shutdown with shelve & c pickle

[zd nex]

zd nex  added the comment:

So I was trying to figure out what is crash it self and it looks to me that it 
is related to import. Do you know how I can properly debug this crash?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39672] SIGSEGV crash on shutdown with shelve & c pickle

[zd nex]

New submission from zd nex :

Hello,

so I was transferring some our old code from Python2.7 to new and find that new 
version seems to crash quite a lot. After some finding (good thing 
faulthandler) I think I tracked it down to to Shelve.__del__ method > going to 
C Pickle module (not python one). Here it is crash itself. Attached zip has 3 
file. When shelve.close is used it does not seem to crash every time. 

$python3.8 -X faulthandler ce_test_2.py
start
end
Fatal Python error: Segmentation fault

Current thread 0x7fb22e299740 (most recent call first):
  File "/usr/lib/python3.8/shelve.py", line 124 in __setitem__
  File "/usr/lib/python3.8/shelve.py", line 168 in sync
  File "/usr/lib/python3.8/shelve.py", line 144 in close
  File "/usr/lib/python3.8/shelve.py", line 162 in __del__
Neoprávněný přístup do paměti (SIGSEGV)


Code for crash is here:


import shelve
import material
data = shelve.open("test3", flag="c",writeback=True)

def test_shelve(data):
for k,v in data.items():
pass

print("start")
test_shelve(data)

#data.close() #fixes SIGSEGV at shutdown
#actually problem is in c pickle module; when Python pickle module is used it 
works

print("end")
#after this it is crash



Code just loads module and shelve and opens file. Then in another function it 
cycles through data and that creates crash in C pickle module at shutdown. 
Weird thing is that when cycle through data is not in function it does not 
crash. Also crash can be avoided when C Pickle is traded for Python Pickle.


In REPL it is quite similar just list on shelve.items() and exit makes Python 
crash.

Python 3.8.1 (default, Dec 22 2019, 08:15:39) 
[GCC 7.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import shelve
>>> import material
>>> data = shelve.open("test3", flag="c",writeback=True)
>>> list(data.items())
[('H1615', Material(name='Třešeň Romana', code='H1615', vars=0))]
>>> exit()
Fatal Python error: Segmentation fault

Current thread 0x7f14a2546740 (most recent call first):
  File "/usr/lib/python3.8/shelve.py", line 124 in __setitem__
  File "/usr/lib/python3.8/shelve.py", line 168 in sync
  File "/usr/lib/python3.8/shelve.py", line 144 in close
  File "/usr/lib/python3.8/shelve.py", line 162 in __del__
Neoprávněný přístup do paměti (SIGSEGV)

Hopefully you can fix this.

--
components: Library (Lib)
files: test_crash_shelve.zip
messages: 362186
nosy: zd nex
priority: normal
severity: normal
status: open
title: SIGSEGV crash on shutdown with shelve & c pickle
type: crash
versions: Python 3.6, Python 3.7, Python 3.8
Added file: https://bugs.python.org/file48899/test_crash_shelve.zip

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com