[issue25442] Shelve consistency issues

[Yanyan Jiang]

Yanyan Jiang added the comment:

Thanks for reminding. It is originally reported with the default setting. We 
conducted further tests with other options of anydbm (dbhash, dbm, gdbm), none 
of them survived crash testing. For the detailed reasoning please refer to an 
OSDI'14 research paper: 
https://www.usenix.org/system/files/conference/osdi14/osdi14-paper-pillai.pdf
 This paper discussed vulnerabilities of GDBM implementation in that paper, and 
these lightweight db implementations have similar problems. We also have tests 
SQLite, and it is much more robust that we have not found ACID violation yet.

Personally I think it is reasonable to have an SQLite backend, as it is much 
safer (plus providing thread safety). Just to see what I can do for that.

--

___
Python tracker 
<http://bugs.python.org/issue25442>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25442] Shelve consistency issues

[Yanyan Jiang]

New submission from Yanyan Jiang:

I am currently working on the file system reliability issues. I have a disk 
driver that is able to simulate crash disk sites after injected power failures. 
This disk is totally compatible with the Linux block driver semantics (refer to 
 https://www.kernel.org/doc/Documentation/block/writeback_cache_control.txt), 
and may create many crash sites that pending blocks are partially flushed into 
the disk which is a common behavior of a commodity disk with write buffer.

Our automated tool confirms the corruptions could happen on a crash site at an 
unclean shutdown (Linux with default ext4 setting). We also found that there 
are some discussions on 
[Stackoverflow](http://stackoverflow.com/questions/4226580/prevent-python-shelve-corruption)
 concerning this issue. I am suggesting to explicitly remind the developers of 
such behaviors.

Suggested documentation enhancement
--
As a minimal database library, `shelve` does not offer as strong ACID 
(atomicity, consistency, isolation and durability) guarantee as a database 
(like SQLite). On certain system configurations, a system crash would lead to a 
corrupted shelve file. If you are using shelve to persistent precious data like 
user's document, we suggest using the following steps to ensure data is not 
lost:

1. Create a copy of the file, say, the temporary.
2. Operate on a copy of the temporary file. Closing a shelve db implies data to 
be flushed to the disk.
3. Rename the temporary file to replace the original file. Renaming is 
carefully treated by a journaled filesystem to be atomic.

--
assignee: docs@python
components: Documentation
messages: 253188
nosy: Yanyan Jiang, docs@python
priority: normal
severity: normal
status: open
title: Shelve consistency issues
type: enhancement
versions: Python 2.7, Python 3.2, Python 3.3, Python 3.4, Python 3.5, Python 3.6

___
Python tracker 
<http://bugs.python.org/issue25442>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com