On 14Dec2013 10:15, Tim Chase <python.l...@tim.thechases.com> wrote: > On 2013-12-14 07:29, JL wrote: > > I have a number of python processes which communicate with each > > other through writing/reading config text files. The python > > ConfigParser is used. I am wondering if it is more CPU-efficient to > > switch to using sqlite database instead of using configuration > > files. If the software does plenty of reading/writing, is it more > > efficient to use config text files or sqlite database? > > I'm pretty sure that the CPU aspect doesn't really play into things. > A few thoughts: > > + You'll be I/O bound most of the time. Even if you used a ramdisk > to reduce disk access delays, accessing multiple .txt files requires > the OS to do permission-checking each time, while a single sqlite > file gets checked once upon opening the DB initially. > + text-files are fragile unless you take extra pains to keep things > atomic > + sqlite guarantee* atomicity, so you either see all-or-nothing > + sqlite is also very efficient for querying
Annoyingly, sqlite: + only lets one process access the db at a time, taking you back to a similar situation as with config files + only lets you access the db from the same thread in which it was opened, outstandingly annoying; I've had to gratuitously refactor code because of this + traditionally, sqlite is extreme fsync() happy; forces a disc level flush on each commit - extremely slow on busy databases, not to mention hard of drives > + sticking with plain-text config files is just asking for some sort > of race-condition or partial-file issue to come up A locking mechanism will be required. Lockfiles work quite well. > + sqlite may give you less CPU load is just an added benefit > > * well, except on NFS shares and other places where file-locking is > unreliable Backing off to config files, making a lock directory is NFS safe. So is opening a lock file for write with zero permissions (low level open with mode=0). Cheers, -- Cameron Simpson <c...@zip.com.au> I drank what?!? - Socrates -- https://mail.python.org/mailman/listinfo/python-list