#20536: File based cache is not safe for concurrent processes
-------------------------------------+-------------------------------------
Reporter: jaap3 | Owner: nobody
Type: Bug | Status: new
Component: Core (Cache system) | Version: master
Severity: Normal | Resolution:
Keywords: filebasedcache | Triage Stage:
concurrency unicodeerror | Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Changes (by gerdemb):
* needs_better_patch: => 0
* needs_tests: => 0
* needs_docs: => 0
Comment:
I've been running something very similar in production for a long time
with no problems. Here's my code:
{{{
def _cull(self):
return
def set(self, key, value, timeout=None, version=None):
key = self.make_key(key, version=version)
self.validate_key(key)
fname = self._key_to_file(key)
dirname = os.path.dirname(fname)
if not os.path.exists(dirname):
try:
os.makedirs(dirname)
except OSError, e:
if e.errno == errno.EEXIST:
pass
else:
raise
tmpfile = tempfile.NamedTemporaryFile(dir=self._dir, delete=False)
if timeout is None:
timeout = self.default_timeout
self._cull()
now = time.time()
pickle.dump(now + timeout, tmpfile, pickle.HIGHEST_PROTOCOL)
pickle.dump(value, tmpfile, pickle.HIGHEST_PROTOCOL)
# tmpfile.flush()
# os.fsync(tmpfile.fileno())
tmpfile.close()
os.rename(tmpfile.name, fname)
}}}
A couple of notes:
1. The only exception I catch is when makedirs() fails due to an already
existing directory. This can happen when two cache keys that point to the
same directory are added at the same time. I have not seen any other
exceptions raised in production.
2. Somewhere I read that to be truly atomic the temporary file must be
flushed and the file system fsynced. I have commented out those two lines
for better performance and have not had any problems, but I am not an
expert here.
3. I moved the cull() function to a management command that I run
periodically with cron. This improves performance, BUT I suspect there
could be conflicts when there are cache writes or reads during a cull
operation. Since I am only culling once a day, my chances of having a
conflict are quite small and I have not seen any errors in production.
--
Ticket URL: <https://code.djangoproject.com/ticket/20536#comment:1>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
--
You received this message because you are subscribed to the Google Groups
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-updates/063.1272d8a0ebe3d8ca752f8210ec1816cf%40djangoproject.com.
For more options, visit https://groups.google.com/groups/opt_out.
