Re: [Django] #15825: File-based caching doesn't cull truly randomly

[Django]

#15825: File-based caching doesn't cull truly randomly
-+
 Reporter:  gkuenning|Owner:  nobody
 Type:  Bug  |   Status:  closed
Component:  Core (Cache system)  |  Version:  master
 Severity:  Normal   |   Resolution:  fixed
 Keywords:   | Triage Stage:  Accepted
Has patch:  1|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+
Changes (by Anssi Kääriäinen ):

 * status:  new => closed
 * resolution:   => fixed


Comment:

 In [changeset:"7be638390e18fcbfaaed638f9908673360c280d3"]:
 {{{
 #!CommitTicketReference repository=""
 revision="7be638390e18fcbfaaed638f9908673360c280d3"
 Fixed #20536 -- rewrite of the file based cache backend

  * Safer for use in multiprocess environments
  * Better random culling
  * Cache files use less disk space
  * Safer delete behavior

 Also fixed #15806, fixed #15825.
 }}}

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/067.0b244b27c4ab31f75f2504c9d3ae543d%40djangoproject.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [Django] #15825: File-based caching doesn't cull truly randomly

[Django]

#15825: File-based caching doesn't cull truly randomly
-+
 Reporter:  gkuenning|Owner:  nobody
 Type:  Bug  |   Status:  new
Component:  Core (Cache system)  |  Version:  master
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:  Accepted
Has patch:  1|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+

Comment (by jaap3):

 Created a pull request that addresses this issue:
 https://github.com/django/django/pull/1514 (also #20536)

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/067.2a92cac88466c5b9b3fca7ff5eb9fe4a%40djangoproject.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [Django] #15825: File-based caching doesn't cull truly randomly

[Django]

#15825: File-based caching doesn't cull truly randomly
-+
 Reporter:  gkuenning|Owner:  nobody
 Type:  Bug  |   Status:  new
Component:  Core (Cache system)  |  Version:  master
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:  Accepted
Has patch:  1|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+
Changes (by jaap3):

 * cc: jaap3 (added)


-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/067.cba8b1c313049d753345a4f47d9d7858%40djangoproject.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [Django] #15825: File-based caching doesn't cull truly randomly

[Django]

#15825: File-based caching doesn't cull truly randomly
-+
 Reporter:  gkuenning|Owner:  nobody
 Type:  Bug  |   Status:  new
Component:  Core (Cache system)  |  Version:  master
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:  Accepted
Has patch:  1|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+

Comment (by susan):

 Just ran the latest patch "merge_patch" against the cache tests; they
 passed. Then, I ran the applied patch against the full test suite, which
 produced a failure. Stack trace here:

 FAIL: test_cull (cache.tests.FileBasedCacheTests)
 --
 Traceback (most recent call last):
   File "../django/tests/cache/tests.py", line 1085, in test_cull
 self.perform_cull_test(35, 24)
   File "../django/tests/cache/tests.py", line 495, in perform_cull_test
 self.assertEqual(count, final_count)
 AssertionError: 23 != 24

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/067.3353a09cfa4cc32a35f4df52ff07ea66%40djangoproject.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [Django] #15825: File-based caching doesn't cull truly randomly

[Django]

#15825: File-based caching doesn't cull truly randomly
-+
 Reporter:  gkuenning|Owner:  nobody
 Type:  Bug  |   Status:  new
Component:  Core (Cache system)  |  Version:  master
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:  Accepted
Has patch:  1|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+

Comment (by anonymous):

 This seems to have been forgotten. Has the patch rotted?

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [Django] #15825: File-based caching doesn't cull truly randomly

[Django]

#15825: File-based caching doesn't cull truly randomly
-+-
   Reporter:  gkuenning  |  Owner:  nobody
   Type:  Bug| Status:  new
  Milestone: |  Component:  Core (Cache system)
Version:  SVN|   Severity:  Normal
 Resolution: |   Keywords:
   Triage Stage:  Accepted   |  Has patch:  1
Needs documentation:  0  |Needs tests:  0
Patch needs improvement:  0  |  Easy pickings:  0
  UI/UX:  0  |
-+-
Changes (by desh):

 * has_patch:  0 => 1
 * ui_ux:   => 0


Comment:

 I've added patch against trunk AND patch that also fixes #15806 (it's
 actually just a result of merge) to avoid merging conflicts.

 I've removed sorted() cause it's make no sense with random culling. I've
 also replaced nested for that traverse directories to delete paths by
 shutils.rmtree, as it does the same.

 Tests changed in two ways: "cull%d" became "cull-%d" and initial inserts
 count for cull test was reduced from 49 to 34, to have 34 unique 2-char
 prefixes after hashing. It's actually a bit rude but in fact, #14250 was
 about as rude as this.

 I see two other ways to solve last problem with tests:
   * change length of directory name from 2 to 3, so there will be 4096
 possible prefixes (conceptually, same hack but from different pov);
   * change the way of culling: delete exactly
 _num_entries//_cull_frequency FILES, not directories (bad thing, cause it
 will not scale with big cache sizes).

 Waiting for review.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To post to this group, send email to django-updates@googlegroups.com.
To unsubscribe from this group, send email to 
django-updates+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-updates?hl=en.

Re: [Django] #15825: File-based caching doesn't cull truly randomly

[Django]

#15825: File-based caching doesn't cull truly randomly
-+-
   Reporter:  gkuenning  |  Owner:  nobody
   Type:  Bug| Status:  new
  Milestone: |  Component:  Core (Cache system)
Version:  SVN|   Severity:  Normal
 Resolution: |   Keywords:
   Triage Stage:  Accepted   |  Has patch:  0
Needs documentation:  0  |Needs tests:  0
Patch needs improvement:  0  |  Easy pickings:  0
-+-

Comment (by jacob):

 See also #15806. Not exactly related, but right around in that same part
 of code and probably worth fixing at the same time.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To post to this group, send email to django-updates@googlegroups.com.
To unsubscribe from this group, send email to 
django-updates+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-updates?hl=en.

Re: [Django] #15825: File-based caching doesn't cull truly randomly

[Django]

#15825: File-based caching doesn't cull truly randomly
-+-
   Reporter:  gkuenning  |  Owner:  nobody
   Type:  Bug| Status:  new
  Milestone: |  Component:  Core (Cache system)
Version:  SVN|   Severity:  Normal
 Resolution: |   Keywords:
   Triage Stage:  Accepted   |  Has patch:  0
Needs documentation:  0  |Needs tests:  0
Patch needs improvement:  0  |  Easy pickings:  0
-+-

Comment (by lukeplant):

 If this is fixed, please note that the tests will need some subtle fixing,
 particularly `FileBasedCacheTests.test_cull`. The tests check for a
 certain number deleted, but because of the way that files are grouped into
 folders in the file based backend, if this is randomized the number
 deleted can change. The 'sorted' call just a few lines above this line was
 added because of this issue, to fix a difference that appeared on
 different platforms, #14250.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To post to this group, send email to django-updates@googlegroups.com.
To unsubscribe from this group, send email to 
django-updates+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-updates?hl=en.

Re: [Django] #15825: File-based caching doesn't cull truly randomly

[Django]

#15825: File-based caching doesn't cull truly randomly
-+-
   Reporter:  gkuenning  |  Owner:  nobody
   Type:  Bug| Status:  new
  Milestone: |  Component:  Core (Cache system)
Version:  SVN|   Severity:  Normal
 Resolution: |   Keywords:
   Triage Stage:  Accepted   |  Has patch:  0
Needs documentation:  0  |Needs tests:  0
Patch needs improvement:  0  |  Easy pickings:  0
-+-
Changes (by carljm):

 * needs_better_patch:   => 0
 * needs_tests:   => 0
 * version:  1.2 => SVN
 * easy:   => 0
 * needs_docs:   => 0
 * stage:  Unreviewed => Accepted


Comment:

 The referenced line of code in trunk is 123, or `doomed =
 [os.path.join(self._dir, k) for (i, k) in enumerate(filelist) if i %
 self._cull_frequency == 0]`

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To post to this group, send email to django-updates@googlegroups.com.
To unsubscribe from this group, send email to 
django-updates+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-updates?hl=en.

[Django] #15825: File-based caching doesn't cull truly randomly

[Django]

#15825: File-based caching doesn't cull truly randomly
---+
 Reporter:  gkuenning  | Owner:  nobody
 Type:  Bug|Status:  new
Milestone: | Component:  Core (Cache system)
  Version:  1.2|  Severity:  Normal
 Keywords: |  Triage Stage:  Unreviewed
Has patch:  0  |
---+
 The culling algorithm in filebased.py uses an essentially random culling
 method, which is well known to lead to an approximation to LRU.  However,
 the randomness is improperly applied.  For example, consider a frequently-
 accessed page that, by luck, happens to sort first in the file list (due
 to an alphatbetically low hash value).  The culling code will always
 delete this file, while a truly random algorithm would only do so 1/n of
 the time.

 The fix is easy: at approximately line 131 in filebased.py, "== 0" should
 be replaced with "== ranno" where ranno is chosen in the immediately
 preceding line with:

 ranno = random.randint(0, self._cull_frequency)

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To post to this group, send email to django-updates@googlegroups.com.
To unsubscribe from this group, send email to 
django-updates+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-updates?hl=en.