Re: Feature request: ttl method for cache

[Malcolm Box]

Ticket filed and pull request for fix
https://code.djangoproject.com/ticket/22606


On 9 May 2014 10:38, Malcolm Box  wrote:

>
> On Thursday, 8 May 2014 16:59:55 UTC+1, Piotr Gosławski wrote:
>>
>> I was fixing my little helper function to behave more like Sean's and I
>> think I've found a bug in locmem. Could you please take a look at this:
>>
>> Either I'm missing something or has_key() is not working correctly for
>> keys without expiration time.
>>
>
> I think you're right - this is related to this bug:
> https://code.djangoproject.com/ticket/22495 and caused by the same patch.
>
> File a ticket, and I'll have a look at fixing.
>
> Malcolm
>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "Django developers" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/django-developers/nwZBw64cD4c/unsubscribe
> .
> To unsubscribe from this group and all its topics, send an email to
> django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at http://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/956bfca6-3ef7-4b4c-ac91-970628ec7d4b%40googlegroups.com
> .
> For more options, visit https://groups.google.com/d/optout.
>



-- 
Malcolm Box
CTO & Co-Founder, Tellybug  http://tellybug.com
malc...@tellybug.com   +44 7766 990123@malcolmbox

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAF3R4sVVpHxBXmB%3D1pZUGv%2BpgvmxbgWxjZCo19OLELFuQw%2BtqQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Feature request: ttl method for cache

[Malcolm Box]

On Thursday, 8 May 2014 16:59:55 UTC+1, Piotr Gosławski wrote:
>
> I was fixing my little helper function to behave more like Sean's and I 
> think I've found a bug in locmem. Could you please take a look at this:
>
> Either I'm missing something or has_key() is not working correctly for 
> keys without expiration time.
>

I think you're right - this is related to this 
bug: https://code.djangoproject.com/ticket/22495 and caused by the same 
patch.

File a ticket, and I'll have a look at fixing.

Malcolm

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/956bfca6-3ef7-4b4c-ac91-970628ec7d4b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Feature request: ttl method for cache

[Piotr Gosławski]

I was fixing my little helper function to behave more like Sean's and I 
think I've found a bug in locmem. Could you please take a look at this:

>>> cache

>>> cache.set('a', 1)
>>> cache.set('b', 2, None)
>>> cache.get('a')
1
>>> cache.get('b')
2
>>> cache.has_key('a')
True
>>> cache.has_key('b')
False

Either I'm missing something or has_key() is not working correctly for keys 
without expiration time.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/b2e0648c-5e8d-4d2e-a899-b354bf4af58f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Feature request: ttl method for cache

[Piotr Gosławski]

It would sacrifice atomicity of incr()/decr() methods and hit their speed 
pretty hard.

W dniu wtorek, 6 maja 2014 16:47:53 UTC+2 użytkownik adamcik napisał:
>
> On Tue, May 06, 2014 at 05:57:28AM -0700, Piotr Gosławski wrote: 
> > W dniu wtorek, 6 maja 2014 11:47:17 UTC+2 użytkownik Florian Apolloner 
> > napisał: 
> > > 
> > > [...] 
> > > Memcached doesn't provide access to the remaining TTL, and I don't see 
> how 
> > > we can reasonably fake this without writing an extra key containing 
> the 
> > > expiration date. 
> > > 
> > >  Would that be unacceptable to add a switch in backend settings and 
> also 
> > as extra argument for set() that would cause django to transparently 
> save 
> > ttl as an extra key? It would obviously be disabled by default and docs 
> > would say about the extra memory this feature needs. 
>
> See 
> https://groups.google.com/d/msg/django-developers/ctKJzBTONu8/opbWqUIcOKgJ 
> for a similar case that has come up before. The tuple solution used there 
> could easily be adapted to store the time the key will expire, giving you 
> the 
> information you need. 
>
> -Thomas 
>
> > 
> > -- 
> > You received this message because you are subscribed to the Google 
> Groups "Django developers" group. 
> > To unsubscribe from this group and stop receiving emails from it, send 
> an email to django-develop...@googlegroups.com . 
> > To post to this group, send email to 
> > django-d...@googlegroups.com. 
>
> > Visit this group at http://groups.google.com/group/django-developers. 
> > To view this discussion on the web visit 
> https://groups.google.com/d/msgid/django-developers/913011d8-f4c4-4e7c-ac8f-6aa85869e669%40googlegroups.com.
>  
>
> > For more options, visit https://groups.google.com/d/optout. 
>
> -- 
> mvh 
> Thomas Kongevold Adamcik 
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/06af0d07-d328-4f31-9582-f4f5a6f3d139%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Feature request: ttl method for cache

[Piotr Gosławski]

W dniu wtorek, 6 maja 2014 16:47:53 UTC+2 użytkownik adamcik napisał:
>
> See 
> https://groups.google.com/d/msg/django-developers/ctKJzBTONu8/opbWqUIcOKgJ 
> for a similar case that has come up before. The tuple solution used there 
> could easily be adapted to store the time the key will expire, giving you 
> the 
> information you need. 
>
> -Thomas 
>
>  
I would sacrifice atomicity of incr()/decr() methods and hit their 
performance pretty hard.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/3e26c0b8-ab60-41b2-aa2d-2c2edc47e9b1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Feature request: ttl method for cache

[Thomas K. Adamcik]

On Tue, May 06, 2014 at 05:57:28AM -0700, Piotr Gosławski wrote:
> W dniu wtorek, 6 maja 2014 11:47:17 UTC+2 użytkownik Florian Apolloner 
> napisał:
> >
> > [...]
> > Memcached doesn't provide access to the remaining TTL, and I don't see how 
> > we can reasonably fake this without writing an extra key containing the 
> > expiration date.
> >
> >  Would that be unacceptable to add a switch in backend settings and also 
> as extra argument for set() that would cause django to transparently save 
> ttl as an extra key? It would obviously be disabled by default and docs 
> would say about the extra memory this feature needs.

See https://groups.google.com/d/msg/django-developers/ctKJzBTONu8/opbWqUIcOKgJ
for a similar case that has come up before. The tuple solution used there
could easily be adapted to store the time the key will expire, giving you the
information you need.

-Thomas

> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Django developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at http://groups.google.com/group/django-developers.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/django-developers/913011d8-f4c4-4e7c-ac8f-6aa85869e669%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

-- 
mvh
Thomas Kongevold Adamcik

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/20140506144753.GC5886%40samfundet.no.
For more options, visit https://groups.google.com/d/optout.

Re: Feature request: ttl method for cache

[Michael Manfre]

On Tue, May 6, 2014 at 8:57 AM, Piotr Gosławski
wrote:

> W dniu wtorek, 6 maja 2014 11:47:17 UTC+2 użytkownik Florian Apolloner
> napisał:
>>
>> [...]
>>
>> Memcached doesn't provide access to the remaining TTL, and I don't see
>> how we can reasonably fake this without writing an extra key containing the
>> expiration date.
>>
>>  Would that be unacceptable to add a switch in backend settings and also
> as extra argument for set() that would cause django to transparently save
> ttl as an extra key? It would obviously be disabled by default and docs
> would say about the extra memory this feature needs.
>

The majority of projects wouldn't use this and it's possible to accomplish
with the existing cache API, so I don't think it makes sense to add this to
Django. This functionality sounds like it is a better fit for a 3rd party
app that provides custom cache backends.

Regards,
Michael Manfre

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAGdCwBuTbm5Ey073fMrRpzT8YESHP3%3D9mAtZO4ONGqf5wZfdAQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Feature request: ttl method for cache

[Piotr Gosławski]

W dniu wtorek, 6 maja 2014 11:47:17 UTC+2 użytkownik Florian Apolloner 
napisał:
>
> [...]
> Memcached doesn't provide access to the remaining TTL, and I don't see how 
> we can reasonably fake this without writing an extra key containing the 
> expiration date.
>
>  Would that be unacceptable to add a switch in backend settings and also 
as extra argument for set() that would cause django to transparently save 
ttl as an extra key? It would obviously be disabled by default and docs 
would say about the extra memory this feature needs.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/913011d8-f4c4-4e7c-ac8f-6aa85869e669%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Feature request: ttl method for cache

[Russell Keith-Magee]

On Tue, May 6, 2014 at 5:47 PM, Florian Apolloner wrote:

> Hi Russ,
>
>
> On Tuesday, May 6, 2014 2:54:52 AM UTC+2, Russell Keith-Magee wrote:
>>
>> As far as the NotImplemented bit goes - I'd rather see this implemented
>> for all officially supported backends, rather than only one backend. The
>> only exception to this would be if there is a fundamental technical reason
>> why TTL can't be implemented (or adequately faked).
>>
>
> Memcached doesn't provide access to the remaining TTL, and I don't see how
> we can reasonably fake this without writing an extra key containing the
> expiration date.
>

/me facepalms

I realise now that I when I took a cursory look at the sample code that was
provided, I read "LocMemCache" as "MemCache", and assumed that there was an
exposed API that could be used to calculate TTL.

If this isn't the case, then I'll retract my support for this feature. If
TTL isn't something memcache supports, then I'm not sure I see the value in
adding this as a feature. The cache API really exists to provide a cache
mockup when you don't have easy access to memcache (e.g., during
development and testing); I'm not sure I see the benefit in adding support
for a feature that isn't' supported by the most important implementation.

Russ %-)

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAJxq849zRU4d_3zo-vn4j7AQqTLLAiXUFmXcsb2XKEBfAHUFiw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Feature request: ttl method for cache

[Florian Apolloner]

Hi Russ,

On Tuesday, May 6, 2014 2:54:52 AM UTC+2, Russell Keith-Magee wrote:
>
> As far as the NotImplemented bit goes - I'd rather see this implemented 
> for all officially supported backends, rather than only one backend. The 
> only exception to this would be if there is a fundamental technical reason 
> why TTL can't be implemented (or adequately faked).
>

Memcached doesn't provide access to the remaining TTL, and I don't see how 
we can reasonably fake this without writing an extra key containing the 
expiration date.

Cheers,
Florian

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/4eac4dbe-5a3e-4c0a-b2ec-4d341e22bc2d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Feature request: ttl method for cache

[Sean Bleier]

Hi Piotr,

For what it's worth, I have an implementation of ttl on django-redis-cache:
https://github.com/sebleier/django-redis-cache/commit/626a3263c428cf59b1428f5fc2aa638efd77346a

It's slightly different from your snippet in that if the key does not exist
(because it is expired or didn't exist in the first place), the returned
value will be 0 instead of None.  None is reserved for non-volatile keys
with no expirations.

–Sean


On Mon, May 5, 2014 at 1:16 PM, Piotr Gosławski
wrote:

> Hi!
>
> Since the contribution workflow is a bit confusing to me, I'll just leave
> it here.
> I think django's cache needs ttl (time to live) method that would return
> time left until specified key expires. I, for instance, needed one when
> writing
> a rate limiter. Without it I would have to store twice as much key-value
> pairs
> in cache. I ended up with a workaround that dynamically binds ttl method,
> but only for locmem and redis, since those are the backends I use.
>
> Here is my workoround, maybe it will be of some use:
> https://github.com/p-tr0/snipets/blob/master/django_cache_ttl.py
>
> I guess the proper way would be to add a method that just raises
> NotImplementedError and then cover it one backend at a time.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at http://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/07aa224a-3716-48c0-8af6-757d875c6d88%40googlegroups.com
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAKFtT_09-ERyrWR%2BukAn0EZ0dc1Kcru5K3h3JXett2GT%3D0fqRw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Feature request: ttl method for cache

[Russell Keith-Magee]

On Tue, May 6, 2014 at 4:16 AM, Piotr Gosławski
wrote:

> Hi!
>
> Since the contribution workflow is a bit confusing to me, I'll just leave
> it here.
> I think django's cache needs ttl (time to live) method that would return
> time left until specified key expires. I, for instance, needed one when
> writing
> a rate limiter. Without it I would have to store twice as much key-value
> pairs
> in cache. I ended up with a workaround that dynamically binds ttl method,
> but only for locmem and redis, since those are the backends I use.
>
> Here is my workoround, maybe it will be of some use:
> https://github.com/p-tr0/snipets/blob/master/django_cache_ttl.py
>
> I guess the proper way would be to add a method that just raises
> NotImplementedError and then cover it one backend at a time.
>

Hi Piotr,

Sounds like a reasonable proposal to me. Feel free to open a ticket, and
maybe try your hand at writing a patch!

As far as the NotImplemented bit goes - I'd rather see this implemented for
all officially supported backends, rather than only one backend. The only
exception to this would be if there is a fundamental technical reason why
TTL can't be implemented (or adequately faked).

Yours
Russ Magee %-)

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAJxq84__RVS7gNC3bXnCTM6689XjMygZn%3D-%2BU00Gka0fr0ErMQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Feature request: ttl method for cache

[Piotr Gosławski]

Hi!

Since the contribution workflow is a bit confusing to me, I'll just leave 
it here.
I think django's cache needs ttl (time to live) method that would return
time left until specified key expires. I, for instance, needed one when 
writing
a rate limiter. Without it I would have to store twice as much key-value 
pairs
in cache. I ended up with a workaround that dynamically binds ttl method,
but only for locmem and redis, since those are the backends I use.

Here is my workoround, maybe it will be of some use:
https://github.com/p-tr0/snipets/blob/master/django_cache_ttl.py

I guess the proper way would be to add a method that just raises
NotImplementedError and then cover it one backend at a time.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/07aa224a-3716-48c0-8af6-757d875c6d88%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.