I have a theory about what's going on. The implementation of Redis in web2py stores the keys in "buckets" or sets, and the name of those sets are stored in cache *with no expiration time* accordingly to the code documentation <https://github.com/web2py/web2py/blob/master/gluon/contrib/redis_cache.py#L76> : "all buckets are indexed in a fixed key that never expires"
For what I see, this is how web2py stores keys and buckets for a given appname: 127.0.0.1:6379> SMEMBERS w2p:appname:___cache_set 1) "w2p:appname:___cache_set:26319660" 2) "w2p:appname:___cache_set:26377475" 3) "w2p:appname:___cache_set:26336873" 4) "w2p:appname:___cache_set:26318136" Every key set can be accessed with the same command, for example: 127.0.0.1:6379> SMEMBERS w2p:appname:___cache_set:26319660 1) "w2p:cache:appname:url_logo_movil" 2) "w2p:cache:appname:menu1" 3) "w2p:cache:appname:url_imagen_default" 4) "w2p:cache:appname:CT" 5) "w2p:cache:appname:url_imagen_logo_newsletter" At the other hand: - My Redis instance is configured to use no more than 20 GB of RAM, with an eviction policy of "*allkeys-lru*", that is evict keys lest recently used. - The *limit of 20gb has been reached* a few times in the past. Accordingly to the Redis docs <https://redis.io/topics/lru-cache#eviction-policies>, one alternative to allkeys-lru is "*volatile-lru*". This option only evict keys that *have an expiration time*. So I deduce that *my confiuguration (allkeys-lru) evics keys no matter they have or not an expiration time*. And considering web2py sets the bucket names with keys that never expire, *my theory is that some keys storing the name of the buckets were evicted*. That's why my code doesn't find some keys that are still indeed cached: that's because my code does the search from bucket names. That's my best guess.. Anyway, I'll try to adapt the code of my app that clears the cache of an application: I think I'll go for subprocess with a command like: $ redis-cli --scan --pattern w2p:cache:appname:* | xargs redis-cli del Any comment or suggestion will be much appreciated. Warm regards, Lisandro. El sábado, 29 de febrero de 2020, 20:19:29 (UTC-3), Lisandro escribió: > > I'm having some trouble with Redis and I thought some of you could > enlighten me :) > > I have a web2py instance running with several apps. > I have a Redis instance working as a cache, with maxmemory set to 20gb and > LRU eviction policy. > Every application connects to the same Redis instance. > > One of those apps is the main one, from wich I perform some administrative > tasks like: > - getting a list of the cache keys of a specific app > - clearing all the cache keys of a specific app. > > In order to get the list of the keys cached by a specific app I use this > custom code: > > > def get_cache_keys(application): > import re > > result = [] > regex = ':*' > prefix = 'w2p:%s' % application > cache_set = 'w2p:%s:___cache_set' % application > r = re.compile(regex) > buckets = redis_conn.smembers(cache_set) > if buckets: > keys = redis_conn.sunion(buckets) > for key in keys: > if r.match(str(key).replace(prefix, '', 1)): > result.append(key) > return result > > In the other hand, if I need to clear all the cached keys of a specific > app, I get the key list (using the above funciton) and then iterate those > keys calling redis_conn.delete(key) > I thought this was ok, but today I've found a problem: *I've found a key > that was generated by an app but it isn't returned by the code above*. > > To be sure, I connected to Redis from the terminal and in fact I was able > to get the key and its value, but running the code above the key isn't > listed. I'v checked the TTL of the key and it is still valid. > It's not the only case: I've found several cases. > > For example, I have a web2py app called "transmedia". > From redis-cli, I want to know the SETs (buckets) that web2py created to > store the keys of that specific app, so I run: > > 127.0.0.1:6379> SMEMBERS w2p:transmedia:___cache_set > 1) "w2p:transmedia:___cache_set:26383112" > 2) "w2p:transmedia:___cache_set:26384550" > 3) "w2p:transmedia:___cache_set:26383115" > 4) "w2p:transmedia:___cache_set:26383117" > 5) "w2p:transmedia:___cache_set:26383436" > 6) "w2p:transmedia:___cache_set:26383118" > 7) "w2p:transmedia:___cache_set:26383113" > 8) "w2p:transmedia:___cache_set:26383111" > 9) "w2p:transmedia:___cache_set:26383495" > 10) "w2p:transmedia:___cache_set:26383440" > 11) "w2p:transmedia:___cache_set:26383170" > 12) "w2p:transmedia:___cache_set:26383116" > > > Then I checked all those SETs to get all the keys stored by the app: > > 127.0.0.1:6379> SMEMBERS w2p:transmedia:___cache_set:26384550 > 1) "w2p:cache:transmedia:url_logo_movil" > 2) "w2p:cache:transmedia:menu1" > 3) "w2p:cache:transmedia:url_imagen_default" > 4) "w2p:cache:transmedia:CT" > 5) "w2p:cache:transmedia:url_imagen_logo_newsletter" > 6) "w2p:cache:transmedia:html_head" > 7) "w2p:cache:transmedia:menu0" > 8) "w2p:cache:transmedia:TEMPLATE" > 9) "w2p:cache:transmedia:url_fondo_personalizado" > 10) "w2p:cache:transmedia:url_favicon" > 11) "w2p:cache:transmedia:C" > 12) "w2p:cache:transmedia:CONFIG" > 13) "w2p:cache:transmedia:url_logo_grande" > 14) "w2p:cache:transmedia:html_body" > 15) "w2p:cache:transmedia:url_logo" > > There we can see *15 keys*. > All the other sets are "(empty list or set)". > > But here is the weird part: the application "transmedia" stored a key that > I don't see in the list: > *w2p:cache:transmedia:url_imagen_publicidad_newsletter* > Checking from redis-cli I can see that the key is still there and its TTL > is still valid: > > 127.0.0.1:6379> GET w2p:cache:transmedia:url_imagen_publicidad_newsletter > "\x80\x02U\x00." > 127.0.0.1:6379> TTL w2p:cache:transmedia:url_imagen_publicidad_newsletter > (integer) 18711 > > > I can confirm this discrepancy if I perform a SCAN like this: > > $ redis-cli --scan --pattern w2p:cache:transmedia:* > w2p:cache:transmedia:url_logo_movil > w2p:cache:transmedia:url_favicon > w2p:cache:transmedia:url_fondo_personalizado > w2p:cache:transmedia:TEMPLATE > w2p:cache:transmedia:menu1 > w2p:cache:transmedia:html_head > w2p:cache:transmedia:url_imagen_logo_newsletter > w2p:cache:transmedia:html_body > w2p:cache:transmedia:url_logo > w2p:cache:transmedia:CT > w2p:cache:transmedia:url_logo_grande > w2p:cache:transmedia:CONFIG > w2p:cache:transmedia:C > w2p:cache:transmedia:url_imagen_publicidad_newsletter > w2p:cache:transmedia:url_imagen_default > w2p:cache:transmedia:menu0 > > > Notice the output now includes the *16 keys*, not 15. > So why isn't the key listed when scanning all the SETS (buckets) that > web2py created for the app? Is there something wrong with my code? > > My main concern is about *clearing all the keys of a given app*. > Right now I can't trust that my code will delete all the keys, because the > function that gets the key list doesn't always include all the keys that > the app stored. > > So another question: *is there a better way to delete all the keys cached > by an app?* > I thought I could use redis-cli from the commandline like this: > > $ redis-cli --scan --pattern w2p:cache:transmedia:* | xargs redis-cli del > > > This would work, but in order to run it from within my main app I would > have to use subprocess. > What do you think? > -- Resources: - http://web2py.com - http://web2py.com/book (Documentation) - http://github.com/web2py/web2py (Source code) - https://code.google.com/p/web2py/issues/list (Report Issues) --- You received this message because you are subscribed to the Google Groups "web2py-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to web2py+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/web2py/af63e3c3-eb6a-4e58-8138-467da1bd0519%40googlegroups.com.
