Re: memcached + fcgi prefork = wrong value for a given key

2012-08-13 Thread v-tuzov
пятница, 11 февраля 2011 г., 13:52:01 UTC+4 пользователь MarcMarc написал:
> Hi,
> 
> 
> 
> My problem looks like this: when using memcached with fcgi on prefork
> 
> mode I`m getting wrong values for a given keys. Values are
> 
> dictionaries with keys, so I`m logging those erros like this:
> 
> 
> 
> Memcached error: shops_categories.get_count.5942 -> {'version': None,
> 
> 'value': 1328, 'key': 'shops_categories.get_count.5935', 'vkey':
> 
> None}
> 
> 
> 
> As shown above requested key was
> 
> shops_categories.get_available_items_count.5942 and fetched value is
> 
> wrong, key for given value is
> 
> shops_categories.get_available_items_count.5935. Strange. This log is
> 
> generated by custom method 'get', which contains no magic, its just a
> 
> wrapper over django.core.cache.cache.get to log a key that was asked
> 
> for and value that django.core.cache.cache.get fetched.
> 
> 
> 
> There are some clues about this strange behaviour:
> 
> 
> 
> - problem seems to be related to start/stop of the fcgi child process.
> 
> When testing with 'ab -n 1000 -c 10' and runfcgi maxchildren=5 and
> 
> maxrequests parameter set to 10 those errors are visible and almost
> 
> constant, but set maxrequests to, let`s say, 100 000 and there are
> 
> none of them.
> 
> 
> 
> - most of the time mixed up values are similar, but this is not a
> 
> rule. For instance:
> 
> 
> 
> 11-02-11 09:50 aktywni.cache.utilsERRORutils:  23
> 
> Memcached error: shops_categories.get_category_descendants.5650 ->
> 
> {'version': None, 'value': 122, 'key':
> 
> 'shops_categories.get_available_items_count.5650', 'vkey': None}
> 
> 
> 
> 11-02-11 09:50 aktywni.cache.utilsERRORutils:  23
> 
> Memcached error: shops_categories.get_available_items_count.5650 ->
> 
> {'version': None, 'value': 1328, 'key':
> 
> 'shops_categories.get_available_items_count.5935', 'vkey': None}
> 
> 
> 
> 11-02-11 09:50 aktywni.cache.utilsERRORutils:  23
> 
> Memcached error: shops_categories.get_category_descendants.5948 ->
> 
> {'version': None, 'value': 0, 'key':
> 
> 'shops_categories.get_available_items_count.5795', 'vkey': None}
> 
> 
> 
> - single view generates around 25-50 cache requests, or even more.
> 
> 
> 
> - when using get_multi tracebacks looks like this one:
> 
> 
> 
>  File "/var/www/htdocs/serwisy/aktywni.pl/lib/django/core/cache/
> 
> backends/memcached.py", line 60, in get_many
> 
>return self._cache.get_multi(map(smart_str,keys))
> 
> 
> 
>  File "/var/www/htdocs/serwisy/aktywni.pl/lib/memcache.py", line 738,
> 
> in get_multi
> 
>retvals[prefixed_to_orig_key[rkey]] = val   # un-prefix returned
> 
> key.
> 
> 
> 
> KeyError: 'shops_categories.get_lowest_categories'
> 
> 
> 
> - I`ve tested my site on threaded fcgi, and there is no such problem
> 
> in
> 
> 
> 
> Maybe there is some problem with connection to memcached left open
> 
> after fcgi process is killed after set amount of requests? Maybe for
> 
> some reason another request from next process is getting values from
> 
> the already killed one? Of course there is no evidence of failure in
> 
> memcached logs in verbose mode.
> 
> 
> 
> Feel free to ask more questions, this is my first apperance on this
> 
> group, I`ve tried to show every piece of evidence : )
> 
> 
> 
> Env summary:
> 
> Ubuntu 10.10 minimal (but there was the same problem on Gentoo)
> 
> memcached 1.4.5
> 
> python-memcached 1.43
> 
> django 1.2.1
> 
> nginx 0.7.67

Hello! So did you find the solution of this problem?
I have the same problem and it would be great if there is a real solutions.

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/django-users/-/uuOIOMHDLN0J.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.



memcached + fcgi prefork = wrong value for a given key

2011-02-11 Thread MarcMarc
Hi,

My problem looks like this: when using memcached with fcgi on prefork
mode I`m getting wrong values for a given keys. Values are
dictionaries with keys, so I`m logging those erros like this:

Memcached error: shops_categories.get_count.5942 -> {'version': None,
'value': 1328, 'key': 'shops_categories.get_count.5935', 'vkey':
None}

As shown above requested key was
shops_categories.get_available_items_count.5942 and fetched value is
wrong, key for given value is
shops_categories.get_available_items_count.5935. Strange. This log is
generated by custom method 'get', which contains no magic, its just a
wrapper over django.core.cache.cache.get to log a key that was asked
for and value that django.core.cache.cache.get fetched.

There are some clues about this strange behaviour:

- problem seems to be related to start/stop of the fcgi child process.
When testing with 'ab -n 1000 -c 10' and runfcgi maxchildren=5 and
maxrequests parameter set to 10 those errors are visible and almost
constant, but set maxrequests to, let`s say, 100 000 and there are
none of them.

- most of the time mixed up values are similar, but this is not a
rule. For instance:

11-02-11 09:50 aktywni.cache.utilsERRORutils:  23
Memcached error: shops_categories.get_category_descendants.5650 ->
{'version': None, 'value': 122, 'key':
'shops_categories.get_available_items_count.5650', 'vkey': None}

11-02-11 09:50 aktywni.cache.utilsERRORutils:  23
Memcached error: shops_categories.get_available_items_count.5650 ->
{'version': None, 'value': 1328, 'key':
'shops_categories.get_available_items_count.5935', 'vkey': None}

11-02-11 09:50 aktywni.cache.utilsERRORutils:  23
Memcached error: shops_categories.get_category_descendants.5948 ->
{'version': None, 'value': 0, 'key':
'shops_categories.get_available_items_count.5795', 'vkey': None}

- single view generates around 25-50 cache requests, or even more.

- when using get_multi tracebacks looks like this one:

 File "/var/www/htdocs/serwisy/aktywni.pl/lib/django/core/cache/
backends/memcached.py", line 60, in get_many
   return self._cache.get_multi(map(smart_str,keys))

 File "/var/www/htdocs/serwisy/aktywni.pl/lib/memcache.py", line 738,
in get_multi
   retvals[prefixed_to_orig_key[rkey]] = val   # un-prefix returned
key.

KeyError: 'shops_categories.get_lowest_categories'

- I`ve tested my site on threaded fcgi, and there is no such problem
in

Maybe there is some problem with connection to memcached left open
after fcgi process is killed after set amount of requests? Maybe for
some reason another request from next process is getting values from
the already killed one? Of course there is no evidence of failure in
memcached logs in verbose mode.

Feel free to ask more questions, this is my first apperance on this
group, I`ve tried to show every piece of evidence : )

Env summary:
Ubuntu 10.10 minimal (but there was the same problem on Gentoo)
memcached 1.4.5
python-memcached 1.43
django 1.2.1
nginx 0.7.67

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.