Re: Issue 142 in memcached: Memcached 1.4.5 using 1.5G at boot time
Comment #3 on issue 142 by ferranbonas: Memcached 1.4.5 using 1.5G at boot time http://code.google.com/p/memcached/issues/detail?id=142 The other machines where I've got memcached 1.4.5 running are: CentOS release 5.4 (Final) libevent-1.1a-3.2.1 libevent-devel-1.1a-3.2.1
Re: Issue 142 in memcached: Memcached 1.4.5 using 1.5G at boot time
Comment #4 on issue 142 by dsallings: Memcached 1.4.5 using 1.5G at boot time http://code.google.com/p/memcached/issues/detail?id=142 Not much is expected to behave properly on libevent-1.1. Can you upgrade libevent and try your bug again? No reason to believe this is related, but even if we were to figure out what was wrong with this version and it *wasn't* libevent, you'd have bad luck running in production with this version of libevent.
Suggestions for deferring DB write using memcached
We're building a system with heavy real-time write volume and looking for a way to decouple db writes from the user request path. We're exploring the approach of buffering updated entities in memcached and writing them back to the database asynchronously. The primary problem that we're concerned about is how to ensure that the entity remains in the cache until the background process has a change to write it. Any advice and/or references would be greatly appreciated.
Re: Suggestions for deferring DB write using memcached
We're building a system with heavy real-time write volume and looking for a way to decouple db writes from the user request path. We're exploring the approach of buffering updated entities in memcached and writing them back to the database asynchronously. The primary problem that we're concerned about is how to ensure that the entity remains in the cache until the background process has a change to write it. Any advice and/or references would be greatly appreciated. You really probably want to use a job server... http://gearman.org/ - write updates async to gearman and update caches in memcached if you want, then have workers write them back to the DB as fast as your system can handle.
Re: Suggestions for deferring DB write using memcached
On Thu, Jun 3, 2010 at 3:06 PM, ehalpern eric.halp...@gmail.com wrote: We're building a system with heavy real-time write volume and looking for a way to decouple db writes from the user request path. We're exploring the approach of buffering updated entities in memcached and writing them back to the database asynchronously. The primary problem that we're concerned about is how to ensure that the entity remains in the cache until the background process has a change to write it. Any advice and/or references would be greatly appreciated. You want a job/work queue for this or just a simple queue service. I've built something that does this for us using Kestrel, a very simple, fast, persistent queue from Twitter that speaks memcached. http://github.com/robey/kestrel http://github.com/robey/kestrelYou can obviously make it as simple or complex as you need, but I'd say it's best to start out as simple as possible. Since kestrel is fast and persistent, you should just be able to write your data to it and move on, with the assumption that a background service will pop items off of kestrel's queue and write them to the database as quickly as possible. If you need the data to be immediately available, you can do something more complex like updating your cache at the same time, so that subsequent reads will get the new data as long as it's still available in the cache. -- awl
Re: Suggestions for deferring DB write using memcached
+1 We use gearman for this. Brian. http://brian.moonspot.net/ On 6/3/10 3:13 PM, dormando wrote: We're building a system with heavy real-time write volume and looking for a way to decouple db writes from the user request path. We're exploring the approach of buffering updated entities in memcached and writing them back to the database asynchronously. The primary problem that we're concerned about is how to ensure that the entity remains in the cache until the background process has a change to write it. Any advice and/or references would be greatly appreciated. You really probably want to use a job server... http://gearman.org/ - write updates async to gearman and update caches in memcached if you want, then have workers write them back to the DB as fast as your system can handle.
Re: Suggestions for deferring DB write using memcached
Oh, and I wrote about one application that does this. http://brian.moonspot.net/logging-with-mysql Brian. http://brian.moonspot.net/ On 6/3/10 3:13 PM, dormando wrote: We're building a system with heavy real-time write volume and looking for a way to decouple db writes from the user request path. We're exploring the approach of buffering updated entities in memcached and writing them back to the database asynchronously. The primary problem that we're concerned about is how to ensure that the entity remains in the cache until the background process has a change to write it. Any advice and/or references would be greatly appreciated. You really probably want to use a job server... http://gearman.org/ - write updates async to gearman and update caches in memcached if you want, then have workers write them back to the DB as fast as your system can handle.
Re: Suggestions for deferring DB write using memcached
Others have suggested their favored tools. I would add beanstalkd to the list http://kr.github.com/beanstalkd/ I like it since it comes with simple (memcached inspired) protocol and can persist jobs. Vladimir On Thu, 3 Jun 2010, ehalpern wrote: We're building a system with heavy real-time write volume and looking for a way to decouple db writes from the user request path. We're exploring the approach of buffering updated entities in memcached and writing them back to the database asynchronously. The primary problem that we're concerned about is how to ensure that the entity remains in the cache until the background process has a change to write it. Any advice and/or references would be greatly appreciated.
