On May 21, 10:07 am, punkish <punk.k...@gmail.com> wrote: > I consistently get results such as above, while I expected the > memcache to slowly fill up and speed up the queries way faster than > only accessing the file based db. Whatever I am doing, it is far > faster to open up the SQLite database every time and query it than it > is to query the memory cached value. (I am creating a $dbh and $sth > every time, and also opening a file and writing to it every time to > kinda emulate a CGI process that starts afresh on every hit of the > browser.
Minor nit: You're not opening the file every time. > What am I doing wrong, or are my expectations wrong? The thing you're doing wrong is taking something very cheap (i.e. not performance critical) and trying to make it faster with memcached. First note that SQLite is quite fast. Any DB whose entire contents can fit into your filesystem cache will return primary key lookups around as fast as memcached. In this case, the entire database is in memory in perl's process space and you're doing non-concurrent access to it and memcached and comparing speeds. SQLite has an advantage here and is not performance critical to your app. For this exact app, you're better off storing the data in a perl-based in-memory cache (hash table with some eviction logic). I've deployed memcached over SQLite in a twisted app that had lots of outstanding read and write queries to be processed through a separate thread. Before sending them off to that thread, I could fire many concurrent requests to memcached and avoid the need to hit the scarce resource at all as concurrency is not an issue there. While it might be slower for what I'm doing, it could do it all in parallel. The basic problem with benchmarking is that it often doesn't start with a problem. In this case, I think you've simplified the problem to the level where you can't really see the benefits. Had this been an actual application, for example, you might want to get 10 keys at a time. That would give you a 10x increase in throughput. You may also have more than one thread/DB doing the work, or your DB is larger than can fit in RAM in which case you can spread the RAM across many machines, workers across many machines/processes/threads/events.
