Wouldn't it make more sense to shift the caching to an image server that
directly serves all the files in the filestore ?
This will also reduce the load on the php application significantly.
From Will's and Amanda's notes, it appears that they are deploying the
rails app in this manner - using nginx to serve the files and forwarding
all other requests to the thins in the back.
Ideally, we would like a distributed filestore - otherwise,sooner or
later we're going to run into a bottleneck on the image server as well.
But at least this is a good first step to take.
Comments ?
Shanti
On 01/28/09 09:18 AM, Akara Sucharitakul wrote:
As we run into filesystem bottlenecks, it makes sense to reduce the
filesystem access by caching thumbnail images in memcached. The
thumbnails are about 6-7 KB in size. We want to ensure there's a hot set
of thumbnails being cached in memcached but not that every image gets
cached. So here's my first draft on how to achieve that:
1. All thumbnails that are accessed by the client and driver are cached
in memcached with a timeout of 90 seconds. Of course, we can still
adjust this. Any access to the thumbnails will check the cache at first.
2. In order to keep the hot set low, we will modify our tag search query
to order by event date with the most recent upcoming events first. As a
result, the set of events searched by each tag are more deterministic
and will not often change. Since the tag search only shows 10 events per
page, the thumbnails for these top ten events for each tag will be held
in memcached.
The mathematical model for this is a little complicated, though. Since
we're dealing with non-uniform distribution of tags and a cap of 10
upcoming events on top of that. But please let me know whether you agree
with this conceptually.
Thanks,
-Akara
