Really a database is the way to go. Any modern database should allow you to check if a value is in there before inserting, so the same tweet won't go in there twice. Additionally, not every user search has to use the up to the minute results. They can go back just a little in time (30 seconds or so) to have batches in case there are multiple people searching for the same thing.
On Oct 2, 7:36 am, Bjoern <bjoer...@googlemail.com> wrote: > Hi, > > just wondering about a best practice thing. Suppose I show results of > specific Twitter searches on a web site. How would I go about caching > the searches? > > The naive approach seems to be to first check in my own database, then > do a twitter search with the since_id parameter to only get results I > don't already have. Then store the results from twitter in the > database, too, and return the merged results to the web site. > > The problem I see is that if multiple user run the same search on my > web site, threading issues might occur (as each user starts a separate > thread on my server). Not only could multiple twitter searches with > the same since_id be executed (maybe forgivable), but trouble starts > when said results are to be inserted in my local database. Different > threads could attempt to insert the same messages into my database. > > One simple solution I could imagine: just use the message ids from > twitter as the primary key in my local database. That way, multiple > threads saving the same message would just overwrite the message with > itself. I actually wonder if that is a common solution - to use the > twitter ids as primary keys (also for users, direct messages...). I > have kind of arrived at the opinion that this would be the way of > least resistance, although I feel a bit uneasy about it. > > An alternative that came to my mind might be to have single threaded > background jobs do the copying of the search results from twitter to > my database, and only show the results from my cache to the web site. > This would cause some lag in the time the search results would appear, > but it would not be too bad. However, if I have a lot of different > searches, it would become infeasible to update all of them > periodically. It would become necessary to only trigger an update when > a user does the search. At that point things might get overly > complicated: presumably I would need some kind of Ajax solution to > trigger the "caching" with the first request, show a spinner while the > updating of my local db is going on, and then show the results from my > local cache/db. The trickiest part being to prevent the starting of > multiple update tasks for the same search. > > All in all the simple solution might be the better way to go? > > Would be interested in hearing your opinions, experiences and > solutions! > > Thanks! > > Björn
