On 5/2/2012 10:14 PM, Steve Howell wrote:
This is slightly off topic, but I'm hoping folks can point me in the right direction.I'm looking for a fairly lightweight key/value store that works for this type of problem: ideally plays nice with the Python ecosystem the data set is static, and written infrequently enough that I definitely want *read* performance to trump all there is too much data to keep it all in memory (so no memcache) users will access keys with fairly uniform, random probability the key/value pairs are fairly homogenous in nature: keys are<= 16 chars values are between 1k and 4k bytes generally approx 3 million key/value pairs total amount of data == 6Gb needs to work on relatively recent versions of FreeBSD and Linux
On my 64bit machine with 64 bit Python, I would consider putting all the values in one data file and creating a key:file-offset dict. Each value would start with length(value) so that is not needed in memory. The dict, once created, could be pickled and unpickled for each run.
-- Terry Jan Reedy -- http://mail.python.org/mailman/listinfo/python-list
