Re: Questions about bsddb
Thanks for the suggestion, I do remember reading that, but I don't think that helped much. I found experimenting around with the different settings, that the cache size is where the problem was. I've got it set to 1.5 GB and it's pretty happy at the moment, and the reduction in build time is a fraction of what it used to be. Thanks again for all the suggestions. Regards, JM -- http://mail.python.org/mailman/listinfo/python-list
Re: Questions about bsddb
Thanks for the info Nick. I plan on accessing the data in pretty much random order, and once the database is built, it will be read only. At this point Im not too concerned about access times, just getting something to work. I've been messing around with both bt and hash with limited success, which led me to think that maybe I was going beyond some internal limit for the data size.It works great on a limited set of data, but once I turn it loose on the full set, usually several hours later, it either causes a hard reset of my machine or the HD grinds on endlessly with no apparent progress. Is there a limit to the size of data you can place per key? Thanks for the MySQL suggestion, I'll take a look. -JM -- http://mail.python.org/mailman/listinfo/python-list
Questions about bsddb
Hello, I need to build a large database that has roughly 500,000 keys, and a variable amount of data for each key. The data for each key could range from 100 bytes to megabytes.The data under each will grow with time as the database is being built. Are there some flags I should be setting when opening the database to handle large amounts of data per key? Is hash or binary tree recommended for this type of job, I'll be building the database from scratch, so lots of lookups and appending of data. Testing is showing bt to be faster, so I'm leaning towards that. The estimated build time is around 10~12 hours on my machine, so I want to make sure that something won't get messed up in the 10th hour. TIA, JM -- http://mail.python.org/mailman/listinfo/python-list
