Maybe port to SQLite? I would not choose dbm these days. Barry
> On 27 Nov 2019, at 01:48, Tim Chase <python.l...@tim.thechases.com> wrote: > > Working with the dbm module (using it as a cache), I've gotten the > following error at least twice now: > > HASH: Out of overflow pages. Increase page size > Traceback (most recent call last): > [snip] > File ".py", line 83, in get_data > db[key] = data > _dbm.error: cannot add item to database > > I've read over the py3 docs on dbm > > https://docs.python.org/3/library/dbm.html > > but don't see anything about either "page" or "size" contained > therein. > > There's nothing particularly complex as far as I can tell. Nothing > more than a straightforward > > import dbm > with dbm.open("cache", "c") as db: > for thing in source: > key = extract_key_as_bytes(thing) > if key in db: > data = db[key] > else: > data = long_process(thing) > db[key] = data > > The keys can get a bit large (think roughly book-title length), but > not huge. I have 11k records so it seems like it shouldn't be > overwhelming, but this is the second batch where I've had to nuke the > cache and start afresh. Fortunately I've tooled the code so it can > work incrementally and no more than a hundred or so requests have to > be re-performed. > > How does one increas the page-size in a dbm mapping? Or are there > limits that I should be aware of? > > Thanks, > > -tkc > > PS: FWIW, this is Python 3.6 on FreeBSD in case that exposes any > germane implementation details. > > > > > > > > -- > https://mail.python.org/mailman/listinfo/python-list > -- https://mail.python.org/mailman/listinfo/python-list
