Re: Large Dictionaries

Claudio Grondi wrote: > Chris Foote wrote: >> Klaas wrote: >> >>>> 22.2s 20m25s[3] >>> >>> 20m to insert 1m keys? You are doing something wrong. >> >> I've put together some simplified test code, but the bsddb >> module gives 1

Re: Large Dictionaries

Richie Hindle wrote: > [Chris] >> Has anyone written a fast hash module which is more optimal for >> large datasets ? > > PyJudy might be what you're looking for, though I've never used it: > > http://www.dalkescientific.com/Python/PyJudy.html > > "Judy's key benefits are scalability, high per

Re: Large Dictionaries

Klaas wrote: 22.2s 20m25s[3] 20m to insert 1m keys? You are doing something wrong. Hi Mike. I've put together some simplified test code, but the bsddb module gives 11m for 1M keys: Number generator test for 100 number ranges with a maximum of 3 wildcard digits. Wed May 17 22:1

Re: Large Dictionaries

Claudio Grondi wrote: > Chris Foote wrote: > >> However, please note that the Python bsddb module doesn't support >> in-memory based databases - note the library documentation's[1] wording: >> >> "Files never intended to be preserved on disk m

Re: Large Dictionaries

lcaamano wrote: > Sounds like PyTables could be useful. > > http://www.pytables.org In browsing their excellent documentation, it seems that all concepts are built around storing and reading HDF5 format files. Not suitable for this project unfortunately. Cheers, Chris -- http://mail.python.o

Re: Large Dictionaries

Paul McGuire wrote: > "Claudio Grondi" <[EMAIL PROTECTED]> wrote in message > news:[EMAIL PROTECTED] >> Chris Foote wrote: >>> Hi all. >>> >>> I have the need to store a large (10M) number of keys in a hash table, >>> based on a tuple

Re: Large Dictionaries

Claudio Grondi wrote: > Chris Foote wrote: >> p.s. Disk-based DBs are out of the question because most >> key lookups will result in a miss, and lookup time is >> critical for this application. >> > Python Bindings (\Python24\Lib\bsddb vers. 4.3.0) and the DLL for

Re: Large Dictionaries

Aahz wrote: > In article <[EMAIL PROTECTED]>, > Roy Smith <[EMAIL PROTECTED]> wrote: >> In article <[EMAIL PROTECTED]>, Chris Foote <[EMAIL PROTECTED]> >> wrote: >>> I have the need to store a large (10M) number of keys in a hash table, &g

Re: Large Dictionaries

Roy Smith wrote: > In article <[EMAIL PROTECTED]>, Chris Foote <[EMAIL PROTECTED]> > wrote: > >> I have the need to store a large (10M) number of keys in a hash table, >> based on a tuple of (long_integer, integer). The standard python >> dictionary wo

Large Dictionaries

Hi all. I have the need to store a large (10M) number of keys in a hash table, based on a tuple of (long_integer, integer). The standard python dictionary works well for small numbers of keys, but starts to perform badly for me inserting roughly 5M keys: # keys dictionary metakit (both usin