[Olivier Langlois]
> ...
> I have kept thinking about the original problem and I now believe that
> the only solution if he wants to store 3.6GB of data in a Python script
> is to recompile Python in 64 bits. I do not know if this is something
> that someone has already done successfully...
I did
An alternative is to use ZODB. For example, you could use the BTree
class for the outermost layers of the nested dict, and a regular dict
for the innermost layer. If broken up properly, you can store
apparently unlimited amount of data with reasonable performance.
Just remember not to iterate ove
://www.quazal.com
> -Original Message-
> From: [EMAIL PROTECTED]
[mailto:python-
> [EMAIL PROTECTED] On Behalf Of Burton
Samograd
> Sent: April 11, 2006 2:10 PM
> To: python-list@python.org
> Subject: Re: Memory limit to dict?
>
> Peter Beattie <[EMAIL PROTECTED]> writ
That is a good observation!
Considering that hash tables to have good performance need to have a
fill rate of 70% or lower means that if you try to fit 3.6 GB of data
into a dict, your memory requirement will be much higher than 3.6GB.
BTW, this brings a new question. How does Python controls dic
Peter Beattie <[EMAIL PROTECTED]> writes:
> I've no idea how far the Windows task manager's resource monitor can be
> trusted -- probably not as far as I could throw a heavy-set St Bernard
> --, but it seems to stop roughly when that monitor records a swap file
> size of 2.2 GB.
Not being a wind
Em Ter, 2006-04-11 às 19:45 +0200, Peter Beattie escreveu:
> I was wondering whether certain data structures in Python, e.g. dict,
> might have limits as to the amount of memory they're allowed to take up.
> Is there any documentation on that?
>
> Why am I asking? I'm reading 3.6 GB worth of BLAST
Just out of curiosity. What is the OS that you are using? I am asking
because I do not know about XP 64bits edition but if you are using
Windows XP 32 bits version, no matter how much RAM you have on your
machine, the OS has an inherent 4GB address space limit per process.
Olivier Langlois
http://
I was wondering whether certain data structures in Python, e.g. dict,
might have limits as to the amount of memory they're allowed to take up.
Is there any documentation on that?
Why am I asking? I'm reading 3.6 GB worth of BLAST output files into a
nested dictionary structure (dict within dict ..
