Simon Strobl a écrit :
(snip)
I would prefer to be able to use the same type of
scripts with data of all sizes, though.
Since computers have a limited RAM, this is to remain a wish. You can't
obviously expect to deal with terabytes of data like you do with a 1kb
text file.
--
Bruno Desthuilliers wrote:
Simon Strobl a écrit :
(snip)
I would prefer to be able to use the same type of
scripts with data of all sizes, though.
Since computers have a limited RAM, this is to remain a wish. You
can't obviously expect to deal with terabytes of data like you do with
a 1kb
Have you considered that the operating system imposes per-process limits
on memory usage? You say that your server has 128 GB of memory, but that
doesn't mean the OS will make anything like that available.
According to our system administrator, I can use all of the 128G.
I thought it would
On Tue, 05 Aug 2008 01:20:08 -0700, Simon Strobl wrote:
I thought it would be practical not to create the dictionary from a
text file each time I needed it. I.e. I thought loading the .pyc-file
should be faster. Yet, Python failed to create a .pyc-file
Probably a good example of
Simon Strobl [EMAIL PROTECTED] wrote:
Well, as I was using Python, I did not expect to have to care about
the language's internal affairs that much. I thought I could simply do
always the same no matter how large my files get. In other words, I
thought Python was really scalable.
It's not
En Mon, 04 Aug 2008 11:02:16 -0300, Simon Strobl [EMAIL PROTECTED]
escribió:
I created a python file that contained the dictionary. The size of
this file was 6.8GB. I thought it would be practical not to create the
dictionary from a text file each time I needed it. I.e. I thought
loading the
Simon Strobl wrote:
Well, as I was using Python, I did not expect to have to care about
the language's internal affairs that much. I thought I could simply do
always the same no matter how large my files get. In other words, I
thought Python was really scalable.
Python the language is
On 4 Aug., 00:51, Avinash Vora [EMAIL PROTECTED] wrote:
On Aug 4, 2008, at 4:12 AM, Jörgen Grahn wrote:
(You might want to post this to comp.lang.python rather than to me --
I am just another c.l.p reader. If you already have done to, please
disregard this.)
Yeah, I hit reply by mistake
On Mon, 04 Aug 2008 07:02:16 -0700, Simon Strobl wrote:
I created a python file that contained the dictionary. The size of this
file was 6.8GB.
Ah, that's what I thought you had done. That's not a dictionary. That's a
text file containing the Python code to create a dictionary.
My guess is
On Fri, 1 Aug 2008 01:05:07 -0700 (PDT), Simon Strobl [EMAIL PROTECTED] wrote:
What does load a dictionary mean?
I had a file bigrams.py with a content like below:
bigrams = {
, djy : 75 ,
, djz : 57 ,
, djzoom : 165 ,
, dk : 28893 ,
, dk.au : 854 ,
, dk.b. : 3668 ,
...
}
In
On 3 Aug 2008 20:36:33 GMT, Jorgen Grahn [EMAIL PROTECTED] wrote:
On Fri, 1 Aug 2008 01:05:07 -0700 (PDT), Simon Strobl [EMAIL PROTECTED]
wrote:
...
If there is no other way to do it, I will have to learn how to use
databases in Python.
If you use Berkeley DB (import bsddb), you don't have
On 3 Aug 2008 20:40:02 GMT, Jorgen Grahn [EMAIL PROTECTED] wrote:
On 3 Aug 2008 20:36:33 GMT, Jorgen Grahn [EMAIL PROTECTED] wrote:
On Fri, 1 Aug 2008 01:05:07 -0700 (PDT), Simon Strobl [EMAIL PROTECTED]
wrote:
...
If there is no other way to do it, I will have to learn how to use
On Aug 4, 2008, at 4:12 AM, Jörgen Grahn wrote:
(You might want to post this to comp.lang.python rather than to me --
I am just another c.l.p reader. If you already have done to, please
disregard this.)
Yeah, I hit reply by mistake and didn't realize it. My bad.
(I assume here that
On Fri, 01 Aug 2008 00:46:09 -0700, Simon Strobl wrote:
Hello,
I tried to load a 6.8G large dictionary on a server that has 128G of
memory. I got a memory error. I used Python 2.5.2. How can I load my
data?
How do you know the dictionary takes 6.8G?
I'm going to guess an answer to my own
Hello,
I tried to load a 6.8G large dictionary on a server that has 128G of
memory. I got a memory error. I used Python 2.5.2. How can I load my
data?
SImon
--
http://mail.python.org/mailman/listinfo/python-list
On Fri, 01 Aug 2008 00:46:09 -0700, Simon Strobl wrote:
I tried to load a 6.8G large dictionary on a server that has 128G of
memory. I got a memory error. I used Python 2.5.2. How can I load my
data?
What does load a dictionary mean? Was it saved with the `pickle`
module?
How about using a
What does load a dictionary mean?
I had a file bigrams.py with a content like below:
bigrams = {
, djy : 75 ,
, djz : 57 ,
, djzoom : 165 ,
, dk : 28893 ,
, dk.au : 854 ,
, dk.b. : 3668 ,
...
}
In another file I said:
from bigrams import bigrams
How about using a database instead of a
Simon Strobl:
I had a file bigrams.py with a content like below:
bigrams = {
, djy : 75 ,
, djz : 57 ,
, djzoom : 165 ,
, dk : 28893 ,
, dk.au : 854 ,
, dk.b. : 3668 ,
...
}
In another file I said:
from bigrams import bigrams
Probably there's a limit in the module size here. You can
Simon Strobl [EMAIL PROTECTED] wrote:
I tried to load a 6.8G large dictionary on a server that has 128G of
memory. I got a memory error. I used Python 2.5.2. How can I load my
data?
Let's just eliminate one thing here: this server is running a
64-bit OS, isn't it? Because if it's a 32-bit OS,
On Fri, 01 Aug 2008 14:47:17 +0100, Sion Arrowsmith wrote:
Simon Strobl [EMAIL PROTECTED] wrote:
I tried to load a 6.8G large dictionary on a server that has 128G of
memory. I got a memory error. I used Python 2.5.2. How can I load my
data?
Let's just eliminate one thing here: this server is
On Fri, 01 Aug 2008 14:47:17 +0100, Sion Arrowsmith wrote:
Simon Strobl [EMAIL PROTECTED] wrote:
I tried to load a 6.8G large dictionary on a server that has 128G of
memory. I got a memory error. I used Python 2.5.2. How can I load my
data?
Let's just eliminate one thing here: this server is
Simon Strobl wrote:
Hello,
I tried to load a 6.8G large dictionary on a server that has 128G of
memory. I got a memory error. I used Python 2.5.2. How can I load my
data?
SImon
Take a look at the python bsddb module. Uing btree tables is fast, and
it has the benefit that once the table is
22 matches
Mail list logo
