Re: Memory Problems in Windows 2003 Server
Thanks Marc, I just tried shelve but it is very slow :( I haven't tried the dbs yet. Andre Marc 'BlackJack' Rintsch a écrit : > On Mon, 15 Oct 2007 11:31:59 +0200, amdescombes wrote: > >> Are there any classes that implement disk based dictionaries? > > Take a look at the `shelve` module from the standard library. > > Or object databases like ZODB or Durus. > > Ciao, > Marc 'BlackJack' Rintsch -- http://mail.python.org/mailman/listinfo/python-list
Re: Memory Problems in Windows 2003 Server
On Mon, 15 Oct 2007 11:31:59 +0200, amdescombes wrote: > Are there any classes that implement disk based dictionaries? Take a look at the `shelve` module from the standard library. Or object databases like ZODB or Durus. Ciao, Marc 'BlackJack' Rintsch -- http://mail.python.org/mailman/listinfo/python-list
Re: Memory Problems in Windows 2003 Server
Yes, I think that might be the issue, perhaps I could implement the solution using several dictionaries instead of just one. Are there any classes that implement disk based dictionaries? Thanks, Andre > > I don't know whether Python dictionaries must live in a contiguous piece of > memory, but if so, that could be the issue. The system DLLs in Server 2003 > have been "rebased" in such a way that they chop up the virtual address > space more than XP. Even though there is more virtual memory available, it > is fragmented. -- http://mail.python.org/mailman/listinfo/python-list
Re: Memory Problems in Windows 2003 Server
AMD <[EMAIL PROTECTED]> wrote: > >I do the reading one line at a time, the problem seems to be with the >dictionary I am creating. I don't know whether Python dictionaries must live in a contiguous piece of memory, but if so, that could be the issue. The system DLLs in Server 2003 have been "rebased" in such a way that they chop up the virtual address space more than XP. Even though there is more virtual memory available, it is fragmented. -- Tim Roberts, [EMAIL PROTECTED] Providenza & Boekelheide, Inc. -- http://mail.python.org/mailman/listinfo/python-list
Re: Memory Problems in Windows 2003 Server
Hi Brad, I do the reading one line at a time, the problem seems to be with the dictionary I am creating. Andre > amdescombes wrote: >> Hi, >> >> I am using Python 2.5.1 >> I have an application that reads a file and generates a key in a >> dictionary for each line it reads. I have managed to read a 1GB file >> and generate more than 8 million keys on an Windows XP machine with >> only 1GB of memory and all works as expected. When I use the same >> program on a Windows 2003 Server with 2GB of RAM I start getting >> MemoryError exceptions! >> I have tried setting the IMAGE_FILE_LARGE_ADDRESS_AWARE on both >> Python.exe and Python25.dll and setting the /3GB flag on the boot.ini >> file to no avail. I still get the MemoryError exceptions. >> >> Has anybody encountered this problem before? >> >> Thanks in advance for any ideas/suggestions. >> >> Best Regards, >> >> André M. Descombes > > I forgot to mention that the OS itself or other processes may be using a > lot of memory. So, just because you have 2GB, that does not mean you can > access all of that at once. I would guess that 25% of memory is in > constant use by the OS. So, do your IO/reads in smaller chunks similar > to the example I gave earlier. > > Brad -- http://mail.python.org/mailman/listinfo/python-list
Re: Memory Problems in Windows 2003 Server
amdescombes wrote: > Hi, > > I am using Python 2.5.1 > I have an application that reads a file and generates a key in a > dictionary for each line it reads. I have managed to read a 1GB file and > generate more than 8 million keys on an Windows XP machine with only 1GB > of memory and all works as expected. When I use the same program on a > Windows 2003 Server with 2GB of RAM I start getting MemoryError exceptions! > I have tried setting the IMAGE_FILE_LARGE_ADDRESS_AWARE on both > Python.exe and Python25.dll and setting the /3GB flag on the boot.ini > file to no avail. I still get the MemoryError exceptions. > > Has anybody encountered this problem before? > > Thanks in advance for any ideas/suggestions. > > Best Regards, > > André M. Descombes I forgot to mention that the OS itself or other processes may be using a lot of memory. So, just because you have 2GB, that does not mean you can access all of that at once. I would guess that 25% of memory is in constant use by the OS. So, do your IO/reads in smaller chunks similar to the example I gave earlier. Brad -- http://mail.python.org/mailman/listinfo/python-list
Re: Memory Problems in Windows 2003 Server
amdescombes wrote: > Hi, > > I am using Python 2.5.1 > I have an application that reads a file and generates a key in a > dictionary for each line it reads. I have managed to read a 1GB file and > generate more than 8 million keys on an Windows XP machine with only 1GB > of memory and all works as expected. When I use the same program on a > Windows 2003 Server with 2GB of RAM I start getting MemoryError exceptions! > I have tried setting the IMAGE_FILE_LARGE_ADDRESS_AWARE on both > Python.exe and Python25.dll and setting the /3GB flag on the boot.ini > file to no avail. I still get the MemoryError exceptions. > > Has anybody encountered this problem before? > > Thanks in advance for any ideas/suggestions. > > Best Regards, > > André M. Descombes How are you reading the large files? IMO, large files are better read in chunks: target_file = open(f, 'rb') while 1: data = target_file.read(8192000) if data: DO SOMETHING else: break The above reads 8MB at a time until the file has been completely read. Change the 8MB to whatever you like. -- http://mail.python.org/mailman/listinfo/python-list
Memory Problems in Windows 2003 Server
Hi, I am using Python 2.5.1 I have an application that reads a file and generates a key in a dictionary for each line it reads. I have managed to read a 1GB file and generate more than 8 million keys on an Windows XP machine with only 1GB of memory and all works as expected. When I use the same program on a Windows 2003 Server with 2GB of RAM I start getting MemoryError exceptions! I have tried setting the IMAGE_FILE_LARGE_ADDRESS_AWARE on both Python.exe and Python25.dll and setting the /3GB flag on the boot.ini file to no avail. I still get the MemoryError exceptions. Has anybody encountered this problem before? Thanks in advance for any ideas/suggestions. Best Regards, André M. Descombes -- http://mail.python.org/mailman/listinfo/python-list
