Re: Memory Problems in Windows 2003 Server
Thanks Marc, I just tried shelve but it is very slow :( I haven't tried the dbs yet. Andre Marc 'BlackJack' Rintsch a écrit : On Mon, 15 Oct 2007 11:31:59 +0200, amdescombes wrote: Are there any classes that implement disk based dictionaries? Take a look at the `shelve` module from the standard library. Or object databases like ZODB or Durus. Ciao, Marc 'BlackJack' Rintsch -- http://mail.python.org/mailman/listinfo/python-list
Re: Memory Problems in Windows 2003 Server
Yes, I think that might be the issue, perhaps I could implement the solution using several dictionaries instead of just one. Are there any classes that implement disk based dictionaries? Thanks, Andre I don't know whether Python dictionaries must live in a contiguous piece of memory, but if so, that could be the issue. The system DLLs in Server 2003 have been rebased in such a way that they chop up the virtual address space more than XP. Even though there is more virtual memory available, it is fragmented. -- http://mail.python.org/mailman/listinfo/python-list
Re: Memory Problems in Windows 2003 Server
On Mon, 15 Oct 2007 11:31:59 +0200, amdescombes wrote: Are there any classes that implement disk based dictionaries? Take a look at the `shelve` module from the standard library. Or object databases like ZODB or Durus. Ciao, Marc 'BlackJack' Rintsch -- http://mail.python.org/mailman/listinfo/python-list
Re: Memory Problems in Windows 2003 Server
AMD [EMAIL PROTECTED] wrote: I do the reading one line at a time, the problem seems to be with the dictionary I am creating. I don't know whether Python dictionaries must live in a contiguous piece of memory, but if so, that could be the issue. The system DLLs in Server 2003 have been rebased in such a way that they chop up the virtual address space more than XP. Even though there is more virtual memory available, it is fragmented. -- Tim Roberts, [EMAIL PROTECTED] Providenza Boekelheide, Inc. -- http://mail.python.org/mailman/listinfo/python-list
Re: Memory Problems in Windows 2003 Server
Hi Brad, I do the reading one line at a time, the problem seems to be with the dictionary I am creating. Andre amdescombes wrote: Hi, I am using Python 2.5.1 I have an application that reads a file and generates a key in a dictionary for each line it reads. I have managed to read a 1GB file and generate more than 8 million keys on an Windows XP machine with only 1GB of memory and all works as expected. When I use the same program on a Windows 2003 Server with 2GB of RAM I start getting MemoryError exceptions! I have tried setting the IMAGE_FILE_LARGE_ADDRESS_AWARE on both Python.exe and Python25.dll and setting the /3GB flag on the boot.ini file to no avail. I still get the MemoryError exceptions. Has anybody encountered this problem before? Thanks in advance for any ideas/suggestions. Best Regards, André M. Descombes I forgot to mention that the OS itself or other processes may be using a lot of memory. So, just because you have 2GB, that does not mean you can access all of that at once. I would guess that 25% of memory is in constant use by the OS. So, do your IO/reads in smaller chunks similar to the example I gave earlier. Brad -- http://mail.python.org/mailman/listinfo/python-list
Memory Problems in Windows 2003 Server
Hi, I am using Python 2.5.1 I have an application that reads a file and generates a key in a dictionary for each line it reads. I have managed to read a 1GB file and generate more than 8 million keys on an Windows XP machine with only 1GB of memory and all works as expected. When I use the same program on a Windows 2003 Server with 2GB of RAM I start getting MemoryError exceptions! I have tried setting the IMAGE_FILE_LARGE_ADDRESS_AWARE on both Python.exe and Python25.dll and setting the /3GB flag on the boot.ini file to no avail. I still get the MemoryError exceptions. Has anybody encountered this problem before? Thanks in advance for any ideas/suggestions. Best Regards, André M. Descombes -- http://mail.python.org/mailman/listinfo/python-list
Re: Memory Problems in Windows 2003 Server
amdescombes wrote: Hi, I am using Python 2.5.1 I have an application that reads a file and generates a key in a dictionary for each line it reads. I have managed to read a 1GB file and generate more than 8 million keys on an Windows XP machine with only 1GB of memory and all works as expected. When I use the same program on a Windows 2003 Server with 2GB of RAM I start getting MemoryError exceptions! I have tried setting the IMAGE_FILE_LARGE_ADDRESS_AWARE on both Python.exe and Python25.dll and setting the /3GB flag on the boot.ini file to no avail. I still get the MemoryError exceptions. Has anybody encountered this problem before? Thanks in advance for any ideas/suggestions. Best Regards, André M. Descombes I forgot to mention that the OS itself or other processes may be using a lot of memory. So, just because you have 2GB, that does not mean you can access all of that at once. I would guess that 25% of memory is in constant use by the OS. So, do your IO/reads in smaller chunks similar to the example I gave earlier. Brad -- http://mail.python.org/mailman/listinfo/python-list
Re: Memory Problems in Windows 2003 Server
amdescombes wrote: Hi, I am using Python 2.5.1 I have an application that reads a file and generates a key in a dictionary for each line it reads. I have managed to read a 1GB file and generate more than 8 million keys on an Windows XP machine with only 1GB of memory and all works as expected. When I use the same program on a Windows 2003 Server with 2GB of RAM I start getting MemoryError exceptions! I have tried setting the IMAGE_FILE_LARGE_ADDRESS_AWARE on both Python.exe and Python25.dll and setting the /3GB flag on the boot.ini file to no avail. I still get the MemoryError exceptions. Has anybody encountered this problem before? Thanks in advance for any ideas/suggestions. Best Regards, André M. Descombes How are you reading the large files? IMO, large files are better read in chunks: target_file = open(f, 'rb') while 1: data = target_file.read(8192000) if data: DO SOMETHING else: break The above reads 8MB at a time until the file has been completely read. Change the 8MB to whatever you like. -- http://mail.python.org/mailman/listinfo/python-list
