Re: Multiprocessing and memory management

2019-07-04 Thread Thomas Jollans
On 03/07/2019 18.37, Israel Brewster wrote: > I have a script that benefits greatly from multiprocessing (it’s generating a > bunch of images from data). Of course, as expected each process uses a chunk > of memory, and the more processes there are, the more memory used. The amount > used per

Re: Multiprocessing and memory management

2019-07-03 Thread Peter J. Holzer
On 2019-07-03 08:37:50 -0800, Israel Brewster wrote: > 1) Determine the total amount of RAM in the machine (how?), assume an > average of 10GB per process, and only launch as many processes as > calculated to fit. Easy, but would run the risk of under-utilizing the > processing capabilities and

Re: Multiprocessing and memory management

2019-07-03 Thread Gary Herron
On 7/3/19 9:37 AM, ijbrews...@alaska.edu wrote: I have a script that benefits greatly from multiprocessing (it’s generating a bunch of images from data). Of course, as expected each process uses a chunk of memory, and the more processes there are, the more memory used. The amount used per

Multiprocessing and memory management

2019-07-03 Thread Israel Brewster
I have a script that benefits greatly from multiprocessing (it’s generating a bunch of images from data). Of course, as expected each process uses a chunk of memory, and the more processes there are, the more memory used. The amount used per process can vary from around 3 GB (yes, gigabytes) to