Re: Sharing Data in Python
I have some Pickled data, which is stored on disk, and it is about 100 MB in size. When my python program is executed, the picked data is loaded using the cPickle module, and all that works fine. If I execute the python multiple times using python main.py for example, each python process will load the same data multiple times, which is the correct behaviour. How can I make it so, all new python process share this data, so it is only loaded a single time into memory? asked the same question on SO, but could not get any constructive responses.. http://stackoverflow.com/questions/10550870/sharing-data-in-python/10551845 If I was running this code on somewhat new hardware I'm not sure I would do anything. Your OS or disk subsystem, possibly both, are probably already caching this data. -- http://mail.python.org/mailman/listinfo/python-list
Re: Sharing Data in Python
raunakgu...@gmail.com wrote: I have some Pickled data, which is stored on disk, and it is about 100 MB in size. When my python program is executed, the picked data is loaded using the cPickle module, and all that works fine. If I execute the python multiple times using python main.py for example, each python process will load the same data multiple times, which is the correct behaviour. How can I make it so, all new python process share this data, so it is only loaded a single time into memory? asked the same question on SO, but could not get any constructive responses.. http://stackoverflow.com/questions/10550870/sharing-data-in-python/10551845 Well a straightforward way of solving this is to use threads. Have your main program spawn threads for processing data. Now what triggers a thread is up to you, your main program can listen to commands, or catch keyboard events ... If you want to make sure your using all CPU cores, you may want to use the module http://docs.python.org/library/multiprocessing.html It emulates threads with subprocesses, and features a way to share memory. I do tend to prefer processes over threads, I find them easier to monitor and control. Cheers, JM -- http://mail.python.org/mailman/listinfo/python-list
Sharing Data in Python
I have some Pickled data, which is stored on disk, and it is about 100 MB in size. When my python program is executed, the picked data is loaded using the cPickle module, and all that works fine. If I execute the python multiple times using python main.py for example, each python process will load the same data multiple times, which is the correct behaviour. How can I make it so, all new python process share this data, so it is only loaded a single time into memory? asked the same question on SO, but could not get any constructive responses.. http://stackoverflow.com/questions/10550870/sharing-data-in-python/10551845 -- http://mail.python.org/mailman/listinfo/python-list
Re: Sharing Data in Python
How can I make it so, all new python process share this data, so it is only loaded a single time into memory? You can have one process as server and client ask for parts of data. You might be able to do something smart with mmap but I can't think of a way. I Linux systems, if you first load the data and then fork, the OS will keep all the read only data shared. -- http://mail.python.org/mailman/listinfo/python-list
