A Tuesday 01 February 2011 19:58:16 Sturla Molden escrigué: > Den 01.02.2011 18:58, skrev Christopher Barker: > > But if you really have big collections of images, you might try > > memory mapped arrays -- as Sturla pointed out they wont' let you > > create monster arrays on a 32 bit python, > > But they will on 64 bit Python :D We can just memory map a temporary > file, and fake as much "memory" as we need. The OS will optimize the > disk access.
What do you mean by 'optimize the disk access'? One of the drawbacks of memory mapped files is precisely that OS cannot distinguish between data that belongs to 'disk' and data that belongs to 'memory'. This normally introduces extreme slowness in other programs when datasets in files exceeds physical memory but have to loaded by OS -- the reason being that OS swaps out most of programs/shared libraries that were in memory in order to be able to load new 'disk' data. The other important drawback of the memory mapped files is that you need to have at very least an amount of virtual memory that is enough to keep all of these data files. In general, you only have virtual memory that is between 1.5x and 2x the physical memory (having more than this is generally regarded as a waste of disk space). This is why I prefer very much reading directly from a file: the OS in this case is able to distinguish between data belonging to 'disk' and data belonging to 'memory'. Is in this case when the OS can really optimize disk access (unless you have complicated setups). > Also consider that hardware is cheap compared to labour, > at least in Europe. Is programming for memory limited 32 bit Python > worth the effort? -- Francesc Alted _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
