> It seems that using 64 bit python is the solution. It's certainly the easy way to access a lot of memory -- and memory is cheap these days.
> But the thing is i would > compile my code and wanna distribute it to the clients.. I don't think 64 bit gets in the way of that -- except that it will only runon 64 bit systems, which may be an issue. > only reason why i want to work on 32 bit system. Sturla, how I can make it > sure that some part of the data is kept on the disk and only the necessary > one in the memory; as this seems to be a solution to my problem. You can "roll your own" and have a disk cache of some sort -- it would be pretty easy to store each image in a *.npz file and load the up as you need them. But it would probably be even easier to use one of the hdf-based libarries, such as pytables -- I htink it will do it all for you. One other option, that I've never tried, is carray, which is an array compressed in memory. Depending on your images, perhaps they would compress a lot (or not ....): https://github.com/FrancescAlted/carray http://mail.scipy.org/pipermail/numpy-discussion/2010-August/052378.html > As i said i > want a 3d visualization out of the numpy array. it works fine for the > downsampled dataset. And to visualize, i have to convert the 16bit data into > 8bit as PIL doesnt support 16 bit data. It's unclear to me what you native data really is: 16 bitgreyscale? 8bit greyscale? either one should fit OK into 32 bit memory, and if 8bit is accurate enough foryour needs, then it should be pretty easy. > stack = numpy.empty((120, 1024, 1024)) numpy defaults to double precision float, np.float64, i.e. 8 bytes per element -- you probably don't want that if you are concerned about memoery use, and have 8 or 16 bit greyscale images. Try: stack = np.empty((120, 1024, 1024), dtype=np.uint8) # (or dtype=np.uint16) > i = 0 > > os.chdir(dirr) > for f in os.listdir(dirr): > > im = Image.open(f) > im = im.convert("L") you ight want to try mode "I". That should give you a 32 bit integer grey scale, which should hold all the 16 bit data without loss -- then you can convert to 16 bit when you bring it into numpy. > a = numpy.asarray(im) > print a.dtype what does this print? it should be np.uint8 > stack[i] = a here, if a is a uint8, numpy will convert it to a float64, to fit into array a -- that's why you want to set the dtype of array a when you create it. > one more thing, it really doesnt work for tiff files at all, i have to > convert them into jpgs as a prior step to this. probably not the best choice either, jpeg is lossy -- is will smear things out a bit, which you may not want. It also only holds 24 bit RGB (I think), which is both a waste and will lose information from 16 bit greyscale. If you have to convert, try PNG, though I'm not sure if it handles 16 bit greyscale, either. I'd look at a lib that can read tiff properly -- some have been suggested here, and you can also use GDAL, which is meant for geo-referenced data, but you can ignore the geo information and just get an image if you want. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion