Re: [gdal-dev] Call for discussion on RFC 45: GDAL datasets and raster bands as virtual memory mappings

2013-12-19 Thread Even Rouault
Le jeudi 19 décembre 2013 04:41:18, Trent Piepho a écrit : Do you see page file activity? If you look at /proc/pid/smaps, you should be able to see the actual status of the mapping of your data file. Probably it is consuming a large number of pages of RAM, but also there should be zero pages

Re: [gdal-dev] Call for discussion on RFC 45: GDAL datasets and raster bands as virtual memory mappings

2013-12-18 Thread Even Rouault
Le mercredi 18 décembre 2013 06:55:50, Frank Warmerdam a écrit : Even, Very impressive work, I am supportive. IMHO it would be wonderful if there was also an mmap() based mechanism where you could ask for the virtual memory chunk and you get it back (if it works) along with stride values

Re: [gdal-dev] Call for discussion on RFC 45: GDAL datasets and raster bands as virtual memory mappings

2013-12-18 Thread Frank Warmerdam
Even, Sorry, I was thinking of mmap() directly to the file, and having something like: CPLVirtualMem CPL_DLL* GDALBandGetVirtualMemAuto( GDALRasterBandH hBand, int *pnPixelSpace, GIntBig *pnLineSpace,

Re: [gdal-dev] Call for discussion on RFC 45: GDAL datasets and raster bands as virtual memory mappings

2013-12-18 Thread Even Rouault
Le mercredi 18 décembre 2013 19:53:37, Frank Warmerdam a écrit : Even, Sorry, I was thinking of mmap() directly to the file, and having something like: CPLVirtualMem CPL_DLL* GDALBandGetVirtualMemAuto( GDALRasterBandH hBand, int *pnPixelSpace,

Re: [gdal-dev] Call for discussion on RFC 45: GDAL datasets and raster bands as virtual memory mappings

2013-12-18 Thread Frank Warmerdam
On Wed, Dec 18, 2013 at 11:46 AM, Even Rouault even.roua...@mines-paris.org wrote: Le mercredi 18 décembre 2013 19:53:37, Frank Warmerdam a écrit : Even, Sorry, I was thinking of mmap() directly to the file, and having something like: CPLVirtualMem CPL_DLL*

Re: [gdal-dev] Call for discussion on RFC 45: GDAL datasets and raster bands as virtual memory mappings

2013-12-18 Thread Frank Warmerdam
On Wed, Dec 18, 2013 at 11:46 AM, Even Rouault even.roua...@mines-paris.org wrote: I'm wondering if there would be ways of making what you propose work with Python Numpy in such a way that a numpy array could be requested which is of this virtual memory. That would also be a nice

Re: [gdal-dev] Call for discussion on RFC 45: GDAL datasets and raster bands as virtual memory mappings

2013-12-18 Thread Trent Piepho
On Wed, Dec 18, 2013 at 11:46 AM, Even Rouault even.roua...@mines-paris.org wrote: Le mercredi 18 décembre 2013 19:53:37, Frank Warmerdam a écrit : I imagined an available virtual method on the band which could be implemented - primarily by the RawBand class to try and mmap() the data and

Re: [gdal-dev] Call for discussion on RFC 45: GDAL datasets and raster bands as virtual memory mappings

2013-12-18 Thread Even Rouault
Le mercredi 18 décembre 2013 21:09:48, Trent Piepho a écrit : On Wed, Dec 18, 2013 at 11:46 AM, Even Rouault even.roua...@mines-paris.org wrote: Le mercredi 18 décembre 2013 19:53:37, Frank Warmerdam a écrit : I imagined an available virtual method on the band which could be implemented

Re: [gdal-dev] Call for discussion on RFC 45: GDAL datasets and raster bands as virtual memory mappings

2013-12-18 Thread Trent Piepho
Do you see page file activity? If you look at /proc/pid/smaps, you should be able to see the actual status of the mapping of your data file. Probably it is consuming a large number of pages of RAM, but also there should be zero pages written to swap. All clean private or clean shared, zero

[gdal-dev] Call for discussion on RFC 45: GDAL datasets and raster bands as virtual memory mappings

2013-12-17 Thread Even Rouault
Hi, This is a call for discussion for RFC 45: GDAL datasets and raster bands as virtual memory mappings Beginning of the RFC inline (the full RFC includes a few colorful schemas !) : == Summary == This document proposes additions to GDAL so that image data of GDAL datasets and raster bands

Re: [gdal-dev] Call for discussion on RFC 45: GDAL datasets and raster bands as virtual memory mappings

2013-12-17 Thread Even Rouault
Le mardi 17 décembre 2013 21:54:31, Even Rouault a écrit : Hi, This is a call for discussion for RFC 45: GDAL datasets and raster bands as virtual memory mappings Here's the link to the RFC : http://trac.osgeo.org/gdal/wiki/rfc45_virtualmem Even -- Geospatial professional services

Re: [gdal-dev] Call for discussion on RFC 45: GDAL datasets and raster bands as virtual memory mappings

2013-12-17 Thread Frank Warmerdam
Even, Very impressive work, I am supportive. IMHO it would be wonderful if there was also an mmap() based mechanism where you could ask for the virtual memory chunk and you get it back (if it works) along with stride values to access in it. This could likely be made to work for most raw based