Re: [gdal-dev] Using SWIG from a Driver
Hi, I think it is possibile using the swig's director feature: Calling back non c/c++ from c/c++ code. Basically you have to develop your driver as a C++ class declared in a swig interface file (.i) declaring virtual functions that have to implements the features of your driver. This driver should be viewed as an abstract driver. The concrete driver is implemented by a swig generated class that calls you java code implementing the virtual functions you declared in you .i file. Regards, Stefano Dr.Eng. Stefano Moratto stefano.mora...@gmail.com stefano.mora...@csiat.it http://www.csiat.it - Traffic Optimization Software On Wed, Feb 22, 2012 at 4:13 PM, Even Rouault even.roua...@mines-paris.orgwrote: Selon Michael Speth spe...@landcareresearch.co.nz: Greetings GDAL devs, I am interested in developing a GDAL driver that calls out to a Java library that accesses a database that we are developing using SWIG's Java Bindings. Are there any examples of existing GDAL drivers using the SWIG bindings? Not that I am aware of, and not in GDAL source tree for sure. If not, how might I use the SWIG library from a new GDAL driver? I've noticed there are 2 usages of Java from a GDAL driver. Frank pointed out to me (ogr/ogrsf_frmts/ili) and mdb (ogr/ogrsf_frmts/mdb). Both of these driver's use JNI. The reason I want to use SWIG is to avoid having to deal with the added configuration of JNI (since SWIG handles that already). I'm not sure if you can call Java code with SWIG. Perhaps, but that's perhaps more a question for a SWIG mailing list. I don't see how the code of the GDAL SWIG bindings could help you since you will use a dedicated API for your Java library. Thanks! -- Michael Speth Research Systems Developer Landcare Research ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] RasterIO in paralel
Thanks for the suggestion but I'm developing a Win32 app. I will take in consideration your previous answer because the app can read images from other sources. Regards, Stefano -- Dr.Eng. Stefano Moratto stefano.mora...@gmail.com stefano.mora...@csiat.it http://www.csiat.it - Traffic Optimization Software On Tue, Feb 8, 2011 at 11:38 PM, Even Rouault even.roua...@mines-paris.orgwrote: Le mardi 08 février 2011 23:33:39, Stefano Moratto a écrit : The openstreet data is a virtual image composed by several tiles at a given zoom level, each tiles is a 256x256 png, so I think if a thead read a 256x256 block i will access only a given png. Yes of course reading different images by different threads will work. If you use GDAL on Unix, be sure that it is built with the --with-threads option, which had to be explicitely passed before GDAL 1.8.0 and is now the default. My previous answer only applies if you divide the reading of a single PNG image into several threads. On Tue, Feb 8, 2011 at 11:14 PM, Even Rouault even.roua...@mines-paris.orgwrote: Le mardi 08 février 2011 23:07:36, Stefano Moratto a écrit : Thanks for the suggestion, I have to read png from openstreetmap. I'm trying to render an A4 paper size sheet so I have a lot of data to move. It may be better to divide the area to be read in blocks and to give to each thread a block. Foreach block a thread should read all the bands. Probabilly I should allign the block to the tile's boundary Unfortunately the PNG format is such that it is not possible to seek to an arbitrary line without decoding the data of all previous scanlines, so this approach won't work. The same would be true for JPEG or GIF. If you can fetch the data as TIFF, you could however try that approach. Stefano Le mardi 08 février 2011 22:41:50, Stefano Moratto a écrit : Hello, given an opened GDAL dataset, is it possibile to perform gdalrasterio calls for each band in a dedicated thread per band in a thead-safe way? E.g. I have a raster image with 3 band (r,g,b). Since I have 4 core I would like to I run a thread for a band r, one for band g and one for band g. This has been discussed recently. See http://lists.osgeo.org/pipermail/gdal- dev/2011-January/027567.html and the following messages. (I'd note that if you process a pixel interleaved image, there are optimizations in the GTiff driver to make the fetching of other bands almost 'free' if you read block by block, so there's probably little to gain, but the results of your experiments are welcome) Regards, Stefano ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
[gdal-dev] RasterIO in paralel
Hello, given an opened GDAL dataset, is it possibile to perform gdalrasterio calls for each band in a dedicated thread per band in a thead-safe way? E.g. I have a raster image with 3 band (r,g,b). Since I have 4 core I would like to I run a thread for a band r, one for band g and one for band g. Regards, Stefano -- Dr.Eng. Stefano Moratto stefano.mora...@gmail.com stefano.mora...@csiat.it http://www.csiat.it - Traffic Optimization Software ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Interrupt long operation
Frank, thank you for your suggestion. I added a GDALProgressFunc as you suggested. In order to prevent blocking in the rasterio now I check a state variable indicating when I have to stop the reading. Since I call rasterio foreach line of the destination bitmap I have added a simple check of the state variable (eg.: if (terminate) then return). Very simple!! I should have think about it by myself. Now my app can browse without blocking the openstreetmap dataset. Best regards. Stefano - Dr.Eng. Stefano Moratto stefano.mora...@gmail.com stefano.mora...@csiat.it http://www.csiat.it - Traffic Optimization Software On Mon, Jan 31, 2011 at 5:39 PM, Frank Warmerdam warmer...@pobox.comwrote: On 11-01-31 04:26 AM, Stefano Moratto wrote: Hello, I need to interrupt a RasterIO and warp calls before they have been ended. The call is invoked in a background thread and It may happen that the user changes the requested area in the main thread (GUI) (eg. a pan/scroll operation). Stefano, Algorithms which take a GDALProgressFunc are likely to be interruptable. Each time the progress function is invoked it has the opportunity to return an indicate that the operation has been cancelled by the user. So the warper can be canceled if you provide a customized progress function. But RasterIO cannot since it has no progress function. If you want IO to be interruptible you would normally do it in modest sized chunks, though this can interfere with optimization of the IO in some cases. Best regards, -- ---+-- I set the clouds in motion - turn up | Frank Warmerdam, warmer...@pobox.com light and sound - activate the windows | http://pobox.com/~warmerdamhttp://pobox.com/%7Ewarmerdam and watch the world go round - Rush| Geospatial Programmer for Rent ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev - ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
[gdal-dev] Interrupt long operation
Hello, I need to interrupt a RasterIO and warp calls before they have been ended. The call is invoked in a background thread and It may happen that the user changes the requested area in the main thread (GUI) (eg. a pan/scroll operation). Any suggestion, Stefano Dr.Eng. Stefano Moratto stefano.mora...@gmail.com stefano.mora...@csiat.it http://www.csiat.it - Traffic Optimization Software ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] GDAL Speed Optimization
Frank, I have increased the cache as you suggested using GDALSetCacheMax and it produced a performance gain. Thanks. I think I can improve more acting on the virtual datataset that I use to adapt the image to the paper size. I have to look into the tms driver in order to understand how it download the tiles to compose the final image. Stefano On Fri, Jun 11, 2010 at 6:17 PM, Frank Warmerdam warmer...@pobox.comwrote: Stefano Moratto wrote: Frank, The source images are a sequence of tiles from openstreetmaps. Every tiles has a fixed size (256 x 256) and the area they covers depends from the zoom level. It is explained very well in http://www.maptiler.org/google-maps-coordinates-tile-bounds-projection. I use interpolation (via a virtual dataset) because my window size and the paper size are not a multiple of 256 x 256. I use reprojection because I've to mix this raster datasource with a vector datasource that has a different projection. Reprojecting my vector data to openstreet projection do not affect the performance. So I think the key factor is the resizing ( I use bilinear interpolation) and the size of the bitmap that may cause paging. ... Can you suggest some other parameters? Stefano, Sorry, I wasn't able to learn much more skimming the code and referenced web page. Have you tried the cache size setting I suggested? Best regards, ---+-- I set the clouds in motion - turn up | Frank Warmerdam, warmer...@pobox.com light and sound - activate the windows | http://pobox.com/~warmerdamhttp://pobox.com/%7Ewarmerdam and watch the world go round - Rush| Geospatial Programmer for Rent -- Dr.Eng. Stefano Moratto stefano.mora...@gmail.com stefano.mora...@csiat.it http://www.csiat.it - Traffic Optimization Software ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] GDAL Speed Optimization
Iván, I evaluated the use of mapnik but I develop on Windows and setup it on that platform is not easy as on Unix/Linux. Also I develop in delphi, so I have do create the bindings by myself I would like to use mapnik in my project as a replacement of my custom render but I have to transform the openstreet data in my own format (I use a relational version of the GDF) because I want not to add another db server. I should have to had a new datasource for mapnik and in order to do that I must be able to recompile it ;-) My application is a desktop application and my customer have to produce some reports for the local area government in order to plan alternative path in case of hazard in the highway. The deployment term in near so I have no time to try the direct use of mapnk. The actual solution works but it is not rapid as I would like ( I have to wait for 30 seconds in order to get an a3 bitmap). Have you experience of mapnik on windows? Regards, Stefano 2010/6/11 Iván Sánchez Ortega i...@sanchezortega.es El día Thursday 10 June 2010 09:49:01, Stefano Moratto dijo: I use GDAL to download tiles (jpeg) from OpenstreetMap and to compose a bitmap of the area that is being viewed. The resulting bitmap (not compressed) is warped and displayed. Hhhhmmm. Personally, I think you're doing it wrong. If you really need speed for custom reprojections of OSM data, the best thing to do is to render the data locally with Mapnik. i.e.: - Download a planet file - Put it in PostGIS via osm2pgsql - Configure Mapnik to use the projection you need - Use Mapnik to get the desired area rendered You might incur in a speed penalty due to the rendering itself, but you guarantee that there will be no artifacts due to the image warping. And also, you might get a boost by reprojecting the vector data inside PostGIS so Mapnik doesn't have to reproject it on-the-fly. I realise that setting Mapnik is quite a lot of work at first, but it's an idea that is worth considering in your case. Cheers, -- Iván Sánchez Ortega i...@sanchezortega.es Un ordenador no es una televisión ni un microondas: es una herramienta compleja. ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev -- Dr.Eng. Stefano Moratto stefano.mora...@gmail.com stefano.mora...@csiat.it http://www.csiat.it - Traffic Optimization Software ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] GDAL Speed Optimization
Frank, The source images are a sequence of tiles from openstreetmaps. Every tiles has a fixed size (256 x 256) and the area they covers depends from the zoom level. It is explained very well in http://www.maptiler.org/google-maps-coordinates-tile-bounds-projection. I use interpolation (via a virtual dataset) because my window size and the paper size are not a multiple of 256 x 256. I use reprojection because I've to mix this raster datasource with a vector datasource that has a different projection. Reprojecting my vector data to openstreet projection do not affect the performance. So I think the key factor is the resizing ( I use bilinear interpolation) and the size of the bitmap that may cause paging. I use the following : hWarp := GDALAutoCreateWarpedVRT (hSized, nil, PChar(FMapSRSWkt), GDALResampleAlg_GRA_NearestNeighbour, 0.5, nil); where hSized is a hSized = F (hDataset) ; F (hDataset) := Virtual Dataset of hDataset. I use it for in memory resizing . It is generated by: (I do not write all the details but I think it is clear) VRTAddSimpleSource (poVRTBand, poSrcBand, Round(anSrcWin[0]), Round(anSrcWin[1]), Round(anSrcWin[2]), Round(anSrcWin[3]), 0, 0, size.cx, size.cy,'Bilinear', 0.0 ); and hDataset = GDALOpen( ... openstreet.xml ) Can you suggest some other parameters? Thanks, Stefano On Thu, Jun 10, 2010 at 4:53 PM, Frank Warmerdam warmer...@pobox.comwrote: Stefano Moratto wrote: The performances are quite acceptable but when I try to print they are not. The drawing to be printed has a resolution larger of the screen (Screen : 1024x1024, Printer 4096 x 4094 in A4 and 9000 x 9000 in A3 approximately). I think tha the bottleneck could be found in: 1) jpeg decompression. 2) bitmap interpolation (I use the low qualitiy settings). 3) warping ( I use an approsimated warping function). An increase of performance could be achieved using 1) hw accelerated functions as IPP or GPU (e.g CUDA) 2) parallel alghorithms that takes advantage from multicore CPU Has someone already approched these problems? Stefano, Are the same number of jpeg tiles being used when producing print products as on screen? I'm assuming the answer is yes, and that the warper is just interpolating a higher resolution result from them. If the answer is yes, then presumably the time decoding the jpegs would not be greater for the prints than for screen display, though it is possible that more re-reading is occuring. If re-reading is an issue you might want to increase the size of the GDAL block cache. Programmatically this can be done with the C function GDALSetCacheMax(). Try setting it to 100 (for 100MB). Are you using the warper just to interpolate to different sizes or are you also applying reprojection or some other non-linear warping? If you are just using it to resize the image then I would suggest using something less general-purpose than the warper to do the interpolation. Seth's CUDA work may also be of interest of course, but before committing to that path I would suggest some analysis of where the time is really being spent. Best regards, -- ---+-- I set the clouds in motion - turn up | Frank Warmerdam, warmer...@pobox.com light and sound - activate the windows | http://pobox.com/~warmerdamhttp://pobox.com/%7Ewarmerdam and watch the world go round - Rush| Geospatial Programmer for Rent -- Dr.Eng. Stefano Moratto stefano.mora...@gmail.com stefano.mora...@csiat.it http://www.csiat.it - Traffic Optimization Software ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
[gdal-dev] GDAL Speed Optimization
I use GDAL in my traffic optimization CAD program. It is a Win32 application written in DELPHI that uses GDAL C API - the binding was autogenerated by my SWIG module for object pascal. I use GDAL to download tiles (jpeg) from OpenstreetMap and to compose a bitmap of the area that is being viewed. The resulting bitmap (not compressed) is warped and displayed. The performances are quite acceptable but when I try to print they are not. The drawing to be printed has a resolution larger of the screen (Screen : 1024x1024, Printer 4096 x 4094 in A4 and 9000 x 9000 in A3 approximately). I think tha the bottleneck could be found in: 1) jpeg decompression. 2) bitmap interpolation (I use the low qualitiy settings). 3) warping ( I use an approsimated warping function). An increase of performance could be achieved using 1) hw accelerated functions as IPP or GPU (e.g CUDA) 2) parallel alghorithms that takes advantage from multicore CPU Has someone already approched these problems? Regards, Stefano -- Dr.Eng. Stefano Moratto stefano.mora...@gmail.com stefano.mora...@csiat.it http://www.csiat.it - Traffic Optimization Software ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: Re: [gdal-dev] GDAL Speed Optimization
I'm working on a VirtualMachine running on an usb hdd and my customer use iMac laptop so I can't change disk. The primary dataset is composed by a lot of jpeg download from internet and then cached to disk. I think the performance slowdown is caused after the creation of the virtual dataset because and it is present only when the output bitmap has the resolution of the printer (considering alwarys the sampre geographic area). All the output bitmaps, exept the dataset's jpeg, are in memory and the workingset of the application does not exeed the 256 MB Can you suggest a profiler? Thanks, Stefano Il giorno 10/giu/2010 10.31, Seth Price s...@pricepages.org ha scritto: Unfortunately, both nearest neighbor and bilinear are probably not CPU bound, so speeding up their processing won't help. They are probably I/O bound, so you might need faster disks. (Though I can test this later in my project.) Have you tried running a profiling tool on GDAL while you're running the warper? I've just started the OpenCL project, so it's still pretty immature. I'm still discussing the best way to integrate an OpenCL warper into existing code. ~Seth On Thu, June 10, 2010 2:26 am, Stefano Moratto wrote: Seth, You have been choosen a very interesting project. I use the following : hWarp := GDALAutoCreateWarpedVRT (hSized, nil, PChar(FMapSRSWkt), GDALResampleAlg_GRA_NearestNeighbour, 0.5, nil); where hSized is a hSized = F (hDataset) ; F (hDataset) := Virtual Dataset of hDataset. I use it for in memory resizing . It is generated by: (I do not write all the details but I think it is clear) VRTAddSimpleSource (poVRTBand, poSrcBand, Round(anSrcWin[0]), Round(anSrcWin[1]), Round(anSrcWin[2]), Round(anSrcWin[3]), 0, 0, size.cx, size.cy,'Bilinear', 0.0 ); and hDataset = GDALOpen( ... openstreet.xml ) Hence I use 1) ,'Bilinear' for zooming 2) NearestNeighbour for warping. What is the state of your project? Stefano On Thu, Jun 10, 2010 at 9:52 AM, Seth Price s...@pricepages.org wrote: 1) hw accelerated functions as IPP or GPU (eg CUDA) This is my (ongoing) Google Summer of Code project, except I'm using OpenCL. :D What resampling algorithm are you using? ~Seth On Thu, June 10, 2010 1:49 am, Stefano Moratto wrote: I use GDAL in my traffic optimization CAD program. It is a Win32 application written in DELPHI that uses GDAL C API - the binding was autogenerated by my SWIG module for object pascal. I use GDAL to download tiles (jpeg) from OpenstreetMap and to compose a bitmap of the area that is being viewed. The resulting bitmap (not compressed) is warped and displayed. The performances are quite acceptable but when I try to print they are not. The drawing to be printed has a resolution larger of the screen (Screen : 1024x1024, Printer 4096 x 4094 in A4 and 9000 x 9000 in A3 approximately). I think tha the bottleneck could be found in: 1) jpeg decompression. 2) bitmap interpolation (I use the low qualitiy settings). 3) warping ( I use an approsimated warping function). An increase of performance could be achieved using 1) hw accelerated functions as IPP or GPU (eg CUDA) 2) parallel alghorithms that takes advantage from multicore CPU Has someone already approched these problems? Regards, Stefano -- Dr.Eng. Stefano Moratto stefano.mora...@gmail.com stefano.mora...@csiat.it http://www.csiat.it - Traffic Optimization Software ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev -- Dr.Eng. Stefano Moratto stefano.mora...@gmail.com stefano.mora...@csiat.it http://www.csiat.it - Traffic Optimization Software ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
